Commit bfa0c615 authored by maciek riedl's avatar maciek riedl
Browse files

Merge branch 'exponential_smoothing_predictor' into 'morphemic-rc2.0'

Miscellaneous corrections and improvements

See merge request !292
parents 4cb7ef9e 8b7728ec
Pipeline #20889 failed with stages
in 27 minutes and 36 seconds
......@@ -66,4 +66,4 @@ COPY ./src/r_predictors/forecasting_real_workload.R /home/r_predictions/
WORKDIR /home/r_predictions/esm_forecaster-0.1.0
CMD ["/bin/sh","-c","python3 /home/r_predictions/esm_forecaster-0.1.0/runtime/Predictor.py /home/r_predictions/esm_forecaster-0.1.0/r_predictors/prediction_configuration.properties > $LOG_FILE 2>&1"]
\ No newline at end of file
CMD ["/bin/sh","-c","python3 -u /home/r_predictions/esm_forecaster-0.1.0/runtime/Predictor.py /home/r_predictions/esm_forecaster-0.1.0/r_predictors/prediction_configuration.properties 2>&1 > $LOG_FILE "]
\ No newline at end of file
......@@ -6,11 +6,15 @@ The exponential smoothing predictor is based on the use of the Holt-Winters [1]
Apart from standard R and Python libraries, the libraries included in the src/requirements.txt file should be available for the Python code to be successfully executed. Moreover the `rapportools`,`gbutils`,`forecast`,`ggplot2`,`properties`,`xts`,`anytime` and `purrr` R libraries should be available (included in the src/r_predictors/r_commands.R file).
[1] https://www.rdocumentation.org/packages/forecast/versions/8.15/topics/forecast.HoltWinters
[2] https://www.rdocumentation.org/packages/forecast/versions/8.15/topics/ets
## Configuration
### Configuration file
The predictor comes with two configuration files which can be used to specify the behaviour of the component. The two files are located in the src/r_predictors directory of the project.
The options which will most probably need to be changed before deployment are the `broker_address`,the `horizon`, the `number_of_days_to_aggregate_data_from` and the `number_of_seconds_to_aggregate_on`.
The options which will most probably need to be changed before deployment are the `broker_address`,the `horizon`, the `number_of_days_to_aggregate_data_from` and the `number_of_seconds_to_aggregate_on`.
It should be noted that the `number_of_seconds_to_aggregate_on` variable is updated at runtime to become the minimum between the configuration value and the horizon value received for the predictions (i.e the prediction interval)
| Option | Description |
-------- |------------ |
......@@ -25,7 +29,7 @@ The options which will most probably need to be changed before deployment are th
|horizon| The number of seconds which should be forecasted into the future|
|path_to_datasets|The absolute path to the datasets, **not** including the final slash ('/') character.|
|application_name|The application name to be used when creating a dataset|
|number_of_seconds_to_aggregate_on| The duration of the monitoring data interval in seconds. Monitoring data inside this interval will be aggregated to a single value (the mean value of all per-second observated/interpolated values) |
|number_of_seconds_to_aggregate_on| The duration of the monitoring data interval in seconds (greater than or equal to one). Monitoring data inside this interval will be aggregated to a single value (the mean value of all per-second observated/interpolated values) |
|number_of_days_to_aggregate_data_from| The number of days which will be used to retrieve monitoring data from when creating a dataset|
|prediction_processing_time_safety_margin_seconds| The number of seconds which will be used as a buffer when performing a prediction in order not to delay predictions and yet use as much data as possible|
|testing_prediction_functionality| If set to 'True', then it will not send a prediction time for which predictions will be requested, but will rather allow the horizon setting to be used to create predictions|
......@@ -70,7 +74,7 @@ docker run <container_name>
### Test execution
To quickly test the functionality of the forecaster, assuming that the EMS (or an ActiveMQ broker) has been (or soon will be) setup and is accessible, that the persistence storage module is available, and that the 'latency' and 'memory' metrics are being published to it, the following commands can be issued in order - provided that the broker-client.jar file is available.
To quickly test the functionality of the forecaster, assuming that the EMS (or an ActiveMQ broker) has been (or will soon be) setup and accessible, that the persistence storage module is available, and that the 'latency' and 'memory' metrics are being published to it, the following commands can be issued in order - provided that the broker-client.jar file is available.
1) Publish metrics to predict:
java -jar broker-client.jar publish3 -Umorphemic -Pmorphemic tcp://localhost:61616 metrics_to_predict [{"metric":"latency","level":3,"publish_rate":10000},{"metric":"memory","level":3,"publish_rate":10000}]
......
......@@ -31,8 +31,9 @@ find_last_timestamp <- function(mydata,next_prediction_time,realtime_mode){
if(realtime_mode){
#return(min(c(possible_timestamp,next_prediction_time)))
if (next_prediction_time>possible_timestamp){
return(next_prediction_time)
return(possible_timestamp)
}else{
print("Possible problem with the requested prediction time, there is already data for a timestamp newer than the time requested to predict for. Returning the newer timestamp, being aware that this will lead to this prediction returning no meaningful output")
return (possible_timestamp)
}
}else{
......@@ -53,6 +54,8 @@ time_unit_granularity <- "sec" # Handle monitoring data using this time unit gra
endpoint_time_unit_granularity <- "seconds"
#configuration_properties <- read.properties(".\\prediction_configuration-windows.properties")
print("Reading properties from the following file:")
print(paste(getwd(),"/prediction_configuration.properties",sep=''))
configuration_properties <- read.properties(paste(getwd(),"/prediction_configuration.properties",sep=''))
realtime_mode <- as.logical(configuration_properties$realtime_mode) #whether or not we should use all datapoints available (True value), or we are only partially using the available dataset (False value) e.g to test the prediction method performance
......
......@@ -42,14 +42,32 @@ def predict_attribute(attribute, configuration_file_location,next_prediction_tim
os.chdir(os.path.dirname(configuration_file_location))
prediction_data_file = get_prediction_data_filename(configuration_file_location)
from sys import platform
if State.testing_prediction_functionality:
print_with_time("Testing, so output will be based on the horizon setting from the properties file and the last timestamp in the data")
print_with_time("Issuing command: Rscript forecasting_real_workload.R "+str(prediction_data_file)+" "+attribute)
command = ['Rscript', 'forecasting_real_workload.R', prediction_data_file, attribute]
# Windows
if platform == "win32":
command = ['Rscript', 'forecasting_real_workload.R', prediction_data_file, attribute]
# linux
elif platform == "linux" or platform == "linux2":
command = ["Rscript forecasting_real_workload.R "+str(prediction_data_file) + " "+ str(attribute)]
#Choosing the solution of linux
else:
command = ["Rscript forecasting_real_workload.R "+str(prediction_data_file) + " "+ str(attribute)]
else:
print_with_time("Issuing command: Rscript forecasting_real_workload.R "+str(prediction_data_file)+" "+attribute+" "+next_prediction_time)
command = ['Rscript', 'forecasting_real_workload.R', prediction_data_file, attribute, next_prediction_time]
# Windows
if platform == "win32":
command = ['Rscript', 'forecasting_real_workload.R', prediction_data_file, attribute, next_prediction_time]
# Linux
elif platform == "linux" or platform == "linux2":
command = ["Rscript forecasting_real_workload.R "+str(prediction_data_file) + " "+ str(attribute)+" "+str(next_prediction_time)]
#Choosing the solution of linux
else:
command = ["Rscript forecasting_real_workload.R "+str(prediction_data_file) + " "+ str(attribute)+" "+str(next_prediction_time)]
process_output = run(command, shell=True, stdout=PIPE, stderr=PIPE, universal_newlines=True)
if (process_output.stdout==""):
......@@ -179,7 +197,7 @@ def calculate_and_publish_predictions(prediction_horizon,maximum_time_required_f
State.connection.send_to_topic('intermediate_prediction.%s.%s' % (id, attribute), prediction_message_body)
State.connection.send_to_topic('training_events',training_events_message_body)
message_not_sent = False
print_with_time("Successfully sent prediction message for "+attribute)
print_with_time("Successfully sent prediction message for "+attribute+" to topic intermediate_prediction.%s.%s" % (id, attribute))
except ConnectionError as exception:
State.connection.disconnect()
State.connection = messaging.morphemic.Connection('admin', 'admin')
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment