| Specifies the number of hyperparameter sampling repetitions. Ensures every experiment is repeated a given number of times. It should be an integer number higher than one. Set `-1` to never see repetitions.

| Int (default=-1)

| `MAX_ITERATIONS`

| Specifies the number of maximum iterations. It should be an integer number higher than zero. Set `-1` for an infinite loop.

| Int (default=2)

| `PARALLEL_EXECUTIONS_PER_ITERATION`

| Specifies the number of parallel executions per iteration. It should be an integer number higher than zero.

| Int (default=2)

| `NUMBER_OF_REPETITIONS`

| Specifies the number of hyperparameter sampling repetitions. Ensures every experiment is repeated a given number of times. It should be an integer number higher than one. Set `-1` to never see repetitions.

| Int (default=-1)

| `PAUSE_AFTER_EVERY_ITERATIONS`

| If higher than zero, pause the workflow after every specified number of iterations. Set `-1` to disable.

| Int (default=-1)

| `TARGET_WORKFLOW`

| Specifies the workflow path from the catalog that should be optimized.

| If True, the selected container platform (docker, singularity, ...) will propagate to the workflow that should be optimized.

| (default=True)

| `TENSORBOARD_PROXYFIED`

| If True, requests to TensorBoard are sent via a proxy server.

| Boolean (default=False)

|===

image::AutoML_Full.png[align=center]

...

...

@@ -273,24 +276,18 @@ The choice of the tuner depends on the following aspects:

In the following, we briefly describe the different tuners proposed by the `Distributed_Auto_ML` workflow:

- *Grid sampling* applies when all variables are discrete, and the number of possibilities is low.

A grid search is a naive approach that will simply try all possibilities making the search extremely long even for medium-sized problems.

- *Grid sampling* applies when all variables are discrete, and the number of possibilities is low. A grid search is a naive approach that will simply try all possibilities making the search extremely long even for medium-sized problems.

- *Random sampling* is an alternative to grid search when the number of discrete parameters to optimize, and the time required for each evaluation is high.

Random search picks the point randomly from the configuration space.

- *Random sampling* is an alternative to grid search when the number of discrete parameters to optimize, and the time required for each evaluation is high. Random search picks the point randomly from the configuration space.

- *QuasiRandom sampling* ensures a much more uniform exploration of the search space than traditional pseudo random.

Thus, quasi random sampling is preferable when not all variables are discrete, the number of dimensions is high, and the time required to evaluate a solution is high.

- *QuasiRandom sampling* ensures a much more uniform exploration of the search space than traditional pseudo random. Thus, quasi random sampling is preferable when not all variables are discrete, the number of dimensions is high, and the time required to evaluate a solution is high.

- *Bayes search* models the search space using gaussian process regression, which allows an estimation of the loss function, and the uncertainty on that estimate at every point of the search space.

Modeling the search space suffers from the curse of dimensionality, which makes this method more suitable when the number of dimensions is low.

- *Bayes search* models the search space using gaussian process regression, which allows an estimation of the loss function, and the uncertainty on that estimate at every point of the search space. Modeling the search space suffers from the curse of dimensionality, which makes this method more suitable when the number of dimensions is low.

- *CMAES search* (Covariance Matrix Adaptation Evolution Strategy) is one of the most powerful black-box optimization algorithm.

However, it requires a significant number of model evaluation (in the order of 10 to 50 times the number of dimensions) to converge to an optimal solution.

This search method is more suitable when the time required for a model evaluation is relatively low.

However, it requires a significant number of model evaluation (in the order of 10 to 50 times the number of dimensions) to converge to an optimal solution. This search method is more suitable when the time required for a model evaluation is relatively low.

- *MOCMAES search* (Multi-Objective Covariance Matrix Adaptation Evolution Strategy) is a multi-objective algorithm optimizing multiple tradeoffs simultaneously.

To do that, MOCMAES employs a number of CMAES algorithms.

- *MOCMAES search* (Multi-Objective Covariance Matrix Adaptation Evolution Strategy) is a multi-objective algorithm optimizing multiple tradeoffs simultaneously. To do that, MOCMAES employs a number of CMAES algorithms.

Here is a table that resumes when to use each algorithm.

|===

...

...

@@ -336,7 +333,7 @@ Here is a table that resumes when to use each algorithm.

=== Objective Functions

The following workflows represent some mathematical functions that can be optimized by the AutoML tuners.

The following workflows represent some mathematical functions that can be optimized by the `Distributed_Auto_ML` tuners.

*Himmelblau_Function:* is a multi-modal function containing four identical local minima. It'susedtotesttheperformanceofoptimizationalgorithms.Formoreinfo,pleaseclickhttps://en.wikipedia.org/wiki/Himmelblau%27s_function[here].

...

...

@@ -382,7 +379,7 @@ The following workflows have common variables with the above illustrated workflo

*Text_Generation:*trainsasimpleLongShort-TermMemory(LSTM)tolearnsequencesofcharactersfrom'The Alchemist'book.It's a novel by Brazilian author Paulo Coelho that was first published in 1988.

...

...

@@ -391,8 +388,14 @@ The following workflows have common variables with the above illustrated workflo

The following workflows contain a search space containing a set of possible neural networks architectures that can be used by `Distributed_Auto_ML` to automatically find the best combinations of neural architectures within the search space.

*Handwritten_Digit_Classification:* train a simple deep CNN on the MNIST dataset using the PyTorch library.

*Handwritten_Digit_Classification:* trains a simple deep CNN on the MNIST dataset using the PyTorch library.

=== Distributed Training

The following workflows illustrate some examples of multi-node and multi-gpu distributed learning.

*TensorFlow_Keras_Multi_Node_Multi_GPU:* is a TensorFlow + Keras workflow template for distributed training (multi-node multi-gpu) with AutoML support.