Skip to contents

OptimizerIrace class that implements iterated racing. Calls irace::irace() from package irace.

Source

Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43--58. doi:10.1016/j.orp.2016.09.002 .

Parameters

instances

list()
A list of instances where the configurations executed on.

targetRunnerParallel

function()
A function that executes the objective function with a specific parameter configuration and instance. A default function is provided, see section "Target Runner and Instances".

For the meaning of all other parameters, see irace::defaultScenario(). Note that we have removed all control parameters which refer to the termination of the algorithm. Use TerminatorEvals instead. Other terminators do not work with OptimizerIrace.

In contrast to irace::defaultScenario(), we set digits = 15. This represents double parameters with a higher precision and avoids rounding errors.

Target Runner and Instances

The irace package uses a targetRunner script or R function to evaluate a configuration on a particular instance. Usually it is not necessary to specify a targetRunner function when using OptimizerIrace. A default function is used that forwards several configurations and instances to the user defined objective function. As usually, the user defined function has a xs, xss or xdt parameter depending on the used Objective class. For irace, the function needs an additional instances parameter.

fun = function(xs, instances) {
 # function to evaluate configuration in `xs` on instance `instances`
}

Archive

The Archive holds the following additional columns:

  • "race" (integer(1))
    Race iteration.

  • "step" (integer(1))
    Step number of race.

  • "instance" (integer(1))
    Identifies instances across races and steps.

  • "configuration" (integer(1))
    Identifies configurations across races and steps.

Result

The optimization result (instance$result) is the best performing elite of the final race. The reported performance is the average performance estimated on all used instances.

Dictionary

This Optimizer can be instantiated via the dictionary mlr_optimizers or with the associated sugar function opt():

mlr_optimizers$get("irace")
opt("irace")

Progress Bars

$optimize() supports progress bars via the package progressr combined with a Terminator. Simply wrap the function in progressr::with_progress() to enable them. We recommend to use package progress as backend; enable with progressr::handlers("progress").

Super class

bbotk::Optimizer -> OptimizerIrace

Methods

Inherited methods


Method new()

Creates a new instance of this R6 class.

Usage


Method clone()

The objects of this class are cloneable with this method.

Usage

OptimizerIrace$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

library(data.table)

search_space = domain = ps(
  x1 = p_dbl(-5, 10),
  x2 = p_dbl(0, 15)
)

codomain = ps(y = p_dbl(tags = "minimize"))

# branin function with noise
# the noise generates different instances of the branin function
# the noise values are passed via the `instances` parameter
fun = function(xdt, instances) {
  ys = branin(xdt[["x1"]], xdt[["x2"]], noise = as.numeric(instances))
  data.table(y = ys)
}

# define objective with instances as a constant
objective = ObjectiveRFunDt$new(
 fun = fun,
 domain = domain,
 codomain = codomain,
 constants = ps(instances = p_uty()))

instance = OptimInstanceSingleCrit$new(
  objective = objective,
  search_space = search_space,
  terminator = trm("evals", n_evals = 1000))

# create instances of branin function
instances = rnorm(10, mean = 0, sd = 0.1)

# load optimizer irace and set branin instances
optimizer = opt("irace", instances = instances)

# modifies the instance by reference
optimizer$optimize(instance)
#> # 2022-11-05 08:18:06 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 3
#> # minNbSurvival: 3
#> # nbParameters: 2
#> # seed: 1426568725
#> # confidence level: 0.95
#> # budget: 1000
#> # mu: 5
#> # deterministic: FALSE
#> 
#> # 2022-11-05 08:18:06 UTC: Iteration 1 of 3
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 1000
#> # currentBudget: 333
#> # nbConfigurations: 55
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          1|         55|         11|     2.345541698|         55|00:00:00|   NA|  NA|    NA|
#> |x|          2|         55|         11|     2.345541698|        110|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|         55|         11|     2.345541698|        165|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|         55|         11|     2.345541698|        220|00:00:00|+1.00|1.00|0.0000|
#> |-|          5|          1|         11|     2.345541698|        275|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:          11    mean value:      2.345541698
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 11   11 3.22820815606974 3.59104853938334       NA
#> 
#> # 2022-11-05 08:18:07 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                  x1               x2
#> 11 3.22820815606974 3.59104853938334
#> # 2022-11-05 08:18:07 UTC: Iteration 2 of 3
#> # experimentsUsedSoFar: 275
#> # remainingBudget: 725
#> # currentBudget: 362
#> # nbConfigurations: 52
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          6|         52|         87|    0.4675979129|         52|00:00:00|   NA|  NA|    NA|
#> |x|          3|         52|         87|    0.4675979129|        103|00:00:00|+1.00|1.00|0.0000|
#> |x|          1|         52|         87|    0.4675979129|        154|00:00:00|+1.00|1.00|0.0000|
#> |x|          5|         52|         87|    0.4675979129|        205|00:00:00|+1.00|1.00|0.0000|
#> |-|          4|          2|         87|    0.4675979129|        256|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:          11    mean value:      2.345541698
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 11   11 3.22820815606974 3.59104853938334       NA
#> 
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                  x1               x2
#> 11 3.22820815606974 3.59104853938334
#> 87 3.12040374902489 2.03166870940795
#> # 2022-11-05 08:18:08 UTC: Iteration 3 of 3
#> # experimentsUsedSoFar: 531
#> # remainingBudget: 469
#> # currentBudget: 469
#> # nbConfigurations: 60
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          7|         60|        146|    0.4238398181|         60|00:00:00|   NA|  NA|    NA|
#> |x|          1|         60|        146|    0.4238398181|        118|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|         60|        146|    0.4238398181|        176|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|         60|        146|    0.4238398181|        234|00:00:00|+1.00|1.00|0.0000|
#> |-|          6|          3|        146|    0.4238398181|        292|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:          11    mean value:      2.345541698
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 11   11 3.22820815606974 3.59104853938334       NA
#> 
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 11  3.22820815606974 3.59104853938334
#> 87  3.12040374902489 2.03166870940795
#> 146 3.11675589596600 2.14282099286353
#> # 2022-11-05 08:18:08 UTC: Iteration 4 of 4
#> # experimentsUsedSoFar: 823
#> # remainingBudget: 177
#> # currentBudget: 177
#> # nbConfigurations: 22
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          8|         22|        179|    0.4066167046|         22|00:00:00|   NA|  NA|    NA|
#> |x|          5|         22|        179|    0.4066167046|         42|00:00:00|+1.00|1.00|0.0000|
#> |x|          1|         22|        179|    0.4066167046|         61|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|         22|        179|    0.4066167046|         80|00:00:00|+1.00|1.00|0.0000|
#> |-|          6|          4|        179|    0.4066167046|         99|00:00:00|+1.00|1.00|0.0000|
#> |!|          2|          4|        179|    0.4066167046|        102|00:00:00|+1.00|1.00|0.0000|
#> |!|          4|          4|        179|    0.4066167046|        103|00:00:00|+1.00|1.00|0.0000|
#> |-|          7|          1|        179|    0.4066167046|        104|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         179    mean value:     0.4066167046
#> Description of the best-so-far configuration:
#>     .ID.               x1               x2 .PARENT.
#> 179  179 3.12986906050738 2.19433019754848      146
#> 
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 179 3.12986906050738 2.19433019754848
#> # 2022-11-05 08:18:09 UTC: Iteration 5 of 5
#> # experimentsUsedSoFar: 927
#> # remainingBudget: 73
#> # currentBudget: 73
#> # nbConfigurations: 8
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          9|          8|        185|    0.4020123373|          8|00:00:00|   NA|  NA|    NA|
#> |x|          1|          8|        185|    0.4020123373|         15|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|          8|        185|    0.4020123373|         22|00:00:00|+1.00|1.00|0.0000|
#> |x|          6|          8|        185|    0.4020123373|         29|00:00:00|+1.00|1.00|0.0000|
#> |-|          2|          2|        185|    0.4020123373|         36|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         179    mean value:     0.4066167046
#> Description of the best-so-far configuration:
#>     .ID.               x1               x2 .PARENT.
#> 179  179 3.12986906050738 2.19433019754848      146
#> 
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> # 2022-11-05 08:18:09 UTC: Iteration 6 of 6
#> # experimentsUsedSoFar: 963
#> # remainingBudget: 37
#> # currentBudget: 37
#> # nbConfigurations: 5
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|         10|          5|        191|    0.4007174838|          5|00:00:00|   NA|  NA|    NA|
#> |x|          7|          5|        191|    0.4007174838|          9|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|          5|        191|    0.4007174838|         13|00:00:00|+1.00|1.00|0.0000|
#> |x|          9|          5|        191|    0.4007174838|         16|00:00:00|+1.00|1.00|0.0000|
#> |-|          3|          3|        191|    0.4007174838|         19|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         179    mean value:     0.4066167046
#> Description of the best-so-far configuration:
#>     .ID.               x1               x2 .PARENT.
#> 179  179 3.12986906050738 2.19433019754848      146
#> 
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> 191 3.15402521858829 2.21962921089567
#> # 2022-11-05 08:18:09 UTC: Iteration 7 of 7
#> # experimentsUsedSoFar: 982
#> # remainingBudget: 18
#> # currentBudget: 18
#> # nbConfigurations: 4
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|         11|          4|        191|    0.4007174838|          4|00:00:00|   NA|  NA|    NA|
#> |x|          6|          4|        191|    0.4007174838|          6|00:00:00|+1.00|1.00|0.0000|
#> |x|          9|          4|        191|    0.4007174838|          7|00:00:00|+1.00|1.00|0.0000|
#> |x|          8|          4|        191|    0.4007174838|         10|00:00:00|+1.00|1.00|0.0000|
#> |-|          7|          3|        191|    0.4007174838|         11|00:00:00|+1.00|1.00|0.0000|
#> |.|          4|          3|        191|    0.4007174838|         11|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         179    mean value:     0.4066167046
#> Description of the best-so-far configuration:
#>     .ID.               x1               x2 .PARENT.
#> 179  179 3.12986906050738 2.19433019754848      146
#> 
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> 191 3.15402521858829 2.21962921089567
#> # 2022-11-05 08:18:09 UTC: Stopped because there is not enough budget left to race more than the minimum (3)
#> # You may either increase the budget or set 'minNbSurvival' to a lower value
#> # Iteration: 8
#> # nbIterations: 8
#> # experimentsUsedSoFar: 993
#> # timeUsed: 0
#> # remainingBudget: 7
#> # currentBudget: 7
#> # number of elites: 3
#> # nbConfigurations: 3
#> # Total CPU user time: 2.455, CPU sys time: 0.024, Wall-clock time: 2.48
#>          x1      x2 configuration  x_domain         y
#> 1: 3.129869 2.19433           179 <list[2]> 0.4066167

# best scoring configuration
instance$result
#>          x1      x2 configuration  x_domain         y
#> 1: 3.129869 2.19433           179 <list[2]> 0.4066167

# all evaluations
as.data.table(instance$archive)
#>              x1         x2           y race step instance configuration
#>   1:  4.6886568 11.5529239 113.3073449    1    1        1             1
#>   2:  4.8117002  2.5361398  12.3998639    1    1        1             2
#>   3:  1.6886930  1.0712526  15.6800673    1    1        1             3
#>   4:  4.5821781  7.8284080  49.8255244    1    1        1             4
#>   5: -0.3309107  0.6326879  53.9870345    1    1        1             5
#>  ---                                                                   
#> 989:  3.1412163  2.1943575   0.4044387    7    1        2           194
#> 990:  3.1419756  2.2104809   0.4020123    7    1       10           185
#> 991:  3.1540252  2.2196292   0.4007175    7    1       10           191
#> 992:  3.1412163  2.1943575   0.4044387    7    1       10           194
#> 993:  3.1412163  2.1943575   0.4044387    7    1        5           194
#>                timestamp batch_nr
#>   1: 2022-11-05 08:18:07        1
#>   2: 2022-11-05 08:18:07        1
#>   3: 2022-11-05 08:18:07        1
#>   4: 2022-11-05 08:18:07        1
#>   5: 2022-11-05 08:18:07        1
#>  ---                             
#> 989: 2022-11-05 08:18:09       36
#> 990: 2022-11-05 08:18:09       37
#> 991: 2022-11-05 08:18:09       37
#> 992: 2022-11-05 08:18:09       37
#> 993: 2022-11-05 08:18:09       38