Skip to contents

OptimizerBatchIrace class that implements iterated racing. Calls irace::irace() from package irace.

Source

Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43–58. doi:10.1016/j.orp.2016.09.002 .

Parameters

instances

list()
A list of instances where the configurations executed on.

targetRunnerParallel

function()
A function that executes the objective function with a specific parameter configuration and instance. A default function is provided, see section "Target Runner and Instances".

For the meaning of all other parameters, see irace::defaultScenario(). Note that we have removed all control parameters which refer to the termination of the algorithm. Use TerminatorEvals instead. Other terminators do not work with OptimizerBatchIrace.

In contrast to irace::defaultScenario(), we set digits = 15. This represents double parameters with a higher precision and avoids rounding errors.

Target Runner and Instances

The irace package uses a targetRunner script or R function to evaluate a configuration on a particular instance. Usually it is not necessary to specify a targetRunner function when using OptimizerBatchIrace. A default function is used that forwards several configurations and instances to the user defined objective function. As usually, the user defined function has a xs, xss or xdt parameter depending on the used Objective class. For irace, the function needs an additional instances parameter.

fun = function(xs, instances) {
 # function to evaluate configuration in `xs` on instance `instances`
}

Archive

The Archive holds the following additional columns:

  • "race" (integer(1))
    Race iteration.

  • "step" (integer(1))
    Step number of race.

  • "instance" (integer(1))
    Identifies instances across races and steps.

  • "configuration" (integer(1))
    Identifies configurations across races and steps.

Result

The optimization result (instance$result) is the best performing elite of the final race. The reported performance is the average performance estimated on all used instances.

Dictionary

This Optimizer can be instantiated via the dictionary mlr_optimizers or with the associated sugar function opt():

mlr_optimizers$get("irace")
opt("irace")

Progress Bars

$optimize() supports progress bars via the package progressr combined with a Terminator. Simply wrap the function in progressr::with_progress() to enable them. We recommend to use package progress as backend; enable with progressr::handlers("progress").

Super classes

bbotk::Optimizer -> bbotk::OptimizerBatch -> OptimizerBatchIrace

Methods

Inherited methods


Method new()

Creates a new instance of this R6 class.

Usage


Method clone()

The objects of this class are cloneable with this method.

Usage

OptimizerBatchIrace$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# \donttest{
library(data.table)

search_space = domain = ps(
  x1 = p_dbl(-5, 10),
  x2 = p_dbl(0, 15)
)

codomain = ps(y = p_dbl(tags = "minimize"))

# branin function with noise
# the noise generates different instances of the branin function
# the noise values are passed via the `instances` parameter
fun = function(xdt, instances) {
  ys = branin(xdt[["x1"]], xdt[["x2"]], noise = as.numeric(instances))
  data.table(y = ys)
}

# define objective with instances as a constant
objective = ObjectiveRFunDt$new(
 fun = fun,
 domain = domain,
 codomain = codomain,
 constants = ps(instances = p_uty()))

instance = OptimInstanceBatchSingleCrit$new(
  objective = objective,
  search_space = search_space,
  terminator = trm("evals", n_evals = 1000))

# create instances of branin function
instances = rnorm(10, mean = 0, sd = 0.1)

# load optimizer irace and set branin instances
optimizer = opt("irace", instances = instances)

# modifies the instance by reference
optimizer$optimize(instance)
#> # 2024-11-27 19:25:22 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 3
#> # minNbSurvival: 3
#> # nbParameters: 2
#> # seed: 2011091433
#> # confidence level: 0.95
#> # budget: 1000
#> # mu: 5
#> # deterministic: FALSE
#> 
#> # 2024-11-27 19:25:23 UTC: Iteration 1 of 3
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 1000
#> # currentBudget: 333
#> # nbConfigurations: 55
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          1|         55|         32|    0.6807327591|         55|00:00:00|   NA|  NA|    NA|
#> |x|          2|         55|         32|    0.6807327591|        110|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|         55|         32|    0.6807327591|        165|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|         55|         32|    0.6807327591|        220|00:00:00|+1.00|1.00|0.0000|
#> |-|          5|          1|         32|    0.6807327591|        275|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:          32    mean value:     0.6807327591
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 32   32 3.37967336177826 2.20616519451141       NA
#> 
#> # 2024-11-27 19:25:24 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                  x1               x2
#> 32 3.37967336177826 2.20616519451141
#> # 2024-11-27 19:25:24 UTC: Iteration 2 of 3
#> # experimentsUsedSoFar: 275
#> # remainingBudget: 725
#> # currentBudget: 362
#> # nbConfigurations: 52
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          6|         52|         66|    0.4358445903|         52|00:00:00|   NA|  NA|    NA|
#> |x|          3|         52|         66|    0.4358445903|        103|00:00:00|+1.00|1.00|0.0000|
#> |x|          1|         52|         66|    0.4358445903|        154|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|         52|         66|    0.4358445903|        205|00:00:00|+1.00|1.00|0.0000|
#> |-|          2|          2|         66|    0.4358445903|        256|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 66
#> Best-so-far configuration:          32    mean value:     0.6807327591
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 32   32 3.37967336177826 2.20616519451141       NA
#> 
#> # 2024-11-27 19:25:25 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                  x1               x2
#> 32 3.37967336177826 2.20616519451141
#> 66 3.12659798739228 2.47875866510817
#> # 2024-11-27 19:25:26 UTC: Iteration 3 of 3
#> # experimentsUsedSoFar: 531
#> # remainingBudget: 469
#> # currentBudget: 469
#> # nbConfigurations: 60
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          7|         60|        155|    0.4063383880|         60|00:00:00|   NA|  NA|    NA|
#> |x|          3|         60|        155|    0.4063383880|        118|00:00:00|+1.00|1.00|0.0000|
#> |x|          5|         60|        155|    0.4063383880|        177|00:00:00|+1.00|1.00|0.0000|
#> |x|          1|         60|        155|    0.4063383880|        235|00:00:00|+1.00|1.00|0.0000|
#> |-|          2|          3|        155|    0.4063383880|        293|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 155
#> Best-so-far configuration:          66    mean value:     0.4358445903
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 66   66 3.12659798739228 2.47875866510817       32
#> 
#> # 2024-11-27 19:25:27 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 66  3.12659798739228 2.47875866510817
#> 32  3.37967336177826 2.20616519451141
#> 155 3.17261652702200 2.31282092945237
#> # 2024-11-27 19:25:28 UTC: Iteration 4 of 4
#> # experimentsUsedSoFar: 824
#> # remainingBudget: 176
#> # currentBudget: 176
#> # nbConfigurations: 21
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          8|         21|        155|    0.4063383880|         21|00:00:00|   NA|  NA|    NA|
#> |x|          5|         21|        155|    0.4063383880|         39|00:00:00|+1.00|1.00|0.0000|
#> |x|          3|         21|        155|    0.4063383880|         57|00:00:00|+1.00|1.00|0.0000|
#> |x|          1|         21|        155|    0.4063383880|         75|00:00:00|+1.00|1.00|0.0000|
#> |-|          2|          3|        155|    0.4063383880|         93|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 155
#> Best-so-far configuration:          66    mean value:     0.4358445903
#> Description of the best-so-far configuration:
#>    .ID.               x1               x2 .PARENT.
#> 66   66 3.12659798739228 2.47875866510817       32
#> 
#> # 2024-11-27 19:25:28 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 66  3.12659798739228 2.47875866510817
#> 32  3.37967336177826 2.20616519451141
#> 155 3.17261652702200 2.31282092945237
#> # 2024-11-27 19:25:29 UTC: Iteration 5 of 5
#> # experimentsUsedSoFar: 917
#> # remainingBudget: 83
#> # currentBudget: 83
#> # nbConfigurations: 10
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          9|         10|        189|    0.3997731773|         10|00:00:00|   NA|  NA|    NA|
#> |x|          1|         10|        189|    0.3997731773|         17|00:00:00|+1.00|1.00|0.0000|
#> |x|          8|         10|        189|    0.3997731773|         24|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|         10|        189|    0.3997731773|         32|00:00:00|+1.00|1.00|0.0000|
#> |-|          6|          4|        189|    0.3997731773|         40|00:00:00|+1.00|1.00|0.0000|
#> |!|          7|          4|        189|    0.3997731773|         41|00:00:00|+1.00|1.00|0.0000|
#> |!|          2|          4|        189|    0.3997731773|         42|00:00:00|+1.00|1.00|0.0000|
#> |!|          3|          4|        189|    0.3997731773|         43|00:00:00|+1.00|1.00|0.0000|
#> |-|          5|          1|        189|    0.3997731773|         44|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         189    mean value:     0.3997731773
#> Description of the best-so-far configuration:
#>     .ID.               x1             x2 .PARENT.
#> 189  189 3.14982869495763 2.308084599866      155
#> 
#> # 2024-11-27 19:25:29 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1             x2
#> 189 3.14982869495763 2.308084599866
#> # 2024-11-27 19:25:30 UTC: Iteration 6 of 6
#> # experimentsUsedSoFar: 961
#> # remainingBudget: 39
#> # currentBudget: 39
#> # nbConfigurations: 4
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|         10|          4|        192|    0.3988000631|          4|00:00:00|   NA|  NA|    NA|
#> |x|          2|          4|        192|    0.3988000631|          7|00:00:00|+1.00|1.00|0.0000|
#> |x|          9|          4|        192|    0.3988000631|         10|00:00:00|+1.00|1.00|0.0000|
#> |x|          4|          4|        192|    0.3988000631|         13|00:00:00|+1.00|1.00|0.0000|
#> |-|          3|          2|        192|    0.3988000631|         16|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 192
#> Best-so-far configuration:         189    mean value:     0.3997731773
#> Description of the best-so-far configuration:
#>     .ID.               x1             x2 .PARENT.
#> 189  189 3.14982869495763 2.308084599866      155
#> 
#> # 2024-11-27 19:25:30 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 189 3.14982869495763 2.30808459986600
#> 192 3.14194228084746 2.30492865459122
#> # 2024-11-27 19:25:31 UTC: Iteration 7 of 7
#> # experimentsUsedSoFar: 977
#> # remainingBudget: 23
#> # currentBudget: 23
#> # nbConfigurations: 4
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|         10|          4|        192|    0.3988000631|          2|00:00:00|   NA|  NA|    NA|
#> |x|          2|          4|        192|    0.3988000631|          4|00:00:00|+1.00|1.00|0.0000|
#> |x|          5|          4|        192|    0.3988000631|          7|00:00:00|+1.00|1.00|0.0000|
#> |x|          8|          4|        192|    0.3988000631|         10|00:00:00|+1.00|1.00|0.0000|
#> |-|          3|          2|        192|    0.3988000631|         12|00:00:00|+1.00|1.00|0.0000|
#> |.|          9|          2|        192|    0.3988000631|         12|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 192
#> Best-so-far configuration:         189    mean value:     0.3997731773
#> Description of the best-so-far configuration:
#>     .ID.               x1             x2 .PARENT.
#> 189  189 3.14982869495763 2.308084599866      155
#> 
#> # 2024-11-27 19:25:31 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 189 3.14982869495763 2.30808459986600
#> 192 3.14194228084746 2.30492865459122
#> # 2024-11-27 19:25:31 UTC: Stopped because there is not enough budget left to race more than the minimum (3).
#> # You may either increase the budget or set 'minNbSurvival' to a lower value.
#> # Iteration: 8
#> # nbIterations: 8
#> # experimentsUsedSoFar: 989
#> # timeUsed: 0
#> # remainingBudget: 11
#> # currentBudget: 11
#> # number of elites: 2
#> # nbConfigurations: 3
#> # Total CPU user time: 9.43, CPU sys time: 0.052, Wall-clock time: 9.484
#> # 2024-11-27 19:25:32 UTC: Starting post-selection:
#> # Configurations selected: 189, 192, 66, 32, 155.
#> # Pending instances: 0, 3, 1, 1, 1.
#> # Seed: 2011091433
#> # Configurations: 5
#> # Available experiments: 11
#> # minSurvival: 1
#> # Markers:
#>      x No test is performed.
#>      c Configurations are discarded only due to capping.
#>      - The test is performed and some configurations are discarded.
#>      = The test is performed but no configuration is discarded.
#>      ! The test is performed and configurations could be discarded but elite configurations are preserved.
#>      . All alive configurations are elite and nothing is discarded.
#> 
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | |   Instance|      Alive|       Best|       Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x|          6|          5|        192|    0.3988000631|          1|00:00:00|   NA|  NA|    NA|
#> |.|          2|          5|        192|    0.3988000631|          1|00:00:00|+1.00|1.00|0.0000|
#> |.|          8|          5|        192|    0.3988000631|          1|00:00:00|+1.00|1.00|0.0000|
#> |.|          9|          5|        192|    0.3988000631|          1|00:00:00|+1.00|1.00|0.0000|
#> |.|          5|          5|        192|    0.3988000631|          1|00:00:00|+1.00|1.00|0.0000|
#> |.|          4|          5|        192|    0.3988000631|          1|00:00:00|+1.00|1.00|0.0000|
#> |!|         10|          5|        192|    0.3988000631|          4|00:00:00|+1.00|1.00|0.0000|
#> |!|          7|          5|        192|    0.3988000631|          5|00:00:00|+1.00|1.00|0.0000|
#> |.|          3|          5|        192|    0.3988000631|          5|00:00:00|+1.00|1.00|0.0000|
#> |-|          1|          1|        192|    0.3988000631|          6|00:00:00|   NA|  NA|    NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration:         192    mean value:     0.3988000631
#> Description of the best-so-far configuration:
#>     .ID.               x1               x2 .PARENT.
#> 192  192 3.14194228084746 2.30492865459122      189
#> 
#> # 2024-11-27 19:25:42 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#>                   x1               x2
#> 192 3.14194228084746 2.30492865459122
#> # Total CPU user time: 18.967, CPU sys time: 0.392, Wall-clock time: 19.375
#>          x1       x2 configuration  x_domain         y
#>       <num>    <num>         <int>    <list>     <num>
#> 1: 3.141942 2.304929           192 <list[2]> 0.3988001

# best scoring configuration
instance$result
#>          x1       x2 configuration  x_domain         y
#>       <num>    <num>         <int>    <list>     <num>
#> 1: 3.141942 2.304929           192 <list[2]> 0.3988001

# all evaluations
as.data.table(instance$archive)
#>             x1         x2           y  race  step instance configuration
#>          <num>      <num>       <num> <int> <int>    <int>         <int>
#>   1:  4.785923  0.7999151  10.9992463     1     1        2             1
#>   2: -2.714076  8.2999152  10.0905500     1     1        2             2
#>   3:  8.535923 12.0499152 108.4497127     1     1        2             3
#>   4:  1.035924  4.5499152  14.8980988     1     1        2             4
#>   5:  2.910923 10.1749152  60.1449255     1     1        2             5
#>  ---                                                                    
#> 991:  3.379673  2.2061652   0.6807328     8     1        1            32
#> 992:  3.126598  2.4787587   0.4358446     8     1        1            66
#> 993:  3.172617  2.3128209   0.4063384     8     1        1           155
#> 994:  3.141942  2.3049287   0.3988001     8     1        6           192
#> 995:  3.141942  2.3049287   0.3988001     8     1        2           192
#>                timestamp batch_nr
#>                   <POSc>    <int>
#>   1: 2024-11-27 19:25:23        1
#>   2: 2024-11-27 19:25:23        1
#>   3: 2024-11-27 19:25:23        1
#>   4: 2024-11-27 19:25:23        1
#>   5: 2024-11-27 19:25:23        1
#>  ---                             
#> 991: 2024-11-27 19:25:42       41
#> 992: 2024-11-27 19:25:42       41
#> 993: 2024-11-27 19:25:42       41
#> 994: 2024-11-27 19:25:42       42
#> 995: 2024-11-27 19:25:42       43
# }