OptimizerIrace class that implements iterated racing. Calls
irace::irace() from package irace.
Source
Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43--58. doi:10.1016/j.orp.2016.09.002 .
Parameters
instanceslist()
A list of instances where the configurations executed on.targetRunnerParallelfunction()
A function that executes the objective function with a specific parameter configuration and instance. A default function is provided, see section "Target Runner and Instances".
For the meaning of all other parameters, see irace::defaultScenario(). Note
that we have removed all control parameters which refer to the termination of
the algorithm. Use TerminatorEvals instead. Other terminators do not work
with OptimizerIrace.
In contrast to irace::defaultScenario(), we set digits = 15.
This represents double parameters with a higher precision and avoids rounding errors.
Target Runner and Instances
The irace package uses a targetRunner script or R function to evaluate a
configuration on a particular instance. Usually it is not necessary to
specify a targetRunner function when using OptimizerIrace. A default
function is used that forwards several configurations and instances to the
user defined objective function. As usually, the user defined function has
a xs, xss or xdt parameter depending on the used Objective class.
For irace, the function needs an additional instances parameter.
fun = function(xs, instances) {
# function to evaluate configuration in `xs` on instance `instances`
}Archive
The Archive holds the following additional columns:
"race"(integer(1))
Race iteration."step"(integer(1))
Step number of race."instance"(integer(1))
Identifies instances across races and steps."configuration"(integer(1))
Identifies configurations across races and steps.
Result
The optimization result (instance$result) is the best performing elite of
the final race. The reported performance is the average performance estimated
on all used instances.
Dictionary
This Optimizer can be instantiated via the dictionary
mlr_optimizers or with the associated sugar function opt():
mlr_optimizers$get("irace")
opt("irace")Progress Bars
$optimize() supports progress bars via the package progressr
combined with a Terminator. Simply wrap the function in
progressr::with_progress() to enable them. We recommend to use package
progress as backend; enable with progressr::handlers("progress").
Super class
bbotk::Optimizer -> OptimizerIrace
Examples
library(data.table)
search_space = domain = ps(
x1 = p_dbl(-5, 10),
x2 = p_dbl(0, 15)
)
codomain = ps(y = p_dbl(tags = "minimize"))
# branin function with noise
# the noise generates different instances of the branin function
# the noise values are passed via the `instances` parameter
fun = function(xdt, instances) {
ys = branin(xdt[["x1"]], xdt[["x2"]], noise = as.numeric(instances))
data.table(y = ys)
}
# define objective with instances as a constant
objective = ObjectiveRFunDt$new(
fun = fun,
domain = domain,
codomain = codomain,
constants = ps(instances = p_uty()))
instance = OptimInstanceSingleCrit$new(
objective = objective,
search_space = search_space,
terminator = trm("evals", n_evals = 1000))
# create instances of branin function
instances = rnorm(10, mean = 0, sd = 0.1)
# load optimizer irace and set branin instances
optimizer = opt("irace", instances = instances)
# modifies the instance by reference
optimizer$optimize(instance)
#> # 2022-11-05 08:18:06 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 3
#> # minNbSurvival: 3
#> # nbParameters: 2
#> # seed: 1426568725
#> # confidence level: 0.95
#> # budget: 1000
#> # mu: 5
#> # deterministic: FALSE
#>
#> # 2022-11-05 08:18:06 UTC: Iteration 1 of 3
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 1000
#> # currentBudget: 333
#> # nbConfigurations: 55
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 1| 55| 11| 2.345541698| 55|00:00:00| NA| NA| NA|
#> |x| 2| 55| 11| 2.345541698| 110|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 55| 11| 2.345541698| 165|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 55| 11| 2.345541698| 220|00:00:00|+1.00|1.00|0.0000|
#> |-| 5| 1| 11| 2.345541698| 275|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 11 mean value: 2.345541698
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 11 11 3.22820815606974 3.59104853938334 NA
#>
#> # 2022-11-05 08:18:07 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 11 3.22820815606974 3.59104853938334
#> # 2022-11-05 08:18:07 UTC: Iteration 2 of 3
#> # experimentsUsedSoFar: 275
#> # remainingBudget: 725
#> # currentBudget: 362
#> # nbConfigurations: 52
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 6| 52| 87| 0.4675979129| 52|00:00:00| NA| NA| NA|
#> |x| 3| 52| 87| 0.4675979129| 103|00:00:00|+1.00|1.00|0.0000|
#> |x| 1| 52| 87| 0.4675979129| 154|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 52| 87| 0.4675979129| 205|00:00:00|+1.00|1.00|0.0000|
#> |-| 4| 2| 87| 0.4675979129| 256|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 11 mean value: 2.345541698
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 11 11 3.22820815606974 3.59104853938334 NA
#>
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 11 3.22820815606974 3.59104853938334
#> 87 3.12040374902489 2.03166870940795
#> # 2022-11-05 08:18:08 UTC: Iteration 3 of 3
#> # experimentsUsedSoFar: 531
#> # remainingBudget: 469
#> # currentBudget: 469
#> # nbConfigurations: 60
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 7| 60| 146| 0.4238398181| 60|00:00:00| NA| NA| NA|
#> |x| 1| 60| 146| 0.4238398181| 118|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 60| 146| 0.4238398181| 176|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 60| 146| 0.4238398181| 234|00:00:00|+1.00|1.00|0.0000|
#> |-| 6| 3| 146| 0.4238398181| 292|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 11 mean value: 2.345541698
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 11 11 3.22820815606974 3.59104853938334 NA
#>
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 11 3.22820815606974 3.59104853938334
#> 87 3.12040374902489 2.03166870940795
#> 146 3.11675589596600 2.14282099286353
#> # 2022-11-05 08:18:08 UTC: Iteration 4 of 4
#> # experimentsUsedSoFar: 823
#> # remainingBudget: 177
#> # currentBudget: 177
#> # nbConfigurations: 22
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 8| 22| 179| 0.4066167046| 22|00:00:00| NA| NA| NA|
#> |x| 5| 22| 179| 0.4066167046| 42|00:00:00|+1.00|1.00|0.0000|
#> |x| 1| 22| 179| 0.4066167046| 61|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 22| 179| 0.4066167046| 80|00:00:00|+1.00|1.00|0.0000|
#> |-| 6| 4| 179| 0.4066167046| 99|00:00:00|+1.00|1.00|0.0000|
#> |!| 2| 4| 179| 0.4066167046| 102|00:00:00|+1.00|1.00|0.0000|
#> |!| 4| 4| 179| 0.4066167046| 103|00:00:00|+1.00|1.00|0.0000|
#> |-| 7| 1| 179| 0.4066167046| 104|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 179 mean value: 0.4066167046
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 179 179 3.12986906050738 2.19433019754848 146
#>
#> # 2022-11-05 08:18:08 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 179 3.12986906050738 2.19433019754848
#> # 2022-11-05 08:18:09 UTC: Iteration 5 of 5
#> # experimentsUsedSoFar: 927
#> # remainingBudget: 73
#> # currentBudget: 73
#> # nbConfigurations: 8
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 9| 8| 185| 0.4020123373| 8|00:00:00| NA| NA| NA|
#> |x| 1| 8| 185| 0.4020123373| 15|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 8| 185| 0.4020123373| 22|00:00:00|+1.00|1.00|0.0000|
#> |x| 6| 8| 185| 0.4020123373| 29|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 2| 185| 0.4020123373| 36|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 179 mean value: 0.4066167046
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 179 179 3.12986906050738 2.19433019754848 146
#>
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> # 2022-11-05 08:18:09 UTC: Iteration 6 of 6
#> # experimentsUsedSoFar: 963
#> # remainingBudget: 37
#> # currentBudget: 37
#> # nbConfigurations: 5
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 10| 5| 191| 0.4007174838| 5|00:00:00| NA| NA| NA|
#> |x| 7| 5| 191| 0.4007174838| 9|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 5| 191| 0.4007174838| 13|00:00:00|+1.00|1.00|0.0000|
#> |x| 9| 5| 191| 0.4007174838| 16|00:00:00|+1.00|1.00|0.0000|
#> |-| 3| 3| 191| 0.4007174838| 19|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 179 mean value: 0.4066167046
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 179 179 3.12986906050738 2.19433019754848 146
#>
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> 191 3.15402521858829 2.21962921089567
#> # 2022-11-05 08:18:09 UTC: Iteration 7 of 7
#> # experimentsUsedSoFar: 982
#> # remainingBudget: 18
#> # currentBudget: 18
#> # nbConfigurations: 4
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 11| 4| 191| 0.4007174838| 4|00:00:00| NA| NA| NA|
#> |x| 6| 4| 191| 0.4007174838| 6|00:00:00|+1.00|1.00|0.0000|
#> |x| 9| 4| 191| 0.4007174838| 7|00:00:00|+1.00|1.00|0.0000|
#> |x| 8| 4| 191| 0.4007174838| 10|00:00:00|+1.00|1.00|0.0000|
#> |-| 7| 3| 191| 0.4007174838| 11|00:00:00|+1.00|1.00|0.0000|
#> |.| 4| 3| 191| 0.4007174838| 11|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 179 mean value: 0.4066167046
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 179 179 3.12986906050738 2.19433019754848 146
#>
#> # 2022-11-05 08:18:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 179 3.12986906050738 2.19433019754848
#> 185 3.14197557616049 2.21048087022291
#> 191 3.15402521858829 2.21962921089567
#> # 2022-11-05 08:18:09 UTC: Stopped because there is not enough budget left to race more than the minimum (3)
#> # You may either increase the budget or set 'minNbSurvival' to a lower value
#> # Iteration: 8
#> # nbIterations: 8
#> # experimentsUsedSoFar: 993
#> # timeUsed: 0
#> # remainingBudget: 7
#> # currentBudget: 7
#> # number of elites: 3
#> # nbConfigurations: 3
#> # Total CPU user time: 2.455, CPU sys time: 0.024, Wall-clock time: 2.48
#> x1 x2 configuration x_domain y
#> 1: 3.129869 2.19433 179 <list[2]> 0.4066167
# best scoring configuration
instance$result
#> x1 x2 configuration x_domain y
#> 1: 3.129869 2.19433 179 <list[2]> 0.4066167
# all evaluations
as.data.table(instance$archive)
#> x1 x2 y race step instance configuration
#> 1: 4.6886568 11.5529239 113.3073449 1 1 1 1
#> 2: 4.8117002 2.5361398 12.3998639 1 1 1 2
#> 3: 1.6886930 1.0712526 15.6800673 1 1 1 3
#> 4: 4.5821781 7.8284080 49.8255244 1 1 1 4
#> 5: -0.3309107 0.6326879 53.9870345 1 1 1 5
#> ---
#> 989: 3.1412163 2.1943575 0.4044387 7 1 2 194
#> 990: 3.1419756 2.2104809 0.4020123 7 1 10 185
#> 991: 3.1540252 2.2196292 0.4007175 7 1 10 191
#> 992: 3.1412163 2.1943575 0.4044387 7 1 10 194
#> 993: 3.1412163 2.1943575 0.4044387 7 1 5 194
#> timestamp batch_nr
#> 1: 2022-11-05 08:18:07 1
#> 2: 2022-11-05 08:18:07 1
#> 3: 2022-11-05 08:18:07 1
#> 4: 2022-11-05 08:18:07 1
#> 5: 2022-11-05 08:18:07 1
#> ---
#> 989: 2022-11-05 08:18:09 36
#> 990: 2022-11-05 08:18:09 37
#> 991: 2022-11-05 08:18:09 37
#> 992: 2022-11-05 08:18:09 37
#> 993: 2022-11-05 08:18:09 38