OptimizerBatchIrace
class that implements iterated racing. Calls
irace::irace()
from package irace.
Source
Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43–58. doi:10.1016/j.orp.2016.09.002 .
Parameters
instances
list()
A list of instances where the configurations executed on.targetRunnerParallel
function()
A function that executes the objective function with a specific parameter configuration and instance. A default function is provided, see section "Target Runner and Instances".
For the meaning of all other parameters, see irace::defaultScenario()
. Note
that we have removed all control parameters which refer to the termination of
the algorithm. Use TerminatorEvals instead. Other terminators do not work
with OptimizerBatchIrace
.
In contrast to irace::defaultScenario()
, we set digits = 15
.
This represents double parameters with a higher precision and avoids rounding errors.
Target Runner and Instances
The irace package uses a targetRunner
script or R function to evaluate a
configuration on a particular instance. Usually it is not necessary to
specify a targetRunner
function when using OptimizerBatchIrace
. A default
function is used that forwards several configurations and instances to the
user defined objective function. As usually, the user defined function has
a xs
, xss
or xdt
parameter depending on the used Objective class.
For irace, the function needs an additional instances
parameter.
Archive
The Archive holds the following additional columns:
"race"
(integer(1)
)
Race iteration."step"
(integer(1)
)
Step number of race."instance"
(integer(1)
)
Identifies instances across races and steps."configuration"
(integer(1)
)
Identifies configurations across races and steps.
Result
The optimization result (instance$result
) is the best performing elite of
the final race. The reported performance is the average performance estimated
on all used instances.
Dictionary
This Optimizer can be instantiated via the dictionary
mlr_optimizers or with the associated sugar function opt()
:
Progress Bars
$optimize()
supports progress bars via the package progressr
combined with a Terminator. Simply wrap the function in
progressr::with_progress()
to enable them. We recommend to use package
progress as backend; enable with progressr::handlers("progress")
.
Super classes
bbotk::Optimizer
-> bbotk::OptimizerBatch
-> OptimizerBatchIrace
Examples
# \donttest{
library(data.table)
search_space = domain = ps(
x1 = p_dbl(-5, 10),
x2 = p_dbl(0, 15)
)
codomain = ps(y = p_dbl(tags = "minimize"))
# branin function with noise
# the noise generates different instances of the branin function
# the noise values are passed via the `instances` parameter
fun = function(xdt, instances) {
ys = branin(xdt[["x1"]], xdt[["x2"]], noise = as.numeric(instances))
data.table(y = ys)
}
# define objective with instances as a constant
objective = ObjectiveRFunDt$new(
fun = fun,
domain = domain,
codomain = codomain,
constants = ps(instances = p_uty()))
instance = OptimInstanceBatchSingleCrit$new(
objective = objective,
search_space = search_space,
terminator = trm("evals", n_evals = 1000))
# create instances of branin function
instances = rnorm(10, mean = 0, sd = 0.1)
# load optimizer irace and set branin instances
optimizer = opt("irace", instances = instances)
# modifies the instance by reference
optimizer$optimize(instance)
#> # 2024-11-27 19:25:22 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 3
#> # minNbSurvival: 3
#> # nbParameters: 2
#> # seed: 2011091433
#> # confidence level: 0.95
#> # budget: 1000
#> # mu: 5
#> # deterministic: FALSE
#>
#> # 2024-11-27 19:25:23 UTC: Iteration 1 of 3
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 1000
#> # currentBudget: 333
#> # nbConfigurations: 55
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 1| 55| 32| 0.6807327591| 55|00:00:00| NA| NA| NA|
#> |x| 2| 55| 32| 0.6807327591| 110|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 55| 32| 0.6807327591| 165|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 55| 32| 0.6807327591| 220|00:00:00|+1.00|1.00|0.0000|
#> |-| 5| 1| 32| 0.6807327591| 275|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 32 mean value: 0.6807327591
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 32 32 3.37967336177826 2.20616519451141 NA
#>
#> # 2024-11-27 19:25:24 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 32 3.37967336177826 2.20616519451141
#> # 2024-11-27 19:25:24 UTC: Iteration 2 of 3
#> # experimentsUsedSoFar: 275
#> # remainingBudget: 725
#> # currentBudget: 362
#> # nbConfigurations: 52
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 6| 52| 66| 0.4358445903| 52|00:00:00| NA| NA| NA|
#> |x| 3| 52| 66| 0.4358445903| 103|00:00:00|+1.00|1.00|0.0000|
#> |x| 1| 52| 66| 0.4358445903| 154|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 52| 66| 0.4358445903| 205|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 2| 66| 0.4358445903| 256|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 66
#> Best-so-far configuration: 32 mean value: 0.6807327591
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 32 32 3.37967336177826 2.20616519451141 NA
#>
#> # 2024-11-27 19:25:25 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 32 3.37967336177826 2.20616519451141
#> 66 3.12659798739228 2.47875866510817
#> # 2024-11-27 19:25:26 UTC: Iteration 3 of 3
#> # experimentsUsedSoFar: 531
#> # remainingBudget: 469
#> # currentBudget: 469
#> # nbConfigurations: 60
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 7| 60| 155| 0.4063383880| 60|00:00:00| NA| NA| NA|
#> |x| 3| 60| 155| 0.4063383880| 118|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 60| 155| 0.4063383880| 177|00:00:00|+1.00|1.00|0.0000|
#> |x| 1| 60| 155| 0.4063383880| 235|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 3| 155| 0.4063383880| 293|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 155
#> Best-so-far configuration: 66 mean value: 0.4358445903
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 66 66 3.12659798739228 2.47875866510817 32
#>
#> # 2024-11-27 19:25:27 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 66 3.12659798739228 2.47875866510817
#> 32 3.37967336177826 2.20616519451141
#> 155 3.17261652702200 2.31282092945237
#> # 2024-11-27 19:25:28 UTC: Iteration 4 of 4
#> # experimentsUsedSoFar: 824
#> # remainingBudget: 176
#> # currentBudget: 176
#> # nbConfigurations: 21
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 8| 21| 155| 0.4063383880| 21|00:00:00| NA| NA| NA|
#> |x| 5| 21| 155| 0.4063383880| 39|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 21| 155| 0.4063383880| 57|00:00:00|+1.00|1.00|0.0000|
#> |x| 1| 21| 155| 0.4063383880| 75|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 3| 155| 0.4063383880| 93|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 155
#> Best-so-far configuration: 66 mean value: 0.4358445903
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 66 66 3.12659798739228 2.47875866510817 32
#>
#> # 2024-11-27 19:25:28 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 66 3.12659798739228 2.47875866510817
#> 32 3.37967336177826 2.20616519451141
#> 155 3.17261652702200 2.31282092945237
#> # 2024-11-27 19:25:29 UTC: Iteration 5 of 5
#> # experimentsUsedSoFar: 917
#> # remainingBudget: 83
#> # currentBudget: 83
#> # nbConfigurations: 10
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 9| 10| 189| 0.3997731773| 10|00:00:00| NA| NA| NA|
#> |x| 1| 10| 189| 0.3997731773| 17|00:00:00|+1.00|1.00|0.0000|
#> |x| 8| 10| 189| 0.3997731773| 24|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 10| 189| 0.3997731773| 32|00:00:00|+1.00|1.00|0.0000|
#> |-| 6| 4| 189| 0.3997731773| 40|00:00:00|+1.00|1.00|0.0000|
#> |!| 7| 4| 189| 0.3997731773| 41|00:00:00|+1.00|1.00|0.0000|
#> |!| 2| 4| 189| 0.3997731773| 42|00:00:00|+1.00|1.00|0.0000|
#> |!| 3| 4| 189| 0.3997731773| 43|00:00:00|+1.00|1.00|0.0000|
#> |-| 5| 1| 189| 0.3997731773| 44|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 189 mean value: 0.3997731773
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 189 189 3.14982869495763 2.308084599866 155
#>
#> # 2024-11-27 19:25:29 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 189 3.14982869495763 2.308084599866
#> # 2024-11-27 19:25:30 UTC: Iteration 6 of 6
#> # experimentsUsedSoFar: 961
#> # remainingBudget: 39
#> # currentBudget: 39
#> # nbConfigurations: 4
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 10| 4| 192| 0.3988000631| 4|00:00:00| NA| NA| NA|
#> |x| 2| 4| 192| 0.3988000631| 7|00:00:00|+1.00|1.00|0.0000|
#> |x| 9| 4| 192| 0.3988000631| 10|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 4| 192| 0.3988000631| 13|00:00:00|+1.00|1.00|0.0000|
#> |-| 3| 2| 192| 0.3988000631| 16|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 192
#> Best-so-far configuration: 189 mean value: 0.3997731773
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 189 189 3.14982869495763 2.308084599866 155
#>
#> # 2024-11-27 19:25:30 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 189 3.14982869495763 2.30808459986600
#> 192 3.14194228084746 2.30492865459122
#> # 2024-11-27 19:25:31 UTC: Iteration 7 of 7
#> # experimentsUsedSoFar: 977
#> # remainingBudget: 23
#> # currentBudget: 23
#> # nbConfigurations: 4
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 10| 4| 192| 0.3988000631| 2|00:00:00| NA| NA| NA|
#> |x| 2| 4| 192| 0.3988000631| 4|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 4| 192| 0.3988000631| 7|00:00:00|+1.00|1.00|0.0000|
#> |x| 8| 4| 192| 0.3988000631| 10|00:00:00|+1.00|1.00|0.0000|
#> |-| 3| 2| 192| 0.3988000631| 12|00:00:00|+1.00|1.00|0.0000|
#> |.| 9| 2| 192| 0.3988000631| 12|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best configuration for the instances in this race: 192
#> Best-so-far configuration: 189 mean value: 0.3997731773
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 189 189 3.14982869495763 2.308084599866 155
#>
#> # 2024-11-27 19:25:31 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 189 3.14982869495763 2.30808459986600
#> 192 3.14194228084746 2.30492865459122
#> # 2024-11-27 19:25:31 UTC: Stopped because there is not enough budget left to race more than the minimum (3).
#> # You may either increase the budget or set 'minNbSurvival' to a lower value.
#> # Iteration: 8
#> # nbIterations: 8
#> # experimentsUsedSoFar: 989
#> # timeUsed: 0
#> # remainingBudget: 11
#> # currentBudget: 11
#> # number of elites: 2
#> # nbConfigurations: 3
#> # Total CPU user time: 9.43, CPU sys time: 0.052, Wall-clock time: 9.484
#> # 2024-11-27 19:25:32 UTC: Starting post-selection:
#> # Configurations selected: 189, 192, 66, 32, 155.
#> # Pending instances: 0, 3, 1, 1, 1.
#> # Seed: 2011091433
#> # Configurations: 5
#> # Available experiments: 11
#> # minSurvival: 1
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded.
#>
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> |x| 6| 5| 192| 0.3988000631| 1|00:00:00| NA| NA| NA|
#> |.| 2| 5| 192| 0.3988000631| 1|00:00:00|+1.00|1.00|0.0000|
#> |.| 8| 5| 192| 0.3988000631| 1|00:00:00|+1.00|1.00|0.0000|
#> |.| 9| 5| 192| 0.3988000631| 1|00:00:00|+1.00|1.00|0.0000|
#> |.| 5| 5| 192| 0.3988000631| 1|00:00:00|+1.00|1.00|0.0000|
#> |.| 4| 5| 192| 0.3988000631| 1|00:00:00|+1.00|1.00|0.0000|
#> |!| 10| 5| 192| 0.3988000631| 4|00:00:00|+1.00|1.00|0.0000|
#> |!| 7| 5| 192| 0.3988000631| 5|00:00:00|+1.00|1.00|0.0000|
#> |.| 3| 5| 192| 0.3988000631| 5|00:00:00|+1.00|1.00|0.0000|
#> |-| 1| 1| 192| 0.3988000631| 6|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 192 mean value: 0.3988000631
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 192 192 3.14194228084746 2.30492865459122 189
#>
#> # 2024-11-27 19:25:42 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 192 3.14194228084746 2.30492865459122
#> # Total CPU user time: 18.967, CPU sys time: 0.392, Wall-clock time: 19.375
#> x1 x2 configuration x_domain y
#> <num> <num> <int> <list> <num>
#> 1: 3.141942 2.304929 192 <list[2]> 0.3988001
# best scoring configuration
instance$result
#> x1 x2 configuration x_domain y
#> <num> <num> <int> <list> <num>
#> 1: 3.141942 2.304929 192 <list[2]> 0.3988001
# all evaluations
as.data.table(instance$archive)
#> x1 x2 y race step instance configuration
#> <num> <num> <num> <int> <int> <int> <int>
#> 1: 4.785923 0.7999151 10.9992463 1 1 2 1
#> 2: -2.714076 8.2999152 10.0905500 1 1 2 2
#> 3: 8.535923 12.0499152 108.4497127 1 1 2 3
#> 4: 1.035924 4.5499152 14.8980988 1 1 2 4
#> 5: 2.910923 10.1749152 60.1449255 1 1 2 5
#> ---
#> 991: 3.379673 2.2061652 0.6807328 8 1 1 32
#> 992: 3.126598 2.4787587 0.4358446 8 1 1 66
#> 993: 3.172617 2.3128209 0.4063384 8 1 1 155
#> 994: 3.141942 2.3049287 0.3988001 8 1 6 192
#> 995: 3.141942 2.3049287 0.3988001 8 1 2 192
#> timestamp batch_nr
#> <POSc> <int>
#> 1: 2024-11-27 19:25:23 1
#> 2: 2024-11-27 19:25:23 1
#> 3: 2024-11-27 19:25:23 1
#> 4: 2024-11-27 19:25:23 1
#> 5: 2024-11-27 19:25:23 1
#> ---
#> 991: 2024-11-27 19:25:42 41
#> 992: 2024-11-27 19:25:42 41
#> 993: 2024-11-27 19:25:42 41
#> 994: 2024-11-27 19:25:42 42
#> 995: 2024-11-27 19:25:42 43
# }