OptimizerIrace
class that implements iterated racing. Calls
irace::irace()
from package irace.
Source
Lopez-Ibanez M, Dubois-Lacoste J, Caceres LP, Birattari M, Stuetzle T (2016). “The irace package: Iterated racing for automatic algorithm configuration.” Operations Research Perspectives, 3, 43--58. doi:10.1016/j.orp.2016.09.002 .
Parameters
instances
list()
A list of instances where the configurations executed on.targetRunnerParallel
function()
A function that executes the objective function with a specific parameter configuration and instance. A default function is provided, see section "Target Runner and Instances".
For the meaning of all other parameters, see irace::defaultScenario()
. Note
that we have removed all control parameters which refer to the termination of
the algorithm. Use TerminatorEvals instead. Other terminators do not work
with OptimizerIrace
.
Target Runner and Instances
The irace package uses a targetRunner
script or R function to evaluate a
configuration on a particular instance. Usually it is not necessary to
specify a targetRunner
function when using OptimizerIrace
. A default
function is used that forwards several configurations and instances to the
user defined objective function. As usually, the user defined function has
a xs
, xss
or xdt
parameter depending on the used Objective class.
For irace, the function needs an additional instances
parameter.
fun = function(xs, instances) {
# function to evaluate configuration in `xs` on instance `instances`
}
Archive
The Archive holds the following additional columns:
"race"
(integer(1)
)
Race iteration."step"
(integer(1)
)
Step number of race."instance"
(integer(1)
)
Identifies instances across races and steps."configuration"
(integer(1)
)
Identifies configurations across races and steps.
Result
The optimization result (instance$result
) is the best performing elite of
the final race. The reported performance is the average performance estimated
on all used instances.
Dictionary
This Optimizer can be instantiated via the dictionary
mlr_optimizers or with the associated sugar function opt()
:
mlr_optimizers$get("irace")
opt("irace")
Progress Bars
$optimize()
supports progress bars via the package progressr
combined with a Terminator. Simply wrap the function in
progressr::with_progress()
to enable them. We recommend to use package
progress as backend; enable with progressr::handlers("progress")
.
Super class
bbotk::Optimizer
-> OptimizerIrace
Examples
library(data.table)
search_space = domain = ps(
x1 = p_dbl(-5, 10),
x2 = p_dbl(0, 15)
)
codomain = ps(y = p_dbl(tags = "minimize"))
# branin function with noise
# the noise generates different instances of the branin function
# the noise values are passed via the `instances` parameter
fun = function(xdt, instances) {
a = 1
b = 5.1 / (4 * (pi^2))
c = 5 / pi
r = 6
s = 10
t = 1 / (8 * pi)
data.table(y = (
a * ((xdt[["x2"]] -
b * (xdt[["x1"]]^2L) +
c * xdt[["x1"]] - r)^2) +
((s * (1 - t)) * cos(xdt[["x1"]])) +
unlist(instances)))
}
objective = ObjectiveRFunDt$new(fun = fun, domain = domain, codomain = codomain)
instance = OptimInstanceSingleCrit$new(
objective = objective,
search_space = search_space,
terminator = trm("evals", n_evals = 1000))
# create instances of branin function
instances = rnorm(10, mean = 0, sd = 0.1)
# load optimizer irace and set branin instances
optimizer = opt("irace", instances = instances)
# modifies the instance by reference
optimizer$optimize(instance)
#> # 2022-06-30 04:24:08 UTC: Initialization
#> # Elitist race
#> # Elitist new instances: 1
#> # Elitist limit: 2
#> # nbIterations: 3
#> # minNbSurvival: 3
#> # nbParameters: 2
#> # seed: 1565435889
#> # confidence level: 0.95
#> # budget: 1000
#> # mu: 5
#> # deterministic: FALSE
#>
#> # 2022-06-30 04:24:08 UTC: Iteration 1 of 3
#> # experimentsUsedSoFar: 0
#> # remainingBudget: 1000
#> # currentBudget: 333
#> # nbConfigurations: 55
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 1| 55| 23| -7.494146773| 55|00:00:00| NA| NA| NA|
#> |x| 2| 55| 23| -7.562743570| 110|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 55| 23| -7.536060338| 165|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 55| 23| -7.525953208| 220|00:00:00|+1.00|1.00|0.0000|
#> |-| 5| 1| 23| -7.561368462| 275|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 23 mean value: -7.561368462
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 23 23 9.9085 3.866 NA
#>
#> # 2022-06-30 04:24:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 23 9.9085 3.866
#> # 2022-06-30 04:24:09 UTC: Iteration 2 of 3
#> # experimentsUsedSoFar: 275
#> # remainingBudget: 725
#> # currentBudget: 362
#> # nbConfigurations: 52
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 6| 52| 63| -9.480394539| 52|00:00:00| NA| NA| NA|
#> |x| 4| 52| 63| -9.445608218| 103|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 52| 63| -9.429700130| 154|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 52| 63| -9.476829986| 205|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 2| 63| -9.490770078| 256|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 23 mean value: -7.562007795
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 23 23 9.9085 3.866 NA
#>
#> # 2022-06-30 04:24:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 23 9.9085 3.8660
#> 63 9.2973 2.4971
#> # 2022-06-30 04:24:09 UTC: Iteration 3 of 3
#> # experimentsUsedSoFar: 531
#> # remainingBudget: 469
#> # currentBudget: 469
#> # nbConfigurations: 60
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 7| 60| 142| -9.735572885| 60|00:00:00| NA| NA| NA|
#> |x| 1| 60| 142| -9.617238779| 119|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 60| 142| -9.578289091| 177|00:00:00|+1.00|1.00|0.0000|
#> |x| 2| 60| 142| -9.592741385| 235|00:00:00|+1.00|1.00|0.0000|
#> |-| 3| 3| 142| -9.571683463| 293|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 63 mean value: -9.501313186
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 63 63 9.2973 2.4971 23
#>
#> # 2022-06-30 04:24:09 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 63 9.2973 2.4971
#> 23 9.9085 3.8660
#> 142 9.4202 2.5383
#> # 2022-06-30 04:24:09 UTC: Iteration 4 of 4
#> # experimentsUsedSoFar: 824
#> # remainingBudget: 176
#> # currentBudget: 176
#> # nbConfigurations: 21
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 8| 21| 174| -9.584832284| 21|00:00:00| NA| NA| NA|
#> |x| 4| 21| 174| -9.542903425| 39|00:00:00|+1.00|1.00|0.0000|
#> |x| 3| 21| 174| -9.524614490| 57|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 21| 174| -9.570553923| 76|00:00:00|+1.00|1.00|0.0000|
#> |-| 1| 4| 174| -9.556341043| 94|00:00:00|+1.00|1.00|0.0000|
#> |!| 7| 4| 174| -9.586310491| 95|00:00:00|+1.00|1.00|0.0000|
#> |!| 6| 4| 174| -9.584058593| 97|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 1| 174| -9.590636658| 98|00:00:00| NA| NA| NA|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 174 mean value: -9.590636658
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 174 174 9.4127 2.5225 142
#>
#> # 2022-06-30 04:24:10 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 174 9.4127 2.5225
#> # 2022-06-30 04:24:10 UTC: Iteration 5 of 5
#> # experimentsUsedSoFar: 922
#> # remainingBudget: 78
#> # currentBudget: 78
#> # nbConfigurations: 8
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 9| 8| 186| -9.628856861| 8|00:00:00| NA| NA| NA|
#> |x| 5| 8| 186| -9.669385870| 15|00:00:00|+1.00|1.00|0.0000|
#> |x| 7| 8| 186| -9.692157376| 22|00:00:00|+1.00|1.00|0.0000|
#> |x| 8| 8| 186| -9.665711767| 29|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 2| 186| -9.660214568| 36|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 174 mean value: -9.594711941
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 174 174 9.4127 2.5225 142
#>
#> # 2022-06-30 04:24:10 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 174 9.4127 2.5225
#> 186 9.4343 2.5283
#> # 2022-06-30 04:24:10 UTC: Iteration 6 of 6
#> # experimentsUsedSoFar: 958
#> # remainingBudget: 42
#> # currentBudget: 42
#> # nbConfigurations: 6
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 10| 6| 191| -9.701155555| 6|00:00:00| NA| NA| NA|
#> |x| 8| 6| 191| -9.644190578| 10|00:00:00|+1.00|1.00|0.0000|
#> |x| 4| 6| 191| -9.597249679| 15|00:00:00|+1.00|1.00|0.0000|
#> |x| 6| 6| 191| -9.591172390| 20|00:00:00|+1.00|1.00|0.0000|
#> |-| 2| 3| 191| -9.600753198| 24|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 174 mean value: -9.605116971
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 174 174 9.4127 2.5225 142
#>
#> # 2022-06-30 04:24:10 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 174 9.4127 2.5225
#> 186 9.4343 2.5283
#> 191 9.4337 2.5179
#> # 2022-06-30 04:24:10 UTC: Iteration 7 of 7
#> # experimentsUsedSoFar: 982
#> # remainingBudget: 18
#> # currentBudget: 18
#> # nbConfigurations: 4
#> # Markers:
#> x No test is performed.
#> c Configurations are discarded only due to capping.
#> - The test is performed and some configurations are discarded.
#> = The test is performed but no configuration is discarded.
#> ! The test is performed and configurations could be discarded but elite configurations are preserved.
#> . All alive configurations are elite and nothing is discarded
#>
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> |x| 11| 4| 191| -9.701155555| 4|00:00:00| NA| NA| NA|
#> |x| 10| 4| 191| -9.701155555| 5|00:00:00|+1.00|1.00|0.0000|
#> |x| 7| 4| 191| -9.713620720| 7|00:00:00|+1.00|1.00|0.0000|
#> |x| 5| 4| 191| -9.712906925| 9|00:00:00|+1.00|1.00|0.0000|
#> |-| 1| 3| 191| -9.670702107| 12|00:00:00|+1.00|1.00|0.0000|
#> |.| 6| 3| 191| -9.654408510| 12|00:00:00|+1.00|1.00|0.0000|
#> |.| 2| 3| 191| -9.652218213| 12|00:00:00|+1.00|1.00|0.0000|
#> +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
#> Best-so-far configuration: 174 mean value: -9.613630177
#> Description of the best-so-far configuration:
#> .ID. x1 x2 .PARENT.
#> 174 174 9.4127 2.5225 142
#>
#> # 2022-06-30 04:24:10 UTC: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks):
#> x1 x2
#> 174 9.4127 2.5225
#> 186 9.4343 2.5283
#> 191 9.4337 2.5179
#> # 2022-06-30 04:24:10 UTC: Stopped because there is not enough budget left to race more than the minimum (3)
#> # You may either increase the budget or set 'minNbSurvival' to a lower value
#> # Iteration: 8
#> # nbIterations: 8
#> # experimentsUsedSoFar: 994
#> # timeUsed: 0
#> # remainingBudget: 6
#> # currentBudget: 6
#> # number of elites: 3
#> # nbConfigurations: 3
#> x1 x2 configuration x_domain y
#> 1: 9.4127 2.5225 174 <list[2]> -9.61363
# best scoring configuration
instance$result
#> x1 x2 configuration x_domain y
#> 1: 9.4127 2.5225 174 <list[2]> -9.61363
# all evaluations
as.data.table(instance$archive)
#> x1 x2 y race step instance configuration
#> 1: 6.5397 14.3165 183.622162 1 1 6 1
#> 2: 0.5583 1.4238 22.139968 1 1 6 2
#> 3: 5.5075 13.7228 164.952862 1 1 6 3
#> 4: -4.9159 3.1294 192.931631 1 1 6 4
#> 5: -0.0423 5.5956 9.914861 1 1 6 5
#> ---
#> 990: 9.4337 2.5179 -9.710766 7 1 8 191
#> 991: 9.4198 2.5237 -9.709481 7 1 8 194
#> 992: 9.4343 2.5283 -9.501032 7 1 6 186
#> 993: 9.4337 2.5179 -9.501883 7 1 6 191
#> 994: 9.4198 2.5237 -9.500599 7 1 6 194
#> timestamp batch_nr
#> 1: 2022-06-30 04:24:08 1
#> 2: 2022-06-30 04:24:08 1
#> 3: 2022-06-30 04:24:08 1
#> 4: 2022-06-30 04:24:08 1
#> 5: 2022-06-30 04:24:08 1
#> ---
#> 990: 2022-06-30 04:24:10 37
#> 991: 2022-06-30 04:24:10 37
#> 992: 2022-06-30 04:24:10 38
#> 993: 2022-06-30 04:24:10 38
#> 994: 2022-06-30 04:24:10 38