Best ways to do hyper-parameter tuning

Specific Domains Machine Learning
,
1

I’d like to tune a model in JLBoost (an awesome, all Julia package by @xiaodai builds on XGBoost, LightGBM, & Catboost).

using RDatasets, DataFrames, JLBoost, MLJ;
d = dataset("MASS", "Boston");
train, test = partition(eachindex(d[:,1]), .7, rng=333);
target = :MedV;
features = setdiff(names(d), [target]);
warm_start = fill(0.0, nrow(d));
dt = d[train,:]; dh = d[test,:];
yt = d[train, target]; yh = d[test, target];    y = d[!, target];
using LossFunctions: L2DistLoss;
loss = L2DistLoss();
#
g_η  = .3 ∪ range(0, 1, length=3)
g_λ  = 0 ∪ range(0, 1, length=3)
g_γ  = 0 ∪ range(0, 1, length=3)
g_md = 6 ∪ (1:10)
G = Iterators.product(g_η, g_λ, g_γ, g_md);
sc=[]; p=[];
@time for g in G
    m = jlboost(dt, target, features, warm_start, loss;
    eta = g[1],
    lambda = g[2],
    gamma = g[3],
    max_depth = g[4]   )
    ŷ = predict(m, dh)
    push!(sc,  rms(ŷ, yh)  )
    push!(p, (  g[1], g[2], g[3], g[4])  )
end
minimum(sc)
p[ findall(x->x==minimum(sc), sc) ]

This does grid search over the entire grid G.
Q1: how can I create a new grid, G1, which is a random subset of G w/ 30 elements?
However, I also wanna include all the default hyper-parameters in G1 as well.

Q2: does anyone know all the options currently available in Julia for tuning hyper-parameters?

Currently the only package tagged hyper-parameter optimization in (https://pkg.julialang.org/docs/) is @baggepinnen’s Hyperopt.jl. It looks promising but I can’t load it bc it requires CMake which isn’t building right now.

2

Thanks.

I have an example of using MLJ to do the hyper parameters search. The JLBoostMLJ is undergoing registration so you need to install by providing the full URL when adding.

It’s not exactly what u r asking for, but I think it will work.

The packages is WIP so appreciate any feedback on usability etc. Thanks.

1 Like
3

Thanks.
In the readme, I think you do the same thing as what I’m already doing. Which is grid search over the entire grid.

using JLBoost, JLBoostMLJ, MLJ
jlb = JLBoostClassifier()
r1 = range(jlb, :nrounds, lower=1, upper = 6)
r2 = range(jlb, :max_depth, lower=1, upper = 6)
r3 = range(jlb, :eta, lower=0.1, upper=1.0)
tm = TunedModel(model = jlb, ranges = [r1, r2, r3], measure = cross_entropy)
m = machine(tm, X, y_cate)
4

I think you just need to use StatsBase: sample and collect on the grid before your loop. You need to add in the default parameters manually though.

using StatsBase: sample
# sample 30
Gs = sample(collect(G), 30)

sc=[]; p=[];
@time for g in Gs
    m = jlboost(dt, target, features, warm_start, loss;
    eta = g[1],
    lambda = g[2],
    gamma = g[3],
    max_depth = g[4]   )
    ŷ = predict(m, dh)
    push!(sc,  rms(ŷ, yh)  )
    push!(p, (  g[1], g[2], g[3], g[4])  )
end
minimum(sc)
p[ findall(x->x==minimum(sc), sc) ]
1 Like