# Best ways to do hyper-parameter tuning
**URL:** https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010
**Category:** Machine Learning
**Tags:** mlj, tuning
**Created:** [January 6, 2020, 12:14am UTC](https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010 "2020-01-06T00:14:31Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)
#### Post date: [January 6, 2020, 12:14am UTC](https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010/1 "2020-01-06T00:14:31Z")
I’d like to tune a model in [JLBoost](https://github.com/xiaodaigh/JLBoost.jl) (an awesome, all Julia package by @xiaodai builds on XGBoost, LightGBM, & Catboost).
```nohighlight
using RDatasets, DataFrames, JLBoost, MLJ;
d = dataset("MASS", "Boston");
train, test = partition(eachindex(d[:,1]), .7, rng=333);
target = :MedV;
features = setdiff(names(d), [target]);
warm_start = fill(0.0, nrow(d));
dt = d[train,:]; dh = d[test,:];
yt = d[train, target]; yh = d[test, target]; y = d[!, target];
using LossFunctions: L2DistLoss;
loss = L2DistLoss();
#
g_η = .3 ∪ range(0, 1, length=3)
g_λ = 0 ∪ range(0, 1, length=3)
g_γ = 0 ∪ range(0, 1, length=3)
g_md = 6 ∪ (1:10)
G = Iterators.product(g_η, g_λ, g_γ, g_md);
sc=[]; p=[];
@time for g in G
m = jlboost(dt, target, features, warm_start, loss;
eta = g[1],
lambda = g[2],
gamma = g[3],
max_depth = g[4] )
ŷ = predict(m, dh)
push!(sc, rms(ŷ, yh) )
push!(p, ( g[1], g[2], g[3], g[4]) )
end
minimum(sc)
p[findall(x->x==minimum(sc), sc)]
```
This does grid search over the entire grid G.
Q1: how can I create a new grid, G1, which is a random subset of G w/ 30 elements?
However, I also wanna include all the default hyper-parameters in G1 as well.
Q2: does anyone know all the options currently available in Julia for tuning hyper-parameters?
Currently the only package tagged hyper-parameter optimization in ([https://pkg.julialang.org/docs/](https://pkg.julialang.org/docs/)) is @baggepinnen’s [Hyperopt.jl](https://github.com/baggepinnen/Hyperopt.jl). It looks promising but I can’t load it bc it requires CMake which isn’t building right now.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [January 6, 2020, 12:28am UTC](https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010/2 "2020-01-06T00:28:20Z")
Thanks.
> **[GitHub - xiaodaigh/JLBoostMLJ.jl: MLJ.jl interface for JLBoost.jl](https://github.com/xiaodaigh/JLBoostMLJ.jl)**
>
> MLJ.jl interface for JLBoost.jl. Contribute to xiaodaigh/JLBoostMLJ.jl development by creating an account on GitHub.
I have an example of using MLJ to do the hyper parameters search. The JLBoostMLJ is undergoing registration so you need to install by providing the full URL when adding.
It’s not exactly what u r asking for, but I think it will work.
The packages is WIP so appreciate any feedback on usability etc. Thanks.
---
### Author:  [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)
#### Post date: [January 6, 2020, 12:38am UTC](https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010/3 "2020-01-06T00:38:36Z")
Thanks.
In the readme, I think you do the same thing as what I’m already doing. Which is grid search over the entire grid.
```nohighlight
using JLBoost, JLBoostMLJ, MLJ
jlb = JLBoostClassifier()
r1 = range(jlb, :nrounds, lower=1, upper = 6)
r2 = range(jlb, :max_depth, lower=1, upper = 6)
r3 = range(jlb, :eta, lower=0.1, upper=1.0)
tm = TunedModel(model = jlb, ranges = [r1, r2, r3], measure = cross_entropy)
m = machine(tm, X, y_cate)
```
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [January 6, 2020, 2:28am UTC](https://discourse.julialang.org/t/best-ways-to-do-hyper-parameter-tuning/33010/4 "2020-01-06T02:28:30Z")
I think you just need to use `StatsBase: sample` and `collect` on the grid before your loop. You need to add in the default parameters manually though.
```julia
using StatsBase: sample
# sample 30
Gs = sample(collect(G), 30)
sc=[]; p=[];
@time for g in Gs
m = jlboost(dt, target, features, warm_start, loss;
eta = g[1],
lambda = g[2],
gamma = g[3],
max_depth = g[4] )
ŷ = predict(m, dh)
push!(sc, rms(ŷ, yh) )
push!(p, ( g[1], g[2], g[3], g[4]) )
end
minimum(sc)
p[findall(x->x==minimum(sc), sc)]
```