# Hyperparameters optimization packages for ML application in Julia

**URL:** <https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519>\
**Category:** Machine Learning\
**Tags:** package\
**Created:** [January 18, 2020, 11:27am UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519 "2020-01-18T11:27:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Simon\_Besnard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simon_besnard/32/9730_2.png) [@Simon\_Besnard](https://discourse.julialang.org/u/Simon_Besnard)\
**Post date:** [January 18, 2020, 11:27am UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/1 "2020-01-18T11:27:03Z")

</div>

Hi all,

I am currently working with Flux.jl package for deep learning model implementation. I was wondering if some of you have had experience with some hyperparameters optimization packages (e.g. using hyperbands optimization algorithm) that can be nicely coupled with Flux.jl.

Thanks for sharing your experiences.

Simon.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [January 18, 2020, 3:10pm UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/2 "2020-01-18T15:10:02Z")

</div>

An implementation of hyperband is provided in [GitHub - baggepinnen/Hyperopt.jl: Hyperparameter optimization in Julia.](https://github.com/baggepinnen/Hyperopt.jl#hyperband)

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [January 18, 2020, 6:10pm UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/3 "2020-01-18T18:10:06Z")

</div>

Another option: [https://github.com/DrChainsaw/NaiveGAflux.jl](https://github.com/DrChainsaw/NaiveGAflux.jl)

I guess it is has some half accidental similarity to hyperband in the sense that it tries to focus training efforts on promising candidates.

If one does not believe in parameter reuse it should be possible to build hyperband on it, e.g. by using elite selection of half the population size until there is only one candidate.

---

<div class="post-metadata">

**Author:** ![luboshanus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luboshanus/32/12163_2.png) [@luboshanus](https://discourse.julialang.org/u/luboshanus)\
**Post date:** [February 27, 2020, 6:20pm UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/4 "2020-02-27T18:20:23Z")

</div>

Hi @baggepinnen,

I’d like to ask you if you could put here an example using `Hyperband` and inside of it is a train\_network. Please, could you do this for me? 🙂 Thank you.

I am getting an error when trying to modify working example with `RandomSample` or `LHSampler` , but I get:

```julia
ERROR: LoadError: syntax: Global method definition around /Users/ies503/.julia/packages/Hyperopt/utzuO/src/samplers.jl:191 needs to be placed at the top level, or use "eval".
Stacktrace:
...
... 

```

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [February 27, 2020, 7:06pm UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/5 "2020-02-27T19:06:14Z")

</div>

_The macro `@phyperopt` works in the same way as `@hyperopt` but distributes all computation on available workers. The usual caveats apply, code must be loaded on all workers etc._

You make it sound so easy! I think this is VERY impressive.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 28, 2020, 2:44am UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/6 "2020-02-28T02:44:32Z")

</div>

Such an example would depend completely on what tools you use to train networks, as Hyperband requires you to save a meaningful state and also resume training from this state. You will find an example that shows how to do this using Optim.jl [here](https://github.com/baggepinnen/Hyperopt.jl/blob/5e7ed134dceb341bef1f79ed2eac9c4ef44e70a4/test/runtests.jl#L139)

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 28, 2020, 2:46am UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/7 "2020-02-28T02:46:10Z")

</div>

Thanks 🙂 It’s really Julia making it easy. Distributing using `Distributed` is significantly easier than threading, as one does not have to worry about thread safety etc.

---

<div class="post-metadata">

**Author:** ![TheLateKronos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thelatekronos/32/12824_2.png) [@TheLateKronos](https://discourse.julialang.org/u/TheLateKronos)\
**Post date:** [April 5, 2022, 6:37am UTC](https://discourse.julialang.org/t/hyperparameters-optimization-packages-for-ml-application-in-julia/33519/8 "2022-04-05T06:37:20Z")

</div>

> [@baggepinnen](#):
>
> as Hyperband requires you to save a meaningful state and also resume training from this state

I found this comment very helpful in my attempt to use your package. Context like this would be much appreciated in the docs (readme). I would be happy to help write, but I don’t really understand enough to be of much use right now I think.