I’m running some ML experiments to draw certain conclusions about whether my new method works as expected. I would like, so to speak, to run multiple experiments in the same file, as it would otherwise be cumbersome. The idea is to use a syntax similar to what Test offers; however, I don’t want to create a test, I just want to run an experiment and extract statistics from it. Is there any framework for this that I’m unaware of?
The idea would be to have something similar to this but I’m curious if there’s a dedicated package that offers specific functionalities for this purpose.
@testset "vanilla_gan" begin
@testset "Origin N(0,1)" begin
noise_model = Normal(0.0f0, 1.0f0)
n_samples = 10000
@testset "N(0,1) to N(23,1)" begin
gen = Chain(Dense(1, 7), elu, Dense(7, 13), elu, Dense(13, 7), elu, Dense(7, 1))
dscr = Chain(
Dense(1, 11), elu, Dense(11, 29), elu, Dense(29, 11), elu, Dense(11, 1, σ)
)
target_model = Normal(23.0f0, 1.0f0)
hparams = HyperParamsVanillaGan(;
data_size=100,
batch_size=1,
epochs=1000,
lr_dscr=1e-4,
lr_gen=1e-4,
dscr_steps=5,
gen_steps=1,
noise_model=noise_model,
target_model=target_model,
)
train_vanilla_gan(dscr, gen, hparams)
ksd = KSD(noise_model, target_model, n_samples, 20:0.1:25)
mae = MAE(noise_model, x -> x .+ 23, n_samples)
mse = MSE(noise_model, x -> x .+ 23, n_sample)
end
...