I am using RandomizedsearchCV to fine tune hyperparameters. I think the default scoring is based on “accuracy”.
-
However I wish to specify a different scoring function such as “recall”. I have been able to use it successfully in Python but any attempt to specify a scoring parameter results in an error in Julia. see below.
-
Moreover everytime I run the model I get a different set of best parameters. I have specified the seed value to return the same parameters but still getting different set of best parameters everytime.
using Distributed,DecisionTree
using StatsBase,ScikitLearn
rmprocs(procs())
c=addprocs(3)
@everywhere using DecisionTree,ScikitLearn,StatsBase
using ScikitLearn.GridSearch: GridSearchCV, RandomizedSearchCV
using Printf,Statistics
const n_iters = 10
sampler(a)=StatsBase.sample(a,n_iters)
param_dist = Dict("pruning_purity_threshold"=> sampler(0.5:0.001:1.0),
"max_depth"=> sampler(4:50),
"min_samples_leaf"=> sampler(1:11),
"min_samples_split"=> sampler(2:22));#,
# "min_purity_increase" =>[0.0001]);
# build a classifier
Random.seed!(123);
clf = DecisionTree.DecisionTreeClassifier()
# run randomized search
nfold=5
random_search = RandomizedSearchCV(clf, param_dist, n_iter=n_iters, random_state=MersenneTwister(123),cv=nfold);
fit!(random_search, X, y)
---------------------------------------
random_search2 = RandomizedSearchCV(clf, param_dist, n_iter=n_iters, random_state=MersenneTwister(123),cv=nfold,
scoring="accuracy_score");
fit!(random_search2, X, y)
UndefVarError: sorted not defined
Stacktrace:
[1] get_scorer(::Symbol) at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\scorer.jl:64
[2] get_scorer at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\scorer.jl:55 [inlined]
[3] #check_scoring#82 at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\cross_validation.jl:435 [inlined]
[4] check_scoring(::DecisionTreeClassifier, ::String) at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\cross_validation.jl:432
[5] _fit!(::RandomizedSearchCV, ::Array{Float64,2}, ::Array{Int64,1}, ::Array{Any,1}) at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\grid_search.jl:258
[6] fit!(::RandomizedSearchCV, ::Array{Float64,2}, ::Array{Int64,1}) at C:\Users\chatura\.julia\packages\ScikitLearn\HK6Vs\src\grid_search.jl:748
[7] top-level scope at In[68]:3rd_place_medal: