I am trying to use Flux for the first time on my dataset. I am using iris.jl as a starting point, just adapting slightly to my dataset.
I always get the error of loss is NaN, regardless of the loss function that I choose.
I know the problem is my data, as I have run the exact same pipeline on other toy datasets without any issue.
Is there a way to set verbose=true in Flux.train to get a bit more info of what is going on? Sorry, I cannot find this in the documentation/source code.
Sadly I cannot share this data, so I am testing different subsets at the moment. If anyone has a clue of what could be the issue, I would appreciate any hints.
I should add that there are no NaN in my dataset.
Thanks!
I’m not very experienced with this stuff, but I sometimes get this error when using a learning rate that is too high. Have you tried lowering it? Usually when I have such issues I trim my dataset to a very small size and check if the generated matrices look ok(like labels corresponding to features).
No, I have not tried to play around with the learning rate, thanks for the idea!
I started to take subsets of my data, but I will definitely try with the learning rate.
Thanks!!
The original learning rate that I tried was 0.5 (as in the iris example). I just tested multiple learning rates in the interval (0.00005, 0.5), but I still get the “loss is NaN” error every time. I will keep checking subsets of my data. Thanks!
There exists one subset of my data that do not produce NaN loss. So, I just need to increase it little by little until I identify the problem. Thanks!
Found the problem: my data had two constant features. So when I tested with isnan on the original matrix, I did not find any problems, but the normalized matrix had NaN because the standard deviation would be zero for these two features.
Just documenting in case someone else runs into this issue!
2 Likes