Hi,
Not sure what the issue could be, but a starting point could be to look at how the loss is evolving during training. You can use callbacks to monitor this when using the train!() function.
It might also be helpful if you could describe what type of data you are dealing with. Do you have discrete or continuous labels? Is it a balanced dataset?