Best Practices for Using "Off-the-Shelf" ML Solutions?

General Usage
,
1

Hi folks! :wave:

I am not a formal ML researcher (I couldn’t tell you how to build a Neural Network from scratch) but I know about ML models and when what species of models make sense for what problems.

A question that I have been struggling with is if I want to use ML models (like from MLJ, (F)Lux, Pytorch, etc.) in my work but do not understand the fine technical details of a model, am I sufficiently qualified to do so? I know some packages in Julia and Python ecosystems brand themselves as “off-the-shelf” ML solutions, but I just worry my lacking of knowledge on the technical side could lead to errors in my potential analysis.

What is considered best practices here in this space? Am I unqualified to use these “off-the-shelf” solutions or would I be the target audience?

Thank you!

~ tcp :deciduous_tree:

2 Likes
2

Just try it. If that is too big of a time investment, try it on a sliced down problem. Actually trying a toy problem first is almost always a good idea.

ML solution creating misleading results can absolutely happen. For many models there are no theoretical guarantees about errors or sane results.
In addition to standard best practices like regularization, train test split etc. I find the following often very useful:

  • You need an intuition about how you expect the solution to look like. Does the ML produce sane results?
  • Try a couple of models or vary some parameters. Do solutions roughly look the same or vary wildly?
  • Simplify your problem until you have good intuition or exact solutions. Run ML on that and get a feel for the errors it produces there.
  • Do you have some less precise/limited/slower ways to solve your problem or some properties of the solution? Compare these against ML.
  • Test the limits of your ML solution. Tweak some examples and see when then model produces bad results.
  • Depending on your domain you might now some invariants that should hold. Like conservation of energy etc. check these.
  • Depending on the model, there may be further specific tools to gain insights, find out about these.

I am pretty sure you are a target audience. That means you can apply models without knowing their inner workings, you just can’t trust them blindly. You need to be critical and apply checks like the above. These don’t require much ML knowledge, but instead domain knowledge.

2 Likes