Working on a package that does some rather generic things with types, I’ve been thinking of adding some type stability checking, performance benchmarks for regression detection, unexpected allocation detections, …
I get the impression that while the packages to do this kind of work are functional and do the job quite well (DispatchDoctor, Cthulhu, AllocCheck, JET… depending on the goal), if one doesn’t integrate them from the get-go into a workflow it is a LOT harder to sift through errors and issues and to know what to prioritise, especially in code that occasionally does somewhat generic manipulation on user-defined types.
I get the impression starting small, making sure new code doesn’t have issues and perhaps then slowly and iteratively propagating stability/sane allocations starting with functions that should never allocate/never have any unexpected type quirks is a good way to go, but I’m not sure if that’s the best approach.
Are there any practical blogposts / postmortems on this topic ? I feel like the advice I see is sound, but more of a tutorial or a ‘here’s what to use’, not much about adding this analysis quite late into a package’s lifetime and having to also deal with the aforementioned problems, and Julia does have the issue that’s it doesn’t really naturally point you towards the better workflows for some things.
It’s difficult to give generic advice here, because there can be a lot of different reasons why something is type unstable or allocates GC memory. The best/most generic advice that I think exists is “think about the entry points in your code, use the analysis tools on those and dive into fixing issues found by them”. This isn’t glamorous and sadly not always easy, but I’m not sure there’s a way around it The upside is that once that’s done, you should be able to add checks like @inferred or JET.jl to your testsuite.
For more specific guidance (e.g. how to fix a specific finding), you’ll have to give a more specific example, I’m afraid.
Well, part of my question was asking for more specific examples / articles discussing their approach.
One such package I’m working on operates on graphs declared by the user, so there are some unknowns in terms of types passed through and functions that are called at each node.
This means some of the code managing these graphs tends to be quite open in terms of what types can sometimes be accepted, and it’s sometimes hard to tell the difference between, say, an allocation caused by a generic type being passed through that could be avoided through some upstream better type handling, and those that can’t (amongst other things).
My experience has been that it is best to do this incrementally, as you suggested. Ensure that each function is type stable/doesn’t allocate unnecessarily as you write it. Doing it after the fact has been more difficult for me.
Psychologically it’s hard to fix these types of problems when every function is 100x slower than it should be and is allocating megabytes it shouldn’t be. You may have to modify or rewrite most of your functions before you see significant overall performance improvement. That can be so daunting when tackled as a single large task that you never can convince yourself to do it.
Alright, thanks for the response, that aligns with my conclusion. I’ll have to find a good dynamic to incrementally improve the package sanity checks.
If ever anybody else comes across some resource showcasing how to deal with some of the more subtle difficulties one might encounter, feel free to ping me or post it here.
The upside is that once that’s done, you should be able to add checks like