I can’t figure out how to avoid runtime dispatch when using SQLite (or understand if the profiler is indicating incorrectly). I ran @profview, and it’s showing most of the time in SQLite.getvalue, I think the call to sqlitevalue. The profiler shows tags runtime-dispatch, GC on the getvalue method. I tried passing in strict when running the query but that didn’t seem to help. I rewrote the code with explicit types all the way down to the function sqlite3_column_int64. Are external calls like that inherently runtime dispatch (or marked that way by the profiler)?
This case is particularly tough with sqlite because there’s never a guarantee that column values will be of any one specific type, which is often leveraged when using sqlite and just putting values of whatever you want in columns. So in SQLite.jl, we were originally much stronger typed, but it kept biting people and it was more frustrating to have queries error than just be slower.
Adding the strict mode should have helped somewhat, but I imagine there’s still room for improvement. The first step to improving things is to have a nice, small, reproducible benchmark we can start from. If you don’t mind opening an issue at the SQLite.jl repo, with some code that builds a table and queries, I can find some time to help look into what we can do. One idea off the top of my head is that we could do better about manually “unrolling” the most common types so we can avoid the dynamic dispatch, even in the non-strict case. This would most likely be in SQLite.getvalue and we’d just include a bit if-elseif-else block to check for the most common types. This sort of approach has worked really well in other data packages. If anyone’s up for giving a PR a shot, I’d be happy to review and help push things forward.