It is indeed JIT compilation latency, and currently there is no good way to avoid that. (You can start julia with julia --compile=min to get rid of much of the latency but then you won’t get the performance of JIT compiled code and that particular commandline option often has weird behavior from my experience.)
The whole issue of JIT latency has been extensively discussed elsewhere (see here and here for example).