Threads are not speeding up computation a lot

Did you try to launch Julia with the parameter:

--gcthreads=8,1

?

8 would be the number of threads for the GC, set it to the number of physical cores of your machine or one or two less.