# Automatic Parallelization in Julia
**URL:** https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543
**Category:** Julia at Scale
**Created:** [July 5, 2020, 1:15am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543 "2020-07-05T01:15:02Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Honza9723](https://discourse.julialang.org/u/Honza9723)
#### Post date: [July 5, 2020, 1:15am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/1 "2020-07-05T01:15:02Z")
Dear All,
I would like to ask you, whether there is some automatic parallelization tool in Julia similar to auto parallelization capabilities of Intel/gcc compilers for Fortran/C++. It would be really awesome if Julia compiler could transform standard serial codes into parallel ones!
Best,
Honza
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [July 5, 2020, 1:27am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/2 "2020-07-05T01:27:25Z")
Yeah, you can do all kinds of things. Depends on what level you call automated though. There’s things like CuArrays and DistributedArrays that recompile your code to GPUs and distributed CPUs respectively, KernelAbstractions.jl that recompiles quite a big set of Julia code to GPUs, and recently things like ModelingToolkit that will take a Julia ODE code and recompile it in a multithreaded way:
[https://mtk.sciml.ai/dev/tutorials/auto\_parallel/](https://mtk.sciml.ai/dev/tutorials/auto_parallel/)
So there’s all kinds of things you can do. You’d have to be a bit more specific.
---
### Author:  [@Honza9723](https://discourse.julialang.org/u/Honza9723)
#### Post date: [July 5, 2020, 1:46am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/3 "2020-07-05T01:46:58Z")
> [@ChrisRackauckas](#):
>
> KernelAbstractions.jl
Thank you for your reply!
I ment “highest-level”/implicit paralellism. As far, as I understand powers of Intel Fortran/C++ compiler, it basically takes a whole program, and (if auto-par option is enabled) it automatically search for parallelizable parts of whole code (loops, etc…), so programmer doesn’t have to care about explicitely declaring which part of the code should be parallelized. Especially for less experienced programmers (like me), I would guess, that good compiler optimization could provide better results than explicit parallelization.
---
### Author:  [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [July 5, 2020, 1:52am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/4 "2020-07-05T01:52:07Z")
Do you have any links for this? I’m unaware of this, but would be really interested in reading about it.
---
### Author:  [@Honza9723](https://discourse.julialang.org/u/Honza9723)
#### Post date: [July 5, 2020, 2:01am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/5 "2020-07-05T02:01:50Z")
There is some description by Intel.
> **[Computers, Monitors & Technology Solutions | Dell USA](https://www.dell.com/)**
>
> Shop the latest Dell computers & technology solutions. Laptops, desktops, gaming pcs, monitors, workstations & servers. FREE & FAST DELIVERY
Also wikipedia article is pretty cool.
> **[Automatic parallelization](https://en.wikipedia.org/wiki/Automatic_parallelization)**
>
> Automatic parallelization, also auto parallelization, or autoparallelization refers to converting sequential code into multi-threaded and/or vectorized code in order to use multiple processors simultaneously in a shared-memory multiprocessor (SMP) machine. Fully automatic parallelization of sequential programs is a challenge because it requires complex program analysis and the best approach may depend upon parameter values that are not known at compilation time.
> The programming control structu...
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [July 5, 2020, 3:05am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543/6 "2020-07-05T03:05:08Z")
As far as I understand, the general philosophy that’s been taken so far by the Julia developers is that an optimization should only be automatically applied if they know for sure that
1. It’s correct / safe to apply the optimization
2. The optimization won’t accidentally hurt performance.
Unfortunately, implicit multi-threading makes both of the above criteria _very_ difficult to satisfy. Even if the safety / correctness concern were satisfied (which is not trivial to do), multi-threading has a lot of overhead. The general heuristic is that it takes about 1 microsecond to spawn a multi-threaded task in Julia which is on the order of 1000 CPU cycles. This means that if I write
```julia
for i in 1:N
f(i)
end
```
if it takes less than ~10 microseconds to run that loop, it was probably a mistake to try and multi-thread it. However, the amount of time the loop takes to run depends not only on `N`, but the details of `f`. Knowledge about how to handle this right is not something our compiler currently has or is likely to have anytime soon.
Instead, we generally insist that the programmer opts in to optimizations like multi-threading explicitly because they know more about their program than the compiler. However, we generally try to make it very easy to opt into these sorts of things which is where things like the performance annotations in base (`Threads.@threads`, `@simd`, `@fastmath`, etc.), and various packages like KernelAbstractions.jl, LoopVectorization.jl and ThreadsX.jl come in.