This isn’t a fully planned feature but @tim.holy has mentioned that this becomes more possible now that invalidation are less of a problem https://github.com/JuliaLang/julia/issues/30488
1 Like
The Pkg documentations are quite a nice place to start.
Yes, I understand your point. From a not-expert final user perspective, it is nice and productive that the program install/setup itself completely. julia’s Pkg allows you to do that with as little interaction as it is practical already. But you can extended to full automatic really easily.
In short, you must ensure two things.
- That the code runs in the correct environment
- That all package get installed if not available before loading
In the ‘typical’ simple scenario, where all your dependencies are registered packages it is very simple. It is generally recommended to use one environment (Project.toml) per project, for reproducibility (for this, is even better if a Manifest.toml is provided too), version compatibility issues and portability. In this example the user will just need to do julia path/to/script.jl.
# At starting your script.jl file
import Pkg
# this solve point 1
Pkg.activate(Base.current_project(@__DIR__))
# this solve point 2
Pkg.instantiate()
# Do your stuff
What does it do?
Point 1 line
@__DIR__: expands to the path of the dir containing the file (script.jl).
current_project: this look for a project file in the dir and parents.
This is more robust than just activate(@__DIR__) because now you can move the script file freely to a subfolder relative to the environment root.
activate: tells Pkg which environment to use (if everything was ok it must be your project environment).
Point 2 line
instantiate: install all the needed packages given the information in the environment (Project.toml and/or Manifest.toml).
This three lines are enough for many common scenarios and are quite flexible, which say about how cool julia’s Pkg is…
9 Likes
With those few lines you just saved me half of my work in my experimental classes where I had to do that analitically and then plot it to excel, really nice, although the syntax is still a little bit cryptic to me but I always take my time when learning something new, thanks!
1 Like
Could you please tell more about your major problems in python performance?
I tried and I am trying to migrate from Python to Julia several times. Julia is the wonderful language! However the time-to-first plot problem and unable to use Julia in standalone scripts really frustrates me.
From other side now Python almost solved all problems with the performance. We have NumPy for vectorized style of programming, Numba that supports typing system and JIT compilation, JAX for the JIT-ted differentiation programming.
I clearly understand the REPL style of Julia. And it is really good for the short numerical calculations! But if you develop enough large simulation in the team of developers, the REPL style sometimes more difficult.
What do you mean by this?
Well, this is probably about to trigger a never-ending thread, but basically: Numpy is great, but it is not written in Python, which reveals by itself the weakness of Python for developing fast code. So, if one is developing something “as numpy”, meaning something that must be performant and does not naturally fit into the operations of an existing package, Python is not great for that. Could you imagine writing Numpy using Numba or JAX, without relying on C++? That is what Julia is about.
Yet, this I think comes with a bit of frustration sometimes. People that are very familiar with writing their codes in Python using these libraries will try to use Julia the same way, meaning vectorizing everything and using black-box libraries, and will not experience much, or any, performance gain. And, in those cases, if the problems one is solving fit well into the functions provided by some fast implementation from a library, clearly there is not much room from improvement. It is different if you are trying to develop that fast library by yourself.
14 Likes
Thank you for the response! Before opening the discussion I want to say some words from my side. I am a real fan of the Julia and I wish to Julia a great success! My goal only is to find a way how to improve my own productivity for the scientific code. Now back to the topic.
Just a simple Unix way scenario: a) by cron (time-based job scheduler in Unix) the script is executed; b) script gets data from the API; c) script plots the data into the file and d) the script is finished.
Hmm… From my point of view the Julia primary feature is multiple dispatch. This is the great feature that makes Julia different from other languages. What does it mean?
For me it means that you can use your own just created types in already imported packages functions. It makes Julia really unique language! But run it with the high performance you need to recompile everything. And this can take an unpredictable time. And I am afraid that we can meet “butterfly effect” or may be effect of a “zip bomb”.
For me this is probably unsolvable problem. Because some kind of “binary freeze” means that we remove the multiple dispatch core feature from the Julia.
But could we do this in another way with the same performance? My answer is this is the NumPy arrays. It does not mean that we can write always NumPy-vectorized non-pythonic style. The NumPy is the core API, core interface that can link all part of codes together. Like TensorFlow based on concept of Tensors.
Yes, this is the little bit dirty. But it works
a) You can use NumPy arrays itself in vectorized style.
b) You can use old plain pythonic style loops over NumPy arrays, you only need add “@jit” Numba annotation to compile it with LLVM to the native code. Yes, in modern Python you can just add @jit to a function to get the LLVM code with native performance.
c) You can use JAX to trace all NumPy operation to convert it to differentiable programming style.
d) You can send NumPy array to any language (C, C++, Fortran) without any overhead.
This solution is not as beautiful as in julia types system with multiple dispatch. This is some kind of Unix engineering way to make things working.
1 Like
Yes, I think that the point is to make everything pleasant and elegant, not doing things that were impossible before. One can write everything in C++ directly as well.
4 Likes
This is exactly the purpose Julia’s built-in array type accomplishes. Instead of making their own somewhat numpy-compatible APIs (that’s what JAX is. It is not a drop-in replacement), Julia libraries can rely on Array/AbstractArray with zero additional overhead.
Going a bit more abstract, multiple dispatch is a means to an end and not an end in itself. Yes, part of why Julia can generate such fast code is that multiple dispatch allows for specialization on certain types. However, that does not mean that a) Julia can’t generate fast code without multiple dispatch, and b) compile-time latency will be eliminated if multiple dispatch is removed. I find it helpful to think about things this way:
Imagine every Python function you wrote was automatically run through Numba. As you can imagine, that would be painfully slow even though there is no multiple dispatch going on. Why do I bring this up? That is (conceptually) Julia’s compilation model. In this light, you can see how Julia is actually much “faster” than one would expect from naively JITing all code all the time.
Now, what if you don’t need this aggressive auto-compilation (e.g. in your cron job)? Julia exposes mechanisms for saying “I don’t care about optimization, just run the code”. See this post for a quick overview.
My intent here is not to write a “you’re holding it wrong” post WRT pre-runtime latency in Julia. I use Python almost exclusively for my own work because the current ML stack doesn’t offer enough ROI to offset its tradeoffs for my particular use-cases.
That said, I think it is good to clear up misconceptions around how Julia works and why the statement above is categorically false outside of the narrow domains that some of us work in. I’ll just close by saying that you may find Julia’s approach to be far closer to the Unix Philosophy than the “numpy-shaped island/silo per library” model in Python land 
8 Likes
It is not only a little dirty, but also requires significantly extra work.
And you cannot optimize every Python construct this way, especially when many different Python packages, functions, classes, etc. are involved.
So you may speed up only the most critical parts this way, but other parts of your program may still be slow.
5 Likes
Going back to the original post. Today I taught a class (in portuguese) on how to create a package and have an overall workflow in the development with Julia, which is available as a supplementary material for this package: GitHub - m3g/SimulacoesTemplate.jl: Template de uma simulação básica para a disciplina Simulações
It is a long class, but there is an index. Maybe that helps. In these pandemia times my feeling that we just have to share everything with everyone is increased.
6 Likes
Could you please tell more about your major problems in python performance?
So, for example with DynamicGrids.jl we run user-supplied functions millions to hundreds of billions of times in simulations and optimisations/sensitivity analysis. We run them threaded and now on GPUs for performance. You can use tools from other julia packages in you functions and you can write regular julia code however you like.
You can put an @inline macro on your method to force it to compiles into the framework code, meaning most of the object generation is elided by the compiler - the static arrays aren’t actually built how you would imagine, the objects aren’t actually rebuilt and allocated anywhere, the native code output is just a bunch of math on registers. You can compile whole chains of models defined in separate packages/scripts into very short native instructions. @generated functions further help compile everything specifically to exactly match your problem, with very little boilerplate to make that happen.
As far as I understand you can’t do most of this with Numpy:
- It won’t elide most of the work that is removed by the compiler in DynamicGrids.jl.
- It wont compile user-defined functions to GPU
- It wont compile loops and other iterators to match array sizes known at compile time
- It wont inline separate components from separate packages and scripts into short native code
- It’s hard to parallelise
- You can’t mix in methods from any other package - normal python code will be too slow.
- You can’t use arrays of user-defined objects with all of the above still working
So maybe some individual benchmark with Numpy will be the same as Julia, but at the scale of a complicated simulation pulling in a bunch of separate scripts and packages, it will be much slower, and in probably not really possible to do in the same way at all.
6 Likes
Sorry, I don’t want to put my python-waste to this thread anymore. 
This is the Julia forum and Julia is really awesome!
However, all problems that you are mentioned above have the clear solution in Python. NumPy is the classical solution, but Python is evolving. For example, if you want to compile you own function to the GPU or make it parallel, it could be done with Numba. Here is the example of the custom function with automatic parallelization compiled to the native code
@numba.jit(nopython=True, parallel=True)
def logistic_regression(Y, X, w, iterations):
for i in range(iterations):
w -= np.dot(((1.0 /
(1.0 + np.exp(-Y * np.dot(X, w)))
- 1.0) * Y), X)
return w
In additional to Numba you can use JAX for the differentiation programming. Or as a modern approach you can transform you task to the computational graph and use PyToch or TensorFlow. Now the PyTorch becomes popular for solving classical problems like ODE.
I am sure that any code in modern Python in worst case could be slow, but not more than 2-3 times slow in compare to the fastest ideal case. Not more. Not centuries.
If you really have problems with python, I can help to optimize your Python code. In my own mathematical modeling I am solving ODE and PDE in Python, and this solutions is really fast on the level of the native code.
Anyway, this is the Julia forum. And I am sure that you are really happy with this beautiful Julia language! And Julia really helps you with your tasks!
1 Like
I am curious, really (please do not take me wrong): if instead of that complete dot product you had to sum only the elements of the sum which are multiples of 7? What would that become?
My problem with Python and alike is that my mental effort necessary to do simple things having to adapt the code to the language is too high.
Ps. Is it easy for you to provide a complete working example of that code? It will be quite interesting for me to play with it for learning both languages.
It’s totally fair to bring up comparisons of Julia and any other language here, and there’s certainly nothing wrong with posting good examples of how you can use Python effectively. I do, however, think this probably belongs in a separate thread–we’ve kind of drifted pretty far from the original topic.
4 Likes
You seem to misunderstand the use case here… I’m not talking working to make an individual model fast as a one-off and working on tailoring it for performance.
My use case is a framework that an organisation of regular ecologists can easily write many concise and composable models using their own object types in regular julia, that are often as fast as hand-coded C. And they run parallel, on GPU etc without code changes.
You could probably write that within an order of magnitude with Numba but one programmer working at this while also writing 20 other packages simply wouldn’t be able to do it in practice. 2x would be very hard to do without @generated functions - I couldn’t do it in C++.
But you are absolutely free to continue using Python! it’s fine for many use-case, I’m just trying to point out some cases where julia makes my life a lot easier, and I can produce tools I couldn’t produce otherwise, and that is why I use it.
1 Like
The Numba backend works actually very similar to the Julia one - it is a JIT compiler which propagates type information when a function is called with specific parameter types and then compiles a specialized version of this function using LLVM.
Thus, for these simple examples both Numba and Julia perform similarly in most cases.
The main difference is the front-end: the Python language has been designed to be interpreted, without emphasis on speed / optimization capabilities. On the other hand, Julia is designed from the ground for JIT compilation and to make the compiler optimization as easy as possible.
The differences usually occur in more complex (realistic) use cases - if you have a Python program with >> 1000 LOC, you usually cannot just add Numba decorators anymore and everything is fast.
Edit:
This is an interesting article describing the Julia design philosophy: https://dl.acm.org/doi/10.1145/3276490
4 Likes
Yeah one of the small upsides of the pandemic was the research centers and most of the professors going more digital and record/share lectures and all sort of information, in the digital age they would often lag behind in that and hopefully from now on its easier to keep yourself updated on their projects and initiatives.
Thanks, I’ve been looking around your github repository and will properly look into it when I can afford the time to do so.
2 Likes
I’m just getting started with Julia and actually the Python and Numba discussion come just in time because the reason I’m giving Julia a go is because it was mentioned by my professor in a class about parallel programming, but he’s a computer engineer and I’m a physicist so obviously I’m looking for a “one language to rule them all” so that I can easily solve my problems, and one of the questions that I still have to ask him, because the classes were based around Java, C and Cuda, is “why can I not use Python with external packages to do it all?”.
And that’s why I’m here, to see if Julia can “rule them all”. Obviously that would be pretty hard so I’m trying to see what can it do betterand where it does fall short.
2 Likes
Hey, glad you consider using Julia for your problems. I wonder how your code is organized. From your post: [quote=“Undercover, post:8, topic:51155”]
Thats my main concern, has if I were for instance doing a fluid simulation and change only its initial values would make me have to pre-compile the whole thing instead of the small portion of the file I just changed.
[/quote]
It sounds like there’s some room for improvement on that side.
If you have some kind of simulation script:
#simulation.jl
# imports
using Plots
# whatever initial values and parameters you have
init = 0:0.05:1
parameter = 3.1
steps = 100
results = []
ts = 0:steps
# simulate
for i0 in init
vals = Vector{Float64}(undef, length(ts))
vals[1] = i0
vals[2:end] = [parameter*vals[i]*(1-vals[i]) for i in 1:steps]
push!(results, vals)
end
#plot and save, whatever you want to do here
p = plot(ts, results[1])
for i in 2:lastindex(ts)
plot!(p, ts, results[i])
end
savefig(p, "mySimulation_$(parameter)_$(steps)steps.png")
(written on a smartphone, might contain mistakes)
Then sure, you’ll experience the lengthy time-to-first-plot every time you change something in the script.
Is this the case or did I read your statement wrongly? (sorry if I did)
In case that’s your problem, you may want to put the thing that does the simulation into a separate function that takes all your initial values and parameters as arguments. You can make that a package (even local if you want to) and import it as usual and then call it with whatever initial conditions you want to try without the need to recompile anything at all.
2 Likes