Yes, that’s basically what I mean and I use this as a counter example. Look at the software stack and hacks, let alone all the manpower needed to make all this work; and as I said, where is the comparison to e.g. Julia? It’s for sure possible to optimise the hell out of something with lots of experts. Imagine these guys working on a fully Julia based solution, one language, concise design and transparent implementation. But if people keep beating a dead horse (sorry for exaggerating), you will never know. But there is some evidence at least, cf. Julia Joins Petaflop Club
4 Likes
If what you want is really fast first time to plot, you might want to consider another plotting package other than Plots.jl. It is a really versatile abstract plotting package but that versatility and abstraction comes at a cost. Just use the backends directly.
Gnuplot Based:
Matplotlib:
Pure Julia:
1 Like
…and my personal favourite which is also lightning fast ;)
4 Likes
Yes, and this is how the modern scientific programming should be!
Most important resource is the human mind. The scientific programming is the really hard world. Only one way to overcome the difficulties is to split a problem on to the different levels of abstraction and think differently on each level.
Here is the example. Consider some scientific programming language connected to the OpenBlas for the fast linear algebra. On the top level you need only to solve your system of linear equations by typing
x = A \ y
On the middle level the high level API is connected to the BLAS and LAPACK interfaces. On the low level there are plenty of specialized kernels for each type of hardware. You should take into the account the support of the SIMD, number of cores, the size of L1, L2, caches and other hardware specific things.
Do we need need one language to rule them all? I am not sure. Did you look on the OpenBlas source code? Did you see how it implemented to achieve the high performance? It is absolutely different level of thinking.
When you use TensorFlow (and other HPC python toolkits) you thinking in Python high level in terms of computational graph, in terms of logic of you model. That allows you to achieve maximum productivity. If you need to add some low-level kernel to you computation graph you go down to the low level of the hierarchy and take the most suitable instrument for that level. And Python is the perfect glue that help to transparent connect all levels into the one very productive system.
2 Likes
Hmm… From the user point of view I can from the entry-level laptop (or even from mobile phone) open the “Google Colab” connected to the arrays of TPU with all python environment ready to work, in few lines build the modern machine learning model, connect it to data storage in AWS, run and in several minutes get the high quality result. For the data of billions degree of freedom!
I do not say that the Julia is worse than Python. Julia is the great language. However Python is also good language for the modern scientific computing and Python is real working example how to make modern HPC.
Consider some typical unix-style workflow: a) by cron (time-based job scheduler in Unix) the script is executed; b) script gets data from the API; c) script plots the data into the file and d) the script is finished. If you will use Julia is this scenario, Julia will be the slowest language in the world. Does it mean that the Julia is the slow language? No! We just use Julia if wrong context. We should use Julia in the REPL style or keep the session open.
It means that the not only language, but context of language using and how we are using the language is the very important thing.
Using the Python is close to use of the modern micro service architecture. Please see my post above about different levels of thinking and about do we really need one language for the all levels. Python is the great language for the high level thinking and is the best glue to build the working system from the different levels.
1 Like
I’m sorry, this as absurd. If you came this far in setting up a workflow, you could easily add a little pool of workers similar to DaemonMode.jl and and call it a day.
Not saying that Python is a bad language, it’s really powerful, but @tamasgal has a good point:
Python is by any means not used for that task because it’s inherently optimal for it, but because it grew to be used in that way, alongside a large ecosystem and because it offers enough flexibility to glue things together while being approachable enough for non-expert users to harness this vast ecosystem.
All of the things you mentioned don’t rely on Python as a language, but on some kind of glue language to hold everything together in a proper shape.
4 Likes
Of course I know, I even hacked those myself!
I think you did not get my point.
All I say is that in my opinion Python is not the perfect glue. I think a language like Julia is the perfect glue as it allows much easier customisations from the user side, while allowing to wrap things in lightweight data structures, that’s all. You can list as many examples as you want, most of them have rigid implementations under the hood and little freedom on the Python side without sacrificing a lot of performance. On top of that, the maintenance is complex and its costs very high. I know this very well because I work in a team at the lower end of the Python chain where we need to maintain high-level API to highly optimised custom implementations which are definitely doable in Julia where everything is directly exposed to the user. That’s my main point.
14 Likes
Maybe you should check out MIT’s “Introduction to Computational Thinking” course. This one is an updated and expanded version of their course on Julia Academy. Everything is available for free online, and it gives both an introduction to the language while discussing some higher level concepts (parallel programming, ray-tracing, event simulation, climate modeling, etc.). I have not finished all the lectures/problem sets myself, but I really like what I’ve seen so far. Also, everything is coded in Pluto.jl notebooks, which IMHO are far superior to Jupyter, though I am biased having never liked Jupyter notebooks in the past.
Also, if you really want to be able to ship code/apps to users (or just reduce latency), then you should watch this video about PackageCompiler.jl if you haven’t seen it already.
Edit: All manner of learning resources can also be found at Get started with Julia.
1 Like
Can I suggest you avoid making comments like this one? You have a very good point when you remind people here that Python wrappers for C++ are good enough for many people, but then you make other odd statements like this that end up cast doubting on the better arguments you’re making. Maybe time to pause on this thread and move on?
3 Likes
This is a very bold statement about scientific programing, which reduces it to the problems that can be adapted to use available tools.
Problems that rely heavily on linear algebra may be the ones that most fit into this description. But optimization problems in general are far from that. Contrary to what happens with linear algebra, in many optimization problems the critical part of the code for performance is not what the solver does, but computation of the objective function and the gradient. Those have to be implemented by the user, thus the most critical part of the code must be implemented by the user. The facility to use the solvers is a must, but is of secondary importance.
You cannot naturally write the computation of a complex objective function depending on conditionals, complex data structures, etc, in a language that does not allow you to write performant loops. For specific problems you might find smart ways to write your code vectorizing everything, or if it is simple enough decorating the code, but in practice this is very cumbersome and at the end everyone falls back to low level languages.
13 Likes
That is exactly the problem that Python JAX solves!
Python JAX can automatically differentiate native Python and NumPy functions. It can differentiate through loops, branches, recursion, and closures… JAX also lets you just-in-time compile your own Python functions into XLA-optimized kernels…
See examples of the parallel ODE solvers on the TPUs by Python JAX. And here we have the same architecture where high level computation graph is compiled to the optimized kernels for the specific hardware.
1 Like
But these are examples with 10 lines of code… the things computed are mostly formulas. The problem here is we cannot prove our points without actually showing complex codes and, of course, nobody will translate a complex code in one way or the other just to prove one is right. What I think is that the fact that the performant parts even of the python libraries being written in low level languages proves our point.
4 Likes
Sorry for that comment from my side. I figure out that it could be some misunderstandings. However in the context of the my full post it means the following.
We discused that the scientific codes could be in different levels of abstractions. High level, middle level, low level. And each level could be implemented on different languages (like OpenBlas). And the complexity is how to glue the different layers in different languages in one system. As an example the Python wrappers of the C++ code. When we use micro services in general we solve the same problem, how to arranges an application as a collection of potentially loosely coupled parts.
I absolutely agree, that these are different things. But some approaches in development of the architecture of the full applications somewhere is very close.
Agree. I stop now. Sorry for disturbing the community.
2 Likes
Really make sense. I have an objective function that is so expensive. It is not realistic to code it in Python. I have to written it in C++ in the old days. Now Julia offers a great tool to implement it and glue it together with all powerful optimization tools available in Julia. That is the main reason I moved to Julia from Python.
2 Likes
For what it’s worth I enjoyed reading your arguments for the pythonverse. And it’s an awesome infrastructure that works really well. Especially in my field which is ai/ml. At our company, we use a combination of julia, R, and Python and I agree that one language to rule them all might not be necessary for everything. But I’m still hoping that we one day will move our entire stack to julia. 
3 Likes
I guess that in some extent this isn’t a new problem and is a problem of scope. For most people, especially the ones that do things by themselves and out of curiosity, Python is as good as it gets, but for people who require more, Python just won’t cut it and they require a better solution that isn’t given by that language because most of what it does is wrap around stuff from other languages.
Sometimes you want to do stuff without getting exposed to the complexity of the low-level details of whatever you’re working on, I just want my car to work doesn’t mean I necessarily care or will modify the way it works, but sometimes you do.
Hopefully Julia does manage to do that, so that the end-user could be exposed to the simple use case and if required dig deeper into what’s happening and still be in Julia code, but in the end, Julia will also be written in C, so if somebody goes too deep they might say the same thing some of you are saying here about Python, although probably it will probably be a very tiny fraction of people considering most of Jula is written in Julia (at least according to the github page is 70% Julia).
I am clearly biased because I am the author of DaemonMode.jl, but I encourage you to use it if you want to use Julia in scripts as Python. In that way, the second time (and following) you run a script like that it takes about a second, you can change the script, and it will run it very quickly (or run another similar script with similar time). It works with ArgParse, and even in combination with more complex packages like Literate.jl. I use it everywhere, and I am very happy using it.
3 Likes
The use case I see for this is for deploying a web server that runs Julia code in background. I am unsure however about the scale up of the memory required. If 50 students run a code (which is fast) the variables they allocate will accumulate on the Julia daemon section?
1 Like
I started programming using C#, and I believed back them that it was the best language, them I learned Java in a bigger extend and it became my new ‘bestie’. The history repeated a couple of times more. My theory is that the language that allows you to communicate/do what you want is ‘de facto’ the best. That’s why when a student come and ask me what language is the best to start I say: “Just pick one, anyone, start ASAP and not quit”.
2 Likes