What don't you like about Julia for "serious work"?

Does it mean that if there is nothing that I don’t like, I don’t engage in serious work? :wink:

13 Likes

No, it means all your work is fun.

12 Likes

Depending upon the kind of threading, I use JULIA_NUM_THREADS set to 2 x the number of hyperthreaded cores, or addprocs(n) where is set to the same value.

The problem is that threading can be oversold. Take pmap, for example:

using Distributed, BenchmarkTools
addprocs(4)                                        # i7 dual core
a = collect(1:size)
@btime pmap(x -> x,a)                      # simplest function

I compared map against pmap (as always, your mileage may vary):

There is a pretty solid three microsecond overhead per task for pmap. OK, it’s not much, but if the task is small then pmap is much slower than map.

I have got some useful increases in speed. I think that progress is about being more picky.

Note that pmap has a batch_size keyword argument that defaults to 1. If you set it to something larger, e.g. batch_size = cld(Size, nworkers()), it should perform better on small/cheap tasks.
Larger batch sizes of course make it less effective at load balancing, so consider the tradeoff of overhead vs load balancing for your specific application.

4 Likes

So you are setting it to the number of hyperthreads? If you are doing computationally intensive things it’s usually better to set to same as number of cores. Hyperthreading is rarely super effective for computationally intensive workloads

2 Likes

Hey, we have had some discussions about how to work with julia and .NET in Interoperability with .NET - #35 by AmburoseSekar and Correct process to use julia code from C# or VB .NET - #22 by AmburoseSekar.
I hope these links may help if you did not know about them !

1 Like

What don’t you like about Julia for “serious work”?

First of all, I would like to say that I like Julia and I like what the core team is trying to achieve. I like to play with Julia. To be honest for me it is more a toy than anything else. I am still struggling to find a real use for it or even use it.

If I need to prototype a model, glue things together do some stats, make some plots or create automated analysis and reports, then I can use R, Python or even Excel. Any of these 3 options will do a better job than Julia because they have better libraries, tooling and ecosystem in general. They also do not suffer from the Time to first plot, have better documentation and larger communities. That also means that you are more likely to find someone to help you solve your problems, understand code that was written by someone else or even find some to hire with that particular skill. I can list more things that are missing from Julia but that is not the point here.

Most people around me who tried to use Julia only see it as an immature fast version of R/Python. However for most people everything is not all about speed. Most people around me would say that they need to get things done quick, in a stable, reliable, secure and reproductible way. But often the Julia workflow is a pain, ecosystem too young, correctness issues, huge memory consumption and bugs like this one that drove me crazy:

That being said, which language is not a pain when you start learning it? None, they all have their good side and bad side… you just have to take the time to learn and appreciate them…

Also interacting with existing C++ libraries is key, because a lot has been built in that language. I know that Julia has CxxWrap, but it does not seem easy to use. Maybe I did not try hard enough or I am too stupid to manage to make it work. And there is no way a large company would rewrite its massive code base from C++ to a young language like Julia. That is never going to happen!

I really can see the benefit of Julia when someone has a need for prototyping something which must be fast and that also run for a long time. But that is really a niche for a language that aspires to be general. Other than that I am not sure what else can it be used for…

If someone or a firm wanted to build a software it would probably be better to build it in C++/Rust/Go/Java… all these languages can create small compact GUI, exe, dll …etc which can be used by other projects or by users with various level of IT skills. Yes Julia can create an (large) exe application (with 20 dlls) with PackageCompiler but again when it comes to creating an exe, app to be distributed, proprietary libraries, software, phone app…etc C++/Rust/Go/Java are better option than Julia.

Where does that leave Julia? In the middle between Python and C++, filling a small niche… And don’t get me wrong, you can do some serious quality work if you are in this niche…

If I had only 1 wish, I would ask for Julia to be as good as C++/Rust at creating exe/dll. That would be a big selling point over Python/R. The other “issues” like TTFP, slow compile time, lack of libraries… are minors issue from my point of view, and I would probably be able to live with them…

5 Likes

At my workplace three things.

  1. Researchers with no interest in computers find it too complicated compared to R and Python. They refuse to use git and write their own packages, and like their code in one file and don’t understand why all Julia code needs to be inside functions. They also love RStudio.
  2. The computer experts find it not as flexible as Rust (or other such languages).
  3. A lot of our code is run in pipelines on the command line, and may take 10 seconds to run, then a startup time of 60 seconds is annoying.
7 Likes

that nails it.

I would say the squeezed middle between {fortran, c, c++, rust} and {r, matlab, python}.

I’ve done projects in Julia, but when doing something small and new will almost always do it in python or r. And have started learning rust.

3 Likes

tracking type stability problems takes time and is not obvious. This is the #1 issue for HPC. For Julia wizards, it may be easy.

1 Like

Au contraire my friend: Julia is not a in a small niche in between, it spans the space from C and its ilk to Matlab and Python, and it is a closed interval (it overlaps those languages).

15 Likes

That is what I want Julia to be.

Alas, Julia is more complicated than the {r, matlab, python} of this world and not as powerful as {fortran, c, c++, rust}s.

Next time I win the lottery I will see if I can’t get the RStudio folks to add Julia to it, like they did with Python. Then Julia has a better chance there.

And I am hoping for standalone thin Julia binaries that one can easily copy and run on random systems.

Until then, the squeezed middle it is.

5 Likes

Next time I win the lottery I will see if I can’t get the RStudio folks to add Julia to it, like they did with Python. Then Julia has a better chance there.

They already did:

9 Likes

Reading datasets from cloud storage. These are often (not always) hive partitioned parquet files in a ‘data lake.’

There is often a metadata store, but this can be elided if you know the layout. It is effortful to build your own file tree to read into memory for some transform logic before writing elsewhere. Parquet is not well supported currently (kudos to the huge solo effort on Parquet2.jl), and its a ubiquitous data lake format since the schema is encoded with the conpressed data.

Its just a lot easier to run a spark cluster to handle that initial read for joins or filtering. So I’m generally managing two environments, a spark or MPP DB one (trino, redshift, etc) and wedging Julia work in there further downstream.

2 Likes

Incidentally, I was asked to evaluate and make a recommendation between Quarto and Jupyter books.

But, Quarto is not RStudio. And adding Julia (or any language) to Quarto is an order of magnitude easier than adding it to RStudio.

One can hope.

4 Likes

Maybe think about contributing to Parquet2.jl? Thats how this changes

2 Likes

Thanks for the hint! I see you just want the full RStudio (I am not exeprienced with) rather than just RMarkdown. IDE activities on the Julia side in the moment focus on Visual Studio code (which I don’t use as well as a diehard emacs user…) .

2 Likes

I don’t know if you speak from your own experience, but before you start boldly generalizing, you may wish to consider that your single data point may well be atypical.

I for one have created systems of decent complexity in all of these languages, except R. I found Matlab and Python fragile and disappointing overall.

And in my experience, Julia is much more powerful than C, C++, and Fortran when designing large systems. It is expressive, obvious, interactive, and high performance is reasonably easy to get. It aids in attaining reproducibility. Programming in C, C++, and Fortran is tedious, time consuming, and overall quite expensive in terms of man power.

20 Likes

Actually, I code in VScode and neovim.

But even R coder I know prefers RStudio. And its taught in the unis I know. And to get people started on coding, it is perfect. The gateway drug to R and now python. So, if it did Julia, would be fantastic.

This thread is quickly becoming unproductive but I’m surprised by this claim, surely Julia’s built-in ND Array and high-level numerical functions (as opposed to np.) and fast for loops (as opposed to remembering exactly what function you need every time) only makes it less complicated than these languages right?

From a user angle, they don’t need to interact with typing system much, so it’s just the surface syntax you’re talking about

13 Likes