Losing Science Marketshare to Non-Python Languages

Offtopic
1

The Julia team could grow the user base by filling the programming language gaps that data scientists are resorting to more difficult languages for, such as Go and Rust. Search the web to see what problems scientists are resorting to learning harder languages like Go and Rust to solve, for example.

For the problems Julia can solve already, start an online campaign via blogs, Reddit, YouTube, Twitter, StackOverFlow, GitHub, etc… to inform scientist how much easier it would be to implement Julia. For the science programming problems Julia can’t solve yet, that’s where the Julia language and it’s libraries should be further developed and informing the science programming community is coming soon.

Edit: I apologize to those my suggestion doesn’t set well with. Sincerely, I can’t even begin to imagine the amount of thought, time and energy that’s been generously poured into making Julia the already amazing accomplishment that it is. Lastly, the quantity and magnitude of prompt responses to the suggestion of a nobody newcomer to the language, like myself, is impressive to say the least. :slightly_smiling_face:

2

Since Julia is a general programming language, there aren’t any.

3 Likes
3

Since Julia is a general programming language, there aren’t any.

This seems like a disingenuously literal interpretation of the statement… there are of course scientific programming problems and workflows for which Julia does not work well. Just because something is possible does not make it efficient or desirable. It would also always be possible to write all of your scientific code in C or assembly.

I agree with OP that it would be good to make Julia more visible in the scientific community, but outreach takes time and resources, and the question is always who has the time and/or motivation to sustain such an effort.

17 Likes
4

All of these things already exist for Julia?

“Julia isn’t as popular/mainstream/well-supported as language X” is a valid criticism, but “be more popular” or “develop more/better packages” aren’t very actionable, and don’t lead to productive discussions.

9 Likes
5

Perhaps we have neglected TikTok and OnlyFans…

32 Likes
6

And even one step further, “science” is too big to be a meaningful entity to discuss too. Nobody works on “science”, people work in subsets like materials physics, ecology, etc. Julia has been making some major inroads in some areas. In pharmacometrics for example, R, MATLAB, and Julia are “the” languages, also discussed along with Perl and Fortran, with Python not even a player in this game. There are other areas where Julia has not made inroads, like in data science it still seems to be R > Python > everything else, and then machine learning, etc.

To have an actionable discussion, the question needs to be much more direct. For example, a lot of people are doing high dimensional data visualizations (umap, tsne, etc.): how could Julia make inroads there? Or rather, should Julia make inroads there?

It seems that the OP’s original interest here wasn’t “science” but data science.

Maybe there are things to be done in data science, but note that the majority of scientists work in “niche” areas which are not data science. So while maybe some of their software may get more use if it was in one of the big areas (cancer biology, bioinformatics, and data science), changing from say particle physics to one of these fields is a major career decision. There’s nothing wrong with the fact that many Julia developers aren’t working in the most hot button area of today: you need to make bets about what will be the hot areas of tomorrow.

And even then, you might know that your area will never be one of the most popular scientific domains, but seeing Julia programs widely used there is not a waste but a nice example of what we should be doing more of.

9 Likes
7

You’re absolutely right that science is not a monolith, but I do repeatedly get the impression from interacting with colleagues and people working on similar problems (i.e. modeling of some kind) that Julia has not even reached the level of passing familiarity that I would expect. The reaction I get is quite often something like “Julia? What is that?”.

2 Likes
8

There has already been an enormous amount of effort expended showcasing and promoting Julia on the forums you mention and others. The material is mostly of very high quality. You can also find this easily. Asking for more or better isn’t a good approach IMO.

People often choose other languages over Julia, eg Rust, not for lack of advertising or even libraries, but for fundamental reasons, such as interoperability and ease of deployment. These are hard to solve and resource intensive and have had to compete with all the other really important stuff; just to grab a couple of things at random: BinaryBuilder and Pkg. Still these problems around interoperability (but not only interop) are being worked on.

So it’s much more than just libraries. And some Python libraries have armies as big as the Julia dev community working on them. It doesn’t make sense to set beating them as a goal. Furthermore Python is written in C, has always relied principally on C extensions. Rust is nearly a drop in replacement.

There are so many other reasons. Rust and Julia are very different. Sometimes people just enjoy writing in Rust more than Julia and vice versa. Maybe a matter of personality.

The questions of exposure and promotion that you mention are important. But, these questions as well have also been discussed in Julia forums in great detail (or at least great quantity) for many years.

11 Likes
10

Yes, these two attributes are paramount.

Regarding a niche to go after, RPA would be great because it requires these two attributes in spades and it’s expected to keep growing rapidly over the next decade. Plus, RPA communities are strong and efficient at spreading the word for their causes. For example, partner with Robot Framework and RoboCorp to make their automations and integrations both simpler and more performant than Python can.

1 Like
11

:joy: Fair enough. :wink:

12

Undoubtedly, but when compared with Rust and Go, neither of which are typically targeted at scientific programming (even if some people may use it for that purpose), Julia is a fine choice.

Rust and Go are of course great languages and have features that Julia does not even try to match (eg Julia is unlikely to become a system programming language like Rust).

6 Likes
13

Having wrappers to particularly good Rust libraries would be a better use of resources than starting to develop a comparable package in Julia. I would rather not start using Rust, I don’t like the look of it.

I call this the two-language solution. :sweat_smile:

4 Likes
14

Then your julia package users that want to change something need to do it in Rust…

That’s the beauty of Julia… if you know how to use a package you also know how to develope it…

4 Likes
15

I went to a HPC conference last week and gave a presentation on using Julia for scaling up your code across the different types of parallelism without changing much of your code, and it was fairly well received by the audience, with many questions. I share this because it seems like a good avenue for academics to inform people about the possible uses of Julia, especially in areas where it undoubtedly has a huge advantage (e.g. HPC and research). The organisers even requested a follow up blog post on the topic to circulate more widely around the university.

The major downside to preparing these talks is that they take a lot of effort to produce, and many people are very busy already, and outreach is usually not credited at all, especially when compared to writing papers.

27 Likes
16

Be the change to you want to see :slight_smile:

I feel at the moment Julia’s parallelism is quite poor compared to its competitors, as it is too easy to segfault, and hard to debug when you do. If be happy to see some better guides on how to avoid that, and debug when it happens.

2 Likes
17

https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/

6 Likes
18

In my perspective, I see Julia’s ‘competitors’ mainly as Matlab, Python and R, and then it’s hard for me to agree with this statement.

Can you clarify which languages you mean?

4 Likes
19

I don’t have to look far to see Julia’s shortcomings that prevent me to advocate for its use.

  1. No static bit fields. This may seem minor, but it actually hit me on my very first professional attempt to promote Julia in production. For the efficiency brought by memory-locality of data used by my algorithm I needed to make a custom memory structure holding a static bit field. As you know, variable-size objects are stored as pointers in structs, so accessing the individual bitfield will usually involve a CPU cache miss. There was a promise for support for that in StaticArrays, but despite a PR (Static Bitsets by chethega · Pull Request #647 · JuliaArrays/StaticArrays.jl · GitHub) it never materialized, to the best of my knowledge, to this day. What is the status of StaticBitsets? · Issue #1 · chethega/StaticBitsets.jl · GitHub. I even did report this shortcoming over a year ago to Sushil Kumar, who was at that time an employee of JuliaComputing team, and I never even got a feedback about appreciation of the problem. This all leads me to believe, that there is a whole mountain of obvious Julia shortcomings and there is by far not enough effort to fix them. Lack of support of static bit field seems like just one of many.

You can guess how disappointing it is for me, especially if you consider how easy is to do similar things in C++ and its STL. It was supposed to be this kind of pure data-science project, that was expected to showcase Julia to my whole organization.

Now, with the advent of C++20 and ever-increasing support for Python in Data science, the competition for Julia in terms of expressive power is even tougher.

There is a progress though: The other obstacle to the success of this project was a lack of support for binary compilation of Julia code. Now it seems to have been solved.

  1. Lack of static code analysis in IDE.

Correct me if I’m wrong, but I see only one modern IDE for Julia: VS Code with Julia addon. It has the philosophy to make code completion database on runtime, rather than statically one it parses the project. I find it a cheap walk-around and a dead-end.

VSCode can only help you with the type of the type-relevant hints only once the code was actually run, and only to the extent, as it was last seen by the Julia parser/interpreter. This is the root to most of the issues I have with Juno. (The other one is that Juno, being an Electron application, is incapable of supporting more than a one window).

Effectively, in summer 2022, the last time I’ve checked I found the overall experience of coding in Juno so much worse than doing the same thing in Python with pyCharm, that I decided that the syntax and speed benefit is not worth it.

I am aware, that doing things statically requires a lot more effort, as a lot of action is done during function dispatching and giving a full support for static analysis may require a lot of heuristics to overcome the infamous Touring’s halting problem. It may be a sad reality that Julia community may simply not afford this amount of manpower in the foreseeable future.

edit: I have mistook Juno with Julia plugin for VSCode. (I used to use Juno even earlier in my career, hence the confusion).

2 Likes
20

The Atom editor which Juno builds on was effectively abandoned years ago, and has also been officially deprecated. For that reason, development of Juno also stopped years ago, and moved to the VS Code Julia plugin.

Were you unaware of this, or are you dismissing the VS Code plugin for some other reason?

3 Likes
21

I can’t comment on 1), since I am not an expert in this use, but there are many areas in which Julia is a tool that provides a huge amount of benefit and utility, it just depends on the problem.

On 2), I think the most common and recommended IDE for Julia is now VS Code? The language server does allow autocomplete which has improved a lot over the years and runs in the background without you having to have an open REPL. Unfortunately the design of multiple dispatch doesn’t have a convenient way to have great autocomplete like in single dispatch languages (Python with typing, C# etc) as discussed in this thread (Why there is no OOP (object oriented programming) in Julia? - #27 by LaurentPlagne).

I think a lot of people come from languages like Python, MATLAB or R which have varying levels of tooling. While Python has PyCharm, which is very good, the tooling that a lot of people use is very limited (Jupyter notebooks, Spyder etc) and coming to Julia with VS code, you get similar or better support in VS code. I still think there’s a long way to go for sure, but Julia is still quite young, and most other languages have at least a decade of lead time developing good tooling. The tooling has improved a lot over time, and I hope this area gets more attention in the future, but it takes a lot of effort and dev time to achieve.

2 Likes