What steps should the Julia community take to bring Julia to the next level of popularity?

[ChrisRackauckas]

@dlakelan

Let’s face it developers cost $150k or more, you want 3-5 of them working on different bits full time at least. That’s $500k a year order of magnitude, and another 10 years or so means $5M

And if academic, double that so it means you need $10M of grant funds towards non-research outputs, of which the university takes half of the dollars and the other half need to probably work towards real milestones with at least half of their time, so then you’re probably needing $20M to make that be 3-5 full time of people via 6-10 going half time on OSS and half time on their actual research topic to write papers and graduate. That of course is the extra money, i.e. you need $20M extra on top of your “actual research” grants to keep that up over that time period because you also need to cover all of the actual research.

In other words, if anyone wants to collaborate on a grant proposal with the Julia Lab please reach out. We put out tons and tons of proposals . And this is also why we’ve been building commercial arms to support more developers.

[ChrisRackauckas]

@josePereiro

I belief adding a “Prerequisites” section (or something similar) to the beginning of many docs would be very helpful, even for “end users” closer to academia.
Sometimes what is missing is just a warning, “before you continue, make sure you read/see/understand the following material”. From my experience (coming from Biology) that could means a lot, to have a more defined path toward understanding the surrounding material of the package.

That’s a good JuliaCon hackathon topic.

[frylock]

I appreciate your comment. I guess I had always considered that throwing money at an issue would help it to be solved somehow, but just taking my own professional experience into consideration it isn’t so much about money, but more about developer time and “mindshare” if you will. Getting the interest of developers, coaxing them to use language A instead of B or C. Or, getting their time.

I’ve never been on the giving or receiving end of money for development of a language, but I have been on projects where we got money to do a thing. And if the thing was composed of a checklist of some sort, the checklist would more or less be completed as it turned out to be possible, but one thing is certain: we would absolutely burn up all of the money.

Money is just the easiest thing to track, but simply tracking money summarizes away all of the important details.

[johnmyleswhite]

@jreileyclark

I’m curious. Does Julia have what it takes to outperform python in ML? Does it actually stand a chance? Or are we just believers who don’t want to follow the well trodden path? If so, how much would it cost to get there? $50,000? $500,000?

I really want to reset expectations here with anyone who has this mindset and is waiting for Julia to become the dominant ML language.

• Julia is almost surely not going to displace Python for ML because Python for ML is actually C++ for ML. As such, replacing the Python ML frameworks requires Julia to be dramatically better than C++, not Python. But I do not believe any of the Julia developers would claim that Julia will fully replace C++ as a language in the tech industry in the next 10 years. But that is the bar for “Julia is the future of ML”.
• There is no plausible way the Julia community can raise enough money to beat other programming languages through funding professional development instead of relying on volunteer-driven development. $50,000 will pay for, at best, 2 weeks of development time by the programmers working on TensorFlow. If Julia wants to win, it will be through volunteer time from high quality developers who do work based on their own passions, not through monetary compensation. For anyone who does not understand the relevant compensation comparison points here, please spend time reading through levels.fyi.

But here’s the thing – neither of those two points reflects at all poorly on Julia as a language – unless you strictly define success as replacing TensorFlow and PyTorch. Deep learning is close to the worst case scenario for Julia – it’s an extremely narrowly confined style of programming where a small core of computational primitives need to be optimized like crazy. It’s better to think of deep learning as the next generation of BLAS than the next generation of programming languages.

[johnmyleswhite]

@frylock

I appreciate your comment. I guess I had always considered that throwing money at an issue would help it to be solved somehow, but just taking my own professional experience into consideration it isn’t so much about money, but more about developer time and “mindshare” if you will. Getting the interest of developers, coaxing them to use language A instead of B or C. Or, getting their time.

Yes, this is the problem to be solved. But I want to very strongly emphasize – this problem is not going to be solved by corporate money. All of the other ideas in this thread are much better approaches to pursue.

[xiaoxi]

@GunnarFarneback

See The Julia Project and Its Entities

Is there any way to know which projects are funded by this?

Sponsor @JuliaLang on GitHub Sponsors

[rveltz]

@johnmyleswhite

because Python for ML is actually C++ for ML

I dont get this. Is it because python is a wrapper low level language like torch or c++ ?

[trung]

As far as I understand, the backend of Tensorflow and PyTorch are written in C++

[sfuerst]

@ChrisRackauckas

That’s not concrete. What packages and which functions?

As someone who has often cursed at the documentation of Julia packages and often looked into the source code to figure out how to use them, I’ll chime in here and give examples, even though I hate pointing at people.

So first of all as a clarification, a thousand thanks to the developers of the packages I’m about to mention as a negative example, I’m really grateful for their work, and without them the development of my own package would not have been possible at all. I am fully aware of how difficult it is to write a good documentation, and I also confess to have failed at it sometimes.

HDF5.jl probably gave me the most headaches. It starts with outdated information about supported data types, (by now much more is possible, like tuples and under certain conditions also structs, but what exactly you have to test yourself). The information about Parallel HDF5 and especially collective I/O is very sparse. And I also failed several times to find information about other things. Unfortunately that was a long time ago now, so I don’t have the details in my head anymore.

So let’s take two examples that I stumbled across this week, and from actually quite well-known and widespread packages. With Makie I always struggle to find the information which keywords I can use when. Try for example in the REPL “help?> Figure”. There is then a “No documentation found”. In addition a search function on the web page, which always gives only “No Files found” after you hit Enter. Why Franklin and not Documenter was used for the page is generally not clear to me. Try to find information about scatter! on the Makie webpage and compare this with geom_point on the RDocumentation/ggplot webpage.

Then I took a look at Genie.jl this week. I have developed a somewhat larger project with shiny in the past under a lot of time pressure, where I still do not completely rule out a reimplementation. The documentation of Genie.jl seems quite good for now, but I need to build a dashboard. And the one from Stipple, which is supposed to be used for this, the documentation actually only consists of the API itself, without any tutorial or the like, which somehow gives you access.

These are all experiences that I didn’t have before in other “bigger” programming languages (maybe with the exception of Scala). So it’s really hard for me to recommend other people to work with Julia, as long as they don’t really need the features. I really know that it’s hard to write good documentation, and failed often enough myself. I am fully aware that writing good documentation is hard, and I have failed often enough myself.

And that while I think Julia is a great language. I don’t know of any other programming language in which I could have developed Vahana.jl. And I think agent-based modeling in general is an area where Julia shines, not just because of on Agents.jl (which has really great documentation, by the way), but because here the computationally intensive parts have to be executed in the programming language in which the model is written.

[greatpet]

@trung

the backend of Tensorflow and PyTorch are written in C++

Are these C++ backends sufficiently decoupled from the Python front-end and sufficiently documented, to make it realistic to write bindings in other languages like Julia? (Assuming someone has the resources to do it.)

[trung]

Both Tensorflow and PyTorch have APIs for C++ so it might be possible to write bindings in Julia but at this point I would rather use a native solution in Julia like Flux/Lux

[xiaoxi]

@greatpet

Are these C++ backends sufficiently decoupled from the Python front-end and sufficiently documented, to make it realistic to write bindings in other languages like Julia?

Torch for R is based on libtorch.

Posit AI Blog: Please allow me to introduce myself: Torch for R

[dlakelan]

@johnmyleswhite

. It’s better to think of deep learning as the next generation of BLAS than the next generation of programming languages.

This is such a good way to put it.

I also think Julia sits in between C++ or Rust, and Python. It’s designed for people who have enough programming skills they could be programming in a static compiled fast language, but who have enough desire for interactivity and dynamism that they don’t want to. It approximately fills the gap that Common Lisp might have in 1990

Python on the other hand was designed from the start as an educational language for programming language learners, a big part of its success is that people like my friend who is a biologist could learn it on the side in his PhD and then munge some sequence data etc… It’s one step above Excel formulas for a lot of people. Of course it could be used for other stuff, but it’s really slow if you’re not calling C/C++ via wrappers.

So I honestly don’t think Julia will become super popular, just like I don’t think calculus textbooks will overtake romance novels. There just aren’t that many people who think of calculus as entertainment. It takes a certain aesthetic to like Julia. If you hate programming and just need to accomplish something you will use an easy language like Python with a library that practically solves your problem already.

If you want to write BLAS or deep learning libs etc you’ll use C++ or Rust or similar.

If you need to explore a design space or do some research oriented code that is fast and interacts with a lot of different math… Julia is the thing.

Still, that doesn’t matter because Julia doesn’t have to be SUPER popular, if we can attract people away from Matlab and away from R and especially in Agent based modeling as someone else said… We are doing well.

[johnmyleswhite]

See the languages section for Tensorflow and PyTorch. Tensorflow is 64% C++ and 20% Python. PyTorch is 46% Python and 43% C++ after a major push to move more logic into Python.

[jreileyclark]

I think this is the bar set by Julia when trying to solve the 2-language problem. “Julia is a solution to the two language problem!”… Except the solution is a 5 language problem.

[dlakelan]

@jreileyclark

“Julia is a solution to the two language problem!”

It’s really a solution to the 2 language problem for people writing some functionality from scratch. When there’s a huge established codebase to do something, you’ve still got to decide whether you’ll link to that or build your own Julia thing.

[jreileyclark]

This is discouraging. Learning a language requires a significant investment of one’s time. You’re starting that, compared to other languages, it won’t pay off but for an aesthetic appeal. That’s simply not enough. It’s not enough to convince new users who can’t relate yet to the syntax benefits, it’s not going to convince companies to build with the language, it won’t convince universities to adopt it in the classroom. There is no opportunity for this language to pay off other than enjoyment? I don’t enjoy programming for programming sake. I enjoy building things, analyzing things. My company won’t pay me to program for pleasure , they’ll pay me to build them things. You basically say this language is dead. If you could convince me if your position I’d never write a line of Julia again, and would never encourage someone to do so.

[dlakelan]

No, not at all! Julia hypercharges your ability to do certain kinds of projects. But only certain kinds of people tend to tackle those kinds of projects. It’s not just aesthetics but aesthetics in computer programming translate into power to achieve things. Typically you can do stuff in Julia you just wouldn’t consider ever in C++ or Rust or Python.

[jreileyclark]

Which is effectively saying what I just said. You’ve taken a 2-language problem and turned it into an N-language problem.

I want to do data science in Julia. I need a fast language with reliable packages that can do it better than python, or else i’m going to use Python. If you’re telling me julia will never be better than Python for my interests and use case, which also happens to be one of the dominant uses of programming today, what’s the point? I believe Julia can do what python does better than Python does it, all in one language. If thats not true then Julia has no purpose.

[dlakelan]

@jreileyclark

I want to do data science in Julia.

Depends a whole lot on what “data science” means to you. If data science means “i want to train multi-level neural networks over and over again on different large datasets” then yeah, go do it in Python using some libraries developed in C++ with a billion dollars and several million hours of human investment to do that one thing incredibly quickly.

If “data science” means to you that you want to express Agent based models and run them millions of times and develop an approximate bayesian computation based estimate of the posterior distribution of the parameters of the model, there is literally NOTHING that does that like Julia will even close.

If you want to develop some Jupyter notebooks for decision makers to interact with and get very quick answers from a big database of data using simple linear regressions and nice plots… Julia again is going to work well.

There’s just a bazillion ways that you can use any language, and whenever what you want to do is a repeated production of widgets, you are likely to do best with whatever enormous specialized widget maker is available. You don’t buy a CNC lathe if you have to stamp out hubcaps.