How do we Julians win big when the situation is so unfair?

I submitted two documentation PRs to simplify these two examples:

Please, when you notice places like this where the docs could be improved, click the “edit” button on github to file a PR, or file an issue requesting that someone make the change.

8 Likes

One way Julians have already won big is discourse. It is amazing to me to be able to communicate with academics and other specialists in broad fields and read their perspective on issues.

One of the areas I get involved in from time to time is modal analysis from test data. My tool of choice currently is Xmodal, technically open source software but with a distribution/maintenance fee. It’s algorithms are as good as, or better than commercial software. This has been written in MATLAB with a MATLAB GUI and is normally distributed as a MATLAB program in MATLAB’s runtime engine. When this gets ported to Julia, I will know we have won big.

5 Likes

I’d definitely agree with the point you’re making here, but I’d be more specific about the group Julia targets. Often, Julia design decisions are biased towards “power users” (e.g. in high performance computing or physics simulations) with very unusual requirements, usually things that are impossible to do in other programming languages (Fortran is too inflexible and Python is too slow). Often, these push the language’s design towards being extremely dynamic and general, at the expense of reliability or ease-of-use.

This group might include computer scientists in academia, but rarely (if ever) includes software engineers (especially systems programmers, who write most of the biggest Python packages). These kinds of people are usually more practical and favor safety over performance or generality.

2 Likes

We have entire recipe library for doing just that.

This uses the https://binarybuilder.org/ cross compilation framework to provide binaries via the Julia packaging system.

Via GitHub - JuliaInterop/Clang.jl: C binding generator and Julia interface to libclang you can parse C headers and generate Julia wrappers for them via ccall. All that we need for external libraries is a C interface. We also have tools for C++, Java, Python, R, and Rust interfaces.

2 Likes

I agree with Chris. Whatever discipline you are in, make the “killer app”. Most users will migrate to the best app available, because eventually, they will have to. This is particularly true in peer reviewed science where good reviewers will question why you did your analysis using an inferior app.

1 Like

The one thing that I learned from helping to develop NumPy/SciPy 25 years ago was that you need to be patient and persistent. In other words keep advertising Julia’s benefits. Only about 1 in 10 users or developers will listen to you, but that is sufficient over time. Eventually, you will reach a critical mass and most users will have heard of it, although they may not use it. The problem that I see now in the astronomical community is that not enough people know what benefits Julia provides. That’s why it is important to talk up the language. Some people will listen and take a look at it.

The following is a brief history of NumPy/SciPy in case you do not know. Numpy was developed by a handful of people mainly in the Science Software Branch of the Space Telescope Science Institute (STScI) in Baltimore. Think the Hubble Space Telescope. In 1996 Joe Harrington and I recommended that Python be used as the common data analysis language in astronomy. Starting in 1998, Rick White and Perry Greenfield at STScI listened to us and were able to convince STSci management to provide some resources to develop Numpy, PyFITS, Pyraf, and matplotlib. After a few years of developing these libraries, we felt comfortable that they were production ready and began encouraging the astronomical community to adopt NumPy. Because of scepticism in the community, it took another few years to get enough institutions on board to get us over the hurdle. The common argument was that will will take decades to rewrite all the FORTRAN and C/C++ code. This is a common fallacy that I still hear today with respect to Julia. In summary, I’d say it took us about 8 to 10 years to get the astronomical community to adopt Python. So Julia still has a few years to go before there is reason for concern.

49 Likes

To me, by far the “killer app” for Julia is data analysis through Quarto notebooks. If you have some stuff you want to read in and analyze and make a report of the results, you can write a document and process all the data in Julia, and it will be faster and more flexible and more mathematically aligned than anything you could do in R or Python. This is particularly true if you want to do a Bayesian model in Turing, or solve differential equations or build agent based models.

I would never ever go back to R or bother learning Python Numpy etc.

3 Likes

I mean, Symbolics.jl still doesn’t have integration… it’s why I moved back to Matlab on a project I’m working on. So I’m not sure I would say that Symbolics is more powerful than SymPy. It has a lot of advanced capabilities, but is missing a lot of basic capabilities.

5 Likes

This reminds me of this recent thoughtful talk by Evan Czaplicki (the creator of the Elm programming language):

He also mentions Julia once or twice.

I gotta say that the best bet for most programming languages is servicing whatever core niches, and a good integrability story with the overall software ecosystem so people can do anytime/anywhere-style of development (even MATLAB seems to be headed this route). This is an aspect of Python that in retrospect, seems to have gotten it past a trajectory that if you squint back in 2011-2012, could have easily ended up with a story of Ruby, R, and other langs eclipsing it by far in its journey.

Now, the good news is that the world has never seen the number of young and inexperienced developers looking for direction as it has now. So the overall “market” (pie-size) is huge. Compared to traditional circles, I think a good number of these devs spend their time on things like Twitch from what I’ve seen. It’d be great to see some experienced Julia developers (or even people with unusual math, technical, or scientific skills) try their hand at streaming. I suspect more and more that’ll become a generational norm shift in audience building. But these are just my observations as a (perhaps out of the loop) older millennial who has seen at least of 2-3 of these “gen effects” shifts in my lifetime.

6 Likes

It’s a game of patience. Python took decades to take off, unlikely it will be any different for Julia. But, if I were to bet on something to do different it is to appeal to the machine learning crowd (heavy python users) and less so scientific computing (matlab). The reason for this is that the former is a more active and vocal crowd, not sure if it is actually larger.

2 Likes

Just to emphasize this, I was hired in 1997 to write some data collection scripts that basically collected data from Banks and encrypted it and emailed it to other banks. They had a bunch of C++ code that did a lot of what was needed but not all. I thought it was insane, and the managers didn’t know anything much about programming. I sat down with an OReilly book about Python and banged out the full set of functionality in a week or so. This promptly got me written up by manager and I started looking for a better job, as did all the other developers. That company crashed and burned and I moved to a financial data analysis firm that was heavy into SAS… I tried to get them to replace a lot of their c-shell scripts (which were a huge technical debt they didn’t understand) with python but the hardest sale was that whitespace had semantic meaning… Basically in 1999 or so if you were suggesting python it was exactly like today suggesting Julia. And R + MySQL instead of SAS and SAS datasets was also a hard sell.

It kinda sucks to be 20 years ahead of the curve in some ways. The best way to deal with it is to just do what makes sense in your language of choice.

Or decide that what matters is getting rich and screwing the public… That’s what all the “success stories” of the last 20 years amount to.

11 Likes

It’s super impressive to me that 90s Python could do so much already, crazy that you were punished for using it to get the job done so fast.

1 Like

Standardisation through ANSI would be a huge step forward over (eg.) Python and put the language on the same footing as Fortran / C / C++.

$0.02

1 Like

And that’s not on accident either. SymPy and other systems already do symbolic integration. They don’t do great on codegen. So Symbolics.jl should do codegen first and not symbolic integration. Just rewriting what already exists is not how you build something new. Over time symbolic integration will exist, but if Symbolics.jl was just a copy of SymPy then there wouldn’t be a reason to use it.

4 Likes

Though it would be really nice to get Rubi into Julia.

It is in the plans.

3 Likes

Sorry for the completely unrelated comment, but for a split second I thought you are referring to some very large community :joy: The Mind moves in a mysterious way… perhaps it’s because these discussions tend to revolve around the size of the community of X language as well.

But more seriously, thanks for sharing this part of the history of NumPy/SciPy, super interesting!

4 Likes

My dream is to see a Julia-based real-world project/startup/system/whatever where we can look back and, after putting the luck/timing aside, we can clearly point out the Julia Language as the main factor of success and/or competitivity. A kind of modern-time Beating the Averages story.

Yes, Julia is not the best option for every conceivable scenario - but at the same time, I am convinced that large ecosystems can be grown in a nice and composable way for various domains beyond mere technical/scientific ones.

I am convinced that such a feat is not something that can happen overnight (even if adequate funding is made available) - but I don’t see anything wrong with actively pushing things in that direction (instead of staying by and just stating that the people/ecosystem needs to grow organically in that direction). Failure is an option.

Also, metaprogramming has lots of unexplored territory in the LLMs world - and given the crazy things that we can do with Julia compiler (+ macros and metaprogramming-related stuff) at runtime, I think Julia is uniquely positioned to thrive in this environment (demo/MVP later).

Tech and general startup funding is a huge huge problem. With a tiny minority controlling trillions of dollars I have no hope for innovation until that changes.

1 Like

I’m not sure this is a feasible aspiration for Julia or any programming language. Consider Python, despite the juggernaut of numpy/scipy, I think anyone would be hard pressed to say their project’s success was primarily due to the existence of Python and those two packages.

Also, the idea of big wins or “Project Y” suddenly moving a critical mass of talent to Julia seems inherently flawed to me. The story behind NumPy shows that it doesn’t happen overnight. I knew Python was gaining traction in astronomy in 2003, but I continued to use IDL for 8 years because I knew it already and it had all the routines I needed. Only when the majority of new code was Python vs IDL did I finally try to make the switch (luckily I found out about Julia at the same time!). It’s going to be small incremental wins that do the trick, and the more users there are the more small wins there will be.

The Zen of Julia should be: Use it, enjoy it, show it off when you do something cool, and don’t worry about its future.

6 Likes