What can we do to make Julia grow fast?

I’ve said this elsewhere, but perhaps this is a good place to reiterate that it seems absolutely crucial that the functionality for calling Julia from other languages, such as R and particularly Python, be robust and reliable. This doesn’t mean that I think Julia can immediately become a “C replacement” for people working on low-level parts of Python and R code, but it’ll simply be impossible for a large fraction of data scientists working in private industry to use Julia unless it cooperates with the tools that everyone else is using. I’m lucky to be in a place where things are flexible enough that I can use Julia, but that may not always be the case, and isn’t the case for a lot of people. It is much easier to convince colleagues that it is ok for me to be using Julia if they can be confident that they can call whatever I wrote from whatever they’re comfortable with. I was a Python skeptic from the moment I encountered it, so Julia was a pretty easy sell for me, but I think there are a lot of people out there who won’t even look at it until they find that they are already using it because somebody else wrote some Julia code that they are calling from Python.

Support-wise this is a tricky issue because most of us here are very happy using Julia and have no desire whatsoever to write Python code for calling it, but at some point many of us are going to have to buckle down and do it or watch the language we love disappear.

8 Likes

In order for any system to grow, you need to convince strangers to adopt it. An important part of persuading strangers is to show them the system you are champion works for what they want to apply it to. So, examples of real world use and success is important. An honest discussion of benefits and shortfalls is necessary as well. I have been using Julia in professional environments for at least a year. Some people in my region have been using it professionally for at least 2 years. We need to share our success stories from industry, academia, consulting, etc. There are a lot of examples of success with Python and R, which is they are widespread. As a community, we need to spread the word on our own successes and what part of that success that Julia uniquely supplied.

3 Likes

More broad package ecosystem and a good way of dealing with data. Both are nicely under way, but things take time :slight_smile:

7 Likes

Persuading people to adopt Julia is difficult because the package ecosystem is less mature compared to other languages. For many users, the strengths of the language alone are not enough to convince them to switch. I think the first step will involve attracting more people to develop the package ecosystem, which will help attract a larger user base. Attracting more developers should be easier as the core language stabilizes.

2 Likes

I think having more Julia beginner videos on Youtube covering more than just beginner stuff is a great way to highlight how easy Julia is to learn and in fact be good at, compared to other languages. This should accelerate the discovery and eyebrow raising process for potential users. For example one may not have time or be motivated enough to read a big part of the documentation to “discover” Julia, but watching a few short videos while having a little snack sounds a lot more lighthearted. And once a motivated person truly discovers Julia, it’s pretty much a one-way ticket to Julia land.

3 Likes

Here are my $0.02 worth:

  1. Plotting must be unobtrusive. It is unacceptable to have to wait for many seconds or minutes before DataFrames or Plots load and processing transitions to your own code.

  2. Debugger must be available.

  3. It must be easy to call Julia from Python or from R.

  4. Discovery of packages must be easy.

  5. There must be a solid selection of success stories and inspirational projects in Julia.

24 Likes

For me, defining “success” as fast growth seems ill-conceived. Even having Julia as a “C replacement” for Python or R misses the point. Julia can only survive as its own thing IMO, and like @mkborregaard wrote, those things take time.

I’ll throw out my plug for Julia Bloggers here, as its a great way to keep building community, getting examples out there for the broader public. Or write the library you wish existed, contribute to other libraries…

10 Likes

@randyzwitch I know that taste varies greatly so this is only an opinion. I think that the blogs such as the one you linked to leave much to be desired as far as aesthetics are concerned. Actually, I think if I could just pick out one detail to change I would say the code blocks. My recommendation, for whatever it’s worth, is that there be proper text highlighting at bare minimum. Beyond that I would recommend a dark theme for a couple reasons, (1) as far as I can tell there are a greater number of programmers who use an editor with a dark theme and it would feel more comfortable to them, (2) The contrast between the dark-theme code and the lighter colored blog will be pleasing and helpful in directing the eyes easily to what the reader is looking for.

Some of these ideas could perhaps even be applied to the main julia homepage too.

Of course, these are just personal opinions.

Yes. Using a language for scientific computing is all about packages. What can Julia do to show off packages? Here are some things.

  1. Complex number support. Simply put, a lot of physicists (especially quantum physicists) and electrical engineers need complex numbers. Package ecosystems which don’t work with them require a lot of extra work since then the user needs to split complex numbers into their real and imaginary parts. MATLAB handles complex numbers quite well across the board, but its “package ecosystem” is ehh… while in R/Python it’s really hard to performantly support both real and complex numbers in the same routines. Julia can do this easily through generic typing and adding appropriate dispatches. If NLsolve.jl, ForwardDiff.jl, Calculus.jl, and Optim.jl were made to fully work with complex numbers, this would be an area where Julia would be uniquely good and thus there would be specific fields where Julia could easily get a foothold.

  2. Arbitrary precision arithmetic. Quite simply, when packages have to be written in C/Fortran, arbitrary precision is pretty much impossible to support. This is why most of MATLAB/Python/R packages can’t do it. You need to look for specific instances where arbitrary precision matters, and show that Julia packages support it. My example is always differential equations, and in that area people always talk about floating point errors and whether things have converged sufficiently. With arbitrary precision, you can (if you have the time) solve the problem with 1e-20 error or something ridiculous, and put all questions about accuracy to rest. This is a great advantage in this field. @dpsanders’s work highlights another use. Also @lbenet’s TaylorIntegration.jl highlights high-precision arithmetic for safe integration of chaotic systems, something which is always a difficult problem to convince people (and yourself) when you have enough accuracy. Find fields where this matters in say optimization, and highlight it using Optim.jl (does this support arbitrary precision? We should check) in a blogpost.

  3. Iterator-based APIs with callbacks. This is more detailed, but in Julia you can build iterative-algorithms in a way that actually uses the iterator interface, and then give (at least in a separate lower-level API) full control over step-by-step behavior, along with allowing users to easily inject any code in the middle since it’s just using the iterator interface. This is a huge amount of control, and so this can definitely be a really nice way to generalize the functionality of iterative linear solvers and optimizers. I think Optim.jl has started to make use of this? The native DifferentialEquations.jl stuff does, and I think IterativeSolvers.jl had a work-in-progress PR here? Then you just have to think up interesting stuff to show with this. You can make an updating animation on what the local landscape and convergence looks like if you have this ability with an optimizer for instance.

  4. Domain-specific libraries. Because speed of package development is accelerated by Julia (at least if you care about performance), it’s much easier to build something very specific to your field. Do it.

  5. Abstract array typing for parallelism. Abstract array types seem almost silly, but if you make your optimization package only use broadcast and then someone throws a GPUArray in there, then if the user passes an f that is also some GPU call (either only uses broadcast or is a GPU kernel written in CUDANative.jl), congrats you have an optimization algorithm which rests entirely on the GPU without transferring the arrays back to the CPU. This can be taken advantage of all over, and really help Julia ride the GPU hype-train better than other languages.

  6. Metapackages. I think Julia’s final underused strength is the fact that all functions compile and will inline small calls. This means that you can easily make functions like (x,y)->f(y,x) which flip arguments, and Julia will inline this. By doing so, you can make something performantly sit on top of a bunch of other packages and slap a single API and documentation together, and really unify things in a way that say “this thing does every variation of X”. JuMP, Plots (well, Plots has some performance issues due to something unrelated here which can be fixed), Bio, and DifferentialEquations do this. I think this is important. People don’t like Python, they like SciPy/NumPy. We need to build these bigger “premiere” packages that are easy to point to. The issue with Julia’s current development is that its so fractured and everything is a single method in some dude’s repo. Users who aren’t developers don’t want to search for these edge packages and vet them themselves. In some sense, packages which pull functionality together, give it a community with some vetting process and community-knowledge is essential for adoption. Julia tends to have a lot more of “single method from recent paper” than “headliners” (SciPy, Pandas, etc.) and most people (for better or worse) just don’t trust those kinds of small projects, especially in terms of long-term support.

31 Likes

Maybe I’m misunderstanding the point of the original post. I linked to JuliaBloggers just to highlight that the community of readers is beginning to emerge, and that anyone who wants to write beginner articles has an instant audience.

How CSS fits into that and growing the Julia community? I mean, I could change the CSS, but I’m thinking that’s not really keeping people from adopting the language…

1 Like

Sorry @randyzwitch …by my comment it probably looks like I’ve gone off on a tangent. Also, I would definitely recommend getting others’ opinions before changing the CSS because I could well be the one with bad taste. Please continue to make blogs and tutorials. They are important and much needed. For that reason I appreciate your effort and the point you made.

My first comment was a bit lengthy but more clearly explains my main point. The main point being that one thing that could help Julia grow is to produce good teaching tools. If we want people to enjoy learning and also to make that process easier for them then it would be good to take aesthetics into consideration.

I’m not a graphic designer but I am willing to help out where I can …please let me know!

A lot of great suggestions have been made. Obviously there are a lot of things we can work on to help julia to continue to grow and mature.

We need a cuddly animal. Seriously.
Perl has the Camel. Python has a python (which are VERY cuddly snakes). Linux has Tux the penguin.
I think the Julia triangle logo is excellent, and we have seen how it is adapted to the various packages like Julia Physics.

There was a discussion on having perhaps a stand at the FOSDEM conference. We need something to be visible at an event. Also something which is a bit of fun.

8 Likes

I used Julia for the first time with my students last year, and the course foundered badly on one simple issue: Students like cute visuals. To make headway with young learner programmers, Julia must offer a single, standalone, easily usable and imageable graphics package for a wide range of purposes like graphing and 3D visualisation. The current stitched-together bag of dependencies on Python is just too complex for beginners to want to be bothered.

I guess another issue is the ability to just start and use Julia (without an IDE) in an arbitrary folder without having to reset any environment variables. Unless I’m missing something, that’s still a little clunky.

1 Like

Could you be more explicit what you mean? Post an example of some graphics you haven’t been able to do easily in Julia?

1 Like

I taught a course using Julia in 2017 February-March, and my experience was very different. I found Plots.jl to be eminently usable (if not perfect — nothing is :wink:) and it has improved steadily ever since. There are always issues, but once I open one, I get advice or a solution very quickly.

Plotting is a hard problem. Everyone is aware that Julia should have a good plotting framework, but it is more of a matter of a lot of hard work than a simple decision to do it. The way you describe the current state of Julia’s plotting ecosystem doesn’t recognize the enormous amount (unpaid) work that went into it — it took a lot more than “stitching together” existing stuff.

7 Likes

I’m going to print that quote on a t-shirt.

6 Likes

What this guy said!

I actually think Optim/NLsolve falls in some of these places, but, as your point pretty much is, there’s no real reason to. I’ll make sure to test arbitrary precision support. We’ve been using a sort of iterative setup in Optim that allows you to pass callbacks for quite some time now, but maybe we should use the actual iterator interface - somebody’s got to do it though :slight_smile: Complex number support is coming very soon.

Yeah. My point is Julia is good at handling generics like this, so there’s no real reason to not support these things. We should make them ubiquitous and put applications of them on display. Performance is one small way to get people to come, but that’s still only a niche group of people who actually care. You need to look to other niches, and the best way to do that is to just have features which you can’t find other places. Complex numbers, arbitrary precision, and the ability to inject essentially arbitrary code for full control is definitely something we can take full advantage of, and then point to as a reason to use Julia’s packages.

3 Likes

You are absolutely correct. That description was disrespectful and I apologise to everyone. It arose more from the fact that I’m currently very short of time than from any animosity - sorry!

The fact of the matter is, though, that in order to do any normal everyday graphics work you need more than one package - each has its own strengths. So you really need to use Plots, which links them (very nicely) together. However the result is that you end up installing a whole bunch of useful backends plus Python. Now, on my machine that’s not a problem, but for students working on university computers it’s more complicated, since they work with distributed images of the original installation, and these became very unmanageable.

First, the images were quite unwieldy, making them difficult to update on all machines as we slowly found problems with the installation. Second, there were several aspects that we never got working properly. Unfortunately I can’t remember offhand exactly what issues these were, but I know one was that the only way we could draw graphics was via a browser.

I hope that helps somewhat. Sorry again for my rude explosion.

1 Like

Actually I believe a good way to make progress in that direction is to have at least one backend (which should be default) that is good enough at more or less everything and easy to install. My understanding is that this is happening little by little with the GR backend which:

  • doesn’t require python, you just download the binaries
  • is very fast
  • can save nice vectorial graphics

But I still think it needs

  • a little more polish (especially when making plots for a paper)
  • nice interactive display (which should be coming soonish, there was already a preview in the JuliaCon videos)

It’s really important to emphasize that a good plotting package really takes a lot of work as the number of possible features/bugs is just too big.

Personally, I think that the experience of somebody who, like you or @Tamas_Papp, takes the gamble and teaches a course in Julia is extremely valuable because you end up:

  • figuring out what are the current limitations from a student’s perspective
  • probably doing some PRs to fix the most urgent problems
3 Likes