What can we do to make Julia grow fast?

For me, defining “success” as fast growth seems ill-conceived. Even having Julia as a “C replacement” for Python or R misses the point. Julia can only survive as its own thing IMO, and like @mkborregaard wrote, those things take time.

I’ll throw out my plug for Julia Bloggers here, as its a great way to keep building community, getting examples out there for the broader public. Or write the library you wish existed, contribute to other libraries…

10 Likes

@randyzwitch I know that taste varies greatly so this is only an opinion. I think that the blogs such as the one you linked to leave much to be desired as far as aesthetics are concerned. Actually, I think if I could just pick out one detail to change I would say the code blocks. My recommendation, for whatever it’s worth, is that there be proper text highlighting at bare minimum. Beyond that I would recommend a dark theme for a couple reasons, (1) as far as I can tell there are a greater number of programmers who use an editor with a dark theme and it would feel more comfortable to them, (2) The contrast between the dark-theme code and the lighter colored blog will be pleasing and helpful in directing the eyes easily to what the reader is looking for.

Some of these ideas could perhaps even be applied to the main julia homepage too.

Of course, these are just personal opinions.

Yes. Using a language for scientific computing is all about packages. What can Julia do to show off packages? Here are some things.

  1. Complex number support. Simply put, a lot of physicists (especially quantum physicists) and electrical engineers need complex numbers. Package ecosystems which don’t work with them require a lot of extra work since then the user needs to split complex numbers into their real and imaginary parts. MATLAB handles complex numbers quite well across the board, but its “package ecosystem” is ehh… while in R/Python it’s really hard to performantly support both real and complex numbers in the same routines. Julia can do this easily through generic typing and adding appropriate dispatches. If NLsolve.jl, ForwardDiff.jl, Calculus.jl, and Optim.jl were made to fully work with complex numbers, this would be an area where Julia would be uniquely good and thus there would be specific fields where Julia could easily get a foothold.

  2. Arbitrary precision arithmetic. Quite simply, when packages have to be written in C/Fortran, arbitrary precision is pretty much impossible to support. This is why most of MATLAB/Python/R packages can’t do it. You need to look for specific instances where arbitrary precision matters, and show that Julia packages support it. My example is always differential equations, and in that area people always talk about floating point errors and whether things have converged sufficiently. With arbitrary precision, you can (if you have the time) solve the problem with 1e-20 error or something ridiculous, and put all questions about accuracy to rest. This is a great advantage in this field. @dpsanders’s work highlights another use. Also @lbenet’s TaylorIntegration.jl highlights high-precision arithmetic for safe integration of chaotic systems, something which is always a difficult problem to convince people (and yourself) when you have enough accuracy. Find fields where this matters in say optimization, and highlight it using Optim.jl (does this support arbitrary precision? We should check) in a blogpost.

  3. Iterator-based APIs with callbacks. This is more detailed, but in Julia you can build iterative-algorithms in a way that actually uses the iterator interface, and then give (at least in a separate lower-level API) full control over step-by-step behavior, along with allowing users to easily inject any code in the middle since it’s just using the iterator interface. This is a huge amount of control, and so this can definitely be a really nice way to generalize the functionality of iterative linear solvers and optimizers. I think Optim.jl has started to make use of this? The native DifferentialEquations.jl stuff does, and I think IterativeSolvers.jl had a work-in-progress PR here? Then you just have to think up interesting stuff to show with this. You can make an updating animation on what the local landscape and convergence looks like if you have this ability with an optimizer for instance.

  4. Domain-specific libraries. Because speed of package development is accelerated by Julia (at least if you care about performance), it’s much easier to build something very specific to your field. Do it.

  5. Abstract array typing for parallelism. Abstract array types seem almost silly, but if you make your optimization package only use broadcast and then someone throws a GPUArray in there, then if the user passes an f that is also some GPU call (either only uses broadcast or is a GPU kernel written in CUDANative.jl), congrats you have an optimization algorithm which rests entirely on the GPU without transferring the arrays back to the CPU. This can be taken advantage of all over, and really help Julia ride the GPU hype-train better than other languages.

  6. Metapackages. I think Julia’s final underused strength is the fact that all functions compile and will inline small calls. This means that you can easily make functions like (x,y)->f(y,x) which flip arguments, and Julia will inline this. By doing so, you can make something performantly sit on top of a bunch of other packages and slap a single API and documentation together, and really unify things in a way that say “this thing does every variation of X”. JuMP, Plots (well, Plots has some performance issues due to something unrelated here which can be fixed), Bio, and DifferentialEquations do this. I think this is important. People don’t like Python, they like SciPy/NumPy. We need to build these bigger “premiere” packages that are easy to point to. The issue with Julia’s current development is that its so fractured and everything is a single method in some dude’s repo. Users who aren’t developers don’t want to search for these edge packages and vet them themselves. In some sense, packages which pull functionality together, give it a community with some vetting process and community-knowledge is essential for adoption. Julia tends to have a lot more of “single method from recent paper” than “headliners” (SciPy, Pandas, etc.) and most people (for better or worse) just don’t trust those kinds of small projects, especially in terms of long-term support.

31 Likes

Maybe I’m misunderstanding the point of the original post. I linked to JuliaBloggers just to highlight that the community of readers is beginning to emerge, and that anyone who wants to write beginner articles has an instant audience.

How CSS fits into that and growing the Julia community? I mean, I could change the CSS, but I’m thinking that’s not really keeping people from adopting the language…

1 Like

Sorry @randyzwitch …by my comment it probably looks like I’ve gone off on a tangent. Also, I would definitely recommend getting others’ opinions before changing the CSS because I could well be the one with bad taste. Please continue to make blogs and tutorials. They are important and much needed. For that reason I appreciate your effort and the point you made.

My first comment was a bit lengthy but more clearly explains my main point. The main point being that one thing that could help Julia grow is to produce good teaching tools. If we want people to enjoy learning and also to make that process easier for them then it would be good to take aesthetics into consideration.

I’m not a graphic designer but I am willing to help out where I can …please let me know!

A lot of great suggestions have been made. Obviously there are a lot of things we can work on to help julia to continue to grow and mature.

We need a cuddly animal. Seriously.
Perl has the Camel. Python has a python (which are VERY cuddly snakes). Linux has Tux the penguin.
I think the Julia triangle logo is excellent, and we have seen how it is adapted to the various packages like Julia Physics.

There was a discussion on having perhaps a stand at the FOSDEM conference. We need something to be visible at an event. Also something which is a bit of fun.

8 Likes

I used Julia for the first time with my students last year, and the course foundered badly on one simple issue: Students like cute visuals. To make headway with young learner programmers, Julia must offer a single, standalone, easily usable and imageable graphics package for a wide range of purposes like graphing and 3D visualisation. The current stitched-together bag of dependencies on Python is just too complex for beginners to want to be bothered.

I guess another issue is the ability to just start and use Julia (without an IDE) in an arbitrary folder without having to reset any environment variables. Unless I’m missing something, that’s still a little clunky.

1 Like

Could you be more explicit what you mean? Post an example of some graphics you haven’t been able to do easily in Julia?

1 Like

I taught a course using Julia in 2017 February-March, and my experience was very different. I found Plots.jl to be eminently usable (if not perfect — nothing is :wink:) and it has improved steadily ever since. There are always issues, but once I open one, I get advice or a solution very quickly.

Plotting is a hard problem. Everyone is aware that Julia should have a good plotting framework, but it is more of a matter of a lot of hard work than a simple decision to do it. The way you describe the current state of Julia’s plotting ecosystem doesn’t recognize the enormous amount (unpaid) work that went into it — it took a lot more than “stitching together” existing stuff.

7 Likes

I’m going to print that quote on a t-shirt.

6 Likes

What this guy said!

I actually think Optim/NLsolve falls in some of these places, but, as your point pretty much is, there’s no real reason to. I’ll make sure to test arbitrary precision support. We’ve been using a sort of iterative setup in Optim that allows you to pass callbacks for quite some time now, but maybe we should use the actual iterator interface - somebody’s got to do it though :slight_smile: Complex number support is coming very soon.

Yeah. My point is Julia is good at handling generics like this, so there’s no real reason to not support these things. We should make them ubiquitous and put applications of them on display. Performance is one small way to get people to come, but that’s still only a niche group of people who actually care. You need to look to other niches, and the best way to do that is to just have features which you can’t find other places. Complex numbers, arbitrary precision, and the ability to inject essentially arbitrary code for full control is definitely something we can take full advantage of, and then point to as a reason to use Julia’s packages.

3 Likes

You are absolutely correct. That description was disrespectful and I apologise to everyone. It arose more from the fact that I’m currently very short of time than from any animosity - sorry!

The fact of the matter is, though, that in order to do any normal everyday graphics work you need more than one package - each has its own strengths. So you really need to use Plots, which links them (very nicely) together. However the result is that you end up installing a whole bunch of useful backends plus Python. Now, on my machine that’s not a problem, but for students working on university computers it’s more complicated, since they work with distributed images of the original installation, and these became very unmanageable.

First, the images were quite unwieldy, making them difficult to update on all machines as we slowly found problems with the installation. Second, there were several aspects that we never got working properly. Unfortunately I can’t remember offhand exactly what issues these were, but I know one was that the only way we could draw graphics was via a browser.

I hope that helps somewhat. Sorry again for my rude explosion.

1 Like

Actually I believe a good way to make progress in that direction is to have at least one backend (which should be default) that is good enough at more or less everything and easy to install. My understanding is that this is happening little by little with the GR backend which:

  • doesn’t require python, you just download the binaries
  • is very fast
  • can save nice vectorial graphics

But I still think it needs

  • a little more polish (especially when making plots for a paper)
  • nice interactive display (which should be coming soonish, there was already a preview in the JuliaCon videos)

It’s really important to emphasize that a good plotting package really takes a lot of work as the number of possible features/bugs is just too big.

Personally, I think that the experience of somebody who, like you or @Tamas_Papp, takes the gamble and teaches a course in Julia is extremely valuable because you end up:

  • figuring out what are the current limitations from a student’s perspective
  • probably doing some PRs to fix the most urgent problems
3 Likes

I did not consider it a gamble, rather an opportunity to teach something that is valuable to the students. It is my impression that many of the students kept on using Julia after the course, despite, of course, constantly grumbling about plots and the constant need to restart the kernel (this was before #265 was fixed). But this situation may be special to economics, where some prominent people are promoting Julia.

Actually, Plots.jl + Interact.jl allowed us to program some “experiments” about numerical methods, allowing students to explore various options visually. I think they found that pretty impressive (I know I did, it was very easy to set up). So even half a year ago I would not have complained about plots, it was an effective teaching tool. Back then we had to restart all the time because of #265, which, combined with the slow loading, was really painful (insert favourite Four Yorkshiremen reference here).

The single reason that makes me hesitate to push Julia universally at the moment is not plots, but the “dataframes” ecosystem. My impressions is that something very nice will eventually evolve, once named tuples are in and the interface stabilizes, but in the meantime large changes should be expected, and anything specific I would teach today would be obsolete very quickly.

9 Likes

I think a very good way to popularize a language is to provide lots of small examples that people find with Google. This is the best form of search engine optimization: People who searched for something seemingly unrelated to Julia will land on a page that solves their problem and introduces them to Julia as a side effect.

Let’s say you want to do a task X. You google it, but don’t find anything useful. So you write 20 lines of Julia code and the task is done. Now, if you would spend an hour putting your code in a small package, adding a couple of tests and a bit of documentation (making sure that the search terms that you googled are prominently included in README.md) and uploading it to github, then everyone who googles those terms in the future will find Julia.

This is something that anyone can do. You don’t have to be involved in a large project or be familiar with lot’s of pre-existing code. Julia makes it particularly easy to create a new project and upload it to github.

As a bonus: The person who is most likely to benefit from the extra work that you put in is you. Five years from now, when faced with the exact same task, you will google it and find our own code and documentation.

10 Likes

I would argue packages are not the correct place to put 20 line snippets in.
In my opinion, asking a stackoverflow question and answering it yourself might be a better strategy.

5 Likes

Another option is adding examples in the documentation (like in Matlab docs Cumulative product - MATLAB cumprod - MathWorks Switzerland).

2 Likes

What do you propose as a guideline for making a package?

I would argue that the number of lines is irrelevant. The relevant question is whether those lines of code do something useful. (Also, by the time docstrings, error messages and tests are written, the original twenty lines will have turned into somewhere between fifty and a hundred lines of code.)

There’s nothing fundamentally “expensive” about writing a package. There’s no finite pool of package names, or limited space in github. Nor does it require more work than copy-pasting the code into stackoverflow (assuming that you do an equally good job documenting the code there as you would have in a package.)

When using somebody else’s code, I much prefer copy-pasting an url into Pkg.clone(...) compared to copy-pasting code into a new file. That way I automatically have a reference to the original source, and I can easily get updates if the code is improved upon.

1 Like