We had a training on testing and package development with Python and Matlab… It is so much easier to do that with Julia… We just need to spread the message… I need to prepare a lunch-lecture on Julia…
15 Likes
times this attitude. Keep moving forward, using and learning Julia. There are many amazing people in the community and wonderful things built with it. That spirit of building and reaching toward launch, as exemplified in this clip, is something rare and worth sustaining.
3 Likes
That’d be correct. The new idea of Julia is that if you’re skilled enough in what you’re doing, you can do almost anything theoretically possible. Like, if some drone startup wants to optimize the drone by differentiating through finite element simulation and then use the sparse hessian to optimize for something, well, that could be something Julia excel, Likely, this would be something very few tools can do in practice, and Julia could get to be one of them.
Julia’s unrivaled capability is its ability to let users connect the dots from seemingly disparate fields and bring them together to create something awesome. That would be where Julia reign supreme. If we can take a stand there, there is almost certainly no defeat.
However, as Steve Jobs said, you cannot connect the dots looking forward. That means you need to explore various applications in the Julia ecosystem and even invite diverse areas of compute to develop in Julia.
I’d say keep inviting new people from different fields. No need for a lot of people. If we can get 2% of the most enthusiastic devs who can then put in 20% of the work to create a package that’s 80% as good as the best tool, but that happens to be able to work well with other tools in the Julia ecosystem, do that.
It looks like we’re in a long game.
2 Likes
I’m a relative newbe and started using julia only a few months ago. My previous programming experience was from the early '90s and earlier in geoscience, using Fortran and SAS for straightforward data analysis. I have no experience of other, modern languages.
I was initially attracted to julia by its promise of speed and capability for data analysis but also because it promised to be good for general purpose computing, too. I have been blown away by what I have been able to do since I started. It has been a steep learning curve, but amazingly productive.
Every time I come to this community for support, I get what I need from a friendly and helpful community of real experts.
However, and this is a ridiculously small “but”, the clear emphasis here on numerical methods over-awes me every time. I know this is a particular focus for julia and its user community, but anyone interested in “general computing” may be intimidated by what they find here.
I’m not for a minute suggesting it is bad that julia’s strengths are reflected in this discourse. That is absolutely how it should be. But, like @mnemnion, I’d welcome more general computing presence, too.
I’m not really able to offer a solution to this, but do think the focus on numerical computing might become a self-reinforcing reality and it really shouldn’t.
I can’t imagine switching away from julia - it’s too effective and I enjoy using it too much. I promote it to colleagues who, like me, are never going to be “hard core” coders. They may occasionally dabble in python. Postgraduate level numerical computing won’t ever be their thing, but julia could and should be!
21 Likes
It’s super unfortunate if this is the effect such sentiment has on you. I no longer spend much energy bemoaning the size of the Julia user base - I think I’ve been convinced by previous discussion that it’s not a problem - but in the past I did, and if that bummed you out, I’m sorry.
In my own mind, the motivation for the sentiment was that this is an amazing thing that you and so many others have built, and I just wish that more people could capitalize on that goodness, that it got more of the support it deserves. All of that is still true!
14 Likes
The person Stefan was responding to has a tendency to post very broad open ended questions bemoaning difficulties and unfairness and I don’t think it was targeted towards you at all.
11 Likes
Oh yeah, I know that it wasn’t. But I’m implicated, because I think that I’ve made similar comments in the past without a full consideration of the effects that those comments would have on the people putting in so much effort towards making this language amazing.
I hope that explaining where I was coming from might offer a different framing, and hopefully reduce the sting. I think it’s inevitable that some people, especially those newly excited about the language, will look around at the fact that it’s not more widely adopted and bemoan that lack - it’s just in the nature of fandom. But I think this can and should be seen in the light of
Wow, this thing is great, I wish everyone could see that
!
rather than
Ugh, it sucks that its so hard to get more people involved
Even if the later sentiment is what’s explicitly staid 
15 Likes
I know I’m chiming in a bit late on this discussion, but I would really want to emphasize the importance of the following comment
The state of Julia, to me, is a lot like the state of Industry 4.0 and digital twins. It’s full of promise, with people working on it, but not very many people are using it. Nevertheless, people who use it well are going to dominate. Moreover, Scientific ML is going to be the tool that makes digital twins actually useful.
The main problem is the fact that people unfamiliar with this area think that deep learning is enough to solve industrial problems. However, the data is generally not informative enough to fully define your operation. A real Industry 4.0 solution requires the seamless integration of expert knowledge and data to achieve something that is greater than the sum of its parts. But this seamless integration requires us to enable experts to easily build performant code that can be automatically differentiated, and iterated over through differential equation systems that combine first principles and black box models. The work that @ChrisRackauckas and his team have done in this area is nothing short of amazing. Julia is pretty much there (Enzyme.jl will really help) and is miles ahead of anyone else in this area. The problem is that all the chatter around AI obfuscates a few important points that people need to be aware of:
- Deep learning isn’t enough to make Industry 4.0 useful. The datasets require expert knowledge to make sense of, which means you really need to take a Scientific Machine Learning (SciML) approach.
- Feature engineering isn’t SciML; it can help but it’s extremely noisy; SciML can solve the problem more thoroughly/reliably by cutting through the noise.
- Physics-Informed ML is not SciML. Training a neural network on a bunch of simulations is a step forward, but the large number of unknown parameters still makes the approach quite brittle, and it’s very difficult to expand models to include more changing unknowns. SciML allows you to break the black box model components down into smaller, more tractable problems.
- SciML is more difficult, because you need people with broad technical knowledge to build these models, so they’re hard to scale. However, a single model of an industrial asset can provide a lot of economic value, so it is worth building properly.
- Julia has a SciML ecosystem that is far and beyond any other ecosystem out there.
The overall idea of SciML is not new: build high-level models with first principles and using small machine learning models to fill in unknowns. This is done all the time in thermodynamics, but has historically been about getting data from carefully controlled experiments to make it easy to “train” these models with high fidelity data free from confounding variables. The technological advancement is the ability to sufficiently train models like these as a deeply nested problem inside a large model with routine data.
12 Likes
Thank you for sharing your wonderful findings. I am trying to do similar things to your research. That is how many people are downloading for a given Julia package.
I mean I would like to get data something like below:
package_uuid, year, request_count
<UUIDExample>, 2020, 10
<UUIDExample>, 2021, 130
<UUIDExample>, 2022, 119
<UUIDExample>, 2022, 1120
I know there is a post Announcing: package download stats! that allows us to investigate package download statistics by myself. Can I get package download statistics “by year” or “by month” instead of “by date”? As far as I know we can get information from the date field in a CSV file named package_requests_by_date, which only covers from 1 month ago to today.
What makes you think that way?
Flux is probably one of the crown jewels of Julia packages, with support perhaps from the Julia compiler team itself, and even then, Pytorch has like 15x advantage in both commit and contributor count.
And PyTorch is only one of three main ML ecosystems in Python. There are also TensorFlow and Jax.
And this was despite Julia being designed with these sorts of data science in mind. Python was not even initially designed for machine learning.
Can you name someone from the Julia compiler team who is seemingly in this support group for Flux please?
That was quite a feat given that it was designed a few years before the term data science was coined.
7 Likes
Flux uses efficient autograd which I’d assume rely on some compiler hacks that only the compiler team knows. But even if it is not right now, if Flux switches to Diffractor, then there would be Keno Fischer in support.
Perhaps I was hallucinating because Linear Algebra was included in Julia whereas Python relied on Numpy to do the same task. And why would there be a need for linear algebra? I could only think of data science and computer graphics. Since computer graphics isn’t a major use case in Julia (at least yet, I can’t know the future), I’d assume it was data science.
Since linear algebra predates computers by hundreds of years, one would think it has applications outside of computer graphics or data science 
Nevertheless, why do you care so much about how popular Julia is? If it works for you, if you enjoy it, use it. Contribute to the ecosystem—if you can and if you want—to improve it. But why get so emotionally invested?
Clearly, there is a big enough circle of people who are motivated to push Julia forward and who see value in it, so why don’t we just let them do it without repeatedly asking them to defend Julia’s position?
6 Likes
There are numerous fields within mathematics, science, and engineering where linear algebra is a useful, if not essential, tool.
3 Likes
If you’re focused on neural network training then I get why you’re so bummed out about competing with Python. A few years back it looked like Julia could maybe competed there, but there’s basically infinite money to pour into making Python + C + C++ work no matter how much effort it takes, and that’s what has happened. Julia’s ML ecosystem has made steady progress but Python has taken off. But that’s only one relatively niche area of technical computing. Yes, it’s important right now, but ML frameworks get replaced faster than anything but web frameworks. Julia is killing it in areas where the language matters like optimization and simulation, which are frankly much bigger and more interesting areas albeit less “hot” right now. Believe it or not, there way more money in simulation than deep learning, although for various reasons it’s hard to convert it into language development money (but we’re trying to change that). And the real sweet spot for Julia is at the intersection: simulation with machine learning. Is SciML. So my advice is to have a broader and longer view and things look pretty exciting.
55 Likes
I’d think that most math, science, technology, etc. that are currently in use are based on linear algebra as a foundation. A few in some fields are trying to replace linear algebra with other constructions such as the geometric algebras or Hopf algebras (which I know nothing about but they seem cool on account of being more general than linear algebra), however linear algebra is basically ubiquitous.
2 Likes
Okay, if it feels like it’s hurting me more than a programming language’s popularity is, then I would say that it’s not the programming language. New programming languages get written all the time. It’s the thing it represents.
Let’s consider what I consider to be Julia’s philosophical nemesis, the Go programming language, Go was designed to reduce waiting time for compilation while providing reasonable performance and make programmers productive. However, if I were in that position, I would’ve not gone with Go’s design, I would’ve gone with a design similar to that of Julia (so me adopting Julia was indeed no coincidence). But Go’s designers were really smart, so what’s going on here? How could a similar goal turn into what’s essentially a completely different philosophy? Go has no overloading, Julia has hundreds.
Go tries to simplify the programs. Go prefers explicitness and simplicity. It didn’t even bother with generic until much later in the language’s life. To program idiomatically in Go means simplifying your thought process.
Meanwhile, Julia tries to encapsulate the program. Complexity underneath the surface is acceptable if it is wrapped nicely in a library. Julia encapsulates complexity and creates a nice and expressive high-level interface. Go is explicit about its types to reduce cognitive load in understanding functions, whereas, in Julia, type is something you don’t have to care about too much in your functions.
So, what’s going on? The answer is that people process information differently, so features imply different cognitive loads for different people. So, the Go programming language was designed like that because it reduces cognitive load for people using it and Julia was designed like that, also, because it reduces cognitive load for people using it.
And this is where it hurts for me, my cognitive style is alien, as if I’m a human in species only. The fact that the Go programming language (and its mindset) has become popular while Julia isn’t anywhere near the top 10 ranks demonstrates that my cognitive style is alien. For most people, their style of thinking was given from birth in the form of common sense, evolved from millions of years of evolution, and they think by relating concepts to common sense. Mine, on the other hand, is a way of thinking deliberately designed via metacognitive effort. And to explain why my way of thinking is vastly different from other people, that would also hurt.
Walking my way all the way into the programming world to find like-minded individuals only to find that I still don’t belong hurts.
By experience, expliciteness is very helpful to convey information, by contrast to encapsulation which instead requires to dig the info (think when you’re discovering an undocumented package you didn’t wrote).
(and btw, that’s why I don’t like the canonical using Foo approach, which blurs the origin of methods)
In that light, using Go in a team has some real advantages against using Julia.
(Not to say Julia has none, of course.)
As for your feelings, it’s sad that you feel that way. It looks to run deep in you and to go much beyond merely programming in Julia. But as I have no education in psychology, I’m pretty sure I’m not the best to help you
In my experience, having used many libraries that implemented some really complicated algorithms, or at least algorithms that I didn’t fully understand, encapsulation helps a lot. It doesn’t matter what language you use, if the algorithm is arcane to you and you don’t know how on earth it works, then you don’t really understand the code on a higher level.
Also, I do like the virtue of simplicity but I don’t like the implementation for several reasons. First of all, static typing, generic/polymorphism, and simplicity don’t mix, and you will need generic and polymorphism for several important libraries, so, going with static typing, it’s self-defeating.
Secondly, my idea is that the code cannot, in practice, be safe, performant, or simple, unless it is easy and expressive enough. This is because otherwise, one might inadvertently create a slow, bug-ridden, unsafe, weird, lisp/<insert language here, could also be your own weird language> interpreter. So, perhaps I better stay away from a Go project before it becomes a Lisp interpreter hell.
Not only is there more money in simulation than deep learning, there’s also better careers because simulation takes a lot more skill. There’s an order of magnitude fewer jobs obviously, but there’s at least two orders of magnitude less competition. Where learning Julia pays dividends though is if you’re a good data scientist on top of being subject matter expert (SME) capable of building simulations. Just having those two backgrounds will make you twice as productive as two separate people (a data scientist + an SME) when it’s needed. Having good enough knowledge of Julia to seamlessly integrate both areas of expertise will make you at least twice as impactful yet again; becoming a 10x programmer this way is not out of the question.
3 Likes