Why almost all the time I saw a comment about Julia on LinkedIn or some other forum, it is treated as a data science / big data / economics language?! Where did this come from?
I know Julia really excels in those fields, but it also happens to be one of the top environments for complex system simulations. Some engineers do not even try Julia because it seems “just” another version of R (I heard that, twice).
Any ideas about what the community can do to change this scenario?
I, myself, always saw Julia as a general-purpose language. Of course, there are some fields in which maybe Julia is not the best selection (like creating a web server maybe), but we see applications very far from the data science field.
Any ideas about what the community can do to change this scenario?
I got this hobby project. Its about “Neural network analysis of uncertainty and sensitivity of deterministic and probabilistic models in conjunction with quantum computing approaches to shortest path optimization algorithms based on geometric algebra”. I think its really cool and even though coding is not quite precisely my area of expertise, I’ve been working on it for a few months as I like it and I am finding the topics interesting. Well, it also includes potentially a significant component related to satellite technologies. I even see that some guys are sending some advanced stuff into space in order to differentiate itself and to get competitive advantage but frankly, as for now I cannot promise that. You think you may be interested in taking a look at it? If so, pls give me a shout. If such a case, I would be happy to provide you with a project description. EDIT: As well as try my best to answer any potential questions.
It is an interesting observation (and probably a call for action), because I always thought that this part is an obvious strong point of Julia, and that we need to justify that on top of this Julia is also good in “data science” scope.
I think you might be seeing this partially because of where you’re searching. LinkedIn is filled with data science/economics types that might just not have the numerical analysis/simulation applications in mind when talking about Julia. In fact, I would argue that, due to the hotness of data science right now, most forums will have more data types than strictly numerical types.
Anecdotally, of all people I’ve met while studying to whom I’ve mentioned Julia, physicists were actually the most likely to have heard of it (although I’m yet to find one that heard enough to move away from MATLAB/C).
Perhaps another reason is that MATLAB is just so dominant in simulation-heavy fields and whenever MATLAB’s performance runs out, people just switch to C/C++ to get things done without considering Julia. Can you really blame them for not wanting to try something new, however, when a billion tutorials exist for C++ and they probably know someone (maybe the research software engineering department) who can help with C++.
I think one problem that people have when switching over to Julia is that there’s a (relatively high) chance that they’ll find a roadblock somewhere. Either lacking in documentation, examples, functionality, or some other specific pain point that exists in Julia. Even though other languages are painful, it’s a pain they’re used to.
Nice! Yes, please, if you can share something with me, I will appreciate
I agree! However, it seems that people from outside the community is not getting the idea.
In fact, maybe the blame is on you @bkamins . You created such an amazing package for Data Scientist that everything became biased
I hope you are right! However, one reason why some people are not migrating might be because they are associating Julia as an R replacement, instead of a Matlab alternative. It can happens if they are looking the same forums as I am.
The reality is that we want to be “on par” with R/Python with data frame manipulation (of course there are places where I think we are superior, but in general R/Python are very mature in this area so it is hard to hope to be significantly better there) and I think that the users should switch because they find solutions like JuMP or SciML ecosystems (to name just two out of much longer list) superior for their “core” task.
I find the opposite. I’m in economics and I find that most researchers who do modelling still use Fortran or Matlab, with some using Python; and people who do causal analysis (basically regressions) mostly stick to either R or Stata, with some Matlab thrown in. However, I grant that the growth of Julia in economics has been rapid and more and more people are starting to pay attention.
Czesc! Absolutely, as for now, nothing is confirmed, however, should you also be interested in reviewing additional info please let me know. I will be very happy to provide more details, ideally in a direct message.
I do believe Julia has some unique advantages. On the other hand, as far as I know, there are situations when there is almost no other option than C / C++ or DPC++. Maybe in the future it will change, however, as for now it is like it is.
It’s funny how many different perspectives from various fields are on display in this thread. From my point of view in the ML (and especially DL)-focused part of the data science space, finding an organization that seriously uses Julia feels like discovering some mythical creature [1]. On the other hand, I also see what feels like a disproportionate amount of resistance to adoption and general push back on the language as a viable option in this space [2]. With those blinders on, it appears that things are going much better in the Physics/Bio/Econ/Ecology/Geo space.
Outside of handful of well-known ones that are super active in the community (e.g. involved in JuliaCon). ↩︎
We’ve all seen previous discussions of such on this very forum. Though for ML my experience is that the same characterization of “it’s just R/Matlab” is often expressed more derisively. ↩︎
Very few people, myself included, have the capacity to ask “what is the best tool for this job” because they are unfamiliar and certainly not proficient with every tool.
Instead they look in their toolbox and reach for the thing they think they can use to do the job.
The thing to do is write code that is not data science / economics related, publish it and make Julia a prominent feature. Then when the search for “julia” they can scroll past the pictures of Julia Roberts and perhaps find out more (why didn’t the smart OGs think about SEO !?!).
Also, I think the lack of UI is an impediment. Not one that concerns me but I can build a Neural Network in Matlab by dragging boxes around. I can build a Neural Network in Enterprise Miner by dragging boxes around. Despite using Julia for 5 years, I cannot seem to drag myself beyond the tutorial stuff in Flux.
Yeah, I fully agree with you guys that its not an easy topic. Anyway, if any of you might be potentially interested please let me know. I will be happy to provide additional information. Best way of contact is by a direct message or an e-mail. As for now, I see, artificial intelligence, quantum computing and satellite technologies as the main areas of focus (but not only).
Ahh, see I’m not looking for organisations, necessarily. I’m just speaking about my experience in my very tunnel-visioned academic “career” which consists of
Looking for the Juilians in my department,
Looking for companies/departments that use Julia,
There is no third step.
Truth is, I’ve not met an economist who uses Julia. My housemate studying financial economics had heard of it, but didn’t want to switch away from JAX.
To me, people just want a smooth experience in whatever they’re doing. Most programmers are scripters, most scripters only use it as a tool. If I’m using a tool I just want one that gets the job done with the least resistance, which is where I feel packages like the Tidyverse absolutely shine.
It’s not that Julia is bad for data science, it’s just that most people just need to glue library functions together to get their answers and Julia’s packages being less monolithic means looking for answers is harder than just searching “do X numpy/pandas/tensorflow/tidyverse”.
I’ve used it at every company I have worked for since 2016 but the code ends up being so specific, it is of little interest to others.
I make it as modular as possible so I can put the pieces on GitHub but even then, I never expect anyone else to actually use it.
I think it is just one of those things. My kind of work doesn’t generate any interesting papers for conferences or even blog posts.
“How to run SQL in batches of 100k records to get round mandated corporate response timeouts in AWS RDP Mysql using Julia’s @Threads and cache them using JDF”
There’s nothing really “Can your programming language do this? Julia can”
Well since economists are the best and brightest bunch of people, why wouldn’t we want to attach Julia to that group?
I’d say probably your perception is the result of good group efforts. In economics, QuantEcon (https://quantecon.org/) has done a tremendous amount of groundwork putting Julia at the forefront of computational economics. Also, a number of computational benchmark type papers (again, in economics) have been published as well and Julia was lucky to be included in those papers. Things like this give Julia some great publicity and other people obvious notice (because Julia looks good in comparison!).
SciML is another great example of this, as is JuMP. There are more and there will continue to be even more.
This is exactly my experience as well. The activation energy required to get over the inertia of sticking with Python/R is just not there right now. That’s not to say it can’t be done, but at least for ML it appears that the real success stories have been around finding niches that aren’t well addressed by library ecosystems in other languages.
But it’s not just about inertia. At least for the kind of work I do, there are still many gaps and papercuts that make it difficult to recommend the language + ecosystem in good faith to my colleagues. Again, there has been much ink spilled about the topic on this very forum, so I do recommend anyone interested to have a look at the various “state of [ML|data science]” threads that have accumulated over the years. That’s not to say there aren’t any on the other side: Pkg.jl alone would’ve prevented a good amount of hair-pulling around the mess that is Python dep management!
And so you might ask, why bother contributing to/maintaining packages you don’t see yourself using any time soon? Here’s where I think having a good community really pays off. Every testimonial is a motivator, every tricky real-world problem provides a concrete direction for development and each new contributor that sticks around is a force multiplier when the existing team is a handful of people working off the sides of their desks. That’s not to say it’s always easy to keep motivated, but all of these help
Pkg.jl is so good. Once you use it you never want to use pip or conda again… I like to joke that this business near my house exists because of how bad Python’s package systems are:
