I kind of went the other way around. I was a big Julia advocate, pushed a couple of clients to use Julia (Flux) for new products.
Now, I mostly use Python with C for speed or Go for projects. I have embraced Python as a glue language and it excels in that role. Also, easier to get support from Claude/GPT. Purely commercial/convenience reasons.
Julia is still my high school sweetheart.
4 Likes
I like that you still gave it a shot. I don’t believe Julia is the solution to everything. For my work and role, I’m almost 100% sure it’s the long-term solution. But other types of jobs will definitely benefit from other languages more than Julia, either by context or by nature of the job.
1 Like
That´s fair, but that indeed depends on learning the language further. The advantage you can get from Julia is if you can fuse multiple vectorized operations you are forced to implement in Matlab into custom functions that take advantage from the structure of your problems. And that depends not only on the language, but leveraging its flexibility to implement new algorithms.
That said, if the bottleneck of the code is in linear algebra operations, which is frequently the case, there´s not much one can do except tune those specific points to use the underlying library as efficiently as possible, and that being properly done in both languages will provide the same performance. Than, you can´t sell Julia for perfomance, other properties of the language must be taken into acccount, if they give any benefit for the workgroup.
2 Likes
I share the frustration of being very much in a minority, even working in a research environment where Julia could/should be able to deliver real benefits. The usual challenges prevail: no time to learn a new language, too much stuff already written in other languages, some things that we would need from Julia still look immature (e.g. still a significant number of packages at version 0.x or even abandoned).
I hope to find a use case where I can show an “easy win” by switching something to Julia and showing a great improvement in productivity and performance, but there aren’t really that many low hanging fruits to pick. We have many big codes with many years of work done on them, sometimes in multiple languages. Taking a leap of faith like the CliMA team did is quite a big ask.
Whilst I’m impressed and intrigued by the unique capabilities of Julia’s multiple dispatch and pre-compilation, the main real-world use cases I’ve employed turned out to be from the humble ccall interface. I’ve had a couple of instances where we could spend weeks battling cython to get it to play nicely and let us wrap a C library in python, then I could literally just fire up a Julia session and write a one-line ccall and it’s done it already. I’ve built on these examples to provide an interface between a powerful expressive programming environment (Julia) and an otherwise opaque and hard to use C API library. But these are just building blocks.
One case where Julia just can’t compete with python is scripting. There are so many cases where someone will just run dosomething.py from the terminal and it does something. It might be a slow, interpreted language, but python scripts get going right away and maybe the script only takes several seconds to run. If I rewrite it as dosomething.jl, then we have to wait several-times-several seconds for pre-compilation to be done before it then does the same task in milliseconds. So to beat python one has to convert the whole script into an application, or create a special sysimage etc and then it’s no longer something that can be distributed to others as simply as a script. I don’t know that we’ll ever see a solution to this because we only get the performance benefit from cases requiring multiple passes of the same code, not a one-time run. This also has implications for trying to create a serverless framework in Julia. If you have an event that triggers a Julia container to spin up and run a function then tear down again, it takes the hit of startup latency every time. To maximise the effectiveness of Julia we would have to change the paradigm to maintain a continuously-running but mostly dormant Julia function that “wakes up” each time an event provides new data to process. Changing the framework to suit the Julia modus operandi is another big ask.
8 Likes
If this is the biggest hurdle, look into DaemonMode.jl.
2 Likes
I sincerely doubt DaemonMode.jl, while useful during development in its own right, is helpful in a setting where you’re deploying your application into production, as is often the case in a commercial setting.
2 Likes
For sure, it seems helpful mostly for people who need/like to run scripts.
1 Like
thanks, that’s certainly something I’ll look at, but it kind of implements what I said about needing a continuously running dormant Julia process instead of a conventional event-driven container workflow. It might be something to consider for on-prem applications but I guess the cloud costs of leaving a Julia daemon running would be unthinkable.
3 Likes
By making a package out of your script (which is trivial) and adding a precompile using PrecompileTools.jl (which can be running the same script once with some toy data) you can reduce the startup time to “several seconds once”. Maybe a bit longer than firing Python, but probably acceptable.
5 Likes
Indeed, and just following JuliaCon 2024 I’ve discovered JuliaScript.jl, which automates using the precompiler and optionally creates custom sysimages. I gave it a go on my simple “python replacement” script and indeed it made a difference. The first cold run from julia took 10s. The first cold run with juliascript took 15s including cache-building, then the subsequent run (from the command line, not julia) took 1.1s using the cached version. Using the packagecompile option to make a sysimage brought that further down to 0.65s. These are probably a bit approximate because the script uses external REST services that might have variable response times.
Again though, all these approaches require the creation of additional artefacts beyond the original .jl script, but I guess there’s no free lunch here.
Sorry but I feel my comments on this topic have somewhat hijacked the original thread, which was more about general experience with Julia vs what the workplace demands.
6 Likes
Possibly, but you may be unable to convince your boss and/or team about this. If this is the case, your best choice could be quietly dusting off your CV and looking for a company that uses Julia.
As a rule, very few companies have the culture to change to a new technology just because it is better (if your company is the exception, your boss would have been pressuring you already to try Julia, Rust, etc on pilot projects).
What happens instead is that employees who care about the quality of their code and want to avoid pointless frustration migrate to other companies, and are replaced by workers who tolerate it well. Eventually projects collapse under technical debt, but this is never blamed on choosing the inferior technology.
You might find this blog interesting:
6 Likes
I’ve been thinking about what you’ve said here, and it makes a lot of sense. On one hand I can hold the knowledge that Julia would make my workplace experience and my company’s output potentially so much better. On the other hand, if my company hasn’t pursued new and improving technologies (which has always appeared as a contradiction for a tech company in my mind, apart from supporting legacy code products) then they likely won’t unless leadership changes their mind.
I guess, in general for scientific programmers I would agree with you. But my field is so deep and interesting to me that I can see myself working here for a long time. When I interviewed for my job, my boss told me that it would be about ten years before I could really understand the field enough to make major contributions. I’m seven years in, and with each year that passes I’ve learned more of what he meant. A colleague of mine said that for him it was more like fifteen years for his own circumstances.
The mindset I’ve been preparing myself to accept is: if my company never adopts Julia, and I choose to stay, then for the sake of my mental health I should accept that I’ve chosen to stay, which means accepting the use of MATLAB and C++ along with their pains.
What bothered me significantly from an early stage is that my boss never entertained the idea of using other languages. One my colleagues values Python (understandably) and our boss dismissed Python without so much as an experiment. While I’m grateful we’re not using Python, I’m not grateful that he’s not giving it a shot. He’s treating Julia the same way. And it’s not out of dislike for these languages - his explanation is on the costs and time (which is also cost) of training, porting, and the nuanced challenges that come with the new language, which I understand and accept as an explanation. But as I’ve said earlier, I can’t help but see it as an investment that will be worth it.
Another perspective I’ve also developed over the years touches on what I said above about my field usually demanding at least ten years of experience and study before making major contributions. Were we to adopt Julia earlier, the path of porting and training would’ve introduced significant overhead costs and negative impacts to products delivered to customers. This would be due to me learning both Julia and the field at the same time, combined with the aforementioned challenge of porting. On the other hand, having Julia in pilot projects as you’ve mentioned would temporarily relieve my MATLAB and C++ pains knowing that at least we’re giving Julia a go with some proof-of-concepts and gradual training over the years.
At any rate, thanks for your insight. I’ve been comforted significantly by the conversations in this thread, and hope others in similar situations have also. Just to receive some empathy means a lot since no one at my workplace empathises with how learning Julia has impacted my workplace experience. I’ve had the feeling since I started this thread that even after a lot of discussion here, I’ll still hold to my decision to stay with my company. But I also started this thread to help me with my mindset about coming to accept my stay in my company a little more. Especially noting that adopting Julia doesn’t mean my workplace experience becomes devoid of difficulties - difficulties will come with interoperability, migration, training, etc. Even in my personal projects I have difficulties in learning Julia, and expanding that to my workplace would be both rewarding and challenging. All this said, would I still switch to Julia at my workplace if I could? Hell yeah.
3 Likes
There is another very viable route in many companies: Every so often, one needs a proof-of-concept. That is, a very small team (often single person) gets to write something new, very rapidly and quick-and-dirty, in order to learn and demonstrate whether something is reasonably possible and what the unexpected problems are. It is the full intention that this kind of prototype / demo will be thrown away, and if successful be replaced with a full rewrite.
Under these conditions, nobody cares about language / technology choice. So if you are the person doing the prototype, you can decide to do it in julia, smuggling the language in with whatever cool thing you’re doing.
This opens the door to new language / technology choices: It allows the organization to gain experience with it, build “libraries” of discarded code snippets that work with the very custom company internals, and demonstrates viability of the language / technology.
If the prototype is very successful, it also gives lots of internal political ammunition to argue “do we really need to rewrite this? Could be better to clean up the messy code and change company policy about languages. Maybe deploy the prototype now to keep customers / investors happy, and schedule a rewrite for Q4 (narrator: q4 of 2030)”.
This is e.g. how python and bash creep in everywhere. It is also reminiscent of the facebook / low-bit-rate-codec story.
11 Likes
Ideally yes, but in practice not always. Scenarios that prevent this include devops refusing to support the language or the stack, the need to communicate with coworkers who did not write but need to understand the code (to port it to another language if it works out, etc), or company policy forbidding it, just because.
The idea of smuggling in new tools which are then recognized for their superiority is very appealing as a story, but it still needs a supportive environment to work out.
Julia (and Python, Rust, …) are well past the point of having to demonstrate viability. This does not mean that they are the perfect match for every project, just that one has to be living under a rock not to be aware of them as competitive tools. The scenario you are talking about was applicable for Julia approximately five years ago.
I hope it works out for you. But be aware that work circumstances that require this kind of thinking may lead to burnout in the long run.
If I was your boss, I would allow you to allocate 10% of your work to Julia (or whatever you like), purely for the sake of your emotional well-being, regardless of productivity consequences (which, incidentally, I would expect to be positive, but that is beside the point).
7 Likes
If you need devops to deploy, you’re way past “prototype” stage.
They are not: In every single environment, one has to demonstrate-through-lived-experience viability to their individual coworkers on their specific problems and company cultures. The standard counter-arguments are aplenty: “rust is not productive enough and impossible to hire for”, “python is unmaintainable and slow, and we need static types for IDE support”, “julia is impossible to hire for, and we need static types for IDE support”.
These are not stupid counter-arguments! But they don’t apply to “quick prototype” situations, and every choice has its downsides. Tradeoffs.
While this is a nice thought, I think the appropriate way to think about it is the following:
For most people, it is impossible to stay productive concentrated on coding for 8 hours.
This is built into your salary and planned for: The workday also contains time spent socializing and coordinating with people, browsing the internet, some slacking off, some very productive stretches, and long debugging sessions.
Asking an employer to grant you 10% for OSS work already concedes that this is your employers decision to make. I object on political grounds: It is your inalienable right as a worker and not their decision, you’re not a slave, and you’re also not freelance consulting for 100 days/year with obscene daily rates and 260 days/year of downtime.
You can totally treat your dayjob as a job, make sure that you stay reasonably productive at it and don’t burn out, and also do your fun stuff (which includes julia side-projects for many people hanging out here). Just don’t ask for permission.
7 Likes
This was tough for me going to remote work. I no longer spend time “chatting at the water cooler.” But it feels weird spending that same time messing around with Julia.
Good reminder for me to stay the course 
6 Likes
I actually care about the language the proof-of-concept is in.
I’m taking a break at work right now to share a MATLAB design issue my boss and I just currently tackled, and this issue keeps reoccurring: because everything in MATLAB is an array, when writing MATLAB data to JSON, MATLAB doesn’t know what needs to be written as a scalar or what needs to kept as a single-length vector, or even a single-element array for that matter. What’s the workaround? Checking each field’s value that needs to be an array, and wrapping it as a cell if it has one element. And that wrapping-checking needs to be done at multiple levels of the field’s value for higher-dimension arrays.
People say prototyping in a higher level language like MATLAB is easier, and it is. While that is still true, the choice between say MATLAB and Julia is not even a question because MATLAB’s design choices call for so many workarounds, with everything being an array (or a cell) being only one of many design choices that make the prototyping stage longer and harder than it should. I have my own MATLAB package filled with versions of built-in matlab functions that function as they “should.” I was just complaining to my boss (and to my colleague yesterday, and another colleague last week) about the inconsistency of MATLAB’s built-in isscalar, isvector, and ismatrix functions that makes them difficult to use for array manipulations higher than 2 dimensions.
Even on the road to examples, the language’s syntax pain-points will be encountered. But I promise I haven’t forgotten to provide some examples, I’ve just been distracted. I would copy-paste some with screenshotted benchmarks, but my work has a lot of stuff I’m not allowed to share.
isscalar(ones(1)) % true
isscalar(ones(1, 1)) % true
isscalar(ones(1, 1, 1)) % true
isscalar(ones(1, 1, 1, 1)) % true
which is “fine” for MATLAB’s “everything’s an array.” However,
isvector(ones(2)) % false, but more of a syntax pain-point - I inputted "2" but got an element with not 2 elements
isvector(ones(1, 2)) % true
isvector(ones(1, 1, 2)) % false - `isvector` not generalised to higher dimensional vectors
isvector(ones(1, 2, 1)) % true - okay, sure
isvector is documented as such that it’s true for N-by-1 or 1-by-N.
ismatrix(ones(3)) % true - again, a syntax pain-point
ismatrix(ones(3, 3)) % true
ismatrix(ones(3, 3, 3)) % true - good
ismatrix(ones(1, 3, 3)) % false - again, not generalisable. Have to use `squeeze` function
ismatrix(ones(1, 1, 3, 3)) % false
ismatrix(squeeze(ones(1, 3, 3))) % true, of course
ismatrix(ones(3, 3, 1)) % true - okay, sure
My point is, prototyping in MATLAB is quicker than low level languages, sure. But MATLAB’s design often needs workarounds to get something simple working, which takes up more time.
You should see my broadcast.m function in MATLAB for keeping operations on multi-dimensional arrays consistent. It’s very long. It prepares inputs for MATLAB’s arrayfun. And compared with Julia broadcasting, still very slow. I refuse to vectorise things in MATLAB, logical errors have come from mistakes. And I refuse to manually type out every nested for loop for multidimensional operations, that is such a waste of time, and making changes to it also takes such a long time.
1 Like
This is turning into a negative-energy venting space, sorry everyone.
On my point about vectorisation mistakes: miss one, and sometimes MATLAB will parse it as valid, e.g. my colleague made an honest mistake in the following equation (a_old) whose value got used in reports given to customers, the downstream logical error was huge and was found by happenstance. I can share this equation because it’s Equation 1.1 of Ainslie (2010, “Principles of Sonar Modelling Performance”, albeit somewhat different because equations get copied between literature sources and can sometimes change either by rationalisation or mistake). Point is, the missed vectorisations didn’t error.
fr = 1:1000;
a_old = 0.001 + 0.003 * (fr/1000).^2 + (0.1094*(fr/1000).^2)/(1+(fr/1000).^2) + (43.75*(fr/1000).^2)/(4100+(fr/1000).^2);
a_fix = 0.001 + 0.003 * (fr/1000).^2 + (0.1094*(fr/1000).^2)./(1+(fr/1000).^2) + (43.75*(fr/1000).^2)./(4100+(fr/1000).^2);
abs_err = sqrt(sum((a_old - a_fix).^2));
rel_err = mean(100 * abs(a_fix) / abs_err); % 4.1498 percent error - non-zero, with significant downstream error accumulation
Still want to do multidimensional operations in MATLAB? Other means for doing so have horrible inconsistencies. For example:
% Input vectors: 3 by 2; output matrix: 3 by 2 - good
interp2(0:10, 0:10, rand(11, 11), linspace(0, 1, 2), linspace(0, 1, 3)')
% Input vectors: 2 by 3; output matrix: 2 by 3 - good
interp2(0:10, 0:10, rand(11, 11), linspace(0, 1, 3), linspace(0, 1, 2)')
% Input vectors: 2 by 3; output matrix: 3 by 2 - inconsistent
interp2(0:10, 0:10, rand(11, 11), linspace(0, 1, 2)', linspace(0, 1, 3))
% Input vectors: 3 by 2; output matrix: 2 by 3 - inconsistent
interp2(0:10, 0:10, rand(11, 11), linspace(0, 1, 3)', linspace(0, 1, 2))
So interp2(x, y, Z, xq, yq') won’t necessarilly produce the same result as interp2(x, y, Z, xq', yq)'.
Have been talking about just prototyping, these issues and more just scale horribly with the size of the project - and my company has some large MATLAB projects.
People keep saying to address my issues to MathWorks, but there are so many issues that I’d essentially be pushing them to do what Julia does. My boss says “anything you can do in one language, you can do in another” which he uses as one of many dismissive reasons. I’d rebut each time but I reached the realisation that there’s no point attempting to have a constructive discussion with someone who’s already made up his mind. At the very least, I’d like my colleagues at work to admit the difficulties of MATLAB that are resolved by the design of Julia. But there’s just not even enough interest for them to look it up themselves. This thread and Julia correspondences in general are very comforting.
Anyway, this is probably getting to inappropriate levels of venting for this platform. Just having another hard day at work right now. But my point is, MATLAB is a BAD language for prototyping. Sneaking Julia into my work here and there has been a big relief. E.g. we have a very complicated collection of equations implemented in C++ that I’ve been tasked with documenting. It has a lot of mistakes not just in the equations but in the control flow that chooses the equations. I’m currently working on a port to Julia’s CAS (Symbolics.jl) with a minimal-as-possible port while using the reactivity of Pluto.jl to present the algorithms in the C++ code. I had already started porting the equations to MATLAB, but MATLAB’s CAS is horrible and you can’t use the same implementations for numerics as for symbolics there, so I’d have to port things twice, and it’s more work in general porting to MATLAB than to Julia, especially since MATLAB doesn’t have multiple dispatch. Also, I kept on making mistakes in the MATLAB implementation - porting to Julia has been easier because of the implementation similarities. I even keep making mistakes in documenting the equations in Microsoft Word (which is a whole other rantable topic) so automating the production of the equations presentations without worrying if the equations agree with the plots that created them, and minimising my mistakes is a big motivation for me to use Julia.
4 Likes
TMI warning.
Okay, re-reading these last two posts of mine have had me prompted to sit down with my boss to have a serious discussion about this on the basis of my mental health. I believe my neurodivergent symptoms may be playing a larger part in my struggles than I’ve realised. I’ve been learning about conditions like misophonia (which I have) where certain qualities of noises and lights induce negative emotions and physical symptoms in people. I might have some neurodivergent symptoms where I need things to be done a certain way, and if they aren’t, it induces negative symptoms in me. This relates to the sentiment expressed:
which I agree with, especially already experientially.
On that note, sorry everyone for the negative energy, and thank you again for your kind and insightful sharing of experiences and suggestions, I’ve been taking each one to heart and consideration in application methods. I’m happy to share such potential TMI as I’ve benefited from others sharing about their neurodivergent difficulties in their spheres of life.
10 Likes
Since you need to work with the code as an author. But my point is that your boss shouldn’t care – do a prototype in julia, then if successful port to whatever you’re using in prod (matlab? javascript? c++? brainfuck?). Do this 3 times, where the julia prototype is always faster to execute and write and has cleaner code, and you have a very strong argument to convince coworkers and your boss to give that a try.
Really the only thing you need to convince your boss of, initially, is that: doing a prototype in a different language and porting to matlab will end up with quicker turnaround as prototype-in-matlab and subsequent quasi-rewrite-to-prod-quality-matlab, at least if it’s specifically YOU who is doing that.
3 Likes