Julia has ruined my work experience

Offtopic
1

Julia’s syntax for the most part is so satisfying. The community is an intellectual bunch that I learn from via a lot of lurking. And the direction Julia is headed in is quite exciting. I’m inspired by both types of package developments/initiations via any-sized collaborations, or even when one person saw/had a need and Julia’s design was excellent enough for them to just create a powerful and appropriately complicated solution on their own.

So what does this mean for me? At my workplace, I am mostly required to program in MATLAB and C++. I think I’ve upset my boss by the number of times I’ve grunted during work. A few colleagues connect with me on the difficulties of the languages we use. But no one is interested in Julia, and for the many years I’ve mentioned Julia, I’ve started to give up and untrain myself from pitching it. I still use it at work from time to time when appropriate.

But I can’t go a day of MATLAB without thinking “this can be done like this in Julia and it would be easier and faster in development and performance” - goodness I hate MATLAB’s OOP, unintuitive semantics and restrictive syntax. For my job, Julia is almost perfect and more effective.

The amount of boilerplate and poorly implemented C++ code we have to manage at work is mind-numbing. And Julia’s design happens to prompt such a large code reusability both within individual packages and throughout the Julia ecosystem. And that even adds to the robustness of the code when many people are using it in their different ways and collaborating on any discovered issues. Slowly my small team and I have been e.g. replacing the numerous versions of interpolation implementations within our code with an array and interpolation library; refactoring the code for flexibility, especially as new feature requests arrive. I’m fully convinced we could do our jobs faster and better in Julia.

On the other hand I understand the cost of migration. But I can’t help but see it as an investment that’s incredibly worth it.

So, to the Julia community, thank you for allowing me this moment to vent my frustrations while simultaneously thanking you for your contributions to the Julia language, and I appreciate the common vision and understanding of what a programming language can be for humanity, at least on the side of scientific computing.

I’d love to hear your views in response to my experience and outlook, in particular to educate me on your experiences and insights, as well as to cheer me up.

62 Likes
2

Can you show us with numeric examples how fast the Julian solution is when compared with the equivalent MATLAB solution.

I believe that numbers speak louder than words.

3 Likes
3

SyslabCC: Suzhou-Tongyuan’s proprietary Julia AOT compiler
Besides an AOT compiler, their product also have MATLAB language support and many toolboxes implemented (or in progress of implementation) with julia to ease the transition. I don’ t know the price, though, you can try contact them.

1 Like
4

Similar experience as a data engineer. I need to do some complex cleaning function on some columns of data. Do I cram the logic into SQL so Snowflake can do it? Maybe I could do it with Python on ingest… But then I either need to cram it into Pandas’ vectorized functions with all their intermediate allocations or use vanilla Python.

I mean, or I could use Julia. Maybe even create some custom types for these columns that automatically clean the data on construction so that the code is declarative and super readable. Easily manage parallelism. Oh, and while I’m at it, use Documenter to produce beautiful docs in CI.

But DE tooling in Julia isn’t great. And I don’t have time to build things from scratch right now. So I’ll probably just write another unruly SQL script.

5 Likes
5

Thank you for sharing this frustration. I hear this issue all the time in my circles.

Try to ask the following question to managers and people with decision power in your company:

What do you look for in the next years? Increase productivity? Reduce costs?Employee satisfaction? …

Then build concrete examples comparing Julia with the current technology. Make estimates of run time, cost reduction, hours of training, etc. Suggest 3rd-party companies to train your team (enterprise support). This is not an academic endeavor, and you need to show them that they have “MathWorks” support in Julia too.

There is no guarantee that this hard work will trigger change, but you will certainly leave a seed in the minds of the managers if you do your homework. This exercise will also make you feel more confident that this is the right time to transition to Julia.

As technical people, we tend to think in terms of technical criteria, but managers most often think in terms of business advantage. You need to align the two to succeed in this battle.

20 Likes
6

I wish there was a talk at every JuliaCon about how to promote Julia to companies and how to apply for grants. Knowing how to do it might encourage more people to give it a try.

18 Likes
7

Amazing idea. We need a community resource for this so we can share strategies that each of us find effective in different contexts.

I think sometimes people also overdo pitches, and it backfires and gives an overzealous impression, so would be good to include in the resource something on “How to not oversell Julia.” :smiley:

37 Likes
8

I usually sidestep uncomfortable language situation in projects by writing Python or Julia code that writes code that the project needs. That should keep people happy as it requires no extra support for a new language. Maybe the build process gets a bit more “complicated”, but not more so than having me click a few buttons every time I want to change the code.

2 Likes
9

I’m very lucky in the sense that I work on a team in an organization that essentially allows us to use whatever tools we are most comfortable with, so that we can add the most value. Adding value is what it’s really all about. I spent a lot of time writing up a value proposition several years ago to get Julia on the approved software list and that’s really the only advice I can give - think about why your use of Julia will allow you to add more value to your organization and then cross your fingers…on the other hand, the flexibility that I’ve been afforded may also be a double-edged sword because I’m not really willing to take advantage of a new employment opportunity if they won’t allow me to use Julia - it’s kind of a line in the sand issue for me at this point, which severely restricts my options. I would keep that in mind too. You may not be using your preferred tool, but by being flexible (or being forced to be), I think you also have other options since you are accustomed to using whatever tools your organization allows…if they do approve the use of Julia, you may end up feeling a bit trapped like I do because the idea of switching to another tool could become a deal breaker…

7 Likes
10

I gave a talk on how I convinced my workplace to incrementally switch from Python to Julia in JuliaCon 2021: https://www.youtube.com/watch?v=EnkfGuH6Qhg (To be fair, my workplace had 6 people, so not exactly a huge company.)

The key points were rewriting one slow important module (that was costing us $$$ in AWS fees), and setting up Python/Julia interop so it didn’t have to be an immediate, all-or-nothing proposition

11 Likes
11

I have many examples of Julia c.f. MATLAB both already coded, and ideas I’d love to compare, would be fun. Once I’ve finished my current project needed soon, I’ll get to making some screenshots of benchmarks. Will write some replies now while I’m waiting for some long MATLAB code to run.

12

A proprietary license :grimacing: but is amazing work all the same. I’ll try to remember to keep watch of that project, the lead sounds very passionate about it.

13

Discourse is advising me to reply to several posts at once.

That’s quite the mental journey of decision making, one which I’m familiar with haha. What’s DE tooling? D for Data, E for… Enrichment?

The funny thing is, many times my colleagues have made complaints and had their “moments” (a few times a year, over the many years I’ve worked with them) about something undesirable in categories such as MATLAB to C++ code gen, distributed computing, poorly designed legacy code, the price and walling of MATLAB packages, and many other things where I think to myself, “I haven’t used Julia in many of these situations, but allegedly and convincingly, Julia has demonstrated as a solution to my colleagues’ and boss’ complaints.” However, one time my boss mentioned Julia to me - ONE TIME - and it was to point out its low popularity and ranking. I feel like people see the evidences they want to see, and I can’t force them to open their mind. And I don’t necessarily want to make my workplace switch to Julia. The least I want is for me to use Julia as my main language, and that would be enough for me. Not possible in team environments. I have from time to time shown them things like a Pluto notebook and its reactivity, or the CAS Julia packages that helped me reverse engineer some horribly implemented equations in C++, but no takers (yet, hopefully). But I also understand people stick to their comfort zones. And as someone comfortable with Julia, I’m technically fighting to establish my comfort zone, and it’s been painful. Thanks for the encouragement, I’ll hang in there. (This response was longer than I expected.)

It would be awesome for us to collaborate on something like a presentation file (by Pluto.jl or PPT or whatever) filled with demonstrations of the most convincing aspects.

I do love the many talks that have been given in the context of workplace/community Julia adoption. My favourite is by Myrthe Scheepers at ASML.

One thing I do at work that I need to control my expression of is, I get frustrated with something about MATLAB, and my colleague asks what’s wrong, and I share what’s wrong and relate it to Julia, all in a tone of voice that’s frustrated. And I don’t think I like the idea of colleagues having memories of Julia mentions in a frustrated voice.

Are these situations where you can’t avoid boilerplate in other languages?

I have a similar experience in a different dimension where my boss and workplace allow incredible flexibility of when and where I work. I’ve recently started looking into my neurodivergence symptoms with my psychologist and psychiatrist, and throughout this journey my boss has been 100% supportive. He just strictly sticks to MATLAB and C++ on the bases of migration challenges: training; legacy code support/migration; customer deployment changes are very challenging in my field… it’s just not worth it in his eyes, and I understand his point of view. I do have a long-term plan of doing some big personal projects related to work that might pay off, but even if it doesn’t I’m still enjoying that journey. I digress… I see what you mean about feeling trapped, I hope you’re still able to enjoy your work, nice job on getting Julia approved at your workplace.

Your experience with a Julia solution developed and performing faster than Python even after hammering away at the Python implementation is almost comical in the context of how many times I’ve heard people have this experience, and how many people are still willing to die on their hill of other programming languages for what Julia can do better. Great talk! @Satvik do your more recent projects still find their footing in Python? Or do you now use Julia at work as the main language? Asking particularly because the pain points you illustrated in your talk were in the Python-Julia interfacing.

3 Likes
14

We use Julia for almost all numeric/data-oriented stuff now, and have moved a lot of the Python code to Julia. We still use Python for other parts of the codebase, like connecting to exchanges and doing the actual trading. At this point 99% of new code I write is Julia, and for the company as a whole it’s about 50-50. The performance differential has become even more ridiculous, an experiment that used to cost ~$125 to run now costs ~.33 cents, and most of those improvements wouldn’t really have been possible in Python.

Interop is still a bit painful, in particular because any Python code called through Julia is not thread-safe, and even forcing Python code to run on the main thread doesn’t solve all the issues.

19 Likes
15

The chicken and egg problem. I know it is really hard, and sympathize with your frustration.

Keep in mind that you are in a very good position though: your “competitor” is MATLAB. If you target this specific language in your comparisons, you will see that it is not that popular either. Show him/her the numbers, and explain the history behind Julia, how it fixes issues in the MATLAB experience, how it can add value to the business…

You would be in a much worse position if your competitor was Python.

3 Likes
16

Good resource to share with colleagues and technical people:

A similar resource for the industry would be great.

4 Likes
17

@kapple DE is Data Engineering!

1 Like
18

I often hear that someone’s enjoyment of working on projects well-suited for Julia has been ruined after returning to other languages.

There is an opinion that people don’t use Julia because they’ve tried it and don’t like it. I’m sure this is true sometimes. It’s a big world. Sometimes a team tries to use it and just finds it frustrating and is so happy to return to the warm comfort of Python. IDK, but I suspect those who have the opposite experience and find Julia liberating are the majority.

But this misses the big segment who have no experience with Julia at all. I suspect this lack of experience is a bigger factor than rejection from experience. I work on a scientific, library-like, product with end users of varying sophistication. So we must have a Python front-end. Using Julia as a backend would be too difficult even if I had buyin. My coworkers are extremely talented. Most are completely convinced that Python + compiled language (Rust in particular) is the best approach to a performant, flexible, rapidly developed, scientific/technical framework. They have no experience with Julia, but are convinced it couldn’t be better, in fact it must be far inferior. There are real disadvantages to Julia: TTFX, language server, hidden dynamic dispatch, etc. If they rejected Julia taking these into account in a cost-benefit analysis I’d be more understanding. The problem is there is no understanding of the benefits.

For our rapidly-developed, dynamic tool, we use Rust+Python, an ancient scripting language and an extremely strict AOT systems language. Rust may be great for writing the fastest, most polished ls and grep. But it is absolutely terrible for rapid development in a new technical field. (Developing in Python and rewriting in Rust is a terrible fit for a project like this; but I won’t elaborate here.) The amount of friction and boiler plate that this causes relative to Julia is absolutely staggering, mind-blowing, etc. And mind-numbing. But the organization has no understanding of this. I guess there is almost no understanding of this in the broader industry. I’d like to get this point across. Since I find occasional hostility to Julia, I like to say that the major point is really independent of Julia. Maybe the answer isn’t Julia. But there has to be something better than this antiquated model.

PS. Somehow I do enjoy a good bit of the Rust work even though, as I said, it is overall poorly suited to our task. I enjoy it for some of the usual reasons: tooling, package management, cohesive language design, performant, etc. And probably for something intangible, or something I can’t articulate

21 Likes
19

So awesome to hear. I want to bring a megaphone to JuliaCon just so I can put that message on repeat: language interop is the key to industry!

I really feel like further development to the Python-Julia interface is what can make Julia a top-10 language within the next decade. All you need is one Julia function in a legacy Python codebase to get our metaphorical foot in the door… then it will snowball once people see the speed up.

What we really need is for that first step to be n% easier. e.g., respect pyproject.toml · Issue #16 · JuliaPy/pyjuliapkg · GitHub would be a nice direction to think more about.

7 Likes
20

@svilupp gave a great talk last year about “Julia-fying” business teams

9 Likes