Nightmare in a simple software development process and its implications for Julia

Offtopic
1

Here’s a simple program development process:
Take a program A.
Let’s call the new version A’.
If the A’ is tested to be better than A, merge the patch.

This simple program development process seems simple, unbiased, and safe right?

Except, even this simple process is flawed.

Imagine if A has a big assumption x.
Now, since A always merges any test that improves, it would likely be very well-optimized.
This means that each code section is very well-tuned and changing it would very easily make it worse.
Now, imagine A has many code sections that make use of the assumption in the design.
Now, what would removing assumption x imply? It would imply a major change to many parts of the code.
However, the law of A being well-optimized means that each code change probably makes A worse.
This means changing several parts at once is pretty much impossible.
This means that A is probably stuck with assumption x.

I thought that this simple development process had no flaws, but it had such a big and glaring flaw.

It seems nothing is easy in this world.

But here comes the part where it might be relevant to Julia. You might think that it might only apply to some software, maybe some package of yours. Except, even Julia (as a language itself) might not be safe.
Julia is extremely optimized for some metrics like performance, ease of use, etc. Considering that Julia is a really greedy language, it is also probably optimized in some hardcore ways.

If there are assumptions in the making of Julia… those assumptions might be staying here for a really long time.

In an ironic twist, “We are greedy” might not just mean wanting more, but also means myopic optimization as in greedy algorithms.

Are we in for a reckoning? I don’t know.

2

I’m still a junior so I might say something wrong and in that case please correct me. But I think there is no escape from that and it should be taken as a sort of law of nature. Everything must have some kind of assumption, you can try to make them as broad as possible, but you can ever escape that, and sooner or later that assumption will not allow the software to move forward.

I think the main solution for this is diversity, this means choosing the right tools for your project, and if you feel like they do not exist, to try and create them, but always with the knowledge that you are going to still make some assumptions, just different ones.

2 Likes
3

Yeah… that’s a good idea if you can. In practice, though, imagine if you had a machine-learning problem and no tool suits you, creating a new machine learning framework for your project would be a monumental task. That being said, if you’re that good, I’d encourage it.

Julia is good in the sense that it allows you to create whatever tool you want more easily than if you had to do it in other languages. It is not the fault of Julia that it is still hard, but the tasks themselves are often monumental.

For me personally, Julia is fine. Just be aware that your packages might be used in some ways you didn’t imagine. Imagine you wrote a package. In a typical language, you might see people use it and say it’s fine. In Julia, somebody might try to auto-diff through it and say there’s an issue and you could be like “What on earth is a derivative?”.

1 Like