Thanks, I liked your explanation a lot. The point on expressiveness is quite weak though. You can write unreadable/unmaintainable one-liners in any language. This is more about the style which your organisation sticks to. If you talk more about a lack of tooling for enforcing style, then I can agree.
About the duck typing leading to unmaintainable code: Dropbox, at one point, enforced strict typing in their Python codebase to alleviate this problem.
The thing I am missing most about Julia for doing serious work is the ability to quickly make changes to the API of modules, functions or types, and be confident that my changes are correct.
Honestly, my Julia projects feel like a glass-house, where changes from a few days ago may randomly explode in my face because there’s a code-path I didn’t hit since then.
Seriously guys, I have annotated the type of every argument and return value on every function. The compiler should be able to tell that this argument no longer has that field I renamed. I get the feeling that language is wasting all of my time on mundane things I shouldn’t have to worry about.
I have annotated the type of every argument and return value on every function
This rarely tells the compiler much it can’t work out anyway. And Julia doesn’t use types for making sure code is correct at compile-time… It’s a dynamic language.
Quickly making API changes and knowing you haven’t broken anything requires tests. Lots of tests.
I’ll suggest one take away from the comments on typing in this thread is that there’s lots of people who feel they’d benefit from better whole program analysis of type errors in large Julia codebases.
I’d argue that another related issue is that the community’s tendency to encourage writing generic code is in conflict with ensuring that code changes have the fewest non-local effects possible. I suspect the community’s recent work to address invalidations, which is a compiler focused effort, could be viewed in terms of the costs to users who have to reason about non-local effects when methods are added.
I think the point is that if the information is there, for code written in a “static way”, then the compiler could do this analysis.
You could have a best-of-both world approach where code that can be statically analyzed is, and dynamic code isn’t.
I can imagine why that’s hard to do though, if for example “statically written” programs are generic.
A separate static analysis package is probably a more realistic goal, but that would definitely be useful
One of my main issues with catching these minor mistakes by tests/running the code is that the algorithm I am implementing has a seriously exponential running time, and there are several cases in the branch-and-bound algorithm that I am not going to hit with the smallest test cases.
This becomes a problem because the kinds of refactorings that introduce this kind of mistake will often introduce it in many places at once, and running a test case that hits that part of the code is going to take at least a minute or two, and since it stops at the first mistake, I have to wait that minute quite a few times to catch all of them.
This hits the nail on the head. I’d be interested in hearing what any of the core devs maybe @jeff.bezanson think ? Are addressing these fundamentally at odds with Julia’s core philosophy?
That does sound annoying. It may help to break up your problem into smaller functions and write fast unit tests - make sure your lower level methods work, and build more complicated methods using them as building blocks.
What i do not like in Julia is the way how it was marketed (or positioned among other languages). To do “serious” work you need to find a team and this is extremely difficult. Leaving aside followers of the Church of Static Compiler and those who is too lazy or preoccupied to learn new language, I usually hear two arguments
In my experience, both arguments are false. I’ve used Julia for web scraping, zulip bots, stock market trading, even wrote small game and it shined every time. There is nothing immature about Julia ecosystem, I was able to solve every problem I’ve encountered without much problem. And yet, nonetheless is all the same every time "we are only doing XXX, why would we want language for scientific calculations? "
So, if the narrative wouldn’t change, this ship will never set sails.
Well, it’s not just marketed towards scientific/technical computing is it? Variable identifiers that can be math symbols, extensive support for overloading operator symbols that many other languages do not support, multiple dispatch as central paradigm (from the manual: “a good fit for mathematical programming”), etc. Its roots in scientific computing and aim for high performance are fairly obvious. And many features will have been chosen/designed over the years to primarily give good support for such use. Possibly at the expense of more easily supporting general programming tasks. The time-to-first-plot issue is one example where you probably would not use Julia for quick-running scripts aimed to be called from the command-line (Bash, Perl, Python make more sense for those).
But it would indeed be good to have many more examples of Julia being applied to the kind of use cases you mention. They can better highlight strengths/weaknesses relative other languages. My personal search for a high-level language that could also offer performance involved trying out Go, Swift, (only looking at) Rust, before diving into Julia. If there had been dozens of general applications written in Julia then that search might have been shorter. The examples out there, and issues discussed on this forum, are very much geared towards science. Many of the discussions here go quite deep into eg specific algorithms, ML problems or data plotting. Unsurprisingly as that kind of use seems to represent the vast majority of Julia usage. But I’d love to be shown otherwise 
Minor inconvenience for me. I spend wasy more time getting to the point where i need to plot. First plot warmup is not noticeable for me.
Compiles != tested. I’ve learned that personally using Scala/Spark.
True, but with a dynamic language like Julia or Python you spend part of your testing effort on checking things a compiler for a statically-typed languages gives you for free. Unless your default testing strategy is to go for 100% code coverage you can still run into errors like incorrect field access, the wrong method being called due to a missed type, etc.
True, and 100% test coverage of the “normal” paths is pretty standard. But 100% test coverage of the error paths? Like what happens if the disk fills up, what happens if the disk is read only, what happens if bit X of the data file is bad? I often put checks in my code for things that may fail but 99.99% of the time won’t. I don’t always check that those failure paths work correctly, usually because of the contortions I have to go through to get them triggered.
Maybe if you could put the standard library into “test” mode and have read or write failures (and corruption), dropped network connections, etc. Then it would be much much easier to build automated tests for these situations. Then there are the situations where somehow the data got into a bad state i.e. database tables that were mucked with outside of the application. The list goes on of error states that could happen but probably won’t.
Why do you think that programing in Julia is more prone to this kind of problem than in other languages? (Sincere question, I don’t see why a change in a function can break more unpredictably a larger code than in other languages).
I am completely with you here. But do you think Julia doesn’t allow one to write pedestrian code or only motivates one to not do it because it offers the possibility of clever one-liners?
My four person audio research team used Julia as our main language for about 18 months. We had lots of reusable internal modules and tens of thousands of lines of Julia code. Eventually we had to give it a way and move to Python, where the team are now feeling much more productive. We started using Julia because it had the promise that it we could do deep learning using CUDA.jl but also be able to write audio processing code that runs in realtime.
The key issues for us were:
I know this question keeps getting asked over and over again in this thread, but were you using Revise during development?
I completely agree with your post about interfaces; I think the Julia development community should really think about how this could be improved: Julia would be a much better language if the situation was improved. The Holy traits being a kludge, the syntax is not nice, but for me the biggest limitation is that you cannot add a trait to a type if you don’t own it. This prevents using traits in many useful situations, like adding one to a Base type. Some packages have been trying to build an interface implementation around traits as they are but the result is not nice. We should really think how to add properly interfaces to Julia and do this for Julia 2.0 or even before if possible…
I’m pretty sure it’s based on how much memory you have so that it doesn’t keep doubling only to exceed available ram and start swapping.