I think we all recognize that getting a 5x improvement is plausible for rewriting a large system without changing the language – algorithmic improvements, redesign, whatever.
Since the language change per se isn’t going to help performance much, I find it annoying that performance improvement stories are promoted so heavily in Julia’s community. I think this bugs other people too, like in Keno’s comment here. Performance isn’t where Julia’s advantage is (compared to other fast languages).
People like reporting numbers in case studies, but I would rather they report numbers that are more reflective of the changes that switching to Julia actually brings – rather than those it doesn’t bring (but which would occur during a redesign without changing language, namely performance).
In fact, I would rather not mention performance at all, rather than reporting silly claims like “faster than Fortran” that (1) irritate other communities, (2) reduce trust of the Julia community, and (3) confuse our own users.
@Mason mentions that
I’m not so pessimistic about the quantifiablity of these properties. For example, “we reduced the size of our codebase by 2/3 because most of it is in published packages” or “a new team member takes half as much time to be productive” or “changing from Float32 to Float64 took one line in Julia vs n lines in C++”. Those are real advantages of Julia. We should be able to quantify and report them objectively.
I would like to see more restraint in reporting performance results, and more enthusiasm in reporting design and development-experience results.