How reproducibility through Manifest is related to LTS/nonLTS? Manifests are reproducible only within a single Julia version anyway.
1 Like
I mentioned it because LTS is useful for people who care about long-term code maintenance, but in many cases it may not be needed. Of course youāre right, a Manifest.toml wonāt save you if thereās a bug in your non-LTS Julia version.
In any case, I would wager that for industrial applications, the package ecosystem is a much more brittle aspect than the actual language itself. In other words, I would trust a non-LTS Julia version way more than I trust any individual package.
Iām definitely on the side of delaying Julia 2.0 as long as possible and havenāt really seen many issues worthy of issuing a breaking release. However, in light of the new threading PSA, is updating/deprecating the threading foot-guns a strong reason to push for 2.0? Or can any relevant problems be solved by non-breaking releases (or just encouraging better code)?
1 Like
I believe we can address those issues with non breaking changes, by basically offering solutions that work better than whatās available right now.
3 Likes
The issue there is only that @threads, nthreads() and threadid() seem part of the same package of multi-threading features, such that it is too tempting to use threadid() to index buffers in a @threads-ed loop. That can be solved simply by creating a new macro, lets say @parallel, or whatever, suggest proper patterns for its use in the documentation, and let @threads and companion be slowly forgotten in some low-level development part of the docs. The name @threads is bad, anyway.
8 Likes
Bringing Julia to the next level of popularity requires to provide an IDE with a lot less issuesā¦
I suggest to set up a work group to discuss (e.g. on Zoom) how this could be achieved.
12 Likes
If a IDE like spyter also including Pluto and Jupyter, that will be wonderful.
You can use Jupyter within VSCode already. As for Pluto, it may take some time
As someone with no CS background and not very familiar with Julia, I find it difficult to understand the highlights of the new Julia releases. They are full of jargon and focus more on the mechanics of Julia than the syntax of the language itself.
I think a second blog post with less jargon, more accessible to a wider audience, and focused on how new versions improve daily workflows as you have done could very helpful.
Is a good language server and better IDE support currently one of the main pain points in Julia? Would it help to have a mechanism similar to what Rustās language server has to raise money?
7 Likes
I think the syntax has largely settled down. Recent releases of Julia mainly improves other things like the compiler, though occasionally some light syntax sugars are added.
3 Likes
Sounds very interesting!
Raising money is one thing, but then we would also need to find a developer and supervise him or herā¦
From the industry point of view:
- Compiler-wise: Static Compialtion, cross compilation (apparently being a priority for the compiler team, so thatās great news)
- Package-wise: Better ONNX support and other file formats.
10 Likes
What do you mean with other file formats?
On your two points you think are lacking for industry adoption:
- Compiler-wise: Static Compilation, cross compilation [ā¦]
- Package-wise: Better ONNX support and other file formats.
Note, you can already compile to static binaries for Linux, Windows and macOS, with PackageCompiler.jl, and such binary executables are already used in productions by companies. [People already cross-compile, at least for Raspberry Pi? Iām not sure how well itās supported.] If you mean static tiny executables, then there are more experimental packages which achieve that for Linux, but they only support a rather limited subset of Julia.
[āPackage-wiseā, e.g. ONNX, is in the hands of the ecosystem, thereās nothing Julia can do about it, or rather IMHO should do about it. Do you disagree?]
Regarding compilation, you can only choose two of these three: generic, fast (in all cases) and tiny binaries (and that trilemma applies not just to Julia but also to all languages as powerful, e.g. Lisp). You can have precompiled fast code (already, e.g. packages in 1.9), but using code, e.g. from the REPL, in arbitrary ways, i.e. generic code, isnāt possible without a compiler, if you want the code to be fast. Why you need the heavy dependency LLVM, or equivalent, meaning a (backend of a) compiler. Julia is more powerful than e.g. C, C++ and Rust, in this way and getting rid of the compiler, at runtime, limits its power. You could still have precompiled fast packages, as you do already since 1.9, but some code would need to be slow, i.e. interpreted (already possible with Julia, people may not be aware of that). You could still support all code (and have e.g. the GC), itās just that the generic part of Julia, or any language, rules out precompiling, since itās for infinitely many types, some (actually vast majority) not yet invented.
I actually had a thought about this (and other things) that kept me from sleeping until 7 am. I have a plan for Julia 2.0, 3.0, and maybe even 4.0ā¦ and will discuss it separately later. I think we want to go to minimally breaking Julia 2.0, but my idea for 3.0 is āmaximally breakingā, to see where that leads (e.g. no heavy dependencies, such as LLVM and excise ALL of the standard library, or as much as I can, though you could add it back through packages) and it could lead to tiny executables without many limitationsā¦ only sometimes slower (similar limitations as Python with its packages written in fast C), as explained above.
It seems decided that 1.10, not 1.9, will be the next LTS. My idea is that all code that would work in 3.0 will work in 2.0, and all code that works in 2.0 will work in 1.x, and code should be tested one latest Julia (3.0) and 1.10 LTS. And hopefully all 1.x code will be supportable in 2.0 with trivial updates, hopefully just by adding Julia1.jl to the environment.
Releasing 2.0 now would only be negative, the actual userbase wants Julia do keep compatibility so (i) their codes will not break for basic reasons (like a Base method changing name)
Basic reasons like changing a name (in Base, or anywhere really, in packages too), in effect means drop the old name, and add the new name. It can be done in a trivially non-breaking way by just adding the new name as a synonym, e.g. people ask for that to make the language more consistent. Some 2.0 proposals are that basic, and the downside is then there are the new and old way, some mental clutter, I suppose the docs should just deemphasize the old way (rather that drop it from the docs). Dropping the old name could be deferred indefinitely, and arguable should just not be done according to Rick Hickey. Some proposed 2.0 are less trivial, i.e. with semantic changes. E.g. I propose Integer ^ Integer should result in Float64 (thus type-stable working for all numbers), i.e. not resulting in an Integer, since thatās in general not possible (for negative powers) and is a footgun of Julia (likely will overflow).
I donāt a have a good answer for 1.10, or 1.9, but note the answer is the same. Each 1.x release is non-breaking, i.e. you donāt need to wait for 1.10 to promote. Nor be that worried whatās exactly in the latest release. Itās always more of the same goodness, plus e.g. much improved speed (of startup, the so-called time-to-first-X, TTFT; used to be called time-to-first-plot, TTFP, the more specific limitation).
Itās understandable people are skeptical of 2.0, because according to semver, it means breaking changes. It the past some ātechnically breaking changesā have been allowed, but anything more visible is deferred to 2.0. I have some in mind that are minimally breaking, an improvement still, more likely to result in correct results, as with my power (^) example above, so in by some definition not breaking at all.
parquet, arrow. there is support but I still regularly run into issues
In my opinion path to popularity lies through academia.
I strongly disagree. In the USA, only about 2% of the population aged 25 and older has a doctorate degree. However, I remember as a new Julia user coming to the community back in 2018 (mainly here on Discourse) being a bit intimidated as I am not an academic and I donāt have a doctorate degree. I got the impression that the community largely consisted of scientists/academics and the like, and I felt like I didnāt fit in (especially considering that the community I was most involved in before was the JavaScript community). I suspect Iām not the only non-academic to dip a toe into the Julia pool and feel that same vibe.
I think the community should be thinking about ways to do the exact opposite - expand the user base outside of academia and ensure the community feels like a place where anyone (regardless of job or education level) can fit in, learn, and achieve great things with this wonderful language.
I took a quick look at this 2021 Python Developers Survey and there are a few points that I find interesting:
- A substantial proportion (29%) of respondents are using Python just for personal/educational/side projects
- Most users are using Python for data analysis and web development
- Two-thirds of respondents said they do not consider themselves a Data Scientist
While this is certainly speculation on my part, I am willing to bet that the vast majority of Python users are outside of academia (if anyone can find data on that, please post).
28 Likes
Iām not sure I agree with this. Academia is good, but itās not sustainable. Academics usually end up in the real world and face a lot of institutional pressure from whoever employs them next to use more ātraditionalā tools like R, Python, or Matlab.
One of the big problems I observe with Julia right now is that most of the regular users are academics. These people have lots of free time and the ability to dictate their own tools, but they are also small islands ā they donāt manage that many people, and canāt enforce Julia use. Their long-term effect (I think) is probably fairly small.
Something we could be better at is encouraging commercial use. Commercial use is a massive boon for programming languages. People use Python because it works, it covers all the bases, and employers want to pay people who can write Python. Julia doesnāt have the same thing! We use Julia because we love the language and think its a better tool, but few of us are under the impression that someone would pay us for our Julia skills.
Iām not sure how to encourage this. I know Julia Computing has been doing some excellent work to onboard people, but thereās still not many places where I can say āABC Corp? Oh yeah, I heard they were a Julia shop!ā
The more commercial adoption we can push for the better. I have no idea how, but I think thatās a good goal to consider.
An additional goal is more education and outreach. At some point the Julia evangelists got quiet and stopped pushing. I include myself as someone who did this. I want to see more good-quality blog posts, more neat projects, more talks, more meetups, etc. Things are not as unified as they once were and I think we could do a little more in representing a vibrant community to the world.
20 Likes
Also ā the Julia Slack has to be retired or for core devs only or something. The fact that weāre having so many discussions in a transient, deleted, and gated network means all our community knowledge building gets thrown in the trash. Itās a really big problem and I think we need to be more thoughtful about where we devote energy to communication.
49 Likes
The more commercial adoption we can push for the better. I have no idea how, but I think thatās a good goal to consider.
Yeah I fully agree. Corporate funding is probably the single most impactful thing that could happen in this direction. Iāve heard pushback on this front before but I refuse to believe that it could be a net negative. (Not that my or their opinions would matter to a company if it decided to pursue this.) Mindshare is really powerful and it would force a lot of devs to actually learn the language with direct application to their business, instead of being like āitās a cool language but I donāt really know how I can use it, or share my Julia code with others.ā
Take ML for example. S4TF failed because Swift had practically no ML community, or any scientific community for that matter. Juliaās in a completely different position, where such efforts, even if abandoned as cavalierly as S4TF, would not be gone to waste. In fact, given the way that the language has been marketed, youād expect Julia to be especially suited for ML.
On the other hand, thereās an acute shortage of devs in this space while the rest of us are āwaitingā for Diffraction and Enzyme to mature. Meanwhile a lot of the grad students doing a lot of the hard work graduate and move on, or the work is only sustained through grant funding. Maintenance becomes a lot harder. Itās unsustainable.
At some point the Julia evangelists got quiet and stopped pushing. I include myself as someone who did this.
I stopped pushing because I realized I detest being marketed a product I wasnāt looking for. āLooks like Python, runs like Cā was something that we pushed for a long time and I honestly believe it did some real damage to how the language (and community) is perceived. Devs everywhere are well-aware of the problems with two languages and if thereās something out there that claims to solve it, theyāre going to have some pretty high expectations. Repeatedly comparing with Python and C++ just fueled the antagonism, especially when they figured out that you couldnāt compile to small binaries for anything beyond toy problems. Marketing should be humble and presented in such a way as to not set people up for disappointment.
11 Likes
In anticipation of JuliaCon, would it be possible for each of the big Julia Github orgs to write a blog post reflecting on the last 12 months of development? With a bit of luck, between JuliaCon and the blog posts, July could become Julia month.
2 Likes