Is Revise.jl an essential part of what Julia is? I think a lot of people here would think so, but one common complaint is that is lives in a personal repo (@tim.holy ). I see this on Hacker News and other things a lot: if it was really part of the compiler and something everyone should use, it should be in the JuliaLang or shipped with Julia. I think when it was developed the separation made sense, but does it still? I mean I don’t think a move would change the actual construction of it, even if it was in JuliaLang/julia it would still mostly be commits of Tim Holy, but I do think there is something to be said about perception and documentation.
Both Revise and PackageCompiler are learned to be essential parts of using Julia, but they have very little if any representation in the documentation. Are they something everyone should have as a core competency, or an add-on? I think it makes sense to build it as an add-on, but not necessary to present these features as an add-on.
I don’t necessarily know what my proposal is here, but I wanted to open the conversation on a group of related topics:
Should Revise move to be like Pkg and PackageCompiler as part of the JuliaLang organization?
Is there a way to better integrate the Pkg, PackageCompiler, and Revise docs into the Julia docs? I understand that it can be nice that they are developed in a way that is separately versioned, but honestly the full Pkg docs are harder to find than they should be for new users. Shouldn’t there just be a section in the docs “The Package Manager” right below “Standard Library”? Why is there so much indirection here? And then if system image building is so essential to the docs, why not include PackageCompiler and Revise as well?
Doesn’t the debugger deserve a mention as well? I mean sure as an experienced user I know Debugger.jl is the one to use, so go to the package and … but other languages document the “standard” debugger as part of the language.
Basically, I think we should really be reconsidering some of the choices that are made for development and what their influence is on the documentation. Just because Pkg is developed in a separate library from Julia itself does not mean it is ergonomic for new users to have to find a new website in order to find the full documentation of Pkg. I have seen this fail in practice for new users, and I think we power users have a bias to believe that since we know how to navigate this it’s easy, but it’s clearly demonstrated as not when for example I see someone try to find out where extensions are documented or package compatibility instructions.
So let’s have a discussion, what’s the right way to improve the fact that our documentation does not actually cover all of the things we call “standard Julia”?
In MATLAB it’s very clear how to find the things like examples, functions, and apps, and I wonder if we should be using similar space to display how to get to the package manager documentation website and the debugger.
I think with Revise, the argument used to be that since it was constantly changing, putting it into Julia itself would slow its development, but I think that argument no longer holds. Revise seems pretty stable and it seems the only real changes happening to it these days are just adapting to changes in julia. Shipping Revise.jl as a core part of julia seems like a relatively straightforward thing to do.
Regarding PackageCompiler.jl, @jeff.bezanson mentioned on Slack recently that both PackageCompiler.jl and StaticCompiler.jl’s functionality should eventually be subsumed by julia itself. For PackageCompiler.jl, that would likely be somewhat straightforward, but doing the same with StaticCompiler.jl would require a lot of work from some very in-demand compiler experts.
Revise should change one more time, but it’s a pretty drastic change (and simplification). Virtually all of the remaining bugs in Revise come from the fact that we have to quasi-interpret module toplevel code when we track revisions, so that we know which expressions generate which methods (this is important not just for Revise’s core mission of handling code changes, but it’s also essential support relied on by our debuggers: you don’t want incorrect line numbers in files as you step through code simply because you inserted a couple of lines of comments higher up in the file). It would be better if we just kept track of that at the time of method definition. Once JuliaSyntax is part of Julia, perhaps this change will be easier.
But does that require Revise to have a separate website? I understand the discussions of shipping with Julia etc., but I think it’s conflating two separate issues.
Revise currently has a separate documentation website in a repo that’s distinct from the rest of core Julia. Unless someone is “in the know”, there is no indication that Revise is something that say 99% of Julia users should know about. It only shows up as a small section in one part of one chapter of the documentation, just a single reference in the whole thing buried here: Workflow Tips · The Julia Language.
Revise should change, but does that mean that this is the right way to present the Revise of today? That’s the question I’m asking. Things will keep getting better, I don’t see that as justification for not making Revise easily accessible or visible from docs.julialang.org and from the result of clicking on the JuliaLang organization. People use it today because of word of mouth and JuliaCon videos. I think we should instead make it blatantly obvious from our websites and documentation.
Interestingly though, PackageCompiler and Pkg are not mentioned. And this is disconnected from the documentation so someone searching the docs will not easily find this stuff.
I’m not saying I’m representative or anything but just to maybe counter what you might have heard on HN, I’ve been using Julia professionally almost every day for the last 10 years and in that time have used Revise maybe for about a month total and PackageCompiler not once.
Of course I get that they’re dev tools and HN is all about the dev experience etc but as someone who uses Julia to do “data science”, simulations, optimisation and general applied stuff these things aren’t all that relevant to me.
Same here, daily Julia enjoyer for years and I rarely use Revise, I just played with PackageCompiler a couple of time. That said there’s other core packages that I don’t use (Sockets, Dates, …), and code (re)loading, compilation, debugging feel like core language features, so it would make sense to have them included when they are mature enough (seems to be the case for Revise).
+1 for Pkg being more visible, it’s a bit lost in middle of NetworkOptions, CRC32c, etc.
I think this concern would be more relevant if the suggestion was to add it into Base or a stdlib, but I think Chris is talking about the “Julia project” more broadly when he writes “part of Julia”, and is talking about the documentation/website/etc (and maybe the JuliaLang github org).
To me it’s a pretty clear that the lack of discoverability on the Pkg documentation is a con we should be thinking about, it does not take very many user discussions to see people run into the search issues I described above. But then the website (https://julialang.org/) feels the Debugger and Revise are also core parts of the language that should at least show up in a discussion to new users, while the documentation does not include a full discussion and instead expects one to read until the 34th section of the 1st chapter (Workflow Tips · The Julia Language) to get a quick note about the existence of Revise and a link to a different documentation.
That is of course not some pre-ordained immovable object. The Pkg documentation is separate from the Julia documentation by choice. It could be moved into the Julia documentation. I am asking if people have thought about this deeply and have consciously had a discussion about the pros and cons of such a decision and have done this because they believe this is actually the ergonomically best way to present this information.
Is there anything which is not in the Base distribution which should be co-located in terms of documented?
If the answer to (1) is yes, what should be in that list?
I could not find such a discussion, so I opened it to the community to ask if people think the status quo is the right way to be documenting such additions.
This has nothing to do with moving any code into JuliaLang/julia or any of the discussions around whether that is good or bad for something that’s in rapid development or not. This is simply how we present ourselves and our tools to the outside world for documentation and downloads. We have conflated the two, but we don’t have to.
Maybe consider loading Revise in VSCode by default as this ? Central documentation would be nice, but great integration inside the default development environment without user ever needing to refer to docs would be better.
However, in the JuliaHub presentation the docs lose the framing of JuliaHub. There is no link to return to the JuliaHub website or search for another package. While Julia packages usually host their own documentation, packages via Rust cargo are usually available from a central documentation server. For example, pola.rs documetation is available below.
I think the reason historical: the current implementation of Pkg was developed around 2018 when Julia 1.0 was being finalized, and it was kept in a separate repo until it replaced the previous implementation in Base.
I think that being in a separate repository is fine (and I hope that all stdlib packages end up in separate repos, one day), but I do agree that its complete docs should live in the manual.
Now about Revise.jl: is an amazing tool that makes my life much easier, but it is still and incomplete solution due to limitations in Julia. It should be advertised more, but at the same time user expectations should be managed.
As for PackageCompiler: I have to admit that I never had to use it. I always tried to debug TTFX issues and kept startup times low for my environments. Improvements in 1.9 mean that I have even less incentive to use it.
I don’t know for sure, but I think it’s most likely that it will never happen. Replacing your Main with a new module (which Julia now supports) may be the better way to go.
I suppose we could move Revise to JuliaLang, if there’s a general consensus that it’s better there. But where do you draw the line? To me, Cthulhu and JET seem to be in the same camp as Revise. My suspicion is that JET in particular will come to occupy a much more central role once we get full static compilation. In the long term, I think PackageCompiler will go away, but JET will only grow in importance.
I believe widely used packages should always be in organisations where other trusted users have full access to. The risk of the package becoming unmaintained should the only developer disappear for whatever reason is too high, and this isn’t a purely theoretical concern.
Access is pretty much orthogonal to picking up maintenance of orphaned packages.
Most of Julia packages have an MIT or similar license, so if one is unmaintained and you want to maintain it, you just fork, rename, generate a different UUID, and you are good to go. If you want to be nice you explain why you did this in the README and refer back to the original package. For some reason this is not though — we should try to establish a culture that encourages forks.
Conversely, providing access does not make a package maintained or even maintainable. What matters is code clarity and organization, unit tests, documentation of the API and internals, basically good software practices. Then the package can be picked up for maintenance easily; lacking these it sometimes makes more sense to abandon it and make a new one.
IMO the ideal structure for a lot of Julia smallish packages wouldn’t be an organization, but a team of 2–3 core maintainers who are familiar with the code. It is good practice to invite people who make a few quality PRs and seem to care about a project. The downside of an organization is that all members get access, and to manage that sanely requires much more monitoring and trust.
