I find it hard to develop in Julia

[adienes]

I remain optimistic that the existence of Base.ispublic will slowly but surely improve the state of affairs. Prior to 1.11, it was really difficult (borderline impossible at times) to know if something was considered public or private.

[stevengj]

@adienes

Prior to 1.11, it was really difficult (borderline impossible at times) to know if something was considered public or private.

I think the general rule was that anything exported or documented in the manual was public, and everything else was private?

The public keyword gave us a programmatic way of detecting whether a non-exported symbol is public. Unfortunately, it also helped turn up some cases of undocumented public/exported symbols: Missing docstrings for public/exported symbols · Issue #52725 · JuliaLang/julia · GitHub

[langestefan]

@stevengj

Unfortunately, it also helped turn up some cases of undocumented public/exported symbols:

Why unfortunately? That’s good right?

[stevengj]

It’s good that we found them—this is indeed a benefit of the public keyword—but it is unfortunate that the omissions existed in the first place.

(This is a “help wanted” issue, by the way: it should be relatively easy to contribute missing docstrings.)

[affans]

But, at least anecdotally, it seems like many packages break after upgrading to a new Julia version. I am still on 1.11 since a bunch of my stuff broke on 1.12. Are packages really use that many undocumented functions? Or do the internals change significantly?

[adienes]

@affans

Are packages really use that many undocumented functions? Or do the internals change significantly?

yes to the former, many packages use internals they “shouldn’t” all the time. for the latter in the case of 1.12 there were also quite significant internal changes to accommodate all the binding partition (const / struct redefinition) stuff.

another example of packages relying on things they shouldn’t that will almost certainly break things on 1.13 is that most hash values will change. even though the docs specifically have warnings to call this out as a possibility, in practice hash values have been very stable for a long time and some packages have come to rely on them.

[JamesNZ]

For a longer time, my main hope has been that Tim Holy has stated improvements on Julia’s lowering mechanism could bring one or more order of magnitude improvements to the speed at which Julia can be interpreted.

Another possible solution is tiered compilation. No idea how much work it is (probably a lot…) but it would be really nice to have. Maybe it’d even be possible to have interpretation be the first tier if that’s faster than -O0 or whatever.

[jakobnissen]

Interesting discussion, @fonsp! I empathize with your pain from new Julia releases, but I can’t say I share it.

I maintain around 10 Julia packages, and most releases don’t require any work from my side to keep those packages working. Granted, the packages are smaller than Pluto, but still, my experience with Julia’s (in)stability is completely different than yours. In fact, I find developing in Python to be far more precarious at new releases!

Why the different experiences? I think this sentence of yours is the answer:

When I struggle with the developer experience of Julia, I am sometimes left feeling like this was somehow my fault. I used the API wrong (but there was no clear documentation). I used internal API (because there was no public API).

Internal APIs are going to change. By definition. If you build a package on top of internal APIs, you’re asking for breakage. I’m sorry to put it this bluntly, but if this is why Pluto.jl keeps breaking, then it’s entirely your fault.

On some level, the discussion could be closed at that, but there are some nuances that are worth discussing.

Are new Julia releases really worth it?

Maybe. The latency has gotten worse the last three of Julia releases (including 1.13).
Since latency is possibly the single worst thing about Julia, it’s reasonable to conclude that newer Julia releases are, on balance, worse, and therefore they are not worth it, even if they caused no code to break.

So, the latency regressions are bad, I think we all agree on that. But new Julia releases have given us lots of great features: Memory, Pkg apps, Project [sources] section, Manifest versioning, trimming, struct revision (even as that is still quite poor), OncePerProcess, public keyword, new atomic operations, and scoped values. And that’s just the last two releases!

Is it too hard to avoid Julia “internals”?

There is some argument to be had that it’s too easy to accidentally rely on Julia internals:

For example, there is no warning when accessing internal symbols, and it’s not clear when reading source code what is internal and what isn’t. Accessing fields of public structs is also typically considered internals, but there is no warning of that, either. Neither is the fact that iterator states is a private detail.

More generally, the behaviour of Julia functions is underspecified, which makes it too easy to accidentally rely on internal behaviour, and what is considered “breaking” in Julia is too often a matter of norms.

Hypothetically, would it be breaking if length(::Vector) began returning UInt? Obviously, that would be breaking. Right?

But why? That is, by what rule is this breaking? I don’t think it’s documented anywhere that it returns an Int. A UInt would still be semantically correct. And plenty of Julia releases has changed the output type of a function. For example, would it be breaking for map(abs, UInt(1):UInt(2)) to return UInt(1):UInt(2)? What if I make my own MyVector - would adding a specialized method map(f, ::MyVector)::MyVector be breaking?

And that’s before I get to functions are types that are so underdocumented that you literally can’t use them, because they have almost no documented behaviour, and therefore nearly all use of them is absuing internal.

If those kinds of breaking changes are the reasons that Pluto breaks on minor releases, then the breakage isn’t defensible. It’s just painful and I completely get your frustration.

If that’s the case, I very much encourage you to make a new topic and discuss the underspecified behaviour that caused breakage. I think you would do the community a service to state your troubles precisely - the responses in this thread makes it clear that we’re not quite clear whether Pluto breaks because you used internal functions, or relied on the observable but undocumented behaviour of a public, underdocumented function.

Did we make a mistake allowing people to use internals?

I’m tempted to draw the most misanthropic conclusion from the above: People can’t handle the ability to access internals, and will abuse it relentlessly before complaining when their code break, so we should never have allowed people to access internals in the first place. People clearly can’t be responsible on this.

The times I’ve been trolling through PkgEval and been looking at failures, I’ve certainly seen my share of authors using blatantly internal functions, and even more than once an author being dismayed or even surprise that the devs dared to change the metaphorical Core.__unsafe_internal_write!.

Where Python is explicit about its “consenting adults” principle, perhaps Julia should have been like Java or Rust - no public, no usage?

The counter-argument is that if Julia was like that, we wouldn’t have had great packages like LoopVectorization, Casette, Pluto, Mooncake, or Zygote, which only work by hooking into Julia internals.

Then again, the historical survival rate of these projects is abysmal. In effect, we don’t have projects like that anyway, because they keep dying (having used up community development resources in the meanwhile).

What could a better system look like?

Of course the following is way too late - Julia is not at that stage of development anymore etc etc, but at least I find it interesting to think how this internals abuse could be mitigated by language design. One could go the well-trodden route of disallowing access to internals - but is that really a good idea? In science, I sometimes want to pick apart someone else’s code and dig into internals to run some non-standard analysis. This is fine as long as I’m responsible to ensure the internal abuse is functioning correctly, and I’ve pinned the version of my dependency. I think it would be a shame to completely remove the ability to do this.

Perhaps we should take inspiration from how Rust handles unsafe code: It requires specific functions, as well as an unsafe_ block which can be grepped for, and even statically disallowed in projects.

Julia could have had a using internals Base: foo, bar statement. Without using internals, internal names would not be importable at all. Then, the General Registry could scan source code for using internals SomePackage, and refuse entry of a package into the registry unless SomePackage was set to a precise version in the Project.toml.

Similarly, getproperty could be unusable on foreign types unless explicitly implemented.

[kellertuer]

I have to second Jakobs points here. I develop a few packages – certainly far far away from the popularity of Plutos – but I did not yet run into the problem that a release broke one of my packages.

On the other hand I do see that sometimes it might be necessary to use functions / features in Julia that might be considered internal. Or one uses them “by accident” thinking they would not break. Then it is of course frustrating. So getting a clear distinction on that is probably good to have.

[MilesCranmer]

The internal API point is fair but I also find that for many things I want to do in Julia, there is just no public API alternative. If we measured of how many of the most popular Julia libraries relied on non-public methods, I’m guessing it would be a decent fraction. But it may very well be that Julia, by being a very flexible interactive language with its stdlib written in the same language, ends up leading people down a path of diving into the internals…

@giordano

However we eventually had a much better experience lately when, instead of fighting every single version of Pkg, we had a discussion with the Pkg developers to have a set of tests in Pkg representing the BinaryBuilder.jl’s Pkg workload, after getting help from them to clean up the use of the Pkg API inside BinaryBuilder.jl.

On this point, the same way that Julia now tests against Revise.jl to avoid breakages there, I wonder if there could be a “PkgEvalMini” for running PkgEval against a small subset of the most popular packages in the ecosystem. PkgEval is so large that it is only used sparingly, and isn’t necessarily a clean signal. A smaller, “most important package” integration suite might be useful

[j-fu]

Now that we have public and tooling like ExplicitImports.jl I think the way forward here is to identify uses of non-public internals and start a discussion between package developers and Julia core devs. IMHO there are several options for particular internal function:

• Can it just be made it public ? (this would include adding docstrings and unit tests)
• Is there a way to create a wrapper API as part of core Julia around internal functionality which is needed by e.g. Pluto ?
• Are there other ways for implementing certain features, without accessing internals ?

So that we slowly can move to develop a stable API to provide functionality which in the moment is only available by using internals.

[TimG]

@jakobnissen

if this is why Pluto.jl keeps breaking, then it’s entirely your fault.

I don’t think this is quite as black and white as you suggest. I am writing an excel backend for PrettyTables. To do so in a way that is functionally complete, I need to be able to write StyledStrings into Excel cells. This simply isn’t possible unless I access StyledStrings internal. So I have a simple choice. Either I don’t provide the functionality at all, or I need to set myself up for breaking changes in future which are “my fault”. This seems sub-optimal somehow.

[bvdmitri]

@jakobnissen

There is some argument to be had that it’s too easy to accidentally rely on Julia internals:

In my experience, it’s not just easy, but is also a standard practice even followed by the Julia itself. I’ve noted previously in this thread my pain with the tracking allocations in Julia so I will give a concrete example.

Let’s check the documentation of the @allocations, ok, good, fine, clear. It works, until it doesn’t. Apparently, such a simple thing can be underspecified and actually even not recommended (what??? how would I know that from the documentation?). What can semantically be more easier than telling me if my function is allocating or not, that it can actually change behavior…

Anyway, people suggest (somewhere in some discussion on GitHub that you’re supposed to know about of course) to look into @timed. Fine, let’s check the documentation for @timed.

help?> @timed
  @timed

  A macro to execute an expression, and return the value of the expression, elapsed time in seconds, total bytes allocated, garbage collection time, an object with
  various memory allocation counters, compilation time in seconds, and recompilation time in seconds. Any lock conflicts where a ReentrantLock had to wait are
  shown as a count.

  In some cases the system will look inside the @timed expression and compile some of the called code before execution of the top-level expression begins. When
  that happens, some compilation time will not be counted. To include this time you can run @timed @eval ....

  See also @time, @timev, @elapsed, @allocated, @allocations, and @lock_conflicts.

  julia> stats = @timed rand(10^6);
  
  julia> stats.time
  0.006634834
  
  julia> stats.bytes
  8000256
  
  julia> stats.gctime
  0.0055765
  
  julia> propertynames(stats.gcstats)
  (:allocd, :malloc, :realloc, :poolalloc, :bigalloc, :freecall, :total_time, :pause, :full_sweep)
  
  julia> stats.gcstats.total_time
  5576500
  
  julia> stats.compile_time
  0.0
  
  julia> stats.recompile_time
  0.0

Do you see something strange? Does the official documentation “hacks” into the stats.gcstats with propertynames and suggest me this is how to use it? What is gcstats.pause? What is gestates.allocd? Is it a short name for allocated ? Or it is an allocation Daemon? I can’t believe allocd is the amount of allocated memory when I do a simple test:

julia> e = @timed 1 + 1
(value = 2, time = 3.333e-6, bytes = 432, gctime = 0.0, gcstats = Base.GC_Diff(432, 0, 0, 8, 0, 0, 0, 0, 0), lock_conflicts = 0, compile_time = 0.0, recompile_time = 0.0)

Further. What is freecall? What is even total_time suggested by the documantation? Is it time spent during GC? Is it time from @timed? What is full_sweep? I can certainly know that by looking into the documentation of gc stats, right? Right?

help?> Base.GC_Diff
  │ Warning
  │
  │  The following bindings may be internal; they may change or be removed in future versions:
  │
  │    •  Base.GC_Diff

  No documentation found for private binding Base.GC_Diff.

  Summary
  ≡≡≡≡≡≡≡

  struct Base.GC_Diff

  Fields
  ≡≡≡≡≡≡

  allocd     :: Int64
  malloc     :: Int64
  realloc    :: Int64
  poolalloc  :: Int64
  bigalloc   :: Int64
  freecall   :: Int64
  total_time :: Int64
  pause      :: Int64
  full_sweep :: Int64

julia> 

Yeah, not very helpful. So in the doctoring of @timed I’m encouraged to hack around with propertynames and figuring out what actually is going on with this struct and how to use @timed. And since @allocations is not recommended and @timed is recommended this is the way I guess. And since Julia suggest to do that then people actually start doing that as their standard routine.

And this is just one simple example. It happens like pretty much all the time when Julia leaks its own internal implementation details and then the response is that we shouldn’t rely on it. And since this is happens all the time people just get used to that and it becomes “the norm”.

[lmiq]

For the allocations I currently changed to use Chairmarks.@b, which is fast and, I guess, reliable.

Can this part of the discussion be split into a different topic? I think it is a common issue in 1.12 but does not exactly has to do with the original discussion, as this breaks a test, not really the packages.

[bvdmitri]

I agree with you that its off-topic and the allocations issue better be discussed somewhere else. I just used it as an example to show that relying on internals is standard practice even within Julia official docs and the focus is on the documentation, not the allocations issue.

Your example with Chairmarks also relies on the same internals, which proves the point. So Chairmarks is going to be broken as soon as Base.GC_Diff will be changed or Base decides to remove Base.gc_alloc_count because both are private internals.

[mihalybaci]

@TimG

This simply isn’t possible unless I access StyledStrings internal. So I have a simple choice. Either I don’t provide the functionality at all, or I need to set myself up for breaking changes in future which are “my fault”.

A third option, as @giordano brought up for his package, would be to talk to the developers and open an issue with StyledStrings and request they make that particular internal code part of the public API. If you are you using it already, then you might even be the best person to write the PR in a way that will not break your code in the future.

[TimG]

The functions I need already exist in the case of StyledStrings and were suggested for my use by the package author. But they aren’t part of the public API at the moment - though that may, of course, change in future.

[Tamas_Papp]

@jakobnissen

Where Python is explicit about its “consenting adults” principle, perhaps Julia should have been like Java or Rust - no public, no usage?

I think this can be handled in a softer way by tooling, eg we already have the excellent ExplicitImports.jl, and once

Check unused explicit imports, e.g. using `ExplicitImports.jl`

is solved it will be integrated automatically into a lot of workflows that use Aqua already.

That said, I think that packages which use internals do it on purpose, not accidentally, and most packages which do it make this very explicit.

About the OP: I mostly experience breakages with packages that reach into the internals up to their imaginary elbows, mostly AD. But they are also mostly very good at communicating this, eg if you try Enzyme.jl on the (yet) unsupported versions, you get warnings.

I agree with those who point out that in the long run, the solution is to factor out those internals to public APIs. However, this is usually takes long since API design is difficult, and people want to get it right if it is to be included in the Julia API.

Experience shows that the best compromise may be a “shim” package that is not officially part of Julia, and contains branching codepaths for various versions, while presenting a consistent interface. This allows experimentation with the API (easier to release a 2.0 than to do that for Julia), and once it stabilizes, it can be included in Julia if necessary.

[juliohm]

@fonsp

In short, I find Julia an unstable and difficult platform to develop for. Precompilation is slow, developer tooling is lacking, and with every Julia release, I fear that my packages will stop working (they often do).

I would like to bring back the initial concerns raised by the OP. Most comments here are about breaking changes, but people forget how long pre-compilation and incomplete tooling are even more serious issues from the view point of a beginner.

The conversation around breaking changes and internal APIs is important, but IMHO it is very minor compared to the full programming experience with Julia. Students in other fields don’t even know what an API is. However, they feel the slowness when they hit Enter in a script or Pluto notebook. And when they get more experienced, they feel the pain of debugging something that is not working as expected.

[Tamas_Papp]

@juliohm

incomplete tooling

Can you, or the OP, please explain what specific tooling you are missing?

It would help focus the discussion.