Should BenchmarkTools.jl be a stdlib?

[Tero_Frondelius]

@thofma

While there won’t be magically more maintainers, people are forced to keep it working. For some people this is therefore a good argument for putting stuff in std or at least keeping it there.

I don’t understand your logic. If people are committing their free time to do something if they don’t need or like something they either move it out or in extreme cases fork their own branch to develop without the extra burden they don’t care about.

You cannot magically get people freely to do what they don’t want to do. You always have two options to get the features you care about. Either do it yourself or pay someone to do it for you. Of course you can always wait that someone else is interested enough to implement it.

If it wasn’t clear I try to say that you cannot force anyone to do anything.

[thofma]

Of course you are right with what you say. I totally agree. You cannot force anyone to do anything. But I don’t think that this is relevant here.

If I make a PR for julia (the language), which adds an amazing feature or improves the language significantly, but which breaks something in stdlib, it won’t get merged. I have to fix the package in stdlib.

[Tero_Frondelius]

@thofma

which breaks something in stdlib, it won’t get merged. I have to fix the package in stdlib.

That’s not how Semver works either you have to fix all packages (impossible because they can be proprietary) or just wait next major release. My point is that there is no difference if the code is included in stdlib or it’s own package.

[thofma]

I did not mention Semver, which indeed is not related.

Since the package is part of a julia release via the stdlib, there must be a version working with the release.

[Tamas_Papp]

@thofma

If I make a PR for julia (the language), which adds an amazing feature or improves the language significantly, but which breaks something in stdlib, it won’t get merged. I have to fix the package in stdlib.

Which also raises the barrier to entry for PRs, resulting in fewer of them. There is no free lunch here, just trade-offs.

Moving a lot of code out of Base was not done to annoy users, it was somewhat of a necessity.

[Liso]

@Tamas_Papp

Moving a lot of code out of Base was not done to annoy users, it was somewhat of a necessity.

Do you have serious information that removing it from Base is just start for removing it from stdlib too?

[Tamas_Papp]

@Liso

Do you have serious information that removing it from Base is just start for removing it from stdlib too?

I don’t see how this question follows from what you quoted, and what “it” refers to in your sentence.

[Liso]

@Tamas_Papp

I don’t see how this question follows from what you quoted, and what “it” refers to in your sentence.

It is quite easy to explain. Topic is about stdlib. People are saying about stdlib. And you are still saying about Base.

Maybe (but it is quite unbelievable) you don’t understand that Base!==stdlib.

Or maybe answer to his question (if you have more info) could explain why you don’t treat stdlib outside Base seriously.

[Tamas_Papp]

@Liso

you don’t understand that Base!==stdlib

At this point, I think I will abandon this discussion. Have a nice day.

[tk3369]

In my mind, stdlib is more “blessed” and more rigorously maintained than third party packages. If I write an app the depends on stdlib then I have a warm feeling that it works correctly and it will be there for a very long time. If I depend on a 3rd party package then I am running a higher risk of using an eventually abandoned package.

So I think my position is to move BenchmarkTools to stdlib.

[Tamas_Papp]

@tk3369

stdlib is more “blessed” and more rigorously maintained than third party packages

I think you might be mistaken about this. There are bugs (not feature requests or enchancements, but actual bugs) in stdlib libraries that are open for months, and when fixed most normal users (who don’t compile from master or PRs) only get these with the next stable release.

At the same there are package maintainers who are religious about fixing bugs, and close everything critical within a few days, or at most weeks. Do a pkg> up, and you have the fixed version.

[Liso]

@tk3369

In my mind, stdlib is more “blessed” and more rigorously maintained than third party packages. If I write an app the depends on stdlib then I have a warm feeling that it works correctly and it will be there for a very long time. If I depend on a 3rd party package then I am running a higher risk of using an eventually abandoned package.

This is easily understandable.

@tk3369

So I think my position is to move BenchmarkTools to stdlib.

But it is also about human power. I would be glad to see big stdlib but I really respect (and quite understand although not every time agree) what people maintaining Julia decided to prefer with their limited resources.

[carstenbauer]

Actually, the reason for me to start this thread and ask the question in the title was mainly about “should BenchmarkTools ship with Julia” and not so much about maintenance.

To make this clear: Conceptually, I’d be perfectly fine with making all current stdlibs external packages, i.e. move them to separate git repos and version them properly, that is make them less special. However, we’d still want most of them to ship with Julia - certainly Pkg as otherwise you couldn’t install any packages I guess. Hence “stdlibs” would just be a list of selected external packages that ship with julia (also, base julia might rely on them, like Random).

I just wanted to raise the question if BenchmarkTools should also be on the list of those selected packages.

[Liso]

@Tamas_Papp

There are bugs (not feature requests or enchancements, but actual bugs) in stdlib libraries that are open for months, and when fixed most normal users (who don’t compile from master or PRs) only get these with the next stable release.

In bigger projects it is acceptable to do small workarounds to not be bitten by bugs. It is also acceptable to wait month to have fixed version before you upgrade (you have always carefully prepare to upgrade). It is acceptable do “abnormal” things like compile master. What is unacceptable is need to rewrite big part of code because one pillar crashed. Even worse is to redesign big part of solution just because you have to switch from abandoned library. Somebody could say that solution is to fork this library. But if this is not planned from begining it is additional cost which everybody like to avoid. And sometimes it is impossible due to resources.

@Tamas_Papp

At the same there are package maintainers who are religious about fixing bugs, and close everything critical within a few days, or at most weeks. Do a pkg> up , and you have the fixed version.

Look at Qt.jl and its dependency on Cxx.jl. Could you expand your idea about good maintainers here? Do you think that it was mistake to count on Cxx.jl’s maintainer?

(BTW talking about religious reminded me Godwin’s law )

[Liso]

@carstenbauer

… ship with Julia” and not so much about maintenance.

If you don’t want to ship untested things you couldn’t avoid (at least some) maintenance.

[Jean_Michel]

The problem I see is not where sits the code for BenchmarkTools or Documenter. The problem is where is the documentation for them. Basically everybody only learns about BenchmarkTools the first time they post a timing example here and are given the answer “you should use BenchmarkTools”.

It seems clear to me that the documentation of BenchmarkTools should be in the manual, so people can learn about quicker. I understand the burden on the developers that imposes, since it means that they have to bless somehow a particular API. They should conservatively “bless” a few packages, starting from obvious ones like BenchmarkTools. But really I think the diffusion of Julia is hampered by the current situation, and things are only going to be worse if we get an influx of new, less experienced, users.

[Tamas_Papp]

@Jean_Michel

everybody only learns about BenchmarkTools the first time they post a timing example here and are given the answer "you should use BenchmarkTools "

The flow of user questions which can be answered by a single link to the manual suggests that perhaps not all users read it in full (even the core part, without standard libs and devdocs). Making the manual bigger would only make this worse.

As Julia matures, I expect that there will be books and other resources that talk about these things, and recommend a set of commonly used packages.

[pkofod]

@Tamas_Papp

As Julia matures, I expect that there will be books and other resources that talk about these things, and recommend a set of commonly used packages.

You can also search the web for “how to time things in julia”, there are many answers there that point to BenchmarkTools.

[carstenbauer]

FWIW, there is a blue box in the manual that recommends BenchmarkTools. However, I agree that it isn’t very prominent.

In my opinion, the Julia documentation has so far served two purposes at the same time:

1. being a look-up manual, i.e. providing technical description of things (see for example here)

2. and being a starting point for learning the language, i.e. having some coherent readable block of text that makes you familiar with the concepts (tutorial like, see for example here)

And pre 0.7 it did, in my opinion and maybe suprinsingly, serve those purposes relatively well (it really was a nice read!).

However, the big refactor of the code, in the course of which things were moved to stdlibs and some (former base) functionality has even been moved to external packages, happend and, in my eyes, the documentation was somewhat left behind and didn’t evolve with the same pace. In particular the parts serving (2) above.
Fundamental things, like answers to questions like “How do I install a package in Julia?”, which you would certainly expect in a document of type (2) above, are now hidden somewhere in the stdlib “Pkg” documentation (you have to scroll quite a bit to find it in the navigation - and that is if you actually know what to look for).
It also doesn’t quite serve point (1) anymore - although much more than point (2). Try to find fft in the new documentation (it was easily found pre 0.7). You won’t succeed, because it’s now in AbstractFFTs.jl + FFTW.jl, external (meta) packages.

In my opinion, we really need to be aware of the (somewhat orthogonal) purposes listed above and should, with increasing complexity of the language and it’s ecosystem, cleary decide what the official julia documentation should be good for. To what extent should we try to keep it as a good introduction? What is it worth as a look-up manual if I can’t even find fft and others? What about stdlibs? Eventually, when stdlibs are separate versioned packages themselves, will the search in the julia doc only be good for finding things in base or will it cover those stdlibs as well (like now)? If so, why not also extend it to other packages that are considerd to be of fundamental importance?

Point (2) can of course be also covered by external materials like books, as @Tamas_Papp’s mentioned. No one keeps you from writing a very nice introduction. And maybe that’s even better because it can be made context specific (a physicist and a computer linguist might consider different functionality important). Also, one can start improving the getting started section of the Julia docs.

In any case, I believe it’s crucial to have the julia docs serve a clear purpose.

(Of course, I still use it as (1) and am glad that it’s there )

[Zach_Christensen]

I’d hope that Julia follows the path of C (and to a lesser extent C++) where the “batteries included” model in languages like Python is exchanged for high performance/usability/maintainability for solving many problems. I don’t know what the litmus test would be for this is. Maybe if BenchmarkTools is likely to be essential in the vast majority of packages. It does seem to fit the central focus of Julia as a high performance language though.