I would not worry about this. A variation of this thread seems to happen every month or so.
9 Likes
I agree, but this is hard to automate. You can keep an eye on #new-packages-feed on Slack (or Zulip) and leave a blocking comment on the registration PR for new packages that lack documentation.
1 Like
If I might make a few final constructive comments before this gets closed, I do think we need a “clearer” ecosystem for users. As I’ve come to understand Julia, it tries to have a minimal standard library with only some basic packages likely to have fairly broad applicability. For example, the LinearAlgebra package is part of the stdlib, and contains basic operations such as dot & cross product, eigenvalues, direct solve via \, etc.
So me, as an end-user, I don’t feel the need to go find a NumPy-like package to provide those functionalities. However, LinearAlgebra doesn’t include “advanced” capabilities like Krylov solvers, polynomial fitting, etc., so for those I need to go find a package. And those packages are typically “single-focused” on doing one thing, and that one thing well. It can be a bit of a maze, then, to find a set of packages that provide all the functionality of something like Matlab’s base install, or of SciPy or NumPy.
It would be nice, IMO, if there was a curated list on Julia’s homepage of “user-recommended” packages for different domains. Something similar to what NumPy’s webpage has:
We’ve got a good start on the official Julia webpage with:
But it doesn’t quite help me if I’m a new user wanting to rewrite a Matlab project that does finite-difference method on a DNS fluids problem. I’d propose a “Recommended Ecosystems” page that plainly lists various ecosystems and packages. I know there are sites like Julia Packages, JuliaHub, and Julia.jl, but it would be nice to have a curated list directly on the Julia homepage, that simplifies things for new users.
The second thing that would be nice would be if it were possible to simply install an entire organization of packages. For instance, I’d love to be able to simply do Pkg.add_organization( "JuliaMath" ) and not have to manually add each package. JuliaMath is, in many ways, that “user-curated” ecosystem of packages I’m looking for as a “novice” user. If I want to use the erfc function (that is part of both Matlab and Python standard libraries) I don’t want to go searching for the package, but rather would be nice if JuliaMath were a simple-to-install “catch-all” for standard mathematics capabilities. Then we could tell users “if your project uses advanced mathematics, add the JuliaMath library”.
The last thing I’ll mention is that it might be nice, perhaps, to consider expanding the standard library. Why do I need to add the SpecialFunctions.jl package to have access to gamma, erf, Bessel functions? While I think I understand this is because Julia doesn’t want to “play favorites”, it doesn’t seem like there are any other projects that compete in this area. Plus, it then seems odd that LinearAlgebra is included in the standard library, when that seems like a package that shouldn’t based on my own observations of what are/aren’t included in the standard library. This kind of harkens back to the first point, I find it odd that LinearAlgebra has something as “obscure” to a layperson as a generalized-eigenvalue solver function, but not a conjugate gradient or GMRES solver. It’s odd that while neither Python or C++ include linear algebra in their standard libraries, that Julia does; and while both Python and C++ include “special functions” in their standard libraries, Julia doesn’t. It just all seems very arbitrary (to me).
Anyways, hopefully those were constructive (rather than grievance-airing) and I’m happy to participate in future discussions on those topics (in separate threads?) and contribute to source where I can.
13 Likes
I think this is another very big problem to getting new users to adopt Julia–the difficulty in recognizing the importance of defaults and informational constraints. Among experienced Julia devs, the assumption is defaults don’t matter, so we don’t have to change them.
However, just because a change is easy for experienced users doesn’t mean it’s easy for new users, who often don’t even know a feature exists. What percentage of Julia users do you think know about julia --project? My guess is less than 5%. (I didn’t know about it until today, and I’ve written a couple of packages!)
Other examples of poor defaults that are easy to miss when you’re learning Julia:
- FMA (not default because it could confuse devs if someone ran a computation on a pre-2008 computer)
- The inability to use positional arguments with keywords (because then, developers wouldn’t be able to change the names of positional arguments without breaking user code)
- Defaulting to single-threading instead of threads=n_cores (held up by complaints this would be a bad default for large-scale computing)
- Extremely long stacktraces filled with nested type declarations (implementation held up by complaints that it could result in users not posting the full stacktrace in Github issues)
- Performance footguns like type instabilities and eager functional primitives (maybe 80% of the mistakes I find in PRs by newer users)
- No compile-time errors when static analysis catches likely mistakes, like method ambiguities.
In theory, Julia is supposed to be just as easy to pick up as Python, like it promises in its advertising. In practice, all these details are just too easy to miss. Just because it’s easy after you learn how to use it, doesn’t mean it’s easy to learn how to use it. Right now, these kinds of defaults mean it takes me a lot longer to teach a new user to learn Julia than it does to teach them Python.
4 Likes
coincidentally enough, I disagree with your preference about the right choice of default as well as with your characterization of the opposition to that choice for each and every item on that list…besides maybe 4.
Yes, but the original question in this thread regarded steps to take for 2.0, and I think that structured documentation is a step the community could take which would allow for automated checks. And, that might imply a breaking change and packages get let behind at 1.x. R has had these checks with cran for decades, and I think its a major strength.
It looks like good work being done in other areas of improving package quality Aqua.jl
1 Like
For instance, I’d love to be able to simply do
Pkg.add_organization( "JuliaMath" )
Organizations could already now publish a wrapper package that reexports all (or at least the best supported) packages in the org. You probably want to exclude experimental ones.
1 Like
I’m not surprised! All of these defaults are examples I chose because they appeal to the kinds of people who are extremely used to writing Julia, have been using it for a long time (possibly pre-1.0) and post on the Discourse and Slack. I picked them because they’re examples where the opinions of a developer and a user would strongly diverge. I’ve noticed that Python developers have a consistent philosophy of prioritizing user preferences, while Julia developers have a philosophy that prioritizes their own needs and those of heavy, experienced Julia users.
100% agreed on all of those, though that’s not a surprise since I might be the one who has opened each of those issues in Base 
, or at least brings them up in gripes when they aren’t fixed.
The philosophy should be: whatever is best for the inexperienced user for the 90% use case
HPC users have to figure something extra out to get a bit more performance? Whatever. That’s the kind of user that comes from C++/Fortran that knows how to read intense API docs. Why should an undergrad get 1 threaded runs and not know why Julia is slow in order to preference the PhD in computer science doing the 1000 core HPC runs not having to add -t 1 to their build script? Why give a giant stacktrace just in case a dev might want the full thing in the future? Just ask for more when you need it (and honestly, what’s chopped off is essentially never useful… at least the {...} new stuff).
We look so much at Jax because it’s faster, when in reality PyTorch has almost all of the mindshare because it’s easier to use.
14 Likes
That’s not the reason. In a language for people that care about numerics, you aren’t allowed to arbitrarily re-associate floating-point operations, because then you are computing something different from what the programmer specified. Even clang & gcc only do this with -ffast-math.
See e.g. X * y + z does not automatically use FMA instruction - #2 by GunnarFarneback
That’s not the main reason. The basic issue with using keywords for positional arguments is that it makes multiple dispatch ambiguous, unless you disallow re-ordering (and even then it is a problem if different methods use different argument names). See e.g. Allow use of named-argument syntax for positional arguments?
Anyway, this is not an example of a “poor default”, but a language choice that you disagree with. Fair enough, everyone has their own tastes, but people who have different taste than you are not being hostile by saying so.
Making type instabilities impossible would require an entirely different sort of language, not a dynamically typed language. (There are lots of great static languages, and liking them is fine! But that’s not what Julia is.)
…Anyway, this kind of unfocused thread, into which every sort of complaint is dumped, is not particularly productive. I’m doubtful I should even be responding, because it just goes around in circles with the subject continually shifting to a new grievance.
17 Likes
I really dislike seeing this thread continue like this.
This is not because of the original post, the content, or the replies. Most of the content is potentially useful and productive. What concerns me is that much of this will be difficult to reference or find in the future. It is exactly because of the potential that I worry about what a waste all the effort in this thread may be. I hope I am wrong.
Please consider starting a new thread. Your time, thoughts, and opinions are worth it.
14 Likes
I do think I’ll create separate threads from my earlier post, but let me make a final-final comment. When we (Coreform) needed to choose a language to serve as our user-subroutine interface for our (in-development) isogeometric analysis solver (in the field of FEM), we chose Julia (although our codebase is otherwise C++).
The “granddaddy codes” typically use Fortran/C for their subroutines, which are used to define custom material models, element formulations, boundary conditions, etc. So users (typically younger users who’ve learned Matlab or Python) have the challenge of learning Fortran/C and manually compiling their code prior to execution – not always trivial to make sure you’re using the right compiler, and you need to compile for each operating system you want to run on (common to “develop” with GUI tools on Windows, simulate full-scale problem on Linux). And of course then there’s the question of how do you debug your code - from stepping through a debugger, to plotting the function, to benchmarking / profiling your code. Even six years ago, when we were just beginning to spec out our code, it was clear that Julia was the best language for our user-subroutines: the best blend of ease-of-use and performance.
Just a simple example, from our “booth” video that we share at conferences we attend (go to 1:40 if the timestamp in the link doesn’t work):
I hope you’ll excuse my self-promotion, but I thought it would be an uplifting note to mention its use.
EDIT: And what finally convinced me of Julia being the perfect fit was the response from this community on a question I asked back in 2018. Even though I’d been dabbling in Julia for three years at that point, that was the first time I finally saw the full potential of Julia. So what’s one step the Julia community should take, you ask? Answer: Keep on answering user’s Discourse questions, knocking them out of the park with the power and simplicity of Julia. Any user can do this, I know that I personally am quite hesitant to contribute to source, but I can certainly try to answer other user’s questions.
22 Likes
What concerns me is that much of this will be difficult to reference or find in the future. It is exactly because of the potential that I worry about what a waste all the effort in this thread may be.
Perhaps, not every thread needs to be “efficient, single issue, useful” thread. This is not a commercial concern anyways.
May be there is a reason this is some kind of an attractor thread (despite snark threats from @mbauman 
)
For me, anything that increases interaction and lets people speak their mind is good.
I personally stopped trying to read this thread about 500 posts ago.
3 Likes
I suggest collecting all the ideas, and eventually spawning threads based on that.
Similar to how @ChrisRackauckas did it with the Revise issue.
Actually this has already been proposed in this thread, If I remember correctly. As is so often, “somebody” should implement it. I understand a person who proposed some Base code change may feel unqualified for a PR. But everyone is qualified to make such a list
1 Like
I’m focusing on a small thing and am hesitant to take part in a broader way in this sort of open ended discussion. But in my opinion, this makes perfect sense. Julia provides a fairly complete set of dense matrix algorithms in LinearAlgebra, providing solutions to most problems that can be solved using LAPACK. Dense linear algebra is an area in which algorithms can be treated as black boxes that will work well without much guidance from the user, at least on problems that are of a manageable size. There are more choices when dealing with iterative methods and they are in a totally different class in terms of what is required of the user to use them effectively. I think that separating that complexity from the much simpler to use and more problem independent dense matrix algorithms is a good call.
Also, I’m not sure who counts as a layperson, but I would argue that generalized eigenvalue problems aren’t really all that rare if you are in the right field. Having access to the dense matrix algorithms for that is certainly helpful to people in some fields in engineering and physics when they have problems that are not too large.
1 Like
Yes, let’s bring this thread to a close. Where it’s been most useful, I think, is where we’ve been able to split out into focused topics. Thanks everyone for your participation!
Folks are always welcome to share what they think would make their life better, but dedicated topics are always going to be the most productive. If you want to continue the discussion, the best way forward would be to create new topics focusing on a single issue at a time — or better yet search to see if your issue has been discussed before and use an existing topic!
38 Likes