Batteries included

General Usage
1

While I got stuck importing a package to save arrays to text files, I’m thinking that Julia doesn’t appear too “batteries included”, as are Python and PHP for example.

How come this difference in setting?

Is it because Julia focuses heavily on community, in the sense that groups of people take care of their packages, and so one takes advantage of the other?

But what is it, even to do basic things, you always have to import packages …
but after some time the packages do not enter the basic installation package?

I’m used to PHP, where I clearly have various ways to save arrays to files … of course PHP is a different thing, but I ask this to understand the philosophy behind Julia a little better …

1 Like
2

Julia definitely takes a different approach, trying to enable packages to be first-class and then putting lots of functionality in them.

I think this post has some good points about some of the problems of batteries included: Python Software Foundation News: Amber Brown: Batteries Included, But They're Leaking

Similar to that post, in my own experience, open source batteries-included projects can often mean the included version of the functionality isn’t great and you need to know the “good package” for doing something, but then the standard library decays more since developer time switches to those packages. (I think it’s different for commercial products like Matlab and Mathematica since there’s lots of employees making everything work together with one goal).

But Julia does include some key batteries some other systems don’t, like a package manager. I think that’s a really strong choice that helps enable packages.

4 Likes
3

I’m not sure I understand in what sense python is “batteries” included, that Julia isn’t. My experience is that I can hardly do anything in python without multiple import statements, moreso than in Julia. Previously, even a lot more was included in Julia Base, though much has been moved out.

Do you really use less import statements in Python?

8 Likes
4

In what ways is Python “batteries included”? Without numpy you can hardly do any computational stuff as far as I can tell. Are you maybe talking about Anaconda Python?

In general as Eric says the Julia philosophy is to provide users with the best possible environment to become developers and implement the functionality they want. This means that the pool of contributors is maximized and you increase the likelihood of domain experts implementing cutting-edge stuff, which wouldn’t be feasible if things were “batteries included” baked into the standard library.

I do agree though that it can feel a bit weird coming from e.g. Matlab or even Python were Anaconda has most things and the libraries tend to be larger so import numpy as np, pandas as pd is often sufficient. Doesn’t take long to get over that thoough in my experience :slight_smile:

8 Likes
5

Haha, I see we are thinking along similar lines. I agree with the sentiment, but I think for many “data wrangly” workflows in Python you can get by with numpy, pandas and matplotlib, while in Julia I often have using CSV, XLSX, DataFrames, LinearAlgebra, Statistics, StatsFuns, [Stats]Plots (e.g. because norm and mean and countmap happen to be in different maths/stats related packages and the data reading capabilities are separated from the DataFrames library).

Again, I don’t think this is a problem at all - I don’t think there’s a reason why DataFrames and CSV should be more closely integrated, and the development speed for both packages is probably (unprovably) higher than in a world where they would be shoehorned into a big package.

6

I see. For this sort of thing, though, you could have meta-packages, either home-made, or more official ones when common practices have emerged.

For example: https://github.com/JuliaStats/StatsKit.jl

As for python packages, I find that I often need lots of subimport statements, import scipy as sc, import scipy.optimize, import scipy.integrate, etc. Functionality is often buried under several layers of subpackages.

4 Likes
7

I don’t make decisions here so take what I say with a grain of salt, but R and Python are the languages I’m familiar with that do this and it really makes maintaining and updating a code base messy (e.g., R has four native class systems). I smaller core language is easier to maintain and optimize. This makes it so even if you have a standard library it can be maintained in separate packages that are updated asynchronously. That means no waiting around for a new patch release of Julia when you need a bug fix in one package.

1 Like
8

Relevant: Julia and batteries from @Tamas_Papp

7 Likes
9

Julia is indeed not batteries included.

Its like TeX. Noone uses TeX without a few dozen packages.
(including LaTeX).

One things that allows that (for both Julia and Tex) is that it comes with a really good package manager.
(I wonder if TeX was one of the first languages to come with a package manager?).

Standard libraries are like second class packages.
They can’t be updated without reinstalling all of julia.
They can’t have breaking changes without julia being considered to have a breaking change

Contibutions to them are slow because they need to go through similar process of careful review, and there are a limitted number of people that can have commit rights to the language itself (for good reason).

Its been observed in some batteries included languages that the standard library is where code goes to die.
Because it is so hard to update anything in there that it kind of bit-rots and becomes out of date,
and nothing is ever removed.
(Probably exception to this is commercial batteries included languages like C# and Mathematica which have large staff mainining every bit of it.)

6 Likes
10

But is then Python ‘batteries included’?

11

You can use Serialization.serialize to save/read data to disk. This is included in the standard library. Have a look at the documentation for the limitations of that method.

12

I am curious what you consider “basic”. Really basic stuff (file I/O, the array interfaces, and standard univariate math functions, just to name a few things) is already in Base.

No, quite the opposite. Historically, things moved out of Base into the standard libraries and other packages. This is a great thing, as they get independent versioning and develop more freely.

It is more productive to ask a concrete question, eg “how do I save this particular array to a file?”, describing your requirements (who needs to read it later, maybe users in some other language or tool?), and possibly providing an MWE that generates the array.

You may not realize this, but arrays in Julia encompass an infinitely rich variety of types, from FillArrays.Zero to SparseArrays. To give a meaningful answer about how to externalize them, we would need to know your constraints.

What you call arrays in PHP aren’t even the same thing, they would be some kind of AbstractDict in Julia. So a package like JSON3.jl, maybe?

You can get a lot of help from this great community, but investing some effort into a concrete, well-defined question usually has the highest returns.

1 Like
13

I think Python is “batteries included” in the sense that it has a very large standard library.

14

I fail to see the big difference here between languages. People are mentioning MATLAB as batteries-included. One time I installed MATLAB without any of the add-on packages (by accident). Literally 75% of my programs didn’t work because they all relied on one of the add-ons. At least in Julia I can add it in a few seconds!

7 Likes
15

There was some discussion about creating a Batteries.jl package a while back, right around the release of 1.0. I’ll give it a shot when I have a free weekend.

1 Like
16

I feel like for this to work, you would really want to use PackageCompiler as otherwise, this import would take a few minutes probably.

1 Like
17

To clarify for people, in PHP an array an “ordered map”, so don’t think you can use Dict in Julia (only default in Julia). There is however OrderedDict in a package. The abstract types, as in here, are for the concrete sub-types, when that applies, and many users can do a lot without thinking about those.

18

Hello,
thank you all for the comments, which I read with real interest.

I’m glad this community is very friendly,
even when they ask questions that maybe who knows how many others have already asked … and then it is clear that all the beginners’ questions are almost always about the usual things … and therefore it can be boring to repeat oneself.

In short, thank you all.

Since I personally found myself well with Python, without having to add too many things - but this is because the type of work I do, mainly ERP and database management, and lately industrial automation (without the need for who knows what mathematics, to be clear) - then I was immediately struck by the difference in setting between Julia / Python regarding the fact that it seems to me that you always have to download packages to have this or that other functionality …

However I understand, there is a broad reasoning behind it, on the usefulness of having libraries curated independently and by people with specific skills, with the advantages of having them perform better than the model of updates used in Python.
I’m not at a level to understand too much, yet, but it suits me, and it seems very reasonable. And as far as I can see, in fact the packages I installed (when I don’t crash … grrr … importing them …) work well.

I have a distant memory of Matlab, even if I used it thoroughly for my thesis and for some exams even earlier, now many years ago, so I mainly compare with how I found myself (I am here, because I currently use it for the automation with PLCs and other scriptint activities) with Python.

Ok, as soon as I can post something more precise, because I understand that it is better time to time, to write the specific problem, to see what solutions there are.

I noticed one thing, it is better to write here in this forum rather than to search the net, because there are sometimes (like the LDL package, which had a successor LDLD2) references to things that may be out of date, or perhaps related to old versions of Julia.

Thank you all, see you soon.

5 Likes
19

I have opened another topic to explain. :slight_smile:

20

Yep. Good plan.

We right now have a lot of that kind of information stored in humans.
Which TBH I think has a lot of advantages, in terms of being able to apply context and experience in giving answers.
(Only downside is scalability. But even then we gain people with experience to answer at at sufficiently close to the rate we gain people who need answers. I guess this eventually fails with sufficiently exponential growth)

Every time you ask a question on Discourse, it then taking information out if humans and puts it on the web so it becomes searchable for others later.

So great, do keep asking questions.

5 Likes