Not only for technical computing: changing the narrative around the usecase for Julia

I’ve also just realized that my .julia folder is quite large and it seems like most of it is due to Conda (sometimes Pkg keeps more than exactly one version of the package so Conda alone can take a few GBs of space). Would it be possible to somehow not have it as a dependency of IJulia?

2 Likes

Alternatively you can install Jupyter globally. You will still need a Jupyter installation, though.

1 Like

When you install IJulia on Linux now, it asks you if you want to use conda to install Jupyter.

1 Like

Surely the biggest problem for julia’s wider uptake right now is not that it doesn’t come bundled with Linux distributions

1 Like

Also, there is a snap for it, which is available wherever snaps are sold (almost everywhere, these days) not sure how up-to-date it is.

Anyway, I don’t think availability is the problem at all, if we’re just talking about getting software developers interested. It might be a challenge for inexperienced programmers, but nobody who does this stuff for a living is going to have trouble getting started with Julia.

In developer communities, Julia is known as a sort of faster alternative to Numpy/R/Matlab and people don’t know it’s a general-purpose language. I’ve had the conversation several times in real life and many times online. For most people (including many in this community), Julia is synonymous with numeric computing. For me, it’s such a waste that people don’t realize how practical Julia is for so many different types of work.

The other main thing that keeps people away, at least from my conversations, is that a lot of Java-first developers seem to have trouble understanding how to use dispatch on abstract types for code reuse rather than inheritance. (mumble mumble Java developers mumble mumble) edit: I do not mean to imply that Julia should try to accommodate them. What we have is better, and if inheritance is added to the language, people are going to use it, and we don’t want that.

5 Likes

Of course not, which is why I didn’t say that it was. :slight_smile: It’s still important even if not “biggest”.

Remember that this entire thread is taking place in the context of changing the narrative that Julia is only for technical computing. Personally, there have been several times when I wanted to write a script to send to someone else, and my first instinct was to reach for Julia, but unlike python, perl, or bash, I cannot expect Julia to be readily available on the other person’s computer, so I ended up just writing it in bash instead. None of this is a problem within the domain of technical computing, but outside of it the ubiquity of being able to just send a simple script or a statically linked portable binary becomes important; distribution becomes important. Packaging a python script as a standalone program is if anything more difficult than doing the same for Julia, but it doesn’t matter because python is available in every distribution (and usually part of the base system). That’s kind of the crux of the argument I’m making.

2 Likes

Because of limited manpower and the rather high barriers to entry (learning the packaging workflow for a Linux distribution is quite a time investment), and in some cases, release cycles, a lot of software lags behind by 6+ months on many distros.

That said, Ubuntu and Debian (testing) both have a 1.* version now, and for Windows and OS X, install binaries are just one click away. I don’t think it is reasonable to expect more for Julia — it is unlikely that it would be installed by default on systems.

1 Like

We need an IJulia written in Julia. Shouldn’t be too hard :slight_smile:

I’m both a big fan of Python’s subprocess module and a vocal critic. It’s a bear to learn, but once you know it, you can do pretty much anything and, as you mention, it abstracts all the difficult parts of IO away.

I have no particular objection to a Popen.communicate-like interface being added to process.jl. This is more of a sidetrack as someone who has both written process handling interfaces and called a lot of processes in scripts:

What’s the actual usecase for writing to stdin and reading from the stdout of the same command or pipeline inside of a Julia script? It’s not like you’re actually going to be using grep or sed as a data filter inside of Julia. I suppose this might be useful for wrapping a tool like pandoc or other more complex text transformation tool, but I honestly don’t see much of a case for capturing the output of something like that either. (The output is eventually going to go to stdout or into a file anyway, which programs like that can handle directly, unless you’re going to do additional transformations on the output, which is already a very specialized usecase.)

As a process library implementer, this is fun problem to try and solve, but as a scripter, I run processes for two main reasons:

  1. As a side-effect: The process just runs and does something on the system. I might pipe something to the process, but I’ll rarely do anything with the output.
  2. To get information that cannot be easily obtained in another way (Julia’s ccall interface makes most of this unnecessary, but sometimes it’s still easier to parse command output than to bother learning the C API)

Using a process as a filter inside of a Julia/Python/Ruby/Perl script is fairly rare and sort of inelegant, since these languages are perfectly capable of doing their own text filtering.

And, as a side note to this side note, the error Popen.communicate makes is that it deals with strings rather streams. stdout and stderr should be returned as IO objects in Julia, against the event that their content may be too large to fit into RAM. The user can then decide if they want to call read or eachline or whatever.

Thanks for volunteering! let us know as soon as you get it working!

4 Likes

IJulia is written in Julia and dropped the Python dependency recently. It’s Jupyter that needs Python and that’s another story.

6 Likes

For what it’s worth I think you’re entirely right about the reasons that one can’t just send a Julia script to someone and expect that it will just work for them. However, I’m not sure that it makes sense for Julia to aspire to be where Perl 5 or Python are in terms of that. Even if that might happen eventually it will take too long. We need other ways to make it possible to send Julia code to people. Being able to compile a small stand-alone program like you can in Go would be great. Being able to send a Julia script with a manifest file including info about the required Julia version and have it “just work” would be great too.

16 Likes

Can’t agree more! I can compile the same code flawlessly to Windows or Linux to one binary. Please don’t be like Rust where it’s difficult to even find out how to compile a standalone binary!

1 Like

My InteractiveCodeSearch.jl is one of the examples that needs that. Another example I’ve seen in the wild is the SVG renderer for VegaLite.jl.

I agree this for grep and sed but there are non-trivial APIs provided as CLIs like I just mentioned. Piping is a good “RPC” for simple things. It’s nice to be good at it.

3 Likes

The communication between the VS Code julia extension and the LanguageServer.jl, and then the communication between the LanguageServer.jl and the SymbolServer.jl is all based on that model. The architecture there is quite different from something like say the VegaLite.jl case that @tkf mentioned already: in the latter case we start a subprocess, it processes something, ends, and we process the return case. The VS Code example, on the other hand, is more a long running RPC type scenario, where multiple processes stay around for a long time and exchange messages via STDIN and STDOUT.

4 Likes

Any volunteers? :sweat_smile:

4 Likes

So I just freed up around 3-4 GB in my .julia directory by deleting some .cache and .trash directories which were hidden and now it is only about 1.1 GB (most of that is for Nemo).

If you were a user of previous Julia editions, such as 0.4, 0.5, or 0.6, then you might have some hidden folders laying around with some gigabytes of cached data.

2 Likes

How about a command line argument that starts Julia in a project environment with the equivalent of a REQUIRE file, and a shebang syntax that does the same thing?

For example, if the first line of my script is:

#!julia --require "julia 1.1; Plots 0.23"

then Julia would make sure that I’m running v1.1 or later, and it would ask me to Pkg.add("Plots") if I hadn’t done so.

(Including an entire manifest is technically better, but would be much more cumbersome. Even for simple scripts it would not comfortably fit on a command line.)

2 Likes

Coming back to your original post (and github pointer(s)). I’m very interested in trying out Julia to hunt down the various many bash/csh/zsh/{yourinitials}sh scripts lying around (rather lengthy processes, so don’t mind the warmup overhead). Many of my colleagues have recently made a similar switch to python, so it’s a nice opportunity for (soft) benchmarking (including UX).
Any chance to see some (albeit minimal) progress on the tutorial you started? This would be very highly appreciated :slight_smile:

2 Likes

I’m working on the last chapter now, which is about string manipulation. I suppose I can start working on the scripts to build out the file and post it in its own thread here for feedback.

Honestly, there isn’t that much to add to the official docs. I just put a few examples made some links to a few things in the manual. It’s not like Python where all the utilities you need for admin/workflow scripting are spread out in a bunch of crazy modules. Basically everything you need is already in Base.

3 Likes