Discussion: Plans for Julia as a general-purpose language?

General Usage
, ,
1

Hi, first of all let me say that I know this is a big subject, but I couldn’t find any serious discussion about it, so I’m asking here.

I think it’s obvious that Julia has a really good philosophical basis, and I believe it has a lot of potential to grow and compete with all the other big languages.

However, I believe that presently, it’s still not ready to be used for most programming projects. But the “good news” is that perhaps it won’t be so hard to fix, if we make it a priority. I would say there are 3 main roadblocks that I’m aware of:

  1. Performance, or: No integration of JIT and interpreter

Right now, every function that runs even once, gets compiled. This is causing really long start-up time, and a really long time-to-plot, which could be easily avoided by compiling only functions surpassing some call count.

Julia already has an interpreter, so perhaps this won’t be so difficult? I’m sure many would be happy to have that as an option.

I should note that the idea has been mentioned here, but I couldn’t find a discussion about it: A Julia interpreter and debugger

Another thing that can really help performance, is caching the compiled functions, and luckily there’s already a mechanism to serialize and unserialize them (iirc tim already opened an issue about it).

  1. Imports

Julia’s import system is probably its weakest point, in terms of user-facing design. Having to literally include files, and then import them separately, is awkward and inefficient…

Every modern language (and even most non-modern ones) lets you include your module using a single command, and makes sure to remove duplication for you.

The funny thing is that you can import a local module, if you just add the local directory to Julia’s path (which creates its own stack of problems). But perhaps it means that it would be very easy to implement?

edit: Just to clarify, I’m talking about importing modules within the same project (local modules), for example a utility module, or a module for common structs.

  1. Executable size

When compiling a trivial program on windows, the build ends up taking about 500 megs, mostly in complementary dlls and such. Python does the same trick of including itself, but needs only 30 megs. Perhaps there’s a simple solution to drastically reduce the build size?

This one is perhaps not so immediate, but all-in-all wouldn’t it be nice if you could send someone a julia program over email, like you can with any other big language?

I have a lot of other tidbits that I would like to improve in Julia, but I’ll save it for another time. I’m writing all this not to complain, but to help point out what I think should be on high-priority if we want Julia to make its steps towards mainstream and professional use.

I wish I could do all of this myself, but this isn’t one man’s work, even if I was familiar with the code-base and had the time to see it through. But if we decide to make this a priority, we can all carry a little bit and get it done together. So, at this point in time, I’m just trying to figure out if enough people agree with me (or perhaps if I’m wrong about it).

So, let’s discuss it. Do you think these issues are important? Do you feel there are other important things missing? I’m interested in your opinion!

6 Likes
2

Latency due to the need to constantly re-compile code is a huge issue (in my opinion this is by far the biggest problem with the language) that everyone is abundantly aware of and it is being worked on. There are many threads in this forum discussing it.

That’s really the crux of it. Once it is possible to easily cache and restrieve compiled code between distinct sessions of the Julia run-time, the above problem is solved.

This is related to the above, but would require “pruning” functions from the cache which would not get used. I’m not sure how easy this is to do. This may be a more difficult problem, I’m not sure how much thought has been given to it so far, it certainly comes up less often than the need for compile caching.

Python can do this because Python is just the interpreter, that’s it. All of Python is just the execution of a single program, the interpreter, it doesn’t need to store actual compiled code for anything the the Python stdlib or anything like that. Of course, this is a bit of a lie, because to do anything useful in Python, one typically needs compiled code to call upon (e.g. numpy, Cython components of pandas), but that compiled code lives with the individual packages, not Python itself. You can still typically get much smaller binaries that way than with Julia, simply because there’s often less compiled code and you can more easily select which binaries you want to bundle.

Perhaps you are simply not aware of how it is possible to load code? I suggest starting here and here. There are quite a few options for how to do this in Julia

  • If you want to bypass the package manager altogether, you can include the file containing a module declaration and do using .ModuleName (ok, that’s 2 commands).
  • You can add the package in the standard way with the package manager with ]add.
  • You can point the package manager to a particular directory with ]dev.
  • You can modify LOAD_PATH (I don’t recommend ever doing this).
6 Likes
3

I think Julia would be better off maintaining it’s focus on scientific computing (which is already a large and diverse application).

There are always trade-offs to be made in language design and shifting Julia’s focus towards general-purpose computing would likely hurt some of it’s strengths when it comes to scientific computing.

9 Likes
4

If you want to bypass the package manager altogether, you can include the file containing a module declaration and do using .ModuleName

But if there’s a mutual dependency, I’ll get WARNING: replacing module Whatever and who knows what kind of issues that might arise due to duplication. It’s just astonishing how unpolished the module system is in Julia compared to other languages.

Btw, I’m talking about importing modules within the same project, for example a utility module, or a module for common structs.

5

How would the goals I outlined hurt any of its strengths? I can’t think of a reason…

2 Likes
6

Are you sure that you are talking about Julia?

Did you by any chance miss the everything about its package manager?

6 Likes
7

You only get a warning if you are trying to replace an existing module. I don’t understand why you think the module system is unpolished, I find it to work extremely well since even before Julia 1.0. Is there something specific you are trying to achieve that you are unable to do? It sounds to me a bit like you don’t know how to do something, perhaps we can help?

I agree with @erezsh in that I don’t see why one would need to sacrifice any of Julia’s strengths in regard to scientific computing in order to make it useful in the general case. It seems to me that it’s already most of the way there. It seems to me that the only serious limitations of Julia in regard to “general purpose” computing are

  • Need to constantly re-compile.
  • Binary size (seems like a smaller issue to me, but certainly it can be a huge problem in some applications).

as have been pointed out. These are of course issues for scientific computing as well.

5 Likes
8

I am not sure that it is necessary to start another separate discussion.

12 Likes
9

Even when you build an executable? I haven’t played with building an executable but I had assumed it compiled the code it could at that point.

As others have mentioned not sure what you mean by this. I guess you could try to compile the “import” and “include” syntax but I usually end up with a “main” file that does all the includes and the individual files just do imports as needed. And if a module is “generic” it gets moved into it’s own source tree/git and I just use the package management provided by Julia.

Last I checked the executable size was huge. However I did notice that all the system libraries ended up copied to the build directory even if they were not used. When I looked at the files the executable actually linked to there where only two. But the combination of the executable and the libraries was larger than Go programs which I thought where huge.

The executable is still a work in progress so it should get better in the future.

I am using Julia as a general purpose language, nothing to huge so I don’t think I’ve gotten hit with any outrageous startup times. However mostly I’ve been using Julia as the server side for web applications so the UI code is all in TypeScript and HTML while the server side just needs to handle the backend data manipulation, thus keeping the Julia code smaller than if everything was in Julia.

10

Thanks for your comments, I always learn from these views. But, hard to discuss without clarity around what “most programming projects” are. The dynamic interactivity with developing a ‘program’ in the REPL or a notebook for computational or data science ‘programming’ has been awesome! Plus, “most programming projects” is coupled to concept of “most programmers” Are you considering entire world in your scope?
One other point of reference from @jeff.bezanson is this Tube that I found most fascinating, again learning about Julia and the broader ‘computer science’ space: " JuliaCon 2019 | What’s Bad About Julia | Jeff Bezanson"

It truly feels great to be part of such a humble community, and believe this is a key factor if one were dropping chips on bets for the future. It seems most here don’t see an either / or, but embracing the best from the diversity of the broader ‘programming ecosphere’ (pycall, rcall, cwrappers etc.) But, I’m still incredibly ignorant at this point and so can’t discuss substantively.

Oh interdasting! You just might have helped me crack a little nut I was wrestling with last night - thanks!!

3 Likes
11

I’ll chime in a little on this. Personally, I found the pacakge manager extremely confusing to use and had quite a bit of trouble understanding how it worked when reading through the manual. In fact, it frustrated me enough to rewrite one of my projects that was functional in Julia into R/C++.

It’s very important to note that I came to Julia from R. For recent projects, I’ve had to pick up Python. In doing so, I had to learn all about these things called “virtual environments”. After learning about that, suddenly, a lot of the super confusing things about Julia seemed to make sense. I haven’t dived back in yet, but it’s very possible that having a better grip on virtual environments will make the package manager and imports more intuitive, such as to discourage me from doing stuff like taking one script in a directory, putting it somewhere else and expecting it to run. I haven’t had time to look back into Julia to know if all my confusion would have been resolved yet, but it’s certainly possible.

Now, I’m not saying that the manual is definitely missing key information, but it was certainly a very large hurdle for me. I think having a stronger Python background would have made the Package Manager more intuitive.

1 Like
12

Could you point out a few risks? I see absolutely none.

1 Like
13

To be fair, there is some friction with how people prefer imports to be handled. For example, there is the opinion that import shouldn’t be used at all and only using with explicit overloading of functions using <Module>.method being the way to do things. However, nothing is stopping people from doing this now.

On a side note. I also think that the demands of a programming language for scientific computing today basically cover and exceed the territory of a general purpose language. I can’t think of a single thing people would consider a “general-purpose language” feature that wouldn’t be expected to work in something like data science. HPC, file management, web scraping, graphics, mathematics, etc. are all expected skills in data science now.

10 Likes
14

This exactly.

You can see this frequently here, whenever people demand certain functionality to be available as ready-to-use functions. I always wonder, why these people believe, that this should be there. Than I think of R and than think, ok, these are users not programmers (no offence meant).

I originally come from general purpose languages, started seriously with assembly on 68000, over C, C++, Java, only the last years (10-15) I mainly used scientific computing languages perl, python, R and of course Julia.

So I guess it is somehow a generation thing how people are looking on a now not-so-new-anymore language.

There are only two things I am missing:

  • the already mentioned stand-alone-ready-to-share executable with only needed libraries (btw this implies precompiled and cached code, as code changes wouldn’t happen anymore per definition)
  • a modern GUI library (always thinking of starting one but shying away realising the effort)

Everything else which is mentioned so far is maybe not so perfect sometimes but totally usable and every problem seems to be overcomeable somehow.

4 Likes
15

I think you are right, but there are differences in priorities. Eg delivering executables may be less of a priority when compared to other features, which is why we don’t have it yet (but we have a ton of nifty features instead).

Most discussions like this topic are actually about similar differences in priorities: its not that Julia doesn’t have some features (yet) because its contributors don’t consider it important, but rather because they wanted to focus on something else. A corollary is that there is little point in asking for these things: one can just contribute towards them, or wait until someone else does it.

2 Likes
16

I am totally shocked by these words.

Watch this 5 min video:

4 Likes
17

Good point. I feel like this ends up being the conclusion of most of these threads

If you mean something that helps build GUIs then there are definitely things in development.

1 Like
18

Take for example a language like Rust. It has been designed with a focus on memory-safe system programming. So they’ve made design decisions when it comes to the syntax and “unsafe” operations that are reasonable for that type of conservative programming where a crash could be a huge security problem, but require extra boiler-plate code that seems unnecessary for a typical scientific programming task.

Another issue is simply a matter of priorities. Features that are useful for scientific computing may be ignored if the language developers are focused on delivering features useful for other types of programming. Again, see Rust and its lack of a REPL and constant generics (useful for type-level array lengths).

Another issue is the selection of functions in the base library. There’s a limit on how much is reasonable to import by default. The more domain specific a language is, the more relevant these functions can be. As an example of a highly domain specific and successful language (and database) take kdb+/q, which is used almost exclusively for the management and analysis of financial market data. There are base functions like “as-of time series table joins” that are very useful in that domain, but would likely not be present in a general purpose language.

A non-technical, but still important, reason is branding and messaging. It’s quite easy to make the case for Julia being the “best scientific programming language”. But can it really stand out as a general-purpose language? It could certainly be a decent option, but I don’t think that’s much of a unique selling point.

3 Likes
19

I really don’t see Julia as a general purpose language. Not that it couldn’t be - it definitely could be.

I see it as scientific programming backend. Most applications, underneath it all are just data management/access(think corporate soft-ware). If you want to hookup 10+k lines of bunch of UI/UX with Julia - I don’t see why you couldn’t. But, would I currently be writing a UI completely in Julia except for the purposes of making a notebook/demo? No. Are people working on this, yes! Are they doing it in pure Julia (not to my knowledge).

The compilation status of Julia is a huge priority right now. Lots of smart people are working on it. That being said, no one ever said that Julias purpose is to build standalone applications. I kind of prefer it not too be the primary focus to be honest. There’s so much over-head in that, and that over-head is a moving target, a bed of sand. It requires a lot of effort once a new piece of hardware comes out, a new OS, etc. It can also force people into bad design decisions, that detract from the languages true strengths.

I’d rather see Julia focus on what matters - the compute power, syntax, availability of easy to use tools, and staying at the funky bleeding edge of scientific applications. I don’t mind waiting 5-35 seconds the first time I launch a program especially if over the course of it running it’s saving me 2 days of run time and 3 weeks of programming time. Especially if I can open up some source code and actually flupping read it - because it actually looks like the math its performing. I also don’t mind writing a bash script so a user can deploy a julia program I’ve written, or the alternatives (ZMQ/harddrive passing data between some kind of front end). Not a big deal to me.

I’m worried if Julia tried to fit into the area of general programming it could come off as unpolished or immature. I mean, imagine being a brand new programmer seeing some buzz about Julias new front end capabilities trying it and finding it’s not nearly as clean as .NET(10 yrs in the game + thousands of people actively working on it). Could get a bad rap for being clunky, when its real advantages right now aren’t in that space. I’m not trying to say .NET is awesome, I’m not a big fan, but it’s an example.

3 Likes
20

Oh - One thing I would like to see developed is access to COM ports or other hardware. Typically that’s done in other languages, but wrappers to allow for those sorts of things would be awesome. Think LabView.