Ensure Julia is used to its full power

Close, same language, Portugal.

Okay so my goal was to keep the workflow in the terminal with Julia, but it seems that the compile time of the script itself is what is really high which would make my workflow unsuitable for scripting.

I’ll look into your guides as Julia is regarded as a really good language, hopefully I can overcome those minor issues and sucefully implement Julia in my future work. Thanks!

That was something that I was looking for because most time we end up using external libraries, such as plots for example, and then we have a few files that we hardly change and then the main file where we change it all the time.

As such I would love to be able to share my project and the user would download and compile everything on his side and just run the program at will without having to worry about anything else.
Ideally the user would also be able to change the “main source code”, the one I wrote, without recompiling the whole thing, for instance they would never change how plots behave and as such it would never recompile.

There are two cases there: Your package is ready and you will only run your simulation with different parameters, etc. In that case, if the simulation is really heavy, even if the compilation takes a lot of time (a minute, lets say), it is unlikely that this compilation time is important for the total running time of your simulation (we assume here that a heavy simulation should take an hour, at least :slight_smile: ). Thus, for really costly computations the compile time is really not important.

Some parts of the package are precompiled when you load them for the first time, so you would not be compiling everything every time, but that is perhaps a detail here.

Where these compilation times can be annoying is when you are developing your package, because one generally performs quick tests all the time. In this case, re-running the script from scratch every time is not the way to go with Julia. You should keep a Julia section alive and use Revise, or reload the files in that Julia section, as explained in the post above that describes possible workflows (everything there is terminal based).

2 Likes

For that I guarantee that Julia is much better than C++ or Fortran. You will host your package in a github account. Your users will only have to “add” your package. Dependencies will be satisfied automatically. If the package does heavy computation, the fact that package gets compiled on execution should not be a problem, as mentioned above.

The source code will be available for them to see and modify if they want to.

Sharing a package in Fortran or C++ which depends on other libraries is very cumbersome. You have to either teach everyone how to compile it, download or install the dependencies, choose the compiler, etc, or share a binary, which must be built for every possible platform independently.

6 Likes

I find very strange nobody mentioned Jupyter yet. For exploratory computation or debugging, Revise.jl inside a Jupyter Notebook gives a “less claustrophobic” experience (in my opinion) than the Julia REPL (note: I have already used a computer only in text-mode for 6 months and, also, I have no file manager, I do everything in Bash; but even so, when you are working with chunks of code, it is annoying to not be able to easily see all code you are considering and using arrow up to find the line to re-execute is tedious, so I consider Jupyter a great quality of life improvement over using the REPL directly for anything you take more than one or two minutes to do).

5 Likes

Keep your julia session open and use Revise.jl (maybe even with this workflow?) instead of restarting julia over and over. Another option would be Pluto.jl.

I don’t know how your script is written, but if it’s not in the form of functions without globals, chances are you’re leaving a lot of performance on the table anyway, regardless of restarting julia or not.

The first part of this already works if you put your code in a project (which is used to manage dependencies). The second part about not having to recompile the code is not currently possible, but may be possible in the future.

That depends on how addicted you are to Vim and gnu-screen or similar alternatives:

2 Likes

It would be interesting to get this working. I already use vim daily for years (but no gnu-screen on my personal machine just tmux in the lab servers). However, unfortunately Jupyter has the advantage of: being a format to share with other researchers; plots and tables are shown right below the cells in a convenient fashion (I use a plugin for tables to be able to interact with them, but only work on Jupiter-like environments). But yes, if you use vim and do not need use plots and tables often this may be a better experience than Jupyter (the lack of vim hotkeys when writing in Jupyter annoys me a little).

There I am using this plugin: https://github.com/mroavi/vim-julia-cell

I do not know how people use Jupyter to share codes, I might be doing something wrong. Everytime I try to do that I find out that every package has to be installed in the specific jupyter section that one/some else opens, such that it is absolutely out of question. Probably I do not know how codes have to be shared.

1 Like

Are you familiar with the startup.jl file? You can add some code that colleagues may use in their session (including calls to the plotting library). This will cost some extra time to initialize Julia, but this is probably less of an issue.

Oh, I keep a Project/Manifest.toml in the same folder and the first cell just import Pkg and activate and instantiate the project. The first time, and if you do not have the right versions, the packages will be installed, but every other time the instantiate will just discover the packages are already installed.

I do the same thing. For a concrete example, see: https://github.com/rdeits/EdgeCameras.jl/tree/master/notebooks and the first cell of https://github.com/rdeits/EdgeCameras.jl/blob/master/notebooks/demo.ipynb

1 Like

Thanks for the tip, but my problem with Jupyter is at another level. I first thought that I could host a notebook on the web, share it with students, and that students could open it and have it running right away. This is not the case, because when the students try to load the project the packages start to be installed, and that can take a lot of time. Worst, depending on how the student save the work done, every time he/she loads the notebook everything starts again. I suppose you are referring to launching Jupyter from whithin Julia. In that case I do not see many advantages in relation to share a code directly, except for some pretty formatting.

That would be really great for portability actually, so returning back to my example where I would make a Julia program and then plot something using Plots.jl I would just send the whole source code and in the first run it would compile that package and on the next runs, regardless of how the user changes the file that performs the plot, it would not recompile Plots.jl?

You may also want to try Juno, which allows you to work in code cells like Jupyter, but works with plain .jl files, and has results showing either in inline pop-ups, in a REPL pane, or (for graphical results) in a plots pane. It’s similar in style to the MATLAB IDE, though unfortunately not quite as polished and responsive.
I find it more convenient than the Jupyter style of results under each cell. I usually use Juno+Revise with a “live” julia session in my everyday work, restarting Julia only occasionally.
There’s also a Julia plugin for VSCode, but that seems to be in a more preliminary stage of development at this point.

1 Like

Actually I have moved to VSCode from Juno. The original Juno developers now work on its Julia integration. It now has almost all the Juno functionality (especially in the Insiders Edition).

Juno is now in maintenance mode IIRC.

2 Likes

I don’t use vim but there’s things like

1 Like

Maybe you could exemplify where are you having these issues, so we are on the same page. For example, this code:

using Plots
x = sort(rand(100)); y = sort(rand(100));
scatter(x,y)
savefig("plot.png")

It takes 15 seconds in my laptop to run, with:

julia simpleplot.jl

(with Julia 1.6, to be released, it takes 10 seconds).

(a similar script in python takes 0.5 seconds)

That ~10 to 15 seconds is the delay associated to the “first plot” we are talking about. Certainly other scripting languages are faster to startup (for now, at least), but this is the “compilation time” that you will experience in everyday use, and that we claim that is irrelevant if what you are doing is a heavy calculation.

edit: On the very first time you use the package it gets compiled and it takes more time, but this occurs only when the package is installed or updated. I think that in Julia 1.6 they moved this first compilation to the installation process (which is something nice, because we are usually willing to wait some time upon package installation, but not so in its first use).

2 Likes

I suppose what counts as “almost all” depends on how you tend to use it. I haven’t tried the Insiders Edition, but based on the official release, at this point I would still recommend Juno. The VSCode plugin still misses some of what I would consider basic functionality for working with code cells, like commands for moving between cells, or executing a cell without moving to the next.

Okay I think I’m getting it, on the first run right after installing Plots I got, on your script:

$ time julia misc/j.jl
julia misc/j.jl  141.87s user 6.04s system 97% cpu 2:31.76 total

On the second and consecutive runs I got:

$ time julia misc/j.jl
julia misc/j.jl  20.74s user 0.48s system 94% cpu 22.490 total

This is on a laptop with a i5-7300HQ @ 2.50GHz (Max 3.50 GHz) Quad Core CPU. So what this means is that to compile and execute this rather small script takes about ≈ 20 seconds?

I also noticed Julia created some files over at ~/.julia which seems to include a lot of things and seems a little big. I’m assuming that this is the folder used to have my Julia environment and it has both the source files and compiled files of all the packages I’m using. I have a few questions regarding that:

  • Usually when I install a package on my system it’s already compiled, making the user download the source code and compile the package in it’s system would take up quite some space after a while of playing around. Can’t it be to download the binaries pre-compiled?
  • How do Julia environments work? Is it on a project based scenario where I would just keep a folder with all of my Julia files and in that folder I would include all of the source code of the modules I’m using, or does it always use the files over at ~/.julia?

I can see that this is a language which is still being worked on, and I really never took in consideration, like somebody previously mentioned here, the time, effort and money people took to develop these languages and make them usable and convenient to use for all sorts of people, so I’m probably asking something that has already been asked before around here or that is being worked on but I’m unsure where should I go to to keep tabs on what can or cannot be done (usually google always provides me with an answer in the first page when it’s something related to Python, Java or C++ with whatever issue I run into).