Workflow for Julia scripts


#1

I am trying to use the Julia REPL to create some Julia scripts. My programming background is Matlab and a little bit of Python and C/C++.
However, I am a little bit unsure, what the best workflow is for creating Julia scripts.
According to the documentation on workflow the best way is to create a module, where all the functions are defined and then include on the REPL a file like myscript.jl, which in turn imports the module.
On the REPL I can then run myscript multiple times and change things, until I arrive at the final script.

Another workflow outside of the REPL I have seen, is IJulia Notebook, which I do not really like.

Ideally I am looking for a workflow like Python or Matlab:
I have my working folder, can put several functions in one file/several files and directly call them from the REPL, without having to include them. Of course, the ability to make a module is great and a big advantage especially over matlab, but it feels for me like too much overhead, if I just want to make some quick calculations / plot some data in a script.
Is there any way to achieve a more Matlab/Python REPL feeling?
What kind of workflow do you use, when you just want to use the new model, which just has the right tool you needed all along?


#2

I really like the package workflow, even for my own projects. Pkg.generate a package, then put all of your functions in the module. Then you write your scripts in the test directory, just adding using PackageName on the top. On top of that, you can fill in your runtests.jl with unit tests to make sure you don’t break anything along the way. In the end, I have plenty of functions, plenty of tests, and then a few scripts in the test folder that actually do my research. The best part is that this doesn’t impede your research (the workflow is pretty smooth and writing tests speeds you up) but also makes it dead simple to publish a package afterwards (just add a README and turn on CI). So you can do all of the first part without ever publishing the package, I just think the structure is a very convenient way to organize code (“library” vs “scripts that run things”)


#3

PkgDev.generate I think…


#4

You don’t need myscript. Write your code inside functions in a module, then at the REPL

julia> using Revise  # to automatically reload your code when modified

julia> using MyModule

julia> compute_flow(23)  # call some function in MyModule

# modify the source of MyModule.compute_flow

julia> compute_flow(23)  # run it again

etc.

How is your workflow in Python?


#5

This looks like what I do. However, what I’d like to know is how to handle directories in a clean way. For instance, I would do

shell> cd c:/Users/PetrKrysl/.julia/v0.7/FinEtools
julia> using FinEtools
julia> include("examples/deformation_linear/dynamics/forced_vibration/3-d/TEST13H_hva.jl")

Just looking at this I kkeep thinking there must be a better way. What do you guys do?


#6

That’s why I do the package thing. Julia already has the module in its path and knows my scripts are in Pkg.dir, so joinpath(Pkg.dir("Package"),"examples")' or whatever you like to name the folder (I usually just put it all in test by habit. I have one project which essentially is running the same script all the time, so I made it the unit test and just use Pkg.test

At one point I was against it, but I think it would be a good idea to have packages be “runnable” by default, I.e formalizing this with a Pkg.run("Package","script1")


#7

This is good. In my circumstances this would need a little bit of tweaking: all of my “projects” are independent computations, which means packaging each of them into its own separate package means that I’d have to either re-member their names, or list the package folder to remind myself of them. There would be dozens.
Right now this is handled by the include: I can use the expansion of the filename to help me along.

Packaging every single of these projects into a single module as submodules is another option. The file would be tremendous however, and hence probably slow to load.


#8

You may be able to use Base.find_in_path to find modules which are not packaged, and construct a path from there. See how I use it in RoguePkg.jl.


#9

I’ve started adopting the adviced workflow, and I must say that for as far as I worked like this, if you want to create any extended functionality that isn’t limited to one research project, this feels to me like the way to go.

Up to now what I have been doing is just have one file which I put my lines in, that includes the packages I want to use + includes the other files directly. This means that you can just edit the function, reevaluate it in the repl, and be sure that everything else will use the new version.

However now that I discovered that inside VSCode you can switch the repl to evaluate things in the context of a certain module, there almost seems no reason anymore to not just throw stuff into a module and work from there.


#10

We need a nice way to deliver scripts inside packages and put them in the path.

Related: Delivering a Julia application within a package

I would be great to have some way to call scripts from the command line using the package name:

julia PkgName script_name.jl --args values

#11

Also, it would be great if you could do it internally from Julia too, maybe something like

julia> include PkgName.script_name

where script_name.jl is located in some .../PkgName/exec/ folder or whatever.

Another idea is to have a scripts folder in the ~/.julia/scripts/ directory or something, so that if you type

julia> include("script_name.jl")

it automatically looks in that folder in addition to the current directory. Also, it would be nice if the syntax could leave out the quotes and the .jl part, so that you can just do

julia> include :script_name

which is much quicker.

Finally, the ~/.julia/scripts/ folder should be accessible via some command like cd(Pkg.scripts)


#12

Great suggestion, thanks!


#13

The master branch of Julz does what you want to do.

  • Julz.new(<pkg_name>, "mit") makes a package
  • Julz.generate("task", <task_name>) makes a task (if you’re in the right directory)
  • using <pkg_name> loads the package with every files inside it (done automagically w/ Revise added)
  • Julz.run(<task_name>) will then run the desired task

All this can be done from the command line also, without the need to explicitly open the REPL


this was born from similar posts:


official juleps issue related to runnable packages (as mentioned by @ChrisRackauckas)