Advice on Command Line Usage on HPC with Julia

Specific Domains
, ,
1

Hello,

I have been working and re-working the method I use to generate and submit runs on my HPC and I’d appreciate any advice or comments on the process (I’m sure there is plenty of room for improvement). Apologies if any of this seems unnecessarily complicated. This is just me fumbling my way through it.

Right now I use ArgParse.jl. I have a parser that basically looks like the following. I use keyword arguments because it makes it easier to enter the commands (since there are so many, otherwise I’d get confused)

function parse_my_commandline()

    s = ArgParseSettings()

    @add_arg_table s begin
        "--parm1"
        arg_type = String
        "--parm2"
        arg_type = String
        "--parm3"
        arg_type = Float64
        "--parm4"
        arg_type = Float64
        "--parm5"
        arg_type = Int
        # ... And so on to ~30 total 
        end

        parsed_args = parse_args(s)
        # Do some stuff based on default values of args, some error checking, etc.
        return parsed_args 

end

This works but the length of the command line is getting a little silly. Is there a better way to do this? Some type of config file I type up and then feed to the parser? Just so I don’t have to type up such a long command at the terminal? Or is this typical?

2 Likes
2

Note that you can add a short form for each argument. -p etc

2 Likes
3

I recommned using the TOML.jl stdlib to read in a .toml file as a dictionary and then use get(dict, :my_parameter_name, default_value) to query the parameters.

1 Like
4

Then for submitting several jobs at once, there should be several TOML files created, right? The good thing about command line arguments is that you can submit different jobs by changing some args in a bash loop, for example.

5

Personally I prefer this over explicit command line args, because if I keep the parameter file around then it automatically documents what parameter values were used for which simulation.

I also think it is quite easy to write a julia script with a loop to generate the needed configurations and dump them to a file, again using TOML.jl.
[Although in practice I never do this and instead generate the files manually as most of the times this is faster.]

But all of this is personal preference, so YMMV.

1 Like
6

Consider DrWatson.jl to prepare and submit jobs without a command line.

1 Like
7

Thanks for the suggestion. I actually am using DrWatson for this project. Did you have a specific functionality of DrWatson in mind to accomplish what I described above? I might’ve missed something in the docs.