Pipeline question

New to Julia
1

Continuing the discussion from Structure and performance questions from a Fortran programmer: functions, global variables, array allocations, input files, and structs:

Is this how you move stuff from place to place in everyday scientific pipelines? Or do you tend to do something else? If you do, sometimes you need to get rid of some fields in these NamedTuples. How do you do that?

Thanks for your help!

2

I think it’s best to think of named tuples as anonymous structs. Convenient, but if you’re always returning the same objects, creating a proper structure is the next logical step. There’s nothing wrong with having lots of small structs.

As for deletion and general modification, Accessors.jl is the way to go

julia> using Accessors

julia> nt = (; x=4, y=10)
(x = 4, y = 10)

julia> nt2 = @delete nt.x
(y = 10,)

(beware that it creates a new named tuple)

3

I didn’t know how to delete an entry. Thank you, @cstjean .

To select a subset:

julia> nt = (; a = "hello", b = 3.14, c = -999, d = :somesymbol)
(a = "hello", b = 3.14, c = -999, d = :somesymbol)

julia> nt[ (:b, :c) ]
(b = 3.14, c = -999)

This also creates a new NamedTuple, I think.

4

Great answers! But is this how data is typically moved around?

5

I’m not sure exactly what you mean by that and I’m not sure whether what I do is “typical” or not, but here is what I do. Suppose you have a data-producing function. I usually write it as

function read_or_produce_data()
   # ... read or calculate variables ...
   val1 = ... some calculation ...
   val2 = ...
   val3 = ...
   ...
   # NamedTuple shorthand
   return (; val1, val2, val3, ...)
   # return (; val1 = val1, val2 = val2, val3 = val3, . . .)
end

You can then use it in different ways. You get the whole thing:

vals = read_or_produce_data()
func1(; vals...) # Send all values as keyword args
func2(; vals[(:val2, :val5)]...) # Send a subset
func3(vals.val2, vals.val4) # Send two as positional args

You get only a subset and discard the others:

# Discard everything else
(; val2, val4) = read_or_produce_data()
# use val2 and val4

Sometimes, you want to ignore the original names:

(par1, par2, ...) = read_or_produce_data()

because NamedTuples can be assigned to a regular Tuple ignoring the names.

In this way, NambedTuples are my main means of moving data around throughout my programs.

6

I think what you’re asking isn’t really clear. Are you asking about movement from storage to memory (I/O), from one place in memory to another (e.g., passing a data structure from one function to another), or from one machine to another (over a network)?

7

@mthelm85 I’m asking the second question, sorry for the ambiguity.

Thanks @ryofurue for the great answer. I didn’t test this myself, but if I understand it correctly, this is pretty cool:

(; val2, val4) = read_or_produce_data()

I mean, even if read_or_produce_data() can return a lot of stuff, I can choose to return specific things without having to remember their positions, i.e. no _, _, val1 = syntax. Is this correct? And if so, I guess this is one of the things that make NamedTuples quite unique for passing data from function to function compared to other containers?

By the way, and regarding the name of that function, I’ve been using DrWatson.jl, which I find super useful and has a function called produce_or_load (I haven’t used that particular function, but it looks great.

8

Exactly. That’s the very reason I use a NamedTuple to return values from my functions. See the toy example below.

julia> read_or_produce() = (; a = "hello", b = 3.14, c = -999, d = :somesymbol)
read_or_produce (generic function with 1 method)

julia> (; d, b) = read_or_produce()
(a = "hello", b = 3.14, c = -999, d = :somesymbol)

julia> b
3.14

julia> d
:somesymbol

julia> a
ERROR: UndefVarError: `a` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
9

By the way,

Thank you for the information. It looks very helpful. That may be what I should use to manage my programs. I’ll explore it.

10

So far I have only used initialize_project, savename, and datadir/plotsdir etc. That’s the tip of the iceberg, but they are excellent in my opinion.

11

Now back to the topic, I believe this issue is especially important for those of us coming from writing infinitely long MATLAB scripts (or functions that are simply obj = myfun(obj)) and entering a much more functional world in Julia.