"Locally global" variable?

Is there a way to make a julia variable locally global (for the lack of a better word) ?

To give a little context, I am currently trying to implement some inter-process-communication using julia.
With what I know (understand very little) about this process, I need to read a named pipe multiple times from a given server.
My example code here is shown again below

using MsgPack, Parameters, Sockets

struct TestStructure
    a::Int64
    b::Vector{Float64}
end

# Prepare MsgPack to understand "TestStructure" structures
MsgPack.msgpack_type(::Type{TestStructure}) = MsgPack.StructType()


function MsgPack_server()

    # Initialize named pipe server
    server = listen("\\\\.\\pipe\\JuliaPipe")
    # Keep waiting for instructions
    while true
        # Connect to server
        sock = accept(server)
        # Wait until reception job is finished
        while isopen(sock)
            # Read data
            msg = readline(sock)
            # De-code data
            test = unpack(msg, TestStructure)  # <--- in the end how to replace this using information coming through the pipe
            # Print data
            println(test)
            # Close connection
            close(sock)
        end
        # Start new connection to server
        sock = accept(server)
        # Wait until send job is finished
        while isopen(sock)
            # Perform some computation
            test2 = TestStructure(3,rand(Float64,5))
            # Code data
            msg = pack(test2)
            # Write data to stream server
            write(sock, msg)
            flush(sock)
            # Close connection
            close(sock)
            # Print data
            println(test2)
        end
    return nothing
end

The variable test is nested in two while-loops so it is not accessible from a higher scope.
However, for my use case, I would like to access such streamed variable, so I can actually use them in some computations. So, how should I make such variables accessible from another iteration of the second while-loop hence the locally global.

You can use a let block

https://docs.julialang.org/en/v1/manual/variables-and-scoping/#Let-Blocks

As far as I understand this (please correct me if I am wrong), a let block protects a local declaration from a global one, and probably in the same manner protects the global variable from the local declaration.
However, I can’t figure how it would help trespass such scope protection…

You can use local on the level of of scope you want:

julia> function f()
         local x
         for i in 1:10
            for j in 1:10
               x = i*j
            end
         end
         x
       end
f (generic function with 1 method)

julia> f()
100


4 Likes

You can use a let over a function, eg

let test = nothing
    function MsgPack_server()
        ...
    end
end

but frankly I would just refactor and pass it around explicitly, much cleaner.

3 Likes

What do you mean by

I would just refactor and pass it around explicitly

As this is the very first time I implement IPC, it is surely a naive implementation, I am open for any advice :slight_smile:

If you want to reuse test, pass it around as an argument, possibly in a container like RefValue if you want to modify it.

(Sorry if I am missing something about the example, it is not very clear).

To clarify a bit my workflow with this program:

I am doing IPC in the following manner : connect to pipe -> read stream -> do something with the streamed data. Sometimes in send

  • strings : which are used to determine what computation should be launched (within the MsgPack_server function)
  • custom structures : which are used as arguments for the computations

Each time I send an information through the pipe, one iteration of the while isopen(sock) loop runs, making the passed data unavailable for the next iteration e.g. :

  1. First iteration : a specific struct is sent over the pipe. test is of type mystruct1 using the type I can store test with appropriate if ...elseif ... end blocks such that ms1 = test
  2. Second iteration : a string is sent over the pipe, let’s say test is of type String and test = "DoSomethingWithMs1" . Again, with appropriate conditions, I can start a ms2 = DoDomethingWithMs1(ms1). But for that I have to know ms1

So basically, I need persistence of previous-iteration variables inside my MsgPack_server() call.

I hope it is a bit clearer now.

So, just have the first iteration return it, and pass it to the second iteration as an argument?

I am sorry, I don’t see what you mean. Currently I am doing something like

while isopen(sock)
msg = readline(sock)
test = TypeAwareUnpack(msg)
if typeof(test) == String
    if test == "DoSomethingWithMs1"
        ms2 = DoSomethingWithMs1(ms1)
    end
else
    if typeof(test) == Mystruct1
        ms1 = test
    end
end

Is this what you are thinking about (at least that’s how I understand your last comment)

I am still not sure I fully understand the question. If

then maybe wrap the code above in a function and also return test?

I think I am starting to see where you are going with this, but in the end I am not sure if it will make much of a difference than using the local proposed by @leandromartinez98.

Wrapping in a function would probably improve performance a bit though.
Anyway, thanks for the advice.

1 Like

The decomposition that @Tamas_Papp is recommending will not only possibly improve performance, but will definitely increase reusability, composability, and maintainability by following the principle of separation of concerns. That is, separate the IPC logic from the application (ie., compute) logic.

1 Like

As I said before, this is surely a naive implementation, and I am all ears for advice. I just do not see at the moment what this decomposition would look like, that’s all. If you have an example for a good practice, feel free to post it :slight_smile: .

One approach would be split your function into two, each containing one of the inner while loops. Call one readPipe and the other writePipe. Then call those like this:

  data = readPipe()
  results = computeSomething(data)
  writePipe(results)

Then your outer while loop could wrap those, or better, put those three lines in a function and calling that function could be the body of your outer while loop.

Another approach could be to use three separate Tasks, two for reading and writing to the pipe and another for computing the results. The Tasks would communicate via Channels. Asynchronous Programming · The Julia Language

So you mean the overall code would look like

function readPipe(server)
    sock = accept(server)
    while isopen(sock)
        msg = readline(sock)
        data = unpack(msg)
        close(sock)
    end
    return data
end

function writePipe(server,data)
    sock = accept(server)
    while isopen(sock)
        msg = pack(sock, data)
        close(sock)
    end
end

function read_compute_write(server)
    data = readPipe(server)
    results = computeSomething(data)
    writePipe(server,results)
end

function MsgPack_server()
    server = listen("path_to_pipe")
    while true
        read_compute_write(server)
    end
end

?
Thanks for the tip about tasks.

1 Like

After writing my last post, I think I understand now how my first one was ambiguous.
I actually need to reuse variables output by the readPipe in numerous iterations of the overall while and not necessarily in the subsequent writePipe e.g

First iteration : data in pipe is of type Data1

data1 = readPipe()

Second iteration : data in pipe is of type Data2

data2 = readPipe()

Third iteration : data in pipe is of type String

result1 = computeSomething(data1,string)

Fourth iteration : data in pipe is of type String

result2 = computeSomething(data1,result1,string)

So it feels like the previous code factorization is not applicable in this case (or more probably this factorization is a bit too aggressive) while the local works fine. Any ideas to improve this ?

Your code between readPipe and writePipe should provide all the necessary logic for determining what actions to take based on the type(s). readPipe and writePipe are still decomposed properly. Don’t intertwine the app logic with the IO logic.

Options for how to accumulate the data read from readPipe would be to put the data in some combination of mutable structs, Dicts, (named) tuples, arrays, or others, depending on your problem. You can either pass the entire collection to computeSomething and have it deal with the multiple options, or have multiple computeSomething methods and deal with the options before calling it.

In short, break the problem into smaller distinct parts and solve those separately. You’ll discover that your problems and solutions will be reusable. Resist temptation to throw it all in a big pile because the pile will become unwieldy fast.

2 Likes

So just to sum up a bit

  • Messaging should only take / return one argument (if server is omitted) : the byte package / the data passed
  • Filtering conditions based on data may be performed inside or outside of computeSomething as convenience.
  • Variables that should be accessible in the main loop should be so through an appropriate container. For instance
mutable struct MyOverallContainer
    data1::Data1
    data2::Data2
    result1::Result1
end

Which would then be passed and modified by each computeSomething call.
By the way I think this would not work for me because all of my structures are actually immutables so I wouldn’t know how to initialize this starting MyOverallContainer, but a Dict would surely work.

Do you agree with this summary ?

Yeah, that was my plan, but I just came up with this code like few hours of work ago, so understanding was the priority.

Yes the summary looks good. Immutable state is usually “a good thing”.