# Data exchange between Julia sessions (same or different machines)
**URL:** https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157
**Category:** Julia at Scale
**Tags:** question, distributed
**Created:** [April 24, 2020, 4:27pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157 "2020-04-24T16:27:28Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@martincornejo](https://discourse.julialang.org/u/martincornejo)
#### Post date: [April 24, 2020, 4:27pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/1 "2020-04-24T16:27:28Z")
Julia is great for implementing distributed computing. I was wondering how could it be possible to take a _decentralized_ approach to this distributed computing. Instead of calling workers from a Julia (master) session (for example through `@spawnat`), have two “independent” Julia sessions interact with each other and exchange data.
One could for example establish a connection with Sockets:
**Machine A**
```julia
# Local example
using Sockets
l = listen(5000) # Port 5000
a = accept(l)
```
**Machine B**
```julia
using Sockets
c = connect(5000) # Connect to port 5000 (local example, for remote: connect(host, port))
```
(I’m not sure how to properly implement sockets, would be also grateful for some help)
Then with the socket connection established, exchange data with some kind of Channel/RemoteChannel
**Machine A**
```julia
# establish connection AB/BA
# create channel result_a
# pre-processing
while statement
b = fetch(result_b)
a = foo(b)
put!(a, result_a)
end
```
**Machine B**
```julia
# establish connection AB/BA
# create channel result_b
# pre-processing
while statement
a = fetch(result_a)
b = bar(a)
put!(b, result_b)
end
```
Any ideas on how to bind a channel with a socket, or how to implement such _decentralized_ computing?
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [April 24, 2020, 5:08pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/2 "2020-04-24T17:08:48Z")
You could serialize/deserialize functions and arguments between processes, with the function only being serialized by name. I believe this is how Julia’s Distributed library works from a high level.
---
### Author:  [@risingganymede](https://discourse.julialang.org/u/risingganymede)
#### Post date: [April 24, 2020, 5:16pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/3 "2020-04-24T17:16:38Z")
For local data exchange, consider using shared memory (e.g. `/dev/shm` on linux) over sockets. Look up Apache Arrow and its subprojects (Arrow Flight, Arrow Plasma Store) for an example of low-overhead data transfer and serialization.
---
### Author:  [@martincornejo](https://discourse.julialang.org/u/martincornejo)
#### Post date: [April 24, 2020, 5:52pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/4 "2020-04-24T17:52:48Z")
@jpsamaroo @risingganymede Thank you for your responses. That’s not exactly what I’m looking for. My goal is not exclusively increase performance by multithreading/clustering, but have two independent Machines collaborating by passing each other some data while keeping most of it private.
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [April 24, 2020, 10:04pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/5 "2020-04-24T22:04:11Z")
Right, both proposed approaches would do that; when starting two separate Julia processes manually, you can then use either sockets or shared memory (plus some form of serialization) to pass only specific pieces of data between processes, or even to make remote procedure calls. Neither approach relies on Julia’s Distributed package, therefore each process can choose which data is shared and which data is private.
---
### Author:  [@martincornejo](https://discourse.julialang.org/u/martincornejo)
#### Post date: [April 24, 2020, 10:23pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/6 "2020-04-24T22:23:33Z")
Thanks @jpsamaroo ! The documentation on serialization is somehow vague, now that I’ve tried it out it seems it is what I was looking for.
@risingganymede I’ll also take a deeper look on shared memory, see if it also fits my needs. Thanks
---
### Author:  [@robsmith11](https://discourse.julialang.org/u/robsmith11)
#### Post date: [April 25, 2020, 12:12am UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/7 "2020-04-25T00:12:01Z")
It would be nice if there were a package that wrapped all this together for fast and convenient sharing of data and execution of code between 2 independent Julia processes (where one was not started as a worker of the other), on the same or different machines.
I miss having the convenient IPC from kdb+/q:
> **[Interprocess communication | Developing | kdb+ and q documentation - Kdb+ and...](https://code.kx.com/q/basics/ipc/)**
>
> TCP/IP is used for communicating between processes. The protocol is extremely simple, as is the message format.
@martincornejo, if you write it, I’d help test and use it. 😉
---
### Author:  [@robsmith11](https://discourse.julialang.org/u/robsmith11)
#### Post date: [April 25, 2020, 9:03am UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/8 "2020-04-25T09:03:26Z")
I wrote a first version with some of the functionality I’d like to have:
> **[GitHub - robsmith11/SimpleIPC.jl](https://github.com/robsmith11/SimpleIPC.jl)**
>
> Contribute to robsmith11/SimpleIPC.jl development by creating an account on GitHub.
It’s still a bit buggy. Evaluation errors seem to be swallowed up and cause the socket to break. Any tips on making it more robust would be appreciated.
---
### Author:  [@martincornejo](https://discourse.julialang.org/u/martincornejo)
#### Post date: [April 25, 2020, 10:25am UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/9 "2020-04-25T10:25:00Z")
I was thinking about it, but I don’t find the current procedure that inconvenient and I’m also short on free time right now. Nice to see you’ve taken the initiative, I’m not familiar with kdb+ nor q but I’ll take a look on your package.
---
### Author:  [@robsmith11](https://discourse.julialang.org/u/robsmith11)
#### Post date: [April 26, 2020, 8:31am UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/10 "2020-04-26T08:31:08Z")
I’ve pushed a fix that should solve the problem with evaluation errors.
Performance is acceptable for some purposes (1-3 milliseconds per function call), but slower than it could be because deserialization isn’t type-stable.
I plan on adding a type-stable function call that should get performance closer to 100 microseconds. At that point, I might add named pipe support as well.
---
### Author:  [@Thomas\_Markusic](https://discourse.julialang.org/u/Thomas_Markusic)
#### Post date: [February 24, 2023, 4:04pm UTC](https://discourse.julialang.org/t/data-exchange-between-julia-sessions-same-or-different-machines/38157/11 "2023-02-24T16:04:35Z")
Thanks for creating SimpleIPC.jl. I am having problems calling remote process user functions (versus julia-native functions), for example, using ipc\_eval. Can you provide some guidance?