# Module scope and distributed computing

**URL:** <https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014>\
**Category:** General Usage\
**Tags:** distributed\
**Created:** [March 1, 2024, 1:31pm UTC](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014 "2024-03-01T13:31:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lgo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lgo/32/48751_2.png) [@lgo](https://discourse.julialang.org/u/lgo)\
**Post date:** [March 1, 2024, 1:31pm UTC](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014/1 "2024-03-01T13:31:53Z")

</div>

I have a rather extensive and inaccessible algorithm that deploys distributed computing. To make it more accessible, I want to wrap as much of the algorithm into the functions of my package.

I manage to initialize my variables on a worker but cannot access them for subsequent processing (see example below). I understand that `x` is not defined in the right scope but is there a smart way to fix this?

```
using Distributed
addprocs(1) 

@everywhere module myPackage
	using Distributed

	# function to create variable on worker
	function createData()
		@everywhere 2 begin
			x = rand(10)
		end
	end

	# function to execute on worker
	function processData(y)
		return x .* y
	end

	# function to be wrapped
	function runProcessData(y)
		return fetch(@spawnat 2 processData(x,y))
	end

	export createData, runProcessData
end

@everywhere using .myPackage
createData()
fetch(@spawnat 2 eval(:x))
runProcessData(10)

```

The only workaround for me was to add `x` as an input to `runProcessData`, fetch it from the worker, and pass it again, which is extremely inefficient in my case.

---

<div class="post-metadata">

**Author:** ![savq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/savq/32/22063_2.png) [@savq](https://discourse.julialang.org/u/savq)\
**Post date:** [March 3, 2024, 6:23am UTC](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014/2 "2024-03-03T06:23:02Z")

</div>

The `@everywhere` macro executes an expression in `Main`, so the `x` in `createData` exists in the `Main` module of process 2, But `processData` assumes `x` exists in `myPackage`.

---

<div class="post-metadata">

**Author:** ![lgo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lgo/32/48751_2.png) [@lgo](https://discourse.julialang.org/u/lgo)\
**Post date:** [March 3, 2024, 3:49pm UTC](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014/3 "2024-03-03T15:49:13Z")

</div>

Thanks for the reply! I think I got that part, though. My train of thought was that if I could fetch `x` from the worker and pass it again when calling `runProcessData` (see below), there should also be a way to avoid this inefficient communication.

```
using Distributed
addprocs(1) 

@everywhere module myPackage
	using Distributed

	# function to create variable on worker
	function createData()
		@everywhere 2 begin
			x = rand(10)
		end
	end

	# function to execute on worker
	function processData(x,y)
		return x .* y
	end

	# function to be wrapped
	function runProcessData(x,y)
		return fetch(@spawnat 2 processData(x,y))
	end

	export createData, runProcessData
end

@everywhere using .myPackage
createData()
runProcessData(fetch(@spawnat 2 eval(:x)),10)

```

As a solution, I defined a function to spawn workers outside the package and now pass the function as an argument.

```
using Distributed
addprocs(1) 

@everywhere module myPackage
    using Distributed

	# function to create variable on worker
	function createData()
		@everywhere 2 begin
			x = rand(10)
		end
	end

	# function to execute on worker
	function processData(x,y)
		return x .* y
	end

	# function to be wrapped
	function runProcessData(y::Float64,spawn::Function)
		fut_obj::Future = spawn(2,y)
		out_arr::Vector{Float64} = fetch(fut_obj)
		return out_arr
	end

	export createData, processData, runProcessData
end

@everywhere begin
	using .myPackage	
	spawnProcessData(i::Int64, y::Float64) = @spawnat i processData(x,y)
end
createData()
runProcessData(10.0,spawnProcessData)

```