# How to Collect Data from a Async Call?

**URL:** <https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825>\
**Category:** General Usage\
**Tags:** distributed, http\
**Created:** [November 26, 2020, 4:30pm UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825 "2020-11-26T16:30:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![DataPsycho](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datapsycho/32/16891_2.png) [@DataPsycho](https://discourse.julialang.org/u/DataPsycho)\
**Post date:** [November 26, 2020, 4:30pm UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/1 "2020-11-26T16:30:22Z")

</div>

Hi, I have build a application where I have a function which collect data from a API call then store it to a Array as follows:

```julia
using DotEnv, HTTP, JSON3

DotEnv.config();

# Function to collect data
function collect_data(method, url, token, body)
    r = HTTP.request(
        method, 
        url,
        ["Content-Type" => "application/json", "X-Token" => ENV["HERA_SECRET"]],
        JSON3.write(body),
        require_ssl_verification = false
    )
    if isequal(r.status, 200)
        try
            return JSON3.read(r.body)["data"]
        catch err
            println("Error while parsing: $err")
        end
    end
    return Dict()
end

# A preallocated array
linear_data = Array{Any}(undef, 1000);

# linear call to API
for i in 1:1000
    linear_data[i] = collect_data("POST", URL, ENV["HERA_SECRET"], params);
end

```

But I want to convert that in to Async call. But to use Distributed jl probably I need a shared array. But shared array do not support the type Any. My plan was to create an array

```julia
using Distributed, SharedArrays

# Add Processes
addprocs(n) 

# Shared array does not support Any type
non_linear_data = SharedArray{Any}(1000)

@sync @distributed for i in 1:1000
    non_linear_data[i] = collect_data("POST", URL, ENV["HERA_SECRET"], params);
end

```

But I am stuck at the distributed part. Is there any better way to resolve that? In Go that would be easy to create a Bidirectional Channel and feed the result into the Channel and later collect the data from the Channel into a slice.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [November 26, 2020, 8:06pm UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/2 "2020-11-26T20:06:50Z")

</div>

FYI there’s [`Transducers.dcollect`](https://juliafolds.github.io/Transducers.jl/dev/reference/manual/#Transducers.dcollect) that implements `collect`-like API based on Distributed.jl. It’d be something like

```julia
linear_data = dcollect(collect_data("POST", URL, ENV["HERA_SECRET"], params) for i in 1:1000)

```

You can also try thread-based version with `tcollect`.

---

<div class="post-metadata">

**Author:** ![DataPsycho](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datapsycho/32/16891_2.png) [@DataPsycho](https://discourse.julialang.org/u/DataPsycho)\
**Post date:** [November 27, 2020, 1:46pm UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/3 "2020-11-27T13:46:12Z")

</div>

Hi thanks for the solution, though I have not tried it out and I was looking for more detail how to do that but not any magic solution. Because I wanted to learn that In general how to solve that problem with @distribute or channel in Julia. But could not find any well described solution. If you know some document I can have a look to be able to learn that?

---

<div class="post-metadata">

**Author:** ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)\
**Post date:** [November 27, 2020, 3:23pm UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/4 "2020-11-27T15:23:44Z")

</div>

You should be able to do this even as a single-processor asynchronous task. For example:

```julia
julia> linear_data = zeros(Int, 10);

# Linear
julia> @time for i in 1:10
           linear_data[i] = (sleep(1); i);
       end
 10.019392 seconds (50 allocations: 1.406 KiB)

# Asynchronous
julia> @time @sync for i in 1:10
           @async linear_data[i] = (sleep(1); i);
       end
  1.014280 seconds (1.78 k allocations: 95.128 KiB)

julia> linear_data
10-element Array{Int64,1}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

```

Of course you may distribute this to speed it up even more.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [November 28, 2020, 4:25am UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/5 "2020-11-28T04:25:02Z")

</div>

If you are curious about implementation strategies of robust and composable parallel `collect`, I explained an implementation strategy I use in Transducer’s tutorial: [Tutorial: Parallelism · Transducers.jl](https://juliafolds.github.io/Transducers.jl/dev/tutorials/tutorial_parallel/#tutorial-parallel-collect)

---

<div class="post-metadata">

**Author:** ![DataPsycho](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datapsycho/32/16891_2.png) [@DataPsycho](https://discourse.julialang.org/u/DataPsycho)\
**Post date:** [November 28, 2020, 5:46am UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825/6 "2020-11-28T05:46:54Z")

</div>

Cool. That’s awesome. I am reading it.  
Thanks