How to set max number of concurrent request with @sync and @async?

General Usage
1

Hi Julians!

@sync and @async is a nice combination to perform concurrent IO, such as reading files from the cloud. However, there seems to be a maximum number of concurrent request to the google cloud storage. Is there any way to set maximum concurrent requests?

Here is a code example:

@sync begin 
        for (blockID, chunkGlobalRange, globalRange, rangeInChunk, rangeInBuffer) in baIter
            @async begin
                       do the request
            end
        end
end
2

anyone could help?

3

Take a look at channels. Basically, you can create a single producer Task that generates new jobs and puts them to a channel and a set of worker Tasks that take jobs from the channel and execute one by one. You can control the number of concurrent requests to Google cloud storage either by starting that many tasks or by setting size of the channel.

4

thanks! this is a good point!