Is there a limit on the number of @async tasks that can be scheduled?

General Usage
1

Hi,

I am using the HTTP.jl module to create multiple websocket connections and process data from them with @async. In the following program I see only 8 tasks ever succeeding. All the connections are going to be up longterm and they serve data at varying rates.

Using Julia 1.6.0
OSs tested: Windows 10 Home, WSL2, Ubuntu 20.04

asynctest.jl

using JSON3, HTTP

const REST_URL = "https://api.pro.coinbase.com"
const WS_URL = "wss://ws-feed.pro.coinbase.com"
const PRODUCTS_URL = REST_URL * "/products"

function get_products()
    products = HTTP.request("GET", PRODUCTS_URL)
    prod_list = JSON3.read(products.body)
    return prod_list
end

prod_list = get_products()
println("Num Symbols:", size(prod_list))

global num_messages = 0
global num_subscribed = 0
@sync for prod in prod_list
    sleep(1)
    symbol = prod.id
    println(symbol)
    println("num_subscribed:", num_subscribed)
    t = @async HTTP.WebSockets.open(WS_URL) do ws
        global num_messages
        global num_subscribed
        subscribe = """{"type":"subscribe", "product_ids":["$symbol"], "channels": ["full"]}""" 
        write(ws, subscribe)
        num_subscribed += 1
        println("async num_subscribed:", num_subscribed)
        while !eof(ws)
            num_messages += 1
            msg = readavailable(ws)
            if (num_messages % 10000) == 0
                println("num_messages:", num_messages)
            end
        end
    end
end

The following output is seen
output

C:\testprogs\julia>julia asynctest.jl
Num Symbols:(162,)
EOS-BTC
num_subscribed:0
MATIC-USD
num_subscribed:0
async num_subscribed:1
async num_subscribed:2
SUSHI-USD
num_subscribed:2
async num_subscribed:3
LINK-USD
num_subscribed:3
async num_subscribed:4
XTZ-BTC
num_subscribed:4
async num_subscribed:5
BCH-GBP
num_subscribed:5
async num_subscribed:6
LRC-BTC
num_subscribed:6
async num_subscribed:7
REN-BTC
num_subscribed:7
async num_subscribed:8
BNT-BTC
num_subscribed:8
YFI-BTC
num_subscribed:8
COMP-BTC
num_subscribed:8
NMR-USD
num_subscribed:8

The async_num_subscribed never goes above 8. I am fairly new to julia and maybe i am making some elementary errors. Looking for some guidance.

Bharat

2

From https://juliaweb.github.io/HTTP.jl/stable/public_interface/#Requests, the part where keyword arguments are described, there is the default limit:

connection_limit = 8 , number of concurrent connections to each host:port.

You can raise it in your code with e.g.

HTTP.WebSockets.open(WS_URL; connection_limit = 10)

But I have no idea what a reasonable max number could be.

3

Thanks for the quick response, that seems to be it. i raised the limit to 1024 (way beyond my requirements), no issue at all handling a few thousand messages/sec with fairly low cpu usage and ~30-50mbps transfer rate.

Bharat

1 Like