Asyncfilter never ends

Hello guys! I’m working on an asynchronous implementation of filter, but it never finishes. The code below is for testing a list of 950 proxies to see which ones are still working and if the response time was less than 5 seconds. It works fine for up to 200 proxies, but for the 950 it never runs out. Any hints as to what it could be? Is there any way to optimize this?

const conf = (readtimeout=5, retry=false)
url = ""
proxies = CSV.File("proxies.csv") |> Tables.matrix |> vec

function test(proxy)
    resp = ""
        resp = HTTP.get(url, proxy="http://$proxy"; conf...).body |> String |> JSON.Parser.parse |> x -> x["ip"]
    if resp != ""
        println(threadid(), " ok   ", proxy)
        return true
        println(threadid(), " fail ", proxy)
        return false

function asyncfilter(f, i)
    if length(i) >= 2
        head = i[1]
        tail = i[2:end]
        fun = @async f(head)
        recursion = @async asyncfilter(f, tail)
        if fetch(@sync fun)
            return [head, fetch(@sync recursion)...]
            return [fetch(@sync recursion)...]
        return i

result = asyncfilter(test, proxies)

I don’t think that’s how @sync works. It has to enclose the @async calls syntactically to wait on them. fetch also already blocks & waits on the return value of the task, so the @sync is superfluous.

That aside, this will be VERY inefficient, due to creating lots and lots of intermediary arrays and splatting them, as well as creating a copy on slicing and more tasks than necessary due to the recursion - it’s probably better to just asyncmap your test function.

The problem continues in asyncmap.

It’s not really possible to further help debug this without your proxies.csv file - what code exactly are you running with asyncmap? The same test function?



what code exactly are you running with asyncmap? The same test function?


result = asyncmap(test, proxies)

Could be rate limiting on the API?

1 Like

This is not the cause because a 5 second timeout has been set. So if one request fails, it should move on to the next one.

const conf = (readtimeout=5, retry=false)

this problem was a bug in the HTTP.jl package.

1 Like