`ELOOP` when `walkdir`

General Usage
1

I modified the source code of walkdir a little bit to skip some irrelevant dirs. However, the code produces ELOOP error about when 2000 directories are processed. I do not understand the error. Is it because there are too many channels that used up the available sockets? (I have checked the dirs, actually there are almost no symbolic links). Does anyone know the direction of investigation?

ERROR: LoadError: IOError: stat: too many symbolic links encountered (ELOOP)                                                                   
Stacktrace:                                                                                                                                    
 [1] check_channel_state at ./channels.jl:125 [inlined]                                                                                        
 [2] take_unbuffered(::Channel{Any}) at ./channels.jl:327                                                                                      
 [3] take! at ./channels.jl:315 [inlined]                                                                                                      
 [4] iterate(::Channel{Any}, ::Nothing) at ./channels.jl:395                                                                                   
 [5] top-level scope at /mnt/dev/zzlistdir.jl:86 [inlined]                                                         
 [6] top-level scope at ./none:0                                                                                                               
 [7] include at ./boot.jl:326 [inlined]                                                                                                        
 [8] include_relative(::Module, ::String) at ./loading.jl:1038                                                                                 
 [9] include(::Module, ::String) at ./sysimg.jl:29                                                                                             
 [10] exec_options(::Base.JLOptions) at ./client.jl:267                                                                                        
 [11] _start() at ./client.jl:436

The source code is here. The main function is walk


STOPS_IN = ["egg-info", "backup", "torch", "tensorflow", "python", "matlab"]
PASS_EXT = [".jpg", ".png", ".avi", ".jpeg", ".tif", ".tiff", ".mp4", ".mp3", ".bmp", ".npy", ".gif", ".txt"]
FILE_EXT = [".zip", ".tar", ".tar.gz", ".rar"]

iskeepdir(d) = !any(x -> occursin(x, lowercase(d)), STOPS_IN)
iskeepfile(f) = any(x -> endswith(lowercase(f), x), FILE_EXT) 
ispasscheck(f) = any(x -> endswith(lowercase(f), x), PASS_EXT)

function walk(root)
    content = nothing
    try
        content = readdir(root)
    catch err
        c = Channel(0)
        close(c)
        return c
    end
    dirs = Vector{eltype(content)}()
    files = Vector{eltype(content)}()
    nd = 0
    for name in content
        ispasscheck(name) && continue
        if isdir(joinpath(root, name))
            iskeepdir(name) && push!(dirs, name)
        else
            iskeepfile(name) && push!(files, name)
        end
    end

    function _it(chnl)
        put!(chnl, (root, dirs, files))
        for dir in dirs
            path = joinpath(root, dir)
            if !islink(path)
                for (root_l, dirs_l, files_l) in walk(path)
                    put!(chnl, (root_l, dirs_l, files_l))
                end
            end
        end
    end

    return Channel(_it)
end
2

Sounds like you have two symbolic links pointing at each other, or multiple in bigger loop?

From: path_resolution(7) - Linux manual page

If the component is found and is a symbolic link (symlink), we first
resolve this symbolic link (with the current lookup directory as
starting lookup directory). Upon error, that error is returned. If
the result is not a directory, an ENOTDIR error is returned. If the
resolution of the symlink is successful and returns a directory, we
set the current lookup directory to that directory, and go to the
next component. Note that the resolution process here can involve
recursion if the prefix (‘dirname’) component of a pathname contains
a filename that is a symbolic link that resolves to a directory
(where the prefix component of that directory may contain a symbolic
link, and so on). In order to protect the kernel against stack
overflow, and also to protect against denial of service, there are
limits on the maximum recursion depth, and on the maximum number of
symbolic links followed. An ELOOP error is returned when the maximum
is exceeded (“Too many levels of symbolic links”).