How to write a concurrent directory walker

New to Julia
1

I’ve created this function:

function getfiles(root, suffix=".txt")
    goodfiles = String[]
    for (root, dirs, files) in walkdir(root)
	for file in files
	    if endswith(lowercase(file), suffix)
		push!(goodfiles, joinpath(root, file))
	    end
	end
    end
    goodfiles
end

It works fine. However, what I’m aiming to do is this:

results = pmap(expensiveFunction(filename), getfiles("."))

Now, as it stands this should ‘just work’; however, I’d really like the getfiles() to be a generator because it is possible it will have to work with directories with 1000s of files.

Any advice on how to change getfiles() into a generator?

Thanks.

1 Like
2

I’ve now got it working concurrently using the first example from http://docs.julialang.org/en/latest/manual/parallel-computing.html#Parallel-Computing-1
(The example that shows the use of RemoteChannels for jobs & results.)