Rdir: search recursive for files with a given name pattern

New to Julia
,
3

I think you are after glob patterns?

https://github.com/vtjnash/Glob.jl

The following should do it:

using Glob

readdir("*.jl", dir)
3 Likes
4

Oh we wrote at the same time :smiley:

5

I do not know why, but on my computer it does not work, neither on Linux
Julia v1.4 nor on MS Windows 10, Julia v1.7.1.
The error message is in both cases:

no method matching readdir(::String, ::String)

1 Like
6

I think you need

readdir(glob"*.jl", dir)

but it doesn’t look like Glob.jl acts recursively. So it doesn’t full solve your problem (though it is easy to write a recursive function).

1 Like
7

Confirmed! :slight_smile: with glob"string" it works.
And you are right! Unfortunately, it does not work recursively.

8

Interesting, I remember using it recursively in the past, maybe I implemented the recursion myself? I don’t remember exactly.

9

You can always provide a glob pattern to look into subdirectories though like glob"/*/*.jl" to look into subdirectories if I am not mistaken.

In any case it is worth opening an issue in the repository with a feature request.

10

This throws an error on my machine:

LoadError: Glob pattern cannot be empty or start with a / character

where is the trick? Does glob"/*/*.jl" only work as an argument to a function?

11

@ellocco please read the README of the package more carefully, I am just typing random snippets of code here from a mobile device. The character / is special apparently and cannot be used at the start of the pattern. Have you tried omitting the character as the error message suggests?

12

You could use the Glob package with walkdir, for example:

import Glob
function rdir(dir::AbstractString, pat::Glob.FilenameMatch)
    result = String[]
    for (root, dirs, files) in walkdir(dir)
        append!(result, filter!(f -> occursin(pat, f), joinpath.(root, files)))
    end
    return result
end
rdir(dir::AbstractString, pat::AbstractString) = rdir(dir, Glob.FilenameMatch(pat))

Then do e.g. rdir("somedir", "*.jl").

7 Likes
13

(It might be useful to add walkdir support directly to Glob.jl. In general, it seems much better to have an iterator for this sort of thing, since a recursive directory tree can get huge.)

2 Likes
14

sorry, I was sure Glob is recursive and I was literally thinking **, sadly:
https://github.com/vtjnash/Glob.jl/issues/19

15

There is an enhanced Glob around:
Eglob
Unfortunately, the documentation does not make clear to me,
if I can specify a specific top directory as an input
parameter to the function, which specifies the starting point
of the recursive search.

16

@stevengj Thanks for your code! It is not easy to understand everything, could you be so kind and comment this line:

For me it is not easy to understand.

17

I defined two methods of the rdir function. One methodd that takes a FilenameMatch pattern, which is defined by the Glob.jl package, can be constructed with fn"...", and is needed by my implementation because that’s what Glob.jl implements occursin for. The other method, for convenience, takes a simple string pattern — it is implemented by simply converting your string to a FilenameMatch and calling the first method.

This way, you can pass either a string or a FilenameMatch to rdir. (The latter provides more options, e.g. there is an option to make it case-insensitive.)

This is a pretty common pattern in Julia.

2 Likes
18

Hi Steven,
thank you for the link to the topic of different methods for one and the same function!
That link really does widen my horizon!
Stefan

19

The proposed function with the variable type “FilenameMatch” in combination with “occursin” has the drawback that “*.lj” works, but “string*.jl” does not. Another option is the variable type “Glob.GlobMatch” in combination with “readdir()” this enables the usage of the joker char / asterisk “*” inside the search string:

function MyLib_RDir(s_dir::AbstractString, s_pat::Glob.GlobMatch)
    files_filtered = String[]
    for (root, dirs, files) in walkdir(s_dir)
        for i_files in readdir(s_pat, root)
            files_filtered = vcat(files_filtered, i_files)
        end
    end
    return files_filtered
end
# Next: add 2nd method to function "MyLib_RDir"
# https://docs.julialang.org/en/v1/manual/methods/
# purpose: convert "String"-type content into 
# "GlobMatch"-type (defined by the Glob.jl package) pattern,
# by utilizing the first methode of this function
MyLib_RDir(s_dir::AbstractString, s_pat::AbstractString) =
    MyLib_RDir(s_dir, Glob.GlobMatch(s_pat))
20

If I am not mistaken, the nice solution proposed by @stevengj will work as per your wishes by doing the minor adjustement:

append!(result, filter!(f -> occursin(pat, f), files), joinpath.(root, files))
1 Like
21

Confirmed, you are not mistaken! - @rafael.guerra Thanks!

But the initial reason for my post is still open, why does the statement “global s_files” in my initial function does the opposite from what is expected, that it is not necessary in this situation is one thing, but that it does the opposite from what I would like to achieve (that it is made global over the whole function space) is strange to me.

22

global s_files would refer to the module scope outside the rdir() function, but this variable is there empty / not defined.

Local variables defined inside a function can be used inside the inner scope of for loops, annotating them as global would have the meaning above. But I might well be wrong.