Isdir in Windows 67 times slower than Linux

I need to list a large number of files and parse their names. My function works but the performance in Windows is 67 times slower despite running on a more powerful computer. The number of files is unknown and the user can select the recursion depth so I have to separate files from folders. The slow point appears to be with isdir. The files are on a shared network drive and both computers are accessing the same data.

Is there a way to speed this up?

Linux

julia> using BenchmarkTools

julia> @btime isdir.(readdir("/home/user/data/2022/Q4",join=true))
  103.151 ms (17677 allocations: 2.06 MiB)
4416-element BitVector:

Windows

julia> using BenchmarkTools

julia> @btime isdir.(readdir("D:\\data\\2022\\Q4",join=true))
  6.993 s (79501 allocations: 5.36 MiB)
4416-element BitVector:

Linux Computer Info

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, haswell)
  Threads: 4 on 4 virtual cores

Windows Computer Info

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65e (2023-01-08 06:45 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 7 on 16 virtual cores
Environment:
  JULIA_NUM_THREADS = 7

Could you time the stat function on both platforms, please?

using BenchmarkTools
@btime stat("/some/path/to/your/directory")

When I profiled my function the calling function that took so much time was isdir but the lowest level function was indeed stat. The overhead in Windows was enormous but minor in Linux.

Linux

214╎    ╎    ╎    ╎    ╎    ╎ 214  @Base/stat.jl:150; stat(path::String)

Windows

74712╎    ╎    ╎    ╎    ╎    ╎ 74717 @Base\stat.jl:150; stat(path::String)

Linux

julia> @btime stat("/home/user/data/2022/Q4")
  2.318 μs (1 allocation: 224 bytes)
StatStruct for "/home/user/data/2022/Q4"
   size: 0 bytes
 device: 73
  inode: 258212007
   mode: 0o040755 (drwxr-xr-x)
  nlink: 2
    uid: 1000 (user)
    gid: 1000 (user)
   rdev: 0
  blksz: 1048576
 blocks: 0
  mtime: 2023-01-09T14:32:08+0100 (71 days ago)
  ctime: 2023-01-09T14:32:08+0100 (71 days ago)

julia> @btime stat.(readdir("/home/user/data/2022/Q4",join=true))
  99.829 ms (17676 allocations: 2.49 MiB)
4416-element Vector{Base.Filesystem.StatStruct}:

Windows

julia> @btime stat("D:\\data\\2022\\Q4")
  1.187 ms (1 allocation: 192 bytes)
StatStruct for "D:\\data\\2022\\Q4"
   size: 0 bytes
 device: 1866537516
  inode: 258212007
   mode: 0o040666 (drw-rw-rw-)
  nlink: 1
    uid: 0
    gid: 0
   rdev: 0
  blksz: 4096
 blocks: 0
  mtime:  (71 days ago)
  ctime:  (71 days ago)

julia> @btime stat.(readdir("D:\\data\\2022\\Q4",join=true))
  7.018 s (79500 allocations: 5.80 MiB)
4416-element Vector{Base.Filesystem.StatStruct}:

How large?

75,331 at the moment

Somewhat old comment but potentially still relevant:

From there:

Linux has a top-level directory entry cache that means that certain queries (most notably stat calls) can be serviced without calling into the file system at all once an item is in the cache. Windows has no such cache, and leaves much more up to the file systems. A Win32 path like C:\dir\file gets translated to an NT path like ??\C:\dir\file, where ??\C: is a symlink in Object Manager to a device object like \Device\HarddiskVolume4. Once such a device object is encountered, the entire remainder of the path is just passed to the file system, which is very different to the centralized path parsing that VFS does in Linux.

Performance impacts in Julia have previously been discussed here:

2 Likes

Thanks for the info. I guess I’ll just have to live with it. Two minutes is still tolerable. Sadly, I won’t be able to convince IT switch it to Linux.

WSL2 might be your friend in that case.

Or there might be a way to pre-filter the files before passing them to isdir? Maybe some files have extensions that you could use to exclude immediately without calling isdir?

I thought WSL2 needs administrator access to install. The speed improvements were also only OK. Other people have to use my code as well so it’s not something I could easily scale.

@barucden, I have considered that as well. I wanted to check the forum for a more elegant solution first though. It will probably be the most practical solution.

Check also this package: ScanDir.jl

On my small Windows 11 laptop, it ran 20x faster on a folder with 75,331 dummy files.

NB:
I have benchmarked the command:
isdir.(scandir(path))

3 Likes

Network drives with Windows are always a problem.

Web search “windows network drives slow” for various tips to speed them up.

I like this blog:

… because he goes over the various solutions that have worked for him over the past 5 years.

It may be the case for someone else but not for me. The network connection is otherwise very fast. There are no noticeable delays when opening a file via Explorer.

The data folders in my case are on a RAID 5 Samba share on a Linux server (only Linux sysadmin in IT) via a 1GBit/s fiberoptic connection.

Note that the actual call for Windows probably goes through libuv here:

Needs more research but I think Julia is using stat or its equivalent on Windows.

isdir(path) calls isdir(stat(path))

isdir(stat) is defined here:

stat calls jl_stat

jl_stat calls uv_fs_stat

uv_fs_stat calls ``fs__stat_handle`

fs__stat_handle

Have you considered using walkdir for this? Scandir.walkdir does sound promising.

In the end I did the workaround suggested by barucden. I limited the isdir checks to paths without extensions. Performance is nearly identical with Linux now even though there may be the rare edge case where a folder looks like it has an extension. Our file organization isn’t that bad so it should never be a problem.

I now feel obligated to suggest ScanDir.jl too :slight_smile: I did not now that package but it looks that it avoids repeated calls to stat, so it should solve the problem without any compromises.