# Isdir in Windows 67 times slower than Linux

**URL:** <https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436>\
**Category:** General Usage\
**Tags:** windows, linux, filesystem\
**Created:** [March 22, 2023, 9:26am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436 "2023-03-22T09:26:17Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 22, 2023, 9:26am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/1 "2023-03-22T09:26:17Z")

</div>

I need to list a large number of files and parse their names. My function works but the performance in Windows is 67 times slower despite running on a more powerful computer. The number of files is unknown and the user can select the recursion depth so I have to separate files from folders. The slow point appears to be with `isdir`. The files are on a shared network drive and both computers are accessing the same data.

Is there a way to speed this up?

Linux

```julia
julia> using BenchmarkTools

julia> @btime isdir.(readdir("/home/user/data/2022/Q4",join=true))
  103.151 ms (17677 allocations: 2.06 MiB)
4416-element BitVector:

```

Windows

```julia
julia> using BenchmarkTools

julia> @btime isdir.(readdir("D:\\data\\2022\\Q4",join=true))
  6.993 s (79501 allocations: 5.36 MiB)
4416-element BitVector:

```

Linux Computer Info

```julia
julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, haswell)
  Threads: 4 on 4 virtual cores

```

Windows Computer Info

```julia
julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65e (2023-01-08 06:45 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 7 on 16 virtual cores
Environment:
  JULIA_NUM_THREADS = 7

```

---

<div class="post-metadata">

**Author:** ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)\
**Post date:** [March 22, 2023, 9:37am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/2 "2023-03-22T09:37:42Z")

</div>

Could you time the `stat` function on both platforms, please?

```julia
using BenchmarkTools
@btime stat("/some/path/to/your/directory")

```

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 22, 2023, 9:50am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/3 "2023-03-22T09:50:11Z")

</div>

When I profiled my function the calling function that took so much time was `isdir` but the lowest level function was indeed `stat`. The overhead in Windows was enormous but minor in Linux.

Linux

```julia
214╎ ╎ ╎ ╎ ╎ ╎ 214 @Base/stat.jl:150; stat(path::String)

```

Windows

```julia
74712╎ ╎ ╎ ╎ ╎ ╎ 74717 @Base\stat.jl:150; stat(path::String)

```

Linux

```julia
julia> @btime stat("/home/user/data/2022/Q4")
  2.318 μs (1 allocation: 224 bytes)
StatStruct for "/home/user/data/2022/Q4"
   size: 0 bytes
 device: 73
  inode: 258212007
   mode: 0o040755 (drwxr-xr-x)
  nlink: 2
    uid: 1000 (user)
    gid: 1000 (user)
   rdev: 0
  blksz: 1048576
 blocks: 0
  mtime: 2023-01-09T14:32:08+0100 (71 days ago)
  ctime: 2023-01-09T14:32:08+0100 (71 days ago)

julia> @btime stat.(readdir("/home/user/data/2022/Q4",join=true))
  99.829 ms (17676 allocations: 2.49 MiB)
4416-element Vector{Base.Filesystem.StatStruct}:

```

Windows

```julia
julia> @btime stat("D:\\data\\2022\\Q4")
  1.187 ms (1 allocation: 192 bytes)
StatStruct for "D:\\data\\2022\\Q4"
   size: 0 bytes
 device: 1866537516
  inode: 258212007
   mode: 0o040666 (drw-rw-rw-)
  nlink: 1
    uid: 0
    gid: 0
   rdev: 0
  blksz: 4096
 blocks: 0
  mtime: (71 days ago)
  ctime: (71 days ago)

julia> @btime stat.(readdir("D:\\data\\2022\\Q4",join=true))
  7.018 s (79500 allocations: 5.80 MiB)
4416-element Vector{Base.Filesystem.StatStruct}:

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 22, 2023, 10:03am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/4 "2023-03-22T10:03:48Z")

</div>

> [@Jeremy](#):
>
> large number of files

How large?

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 22, 2023, 10:04am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/5 "2023-03-22T10:04:55Z")

</div>

75,331 at the moment

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [March 22, 2023, 10:11am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/6 "2023-03-22T10:11:30Z")

</div>

Somewhat old comment but potentially still relevant:

> <https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829>
>
> \# A brief description
> 
> As a Symfony developer, it's always been hard to get a st…able/fast development environment. My current setup is a Ubuntu running under VirtualBox (using vagrant). While page generation is fast, my IDE accesses my PHP files through SMB, which is really (sometimes horribly) slow.
> I'm now trying to use WSL to improve all of this. However, I'm having a major performance issue when using \`/mnt/\*\` folders.
> If I set up a Symfony project under \`/mnt/c\`, it is really slow. If I set it up under \`/home/mikael\`, it is very fast.
> \# Expected results
> 
> Drives mounted under /mnt should be as fast a any other folder.
> \# Actual results
> 
> With a new Symfony 3.1.3 project, under \`/home/mikael\` takes between 100ms and 130ms to generate the home page.
> The same project under \`/mnt/c/\` takes between 1200ms and 1500ms.
> \# Your Windows build number
> 
> 10.0.14393.51
> \# Steps / commands required to reproduce the error
> 
> \`\`\`
> \# Install PHP5
> $ sudo apt-get install -y php5 php5-json
> 
> \# Download Symfony installer
> $ sudo curl -LsS https://symfony.com/installer -o /usr/local/bin/symfony
> $ sudo chmod a+x /usr/local/bin/symfony
> 
> \# Download Symfony
> cd
> symfony new symfony\_test
> 
> \# Start Symfony
> cd symfony\_test
> php bin/console server:run
> \`\`\`
> 
> Open your browser and go to http://127.0.0.1:8000/.
> Once the page is loaded, refresh it (on first request, Symfony had to generate its cache).
> Generation time is displayed on the bottom left
> !\[Image\](http://glados.aperture.fr.nf/up/load/2016/08/2016-08-12\_01-18-38.png)
> 
> You can then do the same under \`/mnt/c/\`
> 
> \`\`\`
> cd /mnt/c/
> symfony new symfony\_test
> cd symfony\_test
> php bin/console server:run
> \`\`\`
> \# Additional information
> 
> I've added my dev folders as excluded folders in Windows Defender, as well as %LOCALAPPDATA%\\lxss.
> I've tried having my project in \`~\` and pointing my IDE to %LOCALAPPDATA%\\lxss\\home\\mikael\\ but as I've later read, there is no supported way of editing WSL files.
> WSL is installed in its default location under C (no strange junction or symlink), which is a healthy SSD.
> My computer is attached to a domain, if this might have any influence.

From there:

> Linux has a top-level directory entry cache that means that certain queries (most notably stat calls) can be serviced without calling into the file system at all once an item is in the cache. Windows has no such cache, and leaves much more up to the file systems. A Win32 path like C:\dir\file gets translated to an NT path like ??\C:\dir\file, where ??\C: is a symlink in Object Manager to a device object like \Device\HarddiskVolume4. Once such a device object is encountered, the entire remainder of the path is just passed to the file system, which is very different to the centralized path parsing that VFS does in Linux.

Performance impacts in Julia have previously been discussed here:

> [@Package load speed: Windows vs Linux](https://discourse.julialang.org/t/package-load-speed-windows-vs-linux/93039/14):
>
> My general sense is that Windows performance may get less attention than Linux performance. It seems much more likely that I’m the first one to recognize an issue on Windows. This could be merely because many of the developers and users of Julia tend to primarily work on Linux. The exchange that Kristoffer linked above revealed an issue that was affecting Windows users more than Linux users. stat is regularly used when accessing a file since it determines basic information such as does the file…

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 22, 2023, 10:18am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/7 "2023-03-22T10:18:45Z")

</div>

Thanks for the info. I guess I’ll just have to live with it. Two minutes is still tolerable. Sadly, I won’t be able to convince IT switch it to Linux.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [March 22, 2023, 10:19am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/8 "2023-03-22T10:19:40Z")

</div>

WSL2 might be your friend in that case.

---

<div class="post-metadata">

**Author:** ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)\
**Post date:** [March 22, 2023, 10:22am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/9 "2023-03-22T10:22:07Z")

</div>

Or there might be a way to pre-filter the files before passing them to `isdir`? Maybe some files have extensions that you could use to exclude immediately without calling `isdir`?

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 22, 2023, 10:32am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/10 "2023-03-22T10:32:02Z")

</div>

I thought WSL2 needs administrator access to install. The speed improvements were also only OK. Other people have to use my code as well so it’s not something I could easily scale.

@barucden, I have considered that as well. I wanted to check the forum for a more elegant solution first though. It will probably be the most practical solution.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [March 22, 2023, 10:37am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/11 "2023-03-22T10:37:36Z")

</div>

Check also this package: [ScanDir.jl](https://docs.juliahub.com/ScanDir/BvP4J/0.4.2/)

On my small Windows 11 laptop, it ran 20x faster on a folder with 75,331 dummy files.

**NB:**  
I have benchmarked the command:  
`isdir.(scandir(path))`

---

<div class="post-metadata">

**Author:** ![blackeneth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blackeneth/32/10353_2.png) [@blackeneth](https://discourse.julialang.org/u/blackeneth)\
**Post date:** [March 22, 2023, 10:07pm UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/12 "2023-03-22T22:07:11Z")

</div>

Network drives with Windows are always a problem.

Web search “windows network drives slow” for various tips to speed them up.

I like this blog:

> **[Slow Access to Network Shares on Windows Computers | Alexander's Blog](https://www.zubairalexander.com/blog/slow-access-to-network-shares-on-windows-computers/)**
>
> In 2017, I wrote an article What to Do When Creating Folders or Accessing Files on Remote Computer is Very Slow. In 2018, I wrote another article titled How to Fix Slow Access to Network Shares. Because the information addressed the same topic, but...

… because he goes over the various solutions that have worked for him over the past _5 years._

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 23, 2023, 7:48am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/13 "2023-03-23T07:48:04Z")

</div>

It may be the case for someone else but not for me. The network connection is otherwise very fast. There are no noticeable delays when opening a file via Explorer.

The data folders in my case are on a RAID 5 Samba share on a Linux server (only Linux sysadmin in IT) via a 1GBit/s fiberoptic connection.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [March 23, 2023, 8:38am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/14 "2023-03-23T08:38:41Z")

</div>

Note that the actual call for Windows probably goes through libuv here:

> <https://github.com/JuliaLang/libuv/blob/fa7058b865e3c4a5a9c9ff511ed3e589ce817a85/src/win/fs.c#L1565>

Needs more research but I think Julia is using `stat` or its equivalent on Windows.

`isdir(path)` calls `isdir(stat(path))`

> <https://github.com/JuliaLang/julia/blob/489d076452130c718c7d77b157b0d503bfc31602/base/stat.jl#L461>

`isdir(stat)` is defined here:

> <https://github.com/JuliaLang/julia/blob/489d076452130c718c7d77b157b0d503bfc31602/base/stat.jl#L371>

`stat` calls `jl_stat`

> <https://github.com/JuliaLang/julia/blob/489d076452130c718c7d77b157b0d503bfc31602/base/stat.jl#L163>

`jl_stat` calls `uv_fs_stat`

> <https://github.com/JuliaLang/julia/blob/489d076452130c718c7d77b157b0d503bfc31602/src/sys.c#L119>

`uv_fs_stat` calls ``fs\_\_stat\_handle`

> <https://github.com/JuliaLang/libuv/blob/fa7058b865e3c4a5a9c9ff511ed3e589ce817a85/src/win/fs.c#L3631>

`fs__stat_handle`

> <https://github.com/JuliaLang/libuv/blob/fa7058b865e3c4a5a9c9ff511ed3e589ce817a85/src/win/fs.c#L1685>

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [March 23, 2023, 9:21am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/15 "2023-03-23T09:21:33Z")

</div>

Have you considered using `walkdir` for this? `Scandir.walkdir` does sound promising.

---

<div class="post-metadata">

**Author:** ![Jeremy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremy/32/295_2.png) [@Jeremy](https://discourse.julialang.org/u/Jeremy)\
**Post date:** [March 23, 2023, 11:43am UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/16 "2023-03-23T11:43:51Z")

</div>

In the end I did the workaround suggested by barucden. I limited the `isdir` checks to paths without extensions. Performance is nearly identical with Linux now even though there may be the rare edge case where a folder looks like it has an extension. Our file organization isn’t that bad so it should never be a problem.

---

<div class="post-metadata">

**Author:** ![barucden](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barucden/32/26154_2.png) [@barucden](https://discourse.julialang.org/u/barucden)\
**Post date:** [March 23, 2023, 12:31pm UTC](https://discourse.julialang.org/t/isdir-in-windows-67-times-slower-than-linux/96436/17 "2023-03-23T12:31:09Z")

</div>

I now feel obligated to suggest ScanDir.jl too 🙂 I did not now that package but it looks that it avoids repeated calls to `stat`, so it should solve the problem without any compromises.