# NCDatasets: performance issues deriving fast Minimum Maximum value as too slow

**URL:** <https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468>\
**Category:** Geo\
**Tags:** performance, netcdf, ncdatasets\
**Created:** [August 21, 2024, 10:51pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468 "2024-08-21T22:51:19Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 21, 2024, 10:51pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/1 "2024-08-21T22:51:19Z")

</div>

🐢 🐢 🐢 🐢

Finding minimum and maximum of a NetCDF files due to _missing_ values is **painfully slow**. It takes me 2 minutes to perform this simple task:

```julia
"NameOutput="SoilWater"
Output_NCDatasets = NCDatasets.NCDataset(Path)
		
Data = Output_NCDatasets[NameOutput]
Pmin, Pmax = extrema(x for x ∈ skipmissing(Data) if !isnan(x))
println(Pmin,"," , Pmax)

```

Where the OUTPUT is:

```julia
OUTPUT (291 × 313 × 31)
    Datatype: Union{Missing, Float32} (Float32)
    Dimensions: lon × lat × time
    Attributes:
     _FillValue = NaN

```

Many thanks for any help you could provide to speedup the process? 😃

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [August 21, 2024, 11:06pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/2 "2024-08-21T23:06:49Z")

</div>

That’s not a very large array, so fully load it into memory first by doing `Array(Data)`, and then do `skipmissing` on the result.

I can’t know exactly because you don’t provide the file.

But the problem is likely actually that NCDatasets.jl doesn’t properly implement the DiskArrays.jl implementation for `CFVariable` so it is calling the netcdf c library for literally every single pixel, rather than loading larger chunks. (@Alexander-Barth this is an example where making `AbstractVariable` a DiskArray has direct benefits to users - iteration and generators are chunked)

Either way for data this size, just load it into memory before use, it will always be faster.

One other way to speed it up immediately is to use Rasters.jl, which should be fast because it just loads to memory - but also will be faster from disk (via `lazy=true`) because it implements DiskArrays.

```julia
rast = Raster(Path; name=:SoilWater)
Pmin, Pmax = extrema(x for x ∈ skipmissing(rast) if !isnan(x))

```

(In a few weeks with the CF standars PR it will be even faster when you can intentionally not convert to `missing` and use the native missing value in `skipmissing`, which turned out to be much faster than what Base does with `missing`)

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 21, 2024, 11:36pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/3 "2024-08-21T23:36:55Z")

</div>

Thanks for providing a solution. Your solution solves the issue and the code is now running very fast thanks to converting Data into Array.

```julia
Data = Array(Data)

```

I wrote a code to animate the output of my hydrological model by **moving a slider.** When I run in **Jupyter notebook in Visual Studio Code** the plots open in **GLMakie** but when I run it in **REPL** it opens as a png. The question how can I force it to open in Makie when I run it in Visual Studio Code when running it in the REPL?

```julia

	function HEATMAP_TIME(;Path=Path, NameOutput="q_land", Layer=1)
		GLMakie.activate!
		Makie.inline!(false) # Make sure to inline plots into Documenter output!
		
		Output_NCDatasets = NCDatasets.NCDataset(Path)
		
		Data = Output_NCDatasets[NameOutput]

		Data = Array(Data)
	
		Dimensions = length(size(Data))
	
		if Dimensions == 3
         N_Lon = size(Data)[1]
         N_Lat = size(Data)[2]
         N_Time = size(Data)[3]
	
		elseif Dimensions == 4
         N_Lon = size(Data)[1]
         N_Lat = size(Data)[2]
         N_Time = size(Data)[4]
		end

		Pmin, Pmax = extrema(x for x ∈ skipmissing(Data) if !isnan(x))
		@show Pmin, Pmax

	
		function DATA_3D_2_2D(Data; iTime=iTime, Dimensions=Dimensions, Layer=Layer)
			if Dimensions == 4
				return Data[:,:, Layer, iTime]
			elseif Dimensions == 3
				return Data[:,:, iTime]
			end
		end
		
		
		Fig = Figure(size=(Width, Height * 4.0))

		Ax_1 = Axis(Fig[1, 1], title=NameOutput, xlabelsize=xlabelSize, ylabelsize=xlabelSize, xticksize=xticksize, xgridvisible=xgridvisible, ygridvisible=xgridvisible)
		
		sg = SliderGrid(Fig[2, 1],
		(label="iTime", range=1:1:N_Time, startvalue=1),
		width=550, tellheight=true)
		
		iTime = sg.sliders[1].value
		
		Data_Time = lift((iTime) -> DATA_3D_2_2D(Data; iTime=iTime, Dimensions), iTime)
	
		Data_Plot = heatmap!(Ax_1, 1:N_Lon, 1:N_Lat, Data_Time, colorrange=(Pmin, Pmax), colormap =:hawaii50)
	
		Colorbar(Fig[1, 2], Data_Plot; label=NameOutput, width=20, ticks = Pmin:(Pmax-Pmin)/5:Pmax)
	
		Fig
	end

	HEATMAP_TIME(;Path=Path_Output, NameOutput="vwc")

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/1/e1ec44f3e5265cdd505756ec967b58f67a385a6c.png)

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [August 21, 2024, 11:39pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/4 "2024-08-21T23:39:40Z")

</div>

Not sure about Makie, I don’t use VSCode - maybe ask a separate question about that people wont find it here.

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 21, 2024, 4:18am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/5 "2024-08-21T04:18:08Z")

</div>

🐢 🐢 🐢 🐢

Thanks for providing a solution. Nevertheless I have the same problem as finding minimum and maximum is **painfully slow** when i am in jupyter notebook in Visual Studio code. It takes me 2 minutes to perform this simple task:

```julia
Pmin, Pmax = extrema(x for x ∈ skipmissing(Output) if !isnan(x))
println(Pmin,"," , Pmax)

```

Where the OUTPUT is:

```julia
OUTPUT (291 × 313 × 31)
    Datatype: Union{Missing, Float32} (Float32)
    Dimensions: lon × lat × time
    Attributes:
     _FillValue = NaN

```

Many thanks for any help you could provide to speedup the process? 😃

---

<div class="post-metadata">

**Author:** ![Qfl3x](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/qfl3x/32/16227_2.png) [@Qfl3x](https://discourse.julialang.org/u/Qfl3x)\
**Post date:** [August 21, 2024, 8:18am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/6 "2024-08-21T08:18:24Z")

</div>

@JosephPollacco  
That shouldn’t take long at all. If it’s fast when you run it in the REPL (_on the same machine_) then the problem is in Jupyter notebook somehow, it that’s the case I suggest you open another thread. If it’s also slow in the REPL, then the problem is in the machine, perhaps the memory is full or it’s busy…etc

My money is on it’s a machine problem.

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 21, 2024, 10:43pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/7 "2024-08-21T22:43:02Z")

</div>

Thanks for your help.

I have a fast machine so the problem is somwhere else. As you recommended I tested the code by running in **REPL** and the speed of execution is as slow as when I run the code in **Jupyter notebook**.

I will openup another tread to see if we can get the statistics of the OUTPUT by using directly NCDatasets.

Thanks for your help

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [August 21, 2024, 10:59pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/8 "2024-08-21T22:59:58Z")

</div>

It runs almost instantaneously (\< 0.1 seconds) for me, even in global scope:

```julia
julia> Output = Array{Union{Missing,Float32}}(rand(Float32, 291,313,31));

julia> @time Pmin, Pmax = extrema(x for x ∈ skipmissing(Output) if !isnan(x))
  0.053098 seconds (45.14 k allocations: 3.076 MiB, 57.64% compilation time)
(5.9604645f-8, 0.99999994f0)

```

Can you give a minimal working example (MWE) of code that reproduces your problem? See also [Please read: make it easier to help you](https://discourse.julialang.org/t/please-read-make-it-easier-to-help-you/14757)

PS. I’ve moved this to a new thread, since it is unrelated to the problem in the original thread. Please don’t “necropost” to ancient threads.

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 21, 2024, 11:43pm UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/9 "2024-08-21T23:43:29Z")

</div>

Thanks.  
The problem has been solved

> [@NCDatasets: performance issues deriving fast Minimum Maximum value as too slow](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/2):
>
> That’s not a very large array, so fully load it into memory first by doing Array(Data), and then do skipmissing on the result. I can’t know exactly because you don’t provide the file. But the problem is likely actually that NCDatasets.jl doesn’t properly implement the DiskArrays.jl implementation for CFVariable so it is calling the netcdf c library for literally every single pixel, rather than loading larger chunks. (@Alexander-Barth this is an example where making AbstractVariable a DiskArray…

This is by transforming  
`Data = Array(Data)`

Thanks,  
Joseph

---

<div class="post-metadata">

**Author:** ![Fliks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fliks/32/2494_2.png) [@Fliks](https://discourse.julialang.org/u/Fliks)\
**Post date:** [August 22, 2024, 9:15am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/10 "2024-08-22T09:15:23Z")

</div>

Not sure, whether this would solve the GLMakie issue in the REPL but it seems as if the GLMakie.activate! misses the brackets and is therefore not called.  
It might be that in the Jupyter notebook GLMakie is already activated and therefore it doesn’t matter, but for the REPL it does.

---

<div class="post-metadata">

**Author:** ![lupemba](https://avatars.discourse-cdn.com/v4/letter/l/e5b9ba/32.png) [@lupemba](https://discourse.julialang.org/u/lupemba)\
**Post date:** [August 22, 2024, 9:57am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/11 "2024-08-22T09:57:32Z")

</div>

> [@JosephPollacco](#):
>
> The question how can I force it to open in Makie when I run it in Visual Studio Code when running it in the REPL?

Have you tried to call

```julia
makie_window = display(GLMakie.Screen(), Fig)

```

I normally use it when I want to open multiple Makie window. If you are running a script and want to block it from finishing until the makie window is closed you can add.

```julia
wait(makie_window)

```

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 23, 2024, 4:11am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/12 "2024-08-23T04:11:56Z")

</div>

Thanks Fliks for solving the issue. when opening with include. But does not work when \*\*Running without debugging \*\*

```julia
 include(raw"D:\MAIN\MODELS\ViewingOutput.jl")

```

a

```julia
		GLMakie.activate!()
		Makie.inline!(false) # Make sure to inline plots into Documenter output!

```

The solution below works for all CASES

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [August 23, 2024, 4:14am UTC](https://discourse.julialang.org/t/ncdatasets-performance-issues-deriving-fast-minimum-maximum-value-as-too-slow/118468/13 "2024-08-23T04:14:54Z")

</div>

This is the best solution for plots with Sliders and works in Visual Studio Code for all scenarios:

- Opening with include,
- Running without debugging

Thanks for this robust solution