What would be the best approach to handle large NetCDF sets?

Specific Domains Geo
,
11

For some reason the data is opened with a singleton time dimension so that when we try to concatenate the data it doesnt work, because the Time dimension is different for every cube.
Therefore we can index into every cube to get the first time array in the time dimension to enable the concatenation.
The following script is working with your data:

using Statistics
using Dates
using YAXArrays
using NetCDF
using DimensionalData
#using CairoMakie

startDate = DateTime(1960, 1, 1)
endDate = DateTime(1960, 1, 1, 6)
timeStep = Dates.Hour(1)
datetimes = startDate:timeStep:(endDate-timeStep) |> collect


filenames::Vector{String} = readdir("data_temperature/", join=true)
#for dt in datetimes
#    push!(filenames, "temperature_data/2m air temperature_0_$(Dates.format(dt, "yyyymmddTHHMMSS")).nc")
#end

cubes = Cube.(filenames)
Cube(cubes)
fullcube = concatenatecubes(getindex.(cubes, Time=1), Ti(datetimes))

# Now we define an inner function that is applied along the axes
function meanweight(xout, xin, y)
       xout .= mean(xin .* cos.(deg2rad.(y))')
       end

# I am to lazy to think about a proper deg2rad function but it needs to be defined:
deg2rad(x) = x

# Now we can apply the meanweight function along the dimensions that are specified
meancube = mapCube(meanweight, (fullcube, YAXArray((fullcube.y,), fullcube.y.val)), indims=(InDims("X","Y"), InDims("Y")), outdims=OutDims())
1 Like