# Error from conversion of missing values to NaNs

**URL:** <https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321>\
**Category:** General Usage\
**Tags:** question, missing-values, netcdf\
**Created:** [January 4, 2024, 12:38am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321 "2024-01-04T00:38:56Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Leon6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leon6/32/205483_2.png) [@Leon6](https://discourse.julialang.org/u/Leon6)\
**Post date:** [January 4, 2024, 12:38am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/1 "2024-01-04T00:38:56Z")

</div>

A is a value I read out of an NetCDF file (32 bit). Its size is 360x180x30.  
Here is my code to convert its missing values into NaNs:  
`A[ismissing.(A)] .= NaN;`

Why does the above script work perfectly fine, but every time when A is composed of integers, it is causing errors like the below:

```julia
**ERROR:** LoadError: InexactError: Int32(NaN)
Stacktrace:
[1] **Int32**
@ ./float.jl:879 [inlined]
[2] **convert**
@ ./number.jl:7 [inlined]
[3] **convert**
@ ./missing.jl:69 [inlined]
[4] **fill!(**A::SubArray{Union{Missing, Int32}, 1, Base.ReshapedArray{Union{Missing, Int32}, 1, Array{Union{Missing, Int32}, 4}, Tuple{}}, Tuple{Vector{Int64}}, false}, x::Float64**)**
@ Base ./multidimensional.jl:1084
[5] **copyto!**
@ ./broadcast.jl:934 [inlined]
[6] **materialize!**
@ ./broadcast.jl:884 [inlined]
[7] **materialize!(**dest::SubArray{Union{Missing, Int32}, 1, Base.ReshapedArray{Union{Missing, Int32}, 1, Array{Union{Missing, Int32}, 4}, Tuple{}}, Tuple{Vector{Int64}}, false}, bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(identity), Tuple{Float64}}**)**
@ Base.Broadcast ./broadcast.jl:881

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 4, 2024, 12:48am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/2 "2024-01-04T00:48:22Z")

</div>

> [@Leon6](#):
>
> every time when A is composed of integers, it is causing errors

Because [`NaN` is exclusively a floating-point value](https://en.wikipedia.org/wiki/NaN) — you can’t store that in an integer array.

> [@Leon6](#):
>
> `InexactError: Int32(NaN)`

If you have a `Vector{Int32}`, then it is stored in memory as a sequence of consecutive 32-bit signed integer (`Int32`) values. Every possible bit pattern of an `Int32` represents some integer — there is no bit pattern that represents a `NaN` without changing the type of the elements.

What you could do is

```julia
A = float(A)
A[ismissing.(A)] .= NaN

```

to ensure that `A` is a floating-point container. Or alternatively:

```julia
Anan = map(x -> ismissing(x) ? NaN : Float64(x), A)

```

which has the advantage that the container type of `Anan` will be concretely `Vector{Float64}`, not `Vector{Union{Float64,Missing}}` — the compiler will know that it can no longer contain missing values and optimize accordingly.

PS. Please quote your code: [PSA: how to quote code with backticks](https://discourse.julialang.org/t/psa-how-to-quote-code-with-backticks/7530)

---

<div class="post-metadata">

**Author:** ![Leon6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leon6/32/205483_2.png) [@Leon6](https://discourse.julialang.org/u/Leon6)\
**Post date:** [January 4, 2024, 12:55am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/3 "2024-01-04T00:55:33Z")

</div>

Thank you so much. This solution works for me.

So NaNs do have their limitations? The other day, we were discussing how good NaNs are when it comes to missing values.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [January 4, 2024, 1:22am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/4 "2024-01-04T01:22:46Z")

</div>

And at that day I also told you that NaNs are for floating points only. But why do you have `missings` when reading a netCDF file? netCDF have no `missings` and quite likely had NaNs to start with.

Grids of Int type normally have a flag value to indicate _nodata_. Int32 are also not common in geographical grids.

---

<div class="post-metadata">

**Author:** ![Leon6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leon6/32/205483_2.png) [@Leon6](https://discourse.julialang.org/u/Leon6)\
**Post date:** [January 4, 2024, 2:23am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/5 "2024-01-04T02:23:05Z")

</div>

Yes, people still use Int32 and missing values in NetCDF files.

Below is an example:  
[https://www.ncei.noaa.gov/thredds-ocean/catalog/ncei/woa/temperature/decav/1.00/catalog.html?dataset=ncei/woa/temperature/decav/1.00/woa18\_decav\_t00\_01.nc](https://www.ncei.noaa.gov/thredds-ocean/catalog/ncei/woa/temperature/decav/1.00/catalog.html?dataset=ncei/woa/temperature/decav/1.00/woa18_decav_t00_01.nc)

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [January 4, 2024, 2:39am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/6 "2024-01-04T02:39:01Z")

</div>

> [@Leon6](#):
>
> and missing values in NetCDF files.

That is impossible. `missing` is Julia only concept. netCDF files **can not** have them

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [January 4, 2024, 2:55am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/7 "2024-01-04T02:55:00Z")

</div>

As you can see here, most of the layers are 32-bits **floats**

```julia
julia> println(gdalinfo("woa18_decav_t00_01.nc"))
Driver: netCDF/Network Common Data Format
Files: woa18_decav_t00_01.nc
Size is 512, 512
Metadata:
  NC_GLOBAL#cdm_data_type=Grid
  NC_GLOBAL#comment=global climatology as part of the World Ocean Atlas project
  NC_GLOBAL#contributor_name=Ocean Climate Laboratory
  NC_GLOBAL#contributor_role=Calculation of climatologies
  NC_GLOBAL#Conventions=CF-1.6, ACDD-1.3
  NC_GLOBAL#creator_email=NCEI.info@noaa.gov
  NC_GLOBAL#creator_institution=National Centers for Environmental Information
  NC_GLOBAL#creator_name=Ocean Climate Laboratory
  NC_GLOBAL#creator_type=group
  NC_GLOBAL#creator_url=http://www.ncei.noaa.gov
  NC_GLOBAL#date_created=2019-07-28
  NC_GLOBAL#date_modified=2019-07-28
  NC_GLOBAL#geospatial_lat_max=90
  NC_GLOBAL#geospatial_lat_min=-90
  NC_GLOBAL#geospatial_lat_resolution=1.00 degrees
  NC_GLOBAL#geospatial_lat_units=degrees_north
  NC_GLOBAL#geospatial_lon_max=180
  NC_GLOBAL#geospatial_lon_min=-180
  NC_GLOBAL#geospatial_lon_resolution=1.00 degrees
  NC_GLOBAL#geospatial_lon_units=degrees_east
  NC_GLOBAL#geospatial_vertical_max=5500
  NC_GLOBAL#geospatial_vertical_min=0
  NC_GLOBAL#geospatial_vertical_positive=down
  NC_GLOBAL#geospatial_vertical_resolution=SPECIAL
  NC_GLOBAL#geospatial_vertical_units=m
  NC_GLOBAL#id=woa18_decav_t00_01.nc
  NC_GLOBAL#institution=National Centers for Environmental Information (NCEI)
  NC_GLOBAL#keywords=Oceans< Ocean Temperature > Water Temperature
  NC_GLOBAL#keywords_vocabulary=ISO 19115
  NC_GLOBAL#license=These data are openly available to the public. Please acknowledge the use of these data with the text given in the acknowledgment attribute.
  NC_GLOBAL#metadata_link=https://www.nodc.noaa.gov/OC5/woa18/
  NC_GLOBAL#naming_authority=gov.noaa.ncei
  NC_GLOBAL#nodc_template_version=NODC_NetCDF_Grid_Template_v2.0
  NC_GLOBAL#processing_level=processed
  NC_GLOBAL#project=World Ocean Atlas Project
  NC_GLOBAL#publisher_email=NCEI.info@noaa.gov
  NC_GLOBAL#publisher_institution=National Centers for Environmental Information
  NC_GLOBAL#publisher_name=National Centers for Environmental Information (NCEI)
  NC_GLOBAL#publisher_type=institution
  NC_GLOBAL#publisher_url=http://www.ncei.noaa.gov/
  NC_GLOBAL#references=Locarnini, R. A., A. V. Mishonov, O. K. Baranova, T. P. Boyer, M. M. Zweng, H. E. Garcia, J. R. Reagan, D. Seidov, K. W. Weathers, C. R. Paver, I. V. Smolyar, 2019: World Ocean Atlas 2018, Volume 1: Temperature. A. V. Mishonov, Technical Ed., NOAA Atlas NESDIS 81
  NC_GLOBAL#sea_name=World-Wide Distribution
  NC_GLOBAL#standard_name_vocabulary=CF Standard Name Table v49
  NC_GLOBAL#summary=Climatological mean temperature for the global ocean from in situ profile data
  NC_GLOBAL#time_coverage_duration=P63Y
  NC_GLOBAL#time_coverage_end=2017-12-31
  NC_GLOBAL#time_coverage_resolution=P01Y
  NC_GLOBAL#time_coverage_start=1955-01-01
  NC_GLOBAL#title=World Ocean Atlas 2018 : sea_water_temperature Annual 1955-2017 1.00 degree
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"woa18_decav_t00_01.nc":lat_bnds
  SUBDATASET_1_DESC=[180x2] lat_bnds (32-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"woa18_decav_t00_01.nc":lon_bnds
  SUBDATASET_2_DESC=[360x2] lon_bnds (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"woa18_decav_t00_01.nc":depth_bnds
  SUBDATASET_3_DESC=[102x2] depth_bnds (32-bit floating-point)
  SUBDATASET_4_NAME=NETCDF:"woa18_decav_t00_01.nc":climatology_bounds
  SUBDATASET_4_DESC=[1x2] climatology_bounds (32-bit floating-point)
  SUBDATASET_5_NAME=NETCDF:"woa18_decav_t00_01.nc":t_an
  SUBDATASET_5_DESC=[1x102x180x360] sea_water_temperature (32-bit floating-point)
  SUBDATASET_6_NAME=NETCDF:"woa18_decav_t00_01.nc":t_mn
  SUBDATASET_6_DESC=[1x102x180x360] sea_water_temperature (32-bit floating-point)
  SUBDATASET_7_NAME=NETCDF:"woa18_decav_t00_01.nc":t_dd
  SUBDATASET_7_DESC=[1x102x180x360] sea_water_temperature number_of_observations (32-bit integer)
  SUBDATASET_8_NAME=NETCDF:"woa18_decav_t00_01.nc":t_sd
  SUBDATASET_8_DESC=[1x102x180x360] t_sd (32-bit floating-point)
  SUBDATASET_9_NAME=NETCDF:"woa18_decav_t00_01.nc":t_se
  SUBDATASET_9_DESC=[1x102x180x360] sea_water_temperature standard_error (32-bit floating-point)
  SUBDATASET_10_NAME=NETCDF:"woa18_decav_t00_01.nc":t_oa
  SUBDATASET_10_DESC=[1x102x180x360] sea_water_temperature (32-bit floating-point)
  SUBDATASET_11_NAME=NETCDF:"woa18_decav_t00_01.nc":t_gp
  SUBDATASET_11_DESC=[1x102x180x360] t_gp (32-bit integer)
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 512.0)
Upper Right ( 512.0, 0.0)
Lower Right ( 512.0, 512.0)
Center ( 256.0, 256.0)

```

The the _nodata_ (called \__FillValue_ in netCDF) is 9.96921e36

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/d/cd27768dd5fc3671ca27fea067c05054e8de0b35.png)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [January 4, 2024, 8:31am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/8 "2024-01-04T08:31:38Z")

</div>

> [@joa-quim](#):
>
> That is impossible. `missing` is Julia only concept. netCDF files **can not** have them

Linking this [reference page](https://docs.unidata.ucar.edu/netcdf-c/current/attribute_conventions.html) for more information on the topic.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [January 4, 2024, 9:24am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/9 "2024-01-04T09:24:44Z")

</div>

This is a good discussion, and we can all learn things about netCDF.  
However - please be nice to each other. I know this is difficult but a few 😀 help a lot.

---

<div class="post-metadata">

**Author:** ![Leon6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leon6/32/205483_2.png) [@Leon6](https://discourse.julialang.org/u/Leon6)\
**Post date:** [January 4, 2024, 9:52am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/10 "2024-01-04T09:52:26Z")

</div>

Thank you. Below is one example of Int32 values in the NetCDF file above (Variable: t\_gp). It seems that the FillValues for such integers are empty spaces?

* * *

[VAR\_NAME] t\_oa  
[VAR\_ID] 14  
[ATTR] standard\_name: “sea\_water\_temperature”  
[ATTR] long\_name: “statistical mean value minus the objectively analyzed mean value for sea\_water\_temperature.”  
[ATTR] \_FillValue: “9.969209968386869e+36”  
[DIM] lon: 360  
[DIM] lat: 180  
[DIM] depth: 102

* * *

[VAR\_NAME] t\_gp  
[VAR\_ID] 15  
[ATTR] long\_name: “The number of grid-squares within the smallest radius of influence around each grid-square which contain a statistical mean for sea\_water\_temperature.”  
[ATTR] \_FillValue: " "  
[DIM] lon: 360  
[DIM] lat: 180  
[DIM] depth: 102

---

<div class="post-metadata">

**Author:** ![Alexander-Barth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexander-barth/32/3692_2.png) [@Alexander-Barth](https://discourse.julialang.org/u/Alexander-Barth)\
**Post date:** [January 4, 2024, 10:40am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/11 "2024-01-04T10:40:02Z")

</div>

Which package are you using to read the netcdf file? In NCDatasets we automatically replace all values equal to the attribute to `_FillValues` (and `missing_value`) to `missing`. You can use `ds["varname"].var` if you do not want this transformation. More information is available here:

[https://alexander-barth.github.io/NCDatasets.jl/stable/dataset/#CommonDataModel.cfvariable](https://alexander-barth.github.io/NCDatasets.jl/stable/dataset/#CommonDataModel.cfvariable)

[https://alexander-barth.github.io/NCDatasets.jl/stable/variables/](https://alexander-barth.github.io/NCDatasets.jl/stable/variables/)

---

<div class="post-metadata">

**Author:** ![Alexander-Barth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexander-barth/32/3692_2.png) [@Alexander-Barth](https://discourse.julialang.org/u/Alexander-Barth)\
**Post date:** [January 4, 2024, 11:04am UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/12 "2024-01-04T11:04:46Z")

</div>

Looking at the opendap link ([OPeNDAP Dataset Query Form](https://www.ncei.noaa.gov/thredds-ocean/dodsC/ncei/woa/temperature/decav/1.00/woa18_decav_t00_01.nc.html)) the \_FillValue attribute of the variable t\_gp is -32767.

In NCDatasets all this metadata is reported in the `show` function (often implicitly called).

To replace the `missing` by another value like NaN I often use the `nomissing` (from NCDatasets).

---

<div class="post-metadata">

**Author:** ![Leon6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leon6/32/205483_2.png) [@Leon6](https://discourse.julialang.org/u/Leon6)\
**Post date:** [January 4, 2024, 2:35pm UTC](https://discourse.julialang.org/t/error-from-conversion-of-missing-values-to-nans/108321/13 "2024-01-04T14:35:35Z")

</div>

Thanks for sharing. Basically something like the below:

```julia
a = nomissing(da)

```

Retun the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type and checks that no missing values are present.

```julia
a = nomissing(da,value)

```

Retun the values of the array da of type Array{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value.