# Avoiding a copy after removing missings from a vector
**URL:** https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866
**Category:** General Usage
**Tags:** copy, missing-values
**Created:** [July 19, 2025, 10:42am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866 "2025-07-19T10:42:36Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@s-baumann](https://discourse.julialang.org/u/s-baumann)
#### Post date: [July 19, 2025, 10:42am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/1 "2025-07-19T10:42:36Z")
In Julia I am trying to write a function that will (without copying) compute something from two vectors. As an example think of Pearson correlation (although it is not that). What I have got is:
```julia
function get_valid_entries(x::Vector, y::Vector)
valid_entries_in_x = .!(ismissing.(x)
valid_entries_in_y = .!(ismissing.(y)
return findall(valid_entries_in_x .& valid_entries_in_y)
end
x = [4.0, missing, 6.0, 5.0, 8.0, 10.0]
y = [2.0, 3.0, missing, 5.0, 6.0, 2.0]
valids = get_valid_entries(x, y)
function do_something(x::Vector{R}, y::Vector{R}) where R<:Real
println("This is do_something(x::Vector{R}, y::Vector{R})")
end
do_something(x[valids], y[valids])
```
This does not work as `x[valids]` and `y[valids]` are `Vector::{Union{Missing,Float64}}`. I was hoping to avoid copies though so I don’t want to just convert the vectors with `Vector{Float64}(x[valids])`.
Is there a way I can avoid a copy? I was hoping I could avoid copying the vector slices now that there are no missings and the data in there presumably doesn’t have to be rewritten.
The other question is if for a long vector if it is generally faster to do the copy (as the copied vector will be continuous in memory)
---
### Author:  [@gdalle](https://discourse.julialang.org/u/gdalle)
#### Post date: [July 19, 2025, 11:18am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/2 "2025-07-19T11:18:04Z")
Can you be more specific on what the final function actually needs to do in your case? And why it has a Real type constraint?
---
### Author:  [@s-baumann](https://discourse.julialang.org/u/s-baumann)
#### Post date: [July 19, 2025, 11:27am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/3 "2025-07-19T11:27:17Z")
The function that I am feeding this into only makes sense for real values. I am using a similar coding pattern to this a few times (trying to remove invalid entries and then feed it into a different function) but two of the things I am doing are Pearson correlation and Spearman correlation (I realise both are in StatsBase but I am trying out some potential improvements).
I guess I could just define those functions as taking Vector{R,Missing} but would prefer to have stricter typing… it just seems cleaner if it is possible.
---
### Author:  [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)
#### Post date: [July 19, 2025, 11:46am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/4 "2025-07-19T11:46:32Z")
It is not possible to avoid the copy from a `Vector{Union{T, MIssing}}` to a `Vector{T}`. I wonder if it _could_ ever be possible, but I suspect no.
Some other ways you could speed it up is to not allocate the three arrays with indices. Instead, compute a single array of valid indices in one pass over `x` and `y`. Alternatively, do two passes, one to count the number of elements in the resulting `Vector{T}`, and another to copy their elements.
Maybe you could use something like this (not super well optimised):
```julia
function get_nonmissing_indices(x::AbstractArray{>:Missing}, xs::Vararg{AbstractArray{>:Missing}})
result = Int[]
vs = (x, xs...)
for i in eachindex(vs...)
if all(v -> !ismissing(@inbounds(v[i])), vs)
push!(result, i)
end
end
result
end
type_remove_missing(::Type{Union{T, Missing}}) where T = T
type_remove_missing(::Type{Any}) = Any
function remove_any_missings(x::AbstractArray{>:Missing}, xs::Vararg{AbstractArray{>:Missing}})
vs = (x, xs...)
idx = get_nonmissing_indices(vs...)
L = length(idx)
removed = map(vs) do v
r = similar(v, type_remove_missing(eltype(v)), (L,))
@inbounds for (inew, iold) in zip(eachindex(r), idx)
r[inew] = v[iold]
end
r
end
(;idx, removed)
end
```
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [July 19, 2025, 11:48am UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/5 "2025-07-19T11:48:57Z")
Lazy `skipmissing`?
---
### Author:  [@joa-quim](https://discourse.julialang.org/u/joa-quim)
#### Post date: [July 19, 2025, 12:06pm UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/6 "2025-07-19T12:06:06Z")
This is one of the reasons I never use `missing` with float arrays (actually, I never using missings anywhere but that’s my taste/needs). Use NaNs instead, which are numbers of the same type the array elements.
---
### Author:  [@s-baumann](https://discourse.julialang.org/u/s-baumann)
#### Post date: [July 19, 2025, 12:15pm UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/7 "2025-07-19T12:15:26Z")
Thanks @jakobnissen. I will have a go with your suggested optimisations. I guess I cannot avoid the copy though. Thanks @Benny, I can’t use skipmissing alone as I am also removing NaNs (I removed that from above for simplicity). I guess I could pair it with a similar thing for NaNs but it ends up being pretty equivalent to what I have.
I completely agree @joa-quim . This is at the outer edge of the codebase I am writing, once its all within my code I don’t use missing. Its hard to totally avoid missing given in most packages it is used as a default value.
---
### Author:  [@eldee](https://discourse.julialang.org/u/eldee)
#### Post date: [July 19, 2025, 12:21pm UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/8 "2025-07-19T12:21:39Z")
> [@s-baumann](#):
>
> I was hoping to avoid copies though so I don’t want to just convert the vectors with `Vector{Float64}(x[valids])`.
By the way, note that `x[valids]` already creates a copy, so that `Vector{Float64}(x[valids])` copies twice. It would be better to use something like `Vector{Float64}(@view x[valids])` or `convert(Vector{Float64}, @view x[valids])`.
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [July 19, 2025, 1:29pm UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/9 "2025-07-19T13:29:27Z")
> [@s-baumann](#):
>
> I completely agree @joa-quim . This is at the outer edge of the codebase I am writing, once its all within my code I don’t use missing. Its hard to totally avoid missing given in most packages it is used as a default value.
There are definitely performance gotchas with `missing`! Depending on your usecase, you might want to convert them to NaNs right after reading from file, for example.
> [@s-baumann](#):
>
> I can’t use skipmissing alone as I am also removing NaNs
There’s Skipper.jl – like `skipmissing`, but more general 🙂
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [July 19, 2025, 2:27pm UTC](https://discourse.julialang.org/t/avoiding-a-copy-after-removing-missings-from-a-vector/130866/10 "2025-07-19T14:27:34Z")
> [@s-baumann](#):
>
> Is there a way I can avoid a copy?
Not if you really need a `Vector{Float64}` output, that’s just a different data structure. I think we could unsafely reuse a contiguous portion of the underlying `Memory{Union{Float64, Missing}}` and treat it as if it were a `Memory{Float64}`, but that’s assuming a lot about `Memory` we can’t rely on.
> [@eldee](#):
>
> `convert(Vector{Float64}, @view x[valids])`
The allocation here doesn’t seem too bad though because `valids` needed allocations to begin with, so not sure what the harm of another couple allocations are.
> [@s-baumann](#):
>
> I can’t use skipmissing alone
There is an analogous SkipNan.jl. You can make an entirely lazy indexable sequence guaranteed to never return `missing` or `NaN` (by erroring at those elements), and if you want to adjust the indices to also skip those elements, you can use `view` with `valids` beforehand. `collect` also removes the `Missing` element type, unlike for a `view` or slice.
```julia
julia> using BenchmarkTools; using SkipNan: skipnan
julia> x = [1,missing,3,NaN,5,NaN,7,missing];
julia> @btime y = skipnan(skipmissing(view($x, $(1:2:8))))
6.800 ns (0 allocations: 0 bytes)
skipnan(skipmissing(Union{Missing, Float64}[1.0, 3.0, 5.0, 7.0]))
julia> y[1], y[2], y[3], y[4]
(1.0, 3.0, 5.0, 7.0)
julia> collect(y) |> typeof
Vector{Float64} (alias for Array{Float64, 1})
julia> collect(view(x, 1:2:8)) |> typeof
Vector{Union{Missing, Float64}} (alias for Array{Union{Missing, Float64}, 1})
```
Worth noting that `skipmissing` and `skipnan` wrap _iterables_, so they can’t subtype `AbstractVector` and don’t support much except `eltype` and indexing if the parent object does. That’s also why the `view` was inside, rather than outside. Not sure how an `AbstractVector`-only `skipmissing` could be feasible because `size` is much harder to compute, and the `size` varies with `setindex!` calls to the parent object, not just `push!`/`insert!`/`delete!`. Still, that could be hypothetically done if we just accept that even more mutations of the parent object are dangerous than for `view`.