# How to modify HDF5 dataset

**URL:** <https://discourse.julialang.org/t/how-to-modify-hdf5-dataset/36413>\
**Category:** General Usage\
**Created:** [March 24, 2020, 3:05am UTC](https://discourse.julialang.org/t/how-to-modify-hdf5-dataset/36413 "2020-03-24T03:05:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bhawkins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bhawkins/32/3778_2.png) [@bhawkins](https://discourse.julialang.org/u/bhawkins)\
**Post date:** [March 24, 2020, 3:05am UTC](https://discourse.julialang.org/t/how-to-modify-hdf5-dataset/36413/1 "2020-03-24T03:05:33Z")

</div>

I’m trying to write a program to read in an HDF5 file with a lot of complicated metadata, do some processing on a large dataset, and write the metadata and modified dataset back out to a new file. I want to avoid explicitly copying each piece of metadata or implementing some generic thing to copy each object (except the interesting dataset) one by one.

My first try was to copy the file, delete the dataset, and write a new one like so

```julia
using HDF5

function update_dataset1(src, dest, dataset)
    cp(src, dest, follow_symlinks=true, force=true)
    f = h5open(dest, "r+")
    d = f[dataset]
    newd = zeros(eltype(d), size(d)...)
    o_delete(d)
    write(f, dataset, newd)
end

```

But this basically doubles the file size because `o_delete` calls `H5Ldelete` which only deletes the reference to the dataset and not the actual object written to file (which becomes unreachable). I could write to a temporary file and shell out to `h5repack` it, I guess. I also tried the following

```julia
function update_dataset2(src, dest, dataset)
    cp(src, dest, follow_symlinks=true, force=true)
    f = h5open(dest, "r+")
    d = f[dataset]
    newd = zeros(eltype(d), size(d)...)
    d .= newd
end

```

But this dies with a `MethodError` not matching `copyto!` with the right types. I can actually do this in Python like so

```python
def update_dataset3(src, dest, dataset):
    shutil.copy(src, dest)
    f = h5py.File(dest, "r+")
    d = f[dataset]
    newd = numpy.zeros_like(np.asarray(d))
    d[:,:] = newd

```

so I’m wondering if there’s just some interface in HDF5.jl that I’m missing.

---

<div class="post-metadata">

**Author:** ![bhawkins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bhawkins/32/3778_2.png) [@bhawkins](https://discourse.julialang.org/u/bhawkins)\
**Post date:** [March 24, 2020, 5:31am UTC](https://discourse.julialang.org/t/how-to-modify-hdf5-dataset/36413/2 "2020-03-24T05:31:50Z")

</div>

I noticed in HDF5.jl that `setindex!` is defined, and so I tried

```julia
function update_dataset4(src, dest, dataset)
    cp(src, dest, follow_symlinks=true, force=true)
    f = h5open(dest, "r+")
    d = f[dataset]
    newd = zeros(eltype(d), size(d)...)
    d[:,:] = newd
end

```

and it does what I want.

So now I’m just wondering what’s the difference between `x[:] = y` and `x .= y`. I think the first calls `setindex!` while the other calls `copyto!`, but I don’t really understand why these are distinct methods.

Edit: Also, is there a shorthand for “all indices of all dimensions” like the `...` in numpy?