# How to update a data set using another data set?
**URL:** https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203
**Category:** Performance
**Tags:** question, inmemorydatasets
**Created:** [April 28, 2022, 10:32am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203 "2022-04-28T10:32:27Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@mostafa1342004](https://discourse.julialang.org/u/mostafa1342004)
#### Post date: [April 28, 2022, 10:32am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/1 "2022-04-28T10:32:27Z")
I have a master data set and I like to update some of its values using a transaction data set. There are some key columns for finding the rows that must be updated in both data sets. I know about `update!` function but the catch for my problem is that I don’t like to replace the values in the master data set, instead I like to ,for example, add the values of the transaction data set to the values in the master data set.
```julia
julia> master=Dataset(x=[1,1,2,3,3],y=[missing,0,2,3,-1]);
julia> transaction=Dataset(x=[1,3],y=[-1,1]);
```
I like
```julia
Dataset(x=[1,1,2,3,3],y=[missing,-1,2,4,0])
```
my data sets are huge and there are many key/value columns, i am looking for a scalable solution.
---
### Author:  [@moeddel](https://discourse.julialang.org/u/moeddel)
#### Post date: [April 28, 2022, 12:45pm UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/2 "2022-04-28T12:45:52Z")
My first idea would be to try a standard [Dictionary](https://docs.julialang.org/en/v1/base/collections/#Dictionaries). However, depending on your requirements [DataStructures.jl](https://github.com/JuliaCollections/DataStructures.jl) offers a larger variety of data structures.
As a side note, your question is stated quite general. More context could help you get better feedback. E.g. if you are looking for something persistent or in-memory.
---
### Author:  [@monopolynomial](https://discourse.julialang.org/u/monopolynomial)
#### Post date: [April 29, 2022, 12:44am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/3 "2022-04-29T00:44:52Z")
```julia
using InMemoryDatasets
using Chain
master=Dataset(x=[1,1,2,3,3],y=[missing,0,2,3,-1]);
transaction=Dataset(x=[1,3],y=[-1,1]);
@chain master begin
compare(transaction,on=:x,eq=(x,y)->x+y)
select!(Not([:obs_id_left,:obs_id_right]))
rename!(first,_)
update!(master, on=:x)
end
```
I compared master and transaction with eq(x,y)=x+y and then used update! to update the output. it should be simpler than leftjoin to understand but i don’t think is faster than it.
---
### Author:  [@Kia\_Kia](https://discourse.julialang.org/u/Kia_Kia)
#### Post date: [May 1, 2022, 10:06am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/4 "2022-05-01T10:06:46Z")
The above mentioned reply does not function on missing values.
``
using InMemoryDatasets
master=Dataset(x=[1,1,2,3,3],y=[missing,0,2,3,-1]);
transaction=Dataset(x=[1,3],y=[-1,1]);
val\_cols=[:y]
key\_cols=[:x]
leftjoin!(master,view(transaction,:,key\_cols),on=key\_cols,obs\_id=true)
function fun(x,y,z)
for i in eachindex(y)
if !ismissing(x[i])
y[i]+=z[x[i]];
end
end
y
end
transforms\_list=[c=\>x-\>fun(master[!,:obs\_id\_right],x,transaction[!,c] for c in val\_cols]
modify!(master,transforms\_list)
select!(master,Not([:obs\_id\_left,:obs\_id\_right]))
---
### Author:  [@DataFrames](https://discourse.julialang.org/u/DataFrames)
#### Post date: [May 3, 2022, 4:14am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/5 "2022-05-03T04:14:21Z")
Splendid! I rewrote it for `DataFrames.jl`
```julia
using DataFrames
master=DataFrame(x=[1,1,2,3,3],y=[missing,0,2,3,-1]);
transaction=DataFrame(x=[1,3],y=[-1,1]);
val_cols=[:y]
key_cols=[:x]
transaction.obs_id_right = 1:nrow(transaction)
leftjoin!(master,view(transaction,:,[key_cols; :obs_id_right]),on=key_cols);
function fun(x,y,z)
for i in eachindex(y)
if !ismissing(x[i])
y[i]+=z[x[i]];
end
end
y
end
transforms_list=[c=>(x->fun(master[!,:obs_id_right],x,transaction[!,c]))=>c for c in val_cols]
transform!(master,transforms_list)
select!(master,Not([:obs_id_right]))
```
---
### Author:  [@mostafa1342004](https://discourse.julialang.org/u/mostafa1342004)
#### Post date: [July 1, 2022, 9:30am UTC](https://discourse.julialang.org/t/how-to-update-a-data-set-using-another-data-set/80203/6 "2022-07-01T09:30:33Z")
Thanks for replies. I opened an issue and it’s added to InMemoryDatasets.jl
```julia
update(master, transaction, on=:x, op=+)
```