# Updating a data with new data, removing redundancies

**URL:** <https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858>\
**Category:** New to Julia\
**Created:** [July 10, 2020, 5:36pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858 "2020-07-10T17:36:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 10, 2020, 5:36pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/1 "2020-07-10T17:36:19Z")

</div>

Suppose I have four corresponding arrays relating to stock market data. Also, suppose I assemble them into a matrix:

```
dataold = [mktId stockId totalVolume price timeStamp]

```

I proceed to gather more data of the stated kind, but at a different moment. This new data will be different at least in the timeStamp array, but may also be different elsewhere (most obviously price and totalVolume). Again, I assemble the data into matrix form:

```
datanew = [mktId stockId totalVolume price timeStamp]

```

The problem that I am trying to solve in the most efficient way possible is the following:

“How to I combine dataold and datanew to get dataall, such that dataold must not contain redundant data?”

By redundant data I mean that rows (in the matrix form [dataold datanew]) that are duplicates except in timeStamp are removed, such that only the first of such rows (i.e., the row with the smallest timeStamp) remain.

I am looking for a general strategy (what is most efficient? Do I need to assemble the matrices, or is there a much better way, for example?) in Julia.

Perhaps someone may even venture a snippet of code that does the job. That would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 10, 2020, 5:47pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/2 "2020-07-10T17:47:13Z")

</div>

This sounds like a job for `leftjoin` using DataFrames. Is there a reason you want to use a matrix? A data frame seems like a much more intuitive structure for this kind of thing.

---

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 10, 2020, 5:52pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/3 "2020-07-10T17:52:03Z")

</div>

Only because matrices are intuitive for me (given my MATLAB background) and because I am not too familiar with DataFrames.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [July 10, 2020, 5:54pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/4 "2020-07-10T17:54:05Z")

</div>

+1 for the suggestion of DataFrames! Doing something like this without seems quite painful. With DataFrames at the very minimum you could just create one long DataFrame (like your `dataall`) and then do `groupby(df,[:mktId,:stockId])` to isolate the data for each stock and then do your filter.

The docs are very helpful [Introduction · DataFrames.jl](https://juliadata.github.io/DataFrames.jl/stable/)

---

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 10, 2020, 6:22pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/5 "2020-07-10T18:22:29Z")

</div>

> [@tbeason](#):
>
> groupby(df,[:mktId,:stockId])

That is very helpful. I have groupby(df,[:mktId,:stockId]) working in my particular case. I may return with a question about how to apply the filter!

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 10, 2020, 6:32pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/6 "2020-07-10T18:32:02Z")

</div>

There is also a function `unique` which can delete rows that are the same for a certain set of columns.

---

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 10, 2020, 7:49pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/7 "2020-07-10T19:49:19Z")

</div>

Can I ask you to apply the filter? In my case, the first step appears to be:

```
groupby(df,[:mktId,:stockId,:totalVolume,:price])

```

That procedure creates many groups. In a particular one of these groups, the elements mktId, stockId, totalVolume and price are the same, but the last element (timeStamps) can be different (if multiple timeStamps exists).

After the first step has been applied, I want each group to only have one timeStamp (the smallest datetime). So, each group should have only one line.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 10, 2020, 7:51pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/8 "2020-07-10T19:51:51Z")

</div>

```julia
combine(groupby(df,[:mktId,:stockId,:totalVolume,:price])) do sdf
    sdf[1, :]
end

```

---

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 10, 2020, 7:56pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/9 "2020-07-10T19:56:46Z")

</div>

> [@pdeffebach](#):
>
> `sdf`

So, you are taking the first index of each group only?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 10, 2020, 8:05pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/10 "2020-07-10T20:05:07Z")

</div>

yes, `combine` will take all of these `DataFrameRow` objects and make them into one big DataFrame.

But you could also do

```julia
unique(df, [:mktId,:stockId,:totalVolume,:price])

```

which would be the same, I think.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [July 10, 2020, 8:47pm UTC](https://discourse.julialang.org/t/updating-a-data-with-new-data-removing-redundancies/42858/11 "2020-07-10T20:47:39Z")

</div>

Be sure the data is sorted properly because those methods will blindly pick the first row. But yes the last solution using `unique` is definitely the way to go in this case if I understand correctly what you want.