# Lag vector by group using another vector as the grouping variable

**URL:** <https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766>\
**Category:** General Usage\
**Tags:** question\
**Created:** [February 19, 2022, 5:37pm UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766 "2022-02-19T17:37:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [February 19, 2022, 5:37pm UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766/1 "2022-02-19T17:37:35Z")

</div>

Hello everyone,

I was wondering if it were possible to lag a vector by group, when the grouping variable is a separate vector.

Here’s a minimum working example:

```julia
firm_id = ["A", "A", "A", "A", "B", "B", "B", "B"] 
revenue = [100, 200, 300, 400, 50, 67, 75, 90]
year = [2001,2002,2003,2004,2001,2002,2003,2004]

```

where you can think of the revenue as a panel data that corresponds to each firm.

I’d like to lag the revenue variable by firm\_id, to get the following result:

```julia
revenue_lag = [missing, 100, 200, 300, missing, 50, 67, 75]

```

In my project I have to perform this operation repeatedly for different values of the parameter I’m optimizing over.

I realize that this can be done by putting firm\_id and revenue into a DataFrame and then lagging it, but I was wondering if it were possible to do so without creating a DataFrame every time I want to do this. My (uninformed) guess is that creating a DataFrame everytime to lag is more time-consuming, please correct me if I’m wrong about this.

I saw that the ShiftedArrays package has a lag() function but couldn’t figure out how it could be applied by a grouping vector.

Thanks!

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [February 19, 2022, 5:59pm UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766/2 "2022-02-19T17:59:37Z")

</div>

No, I would definitely use DataFrames and ShiftedArrays. Probably with Chain or another similar piping package.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 19, 2022, 6:12pm UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766/3 "2022-02-19T18:12:18Z")

</div>

Agree, just make sure you don’t copy everything when you create the DataFrame:

```julia
julia> using DataFrames, BenchmarkTools

julia> firm_id = rand(["A", "B"], 100_000); revenue = rand(Int, 100_000); year = rand(2001:2004, 100_000);

julia> @btime DataFrame(firm_id = $firm_id, revenue = $revenue, year = $year);
  122.900 μs (34 allocations: 2.29 MiB)

julia> @btime DataFrame(firm_id = $firm_id, revenue = $revenue, year = $year; copycols = false);
  2.140 μs (27 allocations: 1.80 KiB)

```

---

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [February 20, 2022, 1:41am UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766/4 "2022-02-20T01:41:05Z")

</div>

Ah that’s a nice trick, thanks!

---

<div class="post-metadata">

**Author:** ![George9000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/george9000/32/23619_2.png) [@George9000](https://discourse.julialang.org/u/George9000)\
**Post date:** [February 20, 2022, 1:46am UTC](https://discourse.julialang.org/t/lag-vector-by-group-using-another-vector-as-the-grouping-variable/76766/5 "2022-02-20T01:46:55Z")

</div>

```julia
using DataFrames, ShiftedArrays
using Chain

firm_id = ["A", "A", "A", "A", "B", "B", "B", "B"]
revenue = [100, 200, 300, 400, 50, 67, 75, 90]
year = [2001,2002,2003,2004,2001,2002,2003,2004]

df = DataFrame(;firm_id, revenue, year, copycols = false)

@chain df begin
    groupby(:firm_id)
    transform(:revenue => lag => :revenue_lag, ungroup = false)
    transform([:revenue, :revenue_lag] => ByRow((x, y) -> x - y) => :diff_prior_year, ungroup = false) 
end

## avoid anonymous function for better performance

diff(x, y) = x - y

@chain df begin
    groupby(:firm_id)
    transform(:revenue => lag => :revenue_lag, ungroup = false)
    transform([:revenue, :revenue_lag] => ByRow(diff) => :diff_prior_year, ungroup = false) 
end

```