# Manipulating Panel Data in Julia
**URL:** https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109
**Category:** General Usage
**Created:** [June 21, 2022, 8:52am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109 "2022-06-21T08:52:22Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@IljaK91](https://discourse.julialang.org/u/IljaK91)
#### Post date: [June 21, 2022, 8:52am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/1 "2022-06-21T08:52:23Z")
Hey everyone,
I was looking for a straightforward way to do common manipulations in panel data within Julia. For instance, let’s say I have a panel of many countries and years. I would like to calculate the growth rate for a variable country by country. What is the most convenient way to do this? Is there a way to do this using packages such as Query.jl?
I just found something that seems pretty close in R: [A new package for panel data analysis in R | R-bloggers](https://www.r-bloggers.com/2019/05/a-new-package-for-panel-data-analysis-in-r-2/)
Would be great if Julia offered something similar!
Thanks a lot!
Ilja
---
### Author:  [@marcpabst](https://discourse.julialang.org/u/marcpabst)
#### Post date: [June 21, 2022, 9:20am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/2 "2022-06-21T09:20:38Z")
Is there any reason you would rather chose `panelr` over using `dplyr` directly? For Julia, I would look into [DataFramesMeta](https://juliadata.github.io/DataFramesMeta.jl/stable/).
---
### Author:  [@IljaK91](https://discourse.julialang.org/u/IljaK91)
#### Post date: [June 21, 2022, 9:33am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/3 "2022-06-21T09:33:04Z")
At the moment I mainly need a lead/lag operator that respects the panel structure, i.e., applies the lag operator for each id in the panel separately, generates NA where necessary, and puts everything back together. If that is possible with DataFramesMeta directly, that would be great! I guess that is also the added convenience of panelr over deplyr.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [June 21, 2022, 9:50am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/4 "2022-06-21T09:50:22Z")
```julia
julia> using DataFrames, ShiftedArrays
julia> df = DataFrame(stock = ["a", "a", "a", "b", "b", "b"], time = repeat(1:3, 2), price = rand(100:120, 6))
6×3 DataFrame
Row │ stock time price
│ String Int64 Int64
─────┼──────────────────────
1 │ a 1 108
2 │ a 2 105
3 │ a 3 107
4 │ b 1 101
5 │ b 2 106
6 │ b 3 109
julia> transform!(groupby(df, :stock), :price => (x -> log.(x) .- log.(lag(x))) => :return)
6×4 DataFrame
Row │ stock time price return
│ String Int64 Int64 Float64?
─────┼───────────────────────────────────────
1 │ a 1 108 missing
2 │ a 2 105 -0.0281709
3 │ a 3 107 0.0188685
4 │ b 1 101 missing
5 │ b 2 106 0.0483186
6 │ b 3 109 0.0279088
```
---
### Author:  [@IljaK91](https://discourse.julialang.org/u/IljaK91)
#### Post date: [June 21, 2022, 10:12am UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/5 "2022-06-21T10:12:13Z")
Thanks, that worked 🙂
Just one last question: This method does not work when there are gaps in the variable time, correct?
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [June 21, 2022, 3:00pm UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/6 "2022-06-21T15:00:12Z")
Well, it assumes that subsequent rows represent one time step. If you have missing observations which aren’t in the data (i.e. not represented by `missing` but absent altogether) you can construct the range of time steps first (something like `minimum(df.date):Day(1):maximum(df.date)`) and then `leftjoin` your data onto that, which will generate the missing observations. Any return (in my example) where one of the two days used to compute it is missing will then be missing.
---
### Author:  [@eirikeb](https://discourse.julialang.org/u/eirikeb)
#### Post date: [September 13, 2022, 6:30pm UTC](https://discourse.julialang.org/t/manipulating-panel-data-in-julia/83109/7 "2022-09-13T18:30:23Z")
At the dangers of over-promoting, but I’ve written a package that takes care of some the issues you mention @IljaK91: [PanelDataTools.jl](https://github.com/eirikbrandsaas/PanelDataTools.jl)
It is at it’s core just a wrapper around some existing packages and solutions, but from my tests it deals well with for example missing times, use `DateTime` to keep track of time instead of row numbers (which is what shifted arrays basically does).