# Tidying data: DataFrame

**URL:** <https://discourse.julialang.org/t/tidying-data-dataframe/40692>\
**Category:** New to Julia\
**Created:** [June 3, 2020, 7:24pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692 "2020-06-03T19:24:11Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 3, 2020, 7:24pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/1 "2020-06-03T19:24:11Z")

</div>

Hello, I have been trying to do some basic Data Analysis with Julia DataFrames and cannot find a way to tidy my data.

I would like to simply use values of one of the columns to group df based on them and later plot. My impression from reading documentation is that Julia’s `groupby` work differently than R’s ` group_by` and I see that grouped df isn’t of a shape I would expect it to be after R called `group_by`. Could you please navigate me the Julia’s function/macro that could be helpful (ideally still in DataFrames.jl since I expect it has to be there)

code example:

```julia
>println(first(df,1))

1×4 DataFrame
│ Row │ slice │ count_ │ img_name │ circ │
│ │ String │ Int64 │ String │ String │
├─────┼───────────────────┼────────┼───────────────────────────────┼─────────────┤
│ 1 │ Image[name=2,dim] │ 1 │ Labeling[\nname=ImageA7.tif;] │ 0,733056435 │
 
groupby(df, :img_name)

#general look of df output renames the same while the type is now I belive SubDataFrame 

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 3, 2020, 7:36pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/2 "2020-06-03T19:36:46Z")

</div>

Glad you are working with DataFrames.

I think there might be some confusion about what `groupby` does in both `dplyr` and `DataFrames`. In both cases, they create a new object, not a modified one. I think you want

```julia
gdf = groupby(df, :img_name)

```

R should work the same way, where grouping gives you a new object. However `R` defines more operations on a grouped data frame. For instance, in R you can rename columns in a grouped data frame. You can’t do this in Julia, you can only rename `DataFrame`s.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 3, 2020, 7:42pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/3 "2020-06-03T19:42:15Z")

</div>

Can you please describe what you want to achieve with some concrete columns then probably we can help you to find a proper command to achieve this.

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 3, 2020, 10:14pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/4 "2020-06-03T22:14:48Z")

</div>

Oh, I see thank you!

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 3, 2020, 10:26pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/5 "2020-06-03T22:26:03Z")

</div>

Thank you for taking the time, I wanted to make a new column based on `img_name` column so that grouped df had a new column f.e. `img_name_Labelling_name` contained only values like `ImageA7` and no additional variable information to later plot `circ` values grouped by values from a new column, here `ImageA7`

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 4, 2020, 6:36am UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/6 "2020-06-04T06:36:37Z")

</div>

If I understand you correctly you could do (you have not provided the code to reproduce your request nor information what plotting package you use so I am using placeholder functions).

```julia
df.img_name_Labelling_name = extract_name(df.img_name)
for sdf in groupby(df, :img_name_Labelling_name)
    plot(sdf.circ) # or whatever plotting you need
end

```

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 5, 2020, 2:04am UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/7 "2020-06-05T02:04:50Z")

</div>

Oh, sorry I though I those snippets should be enough, thank you for the for loop it looked like something I wanted but after modifying it (so basically going around `extract_name`) it hasn’t worked for me giving blanck outputs:

just modified for loop:

```julia
p = plot()
for sdf in groupby(df, :img_name)
    histogram!(sdf.f_circ)
end
p.show()

```

the whole code:

```julia
using CSV, DataFrames, Statistics, RCall, Plots; pyplot();
folder_path = ".../file.csv"

#read csv and rename the Cols of it
df = CSV.read(folder_path, copycols = true, header = 0)
    hd = Dict("Col1" =>"slice", "Col2" =>"count_",
        "Col3" => "img_name", "Col4" => "circ")
        rename!(df, [Symbol("Col$i") for i in 1:size(df,2)])
        rename!(df, hd)

#TIDYING DATA:
#1. shorten redundant names with the help of manually composed Dict:
unique_names = Array(unique!(df.img_name))
sort(unique_names)#copied names from unique_names later
name_dict = Dict(
"Labeling[\nname=ImageA0.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA0",
"Labeling[\nname=ImageA1.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA1",
"Labeling[\nname=ImageA2.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA2",
"Labeling[\nname=ImageA3.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA3",
"Labeling[\nname=ImageA4.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA4",
"Labeling[\nname=ImageA5.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA5",
"Labeling[\nname=ImageA6.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA6",
"Labeling[\nname=ImageA7.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA7",
"Labeling[\nname=ImageA8.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA8",
"Labeling[\nname=ImageA9.tif;\nsource=;\ndimensions=512,512 (X,Y)]"=> "ImageA9")

replace!(df[!, :img_name], name_dict...)
    df.slice = replace.(df[!, :slice], "\nsource=;" => "")
    df.slice = replace.(df[!, :slice], "\ndimensions" => "dim")
    df.slice = replace.(df[!, :slice], "\npixel" => "")
    println("\t **after** renaming and before parsing: ", first(df,5))

#2. Preparing for parsing: replacing dots for comma to parse as Float later
df.f_circ = replace.(df[:, :circ], "0," => "0.")
for i in 1:size(df,1)
    if startswith.(df[i, :circ], "0,") || startswith.(df[i, :circ], "1,")
        df.f_circ[i] = replace.(df[i, :circ], "0," => "0.")
        df.f_circ[i] = replace.(df[i, :f_circ], "1," => "1.")
    end
end

#Investigation
df[!,:f_circ] = parse.(Float16,df[!, :f_circ])
    println("\t **after** renaming and parsing: ",first(df,5))

#groupby for loop **here**

```

output of the last of investiagion `println` (hence outlook of data in df)

```julia
**after** renaming and parsing: 5×5 DataFrame
│ Row │ slice │ count_ │ img_name │ circ │ f_circ │
│ │ String │ Int64 │ String │ String │ Float16 │
├─────┼──────────────────────────────────────────────────────────────┼────────┼──────────┼─────────────┼─────────┤
│ 1 │ Image[\nname=1;dim=14,14 (X,Y);\nmin=325,99; type=BitType)] │ 1 │ ImageA7 │ 0,733056435 │ 0.733 │
│ 2 │ Image[\nname=2;dim=19,20 (X,Y);\nmin=196,123; type=BitType)] │ 2 │ ImageA6 │ 0,787814224 │ 0.7876 │
│ 3 │ Image[\nname=3;dim=14,12 (X,Y);\nmin=50,138; type=BitType)] │ 3 │ ImageA4 │ 0,749961791 │ 0.75 │
│ 4 │ Image[\nname=4;dim=13,14 (X,Y);\nmin=339,141; type=BitType)] │ 4 │ ImageA5 │ 0,715588033 │ 0.716 │
│ 5 │ Image[\nname=5;dim=1,1 (X,Y);\nmin=474,143; type=BitType)] │ 5 │ ImageA1 │ 0 │ 0.0 │

```

sorry for the general messy thread and thanks for looking replies

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 5, 2020, 5:58am UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/8 "2020-06-05T05:58:54Z")

</div>

Seeing the output I am not clear what you are trying to achieve. You have one observation per group so how do you expect the histogram to look like?

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 5, 2020, 2:37pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/9 "2020-06-05T14:37:13Z")

</div>

With one observation in group (here `:img_name` containing `ImageA0` to `ImageA9`) I would be expecting to see just one histrogram from the for loop (with bars on x-axis as `ImageA0` to `ImageA9` and y-axis showing `f_circ` values).

With more observations there I thought to see number of histograms equaled to added observatios (so if `:img_name` contained `ImageA0_A` and `ImageA0_B` to `ImageA9_A` and `ImageA9_B` would be expecting two histograms `_A` and `_B` of the same parameters as described above). But for now, trying to do it on the first case to practice and learn on dummy data

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 5, 2020, 2:45pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/10 "2020-06-05T14:45:16Z")

</div>

> [@Auhen\_Shauchuk](#):
>
> with bars on x-axis

but with one observation (as in the data frame you have shared) it will be “one bar” not “bars”.

* * *

Probably it would be simplest if you could share your data and the code you now use so that we can help you fix it (if sharing them is possible).

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 5, 2020, 4:59pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/11 "2020-06-05T16:59:24Z")

</div>

Yes, I would love that, thanks for the help! Full code is essentially what have I posted previously and is in archive together with used .csv on the google-drive here : [circularity.\_julia\_Shauchuk.zip - Google Drive](https://drive.google.com/file/d/1yKMIZMVUFZ6rHognlFmtu9pm37YX4RLi/view?usp=sharing)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 5, 2020, 8:45pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/12 "2020-06-05T20:45:19Z")

</div>

Here are my comments on your code (I concentrate on major things):

```julia
Array(unique!(df.img_name))

```

is a serious bug - it resizes `:img_name` column in place and corrupts `df`. You will get errors when trying to work with `df` after this operation. Just write:

```julia
unique(df.img_name)

```

instead.

If you fix this all works correctly except that you need to `display` your plot:

```julia
p = plot()
for sdf in groupby(df, :img_name)
    histogram!(sdf.f_circ)
end
display(p)

```

---

<div class="post-metadata">

**Author:** ![Auhen\_Shauchuk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/auhen_shauchuk/32/15225_2.png) [@Auhen\_Shauchuk](https://discourse.julialang.org/u/Auhen_Shauchuk)\
**Post date:** [June 5, 2020, 9:38pm UTC](https://discourse.julialang.org/t/tidying-data-dataframe/40692/13 "2020-06-05T21:38:40Z")

</div>

Ah yeah, it does now I can go on now, thanks for helping me out !