# Revise cannot transform my dataset

**URL:** <https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177>\
**Category:** Performance\
**Tags:** performance, dataframes, revise\
**Created:** [May 9, 2020, 11:55am UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177 "2020-05-09T11:55:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 11:55am UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/1 "2020-05-09T11:55:17Z")

</div>

I am trying to solve the “Predict Data Sales” on kaggle, using the `DataFrames`, `Plots` and `CSV` packages. Here is my code, named `processdata.jl`, so far:

```julia
@time using CSV, DataFrames, Plots
println("finished importing")
#
# Load datasets
@time begin
    itemcat = DataFrame(CSV.File("item_categories.csv"))
    items = DataFrame(CSV.File("items.csv"))
    salestrain = DataFrame(CSV.File("sales_train.csv"))
    shops = DataFrame(CSV.File("shops.csv"))
    test = DataFrame(CSV.File("test.csv"))
end
println("finished loading datasets")

# Droping useless columns 
@time select!(test, Not(:ID))
println("finished droping datasets' columns")

# Transforming data.
@time begin
    getmonth(x) = split(x, '.')[2]
    gsalestrain = groupby(salestrain, :item_id)
    salestrainmonth = transform(gsalestrain, :date => x -> getmonth.(x))
end 
println("finished transforming loading datasets")

```

Now, some details:

- I am `using Revise` and had `includet("processdata.jl")` at the beggining of the session, it was all working perfectly.

- The `salestrain` dataframe has around 3 milion rows.

- Those macros weren’t there originally: I put a time macro on every piece of the code because when I tryied to acess the `salestrainmonth` variable (or any other variable, as a matter of fact) in the REPL, it just took LITERALLY forever: the most I waited for was ten minutes, and it just wasn’t able to finish the command, eventualy I Ctrl+C’ed the process.

And why I say that the problem is with `Revise`? Because when I run this exact code in the command line with `julia processdata.jl`, it takes long but it does the job, here is the output:

```julia
 18.799342 seconds (28.28 M allocations: 1.436 GiB, 3.84% gc time)
finished importing
  7.724473 seconds (11.05 M allocations: 992.623 MiB, 5.77% gc time)
finished loading datasets
  0.000029 seconds (31 allocations: 2.359 KiB)
finished droping datasets' columns
  8.471720 seconds (47.52 M allocations: 2.178 GiB, 10.96% gc time)
finished transforming loading datasets

```

Since I added the macros I tried to reload the julia REPL, but as soon as I entry the first command: `using Revise; includet("processdata.jl")` it prints:

```julia
 16.395183 seconds (27.99 M allocations: 1.425 GiB, 4.79% gc time)
finished importing
  7.853719 seconds (11.02 M allocations: 991.699 MiB, 5.23% gc time)
finished loading datasets
  0.000032 seconds (31 allocations: 2.359 KiB)
finished droping datasets' columns

```

and then it’s just silence and nothing else (except for my notebook’s fan screaming).

@Edit  
To shine some more light into this bug, I also tried to run `julia --compile=min processdata.jl` and, in this case, the only thing it was able to do was the imports. Here is the output, with also the “After interruption” (I Ctrl^C’ed after waiting 5 minutes):

```julia
julia --compile=min processdata.jl 
  9.435052 seconds (10.61 M allocations: 633.672 MiB, 1.69% gc time)
finished importing
^C
signal (2): Interrupção
in expression starting at /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:5
jl_f_arrayref at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a91e01)
jl_fptr_interpret_call at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a91e01)
jl_fptr_interpret_call at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a9209a)
jl_fptr_interpret_call at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a91e01)
jl_fptr_interpret_call at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a91e01)
jl_fptr_interpret_call at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5990a90855)
unknown function (ip: 0x7f5990a904fa)
unknown function (ip: 0x7f5990a91d84)
unknown function (ip: 0x7f5990a92832)
unknown function (ip: 0x7f5990ab0371)
unknown function (ip: 0x7f5990a84d9e)
jl_load at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f5983a7e4bb)
unknown function (ip: 0x7f5983a8a11f)
unknown function (ip: 0x7f5983a8a7e2)
unknown function (ip: 0x7f5983a8a925)
unknown function (ip: 0x5611d8acc4fe)
unknown function (ip: 0x5611d8acc0a7)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x5611d8acc15d)
unknown function (ip: (nil))
Allocations: 220195535 (Pool: 220192984; Big: 2551); GC: 34

```

@Edit2  
(Sorry for that many edits, maybe I should have done more tests before posting, but I thought it wasn’t much more complex than it looked like)

Even if I comment the line:  
` #salestrainmonth = transform(gsalestrain, :date => x -> getmonth.(x))`

The problem still persists. So, only if I **also** comment the line before it:  
`#gsalestrain = groupby(salestrain, :item_id)`

`Revise` is able to `includet` my file. Now, an even stranger phenomena is that if I comment both lines and then copy&paste them _into the REPL_, Julia is able to normally create both `gsalestrain` and `salestrainmonth`. That reinforces my thesis that `Revise` is the one struggling to make it work.

@Edit3  
I was doing some tweaking and discovered the `test ` function for packages, here it goes my output for `test Revise`:

```julia
(@v1.4) pkg> test Revise
    Testing Revise
Status `/tmp/jl_eB5WW1/Manifest.toml`
  [aafaddc9] CatIndices v0.2.1
  [da1fd8a2] CodeTracking v0.5.11
  [dc8bdbbb] CustomUnitRanges v1.0.0
  [340492b5] EndpointRanges v0.2.0
  [97e2ac4a] EponymTuples v0.2.2
  [7876af07] Example v0.5.3
  [aa1ae85d] JuliaInterpreter v0.7.14
  [6f1432cf] LoweredCodeUtils v0.4.4
  [dbb5928d] MappedArrays v0.2.2
  [6fe1bfb0] OffsetArrays v1.0.4
  [bac558e1] OrderedCollections v1.2.0
  [ae029012] Requires v1.0.1
  [295af30f] Revise v2.6.5
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8ba89e20] Distributed 
  [7b1f6079] FileWatching 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [6462fe0b] Sockets 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 
Skipping Base.active_repl
Skipping Base.active_repl_backend
Comparison and line numbering: Test Failed at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:332
  Expression: bt.func == :cube && (bt.file == Symbol(tmpfile) && bt.line == 7)
Stacktrace:
 [1] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:332
 [2] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:212
 [4] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:79
Comparison and line numbering: Test Failed at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:340
  Expression: bt.func == :mult2 && (bt.file == Symbol(tmpfile) && bt.line == 13)
Stacktrace:
 [1] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:340
 [2] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:212
 [4] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:79

Line numbers in backtraces and warnings: Test Failed at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1334
  Expression: bt.file == Symbol(filename) && bt.line == 2
Stacktrace:
 [1] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1334
 [2] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1314
 [4] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:79
Line numbers in backtraces and warnings: Test Failed at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1358
  Expression: bt.file == Symbol(filename) && bt.line == 3
Stacktrace:
 [1] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1358
 [2] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:1314
 [4] top-level scope at /build/julia/src/julia-1.4.1/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:79
Revise is currently tracking the following files in ReviseFileNow [top-level]: ["src/ReviseFileNow.jl"]
[ Info: tracking Base
┌ Warning: skipping git tests because Revise is not under development
└ @ Main ~/.julia/packages/Revise/MgvIv/test/runtests.jl:2378
extra
[ Info: tracking Pkg
┌ Warning: skipping Core.Compiler tests due to lack of git repo
└ @ Main ~/.julia/packages/Revise/MgvIv/test/runtests.jl:2486
┌ Warning: REPL tests skipped
└ @ Main ~/.julia/packages/Revise/MgvIv/test/runtests.jl:2529
Test Summary: | Pass Fail Broken Total
Revise | 638 4 1 643
  PkgData | 1 1
  Package contents | 1 1
  LineSkipping | 8 8
  Equality and hashing | 5 5
  Parse errors | 4 4
  Signature extraction | 4 4
  Comparison and line numbering | 73 2 1 76
  Display | 7 7
  File paths | 209 209
  Base & stdlib file paths | 5 5
  Recursive types (issue #417) | 1 1
  Cross-module extension | 6 6
  @ __FILE__ | 2 2
  Module docstring | 6 6
  Changing docstring | 3 3
  Undef in docstrings | 72 72
  Macro docstrings (issue #309) | 4 4
  Changing @inline annotations | 16 16
  Revising macros | 7 7
  More arg-modifying macros | 4 4
  Line numbers | 5 5
  Line numbers in backtraces and warnings | 4 2 6
  New submodules | 3 3
  Timing (issue #341) | 3 3
  Method deletion | 60 60
  revise_file_now | 4 4
  Evaled toplevel | 4 4
  Revision errors | 37 37
  Retry on InterruptException | 17 17
  get_def | 7 7
  Pkg exclusion | 3 3
  Manual track | 18 18
  Auto-track user scripts | 4 4
  Distributed | 15 15
  Git | 2 2
  Recipes | 11 11
  CodeTracking #48 | 1 1
  Methods at REPL | No tests
  baremodule | 2 2
ERROR: LoadError: Some tests did not pass: 638 passed, 4 failed, 0 errored, 1 broken.
in expression starting at /home/ttv1/.julia/packages/Revise/MgvIv/test/runtests.jl:78
ERROR: Package Revise errored during testing

```

I tried to reinstall it (`Revise`); it didn’t work nor did it changed the output above. Then I tried to reinstall `Julia` as whole, it also didn’t work.

I guess I already made the post too long, so I will just leave the output of `lspci` and `uname -a` and go on another way.

```julia
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:15.1 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #1 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-LP LPC Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
01:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 930M] (rev a2)
02:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL810xE PCI Express Fast Ethernet controller (rev 07)

```

`Linux restofunesto 5.4.15-arch1-1 #1 SMP PREEMPT Sun, 26 Jan 2020 09:48:50 +0000 x86_64 GNU/Linux `

I discovered `Julia` recently, and I gotta tell that it was love at first sight (god knows my feelings for the numpy/pandas schizophrenic syntax), and I really wanna make it work, so I won’t give up on this is problem: but I really value workflow, and since quick-and-dirty coding with the “write code → run script” loop is not possible due to long compilation time and since `Revise` seems to be the only viable alternative to it; I really am lost.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 9, 2020, 7:12pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/2 "2020-05-09T19:12:24Z")

</div>

I’m not sure I understand what you are trying to do here - why do you need `Revise`?

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 9, 2020, 7:38pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/3 "2020-05-09T19:38:44Z")

</div>

You should use `CSV.read` to get a DataFrame.

Also, if you are having problems running your script, I’d probably add in a section where you only take the first 100k rows or something to get a subsample that won’t take as long. That should help you debug.

It is unclear what is going on here but I don’t think the problem is with Revise per se.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 7:40pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/4 "2020-05-09T19:40:34Z")

</div>

I’m just writing code: cleaning and preparing data to, eventually build a model. I need `Revise`, because it seems to be the best way to work in `Julia`. As I said, I am new to the language, so maybe is there something I am missing? My intentions with `Revise` is basically: being able to write the code, test it, fail, correct it, and keep going without having to wait 18s every time I want to run the code just to see the packages getting imported. I also don’t want to do everything on the REPL for a multitude of reasons: no scalability, not being able to save it, not being able to efficiently change something that was done far in the past, etc.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 9, 2020, 7:44pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/5 "2020-05-09T19:44:52Z")

</div>

I guess the question is what do you need to change? `Revise` is usually required for a package development workflow where you are `using` a Package and then want to change source files from that package and see those changes reflected immediately.

For what I think you’re trying to do Revise isn’t really necessary, you can just change one of your functions (e.g. redefine `getmonth(x) = split(x, '.')[3]` if you’ve realised you actually want the third element) and when you call it the next time the new definition will be called without the need to restart Julia.

I’m also not sure why you say you “don’t want to do everything in the REPL”, the development environment seems unrelated to your question? I.e. you can do what I described above in Juno, VSCode, IJulia, or whatever other setup you prefer.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 7:50pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/6 "2020-05-09T19:50:25Z")

</div>

Nice, I didn’t knew `CSV.read`, I used `File` because I saw done in this way somewhere else.

About the first 100k rows, that is a very good idea that I had not though about \*facepalm\*.

I just tested it: first with 100k, same thing. Waited a minute and nothing. Then with 100. Still took long (about 40s) and I got the following error:

```julia
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Warning: likely failure to return to toplevel, try JuliaInterpreter.split_expressions
└ @ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/dEBFI/src/interpret.jl:600
┌ Error: evaluation error starting at /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:23
│ mod = Main
│ ex =
│ quote
│ #= toplevel:20 =#
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:20 =# @time begin
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:21 =#
│ getmonth(x) = begin
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:21 =#
│ (split(x, '.'))[2]
│ end
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:22 =#
│ gsalestrain = groupby(salestrain, :item_id)
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:23 =#
│ salestrainmonth = transform(gsalestrain, :date => (x->begin
│ #= /home/ttv1/Documents/Codes/Julia/Kaggle/PredictFutureSales/processdata.jl:23 =#
│ getmonth.(x)
│ end))
│ end
│ end
│ exception =
│ MethodError: hcat!##kw(::NamedTuple, ::typeof(DataFrames.hcat!), ::DataFrame, ::AbstractDataFrame) is ambiguous. Candidates:
│ (::DataFrames.var"#hcat!##kw")(::Any, ::typeof(DataFrames.hcat!), df1::DataFrame, df2::DataFrame) in DataFrames at /home/ttv1/.julia/packages/DataFrames/3ZmR2/src/dataframe/dataframe.jl:906
│ (::DataFrames.var"#hcat!##kw")(::Any, ::typeof(DataFrames.hcat!), df1::DataFrame, df2::AbstractDataFrame) in DataFrames at /home/ttv1/.julia/packages/DataFrames/3ZmR2/src/dataframe/dataframe.jl:897
│ To resolve the ambiguity, try making one of the methods more specific, or adding a new method more specific than any of the existing applicable methods.
│ Stacktrace:
│ [1] maybe_evaluate_builtin(::JuliaInterpreter.Frame, ::Expr, ::Bool) at /home/ttv1/.julia/packages/JuliaInterpreter/dEBFI/src/builtins.jl:247
└ @ Revise ~/.julia/packages/Revise/MgvIv/src/lowered.jl:106

```

That is some good light.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 9, 2020, 7:55pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/7 "2020-05-09T19:55:58Z")

</div>

On `CSV.read`: It is actually mostly the same thing as what you’re doing, the [full definition](https://github.com/JuliaData/CSV.jl/blob/master/src/CSV.jl) is:

```julia
read(source; copycols::Bool=false, kwargs...) = DataFrame(CSV.File(source; kwargs...), copycols=copycols)

```

I believe it will be deprecated in future CSV releases so that CSV can drop its dependency on DataFrames, so you might as well stick to `DataFrame(CSV.File)`

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 7:57pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/8 "2020-05-09T19:57:27Z")

</div>

I don’t know what I have to change, I’m sorry for not understanding what you’re asking: my background is basically Python; so when I wanted to do something in Python, if was a small project, I would open the interpreter and do it quickly. Now if was something that would involve at least 250 lines of coding, I don’t see why I would not write a script for it.

Even if Revise is not necessary, wouldn’t you agree that it would be nice? I just don’t see the point in hiding from this issue by going around it and write the code in another way.

Sorry, I wasn’t clear, when I said “I don’t want to do everything in the REPL”; I meant that **without** `Revise`. That’s what is nice about `Revise`: I can write my script and use the REPL at the same time. Also, I only use the REPL and Vim, and I still don’t see how can I write a long code in vim without running it many and many times in the process(hence, Revise).

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 9, 2020, 7:57pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/9 "2020-05-09T19:57:50Z")

</div>

As @nilshg said, you don’t need Revise here. You can overwrite previous definitions of these things no problem (not all things, but these things).

Basically, you just want to execute the code that loads and preps that data once. Then the data is loaded in the session and ready for transformation/analysis. Put that code into a separate file if you want, then run that file first. It’s not the Julian way of doing things but it would get it done. Then your other code goes in a separate file, which you can run over and over, but the original data will stay loaded.

What you should move towards is working less at the “global scope” like the code you posted and more with functions.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 9, 2020, 8:02pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/10 "2020-05-09T20:02:57Z")

</div>

I don’t think workflows are dramatically different in Julia and Python for the type of work you’re doing here - you can do this in Juno (Atom) or IJulia in an interactive fashion, iterate quickly, and end up with long-ish structured scripts.

The key difference is as you observe the longer startup time for Julia, so you can’t kill your process and start from scratch as often as in Python. However there isn’t usually a need to restart the process, as you can just redefine functions in the session you’re running (even without Revise) and run them again.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 8:04pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/11 "2020-05-09T20:04:28Z")

</div>

Ok, I guess that would be a good work around. The thing is: when I posted this, I was still ~tweaking the data~ (I stumbled across this problem very soon), hence why the code is all global. Also, there are many datasets, I thought writting a function that reads and returns them all would be just to redundant: why not load it in the beginning and then just keep writing the code? Yes, it would be run again everytime, but after the first time it would run very fast, so to me it would make no difference.

Anyway, your workaround works because I have no problems using `include`, only using `includet`.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 8:10pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/12 "2020-05-09T20:10:03Z")

</div>

But if, for example, I finish some fine work on the REPL, after I am done I am just supposed to dump it? Or, worse, if I am in the shower and my laptop is off the power: the battery dies, so I lose it all? That’s a simple reason for me wanting to write on a text file and not direct into the REPL. Again, I like the REPL because I can do just `bits` of what I intend to do and see if it really does work as I intended, but to expect me to do everything there sound like madness for me.

Maybe I am still not understanding what you’re trying to say, I never used Juno, Atom, IJulia, Vscode or any of the things you mentioned, but I am also not interested in using any other environment besides the REPL, the shell and vim (especially vim, which I am very used to).

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 9, 2020, 8:15pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/13 "2020-05-09T20:15:33Z")

</div>

Haha, yes I think we’re talking past each other - when I’m working on the kind of stuff you’re describing here, I do it in [Juno](https://junolab.org/), which is basically a very fancy text editor (Atom) with a ton of useful Julia specific features (e.g. a plot pane, inline results including a very nice nested clickable view for objects etc).

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 9, 2020, 8:25pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/14 "2020-05-09T20:25:44Z")

</div>

Maybe this post will be useful for you: [julia vim tutorial for newbies](https://discourse.julialang.org/t/julia-vim-tutorial-for-newbies/36636/7). ( but maybe you already know these things)

As others wrote above Revise doesn’t give you anything here (as far as I know). It would be useful if you wrap the pieces of code in your script in functions. Then if you use `includet` the function definitions will be automatically updated the next time you call them from the REPL. As far as I know, the way you are using the script, `include` and `includet` are equivalent in their end effect.

EDIT: maybe I’m wrong with what I wrote above. I thought Revise tracks only changes in function definitions, but from a quick test it seems that it also tracks some changes in stuff happening at top-level (e.g. assigning a new value to a global variable). Anyway I only ever used it for tracking functions so I’m not sure.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 8:27pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/15 "2020-05-09T20:27:03Z")

</div>

Yes, I understand it, I use `vim-slime` so I can send bits of text to the REPL to get inline results and things like that.

Anyway, I wanted to solve this problem because I found `Revise` to be a very nice feature, and I don’t want to circumvent it. Also, think we have a lot of evidence to think the problem here is with `Revise`:

- I can run the script and even `include` it, but not `includet` it.
- `Pkg.test("Revise")` returned a lot of errors, while I didn’t get any errors testing `DataFrames`

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 9, 2020, 8:33pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/16 "2020-05-09T20:33:38Z")

</div>

Does `includet` work for you on other files?  
say a file that contains only `using` statements and function definitions?

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 8:34pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/17 "2020-05-09T20:34:57Z")

</div>

Thanks, but I already have my vim all configured up (including `vim-slime`).

About Revise doesn’t giving me anything, I think that’s not true at all. It’s very clear that my code is not complete. Yes, I can just write it on vim and `vim-slime` it to the REPL, but with Revise it would be one step less. As I said and will say again, I don’t want to run from this problem because there are other forms of working without having to deal with it. Of course I will work on the ways there are currently possible, but giving up a method of work which I consider the best of all the ones that are known to me just because there are others ways of doing it and that particular method is returning some unexpected bugs, that’s not how I do things.

`include` and `includet` are indeed very different in the way I am using the script: there is a lot more script to be written.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [May 9, 2020, 8:40pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/18 "2020-05-09T20:40:20Z")

</div>

I would generally use `includet` for files with just functions, not data or global state. So there Revise will keep your session up-to-date with the functions in that file. Then I’d have a separate file with data and computations and I’d `include` it (possibly many times as I go), to rerun computations as I change or add things, not using Revise.

---

<div class="post-metadata">

**Author:** ![chedieck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chedieck/32/14716_2.png) [@chedieck](https://discourse.julialang.org/u/chedieck)\
**Post date:** [May 9, 2020, 8:42pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/19 "2020-05-09T20:42:34Z")

</div>

Yes it does. The following code, for example:

```julia
using Plots

f(x) = x^2

function myPlot()
    X = 1:0.1:100 
    Y = [f(x) for x in X] 
    plot(X, Y)
end
                             

```

I just ran it with `includet`, tweaked things and worked exactly as expected.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 9, 2020, 8:56pm UTC](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177/20 "2020-05-09T20:56:42Z")

</div>

If you think the issue may be Revise, you should check out the steps here: [Debugging Revise · Revise.jl](https://timholy.github.io/Revise.jl/stable/debugging/)

[Next page](https://discourse.julialang.org/t/revise-cannot-transform-my-dataset/39177.md?page=2)