# CSV ruins scientific notation

**URL:** <https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835>\
**Category:** General Usage\
**Tags:** dataframes, csv\
**Created:** [June 29, 2019, 8:39am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835 "2019-06-29T08:39:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [June 29, 2019, 8:39am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/1 "2019-06-29T08:39:43Z")

</div>

I think it’s a bug in CSV but I’m not 100% sure that’s why I ask here first:

I wrote some code to convert a txt file into a csv file (basically adding a header and commata).

My code:

```julia
fname = input_name = split(split(txt_file,"/")[end], ".")[1]

header = ["x","y","z","d"]

df = CSV.read(txt_file, delim=" ", header=header)
println(df)
CSV.write("../inputs/csv/"*fname*".csv", df)

```

The txt file:

```julia
1234.56789 4232.45455455 -8789.5455 1.0121325235677855e-11

```

The print out:

```julia

fname: test_123
1×4 DataFrame
│ Row │ x │ y │ z │ d │
│ │ Float64 │ Float64 │ Float64 │ Float64 │
├─────┼─────────┼─────────┼──────────┼─────────────┤
│ 1 │ 1234.57 │ 4232.45 │ -8789.55 │ 1.01213e-11 │

```

The csv file:

```julia
x,y,z,d
1234.56789,4232.45455455,-8789.5455,10121325235677855e-27

```

It should have the same scientific representation (the true scientific notation maybe if the other one has more than one digit in front of the decimal point) in my opinion.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 29, 2019, 8:53am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/2 "2019-06-29T08:53:16Z")

</div>

Is your problem the lack of read-write invariance (which I guess cannot be helped), or that the scientific notation is not normalized, eg as in this MWE:

```julia
using CSV, Tables
x = 10121325235677855e-27
io = IOBuffer()
CSV.write(io, [(x = x, )])
String(take!(io))

```

---

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [June 29, 2019, 8:56am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/3 "2019-06-29T08:56:10Z")

</div>

I think CSV should not change the scientific notation or if not possible normalize it to the standard scientific notation but in my case it seems to have a different “standard”

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 29, 2019, 9:18am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/4 "2019-06-29T09:18:47Z")

</div>

> [@Wikunia](#):
>
> I think CSV should not change the scientific notation

Note that eg a `Float64` representation does not contain this information, eg

```julia
julia> parse(Float64, "1.1") == parse(Float64, "11e-1")
true

```

so this expectation is a bit unrealistic.

> [@Wikunia](#):
>
> or if not possible normalize it to the standard scientific notation

Note that it is called [normalized notation](https://en.wikipedia.org/wiki/Scientific_notation#Normalized_notation), but it is not standard in any sense (though quite common). All representations are equally valid.

I wonder if this is a practical concern. Doing it the way it is [currently implemented](https://github.com/JuliaData/CSV.jl/blob/5d756a2c68fba4085c054aa2f7a94c182cbc0d8f/src/write.jl#L65) is extremely efficient and also happens to be valid. Is there a program that has problems with the output?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [June 29, 2019, 10:04am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/5 "2019-06-29T10:04:41Z")

</div>

The integer value to a large power is a bit weird. I wonder if it’s due to printing the shortest text representation of the value. That seemed like a good idea at one point but seems a bit weird now.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 29, 2019, 10:57am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/6 "2019-06-29T10:57:28Z")

</div>

I think it comes from this [this PR](https://github.com/JuliaData/CSV.jl/pull/406), and the rationale seems to be being able to use the output from `grisu` directly.

I am fine with it either way. I am also more use to eyeballing normalized notation, but I rarely look at CSV directly (usually when I am debugging), so I as long as it is valid, it should be fine.

---

<div class="post-metadata">

**Author:** ![Wikunia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wikunia/32/2180_2.png) [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Post date:** [June 29, 2019, 1:21pm UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/7 "2019-06-29T13:21:32Z")

</div>

Thanks for your reply. It’s valid just seems odd. I shouldn’t have called it bug it’s more unexpected behavior but it’s okay if it stays the way it is.

---

<div class="post-metadata">

**Author:** ![pfarndt](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@pfarndt](https://discourse.julialang.org/u/pfarndt)\
**Post date:** [July 1, 2019, 10:32am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/8 "2019-07-01T10:32:05Z")

</div>

> [@Tamas\_Papp](#):
>
> I rarely look at CSV directly (usually when I am debugging), so I as long as it is valid, it should be fine.

Just to have a different option here - I look at CSV files a lot - all my outputs are basically in CSV - and I often look into them to check whether or how fast some algorithm converged (one may call this debugging 😉 ). For this purpose only the order of magnitude of some numbers is important. Now it would be really annoying if I now have to count digits before the `.` and add this number to exponent before I can come to a conclusion about the order of magnitude of a number. This is why we use [normalized notation](https://en.wikipedia.org/wiki/Scientific_notation#Normalized_notation).

---

<div class="post-metadata">

**Author:** ![eljungsk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eljungsk/32/2865_2.png) [@eljungsk](https://discourse.julialang.org/u/eljungsk)\
**Post date:** [July 1, 2019, 11:37am UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/9 "2019-07-01T11:37:59Z")

</div>

> [@Tamas\_Papp](#):
>
> I wonder if this is a practical concern. Doing it the way it is [currently implemented](https://github.com/JuliaData/CSV.jl/blob/5d756a2c68fba4085c054aa2f7a94c182cbc0d8f/src/write.jl#L65) is extremely efficient and also happens to be valid. Is there a program that has problems with the output?

I had this problem recently when trying to read data generated in Julia into a commercial CFD simulation software. The data were silently corrupted, and it took me a while to figure out the reason since most of the numbers looked reasonable when inspecting them in the CFD package.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [July 1, 2019, 12:52pm UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/10 "2019-07-01T12:52:52Z")

</div>

In this case, I think the best route is to open an issue for this package to emit normalized floats.

---

<div class="post-metadata">

**Author:** ![steph\_de\_paris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steph_de_paris/32/7780_2.png) [@steph\_de\_paris](https://discourse.julialang.org/u/steph_de_paris)\
**Post date:** [July 17, 2019, 5:52pm UTC](https://discourse.julialang.org/t/csv-ruins-scientific-notation/25835/11 "2019-07-17T17:52:41Z")

</div>

See a solution (not really “smart”) proposed in [New behaviour due to an update of the package CSV when using CSV.write - #8 by steph\_de\_paris](https://discourse.julialang.org/t/new-behaviour-due-to-an-update-of-the-package-csv-when-using-csv-write/26442/8)

I just use a cast Float 2 String to keep the scientific notation used in the DataFrames.