# Is this an efficient way to write some information into a .csv file using julia?

**URL:** <https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103>\
**Category:** General Usage\
**Tags:** csv, namedtuple, structarrays\
**Created:** [January 25, 2019, 5:59pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103 "2019-01-25T17:59:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bsnyh](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@bsnyh](https://discourse.julialang.org/u/bsnyh)\
**Post date:** [January 25, 2019, 5:59pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/1 "2019-01-25T17:59:34Z")

</div>

Is the following efficient? I was mainly concerned about the `push!` function.

```julia
function writeToFile(filename::String, am::Vector{Am})
	df=DataFrame(name1=Int[], name2=Int[], name3=Int[])
	for a in am
		push!(df,[a.name1,a.name2,a.name3])
        end
        CSV.write(filename,df)
end

```

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [January 25, 2019, 6:02pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/2 "2019-01-25T18:02:40Z")

</div>

```julia
CSV.write(filename, am)

```

should work already if `am` is a `Vector` of `NamedTuples`.

---

<div class="post-metadata">

**Author:** ![bsnyh](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@bsnyh](https://discourse.julialang.org/u/bsnyh)\
**Post date:** [January 25, 2019, 7:05pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/3 "2019-01-25T19:05:05Z")

</div>

@piever. What do you mean by a `NamedTuples`?

I tried a toy example and got an error.

```julia
using CSV
mutable struct myType
	index::Int # Int64
        sIndex::Int # Int64
        myType() = new(0,0)
end

function writeToFile(filename::String, a::Vector{myType})
	@assert length(a)>0
        CSV.write(filename,a)
end

function main()
   aVector = Vector{myType}(6)
   for i in 1:6
	   aVector[i] = myType()
	   aVector[i].index=i
	   aVector[i].sIndex = i
   end
   writeToFile("path to my csv", aVector)
end

main()

```

I got an error, saying that "ArgumentError: no default `Tables.row` implementation for type: Array{myType, 1}.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [January 25, 2019, 9:06pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/4 "2019-01-25T21:06:13Z")

</div>

The following only applies if you have the latest version of Tables.jl (which is 0.1.14), I’m not sure it will work in earlier versions.

A `NamedTuple` is an object of type `(a=1, b="something")`: like a `Tuple` except it has names. An `Array` of `NamedTuples` (for example `v = [(a=i, b=i+1) for i in 1:3]`) can be considered a table and therefore you can do `CSV.write(filename, v)`. I thought `am` was a vector of named tuples. If it isn’t, instead of allocating a `DataFrame` (in case your data is really big and you are afraid this could slow things down) you can simply generate a `NamedTuple` iterator and write it to CSV.

```julia
itr = ((name1=a.name1, name2=a.name2, name3=a.name3) for a in am)
CSV.write(filename, itr)

```

This way you avoid allocating the `DataFrame` and the data is streamed directly from your vector to the CSV. Note that `itr` is probably a better way to convert to `DataFrame`. For example you could just do `DataFrame(itr)` and get the `DataFrame` (avoiding the temporary vectors `[a.name1, a.name2, a.name3]`).

---

<div class="post-metadata">

**Author:** ![bsnyh](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@bsnyh](https://discourse.julialang.org/u/bsnyh)\
**Post date:** [January 25, 2019, 9:43pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/5 "2019-01-25T21:43:15Z")

</div>

@piever, is there a corresponding reversed way? I mean, to read a .csv file into a `NamedTuple`, and then to create a type/mutable struct based on it. 😉

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [January 26, 2019, 1:04pm UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/6 "2019-01-26T13:04:29Z")

</div>

To consider a `DataFrame` as an iterator of rows you can simply do:

```julia
struct MyType
    x::Int
    y::Float64  
end  
df = CSV.read(filename)
struct_vec = map(row -> MyType(row.x, row.y), Tables.rows(df))

```

As an alternative, if your data is stored as a `DataFrame` and therefore in columnar storage (it is basically a list of vectors), you can convert it to a named tuple of arrays with `Tables.columntable(df)` and then create a [StructArray](https://github.com/piever/StructArrays.jl) from it:

```julia
using StructArrays
cols = Tables.columntable(df)
StructArray{MyType}(cols)

```

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [January 27, 2019, 5:38am UTC](https://discourse.julialang.org/t/is-this-an-efficient-way-to-write-some-information-into-a-csv-file-using-julia/20103/7 "2019-01-27T05:38:17Z")

</div>

Yes, note that you can do `table = CSV.File(file) |> Tables.columntable` to materialize a csv file into a NamedTuple of Vectors directly, no intermediate construction into a DataFrame. In general, you can directly materialize a csv file into any Tables.jl-compatible “sink” function, like `SQLite.load!`, `Feather.write`, `IndexedTables.table`, etc. It makes it nice to not have to always intermediate through DataFrames if that’s not needed.