How to write in .parquet (or any compressed extension)

Caique_Rodrigues
Sun, 29 Aug 2021 05:34:52 +0000
General Usage

I’m doing some simulation that the output can be around 1GB to 10GB, it’s not so big but we are probably going to run lots of simulations. Right now we are saving the output in .txt, but I’d like to write it in a better extension and I thought about .parq. I’ve seen the Parquet.jl but I don’t think I understood how to write with it. Here is how I am saving my files today:

using DelimitedFiles

aux1=10
aux2=20
aux3=30
header = [
    "aux1 = "*string(aux1)*"; This is aux1 in units of a.u.1"
    "aux2 = "*string(aux2)*"; This is aux2 in units of a.u.2"
    "aux3 = "*string(aux3)*"; This is aux3 in units of a.u.3"
]

filename = "testing_file"
format = ".txt"
delimiter = ','

output1_vec = collect(1:1:100)
output2_vec = rand(Complex{Float64}, 100)
output3_vec = rand(100)*1e21
output4_vec = rand(100)*1e2
data = [output1_vec output2_vec output3_vec output4_vec] # in matrix form
data_label = ["vec1 [units of au1]", "vec2 [units of au2]", "vec3 [units of au3]", "vec4 [units of au4]"] # in vector form

#-----------------------

open(filename*format; write=true) do f
    for i in header
        write(f, i*"\n")
    end
    write(f, "----------------------------------------\n")
    for i in data_label
        write(f, i)
        if i != data_label[end]
            write(f, ",")
        else
            write(f, "\n")
        end
    end
    writedlm(f, data, delimiter)
end

Which is basically a header and four columns with complex numbers. Any chance to put this in a more compressed extensions?

4 posts - 2 participants