CSV file is being uploaded (multipart/form-data) to my HTTP service. My task is to read content of that file and store it in PostgreSQL table. HTTP request body contains raw data (content of CSV file/files). How to parse & store it in the database table the most efficient way? julia> using HTTP jul…

You can use the write_table! function from https://github.com/lungben/TableIO.jl It implements the COPY FROM STDIN functionality using CSV.jl and LibPQ.jl. https://github.com/lungben/TableIO.jl/blob/ce1f205053cee0270ae2b53c0787d576f70e0c4e/src/postgresql.jl#L32

Welcome to the community! :smiley: As far as parsing is concerned, have a look at CSV.jl . For writing to your database, take a look at LibPQ.jl .

You should use bulk insert. It was discussed in this thread: How to quickly bulk insert into postgres - #6 by lbilli

Thank you for you recommendations. Now, I can clarify my issue more in detail. My CSV file consists of: 1C;2C;3C;4C;5C 1D;2D;3D;4D;5D and it is sent through multipart/form-data, stored in request body (variable: body): f = CSV.File(body, header=0, delim=';', skipto = 4, footerskip = 1) for row i…

Finally, thank you guys. @lungben , I checked your repositories. It looks very interesting. I already found solution, what I wanted to do. f = CSV.File(body, header=0, delim=';', skipto = 4, footerskip = 1) df = f |> DataFrame row_strings = map(eachrow(df)) do row s = "" for c in 1:length…

Your solution may have issues with some special characters because no escaping is done. Problematic could especially be , and " as DaraFrame contents. Using the CSV.jl row iterator CSV.RowWriter(table) has the advantage that escaping is done for you.

Good to know that. I’ll check it and improve my solution. Thank you.

HTTP request body -> PostgreSQL

New to Julia

Jan_Dolinsky February 9, 2021, 2:54pm 4

This is also a useful read: Upload of DataFrame data to databases - #5 by mmiller

1 Like

Topic		Replies	Views
How to quickly bulk insert into postgres Data	14	22158	September 10, 2019
PostgreSQL in Julia: LibPQ.jl Data package , data	9	5975	July 5, 2018
Given a PostgresSQL.jl connection string, what's the easiest way to create a table in the database by copying a dataframe into it? Data	6	930	September 14, 2021
Upload of DataFrame data to databases Data	6	2466	October 19, 2020
How to quickly bulk insert into postgres 1 Data	3	664	August 29, 2019

HTTP request body -> PostgreSQL

Related topics