This is sort of a shot in the dark, I don’t have any reason to believe there’s a better solution to this but I thought I’d give it a try.
I’m trying to insert data into tables on a postgres database. I’m using the excellent LibPQ.jl doing
Data.stream!(df, LibPQ.Statement, cnxn, str)
str is an insert statement and
df is the
DataFrame I want to upload.
Everything works perfectly fine except that it is horrifyingly slow. It’s taking it something like 20 minutes to upload just
5e4 rows. I know that it’s not a connection speed issue since I get plenty of bandwidth on queries. It also seems highly unlikely that this is a Julia or LibPQ performance issue (other than that I might be using a sub-optimal function for doing this).
Does anyone know of some approach that’s not incredibly slow? I feel like this can’t possibly be the way people populate postgres databases…