Best way to download data from a remote URL directly into JuliaDB

General Usage Performance
1

I am trying to download a time series from a remote URL (the European Central Bank Statistical Data Warehouse) into JuliaDB. Would like to simply pass the URL to a loadtable function. I am able to load ttime-series into JuliaDB but only after downloading them using Request.get and saving them to a file. Which seems very inefficient and ugly… This is the code I have tried:

using Requests, JuliaDB
function ecb(ts::String)
    resp = Requests.get(ts)
    Requests.save(resp,"resp.csv")
    eon2=loadtable("resp.csv", skiplines_begin=5, header_exists=false,  colnames=["Date","Eonia"])
    println(eon2[1])
end  
tseries = "198.EON.D.EONIA_TO.RATE"
ecb("http://sdw.ecb.europa.eu/quickviewexport.do?SERIES_KEY=$(tseries)&type=csv")
# Output: (Date = 2018-04-18, Eonia = -0.368)

Thank you very much

2

Not sure of the syntax precisely, but I bet some combination of IterableTables and FileIO will get you there. They have a source/sink model, and I’m relatively certain any sort of bitstream can be used as the source.

3

To answear my own question one can do the following to load a CSV file from a remote URL into JuliaDB (example for a CSV file with two columns, skipping lines 1-5):

using HTTP, JuliaDB
function ecb2(ts)
    newtable = csvread(IOBuffer(HTTP.get(ts).body), skiplines_begin=5, header_exists=false)
    eon2 = table(newtable[1][1], newtable[1][2], names = [:Date,:Eonia])
    println(eon2[1])
end
2 Likes