I’m looking at using Julia to populate an in memory data structure. My first attempt will be to gather the data from the Fed’s yield curve site. There are multiple models, different refresh rates and lashings of nice processing challenges to let me have some fun. The problem is that Julia seems to offer such a vast array of packages I can’t figure out which is the most suitable to use. My hardware environment will be a Proxmox virtual machine on a hardwired lan.
Here’s a brief synopsis of what I “think” I need to do.
1 access Federal Reserve Board - Yield Curve Models and Data
2 grab the most recent csv files into a temp memory area
3 compare the temp memory area with the current active area and throw away stuff I already have
4 add the new data to the in memory structure
Can someone advice me as to which packages to look at please. I am thinking Apache arrow for the in memory model.
thank you
If you just want the yield curves you can look at one of my packages.
https://github.com/tbeason/DailyTreasuryYieldCurve.jl
I’m not really familiar with the difference between the rates that my package gets and what is available at the site you posted. It looks like that site just posts updated CSV files every so often, if that is the case you should just use RemoteFiles.jl to download them on a set schedule and CSV.jl to read them.
None of the files seemed too large so I’m not sure why you sound very concerned about memory. I would just fetch the new file and overwrite the old rather than try to sort out any differences.
5 Likes
WOW! that was FAST!!! I am embarrassed as I didn’t search for a package that ACTUALLY GETS THE RATES 
I am so looking forwards to seeing how you did this. I learn by example. I am always concerned about memory as it’s my way of figuring out how a language works. I think your idea of just overwriting it is great for the first pass then mess around later on.
thanks again for the FAST solution to my problem. Lashings to look at next week.
1 Like