All of a sudden, HTTP has stopped working to retrieve data from the api.eia.gov website. I have included a simple call that works in R and used to work in Julia but now times out no matter how much time I give it. In R, this responds very quickly.
Any thoughts on how to diagnose this would be appreciated.
using HTTP
HTTP.get(“https://api.eia.gov/v2/seriesid/ELEC.SALES.CO-RES.A?api_key=xxxxx”,
connect_timeout=8,retry=false)
Seems not to be on the endpoint. I get this in a browser
{"response":{"total":24,"dateFormat":"YYYY","frequency":"annual","data":[{"period":2024,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20693.69,"sales-units":"million kilowatt hours"},{"period":2023,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19999.33299,"sales-units":"million kilowatt hours"},{"period":2022,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20593.94201,"sales-units":"million kilowatt hours"},{"period":2021,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20625.00501,"sales-units":"million kilowatt hours"},{"period":2020,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20482.50199,"sales-units":"million kilowatt hours"},{"period":2019,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19404.743,"sales-units":"million kilowatt hours"},{"period":2018,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19286.95201,"sales-units":"million kilowatt hours"},{"period":2017,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18614.99999,"sales-units":"million kilowatt hours"},{"period":2016,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18833.722,"sales-units":"million kilowatt hours"},{"period":2015,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18384.93601,"sales-units":"million kilowatt hours"},{"period":2014,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18092.93201,"sales-units":"million kilowatt hours"},{"period":2013,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18528.908,"sales-units":"million kilowatt hours"},{"period":2012,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18220.36501,"sales-units":"million kilowatt hours"},{"period":2011,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18276.79899,"sales-units":"million kilowatt hours"},{"period":2010,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18102.386,"sales-units":"million kilowatt hours"},{"period":2009,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17412.63,"sales-units":"million kilowatt hours"},{"period":2008,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17720.491,"sales-units":"million kilowatt hours"},{"period":2007,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17634.182,"sales-units":"million kilowatt hours"},{"period":2006,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":16951.534,"sales-units":"million kilowatt hours"},{"period":2005,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":16436.381,"sales-units":"million kilowatt hours"},{"period":2004,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15532.168,"sales-units":"million kilowatt hours"},{"period":2003,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15724.839,"sales-units":"million kilowatt hours"},{"period":2002,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15424.846,"sales-units":"million kilowatt hours"},{"period":2001,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":14470.251,"sales-units":"million kilowatt hours"}],"description":"Electricity sales to ultimate customer by state and sector (number of customers, average price, revenue, and megawatthours of sales). \n Sources: Forms EIA-826, EIA-861, EIA-861M","id":"retail-sales"},"request":{"command":"\/v2\/seriesid\/ELEC.SALES.CO-RES.A","params":{"frequency":"annual","data":["sales"],"facets":{"sectorid":["RES"],"stateid":["CO"]},"start":null,"end":null,"sort":[{"column":"period","direction":"desc"}],"offset":0,"length":5000,"api_key":"jj5NBhhIogCYhpgHCfeF5osMLAmqf7GbxSnGgtV7"},"route":"electricity\/retail-sales"},"apiVersion":"2.1.10","ExcelAddInVersion":"2.1.0"}
and from HTTP, I get
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Date: Tue, 02 Dec 2025 01:21:11 GMT
Content-Type: application/json
Content-Length: 795
Connection: keep-alive
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, X-Params
Access-Control-Allow-Methods: GET,POST,PATCH,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Age: 3
Cache-Control: max-age=0, no-cache, no-store
Content-Encoding: gzip
Expires: Tue, 02 Dec 2025 01:21:11 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding, Accept-Encoding
Via: https/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Api-Umbrella-Request-Id: cqrij2jlporgvtlkkicg
X-Cache: MISS
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 6efaf6bf-6e19-4e51-5f9f-c7521bab71d6
X-Frame-Options: SAMEORIGIN
{"response":{"total":24,"dateFormat":"YYYY","frequency":"annual","data":[{"period":2024,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20693.69,"sales-units":"million kilowatt hours"},{"period":2023,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19999.33299,"sales-units":"million kilowatt hours"},{"period":2022,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20593.94201,"sales-units":"million kilowatt hours"},{"period":2021,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20625.00501,"sales-units":"million kilowatt hours"},{"period":2020,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20482.50199,"sales-units":"million kilowatt hours"},{"period":2019,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales
⋮
4592-byte body
Thanks. It’s helpful to know. Cal you tell me what version of HTTP you used for your request?
I’m on [cd3eb016] HTTP v1.10.19 under 1.12
Webservers can be fickle. They might throttle requests based on certain user agents or IPs or other fingerprints. Try matching the exact headers (but especially including user agent and Accept).
Right you are. I connected through my university’s VPN and everything is working. When I disconnect from the VPN, the tests fail. The tests in R work in both cases. No idea why the ‘get’ request would work in R but not in Julia from my default IP address, but at least I have it working again. Thanks for the tip.
The best way to diagnose things like this, at least on the Julia side is to pass verbose=2 to your HTTP.get call. That will show you the exact HTTP text that is sent over the wire. If there’s a way to do the same in R, then you could actually compare the requests, line by line. It doesn’t sound to me like the issue could be VPN; it’s not like R can magically get around a VPN issue. Either the webserver is accessible or not. As Matt said, it could be related to how the client is presenting itself via HTTP.jl (via user agent header, ip, etc.) or you may have some kind of environment proxy settings setup that R is picking up but HTTP.jl isn’t? (we do support the normal linux-y PROXY env variables, but very possible we’re missing a flavor/variation).
But for me, I always have to start w/ the raw text getting sent over the wire and then work from there.