All of a sudden, HTTP has stopped working to retrieve data from the api.eia.gov website. I have included a simple call that works in R and used to work in Julia but now times out no matter how much time I give it. In R, this responds very quickly.
Any thoughts on how to diagnose this would be appreciated.
using HTTP
HTTP.get(“https://api.eia.gov/v2/seriesid/ELEC.SALES.CO-RES.A?api_key=xxxxx”,
connect_timeout=8,retry=false)
Seems not to be on the endpoint. I get this in a browser
{"response":{"total":24,"dateFormat":"YYYY","frequency":"annual","data":[{"period":2024,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20693.69,"sales-units":"million kilowatt hours"},{"period":2023,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19999.33299,"sales-units":"million kilowatt hours"},{"period":2022,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20593.94201,"sales-units":"million kilowatt hours"},{"period":2021,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20625.00501,"sales-units":"million kilowatt hours"},{"period":2020,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20482.50199,"sales-units":"million kilowatt hours"},{"period":2019,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19404.743,"sales-units":"million kilowatt hours"},{"period":2018,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19286.95201,"sales-units":"million kilowatt hours"},{"period":2017,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18614.99999,"sales-units":"million kilowatt hours"},{"period":2016,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18833.722,"sales-units":"million kilowatt hours"},{"period":2015,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18384.93601,"sales-units":"million kilowatt hours"},{"period":2014,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18092.93201,"sales-units":"million kilowatt hours"},{"period":2013,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18528.908,"sales-units":"million kilowatt hours"},{"period":2012,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18220.36501,"sales-units":"million kilowatt hours"},{"period":2011,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18276.79899,"sales-units":"million kilowatt hours"},{"period":2010,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":18102.386,"sales-units":"million kilowatt hours"},{"period":2009,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17412.63,"sales-units":"million kilowatt hours"},{"period":2008,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17720.491,"sales-units":"million kilowatt hours"},{"period":2007,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":17634.182,"sales-units":"million kilowatt hours"},{"period":2006,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":16951.534,"sales-units":"million kilowatt hours"},{"period":2005,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":16436.381,"sales-units":"million kilowatt hours"},{"period":2004,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15532.168,"sales-units":"million kilowatt hours"},{"period":2003,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15724.839,"sales-units":"million kilowatt hours"},{"period":2002,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":15424.846,"sales-units":"million kilowatt hours"},{"period":2001,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":14470.251,"sales-units":"million kilowatt hours"}],"description":"Electricity sales to ultimate customer by state and sector (number of customers, average price, revenue, and megawatthours of sales). \n Sources: Forms EIA-826, EIA-861, EIA-861M","id":"retail-sales"},"request":{"command":"\/v2\/seriesid\/ELEC.SALES.CO-RES.A","params":{"frequency":"annual","data":["sales"],"facets":{"sectorid":["RES"],"stateid":["CO"]},"start":null,"end":null,"sort":[{"column":"period","direction":"desc"}],"offset":0,"length":5000,"api_key":"jj5NBhhIogCYhpgHCfeF5osMLAmqf7GbxSnGgtV7"},"route":"electricity\/retail-sales"},"apiVersion":"2.1.10","ExcelAddInVersion":"2.1.0"}
and from HTTP, I get
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Date: Tue, 02 Dec 2025 01:21:11 GMT
Content-Type: application/json
Content-Length: 795
Connection: keep-alive
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, X-Params
Access-Control-Allow-Methods: GET,POST,PATCH,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Age: 3
Cache-Control: max-age=0, no-cache, no-store
Content-Encoding: gzip
Expires: Tue, 02 Dec 2025 01:21:11 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding, Accept-Encoding
Via: https/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Api-Umbrella-Request-Id: cqrij2jlporgvtlkkicg
X-Cache: MISS
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 6efaf6bf-6e19-4e51-5f9f-c7521bab71d6
X-Frame-Options: SAMEORIGIN
{"response":{"total":24,"dateFormat":"YYYY","frequency":"annual","data":[{"period":2024,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20693.69,"sales-units":"million kilowatt hours"},{"period":2023,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19999.33299,"sales-units":"million kilowatt hours"},{"period":2022,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20593.94201,"sales-units":"million kilowatt hours"},{"period":2021,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20625.00501,"sales-units":"million kilowatt hours"},{"period":2020,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20482.50199,"sales-units":"million kilowatt hours"},{"period":2019,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales
⋮
4592-byte body
Thanks. It’s helpful to know. Cal you tell me what version of HTTP you used for your request?
I’m on [cd3eb016] HTTP v1.10.19 under 1.12
Webservers can be fickle. They might throttle requests based on certain user agents or IPs or other fingerprints. Try matching the exact headers (but especially including user agent and Accept).
Right you are. I connected through my university’s VPN and everything is working. When I disconnect from the VPN, the tests fail. The tests in R work in both cases. No idea why the ‘get’ request would work in R but not in Julia from my default IP address, but at least I have it working again. Thanks for the tip.
The best way to diagnose things like this, at least on the Julia side is to pass verbose=2 to your HTTP.get call. That will show you the exact HTTP text that is sent over the wire. If there’s a way to do the same in R, then you could actually compare the requests, line by line. It doesn’t sound to me like the issue could be VPN; it’s not like R can magically get around a VPN issue. Either the webserver is accessible or not. As Matt said, it could be related to how the client is presenting itself via HTTP.jl (via user agent header, ip, etc.) or you may have some kind of environment proxy settings setup that R is picking up but HTTP.jl isn’t? (we do support the normal linux-y PROXY env variables, but very possible we’re missing a flavor/variation).
But for me, I always have to start w/ the raw text getting sent over the wire and then work from there.
Helpful, thank you. I added verbose=2. I still couldn’t get a response from HTTP in Julia with no VPN. So, I connected via the VPN and did get the headers (see below). Then I retrieved the headers from the same request in R and included those below. I don’t know enough about what these headers mean to know which differences might matter. Fortunately, I can reproduce this problem at will, but unfortunately, I don’t get any helpful information from the verbose switch when I don’t have the VPN connection engaged; the request just times out and returns an error. I can send along the error listing if that would be helpful.
============================
Julia HTTP verbose (with VPN connection)
Access-Control-Allow-Origin: *
Age: 5
Cache-Control: max-age=0, no-cache, no-store
Content-Encoding: gzip
Expires: Tue, 02 Dec 2025 17:02:45 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding, Accept-Encoding
Via: https/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Api-Umbrella-Request-Id: cqs022lpr6nt70ch8dng
X-Cache: MISS
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 6138e534-8950-4fdf-7e51-da360c53f3b6
X-Frame-Options: SAMEORIGIN
{“response”:{“total”:24,“dateFormat”:“YYYY”,“frequency”:“annual”,“data”:[{“period”:2024,“stateid”:“CO”,“stateDescription”:“Colorado”,“sectorid”:“RES”,“sectorName”:“residential”,“sales”:20693.69,“sales-units”:“million kilowatt hours”},
⋮
4584-byte body
“”"
========================
R httr::GET headers reported in the query response (with VPN connection engaged)
date: “Tue, 02 Dec 2025 16:36:19 GMT”
content-type: “application/json”
access-control-allow-headers: “Origin, X-Requested-With, Content-Type, Accept, Authorization, X-Params”
access-control-allow-methods: “GET,POST,PATCH,PUT,DELETE,OPTIONS”
access-control-allow-origin: “*”
age: “2”
cache-control: “max-age=0, no-cache, no-store”
content-encoding: “gzip”
expires: “Tue, 02 Dec 2025 16:36:19 GMT”
pragma: “no-cache”
strict-transport-security: “max-age=31536000; includeSubDomains; preload”
vary: “Accept-Encoding”
vary: “Accept-Encoding”
via: “https/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])”
x-api-umbrella-request-id: “cqrvm0a7tj8nvi875700”
x-cache: “MISS”
x-content-type-options: “nosniff”
x-vcap-request-id: “67ff987c-3e97-44d3-56fe-1b5ef262d338”
x-frame-options: “SAMEORIGIN”
attr(,“class”): “insensitive” “list”
I just signed up for an API key and was able to pull the data via HTTP.jl without any issues…according to Claude, the most likely culprit is that your “network security appliance sees Julia’s User-Agent as ‘suspicious’ or ‘bot-like’” If that’s the case, you can try this:
headers = ["User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"]
res = HTTP.get(
"https://api.eia.gov/v2/seriesid/ELEC.SALES.CO-RES.A?api_key=YOUR_API_KEY",
headers=headers
)
Here’s what I get with verbose without VPN
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Date: Tue, 02 Dec 2025 22:35:14 GMT
Content-Type: application/json
Content-Length: 795
Connection: keep-alive
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, X-Params
Access-Control-Allow-Methods: GET,POST,PATCH,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: max-age=0, no-cache, no-store
Content-Encoding: gzip
Expires: Tue, 02 Dec 2025 22:35:14 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding, Accept-Encoding
Via: https/1.1 api-umbrella (ApacheTrafficServer [cMsSf ])
X-Api-Umbrella-Request-Id: cqs4qa37irh369h55u80
X-Cache: MISS
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 0455e6fc-81c2-4561-7dab-1ecb646ec556
X-Frame-Options: SAMEORIGIN
{"response":{"total":24,"dateFormat":"YYYY","frequency":"annual","data":[{"period":2024,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20693.69,"sales-units":"million kilowatt hours"},{"period":2023,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":19999.33299,"sales-units":"million kilowatt hours"},{"period":2022,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20593.94201,"sales-units":"million kilowatt hours"},{"period":2021,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20625.00501,"sales-units":"million kilowatt hours"},{"period":2020,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales":20482.50199,"sales-units":"million kilowatt hours"},{"period":2019,"stateid":"CO","stateDescription":"Colorado","sectorid":"RES","sectorName":"residential","sales
⋮
4592-byte body
Thanks for the replies.
(1) I tried some different User-Agent headers for my setup. This did not solve the timeout.
(2) With the VPN disconnected, I have been able to retrieve the data in Firefox and Safari by entering the URL directly.
(3) I have also retrieved the data with R with httr::GET and with the following Python code:
import requests
import json
EIA_URL = “https://api.eia.gov/v2/seriesid/ELEC.SALES.CO-RES.A?api_key=xxxxxxx”
response = requests.get(EIA_URL, timeout=8)
data = response.json()
temp = data.get(“response”, {})
print(temp)
All of the other retrieval methods have worked. The only timeout error occurs with my Julia HTTP setup when not on the VPN. I am using Julia 1.12, and the latest version of HTTP. I have run the code from the command line and from within VSCode. I am running OSX 15.7.2 on M2 hardware. I don’t know why any of this would matter, but I pass this information along for completeness.
I think that fairly conclusively points to your IP block being on the endpoint’s blacklist since it works on VPN.
Well, that doesn’t seem to explain why the requests from R, Python, Safari and Firefox all went through. They are from the same (non-VPN) IP address.
Missed that. Well, it’s not that you’re on macOS. I’m on Tahoe.