I’ve struggled to pin down an mwe for this. The problem is that an HTTP query to the ONS Open Geography sparql endpoint fails intermittently - about one time in two or three - so very often.
I don’t think the query is at fault because I can run it at the interactive sparql query page, here, and it runs every time without problem. Instead, I think I’m creating a context in my code that makes the query fail.
Specifically, the problem is related to using an IOBuffer when reading a file from a zip archive. Here is my code:
using HTTP
using JSON3
using ZipFile
const global pcd = ["BS34 5BZ", "WS9 8TP", "DY1 4AL", "WC2H 9JQ", "BD10 8PW", "NE24 3PA", "IP3 9SX", "G43 1AT", "CF83 1BJ", "TF10 9AT", "CT1 2EH", "CF10 3RB", "CB22 4QR", "CB7 4LS", "NE20 0DX", "CT16 1HU", "TW1 2NL", "ME13 8PW", "SE1 7EW", "WD3 9SL", "CH6 5TW", "W1W 7LT", "SE1 2JH", "BT3 9DT", "M11 4DQ", "KT8 9AU", "LS17 9LG", "HA5 1AE", "BT26 6HR", "LS6 1JD", "BS6 5EE", "FK14 7PL", "B43 7AH", "BS20 0HH", "CF71 7BG", "SA32 8HN", "WF4 4RH", "WR8 9DW", "LL13 0YT", "TR11 5JZ", "RM13 9YN", "KA7 4PQ", "NE26 4QR", "TN17 2AB", "WR12 7JU", "CH7 5LE", "GU32 3HX", "SE1 8XX", "G34 9BN", "YO15 1JF", "LL31 9XZ", "W1J 0BD", "CV37 6BB", "NG12 2LU", "BS1 6TY", "EC4R 1AG", "DY10 1QR", "CV37 6HB", "S1 4FW", "CH5 1DR", "SY1 2SZ", "TBC", "NN12 8TN", "BT48 9PJ", "CF43 4RB", "NN12 7SE", "CV35 0BJ", "NW3 1HS", "PL24 2SG", "FK2 7ZT", "HD1 3DH", "TR11 3QY", "BT48 6RG", "GU3 1DQ", "HR2 8AQ", "HA9 0WS", "S62 7TQ", "RG28 7AL", "TW2 7BA", "WF4 4LG"]
function vectortostring(v)
s = ""
for i in v
s !== "" ? s = s * "\", \"" * string(i) : s = s * string(i)
end
return s
end
function postcodeLonLat(PCS)
query = """
PREFIX pcd: <http://statistics.data.gov.uk/def/postcode/unit#>
PREFIX geog: <http://statistics.data.gov.uk/def/hierarchy/best-fit#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX pos: <http://www.w3.org/2003/01/geo/wgs84_pos#>
PREFIX gridNI: <http://statistics.data.gov.uk/def/spatialrelations/epsg-29903/>
PREFIX gridGB: <http://statistics.data.gov.uk/def/spatialrelations/epsg-27700/>
PREFIX foi: <http://publishmydata.com/def/ontology/foi/>
SELECT ?display ?lat ?lon ?eastNI ?northNI ?eastGB ?northGB ?cde ?nme
WHERE
{
?s rdf:type <http://statistics.data.gov.uk/def/postcode/unit> .
?s pcd:postcode1space ?display .
filter(?display in ("$PCS"))
?s geog:country ?ctry ;
pos:lat ?lat ;
pos:long ?lon .
OPTIONAL {?s gridNI:easting ?eastNI ;
gridNI:northing ?northNI .}
OPTIONAL {?s gridGB:easting ?eastGB ;
gridGB:northing ?northGB .}
?ctry foi:code ?cde ;
foi:displayName ?nme .
}
"""
encoded_query = HTTP.escapeuri("query", query)
url = raw"http://statistics.data.gov.uk/sparql"
response = HTTP.post(url, ["Accept" => "application/sparql-results+json"], encoded_query; redirect_method="POST")
ret = JSON3.read(String(response.body))
return ret
end
io = IOBuffer()
r = ZipFile.Reader("junk.zip")
let nf=true
for f in r.files
if f.name == "junk.txt"
write(io, read(f, String))
nf=false
break
end
end
end
close(r)
seekstart(io)
PostcodeString = vectortostring(pcd)
@time result = postcodeLonLat(PostcodeString)
If I delete the block of code using the IOBuffer, everything works fine every time.
The ZipFile in this example contains only one .txt file with a couple of lines of plain ASCII. I don’t think the file is the problem. I think it is the IO but I don’t know how to fix it.
The HTTP stacktrace is very long but this seems to be the salient message:
IOError: stream is closed or unusable
I’m hoping I’ve just missed something simple.
Thanks.