# Parsing XML as IO-stream
**URL:** https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293
**Category:** General Usage
**Created:** [July 10, 2018, 12:59pm UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293 "2018-07-10T12:59:57Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@johann.spies](https://discourse.julialang.org/u/johann.spies)
#### Post date: [July 10, 2018, 12:59pm UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/1 "2018-07-10T12:59:57Z")
I have a lot of XML-files (gzipped - about 1.5Gb in size) containing approximately 400000 elements
(
```julia
```
)
in which I am interested.
The structure of the file:
```julia
...
etc.
```
I have first looked at LightXML but could not figure out how to do streaming. Then I tried EzXML in this way (which worked for a few records, but then just hangs after a random number of records in the `function eenrekord()`):
```julia
using EzXML
using CodecZlib
function eenrekord(x)
r = parsexml(string(x)) # This is where the process hangs - sometimes after 20 - 80 iterations.
ut = nodecontent(findfirst(root(r), "//UID"))
ut, string(r)
end
function xml_records(gzipfile,tabel,zipfile)
l = GzipDecompressorStream(open(gzipfile))
reader = EzXML.StreamReader(l)
count = 0
rcount = 0
conn = dbcon()
list = []
while !done(reader)
if nodetype(reader) == 1 && nodename(reader) == "REC"
rcount += 1
if rcount % 2 == 1
x = expandtree(reader)
ut, xml = eenrekord(x)
push!(list, [zipfile, gzipfile, ut, xml])
count +=1
if count % 20000 == 0
println(count)
# write_list_sql_table - another function not shown here.
list = []
end
end
end
end
if telling % 20000 != 0
println(count)
# write_list_sql_table - another function not shown here.
end
close(conn)
end
gzipfile = "/tmp/some.xml.gz"
zipfile = "somezipfile.zip"
tabel = "core_2018"
xml_records(gzipfile,tabel,zipfile)
```
I have looked at LibExpat.jl of which the README indicates streaming from a file using `xp_streaming_parsefile` but I have no idea how to use it and I could not any example using it.
At this stage I just want to put ` ` as a XML-type record in the SQL table from where I will use PostgreSQL’s XPATH-capabilities to manipulate it. In total it will be about 60 million records I have to process in this way.
I will appreciate any references to good examples or tutorials on how to do this.
And any comments about mistakes I made in the code shown above.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [July 12, 2018, 2:03am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/2 "2018-07-12T02:03:58Z")
On my phone, so forgive the lack of code/links, but I recall running into a similar problem with a python library for streaming XML and the issue was that a huge chunck of each record was maintained in memory - I was only grabbing a single field from very large records, but all of the parts of the record that were not in my query were held onto, and I had to explicitly flush in every loop, otherwise the memory would load up and it would stall.w
I think it’s unlikely that the libraries you tried have this problem, but thought I’d mention it just in case 🙂
---
### Author:  [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)
#### Post date: [July 12, 2018, 4:22am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/3 "2018-07-12T04:22:16Z")
Hi, I’m a developer of EzXML and CodecZlib.
First of all, you don’t need to use CodecZlib because EzXML detects gzip compression and implicitly decompresses it. Also, calling `expandtree` is inefficient and may be unsafe. I’d like to know what you meant by “hang”; does the process run forever without warning messages?
---
### Author:  [@johann.spies](https://discourse.julialang.org/u/johann.spies)
#### Post date: [July 12, 2018, 8:14am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/4 "2018-07-12T08:14:17Z")
Thanks for the remark about EzXML which detects gzip compression.
I have used `expandtree` to be able to get a document which I could use to search for a specific element. Is there another way of doing it?
The process gave no error or warning messages and nothing was happening. I used print statements to determine the exact piece of code where this seemed to go into some endless loop. This happened sometimes after retrieving anything between 12 and 80 records from the same file.
---
### Author:  [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)
#### Post date: [July 12, 2018, 8:35am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/5 "2018-07-12T08:35:39Z")
Using a streaming parser is a little bit difficult; you’ll need to keep track of the current parsing state using a stack or something.
It would be helpful for me to debug the problem if you could create reproducible code with some (smaller) data and file an issue at [https://github.com/bicycle1885/EzXML.jl/issues](https://github.com/bicycle1885/EzXML.jl/issues).
---
### Author:  [@johann.spies](https://discourse.julialang.org/u/johann.spies)
#### Post date: [July 12, 2018, 8:38am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/6 "2018-07-12T08:38:17Z")
Thanks @kevbonham. I have had the same experience in Python where I did:
```julia
element.clear()
gc.collect()
```
I am not exactly sure how I am going to do this in Julia. Will investigate.
---
### Author:  [@johann.spies](https://discourse.julialang.org/u/johann.spies)
#### Post date: [July 12, 2018, 8:46am UTC](https://discourse.julialang.org/t/parsing-xml-as-io-stream/12293/7 "2018-07-12T08:46:32Z")
Thanks @bicycle1885.
I am willling to do that but the data has strict licensing restrictions and I am reluctant to even put a portion of the data out there in public domain. I will file an issue if I can succeed to replicate the problem with “safe” data.