I pull data from a database into a data frame with just two columns, ID and XMLData. I need to parse out a list of 125 (for now) up to 240+ xpaths into new columns of the dataframe.
I broadcast the parse function over the list of search xpaths for each row in the dataframe. The list of search xpaths also contains the output field name to be used. The function returns a named Pair (or is it a NamedTuple?). I convert the results of the broadcast into a NamedTuple which the $AsTable uses as the list of name/values for the new columns in each row.
It works, but for the first dataframe test, almost 4000 rows, it is taking 7+ minutes. I may need to run this for up to 100+ dataframes at a time, with varying row counts.
Is there possibly a faster , or better, way of parsing the list of xpaths into new columns?
Here’s my parsing code:
#-- Packages: EzXML, DataFrames, DataFramesMeta
#-- xml_lookfor : is vector of dictionary items that have "xmlpath" and "fieldname"
#-- keys, will probably change to a vector of NamedTuple items since I don't
#-- need to change it anymore
function parse_xml_path(xqry, xmlstr)
doc = parsexml(cleanxml(xmlstr))
el = findfirst(xqry["xmlpath"], doc.root)
Symbol(xqry["fieldname"]) => (el == nothing || el.content == "" ? missing : el.content)
end;
@time begin
@rtransform! results begin
$AsTable = NamedTuple(parse_xml_path.(xml_lookfor, (:xmldata,)))
end
end