Read array of strings into Dictionary of DataFrames

General Usage Performance
,
1

Hi,
I have an array of strings where each string is a stream of multiple tables. And the final step of my function is to create and return a dictionary:

function readTables(str)		
		nStrs = split(str,"\r\n%T\t")
		fHead=nStrs[1]
		fTail="%E"
		nStrs[end]=replace(nStrs[end],r"%E.*"=>"")
		tableStrings=nStrs[2:end]
		tableNames=match.(r"^[A-Z]+",tableStrings)
		tableNames=[x.match for x in tableNames]
		dictTables=Dict(tableNames.=>CSV.read.(IOBuffer.(tableStrings),delim="\t",header=2, type=String))
	end
	#map(readTables,values(contentP6))

But when i run the map line above, the browser just hangs. I’m using Julia 1.4.2 / Pluto via Chrome. The size of the string array values(contentP6) is about 1.7Gb, 461 elements.

May I know:

  1. Is Dict of 2D Arrays the right way to store these tables?
  2. How to improve the code for better performance?
2

  1. That’s up to you — it could be a sensible solution.

  2. I would consider some solution that does not need to read the file and then make chunks — eg Mmap the file, find the delimiters, pass a view to CSV.File, and read that.

In any case, I would factor this out to smaller functions so that I could benchmark each piece.