Read special characters using CSV.read

[oheil]

@StefanKarpinski

A potential design change would be to add an option to CSV.read to validate strings as UTF-8 and turn that on by default, only letting you read invalid data if you disable that option.

I would object, as it is clearly a breaking change.
Better: validate by default but only producing a Warning when reading invalid strings.
So the new option could be 3 states: Off (no validation, like now), default On (validate but only warn), Error On (validate and error out).

[TimG]

That would work for me, too.

[oheil]

And, perhaps even more important: CSV.jl is one the most prominent packages in respect to performance. I remember the times and discussions about this. Doing a validation by default would be probably a performance draw back, but I can’t estimate the cost. This needs to be considered too.