I was analysing some code today and I am unable to read information from a simple file while keeping type stability. I would like to pass the results to functions that have to be very optimized, so I believe this is an important issue… At the beginning, I had just a CSV file like this a,b test1,1.…

If x in your snippet turns out to be Any that will propagate into simulation. For example, here f1 returns Any (see Body::Any), because the input of g is of type Any: julia> x::Any = 1.0 # ::Any to be explicit that it can assume any value, it is the default 1.0 julia> g(x) = 2 * x g (generic func…

There’s fundamentally no way to make such a reading function type-stable. After all, its type depends on the CSV file content. [image] VictorSeven: I would like to pass the results to functions that have to be very optimized, so I believe this is an important issue… Type-stable tables can be…

[image] VictorSeven: ormation from a simple file while keeping type stability. I would like to pass the results to functions that have to be very optimized, so I believe this is an important issue… Do you have a benchmark that shows that this type instability really costs a lot of performance…

@aplavin Thanks! I think the suggestion is very similar to second snippet of code I posted. If I do function f() fracfile = Tables.columntable(CSV.File("test.csv")) x = fracfile[:b] return x end which I think it’s what you suggest, nothing changes: x is not type stable. Or were you sug…

Perhaps I’m misunderstanding what you are trying to achieve here, but most of the time, the result of reading a file being type unstable does not matter. It’s a one-time cost, and as long as the thing you get of it (in your case, x) has a concrete type, subsequent operations on this object will be t…

Hi, If you really want the file to be type stable then you could convert it to Arrow then read from that in place of the CSV after the fact. You’ll also benefit from a reduction in memory usage and out of core processing if your real data is bigger than your ram.

[image] VictorSeven: I understand that the compiler cannot know the contents of the file beforehand, so type inference is complicated. But I cannot really get why if I use TypedTables.jl or if I explicitly try to cast variables the compiler can still do nothing. As people have said, you can d…

Thank you all for your answers! After @JonasWickman ’s answer and reading carefully the concept of function barrier again I finally understand the full “problem”. In my case the most hardcore computations are inside functions, something like function f() fracfile = DataFrame(CSV.File("test.csv"…

[image] VictorSeven: x = fracfile[: ,:b] result = simulation(x) the key here is guaranteeing that x has a stable type when passed to simulation. If x is a Vector{Any} inside the function f() that will be still problematic. But if it is, say, Vector{Float64}, the function simulation wi…

Yes @lmiq , that was the main motivation under my question. Notice that in the example I wrote before x is not even Vector{Any} but just Any, as checked with @code_warntype. However, as far as I understand from previous answers and the Performance Tips guide, my last snippet should be good and the …

Type-stable reading from a file

General Usage

lmiq February 11, 2025, 5:14pm 4

There’s a related discussion here, in which I had to implement a fast parser for some file: Performance: read data from ascii file, replace `split`

1 Like

Topic		Replies	Views
CSV.jl type stability General Usage csv , type-stability	26	1143	October 22, 2022
CSV Reading (rewrite in C?) Internals & Design	50	5378	October 1, 2018
CSV.read extremely slow wrt readtable Data	14	3708	July 27, 2018
Does the concept of type-stability apply to DataFrames or Tables? Data dataframes , tables , type-stability	21	5973	October 17, 2017
IndexedTables and type-stability of IndexedTables.groupby New to Julia type-stability	9	1215	October 24, 2019

Type-stable reading from a file

Related topics