Failing to import (relatively) large CSV file with Julia and VSC

Specific Domains Data
, ,
13

I have this memory and …

julia> Sys.total_memory()/2^20
7948.6171875

julia> Sys.free_memory()/2^20
1322.1484375

… in a few minutes I created, saved and read this

using CSV, DataFrames, Random


# m=rand(Float32,4*10^4,4*10^3);
# tm=Tables.table(m)  
# CSV.write("m10k_X_1k.csv", tm);
filesize("m10k_X_1k.csv")/1e6  # 1700.485444
t=CSV.File("m10k_X_1k.csv")

julia> DataFrame(t)
40000×4000 DataFrame
   Row │ Column1    Column2    Column3    Column4    Column5    Column6   ⋯
       │ Float64    Float64    Float64    Float64    Float64    Float64   ⋯
───────┼───────────────────────────────────────────────────────────────────
     1 │ 0.45914    0.323786   0.287918   0.973984   0.373564   0.94596   ⋯
     2 │ 0.148653   0.228002   0.775495   0.181961   0.205018   0.974561   
     3 │ 0.545785   0.464947   0.786147   0.203469   0.0108073  0.23109    
     4 │ 0.976365   0.633956   0.177804   0.567126   0.726814   0.539547   
     5 │ 0.295424   0.140896   0.799063   0.0587873  0.92122    0.838071  ⋯
     6 │ 0.698935   0.925796   0.653495   0.885776   0.892522   0.710776   
     7 │ 0.788658   0.389966   0.392589   0.106059   0.15693    0.0068168  
     8 │ 0.469697   0.0133684  0.350079   0.161829   0.224559   0.571217   
     9 │ 0.69308    0.0798342  0.683515   0.327015   0.0991643  0.926583  ⋯
    10 │ 0.686669   0.0751054  0.915328   0.532172   0.871903   0.572102   
    11 │ 0.104822   0.150674   0.199238   0.251358   0.498402   0.975431   
    12 │ 0.138714   0.911603   0.364109   0.187194   0.745571   0.678922   
    13 │ 0.192071   0.892678   0.85928    0.513568   0.601975   0.140359  ⋯
    14 │ 0.268991   0.667317   0.0689645  0.364786   0.504987   0.757816   
    15 │ 0.361199   0.568255   0.381775   0.305797   0.708287   0.658362   
    16 │ 0.295467   0.547508   0.92817    0.408412   0.411363   0.707712   
   ⋮   │     ⋮          ⋮          ⋮          ⋮          ⋮          ⋮     ⋱
 39986 │ 0.63584    0.637439   0.0449622  0.194153   0.159854   0.435054  ⋯
 39987 │ 0.0470939  0.123351   0.745204   0.0304491  0.426111   0.397158   
 39988 │ 0.589865   0.793649   0.19325    0.221891   0.610391   0.810318   
 39989 │ 0.795853   0.300518   0.243793   0.307483   0.34223    0.83377    
 39990 │ 0.820287   0.440265   0.412261   0.309526   0.939981   0.50417   ⋯
 39991 │ 0.215501   0.526097   0.603948   0.155538   0.721428   0.346939   
 39992 │ 0.473193   0.701665   0.963082   0.720776   0.641577   0.614307   
 39993 │ 0.802456   0.277233   0.736046   0.535992   0.54705    0.173352   
 39994 │ 0.274464   0.680145   0.580526   0.244665   0.0791177  0.780809  ⋯
 39995 │ 0.285998   0.0708128  0.447914   0.676254   0.423098   0.530598   
 39996 │ 0.190717   0.0346309  0.775401   0.0556166  0.70802    0.670697   
 39997 │ 0.796287   0.509074   0.748359   0.855386   0.354784   0.647198   
 39998 │ 0.146545   0.507778   0.981678   0.0832134  0.373617   0.625176  ⋯
 39999 │ 0.158659   0.278268   0.444578   0.998274   0.897327   0.357139   
 40000 │ 0.816053   0.587334   0.0366446  0.148506   0.65696    0.47753    
                                        3995 columns and 39969 rows omitted

julia> begin
           println(now())
           t=CSV.File("m10k_X_1k.csv");
           println(now())
           DataFrame(t);
           println(now())
       end
2024-03-31T19:06:28.566
2024-03-31T19:06:41.880
2024-03-31T19:06:52.558