i’m trying to run a code for an mcmc samling , the code runs fine for an iteration of 1000 when i want to do the same for 2^20 iterations it shows me this error:

1123.840121 seconds (1.19 G allocations: 137.227 GiB, 4.54% gc time)


 [1] setindex! at .\array.jl:766 [inlined]
 [2] writecell(::Array{UInt8,1}, ::Int64, ::Int64, ::IOStream, ::String, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:433
 [3] writecell at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:402 [inlined]
 [4] (::getfield(CSV, Symbol("##82#83")){Array{UInt8,1},Base.RefValue{Int64},Int64,IOStream,Int64,CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},UInt8,UInt8})(::Array{Any,1}, ::Int64, ::Symbol) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:302
 [5] writerow(::Array{UInt8,1}, ::Base.RefValue{Int64}, ::Int64, ::IOStream, ::Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}}, ::DataFrameRow{DataFrame,DataFrames.Index}, ::Int64, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\Tables\nGOci\src\utils.jl:70
 [6] #73 at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:177 [inlined]
 [7] (::getfield(CSV, Symbol("##80#81")){getfield(CSV, Symbol("##73#74")){Bool,Bool,Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}},DataFrames.DataFrameRows{DataFrame,DataFrames.Index},CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},Tuple{Symbol,Symbol},Int64,Int64,Array{UInt8,1}}})(::IOStream) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:241
 [8] #open#312(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::getfield(CSV, Symbol("##80#81")){getfield(CSV, Symbol("##73#74")){Bool,Bool,Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}},DataFrames.DataFrameRows{DataFrame,DataFrames.Index},CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},Tuple{Symbol,Symbol},Int64,Int64,Array{UInt8,1}}}, ::String, ::Vararg{String,N} where N) at .\iostream.jl:375
 [9] open(::Function, ::String, ::String) at .\iostream.jl:373
 [10] with at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:240 [inlined]
 [11] #write#72 at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:167 [inlined]
 [12] write(::Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}}, ::DataFrames.DataFrameRows{DataFrame,DataFrames.Index}, ::String, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:162
 [13] #write#67(::Char, ::Char, ::Nothing, ::Nothing, ::Char, ::Char, ::Char, ::Nothing, ::Bool, ::String, ::getfield(CSV, Symbol("##68#70")), ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CSV.write), ::String, ::DataFrame) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:154
 [14] write(::String, ::DataFrame) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:147
 [15] top-level scope at .\In[43]:10

and this is the code :

function relay_basis(alpha,delta,name::String,n)
    chi = fill(sqrt(0.06), n)
    phi = im*tanh.(chi)
    omega = 1.0/prod(cosh.(chi))^2
    syms, op = qrelay_op(n, phi, alpha, delta)
    op_a, op_ab, mat, coef = op_mat(op)

    op_q2 = [syms.apH[1], syms.apV[1], syms.bpH[end], syms.bpV[end]]
    op_q1 = [syms.apH[2:end]..., syms.apV[2:end]..., syms.bpH[1:end-1]..., syms.bpV[1:end-1]...]
    global mask_q1 = [op in op_q1 for op in op_a];
    global mask_q2 = [op in op_q2 for op in op_a];
    global qq = [x in syms.apH || x in syms.bpV ? 1 : 0 for x in op_a]
    pdet0 = pdet_maker(0.04, 1e-5)
    global qrs = QRelaySampler(mat, coef, omega, pdet0)
    targetcache=Dict{Vector{Int}, Float64}()
    target(x::Vector)= (qrs.prob(qq, x, mask_q1) * qrs.prob(x))
    dist= qrs.psetproposal           #the proposal distribution
    global x=fill([0,0,0,0], Iteration)
    global selected=[]

    @time for i in 2:Iteration
        current_x= x[i-1]
        proposed_x= rand(dist(current_x))
        C= min(1,target(proposed_x)/target(current_x))
        if rand() < C
            x[i]= proposed_x
            push!(selected, x[i])
    funcQ(v) = qrs.prob(qq, v, mask_q2)
    return selected

global all_results=[]
for i = 0:12
    beta = i*pi/12
    name = string(i)
    println("beta:", beta)
    relay_basis(pi/4, beta, name,2)

Unfortunately the manual says little about this kind of error. Do you have enough HD to store the result?


oh actually no i don’t have it is there any other way i can solve this problem with?

It doesn’t solve your disk space problem (for that, either don’t write so much to disk, or buy some more. disk space is cheap), but eliminate your global variables. Performance Tips · The Julia Language.

Edit, FWIW, you can save a small factor in space by writing in a binary format or compressing your data in memory before writing. But if it fills the disk, do you really need to save all of it?


yeah i really do need to save all the outputs cause i’ll have to compute a probability afterwards using these outputs

now i put the code on another PC which has more than enough space in the hard drive but the same problem keeps happening when i want to run the code for 800_000 iterations for 13 times as you can see in the code i don’t know what’s the solution or is there any other way to store these outputs, the outputs are arrays of length=8

See if you can figure out how to write results incrementally. Your relay_basis function accumulates results in selected, and then in your lower loop you append selected to all_results. It doesn’t appear that you need these in additional iterations of either loop, so perhaps you can either just write selected to a file instead of accumulating all_results, and/or you might write checkpoints from relay_basis so you don’t need to accumulate everything in selected.

1 Like

i appended selected to all_results because selected is going to be taking different lists of arrays, i don't know any other way to make this happen and also i'll need to call funcQfor eachselected`, i don’t know any other way to make this happens.

i do this in the lower loop

for i = 0:5
    beta = i*pi/5
    name = string(i)
    selected, funcQ, qrs= relay_basis(pi/4, beta, name,2)
    println("beta:", beta)
    CSV.write("E:\\m.csv",selected=selected,delim = ';',append=true)

but when i want to read the file, it’s empty

I don’t see selected as a keyword argument to CSV.write. Perhaps this will work better

CSV.write("E:\\m.csv",selected,delim = ';',append=true)
1 Like