ReadOnlyMemory

General Usage
1

i’m trying to run a code for an mcmc samling , the code runs fine for an iteration of 1000 when i want to do the same for 2^20 iterations it shows me this error:

beta:0.0
1123.840121 seconds (1.19 G allocations: 137.227 GiB, 4.54% gc time)

ReadOnlyMemoryError()

Stacktrace:
 [1] setindex! at .\array.jl:766 [inlined]
 [2] writecell(::Array{UInt8,1}, ::Int64, ::Int64, ::IOStream, ::String, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:433
 [3] writecell at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:402 [inlined]
 [4] (::getfield(CSV, Symbol("##82#83")){Array{UInt8,1},Base.RefValue{Int64},Int64,IOStream,Int64,CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},UInt8,UInt8})(::Array{Any,1}, ::Int64, ::Symbol) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:302
 [5] writerow(::Array{UInt8,1}, ::Base.RefValue{Int64}, ::Int64, ::IOStream, ::Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}}, ::DataFrameRow{DataFrame,DataFrames.Index}, ::Int64, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\Tables\nGOci\src\utils.jl:70
 [6] #73 at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:177 [inlined]
 [7] (::getfield(CSV, Symbol("##80#81")){getfield(CSV, Symbol("##73#74")){Bool,Bool,Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}},DataFrames.DataFrameRows{DataFrame,DataFrames.Index},CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},Tuple{Symbol,Symbol},Int64,Int64,Array{UInt8,1}}})(::IOStream) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:241
 [8] #open#312(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::getfield(CSV, Symbol("##80#81")){getfield(CSV, Symbol("##73#74")){Bool,Bool,Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}},DataFrames.DataFrameRows{DataFrame,DataFrames.Index},CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))},Tuple{Symbol,Symbol},Int64,Int64,Array{UInt8,1}}}, ::String, ::Vararg{String,N} where N) at .\iostream.jl:375
 [9] open(::Function, ::String, ::String) at .\iostream.jl:373
 [10] with at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:240 [inlined]
 [11] #write#72 at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:167 [inlined]
 [12] write(::Tables.Schema{(:all_results, :beta),Tuple{Any,Float64}}, ::DataFrames.DataFrameRows{DataFrame,DataFrames.Index}, ::String, ::CSV.Options{UInt8,UInt8,Nothing,Tuple{},getfield(CSV, Symbol("##68#70"))}) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:162
 [13] #write#67(::Char, ::Char, ::Nothing, ::Nothing, ::Char, ::Char, ::Char, ::Nothing, ::Bool, ::String, ::getfield(CSV, Symbol("##68#70")), ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CSV.write), ::String, ::DataFrame) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:154
 [14] write(::String, ::DataFrame) at C:\Users\Administrator\.julia\packages\CSV\vyG0T\src\write.jl:147
 [15] top-level scope at .\In[43]:10

and this is the code :

function relay_basis(alpha,delta,name::String,n)
    
    chi = fill(sqrt(0.06), n)
    phi = im*tanh.(chi)
    omega = 1.0/prod(cosh.(chi))^2
    syms, op = qrelay_op(n, phi, alpha, delta)
    op_a, op_ab, mat, coef = op_mat(op)

    op_q2 = [syms.apH[1], syms.apV[1], syms.bpH[end], syms.bpV[end]]
    op_q1 = [syms.apH[2:end]..., syms.apV[2:end]..., syms.bpH[1:end-1]..., syms.bpV[1:end-1]...]
    global mask_q1 = [op in op_q1 for op in op_a];
    
    global mask_q2 = [op in op_q2 for op in op_a];
    global qq = [x in syms.apH || x in syms.bpV ? 1 : 0 for x in op_a]
           
    pdet0 = pdet_maker(0.04, 1e-5)
    global qrs = QRelaySampler(mat, coef, omega, pdet0)
    targetcache=Dict{Vector{Int}, Float64}()
    target(x::Vector)= (qrs.prob(qq, x, mask_q1) * qrs.prob(x))
    Iteration=2^20
    dist= qrs.psetproposal           #the proposal distribution
    global x=fill([0,0,0,0], Iteration)
    x[1]=qq
    global selected=[]

    @time for i in 2:Iteration
        
        current_x= x[i-1]
        proposed_x= rand(dist(current_x))
        C= min(1,target(proposed_x)/target(current_x))
        
        
        if rand() < C
            
            
            x[i]= proposed_x
            push!(selected, x[i])
        
        else
            
            x[i]=current_x
        end
        
    end
    
    funcQ(v) = qrs.prob(qq, v, mask_q2)
    
    return selected
    
end

global all_results=[]
open("file.csv","a")
for i = 0:12
    beta = i*pi/12
    name = string(i)
    println("beta:", beta)
    relay_basis(pi/4, beta, name,2)
    push!(all_results,selected)
    DF=DataFrame(all_results=all_results,beta=beta)
    CSV.write("file.csv",DF)
end
2

Unfortunately the manual says little about this kind of error. Do you have enough HD to store the result?

2 Likes
3

oh actually no i don’t have it is there any other way i can solve this problem with?

4

It doesn’t solve your disk space problem (for that, either don’t write so much to disk, or buy some more. disk space is cheap), but eliminate your global variables. https://dl.acm.org/doi/10.1145/953353.953355 Performance Tips · The Julia Language.

Edit, FWIW, you can save a small factor in space by writing in a binary format or compressing your data in memory before writing. But if it fills the disk, do you really need to save all of it?

2 Likes
5

yeah i really do need to save all the outputs cause i’ll have to compute a probability afterwards using these outputs

6

now i put the code on another PC which has more than enough space in the hard drive but the same problem keeps happening when i want to run the code for 800_000 iterations for 13 times as you can see in the code i don’t know what’s the solution or is there any other way to store these outputs, the outputs are arrays of length=8

7

See if you can figure out how to write results incrementally. Your relay_basis function accumulates results in selected, and then in your lower loop you append selected to all_results. It doesn’t appear that you need these in additional iterations of either loop, so perhaps you can either just write selected to a file instead of accumulating all_results, and/or you might write checkpoints from relay_basis so you don’t need to accumulate everything in selected.

1 Like
8

i appended selected to all_results because selected is going to be taking different lists of arrays, i don't know any other way to make this happen and also i'll need to call funcQfor eachselected`, i don’t know any other way to make this happens.

9

i do this in the lower loop

open("E:\\m.csv","a")
for i = 0:5
    beta = i*pi/5
    name = string(i)
    selected, funcQ, qrs= relay_basis(pi/4, beta, name,2)
    println("beta:", beta)
    CSV.write("E:\\m.csv",selected=selected,delim = ';',append=true)
    
end

but when i want to read the file, it’s empty

10

I don’t see selected as a keyword argument to CSV.write. Perhaps this will work better

CSV.write("E:\\m.csv",selected,delim = ';',append=true)
1 Like