mutable struct InterpreterHooks
closure::Ptr{Nothing}
surface_create::Ptr{Nothing}
surface_destroy::Ptr{Nothing}
context_create::Ptr{Nothing}
context_destroy::Ptr{Nothing}
end
and assign to the fields pointers to available data structures.
I’m debugging now for some time random memory problems and i can see (in a rr trace) that not only this structure is cleaned up in GC, but also the field entries get ‘free’-calls, so there is a probability that contents is overwritten.
I read the documentation that Ptr{CVoid}/Ptr{Nothing} is pointing to memory julia isn’t maintaining. So why do i see the free calls? And how can this be avoided?
I’m not 100% on what you are doing. But is sounds like you are passing an instance of InterpreterHooks to a C library, but it is being garbage. And the various pointers in this structure point to other Julia objects that are also being garbage collected.
In the manual under “Calling C and Fortran Code” in the sub section "Garbage Collection Safety it has:
The suggested way to handle this is to make a global variable of type Array{Ref,1} to hold these values, until the C library notifies you that it is finished with them.
So you need to create a global variable and store the instance of InterpreterHooks along with all the child objects in it. This will keep the garbage collector from freeing the objects.
The instance of InterpreterHook (let’s call it) h has e.g this entry h.closure which points to e.g. a CairoSurface (datatype in Cairo.jl) which has own create and finish functions (let’s call it) cs (h.closure = cs). h is used correctly and the GC correctly discards h. The only thing is, that GC uses free(h.closure) and then after that point the cs is considered free memory and might be reallocated (which causes the random memory problems).
One thing I was thinking about was if you could do something like:
struct IhExternal
closure::Ptr{Nothing}
surface_create::Ptr{Nothing}
surface_destroy::Ptr{Nothing}
context_create::Ptr{Nothing}
context_destroy::Ptr{Nothing}
end
struct InterpreterHooks
closure::CairoSurface
external::Ref{IhExternal}
end
When you need to pass the structure to the C library, you pass InterpreterHooks.external. I think that would work. If you wanted to update IhExternal you could probably make it mutable then you wouldn’t need the Ref.
Openly: i think the freeing of Ptr{Nothing} is very close to a bug, at least it’s somehow unexpected behaviour and i’m missing an explaination why this is needed (i had already similar cases in GC).
I’ll try your change and maybe the better plan is to encapsulate the ‘whole’ process into a function, currently it’s tested as script and that might give GC the wrong impression.
Have you read the tutorial on calling c code: Calling C and Fortran Code · The Julia Language? I believe, the problem you are encountering is that you are referencing an object with a pointer, but that object might move out of scope and therefore gets cleaned by the GC. You probably need to protect the objects you are referencing with GC.@preserve, any time you use their pointer. If you are putting pointers in a struct, you still need to keep the original objects somewhere, to prevent them from getting cleaned up. I believe a common practice is too store them in an IDDict, that is either a global or is stored inside the struct.
First of all, there’s no “GC freeing Ptr{Nothing}”. The GC didn’t allocate them in the first place and will never free them.
They can only be freed by your own code, for example via a finalilzer and if that is the case you just need to make sure the object you registered the finalizer on is valid.
For that to work, you need to figure out the scope you need to keep the object valid. The solution you’ll need depends on that.
You need a much more detailed description of the problem. For example, who created what (GC, C code?), how are different objects passed to C (by pointer? copy? indirectly via a field?), what’s the scope that C referencing them (a call? the time another object is valid? global?).
A few things on some of the comments above.
No that’s absolutely the last resort.
Not directly and creating a Ref here doesn’t help with anything really.
No it isn’t. And a few reason it might feel like a bug is that even when you are using an object in your code, in general the compiler does not have to keep that object valid. The keeping-object-valid-for-c must always be done explicitly.
If you are talking about the scope where the GC preserve things then,
The “scope” is always either global (in which case there’s no tool that can theoretically do that) or explicit GC.@preserve or ccall and require no special tool to see it.
If you mean the scope you need to preserve the object,
Then it’s entirely up to the C code and no tool can really analyse that. (You’ll need an analyser of the C code you call to even get started)
If you’ve seen the call in rr it should be very easy to figure out what the caller was = = …
using Cairo
using CairoScript
using Compat, Colors
import Compat.String
@compat import Base.show
using Test
include("test_painting.jl")
pkg_dir = dirname(dirname(@__FILE__))
# Image Surface
@testset "Interpreter Run" begin
# run a script that includes a writing operation
surf = CairoImageSurface(256, 256, Cairo.FORMAT_ARGB32)
GC.@preserve surf h c ci begin
testfile = joinpath(pkg_dir,"data","a1.cs");
h = CairoScript.InterpreterHooks()
h.surface_create = CairoScript.surf_create_c[]
h.closure = surf.ptr
ci = CairoScript.Interpreter()
c = CairoScript.interpreter_install_hooks(ci,h)
status = CairoScript.interpreter_run(c,testfile)
@test status == 0
finish(surf)
#surf.ptr = C_NULL;
end
i break in cairo_image_surface_create which is called in CairoImageSurface, break in cairo_script_interpreter_run called in CairoScript.Interpreter_run and in cairo_surface_destroy which is called in the finalizer to surf.
Afaics the finalizer is called before the interpreter_run.
Your GC.@preserve is enclosing too much and it’s not doing anything.
Assuming Cairo.jl is at least making sure the arguments are used correctly you should only need to preserve from CairoScript.interpreter_run to finish(surf). Looking at the Cairo.jl code though it’s doing this completly wrong. Almost none of the ccall with a cairo pointer is done correctly and that should be fixed first.
GC.@preserve preserves the object you passed to it, not the variable. so you aren’t preserving any of h c ci.
This doesn’t make sense. Are you sure it is the finalizer for the correct object?
Also, Cairo.jl should really track these implicit dependencies and make sure the user never need to preserve any memory explilcitly.
Just to confirm: You mean storing the pointer to the libcairo allocated memory in e.g. surf.ptr?
Can you please point to an example where it’s done correctly?
i can see the output (pointer to cairo_surface) as an adress, the same as an input to cairo_surface_destroy. This is also how i founded my (false) theory on GC freeing, i set a watchpoint to that adress and found ‘free’ as the function applied to it.
Again, for confirmation: Storing the pointers to libcairo structures?
No that’s not the problem. The problem is that it must NOT use surf.ptr directly as argument for ccall. It must use surf instead and define unsafe_convert to do the conversion. FWIW there seems to be a single ccall that it is passing in surf but that seems to be a bug atm…
No, it’s about storing the reference to all the other julia objects that is managing cairo objects referenced implicitly.