I need this for testing, it looks like i have the situation that the OS is buffering the write operation to a file. So in the test for isfile this is true, but the content has not been fully written.

I’m 99% sure that if you don’t see the file content update, it’s not the OS but actually the user space that’s buffering the content. OS buffering happens when write to the underlying media is delayed and shouldn’t be observable when reading the file from the same host, and should be only observable when, say, if the connection (usb/network) to the media is interrupted before the buffer is flushed. In another word, it wasn’t the OS’s doing (or it would have just give you the buffered content). It’s the user space doing (possibly libc’s) so the OS has no way to tell you anything about it. You’ll have to ask the userspace code that does the buffering, possibly by just flushing the output.

Testing if there’s anyone opening the file should be possible, assuming you have permission to query the openning process. I believe fuser, lsof etc use procfs on Linux. I’m not aware of any library implementation and I’m not sure how it’s implemented elsewhere. (I think windows may simply block you from openning it so you may not have this issue there?)

I agree with most of your post and the cases where i have seen (excessive) OS buffering it was in an server/VM/container environment. In this case there might be a user space problem, but unfortunately i cannot control the actual file operations, as this is inside a library and julia side is ccalling the respective functions. To be more concrete, we’re talking about the cairo surface create, use and destroy logic you’ll find in Cairo.jl/Cairo.jl at master · JuliaGraphics/Cairo.jl · GitHub which is applied to CairoScriptSurfaces in Cairo.jl/Cairo.jl at master · JuliaGraphics/Cairo.jl · GitHub. We have the open issue of CairoScript should probably not use finalizers · Issue #215 · JuliaGraphics/Cairo.jl · GitHub and Andreas Noack created the fix Cairo.jl/test/runtests.jl at master · JuliaGraphics/Cairo.jl · GitHub, as (you can see in 215) he claims, only GC.cg() triggered the cairo_destroy_surface that writes the file in full. I don’t agree on this (finalizer) after visiting the cairo source code some time, a single call to destroy should be enough and actuall is on linux. Seems not to be on OSX, and recently i borrowed a Macbook (i don’t have continuos access to it) and have seen the situation, that files were available in the filesystem only after some seconds. So my new theory is, his observation is correct, however it’s not the GC.gc() triggering the destroy, but the effect is only happening after some time.

It just doesn’t really make sense that the OS will hold the change you’ve told it without letting anyone see. It’ll likely even be bad for performance since the OS has to keep multiple copies of the file content. To really debug, you should log all the cairo calls. You can then either see whether you are calling the functions in the order you think you are or you can then create a C-repro. Doesn’t seem like anyone have done that in that issue. From a 5min read of cairo code, the flush isn’t controlled by the surface, it is controlled by the device which is the script in this case. For the constructor you use, the script is never explicitly dereferenced after the surface acquires a reference of it. It doesn’t seems that cairo has the floating reference like gobject does so the flushing will still depend on when the script is freed.

to 2. Actually at the time of this discussion i did this (gdb of julia-debug, debug version, breakpointing cairo_x calls) on linux and it did look good (destroy → file available). I don’t have access to OSX, so i trusted Andreas’ solution. to 3. I thought for some time into the same direction: What if there is no connection between surface and device destroy? But then the libcairo docu reads: “Decreases the reference count on surface by one. If the result is zero, then surface and all associated resources are freed” and i see the same behavior on e.g. CairoSVGSurface(filename which test nicely on linux and OSX without destroying the device. But i’ll look into it, again.

Maybe a last question here: Do you have a recommendation for 2 (log all calls, on linux)? I just failed trying to use ltrace on a julia process, i see the startup, but then subprocesses or threads are doing the magic.

[image] lobingera: If the result is zero, then surface and all associated resources are freed The device is a seperate object you construct. It is obviously not associated with the surface. [image] lobingera: That ‘tracing’ triggered my question here: Testing GC and finalizers I saw that but I have no idea what you are asking about. finalizers are just normal functions. You test it by running it. Either directly, or indirectly via finalize or gc. [image] lobingera: Do you have a recommendation for 2 This is a scripting (in the sense that there’s no separate AOT compilation stage for the user) and you have full control over all the source code so just do a grep over the code?

[image] yuyichao: The device is a seperate object you construct. It is obviously not associated with the surface. I tend to disagree, as the sequence of creating is: cairo_script_create for the device, cairo_script_surface_create based on the device - while i agree the julia code doesn’t hold reference, libcairo does. Cairoscript is a very specific part of libcairo which is mainly for debugging and not clearly documented. It’s also mentioned in the documentation, that I/O for libcairo should be rather done with streams, then file access. If the file is only closed on destroy device, then referencing is needed, but actually surface finish sounds like the method. [image] yuyichao: Either directly, or indirectly via finalize or gc . Well, this indirectly is my point here. I’d like to write a test in such a way, that the finalizer is called and i can test that it was called - without creating state or something global. On REPL i (obviously <1.0) had situation where i wasn’t sure. [image] yuyichao: This is a scripting (in the sense that there’s no separate AOT compilation stage for the user) and you have full control over all the source code so just do a grep over the code? I guess you mean putting the whole Pkg.test inside code_lowered or code_llvm? I thought about that, but didn’t make progress. I just have a longer experience in using gbd/strace/ltrace.

[image] lobingera: i agree the julia code doesn’t hold reference No. The whole point is that you didn’t release the reference to the script explicitly. [image] lobingera: I’d like to write a test in such a way, that the finalizer is called and i can test that it was called - without creating state or something global. Well, there’s nothing special about indirect. It’s a function with no return value so you can only observe side effects. You’ll have to figure out how to do that for your specific case. Julia does not provide a way to tell you everytime a finalizer is called (other than calling the finalizer) just like it doesn’t tell you everytime a function without a return value is called. [image] lobingera: I guess you mean putting the whole Pkg.test inside code_lowered or code_llvm? No. I mean “grep over the code” as in just edit the source code.

How to test if a file is existing and 'closed'?

General Usage

lobingera August 25, 2018, 6:12pm 7

btw: That ‘tracing’ triggered my question here: Testing GC and finalizers

Topic		Replies	Views
Windows specific UV error Internals & Design	10	1550	May 21, 2017
Testing GC and finalizers Internals & Design	2	799	August 21, 2018
Opening and closing file overhead New to Julia question , performance	8	1796	May 1, 2018
Can I mimick finalizing for immutables / is an immutable file handler unfeasible? Internals & Design file , finalization , immutable	16	1031	October 15, 2023
Clean way to determine if a file open? New to Julia question	8	2371	September 23, 2019

How to test if a file is existing and 'closed'?

Related topics