Video artefacts when using record in Makie under Pluto

I’m using Makie under Pluto to visualise data received as json or parquet.
Visualisation is in form of animated boxes (<30 per frame) with time dependent position, size and orientation where main changes are in position.
Plotting is done by extracting position, size, orientation from the source data for relevant timestamps (all objects expected to be seen at one instant share the same timestamp) and assigning those vectors to observables.
The observables are used as sources for a meshscatter plot showing the scene.
The observables are updated by an on(t) construct with t representing time.
I have used two different ways to change time - either by selecting frame number with a NumberField for precise control or a PlutoUI.clock for animating and interactive session. Both work just fine.

But - when I have used the record function to create a video of a running scenario there are some instances where some shapes get some aspect wrong for a single frame, eg it stays in the right place but for one frame gets an incorrect size - and that size is not random but in the same range as other objects in the scene. I have expected the record function to be even more solid for creating correct content as it does not really have any real-time requirements (?) as it is iterating over the desired time span and can take any time it needs to render each frame.

Primarily interested in knowing if this is a known bad approach to animation and if so how to do it better, or if someone has had similar issues?
Maybe running a Makie animation under Pluto is pushing things a little too far and I need to go for a standalone application, but it is darn convenient…

This is impossible to judge without a minimal working example…
If you have something I can easily reproduce, best open an issue at Makie:
https://github.com/JuliaPlots/Makie.jl/issues/new

Fair point, I will try to cook it down to a MWE

No MWE yet, but examples of the behaviour and an extension of the problem description.

Looking at the images of three consecutive frames from the video it seems like the shapes are actually doubled or at least drawn twice for every position. In the middle frame there is a position offset for many shapes and also a size change for some. In the last frame things are back to “normal”. As the shapes are intended to be drawn very transparent I think the visual appearance of non-doubled shapes in the middle frame is correct. Updating the animation loop assigns new vectors to the observables controlling the mesh scatter plot so there should just be one entry per “real-world” shape in the scatter.



is this WGLMakie or GLMakie? Can you reproduce the behavior outside Pluto?

Its WGLMakie, not had time to try it outside Pluto but I’m for sure considering that.

I’d try GLMakie if possible… Recording videos with WGLMakie is really sub optimal, since its really complicated to get a frame out of the browser…

I sure can try GLMakie, but then I assume you mean outside of Pluto?
As I understand I need to use WGLMakie if i do anything in a browser ie in Pluto.

No, you can run GLmakie just fine in Pluto, but it will display just images… But you can open a window as well, since all Julia code is executed outside the browser…

How do I open a window? Tried GLMakie and it works fine for all 2D Axis plot but I get nothing (= empty space in the workbook) for the intended 3D Axis plots. Getting an “external” window would work fine if I just know how to do it, @sdanisch

GLMakie.inline!(false)
display(figure)

Should work!

Yes, that did work, giving a window outside the browser just as promised.
Still running as a workbook in Pluto though and still see some of the same size-changing
behaviour.

(Edited this post removing description of true weirdness since the problem was user (me :wink: error, being tripped up by QuickTime keeping showing a file long after its was replaced. I get spoiled by working in a reactive environment where everything (well, almost apparently…) get updated and kept in sync. Sry, @sdanisch my bad)

That sounds pretty crazy…
Can you really try to make an minimal working example?
Like, just use some random data for the paths and see if you can reproduce the behaviour?
It would be really great to get to the bottom of this…
You can have a look at the record code:
https://github.com/JuliaPlots/Makie.jl/blob/master/src/display.jl#L595
It just steps through the iterator and calls recordframe! which could be replaced by saving a file…

1 Like

I realised this may be a problem of synchronous updates as I have an animation level based on a single time variable t, but in the on(t) handling I update the three observables holding values for position, size and rotation of the mesh scatter plot individually by empty indexing assignment: positions[] = ... etc.
Hoping to have found the source of the problem I changed the assignment of position and size to use positions.val = ... and only at the very end doing rotations[] = ... thinking that would trigger correct updating of the mesh scatter, taking in account the updated positions and sizes too. But no, that made the objects stay in place and only update the (very small) rotations.
But it feels like a “proper” synchronous update of all the source data to the meshscahter would at least be helpful. Is there a better, or well since this did not work - is there a working way to achieve that?

@sdanisch @fonsp ?

Hoping to have found the source of the problem I changed the assignment of position and size to use positions.val = ... and only at the very end doing rotations[] = ... thinking that would trigger correct updating of the mesh scatter,

That sounds kind of correct … We’re working on implementing synchronous updates that work more intuitively, but until then I guess something like that has to do…

If you can create a minimal working example that I can run and debug (best just a meshscatter call, with the update you’re trying to do), I can see if I can find the underlying problem!

Hi. Finally got around to creating a MWE: Pluto notebook here
But in doing so I had a lot of small issues. On my machine I needed to close and reopen the workbook to get changed behaviour when I switched between empty-index and .val assignment to position and markersize. The workbook takes a long time (say 20s) to start (not sure if that is an issue in itself I know there is a startup time for all workbooks). I also get rather frequent stack traces in the window used to start Pluto:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.3 (2021-09-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pluto

julia> Pluto.run()

Opening http://localhost:1234/?secret=ZRcGOkc9 in your default browser... ~ have fun!

Press Ctrl+C in this terminal to stop Pluto

      From worker 2:	An exception was thrown in JS: TypeError: undefined is not an object (evaluating 'scene.wgl_camera')
      From worker 2:	Additional message: Error while processing message {"payload":{"__javascript_type__":"JSCode","payload":{"source":"    WGLMakie.insert_plot(...__eval_context__[0])\n","context":[["12575459940635311916",[{"transparency":"1448775247744666414","vertexarrays":{"position":{"flat":{"type":"Float32Array","data":[0,-1,0,1,1,-1,1,1]},"type_length":2},"uv":{"flat":{"type":"Float32Array","data":[0,0,0,0,0,0,0,0]},"type_length":2}},"visible":"17693789232690750967","uniform_updater":"3943124913470469672","attribute_updater":"9009309181735591475","faces":[0,1,2,1,3,2],"overdraw":"11032338631077554446","name":"lines-596306099531251942","uuid":"596306099531251942","uniforms":{"color":{"__javascript_type__":"TypedVector","payload":[0,0,0,0.4000000059604645]},"linewidth_start":2,"color_end":{"__javascript_type__":"TypedVector","payload":[0,0,0,0.4000000059604645]},"linewidth_end":2,"color_start":{"__javascript_type__":"TypedVector","payload":[0,0,0,0.4000000059604645]},"resolution":{"__javascript_type__":"TypedVector","payload":[1000,1000]},"linewidth":2,"lightposition":{"__javascript_type__":"TypedVector","payload":[1,1,1]},"model":[1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1]},"vertex_source":"#version 300 es\nprecision mediump int;\nprecision mediump float;\nprecision mediump sampler2D;\nprecision mediump sampler3D;\n// Instance inputs: \nin vec2 position;\nvec2 get_position(){return position;}\nin vec2 uv;\nvec2 get_uv(){return uv;}\n\n// Uniforms: \nuniform vec4 color;\nvec4 get_color(){return color;}\nuniform float linewidth_start;\nfloat get_linewidth_start(){return linewidth_start;}\nuniform vec4 color_end;\nvec4 get_color_end(){return color_end;}\nuniform float linewidth_end;\nfloat get_linewidth_end(){return linewidth_end;}\nuniform vec4 color_start;\nvec4 get_color_start(){return color_start;}\nuniform vec2 resolution;\nvec2 get_resolution(){return resolution;}\nuniform float linewidth;\nfloat get_linewidth(){return linewidth;}\nuniform mat4 model;\nmat4 get_model(){return model;}\n\n\n\n\n// Per instance attributes: \nin vec2 segment_start;\nvec2 get_segment_start(){return segment_start;}\nin vec2 segment_end;\nvec2 get_segment_end(){return segment_end;}\n\nuniform mat4 projection;\nuniform mat4 view;\n\nvec2 screen_space(vec4 position)\n{\n    return vec2(position.xy / position.w) * get_resolution();\n}\nvec3 tovec3(vec2 v){return vec3(v, 0.0);}\nvec3 tovec3(vec3 v){return v;}\n\nvec4 tovec4(vec3 v){return vec4(v, 1.0);}\nvec4 tovec4(vec4 v){return v;}\n\nout vec4 frag_color;\n\nvoid main()\n{\n    mat4 pvm = projection * view * get_model();\n    vec4 point1_clip = pvm * vec4(tovec3(get_segment_start()), 1);\n    vec4 point2_clip = pvm * vec4(tovec3(get_segment_end()), 1);\n    vec2 point1_screen = screen_space(point1_clip);\n    vec2 point2_screen = screen_space(point2_clip);\n    vec2 dir = normalize(point2_screen - point1_screen);\n    vec2 normal = vec2(-dir.y, dir.x);\n    vec4 anchor;\n    float thickness;\n\n    if(position.x == 0.0){\n        anchor = point1_clip;\n        frag_color = tovec4(get_color_start());\n        thickness = get_linewidth_start();\n    }else{\n        anchor = point2_clip;\n        frag_color = tovec4(get_color_end());\n        thickness = get_linewidth_end();\n    }\n    frag_color.a = frag_color.a * min(1.0, thickness * 2.0);\n    // I think GLMakie is drawing the lines too thick...\n    // untill we figure out who is right, we need to add 1.0 to linewidth\n    thickness = thickness > 0.0 ? thickness + 1.0 : 0.0;\n    normal *= (((thickness) / 2.0) / get_resolution()) * anchor.w;\n    // quadpos y (position.y) gives us the direction to expand the line\n    vec4 offset = vec4(normal * position.y, 0.0, 0.0);\n    // start, or end of quad, need to use current or next point as anchor\n    gl_Position = anchor + offset;\n\n}\n\n","fragment_source":"#version 300 es\nprecision mediump int;\nprecision mediump float;\nprecision mediump sampler2D;\nprecision mediump sampler3D;\n\nout vec4 fragment_color;\n\n// Uniforms: \nuniform vec4 color;\nvec4 get_color(){return color;}\nuniform float linewidth_start;\nfloat get_linewidth_start(){return linewidth_start;}\nuniform vec4 color_end;\nvec4 get_color_end(){return color_end;}\nuniform float linewidth_end;\nfloat get_linewidth_end(){return linewidth_end;}\nuniform vec4 color_start;\nvec4 get_color_start(){return color_start;}\nuniform vec2 resolution;\nvec2 get_resolution(){return resolution;}\nuniform float linewidth;\nfloat get_linewidth(){return linewidth;}\nuniform mat4 model;\nmat4 get_model(){return model;}\n\n\nin vec4 frag_color;\n\nvoid main() {\n    fragment_color = frag_color;\n}\n","instance_attributes":{"segment_start":{"flat":{"type":"Float32Array","data":[null,null,null,null,null,null,null,null]},"type_length":2},"segment_end":{"flat":{"type":"Float32Array","data":[null,null,null,null,null,null,null,null]},"type_length":2}}}]]]}},"msg_type":"2"}
      From worker 2:	Stack trace:
      From worker 2:	    add_plot@/Users/klint/.julia/packages/WGLMakie/0tzNl/src/wglmakie.js:65
      From worker 2:	    @/Users/klint/.julia/packages/WGLMakie/0tzNl/src/wglmakie.js:50
      From worker 2:	    forEach@[native code]
      From worker 2:	    insert_plot@/Users/klint/.julia/packages/WGLMakie/0tzNl/src/wglmakie.js:49
      From worker 2:	    anonymous@
      From worker 2:	    @/Users/klint/.julia/packages/JSServe/E7QrV/js_dependencies/JSServe.js:168
      From worker 2:	    process_message@/Users/klint/.julia/packages/JSServe/E7QrV/js_dependencies/JSServe.js:365
      From worker 2:	    forEach@[native code]
      From worker 2:	    process_message@/Users/klint/.julia/packages/JSServe/E7QrV/js_dependencies/JSServe.js:369
      From worker 2:	    process_message@/Users/klint/.julia/packages/JSServe/E7QrV/js_dependencies/JSServe.js:340
      From worker 2:	    @/Users/klint/.julia/packages/JSServe/E7QrV/js_dependencies/JSServe.js:451
      From worker 2:	An exception was thrown in JS: TypeError: undefined is not an object (evaluating 'scene.wgl_camera')
      From worker 2:	Additional message: Error while processing message {"payload":{"__javascript_type__":"JSCode","payload":{"source":"    WGLMakie.insert_plot(...__eval_context__[0])\n","co

and so on.
Also when I tried adjusting the rotation to be yaw(2t) with empty-bracket assignment to make the cube spin faster around its own axis the display was flickering like like it was trying to display both versions, until I (yet another time) restarted Pluto.@sdanisch

@sdanisch Was this of any help? (Sry for kicking this thread again…)