RefValue errors with LibSerialPort

yakir12 · October 26, 2020, 2:55pm

I’m honestly not sure if this has something to do with LibSerialPort or if it’s just me, hence the Discourse post as opposed to an issue…

I’m repeatedly reading off of a serial port inside an async loop, and while trying to write to the same port things get stuck (I’m controlling this from WGLMakie). If I look into that async task, I get this:

Task (failed) @0x00007f72b4851ae0
TaskFailedException:
InterruptException:
Stacktrace:
 [1] RefValue at ./refvalue.jl:8 [inlined]
 [2] Ref at ./refpointer.jl:96 [inlined]
 [3] read(::SerialPort, ::Type{UInt8}) at /home/yakir/.julia/packages/LibSerialPort/Ukfil/src/high-level-api.jl:340
 [4] decode(::SerialPort) at /home/yakir/.julia/packages/COBS/05DPe/src/COBS.jl:108
 [5] (::var"#1#2"{Arduino})() at /home/yakir/.julia/dev/SimpleFanControl/src/simpler.jl:65
 [6] lock(::var"#1#2"{Arduino}, ::ReentrantLock) at ./lock.jl:161
 [7] sample!(::Arduino) at /home/yakir/.julia/dev/SimpleFanControl/src/simpler.jl:63
 [8] (::var"#5#8")() at ./task.jl:356
wait at ./task.jl:267 [inlined]
macro expansion at /home/yakir/.julia/dev/SimpleFanControl/src/simpler.jl:84 [inlined]
(::var"#3#6")() at ./task.jl:356

You can see my code here, but my issue might have to do with my misunderstanding of how serial ports work:

do I need to flush the port before every read/write?
Do I need to use locks to avoid simultaneously reading/writing from/into the port?
What happens if the device connected to the port (in my case an Arduino) sends messages faster than the computer retrieves them?

Appreciate any insight you might have!

contradict · October 26, 2020, 5:53pm

I think the stack trace here is from pressing Ctrl-C to stop the process. I suspect it is hanging because COBS is not self-synchronizing, if the flush happens to discard the leading length byte, the next byte in the sequence becomes the length and things go off the rails.

There is definitely no need to flush the write buffer, so maybe try only flushing the read buffer by calling sp_flush(a.sp, SP_BUF_INPUT) in sample!and removing the call to flush on the write side.

yakir12 · October 26, 2020, 6:07pm

Thank you for taking such a in-depth look!
Seems like it’s better, but I can still get some strange behavior. After reloading the page (again, this might have to do with how JSServe acts and my specific application) it stalls, and when I look into the async reading task I get:

 julia> reading
Task (failed) @0x00007f7f69965fc0
TaskFailedException:
AssertionError: isempty(session.message_queue)
Stacktrace:
 [1] send(::JSServe.Session, ::Dict{Symbol,Any}) at /home/yakir/.julia/packages/JSServe/9Z6uA/src/session.jl:83
 [2] #send#28 at /home/yakir/.julia/packages/JSServe/9Z6uA/src/session.jl:76 [inlined]
 [3] (::JSServe.JSUpdateObservable)(::Array{Any,1}) at /home/yakir/.julia/packages/JSServe/9Z6uA/src/observables.jl:14
 [4] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [5] invokelatest at ./essentials.jl:709 [inlined]
 [6] setindex!(::Observable{Array{Any,1}}, ::Array{Any,1}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:207
 [7] setindex!(::Observable{Array{Any,1}}, ::Array{Any,1}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201
 [8] (::WGLMakie.var"#7#8"{Observable{Array{Any,1}},Symbol})(::Tuple{typeof(setindex!),Tuple{Array{Point{2,Float32},1},Colon}}) at /home/yakir/.julia/packages/WGLMakie/QIWts/src/serialization.jl:190
 [9] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [10] invokelatest at ./essentials.jl:709 [inlined]
 [11] setindex!(::Observable{Tuple{Function,Tuple}}, ::Tuple{typeof(setindex!),Tuple{Array{Point{2,Float32},1},Colon}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:207
 [12] setindex!(::Observable{Tuple{Function,Tuple}}, ::Tuple{typeof(setindex!),Tuple{Array{Point{2,Float32},1},Colon}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201
 [13] setindex!(::ShaderAbstractions.ArrayUpdater{Array{Point{2,Float32},1}}, ::Array{Point{2,Float32},1}, ::Function) at /home/yakir/.julia/packages/ShaderAbstractions/vrRjy/src/types.jl:46
 [14] setindex! at /home/yakir/.julia/packages/ShaderAbstractions/vrRjy/src/types.jl:59 [inlined]
 [15] (::ShaderAbstractions.var"#9#10"{ShaderAbstractions.Buffer{Point{2,Float32},Array{Point{2,Float32},1}}})(::Array{Point{2,Float32},1}) at /home/yakir/.julia/packages/ShaderAbstractions/vrRjy/src/types.jl:146
 [16] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [17] invokelatest at ./essentials.jl:709 [inlined]
 [18] setindex!(::Observable{Array{Point{2,Float32},1}}, ::Array{Point{2,Float32},1}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:207
 [19] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [20] MapUpdater at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:326 [inlined]
 [21] (::Observables.OnUpdate{Observables.MapUpdater{typeof(first),Array{Point{2,Float32},1}},Tuple{Observable{Tuple{Array{Point{2,Float32},1},Array{Point{2,Float32},1}}}}})(::Tuple{Array{Point{2,Float32},1},Array{Point{2,Float32},1}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [22] setindex!(::Observable{Tuple{Array{Point{2,Float32},1},Array{Point{2,Float32},1}}}, ::Tuple{Array{Point{2,Float32},1},Array{Point{2,Float32},1}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [23] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [24] MapUpdater at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:326 [inlined]
 [25] (::Observables.OnUpdate{Observables.MapUpdater{WGLMakie.var"#73#80"{Observable{StepRange{Int64,Int64}},Observable{StepRange{Int64,Int64}}},Tuple{Array{Point{2,Float32},1},Array{Point{2,Float32},1}}},Tuple{Observable{Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}}}})(::Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [26] setindex!(::Observable{Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}}, ::Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [27] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [28] (::Observables.MapUpdater{WGLMakie.var"#70#77"{Combined{AbstractPlotting.lines,Tuple{CircularBuffer{Point{2,Float32}}}}},Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}})(::CircularBuffer{Point{2,Float32}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:326
 [29] (::Observables.OnUpdate{Observables.MapUpdater{WGLMakie.var"#70#77"{Combined{AbstractPlotting.lines,Tuple{CircularBuffer{Point{2,Float32}}}}},Base.ReinterpretArray{Point{2,Float32},1,Tuple{Point{2,Float32},Point{2,Float32}},GeometryBasics.TupleView{Tuple{Point{2,Float32},Point{2,Float32}},2,1,CircularBuffer{Point{2,Float32}}}}},Tuple{Observable{CircularBuffer{Point{2,Float32}}}}})(::CircularBuffer{Point{2,Float32}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [30] setindex!(::Observable{CircularBuffer{Point{2,Float32}}}, ::CircularBuffer{Point{2,Float32}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [31] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [32] MapUpdater at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:326 [inlined]
 [33] (::Observables.OnUpdate{Observables.MapUpdater{AbstractPlotting.var"#183#185"{Int64},CircularBuffer{Point{2,Float32}}},Tuple{Observable{Tuple{CircularBuffer{Point{2,Float32}}}}}})(::Tuple{CircularBuffer{Point{2,Float32}}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [34] setindex!(::Observable{Tuple{CircularBuffer{Point{2,Float32}}}}, ::Tuple{CircularBuffer{Point{2,Float32}}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [35] setindex!(::Observable{Tuple{CircularBuffer{Point{2,Float32}}}}, ::Tuple{CircularBuffer{Point{2,Float32}}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201
 [36] (::AbstractPlotting.var"#201#203"{DataType,Observable{Tuple{CircularBuffer{Point{2,Float32}}}}})(::Tuple{}, ::Tuple{CircularBuffer{Point{2,Float32}}}) at /home/yakir/.julia/packages/AbstractPlotting/JCbJs/src/interfaces.jl:626
 [37] (::Observables.OnUpdate{AbstractPlotting.var"#201#203"{DataType,Observable{Tuple{CircularBuffer{Point{2,Float32}}}}},Tuple{Observable{Tuple{}},Observable{Tuple{CircularBuffer{Point{2,Float32}}}}}})(::Tuple{CircularBuffer{Point{2,Float32}}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [38] setindex!(::Observable{Tuple{CircularBuffer{Point{2,Float32}}}}, ::Tuple{CircularBuffer{Point{2,Float32}}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [39] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [40] MapUpdater at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:326 [inlined]
 [41] (::Observables.OnUpdate{Observables.MapUpdater{typeof(tuple),Tuple{CircularBuffer{Point{2,Float32}}}},Tuple{Observable{CircularBuffer{Point{2,Float32}}}}})(::CircularBuffer{Point{2,Float32}}) at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:293
 [42] setindex!(::Observable{CircularBuffer{Point{2,Float32}}}, ::CircularBuffer{Point{2,Float32}}; notify::Observables.var"#9#11") at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:205
 [43] setindex! at /home/yakir/.julia/packages/Observables/A7I8T/src/Observables.jl:201 [inlined]
 [44] sample!(::Arduino) at /home/yakir/.julia/dev/SimpleFanControl/src/simpler.jl:72
 [45] (::var"#5#8")() at ./task.jl:356
wait at ./task.jl:267 [inlined]
macro expansion at /home/yakir/.julia/dev/SimpleFanControl/src/simpler.jl:84 [inlined]
(::var"#3#6")() at ./task.jl:356

COBS.jl is mine, is there anything specific I can do to remidy that, I’d love to improve it if possible!

contradict · October 26, 2020, 7:06pm

That one is less clear to me, but I suspect the reload invalidates whatever channel read was using to communicate back to the web page. You probably need to re-start reading when the page reloads.

By self-synchronizing I mean that each message sent over the serial port has an unambiguous start and end. This means that if you start reading part way through a message, you can discard bytes until you get to the next full packet. Common techniques are start end bytes (with escaping to include those bytes like SLIP or HDLC), using a format that re-encodes everything using a restricted byte range (sending text and parsing it on the other end), or using checksums (still can use framing bytes, but only packets that also pass a checksum calculation are valid). Encoding the packet length is always tricky, some protocols avoid this entirely by always sending a fixed length packet or by separately framing a fixed-length header encoding the data length. Other fun things to look at:

Modbus

MAVLink

A good search term is “framing protocol” or “serial framing”.

Implementing your own serial framing protocol is a fun project, but test it well, there are a tremendous number of corner cases to handle in a working protocol.

yakir12 · October 26, 2020, 7:12pm

Right now it would be great to have one package that was very good at solving the framing problem. I encountered this problem a while ago, which led to COBS.jl (see this thread for instance). But I did that because there was no alternative back then, do you know if there is one now? I’d welcome any PR you might have to COBS.jl.

contradict · October 26, 2020, 9:54pm

I’ll take a closer look if I get a chance and see if I can figure out exactly what is going on, I think I misread your code the first time through. It does look like a zero byte should unambiguously end a packet, so I’m not sure what is going on.

yakir12 · October 27, 2020, 7:26am

I’m gonna try to switch to a model where the computer asks the Arduino for a message and then and only then does the Arduino send one. This way the communication will be on the computer’s terms instead of the Arduino continuously spamming the channel and the computer trying to catch up. I’ll push my attempt and report back here…

yakir12 · October 27, 2020, 11:39am

OK, that issue got resolved thanks to the magical Simon Danisch. The problem was that I was adding more and more listeners to an observable by reloading the webpage. Bu clearing them off first I managed to remove that problem altogether. I did change models to the arduino sending messages only when asked to, that might have helped. In any case, it’s working now, which is awesome!