Serial synchronization issue and request for some general advice

General Usage Performance
1

Dear Julia Community!

I am currently working on a university project where data is streamed to a Raspberry Pi 4 and needs to be processed in a soft real-time environment. The projects goal is to provide infrastructure for the development of data processing algorithms. These algorithms also need to run in soft real-time on the Raspberry Pi.

The data stream is characterized as following

  • 3 channels of 3-byte (24-bit) ADC samples
  • The channels are recorded simultaneously and continuously at a rate of 50 kHz using a STM32H7 microcontroller
  • Data is transferred to the Raspberry Pi using the USB VCP in batches of 50 samples of all channels, which equals to 3 channels * 3 bytes * 50 samples = 450 bytes of data per transaction. On the Raspberry Pi side, the data is extracted in the extract_data function of my code

Which results in a data rate of 3 channels * 3 bytes * 50 kHz = 450 kBytes per second (+ some overhead)

The code of the main file is here:

Data synchronization problem
The data stream is initiated by sending a start command to the STM32 microcontroller. To prevent data loss, all the functions are compiled before starting the data stream, which works fine. I also try to empty the serial receive buffer before starting the data stream.

function fcm_flush(port)
  sp_flush(port, SP_BUF_BOTH)
end

This does not work and data is misaligned. Even after multiple restarts, the problem persists. However, it does not appear to be a problem when trying the same in an equivalent Python script. The problem also disappears if i first access the data stream using the Python script, then stop the stream and then using the Julia Script.

General advice
Is there some more elegant way of designing the extract_data function? At the moment, it just iterates over the data and, using bit-manipulation, extracts the data for each channel from the stream. It does work, with the justifiable premise of aligned data, but I feel like there must be a more julian way of doing the job.
It would also be very nice to be able to compile the whole program and run it as an executable. Is the package compiler suitable for this?
I am currently using version 1.6.4

Thanks a lot for reading, any thoughts on this are appreciated!

2

It would probably be useful to figure out exactly how the frames are misaligned and whether it’s consistent. One thing I noticed is that the way flushing works seems different in python vs your julia code. In Python it’s only the read buffer which is flushed (by reading all available input), whereas in Julia you’re using SP_BUF_BOTH (do you need SP_BUF_INPUT?)

Another thing is that you’ve got several more calls to flush, stop, etc in Julia. Do any of these commands result in an acknowledgement packet being sent which needs to be read? If you flush the reader too soon, such bytes could arrive asynchronously after the flush. (If you’re expecting an acknowledgement packet with a particular size it should be more reliable to read it explicitly and discard, rather than just hoping a flush will clean things up.)

The timeouts also seem different, though having problems with that should presumably result in Timout() being thrown.

Is there some more elegant way of designing the extract_data function?

I actually think this looks pretty much fine. If you’re reading bespoke binary formats from a microcontroller there’s usually something weird about the format so I find that some explicit bit shifting is quite a reasonable and fast way to unpack such formats.

You can use reinterpret() to reinterpret bits between some format and another, and functions like bswap() and ntoh() to handle endianness.

It would be neat if there were a general purpose helper library for deserializing binary packet formats but I’m not aware of one myself.

1 Like