Function only works if println statements are included

General Usage
1

Hey, I have the following piece of code that works as intended if the commented lines with the println statements are included. If they are commented out the if statement that compares the dictionaries evaluates to true only once and then never again. However, it does if the println statements are included as mentioned above, which doesn’t make sense to me at all.

I know there are large pieces of code missing but I don’t know how to transform this into a minimal example as there are a lot of things happening under the hood. My guess is something related to compilation that is easy to spot for a proficient Julia programmer.

@kwdef mutable struct PoseLimitedDatasetIterationState{TWrappedDataset <: AbstractSimulationFrameDataset, VS, N <: Integer}
    sensor_locations_wgs84::Dict{String, Tuple{Float64, Float64}} = Dict{String, Tuple{Float64, Float64}}()
    sensor_rotations_deg::Dict{String, Float64} = Dict{String, Float64}()
    sensor_heights_m::Dict{String, Float64} = Dict{String, Float64}()

    wrapped_dataset_iterator::Iterators.Stateful{TWrappedDataset, VS, N}
end

function Base.iterate(dataset::PoseLimitedSimulationFrameDataset)
    return iterate(dataset, PoseLimitedDatasetIterationState(wrapped_dataset_iterator = Iterators.Stateful(dataset.dataset)))
end

function Base.iterate(dataset::PoseLimitedSimulationFrameDataset, state::PoseLimitedDatasetIterationState)
    dataset_iteration_tuple = nothing

    end_wrapped_iteration = false
    start_time = time()

    while !end_wrapped_iteration && !isempty(state.wrapped_dataset_iterator) 
        wrapped_dataset_iteration_value = popfirst!(state.wrapped_dataset_iterator)
                
        simulation_frame = get_simulation_frame(dataset)

        if !isnothing(simulation_frame)
            sensor_locations_wgs84 = Dict{String, Tuple{Float64, Float64}}(map(n -> n => get_sensor_location_wgs84(simulation_frame, n), get_sensor_names(simulation_frame)))
            sensor_rotations_deg = Dict{String, Float64}(map(n -> n => get_sensor_rotation_deg(simulation_frame, n), get_sensor_names(simulation_frame)))
            sensor_heights_m = Dict{String, Float64}(map(n -> n => get_sensor_height_m(simulation_frame, n), get_sensor_names(simulation_frame)))

            if state.sensor_locations_wgs84 != sensor_locations_wgs84 || state.sensor_rotations_deg != sensor_rotations_deg || state.sensor_heights_m != sensor_heights_m
                state.sensor_locations_wgs84 = sensor_locations_wgs84
                state.sensor_rotations_deg = sensor_rotations_deg
                state.sensor_heights_m = sensor_heights_m
                
                dataset_iteration_tuple = (wrapped_dataset_iteration_value, state)
                end_wrapped_iteration = true



            #     println("new pose")
            # else
            #     println("no new pose")
            end
        end

        # this yield statements does the trick to
        # yield()

        if time() - start_time > dataset.timeout_s
            end_wrapped_iteration = true
        end
    end

    return dataset_iteration_tuple
end
2

You will get 20x more help if you make the example runnable.

2 Likes
3

It doesn’t, and that indicates that probably the problem is not there. Is this being run multi-threaded? From my experience, bugs that manifest when one add or remove print statements are always some memory corruption, non-initialized variables, something like that.

1 Like
4

Alright, I think I got a clue now. Thanks for your reply. The program indeed incorporates async tasks, which are necessary to run so that the mentioned if statement can evaluate to true.

If I include the yield statement in my edited original post the problem doesn’t occur anymore, as well. So the problem seems to be related with task scheduling. I guess the other tasks (which are required to run) are not scheduled anymore unless the yield or println statements are included.

5

You are probably blocking the task scheduler loop from running when you do not yield. Note that your @async tasks are all trying to run on a single thread. You may be interested in Threads.@spawn to run tasks on other threads. Make sure to start Julia with multiple threads.

1 Like
6

The tasks are created using the @spawn macro just as it is recommended in the documentation. Julia is started with multiple threads, as well. Still, this leaves me a bit confused and shows that I did not really understand how task scheduling works in Julia and when or where yield statements are required.

7

Basically you are running your loop on the thread where the scheduler event loop is also running.

@gbaraldi has proposed a fixed here:

1 Like
8

Thanks for that reference. I think I narrowed it down. So, there is a task that runs on a different thread that is receiving data using ZMQ.jl. This task updates the simulation_frame object in the code piece above based on the received data. Only if there are new values in that frame the condition in the if statement comparing the dictionaries can be true.

I’m guessing that this receiving operation of ZMQ.jl somehow requires the UV loop to run, but since the main thread never has time to go back to it the task will never receive any new data. This is why it doesn’t even matter that it runs on a separate thread. Including the yield statement allows the UV loop to be executed again and the new data can be received, which in turn updates the simulation_frame object and the condition can be true.

Those are my thoughts so far, maybe it helps someone with similar problems.

My current solution to this:

function main()
    # main program code usually run on the main thread
end

# main()
# instead of calling main directly it is wrapped inside a task
wait(@spawn main())
9

Julia uses a working fork of libuv for multitasking and I/O.

So yes, those two things would be linked.