Distributed and logging

General Usage
,
1

I am trying to add a timestamp to a logger using LoggingExtras. I would like that timestamp to be propagated across workers, but I haven’t been able to figure out how.

Here’s my MWE:

module TestParLogging

    using Distributed
    using Dates, LoggingExtras

    function foo()
        timestamp_logger(logger) = TransformerLogger(logger) do log
            merge(log, (; message =
                        "$(Dates.format(now(), "mm-dd HH:MM:SS")) $(log.message)"))
        end
        TeeLogger(
            ConsoleLogger(stdout) |> timestamp_logger
        ) |> global_logger

        pmap(1:4) do i
            @info "Worker $(myid()) is starting $i"
            sleep(5)
        end
    end
    export foo

end

When I run this with a single worker I see the logging time stamps, as expected:

[ Info: 08-16 12:14:07 Worker 1 is starting 1
[ Info: 08-16 12:14:12 Worker 1 is starting 2
[ Info: 08-16 12:14:17 Worker 1 is starting 3
[ Info: 08-16 12:14:22 Worker 1 is starting 4

Then I try to run in parallel by loading julia julia -p 4, then these commands:

using Distributed
@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere Pkg.instantiate()
@everywhere using TestParLogging
foo()

The parallelization works, but the time stamps are not shown:

[ Info: Worker 4 is starting 3
[ Info: Worker 5 is starting 4
[ Info: Worker 3 is starting 2
[ Info: Worker 2 is starting 1

What am I doing wrong?

1 Like