# HPC Slurm Logging Peer Check

**URL:** <https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732>\
**Category:** General Usage\
**Tags:** hpc, logging, io, slurm\
**Created:** [October 8, 2023, 5:40pm UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732 "2023-10-08T17:40:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![TI36XPro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ti36xpro/32/33658_2.png) [@TI36XPro](https://discourse.julialang.org/u/TI36XPro)\
**Post date:** [October 8, 2023, 5:40pm UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732/1 "2023-10-08T17:40:37Z")

</div>

Hello,

I am looking for some help understanding what is probably a familiar problem to those who run Julia regularly on an HPC with Slurm. I’ll explain the issue and then share what I’ve found. Hopefully someone can help me interpret this and understand the issue.

I have a long running script instrumented with `@info`s so I can see information like a summary of the run echoed at the beginning and then regular progress updates (say once every 1000 iterations). I’d like these output to a file during the run.

My slurm script contains the following directives

```julia
#SBATCH --job-name=my_run
#SBATCH --output=my_run.o-%j
#SBATCH --error=my_run.e-%j

```

I understand that the `@info` statements are getting output to `stderr` and indeed when the run finally finishes I see the output in the `my_run.e-` file. But this doesn’t solve my problem as I’d like to see it _during_ the run to check on how it is going.

From what I have read [here](https://discourse.julialang.org/t/flushing-output-for-logging-immediately-without-buffering/26800/4) and [here](https://discourse.julialang.org/t/outputting-to-console-slurm-log-files-in-real-time/10436) it seems this is an issue with Slurm buffering. The buffer doesn’t get filled and so it waits till the end of the run to flush it, something like that (please correct me if I am wrong).

My first attempt to fix this issue followed the approach described [here](https://discourse.julialang.org/t/flushing-output-for-logging-immediately-without-buffering/26800/4). I added the following function to my package and then call it before starting the main function.

```julia
logger = TeeLogger(
    MinLevelLogger(FileLogger(logfile), Logging.Debug),
    MinLevelLogger(ConsoleLogger(stderr, Logging.BelowMinLevel), Logging.Info)
)
global_logger(logger)

```

I understand that `FileLogger` is supposed to flush every time a message is written. That doesn’t appear to be the case though. The log file gets created but I never see anything show up until the end of the run. Also, it seems `ConsoleLogger` also does not automatically flush with each message (I didn’t see any kwargs described that might enable it to, did I miss one?).

I also tried what was described [here](https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe) and submitted my run as follows

```bash
stdbuf -i0 -o0 -e0 sbatch myrun.slurm

```

But that didn’t make a difference either.

Finally, what did work was after every `@info` I added a `flush(stderr)` and now finally it writes to the `my_run.e-` file in real time. I’m a bit disappointed though as adding that after every single `@info` is cumbersome. Is that really the best way to do this? Or is there a better/smarter way to make it so every single `@info` automatically is flushed?

Another alternative I could define my own function I guess that does the following and then replace all the `@info` calls.

```julia
function print_info(msg)
@info "My run info is %s" msg
flush(stderr)
end

```

---

<div class="post-metadata">

**Author:** ![TI36XPro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ti36xpro/32/33658_2.png) [@TI36XPro](https://discourse.julialang.org/u/TI36XPro)\
**Post date:** [October 9, 2023, 1:07am UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732/2 "2023-10-09T01:07:30Z")

</div>

I think I got this resolved.

Instead of `ConsoleLogger` I switched to `FormatLogger` from LoggingExtras.jl. Once I did that, I started getting real time updates during the slurm job run.

Also regarding `FileLogger`, I must have been being impatient because I ran it again a bit ago and started getting messages sent to the log file in real time as well.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 9, 2023, 10:24am UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732/3 "2023-10-09T10:24:06Z")

</div>

That is a good tip. Maybe it could be documented somewhere in ClusterManagers.jl

As an aside, this is a pretty common finding in HPC. If you use PBSPro I think you need to look for job ‘peeking’

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [October 9, 2023, 1:03pm UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732/4 "2023-10-09T13:03:03Z")

</div>

I use a separate log file that I open and write to using `println` and it outputs immediately. It is cumbersome to use. Maybe I should look into a logging package.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 9, 2023, 1:05pm UTC](https://discourse.julialang.org/t/hpc-slurm-logging-peer-check/104732/5 "2023-10-09T13:05:21Z")

</div>

@tbeason This is a function of your cluster filesystem setup. The file is being written by the first compute host allocated to you. I guess you are writing to a shared filesystem.

With PBS you can define rules for how the output files are copied back to main/shared storage. In the jargon this is ‘staging’