@test_logs behavior when using with_logger

When trying to test for logging inside of a custom logger, the @test_logs macro seems to be unable to capture the logs.

Here is an MWE

using Test
using Logging
@test_logs (:error, "test") match_mode = :any Logging.with_logger(Logging.SimpleLogger()) do 
       @error "test"
       end

Which returns empty captured logs

┌ Error: test
└ @ Main REPL[58]:2
Log Test Failed at REPL[58]:1
  Expression: Logging.with_logger(Logging.SimpleLogger()) do 
    #= REPL[58]:2 =# @error "test"
end
  Log Pattern: :error match_mode = :any
  Captured Logs: 

Anyone has experienced this and knows how to fix?