Subject: Initializing file I/O so that code is agnostic to running locally/on cloud

General Usage
, ,
1

Hello,

We are developing a Julia app, and we want to close the gap between running our code in development and production by making the bulk of our code agnostic to whether it’s running in either stage.

We have already set up logging in a way that the code is agnostic to whether the logs are development or production logs. However, we also want to be able to read and write other non-log files to different sources and destinations in such a stage agnostic way.

For logs we have implemented custom loggers that overwrite the Base.Logging.handle_message() function. At the very start of our code we set the main logger that we want using Base.Logging.global_logger() and then for most of our code we call macros such as @info() and @error() in a logger agnostic way.

What would be a good way to follow the same approach for reading/writing non-log files? My understanding is that macros such as @info() and @error() are for logs but that for reading or writing files we should be using functions like Base.open() / Base.read() / Base.write().

module MyProjectMain

import Logging: global_logger, ConsoleLogger, BelowMinLevel

include(joinpath(@__DIR__, "src/logging/logging_interface.jl"))
import .MyProjectLogging: FilesystemLogger, S3Logger, CloudwatchLogger, create_log_stream!, get_task_metadata
include(joinpath(@__DIR__, "src/run_app.jl"))
import .MyProjectApp: run_app


function julia_main()::Cint

    # 1. Initialize log destinations

    if ENV["STAGE"] == "Development"
        logger_summary = ConsoleLogger(stderr, BelowMinLevel)
        logger = FilesystemLogger(filefolder="/folder/")
    elseif ENV["STAGE"] == "Production"
        logger_summary = CloudwatchLogger(log_group="group")
        create_log_stream!(logger_summary, "stream", logger_summary.group)
        logger = S3Logger(bucket="bucket", filefolder="prefix/")
    else
        throw("Error. Set env var 'STAGE' = 'Development' | 'Production'")
    end
    global_logger(logger)

    # 2. Initialize sources/destinations for other non-log files

    # if ENV["STAGE"] == "Development"
    # ???
    # elseif ENV["STAGE"] == "Production"
    # ???

    # 3. Run the app

    run_app()

end

julia_main(logger, logger_summary)

end
2

For non-log files it probably depends on the data you want to write or read.

For example, if you want to read toml, json, csv files, there are delicate packages for that.
Typically these packages can deal with IO streams.

For example, if you have a toml file, you could use TOML · The Julia Language which works with IO streams as input.

using TOML
io = open("config.toml")
data = TOML.parse(io)
close(io)

So, you might define your own function open_resource(...) which opens the right IO stream on demand.

function open_resource(fn)
   if ENV["STAGE"] == "Development"
        return open(fn)
    elseif ENV["STAGE"] == "Production"
        return # ... your code for another context
    end
end

However, a better syntax might be to support the open do ... approach (which closes files automatically)

open("config.toml") do f 
   data = TOML.pares(io)
end

(See Networking and Streams · The Julia Language )

If you want to support this you could do

function open_resource(f::Function, fn)
   io = open_resource(fn)
   f(io)
   close(io)
end

then the example above becomes

open_resource("config.toml") do f 
   data = TOML.pares(io)
end

I didn’t run the code, sorry if there are typos…

1 Like