# Subject: Initializing file I/O so that code is agnostic to running locally/on cloud

**URL:** https://discourse.julialang.org/t/subject-initializing-file-i-o-so-that-code-is-agnostic-to-running-locally-on-cloud/85769
**Category:** General Usage
**Tags:** question, io, code-organization
**Created:** [August 15, 2022, 11:51am UTC](https://discourse.julialang.org/t/subject-initializing-file-i-o-so-that-code-is-agnostic-to-running-locally-on-cloud/85769 "2022-08-15T11:51:03Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![Pablo](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Pablo](https://discourse.julialang.org/u/Pablo)
#### Post date: [August 15, 2022, 11:51am UTC](https://discourse.julialang.org/t/subject-initializing-file-i-o-so-that-code-is-agnostic-to-running-locally-on-cloud/85769/1 "2022-08-15T11:51:03Z")

</div>

Hello,

We are developing a Julia app, and we want to close the gap between running our code in development and production by making the bulk of our code agnostic to whether it’s running in either stage.

We have already set up logging in a way that the code is agnostic to whether the logs are development or production logs. However, we also want to be able to read and write other non-log files to different sources and destinations in such a stage agnostic way.

For logs we have implemented custom loggers that overwrite the Base.Logging.handle\_message() function. At the very start of our code we set the main logger that we want using Base.Logging.global\_logger() and then for most of our code we call macros such as @info() and @error() in a logger agnostic way.

What would be a good way to follow the same approach for reading/writing non-log files? My understanding is that macros such as @info() and @error() are for logs but that for reading or writing files we should be using functions like Base.open() / Base.read() / Base.write().

```julia
module MyProjectMain

import Logging: global_logger, ConsoleLogger, BelowMinLevel

include(joinpath(@ __DIR__ , "src/logging/logging_interface.jl"))
import .MyProjectLogging: FilesystemLogger, S3Logger, CloudwatchLogger, create_log_stream!, get_task_metadata
include(joinpath(@ __DIR__ , "src/run_app.jl"))
import .MyProjectApp: run_app

function julia_main()::Cint

    # 1. Initialize log destinations

    if ENV["STAGE"] == "Development"
        logger_summary = ConsoleLogger(stderr, BelowMinLevel)
        logger = FilesystemLogger(filefolder="/folder/")
    elseif ENV["STAGE"] == "Production"
        logger_summary = CloudwatchLogger(log_group="group")
        create_log_stream!(logger_summary, "stream", logger_summary.group)
        logger = S3Logger(bucket="bucket", filefolder="prefix/")
    else
        throw("Error. Set env var 'STAGE' = 'Development' | 'Production'")
    end
    global_logger(logger)

    # 2. Initialize sources/destinations for other non-log files

    # if ENV["STAGE"] == "Development"
    # ???
    # elseif ENV["STAGE"] == "Production"
    # ???

    # 3. Run the app

    run_app()

end

julia_main(logger, logger_summary)

end

```

---

<div class="post-metadata">

### Author: ![SteffenPL](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/steffenpl/32/206270_2.png) [@SteffenPL](https://discourse.julialang.org/u/SteffenPL)
#### Post date: [August 15, 2022, 2:05pm UTC](https://discourse.julialang.org/t/subject-initializing-file-i-o-so-that-code-is-agnostic-to-running-locally-on-cloud/85769/2 "2022-08-15T14:05:15Z")

</div>

For non-log files it probably depends on the data you want to write or read.

For example, if you want to read `toml`, `json`, `csv` files, there are delicate packages for that.  
Typically these packages can deal with `IO` streams.

For example, if you have a toml file, you could use [TOML · The Julia Language](https://docs.julialang.org/en/v1/stdlib/TOML/#TOML.parse) which works with `IO` streams as input.

```julia
using TOML
io = open("config.toml")
data = TOML.parse(io)
close(io)

```

So, you might define your own function `open_resource(...)` which opens the right `IO` stream on demand.

```julia
function open_resource(fn)
   if ENV["STAGE"] == "Development"
        return open(fn)
    elseif ENV["STAGE"] == "Production"
        return # ... your code for another context
    end
end

```

However, a better syntax might be to support the `open do ...` approach (which closes files automatically)

```julia
open("config.toml") do f 
   data = TOML.pares(io)
end

```

(See [Networking and Streams · The Julia Language](https://docs.julialang.org/en/v1/manual/networking-and-streams/#Text-I/O) )

If you want to support this you could do

```julia
function open_resource(f::Function, fn)
   io = open_resource(fn)
   f(io)
   close(io)
end

```

then the example above becomes

```julia
open_resource("config.toml") do f 
   data = TOML.pares(io)
end

```

* * *

I didn’t run the code, sorry if there are typos…