Help Needed: S3 Connection Error with MinIO in Development Environment

Specific Domains Data
,
10

Thank you, @ericphanson, for providing such a great initial insight. Your comment about the missing values really caught my attention. At first, I was quite surprised, so I decided to bring in my teammate, @tencnivel, for a pair programming session to dive deeper into this.

We updated the HTTP package, but unfortunately, we’re still encountering the same issue with the error:

SubArray{UInt8,1,Memory{UInt8},Tuple{UnitRange{Int64}},true} to an object of type

SubArray{UInt8,1,Vector{UInt8},Tuple{UnitRange{Int64}},true}

Very strange indeed. My teammate (shoutout to @tencnivel) came across this post and read it and share it to me:

I downgraded my julia version (1.11 => 1.10) and tada everything works fine.

By the way, here’s my updated script that works well with MinIO and likely other standard S3 integrations (I don’t see any working example anywhere, maybe it will help).

import Pkg

# Activate the project environment located at the script's directory
Pkg.activate(@__DIR__)
Pkg.add("AWS")
Pkg.add("AWSS3")
Pkg.instantiate()  # Ensures all dependencies are installed
using AWS
using AWSS3


struct MinioConfig <: AbstractAWSConfig
    endpoint::String
    region::String
    require_ssl_verification::Bool
    creds
 end
 AWS.region(c::MinioConfig) = c.region
 AWS.credentials(c::MinioConfig) = c.creds

struct SimpleCredentials
    access_key_id::String
    secret_key::String
    token::String
end
AWS.check_credentials(c::SimpleCredentials) = c

#ENV["AWS_SECRET_ACCESS_KEY"] = "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"
#ENV["AWS_ACCESS_KEY_ID"] = "3nbu1tyiaQ2lN0A5DYmp"

function AWS.generate_service_url(aws::MinioConfig, service::String, resource::String)
    service == "s3" || throw(ArgumentError("Can only handle s3 requests to Minio"))
    return string(aws.endpoint, resource)
end

aws_config = AWS.global_aws_config(MinioConfig("http://127.0.0.1:9000", "",false, SimpleCredentials("3nbu1tyiaQ2lN0A5DYmp", "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8", "")))

@info "after aws_config"
try
    p = S3Path("s3://testapi/test1.txt")  # provides an filesystem-like interface
    #buckets = s3_list_buckets(;aws_config)
    res = read(p, byte_range=1:4)  # returns b"some"
    @info "res:" String(res)
catch e
    println("Stack trace:")
    showerror(stderr, e, Base.catch_backtrace())
    println(aws_config)
end
@info "after s3"

Btw, I noticed that everyone in this thread is helping others. Feel free to tag me if anyone encounters issues with S3 (and other AWS vendors) —I’d be happy to assist!

thx guys !!
cc @ericphanson , @tencnivel , @drizk1

1 Like