Help Needed: S3 Connection Error with MinIO in Development Environment

Specific Domains Data
,
1

Hello community,

I’m trying to connect to a MinIO instance (an S3-compatible provider) in my development environment using Julia and the AWS and AWSS3 packages. I’ve written a script to list buckets, but I’m encountering a MethodError that I don’t fully understand.

import Pkg

# Activate the project environment located at the script's directory
Pkg.activate(@__DIR__)
Pkg.add("AWS")
Pkg.add("AWSS3")
Pkg.instantiate()  # Ensures all dependencies are installed

# Now, load your packages
using AWS
using AWSS3

# Create AWSConfig object with positional arguments
ws_credentials = AWSCredentials()  # Replace with your access key ID and secret access key
aws_config = AWSConfig(creds=ws_credentials)  # Replace with your AWS region
println("generate aws_config")

# Test the connection by listing buckets
try
    println("avant list buckets")
    buckets = AWSS3.s3_list_buckets()
    println("Buckets:")
    for bucket in buckets
        println("- ", bucket.name)
    end
catch e
    println("plop plop")
    println("An error occurred: $(e)")
end

println("end of script")

Here’s the error message I receive when running the script:
An error occurred: MethodError(convert, (SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, UInt8[0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x34, 0x30, 0x33, 0x20, 0x46, 0x6f, 0x72, 0x62, 0x69, 0x64, 0x64, 0x65, 0x6e, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2d, 0x69, 0x64, 0x3a, 0x20, 0x4a, 0x34, 0x58, 0x48, 0x4a, 0x4b, 0x47, 0x5a, 0x46, 0x48, 0x47, 0x31, 0x31, 0x4e, 0x42, 0x4a, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x69, 0x64, 0x2d, 0x32, 0x3a, 0x20, 0x54, 0x75, 0x71, 0x68, 0x58, 0x61, 0x39, 0x42, 0x52, 0x59, 0x58, 0x41, 0x79, 0x68, 0x63, 0x58, 0x79, 0x31, 0x4a, 0x36, 0x33, 0x46, 0x32, 0x74, 0x4c, 0x32, 0x72, 0x34, 0x57, 0x44, 0x31, 0x70, 0x59, 0x34, 0x4d, 0x67, 0x42, 0x2b, 0x4b, 0x39, 0x6b, 0x55, 0x65, 0x69, 0x37, 0x54, 0x2b, 0x71, 0x79, 0x30, 0x57, 0x69

I set the variable as ENV var (it is a dev local values so I can share):

**julia>** ENV["AWS_ACCESS_KEY_ID"] = "3nbu1tyiaQ2lN0A5DYmp"

"3nbu1tyiaQ2lN0A5DYmp"

**julia>** ENV["AWS_CONTAINER_CREDENTIALS_FULL_URI"] = "http://127.0.0.1:9000"

"http://127.0.0.1:9000"

**julia>** ENV["AWS_SECRET_ACCESS_KEY"] = "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"

"WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"

What I’ve tried:

  1. Double-checked that the MinIO credentials and endpoint are correctly configured.
  2. Verified that the MinIO server is accessible via other tools (e.g., mc client or a browser).
  3. Used the MinIO library but same issues

Julia Version: 1.10

Do you have any ideas?

2

Has anyone successfully uploaded files to S3 using the AWS library? Alternatively, has anyone used a different library or method to achieve this?

Thank youuuu

1 Like
3

You can use duckdb to connect to an s3 file

4

I have used Mino and AWS both successfully with AWS.jl. I’m not sure what problem you are facing, can you include a full stacktrace to show where the error is happening?

5

Thanks for you replay.

Based on the link you provided, I updated base code:

Pkg.activate(@__DIR__)
Pkg.add("AWS")
Pkg.instantiate()  # Ensures all dependencies are installed

using AWS: @service
@service S3


@info "before creds"
awscreds = AWS.AWSCredentials(; profile=nothing)
aws_config = AWSConfig(; creds=awscreds, region="eu-central-1")
@info "after aws_config"
try
    S3.list_objects("testapi";aws_config)
catch e
    @info e
    println("Stack trace:")
    println(Base.catch_backtrace())
    println(aws_config)
end
@info "after s3"

Stack trace:

WARNING: replacing module S3.

**[ Info:** before creds

**[ Info:** after aws_config

**[ Info:** MethodError(convert, (SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, UInt8[0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x33, 0x30, 0x31, 0x20, 0x4d, 0x6f, 0x76, 0x65, 0x64, 0x20, 0x50, 0x65, 0x72, 0x6d, 0x61, 0x6e, 0x65, 0x6e, 0x74, 0x6c, 0x79, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2d, 0x72, 0x65, 0x67, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x61, 0x70, 0x2d, 0x73, 0x6f, 0x75, 0x74, 0x68, 0x2d, 0x31, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2d, 0x69, 0x64, 0x3a, 0x20, 0x51, 0x57, 0x48, 0x54, 0x33, 0x51, 0x45, 0x51, 0x50, 0x35, 0x33, 0x32, 0x57, 0x51, 0x58, 0x41, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x69, 0x64, 0x2d, 0x32, 0x3a, 0x20, 0x44, 0x4c, 0x62, 0x45, 0x33, 0x44, 0x75, 0x58, 0x31, 0x50, 0x6c, 0x45, 0x59, 0x75, 0x74, 0x59, 0x43, 0x59, 0x68, 0x6a, 0x6c, 0x37, 0x45, 0x4f, 0x75, 0x61, 0x76, 0x62, 0x71, 0x33, 0x39, 0x77, 0x78, 0x35, 0x77, 0x4c, 0x34, 0x36, 0x4c, 0x52, 0x68, 0x54, 0x52, 0x37, 0x66, 0x2b, 0x7a, 0x4f, 0x39, 0x49, 0x2b, 0x66, 0x47, 0x59, 0x75, 0x38, 0x59, 0x59, 0x70, 0x52, 0x4b, 0x64, 0x65, 0x6e, 0x70, 0x32, 0x63, 0x63, 0x6f, 0x63, 0x4f, 0x51, 0x57, 0x73, 0x63, 0x3d, 0x0d, 0x0a, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x78, 0x6d, 0x6c, 0x0d, 0x0a, 0x54, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x65, 0x72, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x63, 0x68, 0x75, 0x6e, 0x6b, 0x65, 0x64, 0x0d, 0x0a, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x53, 0x75, 0x6e, 0x2c, 0x20, 0x32, 0x34, 0x20, 0x4e, 0x6f, 0x76, 0x20, 0x32, 0x30, 0x32, 0x34, 0x20, 0x31, 0x39, 0x3a, 0x33, 0x39, 0x3a, 0x32, 0x38, 0x20, 0x47, 0x4d, 0x54, 0x0d, 0x0a, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x3a, 0x20, 0x41, 0x6d, 0x61, 0x7a, 0x6f, 0x6e, 0x53, 0x33, 0x0d, 0x0a, 0x0d, 0x0a]), 0x0000000000007c9f)

Stack trace
Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x0000000100eda61f, Ptr{Nothing} @0x0000000100eda557, Ptr{Nothing} @0x0000000100edf75b, Ptr{Nothing} @0x000000030a5ec20b, Ptr{Nothing} @0x000000030a5ec2b7, Ptr{Nothing} @0x0000000173a2c0af, Ptr{Nothing} @0x0000000173a34053, Ptr{Nothing} @0x0000000173a340fb, Ptr{Nothing} @0x00000001739e0047, Ptr{Nothing} @0x00000001739e8073, Ptr{Nothing} @0x00000001739f061b, Ptr{Nothing} @0x00000001739f8073, Ptr{Nothing} @0x00000001739f80af, Ptr{Nothing} @0x000000017382c5b3, Ptr{Nothing} @0x000000017382cd77, Ptr{Nothing} @0x00000001738201b3, Ptr{Nothing} @0x000000017382030b, Ptr{Nothing} @0x0000000100eee6ef, Ptr{Nothing} @0x000000030a2cc0fb, Ptr{Nothing} @0x00000001737f019b, Ptr{Nothing} @0x0000000173800b1b, Ptr{Nothing} @0x000000017380828b, Ptr{Nothing} @0x00000001738102bb, Ptr{Nothing} @0x000000017381806b, Ptr{Nothing} @0x00000001738180ab, Ptr{Nothing} @0x0000000100eee6ef, Ptr{Nothing} @0x000000030a2900f7, Ptr{Nothing} @0x00000001737dc19f, Ptr{Nothing} @0x00000001737dc1e3, Ptr{Nothing} @0x00000001737804f7, Ptr{Nothing} @0x000000017378846f, Ptr{Nothing} @0x0000000173790017, Ptr{Nothing} @0x000000017379004b, Ptr{Nothing} @0x000000030a0bc413, Ptr{Nothing} @0x00000001737500f3, Ptr{Nothing} @0x00000001737680c7, Ptr{Nothing} @0x00000001737704d3, Ptr{Nothing} @0x0000000173778017, Ptr{Nothing} @0x000000017377804b, Ptr{Nothing} @0x000000011feb8637, Ptr{Nothing} @0x0000000309f9c073, Ptr{Nothing} @0x0000000173710b57, Ptr{Nothing} @0x0000000304a5814f, Ptr{Nothing} @0x0000000304a58223, Ptr{Nothing} @0x0000000100efe31f, Ptr{Nothing} @0x0000000100efdd1f, Ptr{Nothing} @0x0000000100efc9d7, Ptr{Nothing} @0x0000000100efcccb, Ptr{Nothing} @0x0000000100efcff3, Base.InterpreterIP in top-level CodeInfo for Main at statement 6, Ptr{Nothing} @0x0000000100f1648b, Ptr{Nothing} @0x0000000100f1620b, Ptr{Nothing} @0x0000000100f1719f, Ptr{Nothing} @0x0000000131597eb7, Ptr{Nothing} @0x0000000131121f67, Ptr{Nothing} @0x000000011f00c04b, Ptr{Nothing} @0x000000011f00c087, Ptr{Nothing} @0x0000000100efe31f, Ptr{Nothing} @0x0000000100efdd1f, Ptr{Nothing} @0x0000000100efc9d7, Ptr{Nothing} @0x0000000100efcff3, Base.InterpreterIP in top-level CodeInfo for Main at statement 0, Ptr{Nothing} @0x0000000100f1648b, Ptr{Nothing} @0x0000000100f1620b, Ptr{Nothing} @0x0000000100f1719f, Ptr{Nothing} @0x000000013d38cb2b, Ptr{Nothing} @0x000000013d2235c3, Ptr{Nothing} @0x000000013d2236cb, Ptr{Nothing} @0x000000013d311b8f, Ptr{Nothing} @0x000000013d317037, Ptr{Nothing} @0x000000013d29ae2f, Ptr{Nothing} @0x000000013d22372b, Ptr{Nothing} @0x000000013d3e40d3, Ptr{Nothing} @0x000000013d28495b, Ptr{Nothing} @0x0000000100eef12f, Ptr{Nothing} @0x000000013146df2f, Ptr{Nothing} @0x000000013146efef, Ptr{Nothing} @0x0000000130d96803, Ptr{Nothing} @0x0000000100f47c6b, Ptr{Nothing} @0x0000000100f47b77]

AWSConfig((3nbu1tyiaQ2lN0A5DYmp, WEO..., 146138512-12-31T23:59:59), "eu-central-1", "json")

**[ Info:** after s3

Wdyt?

6

better stack trace:

Stacktrace:

[1] **readuntil(**buf::IOBuffer, find_delimiter::typeof(HTTP.Parsers.find_end_of_header)**)**

@ HTTP.IOExtras ~/.julia/packages/HTTP/aTjcj/src/IOExtras.jl:139

[2] **readuntil(**t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function, sizehint::Int64**)**

@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:296

[3] **readuntil(**t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function**)**

@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:292

[4] **readheaders(**io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, message::HTTP.Messages.Response**)**

@ HTTP.Messages ~/.julia/packages/HTTP/aTjcj/src/Messages.jl:537

[5] **startread(**http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}**)**

@ HTTP.Streams ~/.julia/packages/HTTP/aTjcj/src/Streams.jl:170

[6] **macro expansion**

@ ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:67 [inlined]

[7] **macro expansion**

@ ./task.jl:498 [inlined]

[8] **request(**::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Base.BufferStream, iofunction::Nothing, verbose::Int64, kw::@Kwargs{require_ssl_verification::Bool}**)**

@ HTTP.StreamRequest ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:57

[9] **request(**::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

@ HTTP.ConnectionRequest ~/.julia/packages/HTTP/aTjcj/src/ConnectionRequest.jl:108

[10] **request(**::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::URIs.URI, ::Vararg{Any}; kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

@ HTTP.ExceptionRequest ~/.julia/packages/HTTP/aTjcj/src/ExceptionRequest.jl:19

[11] **request(**::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

@ HTTP.MessageRequest ~/.julia/packages/HTTP/aTjcj/src/MessageRequest.jl:66

[12] **request(**::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

@ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/aTjcj/src/BasicAuthRequest.jl:28

[13] **request(**::Type{HTTP.TopRequest.TopLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, ::String, ::Vararg{Any}; kwargs::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

@ HTTP.TopRequest ~/.julia/packages/HTTP/aTjcj/src/TopRequest.jl:15

[14] **macro expansion**

@ ~/.julia/packages/Mocking/rzcpa/src/mock.jl:50 [inlined]

[15] **(::AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}})()**

@ AWS ~/.julia/packages/AWS/E9zQ4/src/utilities/request.jl:221

[16] **(::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(**; kwargs::@Kwargs{}**)**

@ Base ./error.jl:300

[17] **(::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()**
7

If I paste vector into the REPL as v=UInt8[...] and run println(String(copy(v))), I see

HTTP/1.1 301 Moved Permanently
x-amz-bucket-region: ap-south-1
x-amz-request-id: QWHT3QEQP532WQXA
x-amz-id-2: DLbE3DuX1PlEYutYCYhjl7EOuavbq39wx5wL46LRhTR7f+zO9I+fGYu8YYpRKdenp2ccocOQWsc=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sun, 24 Nov 2024 19:39:28 GMT
Server: AmazonS3

Not sure if that is helpful or not.

I can’t find this line, IOExtras.jl:139 in HTTP.jl’s source code unless I go back to pre-v1.0. Are you possibly on a very old version of HTTP.jl? Try updating to the latest version.

8

pkg> status HTTP

Status ~/perso/docker/minio/Project.toml

⌅ [cd3eb016] HTTP v0.9.17

Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use status --outdated

I have now the following error:

[ Info: after aws_config

Stack trace:

MethodError: Cannot `convert` an object of type

SubArray{UInt8,1,Memory{UInt8},Tuple{UnitRange{Int64}},true} to an object of type

SubArray{UInt8,1,Vector{UInt8},Tuple{UnitRange{Int64}},true}

The function `convert` exists, but no method is defined for this combination of argument types.

Closest candidates are:

SubArray{T, N, P, I, L}(::Any, ::Any, ::Any, ::Any) where {T, N, P, I, L}

@ Base subarray.jl:19

convert(::Type{T}, ::T) where T

@ Base Base.jl:126

convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray

@ LinearAlgebra /Applications/Julia-1.11.app/Contents/Resources/julia/share/julia/stdlib/v1.11/LinearAlgebra/src/factorization.jl:104

...

Stacktrace:

[1] **readuntil(**buf::IOBuffer, find_delimiter::typeof(HTTP.Parsers.find_end_of_header)**)**

@ HTTP.IOExtras ~/.julia/packages/HTTP/aTjcj/src/IOExtras.jl:139

[2] **readuntil(**t::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, f::Function, sizehint::Int64**)**

@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:296

[3] **readuntil(**t::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, f::Function**)**

@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:292

[4] **readheaders(**io::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, message::HTTP.Messages.Response**)**

@ HTTP.Messages ~/.julia/packages/HTTP/aTjcj/src/Messages.jl:537

[5] **startread(**http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}**)**

@ HTTP.Streams ~/.julia/packages/HTTP/aTjcj/src/Streams.jl:170

[6] **macro expansion**

@ ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:67 [inlined]

[7] **macro expansion**

@ ./task.jl:498 [inlined]

[8] **request(**::Type{StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Base.BufferStream, iofunction::Nothing, verbose::Int64, kw::@Kwargs{require_ssl_verification::Bool}**)**

@ HTTP.StreamRequest ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:57

[9] **request(**::Type{ConnectionPoolLayer{StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**

I think we made progress, but I’m still facing some strange issues.

My current setup involves Julia 1.11 running on a Mac with Apple Silicon (M2), and MinIO is hosted via Docker.

Wdyt, to troubleshoot further, using the same configuration on an Ubuntu system to see if the issue persists across platforms.

9

You need a newer version of the package HTTP.jl. Press ] to go into the package mode and try add HTTP@1.10