# Help Needed: S3 Connection Error with MinIO in Development Environment
**URL:** https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957
**Category:** Data
**Tags:** question, package
**Created:** [November 22, 2024, 1:29pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957 "2024-11-22T13:29:36Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 22, 2024, 1:29pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/1 "2024-11-22T13:29:36Z")
Hello community,
I’m trying to connect to a MinIO instance (an S3-compatible provider) in my development environment using Julia and the `AWS` and `AWSS3` packages. I’ve written a script to list buckets, but I’m encountering a `MethodError` that I don’t fully understand.
```julia
import Pkg
# Activate the project environment located at the script's directory
Pkg.activate(@ __DIR__ )
Pkg.add("AWS")
Pkg.add("AWSS3")
Pkg.instantiate() # Ensures all dependencies are installed
# Now, load your packages
using AWS
using AWSS3
# Create AWSConfig object with positional arguments
ws_credentials = AWSCredentials() # Replace with your access key ID and secret access key
aws_config = AWSConfig(creds=ws_credentials) # Replace with your AWS region
println("generate aws_config")
# Test the connection by listing buckets
try
println("avant list buckets")
buckets = AWSS3.s3_list_buckets()
println("Buckets:")
for bucket in buckets
println("- ", bucket.name)
end
catch e
println("plop plop")
println("An error occurred: $(e)")
end
println("end of script")
```
Here’s the error message I receive when running the script:
`An error occurred: MethodError(convert, (SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, UInt8[0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x34, 0x30, 0x33, 0x20, 0x46, 0x6f, 0x72, 0x62, 0x69, 0x64, 0x64, 0x65, 0x6e, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2d, 0x69, 0x64, 0x3a, 0x20, 0x4a, 0x34, 0x58, 0x48, 0x4a, 0x4b, 0x47, 0x5a, 0x46, 0x48, 0x47, 0x31, 0x31, 0x4e, 0x42, 0x4a, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x69, 0x64, 0x2d, 0x32, 0x3a, 0x20, 0x54, 0x75, 0x71, 0x68, 0x58, 0x61, 0x39, 0x42, 0x52, 0x59, 0x58, 0x41, 0x79, 0x68, 0x63, 0x58, 0x79, 0x31, 0x4a, 0x36, 0x33, 0x46, 0x32, 0x74, 0x4c, 0x32, 0x72, 0x34, 0x57, 0x44, 0x31, 0x70, 0x59, 0x34, 0x4d, 0x67, 0x42, 0x2b, 0x4b, 0x39, 0x6b, 0x55, 0x65, 0x69, 0x37, 0x54, 0x2b, 0x71, 0x79, 0x30, 0x57, 0x69`
I set the variable as ENV var (it is a dev local values so I can share):
```julia
**julia>** ENV["AWS_ACCESS_KEY_ID"] = "3nbu1tyiaQ2lN0A5DYmp"
"3nbu1tyiaQ2lN0A5DYmp"
**julia>** ENV["AWS_CONTAINER_CREDENTIALS_FULL_URI"] = "http://127.0.0.1:9000"
"http://127.0.0.1:9000"
**julia>** ENV["AWS_SECRET_ACCESS_KEY"] = "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"
"WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"
```
**What I’ve tried:**
1. Double-checked that the MinIO credentials and endpoint are correctly configured.
2. Verified that the MinIO server is accessible via other tools (e.g., `mc` client or a browser).
3. Used the MinIO library but same issues
Julia Version: 1.10
Do you have any ideas?
---
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 22, 2024, 1:39pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/2 "2024-11-22T13:39:35Z")
Has anyone successfully uploaded files to S3 using the AWS library? Alternatively, has anyone used a different library or method to achieve this?
Thank youuuu
---
### Author:  [@drizk1](https://discourse.julialang.org/u/drizk1)
#### Post date: [November 22, 2024, 2:58pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/3 "2024-11-22T14:58:37Z")
You can use [duckdb](https://duckdb.org/docs/extensions/httpfs/s3api.html) to connect to an s3 file
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [November 24, 2024, 2:05pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/4 "2024-11-24T14:05:18Z")
I have used Mino and AWS both successfully with AWS.jl. I’m not sure what problem you are facing, can you include a full stacktrace to show where the error is happening?
---
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 24, 2024, 7:43pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/5 "2024-11-24T19:43:47Z")
> [@ericphanson](#):
>
> can you include a full stacktrace to show where the error is happening
Thanks for you replay.
Based on the link you provided, I updated base code:
```julia
Pkg.activate(@ __DIR__ )
Pkg.add("AWS")
Pkg.instantiate() # Ensures all dependencies are installed
using AWS: @service
@service S3
@info "before creds"
awscreds = AWS.AWSCredentials(; profile=nothing)
aws_config = AWSConfig(; creds=awscreds, region="eu-central-1")
@info "after aws_config"
try
S3.list_objects("testapi";aws_config)
catch e
@info e
println("Stack trace:")
println(Base.catch_backtrace())
println(aws_config)
end
@info "after s3"
```
Stack trace:
```julia
WARNING: replacing module S3.
**[ Info:** before creds
**[ Info:** after aws_config
**[Info:** MethodError(convert, (SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, UInt8[0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x33, 0x30, 0x31, 0x20, 0x4d, 0x6f, 0x76, 0x65, 0x64, 0x20, 0x50, 0x65, 0x72, 0x6d, 0x61, 0x6e, 0x65, 0x6e, 0x74, 0x6c, 0x79, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2d, 0x72, 0x65, 0x67, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x61, 0x70, 0x2d, 0x73, 0x6f, 0x75, 0x74, 0x68, 0x2d, 0x31, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2d, 0x69, 0x64, 0x3a, 0x20, 0x51, 0x57, 0x48, 0x54, 0x33, 0x51, 0x45, 0x51, 0x50, 0x35, 0x33, 0x32, 0x57, 0x51, 0x58, 0x41, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x69, 0x64, 0x2d, 0x32, 0x3a, 0x20, 0x44, 0x4c, 0x62, 0x45, 0x33, 0x44, 0x75, 0x58, 0x31, 0x50, 0x6c, 0x45, 0x59, 0x75, 0x74, 0x59, 0x43, 0x59, 0x68, 0x6a, 0x6c, 0x37, 0x45, 0x4f, 0x75, 0x61, 0x76, 0x62, 0x71, 0x33, 0x39, 0x77, 0x78, 0x35, 0x77, 0x4c, 0x34, 0x36, 0x4c, 0x52, 0x68, 0x54, 0x52, 0x37, 0x66, 0x2b, 0x7a, 0x4f, 0x39, 0x49, 0x2b, 0x66, 0x47, 0x59, 0x75, 0x38, 0x59, 0x59, 0x70, 0x52, 0x4b, 0x64, 0x65, 0x6e, 0x70, 0x32, 0x63, 0x63, 0x6f, 0x63, 0x4f, 0x51, 0x57, 0x73, 0x63, 0x3d, 0x0d, 0x0a, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x78, 0x6d, 0x6c, 0x0d, 0x0a, 0x54, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x65, 0x72, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x63, 0x68, 0x75, 0x6e, 0x6b, 0x65, 0x64, 0x0d, 0x0a, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x53, 0x75, 0x6e, 0x2c, 0x20, 0x32, 0x34, 0x20, 0x4e, 0x6f, 0x76, 0x20, 0x32, 0x30, 0x32, 0x34, 0x20, 0x31, 0x39, 0x3a, 0x33, 0x39, 0x3a, 0x32, 0x38, 0x20, 0x47, 0x4d, 0x54, 0x0d, 0x0a, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x3a, 0x20, 0x41, 0x6d, 0x61, 0x7a, 0x6f, 0x6e, 0x53, 0x33, 0x0d, 0x0a, 0x0d, 0x0a]), 0x0000000000007c9f)
Stack trace
Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x0000000100eda61f, Ptr{Nothing} @0x0000000100eda557, Ptr{Nothing} @0x0000000100edf75b, Ptr{Nothing} @0x000000030a5ec20b, Ptr{Nothing} @0x000000030a5ec2b7, Ptr{Nothing} @0x0000000173a2c0af, Ptr{Nothing} @0x0000000173a34053, Ptr{Nothing} @0x0000000173a340fb, Ptr{Nothing} @0x00000001739e0047, Ptr{Nothing} @0x00000001739e8073, Ptr{Nothing} @0x00000001739f061b, Ptr{Nothing} @0x00000001739f8073, Ptr{Nothing} @0x00000001739f80af, Ptr{Nothing} @0x000000017382c5b3, Ptr{Nothing} @0x000000017382cd77, Ptr{Nothing} @0x00000001738201b3, Ptr{Nothing} @0x000000017382030b, Ptr{Nothing} @0x0000000100eee6ef, Ptr{Nothing} @0x000000030a2cc0fb, Ptr{Nothing} @0x00000001737f019b, Ptr{Nothing} @0x0000000173800b1b, Ptr{Nothing} @0x000000017380828b, Ptr{Nothing} @0x00000001738102bb, Ptr{Nothing} @0x000000017381806b, Ptr{Nothing} @0x00000001738180ab, Ptr{Nothing} @0x0000000100eee6ef, Ptr{Nothing} @0x000000030a2900f7, Ptr{Nothing} @0x00000001737dc19f, Ptr{Nothing} @0x00000001737dc1e3, Ptr{Nothing} @0x00000001737804f7, Ptr{Nothing} @0x000000017378846f, Ptr{Nothing} @0x0000000173790017, Ptr{Nothing} @0x000000017379004b, Ptr{Nothing} @0x000000030a0bc413, Ptr{Nothing} @0x00000001737500f3, Ptr{Nothing} @0x00000001737680c7, Ptr{Nothing} @0x00000001737704d3, Ptr{Nothing} @0x0000000173778017, Ptr{Nothing} @0x000000017377804b, Ptr{Nothing} @0x000000011feb8637, Ptr{Nothing} @0x0000000309f9c073, Ptr{Nothing} @0x0000000173710b57, Ptr{Nothing} @0x0000000304a5814f, Ptr{Nothing} @0x0000000304a58223, Ptr{Nothing} @0x0000000100efe31f, Ptr{Nothing} @0x0000000100efdd1f, Ptr{Nothing} @0x0000000100efc9d7, Ptr{Nothing} @0x0000000100efcccb, Ptr{Nothing} @0x0000000100efcff3, Base.InterpreterIP in top-level CodeInfo for Main at statement 6, Ptr{Nothing} @0x0000000100f1648b, Ptr{Nothing} @0x0000000100f1620b, Ptr{Nothing} @0x0000000100f1719f, Ptr{Nothing} @0x0000000131597eb7, Ptr{Nothing} @0x0000000131121f67, Ptr{Nothing} @0x000000011f00c04b, Ptr{Nothing} @0x000000011f00c087, Ptr{Nothing} @0x0000000100efe31f, Ptr{Nothing} @0x0000000100efdd1f, Ptr{Nothing} @0x0000000100efc9d7, Ptr{Nothing} @0x0000000100efcff3, Base.InterpreterIP in top-level CodeInfo for Main at statement 0, Ptr{Nothing} @0x0000000100f1648b, Ptr{Nothing} @0x0000000100f1620b, Ptr{Nothing} @0x0000000100f1719f, Ptr{Nothing} @0x000000013d38cb2b, Ptr{Nothing} @0x000000013d2235c3, Ptr{Nothing} @0x000000013d2236cb, Ptr{Nothing} @0x000000013d311b8f, Ptr{Nothing} @0x000000013d317037, Ptr{Nothing} @0x000000013d29ae2f, Ptr{Nothing} @0x000000013d22372b, Ptr{Nothing} @0x000000013d3e40d3, Ptr{Nothing} @0x000000013d28495b, Ptr{Nothing} @0x0000000100eef12f, Ptr{Nothing} @0x000000013146df2f, Ptr{Nothing} @0x000000013146efef, Ptr{Nothing} @0x0000000130d96803, Ptr{Nothing} @0x0000000100f47c6b, Ptr{Nothing} @0x0000000100f47b77]
AWSConfig((3nbu1tyiaQ2lN0A5DYmp, WEO..., 146138512-12-31T23:59:59), "eu-central-1", "json")
**[ Info:** after s3
```
Wdyt?
---
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 24, 2024, 8:34pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/6 "2024-11-24T20:34:58Z")
better stack trace:
```julia
Stacktrace:
[1] **readuntil(**buf::IOBuffer, find_delimiter::typeof(HTTP.Parsers.find_end_of_header)**)**
@ HTTP.IOExtras ~/.julia/packages/HTTP/aTjcj/src/IOExtras.jl:139
[2] **readuntil(**t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function, sizehint::Int64**)**
@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:296
[3] **readuntil(**t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function**)**
@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:292
[4] **readheaders(**io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, message::HTTP.Messages.Response**)**
@ HTTP.Messages ~/.julia/packages/HTTP/aTjcj/src/Messages.jl:537
[5] **startread(**http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}**)**
@ HTTP.Streams ~/.julia/packages/HTTP/aTjcj/src/Streams.jl:170
[6] **macro expansion**
@ ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:67 [inlined]
[7] **macro expansion**
@ ./task.jl:498 [inlined]
[8] **request(**::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Base.BufferStream, iofunction::Nothing, verbose::Int64, kw::@Kwargs{require_ssl_verification::Bool}**)**
@ HTTP.StreamRequest ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:57
[9] **request(**::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
@ HTTP.ConnectionRequest ~/.julia/packages/HTTP/aTjcj/src/ConnectionRequest.jl:108
[10] **request(**::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::URIs.URI, ::Vararg{Any}; kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
@ HTTP.ExceptionRequest ~/.julia/packages/HTTP/aTjcj/src/ExceptionRequest.jl:19
[11] **request(**::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
@ HTTP.MessageRequest ~/.julia/packages/HTTP/aTjcj/src/MessageRequest.jl:66
[12] **request(**::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
@ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/aTjcj/src/BasicAuthRequest.jl:28
[13] **request(**::Type{HTTP.TopRequest.TopLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, ::String, ::Vararg{Any}; kwargs::@Kwargs{require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
@ HTTP.TopRequest ~/.julia/packages/HTTP/aTjcj/src/TopRequest.jl:15
[14] **macro expansion**
@ ~/.julia/packages/Mocking/rzcpa/src/mock.jl:50 [inlined]
[15] **(::AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}})()**
@ AWS ~/.julia/packages/AWS/E9zQ4/src/utilities/request.jl:221
[16] **(::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(**; kwargs::@Kwargs{}**)**
@ Base ./error.jl:300
[17] **(::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#41#43", AWS.var"#40#42"{Request, DataType, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()**
```
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [November 24, 2024, 8:55pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/7 "2024-11-24T20:55:31Z")
> [@OgmaJ](#):
>
> ```julia
> **[Info:** MethodError(convert, (SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, UInt8[0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x33, 0x30, 0x31, 0x20, 0x4d, 0x6f, 0x76, 0x65, 0x64, 0x20, 0x50, 0x65, 0x72, 0x6d, 0x61, 0x6e, 0x65, 0x6e, 0x74, 0x6c, 0x79, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2d, 0x72, 0x65, 0x67, 0x69, 0x6f, 0x6e, 0x3a, 0x20, 0x61, 0x70, 0x2d, 0x73, 0x6f, 0x75, 0x74, 0x68, 0x2d, 0x31, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2d, 0x69, 0x64, 0x3a, 0x20, 0x51, 0x57, 0x48, 0x54, 0x33, 0x51, 0x45, 0x51, 0x50, 0x35, 0x33, 0x32, 0x57, 0x51, 0x58, 0x41, 0x0d, 0x0a, 0x78, 0x2d, 0x61, 0x6d, 0x7a, 0x2d, 0x69, 0x64, 0x2d, 0x32, 0x3a, 0x20, 0x44, 0x4c, 0x62, 0x45, 0x33, 0x44, 0x75, 0x58, 0x31, 0x50, 0x6c, 0x45, 0x59, 0x75, 0x74, 0x59, 0x43, 0x59, 0x68, 0x6a, 0x6c, 0x37, 0x45, 0x4f, 0x75, 0x61, 0x76, 0x62, 0x71, 0x33, 0x39, 0x77, 0x78, 0x35, 0x77, 0x4c, 0x34, 0x36, 0x4c, 0x52, 0x68, 0x54, 0x52, 0x37, 0x66, 0x2b, 0x7a, 0x4f, 0x39, 0x49, 0x2b, 0x66, 0x47, 0x59, 0x75, 0x38, 0x59, 0x59, 0x70, 0x52, 0x4b, 0x64, 0x65, 0x6e, 0x70, 0x32, 0x63, 0x63, 0x6f, 0x63, 0x4f, 0x51, 0x57, 0x73, 0x63, 0x3d, 0x0d, 0x0a, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x20, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x78, 0x6d, 0x6c, 0x0d, 0x0a, 0x54, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x65, 0x72, 0x2d, 0x45, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e, 0x67, 0x3a, 0x20, 0x63, 0x68, 0x75, 0x6e, 0x6b, 0x65, 0x64, 0x0d, 0x0a, 0x44, 0x61, 0x74, 0x65, 0x3a, 0x20, 0x53, 0x75, 0x6e, 0x2c, 0x20, 0x32, 0x34, 0x20, 0x4e, 0x6f, 0x76, 0x20, 0x32, 0x30, 0x32, 0x34, 0x20, 0x31, 0x39, 0x3a, 0x33, 0x39, 0x3a, 0x32, 0x38, 0x20, 0x47, 0x4d, 0x54, 0x0d, 0x0a, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x3a, 0x20, 0x41, 0x6d, 0x61, 0x7a, 0x6f, 0x6e, 0x53, 0x33, 0x0d, 0x0a, 0x0d, 0x0a]), 0x0000000000007c9f)
>
> ```
If I paste vector into the REPL as `v=UInt8[...]` and run `println(String(copy(v)))`, I see
```julia
HTTP/1.1 301 Moved Permanently
x-amz-bucket-region: ap-south-1
x-amz-request-id: QWHT3QEQP532WQXA
x-amz-id-2: DLbE3DuX1PlEYutYCYhjl7EOuavbq39wx5wL46LRhTR7f+zO9I+fGYu8YYpRKdenp2ccocOQWsc=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sun, 24 Nov 2024 19:39:28 GMT
Server: AmazonS3
```
Not sure if that is helpful or not.
> [@OgmaJ](#):
>
> ```julia
> [1] **readuntil(**buf::IOBuffer, find_delimiter::typeof(HTTP.Parsers.find_end_of_header)**)**
>
> @ HTTP.IOExtras ~/.julia/packages/HTTP/aTjcj/src/IOExtras.jl:139
>
> ```
I can’t find this line, `IOExtras.jl:139` in HTTP.jl’s source code unless I go back to pre-v1.0. Are you possibly on a very old version of HTTP.jl? Try updating to the latest version.
---
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 24, 2024, 9:45pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/8 "2024-11-24T21:45:57Z")
**pkg\>** status HTTP
**Status** `~/perso/docker/minio/Project.toml`
⌅ [cd3eb016] HTTP v0.9.17
**Info** Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`
I have now the following error:
**[ Info:** after aws\_config
```julia
Stack trace:
MethodError: Cannot `convert` an object of type
SubArray{UInt8,1,Memory{UInt8},Tuple{UnitRange{Int64}},true} to an object of type
SubArray{UInt8,1,Vector{UInt8},Tuple{UnitRange{Int64}},true}
The function `convert` exists, but no method is defined for this combination of argument types.
Closest candidates are:
SubArray{T, N, P, I, L}(::Any, ::Any, ::Any, ::Any) where {T, N, P, I, L}
@ Base subarray.jl:19
convert(::Type{T}, ::T) where T
@ Base Base.jl:126
convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray
@ LinearAlgebra /Applications/Julia-1.11.app/Contents/Resources/julia/share/julia/stdlib/v1.11/LinearAlgebra/src/factorization.jl:104
...
Stacktrace:
[1] **readuntil(**buf::IOBuffer, find_delimiter::typeof(HTTP.Parsers.find_end_of_header)**)**
@ HTTP.IOExtras ~/.julia/packages/HTTP/aTjcj/src/IOExtras.jl:139
[2] **readuntil(**t::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, f::Function, sizehint::Int64**)**
@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:296
[3] **readuntil(**t::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, f::Function**)**
@ HTTP.ConnectionPool ~/.julia/packages/HTTP/aTjcj/src/ConnectionPool.jl:292
[4] **readheaders(**io::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, message::HTTP.Messages.Response**)**
@ HTTP.Messages ~/.julia/packages/HTTP/aTjcj/src/Messages.jl:537
[5] **startread(**http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}**)**
@ HTTP.Streams ~/.julia/packages/HTTP/aTjcj/src/Streams.jl:170
[6] **macro expansion**
@ ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:67 [inlined]
[7] **macro expansion**
@ ./task.jl:498 [inlined]
[8] **request(**::Type{StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Base.BufferStream, iofunction::Nothing, verbose::Int64, kw::@Kwargs{require_ssl_verification::Bool}**)**
@ HTTP.StreamRequest ~/.julia/packages/HTTP/aTjcj/src/StreamRequest.jl:57
[9] **request(**::Type{ConnectionPoolLayer{StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::@Kwargs{iofunction::Nothing, require_ssl_verification::Bool, response_stream::Base.BufferStream}**)**
```
I think we made progress, but I’m still facing some strange issues.
My current setup involves Julia 1.11 running on a Mac with Apple Silicon (M2), and MinIO is hosted via Docker.
Wdyt, to troubleshoot further, using the same configuration on an Ubuntu system to see if the issue persists across platforms.
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [November 24, 2024, 10:27pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/9 "2024-11-24T22:27:34Z")
You need a newer version of the package HTTP.jl. Press ] to go into the package mode and try `add HTTP@1.10`
---
### Author:  [@OgmaJ](https://discourse.julialang.org/u/OgmaJ)
#### Post date: [November 26, 2024, 9:16pm UTC](https://discourse.julialang.org/t/help-needed-s3-connection-error-with-minio-in-development-environment/122957/10 "2024-11-26T21:16:02Z")
Thank you, @ericphanson, for providing such a great initial insight. Your comment about the missing values really caught my attention. At first, I was quite surprised, so I decided to bring in my teammate, @tencnivel, for a pair programming session to dive deeper into this.
We updated the HTTP package, but unfortunately, we’re still encountering the same issue with the error:
```julia
SubArray{UInt8,1,Memory{UInt8},Tuple{UnitRange{Int64}},true} to an object of type
SubArray{UInt8,1,Vector{UInt8},Tuple{UnitRange{Int64}},true}
```
Very strange indeed. My teammate (shoutout to @tencnivel) came across this post and read it and share it to me:
> [@Memory{T} in 1.11](https://discourse.julialang.org/t/memory-t-in-1-11/111811):
>
> Hi, While running my code with the 1.11 alpha 2 version of Julia, it broke due (I think) to a change introduced in Julia 1.11. In the following code: d = IOBuffer("hello") dd = d.data dd is a Vector{UInt8} in julia 1.10 but is a Memory{UInt8} in 1.11. Which broke my code because there is no insert! function for Memory{UInt8} 1/ the Julia documentation says it is “One-dimensional dense array with element of type T.” What is the difference with Vector{T} ? And in which case should I use Memo…
I downgraded my julia version (1.11 =\> 1.10) and tada everything works fine.
By the way, here’s my updated script that works well with MinIO and likely other standard S3 integrations (I don’t see any working example anywhere, maybe it will help).
```julia
import Pkg
# Activate the project environment located at the script's directory
Pkg.activate(@ __DIR__ )
Pkg.add("AWS")
Pkg.add("AWSS3")
Pkg.instantiate() # Ensures all dependencies are installed
using AWS
using AWSS3
struct MinioConfig <: AbstractAWSConfig
endpoint::String
region::String
require_ssl_verification::Bool
creds
end
AWS.region(c::MinioConfig) = c.region
AWS.credentials(c::MinioConfig) = c.creds
struct SimpleCredentials
access_key_id::String
secret_key::String
token::String
end
AWS.check_credentials(c::SimpleCredentials) = c
#ENV["AWS_SECRET_ACCESS_KEY"] = "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8"
#ENV["AWS_ACCESS_KEY_ID"] = "3nbu1tyiaQ2lN0A5DYmp"
function AWS.generate_service_url(aws::MinioConfig, service::String, resource::String)
service == "s3" || throw(ArgumentError("Can only handle s3 requests to Minio"))
return string(aws.endpoint, resource)
end
aws_config = AWS.global_aws_config(MinioConfig("http://127.0.0.1:9000", "",false, SimpleCredentials("3nbu1tyiaQ2lN0A5DYmp", "WEOaG7gFONUK185aXE5s0JZpTXV04chpjGXrfxq8", "")))
@info "after aws_config"
try
p = S3Path("s3://testapi/test1.txt") # provides an filesystem-like interface
#buckets = s3_list_buckets(;aws_config)
res = read(p, byte_range=1:4) # returns b"some"
@info "res:" String(res)
catch e
println("Stack trace:")
showerror(stderr, e, Base.catch_backtrace())
println(aws_config)
end
@info "after s3"
```
Btw, I noticed that everyone in this thread is helping others. Feel free to tag me if anyone encounters issues with S3 (and other AWS vendors) —I’d be happy to assist!
thx guys !!
cc @ericphanson , @tencnivel , @drizk1