Using AWSS3.jl getting the following error

General Usage
1

Hi all :slight_smile:

I’m debugging this error now for a while. I think it’s related with a different post I made a while ago.
So I’m using a couple of threads to do stuff async. Once in while the main/ starting thread just stops. But two or three already started threads run fine. This time all the above happened and the whole script crashed with the following error.

nested task error: AWS.AWSExceptions.AWSException: 400 -- AWSException

    HTTP.ExceptionRequest.StatusError(400, "PUT", "/company-snowflake/TT/Prod/ProjectName/FILLED_DATA.bytes", HTTP.Messages.Response:
    """
    HTTP/1.1 400 Bad Request
    x-amz-request-id: <random string>
    x-amz-id-2: <random string>
    Content-Type: application/xml
    Transfer-Encoding: chunked
    Date: Thu, 07 Apr 2022 17:08:28 GMT
    Server: AmazonS3
    Connection: close

    [Message Body was streamed]""")

    (empty body)

    Stacktrace:
      [1] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :require_ssl_verification, :response_stream), Tuple{Nothing, Bool, Base.BufferStream}}})
        @ HTTP.ExceptionRequest C:\.julia\packages\HTTP\qszg7\src\ExceptionRequest.jl:22
      [2] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:require_ssl_verification, :response_stream), Tuple{Bool, Base.BufferStream}}})
        @ HTTP.MessageRequest C:\.julia\packages\HTTP\qszg7\src\MessageRequest.jl:66
      [3] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}; kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:require_ssl_verification, :response_stream), Tuple{Bool, Base.BufferStream}}})
        @ HTTP.BasicAuthRequest C:\.julia\packages\HTTP\qszg7\src\BasicAuthRequest.jl:28
      [4] #request#1
        @ C:\.julia\packages\HTTP\qszg7\src\TopRequest.jl:15 [inlined]
      [5] macro expansion
        @ C:\.julia\packages\Mocking\MsKoy\src\mock.jl:29 [inlined]
      [6] macro expansion
        @ C:\.julia\packages\AWS\1kisM\src\utilities\request.jl:161 [inlined]
      [7] macro expansion
        @ C:\.julia\packages\Retry\vS1bg\src\repeat_try.jl:192 [inlined]
      [8] _http_request(http_backend::AWS.HTTPBackend, request::AWS.Request, response_stream::IOBuffer)
        @ AWS C:\.julia\packages\AWS\1kisM\src\utilities\request.jl:147
      [9] macro expansion
        @ C:\.julia\packages\Mocking\MsKoy\src\mock.jl:29 [inlined]
     [10] macro expansion
        @ C:\.julia\packages\AWS\1kisM\src\utilities\request.jl:88 [inlined]
     [11] macro expansion
        @ C:\.julia\packages\Retry\vS1bg\src\repeat_try.jl:192 [inlined]
     [12] submit_request(aws::AWS.AWSConfig, request::AWS.Request; return_headers::Nothing)
        @ AWS C:\.julia\packages\AWS\1kisM\src\utilities\request.jl:85
     [13] (::AWS.RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Any}; aws_config::AWS.AWSConfig, feature_set::AWS.FeatureSet)
        @ AWS C:\.julia\packages\AWS\1kisM\src\AWS.jl:284
     [14] #put_object#170

Thank you for you help

2

As far as I can find (and I’ve spent some time on it) release versions of Julia do not have access to a working multithreaded HTTP client. With AWS.jl, you can use HTTP.jl or Downloads.jl but both have issues; see Backends · AWS.jl for the documentation on that.

Julia 1.8 and Julia 1.7.3 (unreleased) have multithreading fixes for Downloads.jl, which could fix this if you use one of those versions and switch to the Downloads backend for AWS.jl.

1 Like