I have a loop that writes bytes to S3. It works about 95% of the time. Then periodically, I get a cryptic error from the depths of HTTP.jl about SSL_ERROR_SYSCALL.
The function looks like this
using Retry: @repeat
using AWSS3
using AWS.AWSExceptions: AWSException
function write_and_log(path::S3Path, bytes::Vector{UInt8})
@repeat 4 try
@info "START: Running put to s3" path length(bytes)
write(path, bytes)
@info "END: successfully put file to s3" path
catch e
@retry if e isa HTTP.Exceptions.HTTPError || e isa AWSException
bt = catch_backtrace()
@error "END: Failed to put file to s3. Will try again." path exception=(e, bt)
end
@error "END: Failed to write file to s3" path=path exception=(e, catch_backtrace())
end
end
The error thrown looks like this:
┌ Error: END: Failed to put file to s3. Will try again.
│ job.path = p"s3://ppad-processing-processed-datanode-output-prod/rollout2024q2-150/consolidated_rollout2024q2-150_innetwork/data_version_p=1.1.0/versioneddatasourceid_p=150/sourcefileid_p=1446234/batch_2_chunk_0.parq"
│ exception =
│ HTTP.RequestError:
│ HTTP.Request:
│ HTTP.Messages.Request:
│ """
│ PUT /ppad-processing-processed-datanode-output-prod/rollout2024q2-150/consolidated_rollout2024q2-150_innetwork/data_version_p%3D1.1.0/versioneddatasourceid_p%3D150/sourcefileid_p%3D1446234/batch_2_chunk_0.parq HTTP/1.1
│ Content-Type: application/octet-stream
│ User-Agent: AWS.jl/1.0.0
│ Host: s3.us-east-1.amazonaws.com
│ x-amz-date: 20240619T154349Z
│ x-amz-content-sha256: a167eba412e99ca1b5a51fab79f014a783fab46274cde03b5fe2b94a65434381
│ Content-MD5: Hf6y6YV/Vt4O6NhptirfpQ==
│ x-amz-security-token: # [redacted] because I don't know if this should be shared :)
│ Authorization: AWS4-HMAC-SHA256 Credential=[redacted]/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=b94a0794fdbe81efa864a85afb59b4c22697ae92f192294d5a1ef80750be9b78
│ Accept: */*
│ Content-Length: 302537168
│ Accept-Encoding: gzip
│
│
│ ⋮
│ 302537168-byte body
│ """Underlying error:
│ IOError: SSL_ERROR_SYSCALL
│ Stacktrace:
│ [1] (::HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Type, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
│ @ HTTP.ConnectionRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\ConnectionRequest.jl:143
│ [2] (::HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
│ @ HTTP.RetryRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\RetryRequest.jl:35
│ [3] manageretries
│ @ C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\RetryRequest.jl:30 [inlined]│ [4] (::HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64, retry::Bool})
│ @ HTTP.CookieRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\CookieRequest.jl:42
│ [5] managecookies
│ @ C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\CookieRequest.jl:19 [inlined]
│ [6] (::HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64, retry::Bool})
│ @ HTTP.HeadersRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\HeadersRequest.jl:71
│ [7] defaultheaders
│ @ C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\HeadersRequest.jl:14 [inlined]
│ [8] (::HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Base.BufferStream, kw::@Kwargs{verbose::Int64, retry::Bool})
│ @ HTTP.RedirectRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\RedirectRequest.jl:17
│ [9] redirects
│ @ C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\RedirectRequest.jl:14 [inlined]
│ [10] (::HTTP.MessageRequest.var"#makerequest#3"{HTTP.MessageRequest.var"#makerequest#1#4"{HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}; copyheaders::Bool, response_stream::Base.BufferStream, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{redirect::Bool, retry::Bool})
│ @ HTTP.MessageRequest C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\MessageRequest.jl:35
│ [11] makerequest
│ @ C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\clientlayers\MessageRequest.jl:24 [inlined]
│ [12] request(stack::HTTP.MessageRequest.var"#makerequest#3"{HTTP.MessageRequest.var"#makerequest#1#4"{HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}, method::String, url::URIs.URI, h::Vector{Pair{SubString{String}, SubString{String}}}, b::Vector{UInt8}, q::Nothing; headers::Vector{Pair{SubString{String}, SubString{String}}}, body::Vector{UInt8}, query::Nothing, kw::@Kwargs{redirect::Bool, retry::Bool, response_stream::Base.BufferStream})
│ @ HTTP C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\HTTP.jl:457
│ [13] #request#20
│ @ HTTP C:\Users\mrufsvold\.julia\packages\HTTP\Y2JKB\src\HTTP.jl:315 [inlined]
│ [14] macro expansion
│ @ C:\Users\mrufsvold\.julia\packages\Mocking\Q17aB\src\mock.jl:29 [inlined]
│ [15] (::AWS.var"#48#50"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol},
Vector{Any}}})()
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\utilities\request.jl:225
│ [16] (::Base.var"#96#98"{Base.var"#96#97#99"{AWS.AWSExponentialBackoff, AWS.var"#49#51", AWS.var"#48#50"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(; kwargs::@Kwargs{})
│ @ Base .\error.jl:308
│ [17] (::Base.var"#96#98"{Base.var"#96#97#99"{AWS.AWSExponentialBackoff, AWS.var"#49#51", AWS.var"#48#50"{AWS.Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()
│ @ Base .\error.jl:291
│ [18] _http_request(http_backend::AWS.HTTPBackend, request::AWS.Request, response_stream::IOBuffer)
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\utilities\request.jl:250
│ [19] macro expansion
│ @ C:\Users\mrufsvold\.julia\packages\Mocking\Q17aB\src\mock.jl:29 [inlined]
│ [20] (::AWS.var"#41#44"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}})()
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\utilities\request.jl:134
│ [21] (::AWS.var"#42#46"{AWS.var"#41#44"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer})()
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\utilities\request.jl:149
│ [22] (::Base.var"#96#98"{Base.var"#96#97#99"{AWS.AWSExponentialBackoff, AWS.var"#43#47"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#42#46"{AWS.var"#41#44"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})(; kwargs::@Kwargs{})
│ @ Base .\error.jl:296
│ [23] (::Base.var"#96#98"{Base.var"#96#97#99"{AWS.AWSExponentialBackoff, AWS.var"#43#47"{AWS.AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#42#46"{AWS.var"#41#44"{AWS.AWSConfig, AWS.Request, IOBuffer, Vector{Int64}}, IOBuffer}}})()
│ @ Base .\error.jl:291
│ [24] submit_request(aws::AWS.AWSConfig, request::AWS.Request; return_headers::Nothing)
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\utilities\request.jl:200
│ [25] (::AWS.RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Any}; aws_config::AWS.AWSConfig, feature_set::AWS.FeatureSet)
│ @ AWS C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\AWS.jl:287
│ [26] RestXMLService
│ @ C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\AWS.jl:251 [inlined]
│ [27] #put_object#172
│ @ C:\Users\mrufsvold\.julia\packages\AWS\SchLh\src\services\s3.jl:5754 [inlined]
│ [28] s3_put(aws::AWS.AWSConfig, bucket::SubString{String}, path::String, data::Vector{UInt8}, data_type::String, encoding::String; acl::String, metadata::Dict{String, String}, tags::Dict{String, String}, parse_response::Bool, kwargs::@Kwargs{})
│ @ AWSS3 C:\Users\mrufsvold\.julia\packages\AWSS3\8cxdr\src\AWSS3.jl:1037
│ [29] s3_put
│ @ AWSS3 C:\Users\mrufsvold\.julia\packages\AWSS3\8cxdr\src\AWSS3.jl:985 [inlined]
│ [30] write(fp::S3Path{Nothing}, content::Vector{UInt8}; part_size_mb::Int64, multipart::Bool, returns::Symbol, other_kwargs::@Kwargs{})
│ @ AWSS3 C:\Users\mrufsvold\.julia\packages\AWSS3\8cxdr\src\s3path.jl:696
│ [31] write
│ @ C:\Users\mrufsvold\.julia\packages\AWSS3\8cxdr\src\s3path.jl:674 [inlined]
│ [32] macro expansion
│ @ c:\Users\mrufsvold\Projects\DIL-price-transparency-psd\TableConsolidator.jl\src\jobs\GetSendJob.jl:17 [inlined]
│ [33] macro expansion
│ @ C:\Users\mrufsvold\.julia\packages\Retry\vS1bg\src\repeat_try.jl:192 [inlined]
│ [34] write_and_log(job::Main.TableConsolidator.SendJob)
│ @ Main.TableConsolidator c:\Users\mrufsvold\Projects\DIL-price-transparency-psd\TableConsolidator.jl\src\jobs\GetSendJob.jl:15
│ [35] (::Main.TableConsolidator.var"#60#61"{Channel{Main.TableConsolidator.SendJob}})()
│ @ Main.TableConsolidator c:\Users\mrufsvold\Projects\DIL-price-transparency-psd\TableConsolidator.jl\src\TableConsolidator.jl:71
└ @ Main.TableConsolidator c:\Users\mrufsvold\Projects\DIL-price-transparency-psd\TableConsolidator.jl\src\jobs\GetSendJob.jl:22
Some version info:
julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39 (2023-12-25 18:01 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, tigerlake)
Threads: 1 on 8 virtual cores
(TableConsolidator) pkg> status
Project TableConsolidator v0.1.0
Status `C:\Users\mrufsvold\Projects\DIL-price-transparency-psd\TableConsolidator.jl\Project.toml`
⌃ [fbe9abb3] AWS v1.90.3
[1c724243] AWSS3 v0.11.2
[336ed68f] CSV v0.10.14
[0f8b85d8] JSON3 v1.14.0
⌃ [e6f89c97] LoggingExtras v1.0.2
[98105f81] LoggingFormats v1.5.0
⌃ [98572fba] Parquet2 v0.2.19
⌃ [2dfb63ee] PooledArrays v1.4.2
[20febd7b] Retry v0.4.1
⌃ [bd369af6] Tables v1.10.1
⌃ [28d57a85] Transducers v0.4.78
⌃ [9d95f2ec] TypedTables v1.4.3
[56ddb016] Logging
[9a3f8284] Random
Info Packages marked with ⌃ have new versions available and may be upgradable.
The files are not huge, around 300MB. And the retry loop usually succeeds the second time. I notice that, when it is going to fail, it hangs for a long time before finally throwing this error.
Edit: I can’t seem to reproduce this error in isolation. This write function is getting called in a program that has a number of async read operations. So maybe there is an issue with SSL being used from multiple threads authenticating the reads at the same time as my write loop?
Edit2: openssl - SSL_read failing with SSL_ERROR_SYSCALL error - Stack Overflow indicates that S3 might be closing the connection before getting to the actual EOF. So maybe my write thread is getting interrupted, S3 drops the connection, and then when it tries to write again, we hit this error. If that’s the case, I’m not sure how to convince Julia that a task should not be interrupted.
Edit3: I’m more and more convinced that this is happening because of task switching. Consistently, I hit this error when the start of a write step unblocks a new batch of reads. I think what might be happening is that I take! a send job, it starts, but that unblocks upstream put!s for read tasks. So then the writer is interrupted to consume the batch of files. And then S3 closes the connection. The retry succeeds immediately because upstream tasks are blocked again.
I see that there is no API to tell a task not to yield. So is this a bug in AWSS3 that it doesn’t more gracefully handle retries of multipart uploads?