AWS.jl - global_aws_config can't find ARN

General Usage
1

I have an issue with global_aws_config. It works perfectly fine in SageMaker Notebook:

aws = global_aws_config(; region="us-west-2")

AWSConfig(arn:aws:iam::123456789:instance-profile/BaseNotebookInstanceEc2InstanceRole 
(STUFFSTUFFSTUFF, +ST..., STU..., 2021-10-13T08:03:03), "us-west-2", "json")

but when running the same line of code in SageMaker Studio:

aws = global_aws_config(; region="us-west-2")

KeyError: key "RoleArn" not found

Stacktrace:
 [1] getindex(h::Dict{String, Any}, key::String)
   @ Base ./dict.jl:482
 [2] ecs_instance_credentials()
   @ AWS /opt/julia/packages/AWS/RCBqC/src/AWSCredentials.jl:338
 [3] AWSCredentials(; profile::Nothing, throw_cred_error::Bool)
   @ AWS /opt/julia/packages/AWS/RCBqC/src/AWSCredentials.jl:126
 [4] #global_aws_config#53
   @ /opt/julia/packages/AWS/RCBqC/src/AWS.jl:78 [inlined]
 [5] top-level scope
   @ In[14]:2
 [6] eval
   @ ./boot.jl:360 [inlined]
 [7] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1116

Has anyone else had issue with AWS.jl package and using the aws_config / global_aws_config? I read in the AWS.jl documentation that you can create a text file for it to check but it doesn’t provide an example or much information on how the file is supposed to be structured or where it needs to be placed.

I have tried specifying the ARN within the aws_config:

aws = global_aws_config(creds = AWSCredentials(user_arn = "arn:aws:iam::123456789:role/service-
role/stuffstuffstuff"), region = "us-west-2")

but this just spits out an error:

MethodError: no method matching AWSCredentials(; user_arn="arn:aws:iam::123456789:role/service-
role/stuffstuffstuff")
Closest candidates are:
  AWSCredentials(; profile, throw_cred_error) at 
/opt/julia/packages/AWS/RCBqC/src/AWSCredentials.jl:107 got unsupported keyword argument "user_arn"
  AWSCredentials(::Any, ::Any) at /opt/julia/packages/AWS/RCBqC/src/AWSCredentials.jl:71 got 
unsupported keyword argument "user_arn"
  AWSCredentials(::Any, ::Any, ::Any) at /opt/julia/packages/AWS/RCBqC/src/AWSCredentials.jl:71 got 
unsupported keyword argument "user_arn"
  ...

Stacktrace:
 [1] kwerr(kw::NamedTuple{(:user_arn,), Tuple{String}}, args::Type)
   @ Base ./error.jl:157
 [2] top-level scope
   @ In[16]:1
 [3] eval
   @ ./boot.jl:360 [inlined]
 [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1116

I don’t know why user_arn is not supported because that is the text used when I check
?AWSCredentials

it shows:

AWSCredentials
When you interact with AWS, you specify your AWS Security Credentials to verify who you are and 
whether you have permission to access the resources that you are requesting. AWS uses the security 
credentials to authenticate and authorize your requests. The fields access_key_id and secret_key hold 
the access keys used to authenticate API requests (see Creating, Modifying, and Viewing Access 
Keys). Temporary Security Credentials require the extra session token field. The user_arn and 
account_number fields are used to cache the result of the aws_user_arn and aws_account_number 
functions.

AWS.jl searches for credentials in a series of possible locations and stops as soon as it finds 
credentials. The order of precedence for this search is as follows:

Passing credentials directly to the AWSCredentials constructor
Environment variables
Shared credential file (~/.aws/credentials)
AWS config file (~/.aws/config)
Assume Role provider via the aws config file
Instance metadata service on an Amazon EC2 instance that has an IAM role configured
Once the credentials are found, the method by which they were accessed is stored in the renew field 
and the DateTime at which they will expire is stored in the expiry field. This allows the credentials to be 
refreshed as needed using check_credentials. If renew is set to nothing, no attempt will be made to 
refresh the credentials. Any renewal function is expected to return nothing on failure or a populated 
AWSCredentials object on success. The renew field of the returned AWSCredentials will be discarded 
and does not need to be set.

To specify the profile to use from ~/.aws/credentials, do, for example, AWSCredentials(profile="profile-
name").

Any help or direction will be appreciated!

2

Got a response from AWS support:

I was able to reproduce the issue in SageMaker Studio. Further, checking the Stacktrace , I can see the error is from the following code

function ecs_instance_credentials()
    if !haskey(ENV, "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
        return nothing
    end

    uri = ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]

    response = @mock HTTP.request("GET", "http://169.254.170.2$uri ")
    new_creds = String(response.body)
    new_creds = JSON.parse(new_creds)

    expiry = DateTime(rstrip(new_creds["Expiration"], 'Z'))

    return AWSCredentials(
        new_creds["AccessKeyId"],
        new_creds["SecretAccessKey"],
        new_creds["Token"],
        new_creds["RoleArn"];
        expiry=expiry,
        renew=ecs_instance_credentials,
    )
end

as in Github repo

AWS.jl/src/AWSCredentials.jl

Line 325 in d387cc5

function ecs_instance_credentials()

The issue is with ecs_instance_credentials . The response of ECS meta data server doesn’t include a field called RoleArn . You could check this by running

curl http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | python -m json.tool

in the KernelGateway App terminal. The response will be something like

{
    "AccessKeyId": "ABCD1234
    "SecretAccessKey": "abcdABCD1234",
    "Token": "abcdABCD1234/////||||||",
    "Expiration": "2021-10-15T02:13:17Z",
    "Code": "Success",
    "Message": ""
}

This is the JSON file that will be further processed. That’s the reason why we see KeyError: key "RoleArn" not found .

The issue is with how AWS.jl is implemented. In official AWS SDK/CLI, credentials in SageMaker Studio/Notebook are processed in a special way since they are not standard EC2/ECS environment. They are managed services built upon ECS/EC2 and may modify meta data server. The response is from _sagemaker-instance-credentials which is a modified API.

AWS.jl is not owned by AWS and we have no control over their development. You may want to raise this issue with JuliaCloud so that they could fix this issue.

Unfortunately, for now, Julia is not officially supported by AWS SDK so I guess you still need to use this community SDK. The recommended way is to work with developers of AWS.jl to fix this issue, which is a long term solution. You can fix this issue with a folk repo as a temporary solution. You may clone the original repo and fix the issue by yourself. Then, you should configure your custom image to install from your own folk with the fix. According to my experience, this issue may be mitigated by the following steps:
(1) calling get_caller_identify with retrieved credentials from ECS meta data server, i.e. with AccessKeyId , SecretAccessKey and Token
(2) with the retrieved role name, call get_role to obtain the full role path so that a complete role ARN could be constructed
(3) pass this role ARN to AWSCredentials
An example implementation in Python could be found via https://github.com/aws/sagemaker-python-sdk/blob/c95c75a96aea50928c58832eb9698c478e504bc4/src/sagemaker/session.py#L3536