Help translating curl command to HTTP.jl

New to Julia
,
1

I would like to scrape data using the Spotify API and I cannot seem to get the example curl command to translate to HTTP.jl

I am trying to get the accesstoken using my credentials. The relevant (working) curl command run through Julia is:

run(`curl -X "POST" -H "Authorization: Basic $refreshtoken" -d grant_type=client_credentials https://accounts.spotify.com/api/token -o creds.json`)

My attempt in HTTP.jl is:


HTTP.request("POST", 
    "https://accounts.spotify.com/api/token", 
    ["Authorization" => "Basic: $refreshtoken"],
    "grant_type=client_credentials")
# OR
 HTTP.request("POST", 
           "https://accounts.spotify.com/api/token", 
           ["Authorization" => "Basic $refreshtoken"],
           "grant_type=client_credentials", status_exception=false)

However I get an error:

ERROR: HTTP.ExceptionRequest.StatusError(400, "POST", "/api/token", HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
date: Fri, 22 Jan 2021 17:06:19 GMT
content-type: text/html; charset=utf-8
Content-Length: 1015
...
"""

EDIT:
The following also works in python

import requests
out = requests.post("https://accounts.spotify.com/api/token",
              {"grant_type":"client_credentials"},
              headers={"Authorization": f"Basic {refreshtoken}"})

Help is much appreciated!

2

Grasping at straws here, but your curl and python have no colon after Basic but the HTTP header value does.

1 Like
3

Thanks! Good guess but unfortunately same error after removing the colon. I’ll update the post

4

Shouldn’t the body of the request be a JSON object? What happens if you do:

 HTTP.request(
    "POST", 
    "https://accounts.spotify.com/api/token", 
    ["Authorization" => "Basic $refreshtoken"],
    "{'grant_type': 'client_credentials'}",
    status_exception=false
)
1 Like
5

Unfortunately I get the same error.

This is the example in the spotify docs:

curl -X "POST" -H "Authorization: Basic ZjM4ZjAw...WY0MzE=" -d grant_type=client_credentials https://accounts.spotify.com/api/token
6

What about:

 HTTP.request(
    "POST", 
    "https://accounts.spotify.com/api/token", 
    [
        "Authorization" => "Basic $refreshtoken",
        "Content_Type" => "application/x-www-form-urlencoded"
    ],
    "{'grant_type': 'client_credentials'}",
    status_exception=false
)
1 Like
7

Thank you for you effort!
Still the same error.

8

Once I have the token I can make request with HTTP.jl

e.g.

py"""
import requests
refreshtoken=$refreshtoken
out = requests.post("https://accounts.spotify.com/api/token",{"grant_type":"client_credentials"},headers={"Authorization": f"Basic {refreshtoken}"})
out = out.json()
"""
tkn = get(PyObject(py"out"), "access_token")
resp = HTTP.get("https://api.spotify.com/v1/playlists/1Sbiel38MM24OQi5Sc8cL2", 
    ["Content-Type"=>"application/json", 
    "Accept"=>"application/json", 
    "Authorization"=>"Bearer $tkn"]
    )

julia> JSON3.read(resp.body)
JSON3.Object{Array{UInt8,1},Array{UInt64,1}} with 15 entries:
  :collaborative => false
  :description   => "Una Playlist con los éxitos actuales del Pop en español!! <a href=\"https://open.spotify.com/artist/0vR2qb8m9WHeZ5ByCbimq2\">Reik</…
  :external_urls => {…
  :followers     => {…
  :href          => "https://api.spotify.com/v1/playlists/1Sbiel38MM24OQi5Sc8cL2"
  :id            => "1Sbiel38MM24OQi5Sc8cL2"
  :images        => JSON3.Object[{…
  :name          => "Mundo Pop En Español"
  â‹®              => â‹®
9

I signed up for an API key and when I do this:

HTTP.request(
    "POST", 
    "https://accounts.spotify.com/api/token", 
    [
        "Authorization" => "Basic $my_client_id:$my_client_secret",
        "Content-Type" => "application/x-www-form-urlencoded",
    ],
    "grant_type=client_credentials"
)

I get an “invalid client” error, which is different from the bad request error you’re getting. The Spotify docs say:

Authorization Required.
Base 64 encoded string that contains the client ID and client secret key. The field must have the format: Authorization: Basic <base64 encoded client_id:client_secret>

Are you following the client_id:client_secret format?

1 Like
10

Small sidenote. You can also authenticate by placing the credentials in the url like

"https://$usename:$password@accounts.spotify.com/api/token"

to save some logic on your side (source). Also, basic authorization always requires an username and password, as mentioned by @mthelm85.

2 Likes
11

I bas64 encode the client_id and client_secret:

refreshtoken = clientid * ':' * clientsecret
refreshtoken = base64encode(refreshtoken)

I can directly interpolate this into the run(curl ...) and python versions and it works

12

Sorry I know very little about authentication etc. I am trying the following but still getting a 400 error:

HTTP.request(
           "POST", 
           "https://$user:$pw@accounts.spotify.com/api/token",
           [],"grant_type=client_credentials",
           status_exception=false
       )

where user and pw are my spotify user name an password, respectively. Am I misunderstanding this?

EDIT: Also tried with clientid and clientsecret without any luck.

13

No need to apologise :slightly_smiling_face:

I meant user and password as an example. Clientid and clientsecret sounds like it should work. I guess, but can’t test since I’m not at a pc, that now the grant_type is passed in the HTTP body instead of a header.

By the way, I understand your struggles completely. HTTP.jl is also usually a struggle for me about what should be where exactly

1 Like
14

I’ve gotten it to work with another API (Chartmetric API Documentation) earlier so I was kind of surprised it didn’t work now. It seems quite similar. In case someone is interested in music data this is the library for the Chartmetric API (GitHub - danielw2904/ChartmetricScraper.jl). In the token.jl file I am doing a very similar request.

15

When I do:

curl -v  -X "POST" -H "Authorization: Basic $refreshtoken" -d grant_type=client_credentials https://accounts.spotify.com/api/token

What is shown being requested is:

> Host: accounts.spotify.com
> user-agent: curl/7.74.0
> accept: */*
> authorization: Basic refreshtoken
> content-length: 29
> content-type: application/x-www-form-urlencoded

A few posts ago you tried adding content-type but you for had converted the data to json, try adding content-type to the headers. You might also try adding “user-agent” and “accept” to the headers as well. So:

HTTP.request("POST", 
    "https://accounts.spotify.com/api/token", 
    [
        "authorization" => "Basic: $refreshtoken",
        "user-agent" => "curl/7.74.0",
        "accept" => "*/*",
        "content-type" => "application/x-www-form-urlencoded"
    ],
    "grant_type=client_credentials")
1 Like
16

You could also try changing the body to:

Vector{UInt8}("grant_type=client_credentials")

I’m not 100% sure if the passing a string vs passing a Vector results in the same data being sent to the server.

1 Like
17

Thnak you for your suggestions!
With this the response is

{"error":"invalid_client"}
18

Here’s a debugging suggestion: in a second terminal, run

nc -l 8080

and then replace https://accounts.spotify.com with http://localhost:8080. You can then compare exactly what’s the difference between the two requests. After that you don’t have to come up with guesses but you can just consult the HTTP.jl documentation to get what you need.

3 Likes
19

Thank you that solved it. I feel like I tried this before based on previous comments but anyways this is the command that works:

HTTP.post("https://accounts.spotify.com/api/token", 
    [
    "Authorization" => "Basic $refreshtoken", 
    "Accept" => "*/*", 
    "Content-Type" => "application/x-www-form-urlencoded"
    ],"grant_type=client_credentials")
20

Oh man that is my fault. Your suggestion works except

        "authorization" => "Basic: $refreshtoken",

should be

        "authorization" => "Basic $refreshtoken",

without the colon. I messed that up in the post
Thanks again for your help!

1 Like