GCP.jl - Google Cloud Platform APIs / BigQuery

Package Announcements Package announcements
#1

GCP.jl provides a Google Cloud Platform BigQuery API client and the ability to auto-generate new client APIs from Google’s discovery service. This is a 100% Julia implementation not reliant on command line installation of Google Cloud SDK or Python 2 (which Google Cloud SDK requires).

BigQuery API is generated, functional and tested for getting data in and out of the BigQuery service. Compare to GBP, GoogleCloud and GoogleCloudObjectStores.

GCP.jl is available from the official Julia package registry.

Example usage

using GCP
using GCP.Bigquery

CredentialFilename = expanduser("~/secrets/your-project.json")

p = GCP.Project(CredentialFilename, ["cloud-platform"])
dataset = Bigquery.Dataset()
dataset.datasetReference = Bigquery.DatasetReference()
dataset.datasetReference.datasetId = "your-dataset"
res = p.bigquery.datasets.insert(dataset)

@info res

Why? This is an itch I needed to scratch. For anyone who needs BigQuery access, or needs to generate further Google SDKs, I hope this will help you. At the time of writing 251 more APIs might be generated.

Cheers

24 Likes
#2

@Rana_Ian This is great, thank you for pulling this together. I might be missing it in my quick glance at the test cases, but I’m curious if you have an example you could point me to of pulling down a query result to a dataframe locally using GCP.jl?

I have a solution that I’m using (I’ve written an alternative to the way the GBQ module handles this that is a touch more vectorized and checks data types), but I would like to move away from the reliance on the command line installation.

I’m happy to open PRs to add any of the above items if they’re not in there and it is useful.

2 Likes
#3

I am able to connect but how do we query tables?

CredentialFilename = expanduser("~/xxx/creds.json")
p = GCP.Project(CredentialFilename, ["cloud-platform"])
dataset = Bigquery.Dataset()
dataset.datasetReference = Bigquery.DatasetReference()

I checked the examples and I didn’t see any example of selecting specific tables - there is the “get” command, but this didn’t seem like it worked as a querying tool…

Is there anything that is similar to Python’s google.cloud where one can just use a SELECT statement on the tables??

#4

I had a similar question last year; let me know if you come across anything that works for you @Billpete002 - I’d be open to collaborating on a project to have something closer to R’s bigquery, since R, like Julia, doesn’t have the dedicated API like Python for BQ.