FigShare's API

yakir12 · May 31, 2022, 1:02pm

Depending on how easy it is, I would like to programmatically upload data to FigShare using their API. They have a Jupyter notebook showing how to do so here. It looks like they stream the data in chunks.

Has anyone tried this or similar before…?

quinnj · May 31, 2022, 1:59pm

Looks like it would be a pretty straightforward translation to Julia using HTTP.jl + Base file methods. Nothing too complicated or fancy.

yakir12 · May 31, 2022, 2:11pm

I’ll try to play around with it, see what I get…

The ultimate goal is to generate an Artifacts.toml file containing all the artifacts from the uploaded zip files of locally stored raw-data.

aplavin · May 31, 2022, 9:11pm

Not figshare, but openscienceframework (osf.io): see my Alexander Plavin / OpenScienceFramework.jl · GitLab package. It provides filesystem-like highlevel API (readdir, write, read, …) to remote files, and an automated way to create & upload Julia artifacts. The latter isn’t really documented, but see an example in tests: test/runtests.jl · master · Alexander Plavin / OpenScienceFramework.jl · GitLab.

yakir12 · June 1, 2022, 9:14am

Wow, OSF looks very cool. But unless I misunderstood, they don’t offer hosting the data themselves, instead, they connect between all the different (current) services involved in collaborative research and publication.

One of the points behind services like FigShare (and others) is to promise that the versioned data will be there “forever”. So a link to the data will “always” work and point to that specific (version of the) data. This way a committed version of the code (what with a Manifest file etc) will always successfully work with the linked raw-data.

OpenScienceFramework.jl looks very comprehensive. Thanks for sharing. I think my aim is a lot more modest.

aplavin · June 1, 2022, 12:22pm

I think there is indeed this possibility to host data somewhere else, but I never used it myself. Actually, I only use OSF as a place to host versioned datasets. A nice feature is that it can provide direct download links for private files as well (a code is included in the URL).

That’s exactly my usecase for OSF. Files can be deleted, of course, but old versions cannot be replaced with different content.
Add Julia artifacts system, and you get reproducibility “forever”.

oxinabox · June 1, 2022, 12:55pm

I think we hit FigShare’s API in DataDepsGenerators.jl

github.com

oxinabox/DataDepsGenerators.jl/blob/master/src/APIs/Figshare.jl

struct Figshare <: DataRepo
end

base_url(::Figshare) = "https://api.figshare.com/v2/articles"
# Note: Unlike other generators, FigShare base_url is sometimes a "directory" and sometimes a file
# But it is OK, because Figshare also overloads all the functions that use `base_url`
# Most importantly `mainpage_url`

description(repo::Figshare, mainpage) = mainpage["description"]

author(::Figshare, mainpage) = [value["full_name"] for value in mainpage["authors"]]

license(::Figshare, mainpage) = mainpage["license"]["name"] * " (" *mainpage["license"]["url"] * ")"

published_date(::Figshare, mainpage) = mainpage["published_date"]

dataset_cite(::Figshare, mainpage) = mainpage["citation"]

function get_urls(repo::Figshare, page)
    urls = []

This file has been truncated. show original

When querying for URL to download from and other metadata.

github.com

oxinabox/DataDepsGenerators.jl/blob/master/src/APIs/JSONLD/JSONLD.jl

abstract type JSONLD <: DataRepo
end

include("JSONLD_Web.jl")
include("JSONLD_DOI.jl")

data_fullname(::JSONLD, mainpage) = mainpage["name"]
website(::JSONLD, mainpage_url, mainpage) = mainpage_url
description(repo::JSONLD, mainpage) = filter_html(getfirst(mainpage, "description"))


function author(repo::JSONLD, mainpage)
    inner(authors::Vector) = map(inner, authors)
    inner(::Missing) = missing
    inner(author::Dict) = getfirst(author, "name")

    authors = getfirst(mainpage, "author", "creator") |> inner
    if !(authors isa AbstractVector) # then there was just one
        return [authors]
    end

This file has been truncated. show original

If i recall correctly figshare embeds JSON-LD on their pages.
Which we also hit in DataDepsGenerators.jl

github.com

oxinabox/DataDepsGenerators.jl/blob/master/src/APIs/JSONLD/JSONLD.jl

abstract type JSONLD <: DataRepo
end

include("JSONLD_Web.jl")
include("JSONLD_DOI.jl")

data_fullname(::JSONLD, mainpage) = mainpage["name"]
website(::JSONLD, mainpage_url, mainpage) = mainpage_url
description(repo::JSONLD, mainpage) = filter_html(getfirst(mainpage, "description"))


function author(repo::JSONLD, mainpage)
    inner(authors::Vector) = map(inner, authors)
    inner(::Missing) = missing
    inner(author::Dict) = getfirst(author, "name")

    authors = getfirst(mainpage, "author", "creator") |> inner
    if !(authors isa AbstractVector) # then there was just one
        return [authors]
    end

This file has been truncated. show original

Depending on your use case you could potentially use DataDepsGenerators.jl + DataDeps.jl for the download.
Else feel free to browse that could and steal things.