FigShare's API

Depending on how easy it is, I would like to programmatically upload data to FigShare using their API. They have a Jupyter notebook showing how to do so here. It looks like they stream the data in chunks.

Has anyone tried this or similar before…?

Looks like it would be a pretty straightforward translation to Julia using HTTP.jl + Base file methods. Nothing too complicated or fancy.

1 Like

I’ll try to play around with it, see what I get… :grimacing:

The ultimate goal is to generate an Artifacts.toml file containing all the artifacts from the uploaded zip files of locally stored raw-data.

Not figshare, but openscienceframework ( see my Alexander Plavin / OpenScienceFramework.jl · GitLab package. It provides filesystem-like highlevel API (readdir, write, read, …) to remote files, and an automated way to create & upload Julia artifacts. The latter isn’t really documented, but see an example in tests: test/runtests.jl · master · Alexander Plavin / OpenScienceFramework.jl · GitLab.

1 Like

Wow, OSF looks very cool. But unless I misunderstood, they don’t offer hosting the data themselves, instead, they connect between all the different (current) services involved in collaborative research and publication.

One of the points behind services like FigShare (and others) is to promise that the versioned data will be there “forever”. So a link to the data will “always” work and point to that specific (version of the) data. This way a committed version of the code (what with a Manifest file etc) will always successfully work with the linked raw-data.

OpenScienceFramework.jl looks very comprehensive. Thanks for sharing. I think my aim is a lot more modest.

I think there is indeed this possibility to host data somewhere else, but I never used it myself. Actually, I only use OSF as a place to host versioned datasets. A nice feature is that it can provide direct download links for private files as well (a code is included in the URL).

That’s exactly my usecase for OSF. Files can be deleted, of course, but old versions cannot be replaced with different content.
Add Julia artifacts system, and you get reproducibility “forever”.

I think we hit FigShare’s API in DataDepsGenerators.jl

When querying for URL to download from and other metadata.

If i recall correctly figshare embeds JSON-LD on their pages.
Which we also hit in DataDepsGenerators.jl

Depending on your use case you could potentially use DataDepsGenerators.jl + DataDeps.jl for the download.
Else feel free to browse that could and steal things.

1 Like