# Dataset access - Python

**URL:** <https://discourse.julialang.org/t/dataset-access-python/30048>\
**Category:** Data\
**Created:** [October 18, 2019, 9:23am UTC](https://discourse.julialang.org/t/dataset-access-python/30048 "2019-10-18T09:23:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 18, 2019, 9:23am UTC](https://discourse.julialang.org/t/dataset-access-python/30048/1 "2019-10-18T09:23:41Z")

</div>

@oxinabox will set me right here. I recall looking at [https://github.com/oxinabox/DataDepsGenerators.jl](https://github.com/oxinabox/DataDepsGenerators.jl)

There is a Python standard for accessing datasets which was mentioned at the time.  
Or am I imagining this? I ask since I Am working on a Python shop.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 18, 2019, 2:35pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/2 "2019-10-18T14:35:24Z")

</div>

For standard you are probably linking the JSON-LD / [Schema.org](http://Schema.org) thing.  
[Dataset - Schema.org Type](https://schema.org/Dataset)  
[Google AI Blog: Facilitating the discovery of public datasets](https://ai.googleblog.com/2017/01/facilitating-discovery-of-public.html)

This is how DataDepsGenerators uses it:  
[https://github.com/oxinabox/DataDepsGenerators.jl/tree/master/src/APIs/JSONLD](https://github.com/oxinabox/DataDepsGenerators.jl/tree/master/src/APIs/JSONLD)

For python tools:  
Look at Quilt and frictionlessdata (actually for most languages including julia)  
IIRC both are commercial concerns.  
Niether can consume JSON-LD directly AFAICK  
but frictionlessdata has their own competing standard  
They seem like good folk.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 18, 2019, 3:23pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/3 "2019-10-18T15:23:59Z")

</div>

Thankyou.

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [October 18, 2019, 4:41pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/4 "2019-10-18T16:41:17Z")

</div>

I was just reviewing this package last night, looks great.  
Quick question → Having trouble using the `get_dataurls_from_webserver_index` function in `misc_extractors.jl`  
[https://github.com/oxinabox/DataDepsGenerators.jl/blob/master/src/misc\_extractors.jl](https://github.com/oxinabox/DataDepsGenerators.jl/blob/master/src/misc_extractors.jl)  
Was trying to use this on a public directory such as this:  
[https://download.bls.gov/pub/time.series/sm/](https://download.bls.gov/pub/time.series/sm/)

What is proper syntax to use that function? Thanks!!

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [October 18, 2019, 8:18pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/5 "2019-10-18T20:18:50Z")

</div>

Apologies, scratch that, had tried to use `import` instead of `using`.

```julia
using DataDepsGenerators: get_dataurls_from_webserver_index

```

…and it works!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 18, 2019, 8:24pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/6 "2019-10-18T20:24:32Z")

</div>

I suspect that one would benifit from a bit of generalizing:

```julia
function get_dataurls_from_webserver_index(datapage_url)
    datapage = getpage(datapage_url)

    data_hrefs = (attr(ele, "href") 
        for ele in eachmatch(sel"a", datapage.root)) 
        if !match(r"^(To )?Parent( Directory)?$"i, text_only(ele)) && !match(r"(To )?Index"i, text_only(ele)
    ) 
     
    data_urls = joinpath.(datapage_url, data_hrefs)
end

```

Related is the issue to have a proper `generator ` API based on using that interactively  
[https://github.com/oxinabox/DataDepsGenerators.jl/issues/3](https://github.com/oxinabox/DataDepsGenerators.jl/issues/3)

---

<div class="post-metadata">

**Author:** ![pontus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pontus/32/10438_2.png) [@pontus](https://discourse.julialang.org/u/pontus)\
**Post date:** [October 18, 2019, 9:00pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/7 "2019-10-18T21:00:28Z")

</div>

> [@oxinabox](#):
>

Ok, cool! Put this in the misc\_extractors file and reloaded kernel. (added as a ‘v2’ function, and had to add an `end`).  
Tried to give it a spin but am getting an error:

```julia
get_dataurls_from_webserver_index_v2("https://data.ok.gov/dataset/health-care-cost-growth/")

```

Out:  
`UndefVarError: ele not defined`  
with the previous version, it produces:

```julia
AssertionError: Gumbo.HTMLElement{:a}:
<a href="/group/okstatestat">
  OKStateStat
</a>

```

Which is because that isn’t a valid URL to feed it for this function, will try and find a better example.  
Many thanks for your consideration here!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 18, 2019, 9:46pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048/8 "2019-10-18T21:46:50Z")

</div>

probably a small typo