# Pull data from websites in Julia

**URL:** <https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112>\
**Category:** General Usage\
**Created:** [December 1, 2022, 7:25pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112 "2022-12-01T19:25:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Albert\_Zevelev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albert_zevelev/32/11844_2.png) [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)\
**Post date:** [December 1, 2022, 7:25pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/1 "2022-12-01T19:25:31Z")

</div>

As ski resorts begin to open, they update data on their websites throughout the day, every day.  
Two factors skiiers care about are: (1) # trails currently open, (2) current Snow base

Each resort website posts this info:  
[Park City trails](https://www.parkcitymountain.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/9/6/96b302292285ccde92c13900e1e73e8b2c79dff3.png)  
[Park City snow](https://www.parkcitymountain.com/the-mountain/mountain-conditions/snow-and-weather-report.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/3/4/34b30108c87668dd0b71b4983eb044470286c888.png)

[Vail Trails](https://www.vail.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/0/c/0c556eeb86ed9b95405b0a5cf15a64f4d52838d6.png)  
[Vail Snow](https://www.vail.com/the-mountain/mountain-conditions/snow-and-weather-report.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/c/8/c88402889cfea142b7ea61c121c9e999beb13bc5.png)

[Whistler Trails](https://www.whistlerblackcomb.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/7/5/75bc737c8f6192d48a3f603d885c28feca8cca8b.png)  
[Whistler Snow](https://www.whistlerblackcomb.com/the-mountain/mountain-conditions/snow-and-weather-report.aspx): ![image](https://global.discourse-cdn.com/julialang/original/3X/d/d/dd6db422d21b5e020c1ae5bca4060d8160d7caeb.png)

Can I automatically pull this data from the links above using Julia?  
([skicentral.com](https://www.skicentral.com/openareas-skireport.html#south) does this, but not particularly well & I’d like to learn how to do it in Julia if possible)

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [December 1, 2022, 7:34pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/2 "2022-12-01T19:34:51Z")

</div>

Scraping with Julia: [Scraping web pages with Julia HTTP & Gumbo: Tutorial](https://julia.school/julia/scraping/)  
Someone’s scraping project for ski snow reports, but not Julia: [Web Scraping for custom API - DEV Community 👩‍💻👨‍💻](https://dev.to/sofiajonsson/web-scraping-for-custom-api-6dp)

You should be able to combine parts from both to get what you want.

Scraping is notoriously fussy and fragile.

You can get more results at once from [Snow Report | Colorado Ski Country USA](https://www.coloradoski.com/snow-report), for example

---

<div class="post-metadata">

**Author:** ![Albert\_Zevelev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albert_zevelev/32/11844_2.png) [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)\
**Post date:** [December 1, 2022, 8:52pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/3 "2022-12-01T20:52:30Z")

</div>

Here is my very raw, very naive attempt.  
What’s amazing is I have zero experience “webscraping” & I still got it to work.

```julia
using HTTP;
l_pc ="https://www.parkcitymountain.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";
l_v ="https://www.vail.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";
l_w = "https://www.whistlerblackcomb.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";

# Number of "runs" open in Park City
r = HTTP.get(l_pc); # get link 
rs = String(r.body); # make into long string 
a1="id=\"runs\">\r\n\r\n <div class=\"terrain_summary__circle\"\r\n data-open="
a2=findfirst(a1, rs) # findall 
a3 = rs[( a2[end] +2 ) : ( a2[end] +4)] 
#################

# Number of "runs" open in Vail 
r = HTTP.get(l_v); # get link 
rs = String(r.body); # make into long string 
a1="id=\"runs\">\r\n\r\n <div class=\"terrain_summary__circle\"\r\n data-open="
a2=findfirst(a1, rs) # findall 
a3 = rs[( a2[end] +2 ) : ( a2[end] +4)] 
#################

# Number of "runs" open in Whistler
r = HTTP.get(l_w); # get link 
rs = String(r.body); # make into long string 
a1="id=\"runs\">\r\n\r\n <div class=\"terrain_summary__circle\"\r\n data-open="
a2=findfirst(a1, rs) # findall 
a3 = rs[( a2[end] +2 ) : ( a2[end] +4)] 
#################

```

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [December 1, 2022, 9:00pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/4 "2022-12-01T21:00:12Z")

</div>

All three of those are owned by Vail Resorts, so it’s not too surprising that the snow report html is similar. You’ll need something different for other resorts.

---

<div class="post-metadata">

**Author:** ![Albert\_Zevelev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albert_zevelev/32/11844_2.png) [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)\
**Post date:** [December 1, 2022, 9:02pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/5 "2022-12-01T21:02:25Z")

</div>

Yeah, I’m currently on Epic pass…  
Here is a cleaner way to do it w/ a String. (still no good & today morning I didn’t know what “webscraping” meant)

```julia
using HTTP;
l_pc ="https://www.parkcitymountain.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";
l_v ="https://www.vail.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";
l_w = "https://www.whistlerblackcomb.com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx";

#location of string w/ # Runs open.
a1="id=\"runs\">\r\n\r\n <div class=\"terrain_summary__circle\"\r\n data-open="
RUNS = [];

for resort in [l_pc l_v l_w] 
    r = HTTP.get(resort); # get link
    rs = String(r.body); # make into string 
    a2=findfirst(a1, rs) # findall get index w/ Number of Runs open
    num_runs_open = rs[( a2[end] +2 ) : ( a2[end] +4)] # get # Runs open 
    push!(RUNS, num_runs_open)
end 

julia> RUNS
3-element Vector{Any}:
 "120"
 "86\""
 "30\""

```

---

<div class="post-metadata">

**Author:** ![chris-b1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris-b1/32/14165_2.png) [@chris-b1](https://discourse.julialang.org/u/chris-b1)\
**Post date:** [December 1, 2022, 9:39pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/6 "2022-12-01T21:39:09Z")

</div>

Here’s a little more robust solution using an HTML/XML parser. The heavy lifting is the `"//div"...` line, which is an XPath query to search XML - in this case finding a div with the attribute `data-terrain-status-id` equal to `"runs"` then selecting the next child element.

```julia
using EzXML
using HTTP

function get_open(url)
    r = HTTP.get(url)
    tree = EzXML.parsehtml(r.body)
    node = findfirst("//div[@data-terrain-status-id=\"runs\"]/div", tree)
    val = node["data-open"]
    return parse(Int, val)
end

```

---

<div class="post-metadata">

**Author:** ![j-fu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j-fu/32/11373_2.png) [@j-fu](https://discourse.julialang.org/u/j-fu)\
**Post date:** [December 1, 2022, 10:19pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/7 "2022-12-01T22:19:31Z")

</div>

Try [Gumbo.jl](https://github.com/JuliaWeb/Gumbo.jl). It parses HTML and gives you structured access to the elements.

[Here](https://github.com/j-fu/coronaplot/blob/28c492795521a94e62c27aa1334e44efdeffec87/code/plot_infected.jl#L96) is an example from the time of the coronaplotting craze.

---

<div class="post-metadata">

**Author:** ![Albert\_Zevelev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albert_zevelev/32/11844_2.png) [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)\
**Post date:** [December 2, 2022, 12:27am UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/8 "2022-12-02T00:27:09Z")

</div>

@chris-b1 thanks!

```julia
using HTTP, EzXML;
s1 = "https://www.";
names = ["parkcitymountain" "vail" "whistlerblackcomb" "beavercreek" "breckenridge" "northstarcalifornia"];
sruns = ".com/the-mountain/mountain-conditions/terrain-and-lift-status.aspx"
ssnow = ".com/the-mountain/mountain-conditions/snow-and-weather-report.aspx"
RUNS = [];
for resort in names
    link_runs = s1 * resort * sruns 
    r = HTTP.get(link_runs)
    tree = EzXML.parsehtml(r.body)
    #
    node_runs = findfirst("//div[@data-terrain-status-id=\"runs\"]/div", tree)
    num_runsopen = node_runs["data-open"] |> x -> parse(Int, x)
    num_runstotal = node_runs["data-total"] |> x -> parse(Int, x)
    #
    push!(RUNS, [resort num_runsopen num_runstotal])
    #
    link_snow = s1 * resort * ssnow 
    r = HTTP.get(link_snow)
    tree = EzXML.parsehtml(r.body)
    # How do we get: snowfall.Depth.Inches etc???
end 
pushfirst!(RUNS, ["Resort" "Runs Open" "Runs Total"])
RUNS = vcat(RUNS...)

```

Gives  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/3/a/3a108d8bbe44295a7b4fab47016cf754ae76c66e.png)

I wonder if there is an easy way to scrape all the ski resort websites from [https://www.epicpass.com/](https://www.epicpass.com/)?

Also it looks more complicated to scrape [snowfall](https://www.parkcitymountain.com/the-mountain/mountain-conditions/snow-and-weather-report.aspx) “BASE DEPTH” & “CURRENT SEASON”.  
(not sure the approach used for RUNS above would work…)

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [December 2, 2022, 3:43pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/9 "2022-12-02T15:43:56Z")

</div>

Try scraping for the individual links at [Snow and Weather Reports | Snow.com](https://www.snow.com/mountainconditions/snowandweatherreports.aspx)

---

<div class="post-metadata">

**Author:** ![chris-b1](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chris-b1/32/14165_2.png) [@chris-b1](https://discourse.julialang.org/u/chris-b1)\
**Post date:** [December 2, 2022, 4:26pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/10 "2022-12-02T16:26:31Z")

</div>

> Also it looks more complicated to scrape [snowfall](https://www.parkcitymountain.com/the-mountain/mountain-conditions/snow-and-weather-report.aspx) “BASE DEPTH” & “CURRENT SEASON”.  
> (not sure the approach used for RUNS above would work…)

A trick which can be helpful, if is you select the element in Chrome (right click, Inspect) there is an option to copy an XPath query to an element. Sometimes you’ll want to clean up or modify that query, but it can be a helpful starting point.

```julia
//*[@id="snow_report_1"]/div[2]/ul/li[6]/div/h5/text()

```

![image](https://global.discourse-cdn.com/julialang/original/3X/1/5/15e7d4a6998ea2a5f92d754c45ff1f155e491de7.png)

---

<div class="post-metadata">

**Author:** ![Albert\_Zevelev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albert_zevelev/32/11844_2.png) [@Albert\_Zevelev](https://discourse.julialang.org/u/Albert_Zevelev)\
**Post date:** [December 2, 2022, 8:33pm UTC](https://discourse.julialang.org/t/pull-data-from-websites-in-julia/91112/11 "2022-12-02T20:33:17Z")

</div>

mehhh, no luck pulling snow data (base depth & current season), but thanks anyways…