# Getting data directly from a website

**URL:** https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241
**Category:** Performance
**Created:** [July 7, 2018, 10:43pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241 "2018-07-07T22:43:35Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![mvcavalcanti](https://avatars.discourse-cdn.com/v4/letter/m/919ad9/32.png) [@mvcavalcanti](https://discourse.julialang.org/u/mvcavalcanti)
#### Post date: [July 7, 2018, 10:43pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/1 "2018-07-07T22:43:35Z")

</div>

Hi. I’m working on a script in which I do some analysis about energy prices. The first step to process these values is to get them. They are available on a public web site, through a table of five columns with historical prices. Today, what I do is to write them manually in a .csv or a .dat file, then I read it with Julia.

I think that this procedure is very inefficient, so I’d like to know if there is a way to read the table from this website with Julia. Data is available [here](https://www.ccee.org.br/portal/faces/pages_publico/o-que-fazemos/como_ccee_atua/precos/precos_medios?_afrLoop=48390325021139&_adf.ctrl-state=e2nvm5aes_1#!%40%40%3F_afrLoop%3D48390325021139%26_adf.ctrl-state%3De2nvm5aes_5).

Thanks.

---

<div class="post-metadata">

### Author: ![non-Jedi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/non-jedi/32/3645_2.png) [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)
#### Post date: [July 7, 2018, 10:51pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/2 "2018-07-07T22:51:37Z")

</div>

That site renders all of its content using javascript, so downloading and manually parsing html isn’t an option. Your best bet if you want a julia solution is probably going to be using pycall.jl with one of the many python packages for scraping the web. Only a few support javascript-rendered stuff; I’d probably start by trying out selenium.

---

<div class="post-metadata">

### Author: ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [July 7, 2018, 11:55pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/3 "2018-07-07T23:55:36Z")

</div>

Try [this](https://www.ccee.org.br/preco/precoMedio.do) URL instead.

In general, any data loaded to a page asynchronously still should have its source. In many cases, you can inspect network requests that your browser does (e.g. in Firefox: Menu → Web Developer → Toggle Tools → Network tab in the newly opened window) and find the one that loads the data. Corresponding resource is often static and easy to grab, like the one I posted above.

---

<div class="post-metadata">

### Author: ![mvcavalcanti](https://avatars.discourse-cdn.com/v4/letter/m/919ad9/32.png) [@mvcavalcanti](https://discourse.julialang.org/u/mvcavalcanti)
#### Post date: [July 8, 2018, 12:05am UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/4 "2018-07-08T00:05:14Z")

</div>

@non-Jedi thank you very much! I’ll read more about pycall and selenium, it seems very useful.

@dfdx very interesting! But to get the data from this URL, do you suggest to use one of @non-Jedi options or something else?

---

<div class="post-metadata">

### Author: ![non-Jedi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/non-jedi/32/3645_2.png) [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)
#### Post date: [July 8, 2018, 12:56am UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/5 "2018-07-08T00:56:10Z")

</div>

Ah! Very good. If the resource is available statically, probably beautiful soup or one of the many python libraries built for scraping but simpler than selenium would be better. However, for the simple task of parsing html for the data you need, I’d try one of the html parsing libraries available in Julia rather than python: [https://github.com/search?q=html%20language%3Ajulia](https://github.com/search?q=html%20language%3Ajulia)

Possibly look at EzXML or Gumbo.

---

<div class="post-metadata">

### Author: ![Liso](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [July 8, 2018, 12:08pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/6 "2018-07-08T12:08:22Z")

</div>

**Edit:**  
cookies parameter help to get proper data:

```julia
r = HTTP.get("https://www.ccee.org.br/preco/precoMedio.do", cookies=true);

```

* * *

`HTTP.jl` ~~was unsuccesful here too:~~

```julia
0.7.0-beta.189> r = HTTP.get("https://www.ccee.org.br/preco/precoMedio.do");

0.7.0-beta.189> String(r.body)
"<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>302 Found</title>\n</head><body>\n<h1>Found</h1>\n<p>The document has moved <a href=\"https://login.ccee.org.br:443/oam/server/obrareq.cgi?encquery%3DZYUTYHbstezO3EAE%2FL6JC1pA2hoxJ4M8DcqXXvRIS%2FzRduVNGiWRZT1vy1ILgB9%2BuSxtZZU6dWtKAFdaB7wCNm2ozR1%2FU56731QJK7A0iDOFSIAqPBoLc6t61ju%2Fi8HpDE0UxlDKXQR61q794LYOF7GT%2BN15JSTZHZZjeT5TqJ7cw5A6li0zMZQcekPGrLscV00v5VNu3vsiS3svRYbL6dU4L1CjEFpgpSwZZKtfMKdGl4rzSfQZjx4ah1OYH5Tn3B9mPFRw7dVk3DqAsszZUXHv4fuPyzmPKak1eSBDL%2F78lCo6L25l8KreuL3t%2BZVt%20agentid%3Dagentewg%20ver%3D1%20crmethod%3D2\">here</a>.</p>\n</body></html>\n"

```

BTW it could be 4 liner in Python:

```python
import requests
import pandas as pd
r = requests.get("https://www.ccee.org.br/preco/precoMedio.do")
df = pd.read_html(io.StringIO(r.text))[0]

```

It could be simpler but `pd.read_html` ended with similar problem like `HTTP.jl`(without cookies parameter) → `HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.`

---

<div class="post-metadata">

### Author: ![Liso](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [July 8, 2018, 3:42pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/8 "2018-07-08T15:42:19Z")

</div>

I am learning Julia too, so I just tried to get data with “brutal force”.

I would really understand if you wont like it. 😉

```julia
using HTTP, DataFrames

r = HTTP.get("https://www.ccee.org.br/preco/precoMedio.do", cookies=true);
data = String(r.body);

function replacer(a, b) for i in b a=replace(a, i) end; a end;

csv_data = "Mês,SE/CO,S,NE,N\n" * # adding header
   replacer(
     data[findfirst(r"<tbody.*?>(.*)</tbody>"s, data)], # just data for table body
     ["\t"=>"", # clear the mess
      "\r\n"=>"",  
      r"<tr.*?>"=>"",  
      "</tr>"=>"\n", # <tr>...</tr> as lines
      r"</td><td>"=>"\",\"", # comma between values + quotas
      r"<td>(.*?)</td>"=>s"\"\1\"", # quotas to begin and end of line
      r"<.?tbody>"=>""]); # clear the mess

df = readtable(IOBuffer(csv_data))
182×5 DataFrame
│ Row │ Mês │ SE_CO │ S │ NE │ N │
├─────┼─────────┼────────┼────────┼────────┼────────┤
│ 1 │ 06/2018 │ 472,87 │ 472,87 │ 441,96 │ 441,96 │
│ 2 │ 05/2018 │ 325,46 │ 325,46 │ 211,57 │ 159,47 │
│ 3 │ 04/2018 │ 109,71 │ 109,71 │ 108,64 │ 51,13 │
│ 4 │ 03/2018 │ 219,23 │ 219,23 │ 218,14 │ 40,16 │
│ 5 │ 02/2018 │ 188,79 │ 188,54 │ 178,54 │ 42,91 │
⋮
│ 177 │ 10/2003 │ 25,82 │ 26,30 │ 19,88 │ 25,73 │

```

---

<div class="post-metadata">

### Author: ![mvcavalcanti](https://avatars.discourse-cdn.com/v4/letter/m/919ad9/32.png) [@mvcavalcanti](https://discourse.julialang.org/u/mvcavalcanti)
#### Post date: [July 8, 2018, 6:09pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/9 "2018-07-08T18:09:20Z")

</div>

@Liso WOW! Thank you very much!! It was very very helpful. But unfornunately I had a problem running your code. I got this with csv\_data

“Mês,SE/CO,S,NE,N\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n"\1"\n”

Then I got this error with df:

ERROR: DimensionMismatch(“Number of columns (1) and number of column names (5) are not equal”)

Could you help just one more time, please? 😁

Thanks

---

<div class="post-metadata">

### Author: ![Liso](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [July 8, 2018, 7:52pm UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/10 "2018-07-08T19:52:38Z")

</div>

Seems Julia 0.6.x has some problems with replace regex. This is working for me in 0.6.1 →

```julia
julia> csv_data = "Mês,SE/CO,S,NE,N\n" * # adding header
         replacer(
            data[findfirst(r"<tbody.*?>(.*)</tbody>"s, data)], # just data for table body
            ["\t"=>"", # clear the mess
             "\r\n"=>"",  
             r"<tr.*?>"=>"",  
             "</tr>"=>"\n", # <tr>...</tr> as lines
             r"</td><td>"=>"\",\"", # comma between values + quotas
             r"<[/]?td>"=>"\"", # quotas to begin and end of line
             r"<[/]?tbody>"=>""]); # clear the mess

```

---

<div class="post-metadata">

### Author: ![mvcavalcanti](https://avatars.discourse-cdn.com/v4/letter/m/919ad9/32.png) [@mvcavalcanti](https://discourse.julialang.org/u/mvcavalcanti)
#### Post date: [July 9, 2018, 12:09am UTC](https://discourse.julialang.org/t/getting-data-directly-from-a-website/12241/11 "2018-07-09T00:09:04Z")

</div>

@Liso It worked perfectly now! Thank you very much, you helped me a lot. 😀