# Downloading all information from Ctrl-Shift-I

**URL:** https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430
**Category:** New to Julia
**Created:** [May 19, 2021, 11:11am UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430 "2021-05-19T11:11:50Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)
#### Post date: [May 19, 2021, 11:11am UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430/1 "2021-05-19T11:11:50Z")

</div>

To my dismay, I often find that the data I am after on a website disappears with the following line of code:

using Gumbo  
using Cascadia

page = parsehtml(read(download(url), String))  
collected = string(page)

Yet, when I use inspect Ctrl-Shift-I in the browser, the information is there, clear as day.

So, my question is simple. How do I download all the data contained in Ctrl-Shift-I, convert it to string, and fetch the parts I want using the regularities that are there?

---

<div class="post-metadata">

### Author: ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)
#### Post date: [May 19, 2021, 11:33am UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430/2 "2021-05-19T11:33:38Z")

</div>

It is possible that page is uploaded dynamically through the AJAX. In this case you either need to do necessary calls yourself or in a more complicated scenarios you can use something like Selenium.

---

<div class="post-metadata">

### Author: ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)
#### Post date: [May 19, 2021, 11:47am UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430/3 "2021-05-19T11:47:44Z")

</div>

What do you mean by “make necessary calls yourself” as opposed to using Selenium?

My task is not very complex, I don’t think. Could you ellaborate what you mean by making a call? Perhaps provide an example of such?

---

<div class="post-metadata">

### Author: ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)
#### Post date: [May 19, 2021, 11:58am UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430/4 "2021-05-19T11:58:27Z")

</div>

I mean, you can go through the page source, identify ajax calls and since these calls just requests to some other resource, use them. Or you can use `Network` tab in browser Web Developers Tools (usually you can run it with F12 key) and after refresh you can see all sorts of intermediate calls, which can be used.

As an example, consider this page: [https://www.nasdaq.com/market-activity/earnings](https://www.nasdaq.com/market-activity/earnings) If you try to `download` it, corresponding html will be empty. But, you can turn on Tools, open Networks tab, refresh page and after some investigation you will find, that there is a call [https://api.nasdaq.com/api/calendar/earnings?date=2021-05-19](https://api.nasdaq.com/api/calendar/earnings?date=2021-05-19) which actually populate the table. So, instead of downloading earnings html page, you can request directly `api.nasdaq.com` and process response data.

It is different from Selenium, which is basically full browser and it executes all javascripts on the page and you do not need to work through the calls or read source. You can just grab resulting html page.

---

<div class="post-metadata">

### Author: ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [May 19, 2021, 12:37pm UTC](https://discourse.julialang.org/t/downloading-all-information-from-ctrl-shift-i/61430/5 "2021-05-19T12:37:36Z")

</div>

What @Skoffer refers to is that on modern (responsive) web pages typically the content you see is loaded on a second step.  
The first step is that the browser loads html, css and javascript code.  
Second step is, that the javascript code is executed and fills additional content into html containers. This additional content is loaded from the servers using javascript AJAX protocol.

`download(url)`  
does only the first step of this process.

Selenium is what you need, as @Skoffer recommended.
