Scraping youtube webcomments with WebDriver.jl

General Usage
1

Hi All,

I am trying to replicate this post How to Scrape Youtube Comments with Python | by François St-Amant | Towards Data Science in Julia. The author scrapes comments from yt videos using selenium driver and does some sentiment analysis on them.

The author does something like

import time
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

data=[]
with Chrome(executable_path=r'C:\Program Files\chromedriver.exe') as driver:
    wait = WebDriverWait(driver,15)
    driver.get("https://www.youtube.com/watch?v=kuhhT_cBtFU&t=2s")

    for item in range(200): 
        wait.until(EC.visibility_of_element_located((By.TAG_NAME, "body"))).send_keys(Keys.END)
        time.sleep(15)

    for comment in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#content"))):
        data.append(comment.text)

I want to rewrite it with WebDriver.jl, but I have not found an equivalent of WebDriverWait and expected_conditions in the library. Does anyone tried something like this or has some experience with it?

Thanks for an answer.
Tomas

2

I’m not familiar with the Module but there is implicitly_wait which seems to do something similar

Sets a sticky timeout to implicitly wait for an element to be found,
or a command to complete.

and as this is PyCall you can always call the Python object directly without being forced to use the wrapper. Sadly I leave that as an exrcise :slight_smile:

3

Thanks, every little helps.