lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

pythonIntroCrawlingChapter4Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter3Title

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

# Launch Chrome WebDriver and open a browser window
driver = webdriver.Chrome()

# Navigate to the 'Markets' page on Yahoo Finance
driver.get('https://finance.yahoo.com/markets/')

# Wait until the page is fully loaded (maximum wait time: 10 seconds)
wait = WebDriverWait(driver, 10)

# Locate the 'Americas' section
try:
    # Find h3 tag with text 'Americas'
    americas_section = wait.until(EC.presence_of_element_located((By.XPATH, "//h3[text()='Americas']")))

    # Scroll to the 'Americas' section
    actions = ActionChains(driver)

    # Move mouse to the 'Americas' section
    actions.move_to_element(americas_section).perform()

    # Locate the table in the parent element of the 'Americas' section
    parent_section = americas_section.find_element(By.XPATH, "./ancestor::section[contains(@data-testid, 'world-indices')]")

    # Find the table
    table = parent_section.find_element(By.XPATH, ".//table")

    # Extract the table headers
    headers = [header.text for header in table.find_elements(By.XPATH, ".//th")]

    # Extract the table rows
    rows = table.find_elements(By.XPATH, ".//tbody/tr")

    # Initialize a list to store data
    table_data = []

    # Extract column data from each row and add it to the list
    for row in rows:
        # Extract column data from each row
        columns = row.find_elements(By.XPATH, ".//td")

        row_data = {}  # First create an empty dictionary

        # Assume headers list and columns list have the same length
        for i in range(len(headers)):
            # Get the ith element from the headers list
            header = headers[i]

            # Get the text of the ith element in the columns list
            column_value = columns[i].text
            
            # Add to the dictionary with header as key and column_value as value
            row_data[header] = column_value

        # Add the data to the list
        table_data.append(row_data)

    # Convert the extracted data into a pandas DataFrame
    df = pd.DataFrame(table_data)

    # Select and print only the 'Symbol' and 'Price' columns from the sorted DataFrame
    df_filtered = df[['Symbol', 'Price']]

    # Print the sorted data
    print(df_filtered)

except Exception as e:
    print(f"ERROR: WebDriver error occurred: {str(e)}")

finally:
    # Close the browser
    driver.quit()

# Stock Data Crawling

Stock and chart data have the `dynamic` characteristic of changing constantly over time.

Tracking these real-time changes is crucial for data extraction.

Until now, we have learned methods to extract `static` data that doesn't change rapidly, using libraries like requests and BeautifulSoup.

However, to extract `dynamic data` controlled by JavaScript, you'll need to use a library called **Selenium**.

In the practice section, you can check out the code that uses Selenium to crawl the U.S. stock indexes from Yahoo Finance.

> Note: For more detailed information about Selenium, you can check out `Essential Knowledge for Automating Work`, Chapter 3 `Collecting Data through Web Crawling`, in the 3rd course of <a href="/packages/automation-intro" target="_blank">Automation with Python and AI</a>.

<br />

## Practice

Press the _`Run Code`_ button on the right side of the screen to check the crawling results or modify the code!

selenium