lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter3Title

pythonIntroCrawlingChapter4Title

import requests
from bs4 import BeautifulSoup

# URL of the web page to crawl
url = 'https://example.com'

# Use the requests library to fetch the HTML content of the web page
response = requests.get(url)

# Convert the HTML content of the web page to a string
html_content = response.text

# Create a Beautiful Soup object
soup = BeautifulSoup(html_content, 'html.parser')

# Extract the desired data
# Example: Get all the titles (h1 tags) from the web page
titles = soup.find_all('h1')

# Example: Get all the paragraphs (p tags) from the web page
paragraphs = soup.find_all('p')

# Print the extracted data
for title in titles:
    print(title.text)

print('===')

# Print the extracted data
for paragraph in paragraphs:
    print(paragraph.text)

# Web Crawling - Collecting Data with Python Code

Have you heard of the term `Web Crawling`?

Web Crawling refers to the process of **automatically collecting data** from websites. This task is performed by an automated software (bot) known as a Crawler (or Spider), which visits various web pages to extract desired data.

It is widely used in **data collection**, **indexing** (a system that helps retrieve specific information efficiently), and product data aggregation for online price comparison platforms.

When using web crawling for personal or commercial purposes beyond non-profit activities, it's important to respect the website's Terms of Service and be aware of the legal aspects such as privacy and copyright.

<br />

## Crawling Process

1. `Requesting and Collecting Web Pages`: The crawler sends an HTTP request to a web page and retrieves its HTML content from the server.

2. `Data Parsing`: The received HTML tags of the web page are analyzed to extract necessary data such as text, links, images, etc.

3. `Data Storage`: The extracted data is saved in a database or file.

4. `Repetition`: Steps 1-3 are repeated, requesting, collecting, and storing new web pages until set conditions are met.

<br />

## Technologies Used

- `HTML`: The language that defines the structure and content of web pages.

- `HTTP Request`: Requests web page data from a web server.

- `Parsing`: Parsing refers to **syntax analysis**,  the process of extracting specific data from structured content. In Python, libraries like Beautiful Soup and lxml are often used for HTML parsing.

<br />

## Use Cases of Web Crawling

1. `Search Engine Optimization (SEO) and Indexing`

   - Search engines (Google, Bing, etc.) use web crawlers to collect web pages, then index and rank these pages on search engine result pages based on the collected data.

2. `Data Analysis and Market Research`

   - Analyzing data from commercial websites to study market trends, price changes, product reviews, etc.

3. `Social Media Analysis`

   - Collecting data from social media platforms to analyze user opinions, trends, and social reactions.

4. `Academic Research`

   - Researchers use web crawling to collect academic materials, open data sets, news articles, and utilize them in research.

5. `Automated Monitoring`

   - Continuously monitoring real-time data such as stock prices, exchange rates, and weather information to track changes.

<br />

## Practice

Click on the _`Run Code`_ button on the right-hand side of the screen to review the crawling results or edit the code!

python_execution