lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter3Title

pythonIntroCrawlingChapter4Title

# Wikipedia allows public crawling.
# However, when crawling, it's a good practice to check the site's robots.txt file
# to ensure crawling is permitted.

# Library for sending network requests
import requests

# Library for parsing web pages
from bs4 import BeautifulSoup

# Wikipedia Main Page URL
url = 'https://en.wikipedia.org/wiki/Main_Page'

# Fetch the web page using requests
response = requests.get(url)

# Parse the HTML content of the web page into a BeautifulSoup object
soup = BeautifulSoup(response.text, 'html.parser')

# Retrieve the content of the title tag from the web page
page_title = soup.find('title').text
print(f"Page Title: {page_title}")

# Legal and Ethical Responsibilities in Web Crawling

Many websites prohibit or restrict crawling through their terms of service or `robots.txt` (a document indicating whether web crawlers are allowed to crawl). Therefore, when performing web crawling, it's crucial to carefully consider your legal and ethical responsibilities.

<br />

## Legal Responsibilities

1. `Copyright Law`: Most website content is protected by copyright. Therefore, when crawling and reusing website data, be careful not to violate copyright laws. Special caution is required when using collected data for commercial purposes or making it public.

2. `Data Protection Laws`: Numerous countries enforce strict regulations on the collection and use of personal information. If personal data is collected through web crawling, you must comply with the data protection laws of the respective country.

3. `Terms of Service`: A website's terms of service define the rules on how its data can be used. Many websites have clauses prohibiting or limiting crawling, so it's important to review the terms of service beforehand.

<br />

## Ethical Responsibilities

1. `Minimize Server Load`: Crawling can burden website servers. Excessive crawling can lead to server overloads, disrupting normal service operation. Therefore, it's essential to adjust the crawling frequency appropriately and minimize server load.

2. `Adherence to robots.txt`: A website's `robots.txt` file designates pages that crawlers should not access. For ethical crawling, you must adhere to the instructions in this file.

3. `Transparency in Data Use`: When using collected data, be transparent about the source and method of collection. Also, avoid distorting data or spreading misinformation.

<br />

## Practice

Click the _`Run Code`_ button on the right side of the screen to check the crawling results or modify the code!

python_execution