lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

pythonIntroCrawlingChapter3Title

lesson9Title

lesson10Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter4Title

import requests
from bs4 import BeautifulSoup

# Example of handling an HTML response
html_url = 'https://example.com'
html_response = requests.get(html_url)
html_soup = BeautifulSoup(html_response.text, 'html.parser')
title = html_soup.find('title').text
print("Web Page Title:", title)

# Example of handling a JSON response
json_url = 'https://api.github.com/'
json_response = requests.get(json_url)
json_data = json_response.json()
print("JSON Response Data:", json_data)

# Handling HTML and JSON Responses

When using web APIs or scraping web pages, the responses from the server can come in various formats.

<br />

## 1. HTML Responses

For web pages, responses usually come in **HTML** format. You can parse and extract data from static HTML using libraries like `BeautifulSoup`.

In this case, after requesting an HTML page with `requests`, you can parse and extract the desired data using `BeautifulSoup`.

```python title="HTML Data Extraction Example"
import requests
from bs4 import BeautifulSoup

response = requests.get('https://example.com')
soup = BeautifulSoup(response.text, 'html.parser')

# Extract webpage title
title = soup.find('title').text
```

<br />

## 2. JSON Responses

API responses from servers are mostly in JSON format.

When you receive a JSON response, you can convert it into a Python object using the `response.json()` method.

Conversion takes place in the form of a Python dictionary, making it easy to utilize the data in Python code.

Here, you can also use Python's `json` module to convert JSON data into a Python object.

```python title="JSON Data Usage Example"
import requests
from bs4 import BeautifulSoup

response = requests.get('https://httpbin.org/get')

# Parse JSON data
data = response.json()

url = data['url']

# Print JSON data
print(data)

print('-' * 20)

# Utilize JSON data
print("Host:", url)
```

<br />

## Practice

Click the _`Run Code`_ button on the right side of the screen to verify crawling results or adjust the code!

python_execution