lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

lesson7Title

lesson8Title

pythonIntroCrawlingChapter3Title

lesson9Title

lesson10Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter2Title

pythonIntroCrawlingChapter4Title

import requests
from requests.exceptions import HTTPError, Timeout

# URL to request
url = 'https://httpbin.org/get'

try:
    # Send HTTP GET request, if no response within 5 seconds, raise Timeout exception
    response = requests.get(url, timeout=5)
    # If timeout is set to 0.1, a Timeout exception will occur if there is no response within 0.1 seconds
    
    # Raise an exception if there's an issue with the response status code
    response.raise_for_status()

except HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except Timeout as timeout_err:
    print(f'Timeout error occurred: {timeout_err}')
except Exception as err:
    print(f'An error occurred: {err}')
else:
    print('Success!')
    print('-' * 20)
    # Print response content
    print(response.text)

# HTTP Headers, Parameters, and Exception Handling

Interacting with web servers and APIs requires specifying requests with `headers`, `parameters`, and handling errors through `exception handling`.

 

## HTTP Headers

`HTTP headers` are part of the HTTP request and response messages, containing additional information transferred between a client and a server.

These headers are sent separately from the message body and contain data about the properties of the message, the content of the body, and specific settings of the client and server.

They are generally divided into `Request Headers` and `Response Headers`, each providing additional information about the request and response, respectively.

 

### Usage

- Add headers to an HTTP request using the `headers` parameter.

```python title="Adding Headers to an HTTP Request"
import requests

# Define headers
headers = {'User-Agent': 'My User Agent 1.0'}

# Add header to GET request
response = requests.get('https://api.github.com', headers=headers)
```

 

## HTTP Parameters

`HTTP parameters` are pieces of information sent from the client (web browser) to the server, added in the form of name-value pairs in the URL to specify HTTP requests (e.g., search queries, page numbers).

These parameters are added to the end of the URL with a `?`, followed by the `name=value` format. Multiple parameters are separated by an `&`.

```bash title="Example of Adding URL Parameters"
# Add value 'codefriends' to parameter name 'query'
# Add value 'ascending' to parameter name 'sort'
https://example.com/search?query=codefriends&sort=ascending
```

### Usage

- Use the `params` parameter to add parameters to a GET request.

```python title="Example of Adding Parameters"
import requests

# Add parameters query=python&page=2
parameters = {'query': 'python', 'page': 2} 
response = requests.get('https://api.github.com/search', params=parameters)
```

 

## Exception Handling Methods

If server error responses (4xx, 5xx status codes) are not handled appropriately, the program may crash or errors could occur.

Note that **4xx** status codes indicate client errors, and **5xx** status codes indicate server errors.

The requests library handles exceptions with the `response.raise_for_status()` method.

The raise_for_status() method raises an HTTPError if the response status code is 400 or greater.

```python title="Example of HTTP Error Exception Handling"
try:
 response = requests.get('https://api.github.com')
 response.raise_for_status()
except requests.exceptions.HTTPError as err:
 print(f'HTTP error occurred: {err}')
```

 

## Timeout Handling

If a request does not complete within a specified time, it is advisable to raise a timeout exception to prevent waiting indefinitely.

A timeout sets a maximum wait time for the request, using the `timeout` parameter.

If the maximum wait time is exceeded, a requests.exceptions.Timeout exception is raised.

```python title="Example of Timeout Handling"
try:
 # Wait up to 10 seconds
 response = requests.get('https://api.github.com', timeout=10)
except requests.exceptions.Timeout as err:
 print(f'Request timed out: {err}')
```

 

## Practice

Click the _`Run Code`_ button on the right side of the screen to view the crawling results or modify the code!

python_execution