Home
Head's Up: I'm in the middle of upgrading my site. Most things are in place, but there are something missing and/or broken including image alt text. Please bear with me while I'm getting things fixed.

Snippet Get Web Page in Python

Here's the simple snippet I use to scrape basic web pages in Python.

python 
def get_web_page(url):
    try:
        with urllib.request.urlopen(url) as response:
            return response.read().decode("utf-8")
    except:
        return ""

Most of the examples I see only have the middle part :

python 
with urllib.request.urlopen(url) as response:
	return response.read().decode("utf-8")

The problem is that code crashes if the server sends an error code back. Wrapping it with the try lets you handle that.

Here's a full sample :

python 
#!/usr/bin/env python3

import urllib.request

def get_web_page(url):
    try:
        with urllib.request.urlopen(url) as response:
            return response.read().decode("utf-8")
    except:
        return ""

if __name__ == "__main__":
    html_doc = get_web_page("https://www.alanwsmith.com/")
    print(html_doc)