Here’s an example that I’ve used to get all the text from Paul Graham’s website:
$ wget --mirror --wait=15 --random-wait --adjust-extension --domains=paulgraham.com --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" https://paulgraham.com/
The --mirror parameter is short for -r -N -l inf --no-remove-listing:
| Parameter | Description |
|---|---|
-r, --recursive | Enables recursive downloading (following links) |
-N, --timestamping | Only downloads newer files than local copies |
-l inf, --level=inf | Sets the recursion level to infinite |
--no-remove-listing | Keep “.listing” files that are created to keep track of directory listings |
The rest of the used parameters:
| Parameter | Description |
|---|---|
--adjust-extension | Make sure that “.html” is added to the files |
--domains | Comma-separated list of domains to be followed |
--user-agent | Sends the given “User-Agent” header to the server |
Other useful parameters:
| Parameter | Description |
|---|---|
--page-requisites | Downloads things as inlined images, sounds, and referenced stylesheets |
--span-hosts | Allows downloading files from links that point to different hosts |
--convert-links | Converts links to local links (allowing local viewing) |
--no-check-certificate | Bypasses SSL certificate verification. |
--directory-prefix=/my/directory | Sets up the destination directory. |
-e robots=off | Ignores robots.txt instructions. |
Discover more from Coursity
Subscribe to get the latest posts sent to your email.