Crawl Lab

Toolkit to make your crawlers and scrapers suffer 😈

Status Code Pages

JavaScript Rendered Pages

/js/inline - Inline JavaScript renders the page content
/js/external - External JavaScript file renders the page content
/js/image.png - JavaScript builds an image element dynamically

Special Pages

/long-response?responseAfter=5 - Delayed response after specified seconds
/duplicates/1 - First duplicate page content
/duplicates/2 - Second duplicate page (identical content)
/redirect/temporary-to-200 - 302 redirect to /200
/redirect/permanent-to-200 - 301 redirect to /200
/redirect/cycle-a - 302 redirect to cycle-b (infinite cycle A → B → A)
/redirect/cycle-b - 302 redirect to cycle-a (infinite cycle B → A → B)
/headers - Echo request headers
/uuid - Random UUID generator
/random - Random text content on every request
/empty - 204 No Content response

Content Type Pages

/markdown - Content-Type: text/markdown
/json - Content-Type: application/json
/xml - Content-Type: application/xml
/html - Content-Type: text/html (unstyled)
/text - Content-Type: text/plain
/csv - Content-Type: text/csv
/tsv - Content-Type: text/tab-separated-values

Forum Pages

/forum?page=1 - Forum discussion page 1
/forum?page=2 - Forum discussion page 2
/forum?page=10 - Forum discussion page 10 of 20
/forum?page=20 - Forum discussion page 20 of 20
/forum/shop/1 - Book shop discussion thread 1
/forum/shop/2 - Book shop discussion thread 2
/forum/shop/3 - Book shop discussion thread 3

Size Pages

/100Kb - Proxy instruction for 100KB external file
/1Mb - Proxy instruction for 1MB external file
/10Mb - Proxy instruction for 10MB external file

Asset Pages

/pdf - Serve PDF document (public/files/pdf/sample.pdf)
/simple.pdf - Alias for /pdf
/image.png - Serve PNG image (public/images/sample.png)
/js/image.png - JavaScript-rendered image page

All responses aim to provide 200+ characters for scraper validation and have no styling.

This project is open source on Github