Exercise 1: JavaScript-Heavy Site Scraper
Create a Selenium-based scraper that: - Handles infinite scroll pages - Waits for dynamic content to load - Extracts data from JavaScript-rendered elements - Handles pop-ups and modals - Takes screenshots of important pages
Hint: Try scraping a modern SPA (Single Page Application) website.
Exercise 2: High-Performance Async Scraper
Build an async scraper that: - Scrapes 100+ URLs concurrently - Uses connection pooling - Implements rate limiting with semaphores - Handles errors without stopping the entire process - Saves results as they're collected (streaming)
Exercise 3: Production Scrapy Project
Create a complete Scrapy project with: - Multiple spiders for different websites - Custom middleware for request handling - Item pipelines for data validation and storage - Settings for different environments (dev/prod) - Logging and monitoring
Exercise 4: Database-Backed Scraping System
Build a system that: - Stores scraped data in PostgreSQL or MySQL - Tracks scraping history and status - Supports scheduled scraping (cron jobs) - Provides a simple API to query scraped data - Includes data deduplication
Exercise 5: Anti-Bot Bypass
Research and implement techniques to: - Rotate User-Agents - Handle cookies and sessions - Use proxies (if available) - Mimic human behavior (random delays, mouse movements) - Handle CAPTCHAs (using services like 2captcha)
Note: Only use these techniques on websites you have permission to scrape!
Solutions
These exercises are challenging! Take your time and refer to documentation. Solutions demonstrate production-ready patterns.