Local handler debugging
You can easily test handler without running full-featured server. Here’s how you can do that for the DemoScraper that we have developed in the tutorial.
Add import in the beginning of the file:
from sneakpeek.scraper.runner import ScraperRunner
And add the following lines to the end of the file:
async def main():
result = await ScraperRunner.debug_handler(
DemoScraper(),
config=ScraperConfig(
params=DemoScraperParams(
start_url="https://www.ycombinator.com/",
max_pages=20,
).dict(),
),
middlewares=[
RequestsLoggingMiddleware(),
],
)
logging.info(f"Finished scraper with result: {result}")
if __name__ == "__main__":
asyncio.run(main())
For the argument ScraperRunner.debug_handler takes:
An instance of your scraper handler
Scraper config
[Optional] Middleware that will be used in the handler (see full list of the middleware here)
Now you can run you handler as an ordinary Python script. Given it’s in demo_scraper.py file you can use:
python3 demo_scraper.py