User Agent injector

This middleware automatically adds User-Agent header if it’s not present. It uses fake-useragent in order to generate fake real world user agents.

Configuration of the middleware is defined in UserAgentInjecterMiddlewareConfig.

How to configure middleware for the SneakpeekServer (will be used globally for all requests):

from sneakpeek.middleware.user_agent_injecter_middleware import UserAgentInjecterMiddleware, UserAgentInjecterMiddlewareConfig

server = SneakpeekServer.create(
    ...
    middleware=[
        UserAgentInjecterMiddleware(
            UserAgentInjecterMiddlewareConfig(
                use_external_data = True,
                browsers = ["chrome", "firefox"],
            )
        )
    ],
)

How to override middleware settings for a given scraper:

from sneakpeek.middleware.user_agent_injecter_middleware import UserAgentInjecterMiddlewareConfig

scraper = Scraper(
    ...
    config=ScraperConfig(
        ...
        middleware={
            "user_agent_injecter": UserAgentInjecterMiddlewareConfig(
                use_external_data = False,
                browsers = ["chrome", "firefox"],
            )
        }
    ),
)