User Agent injector
This middleware automatically adds User-Agent
header if it’s not present.
It uses fake-useragent in order to generate fake real world user agents.
Configuration of the middleware is defined in UserAgentInjecterMiddlewareConfig
.
How to configure middleware for the SneakpeekServer
(will be used globally for all requests):
from sneakpeek.middleware.user_agent_injecter_middleware import UserAgentInjecterMiddleware, UserAgentInjecterMiddlewareConfig
server = SneakpeekServer.create(
...
middleware=[
UserAgentInjecterMiddleware(
UserAgentInjecterMiddlewareConfig(
use_external_data = True,
browsers = ["chrome", "firefox"],
)
)
],
)
How to override middleware settings for a given scraper:
from sneakpeek.middleware.user_agent_injecter_middleware import UserAgentInjecterMiddlewareConfig
scraper = Scraper(
...
config=ScraperConfig(
...
middleware={
"user_agent_injecter": UserAgentInjecterMiddlewareConfig(
use_external_data = False,
browsers = ["chrome", "firefox"],
)
}
),
)