Scrapy Send Stats To A Url Passed As Argument As A Post Request Every 5 Minutes

I need to send the crawler stats to a URL which is passed on as a spider argument. I need to make a POST request at regular intervals of 5 minutes. How can I do that?

Solution 1:

You will probably want to write an extension that simply makes a post request every 5 minutes. You can make these requests either using scrapy's own mechanisms (e.g. engine.download()), or you can use a different async http client (e.g. treq)

If you're not sure how to structure your extension, you can take a look at logstats.py which does a similar thing, except not logging over http.

Since you're writing an extension anyway, I'd recommend making the url and interval settings, but that choice is up to you.