scrapy

• http://doc.scrapy.org/en/1.0/intro/tutorial.html
• http://doc.scrapy.org/en/latest/topics/practices.html
• https://stackoverflow.com/questions/27805952/scrapy-set-depth-limit-per-allowed-domains

Scrapy Shell

Start shell: scrapy shell "http://example.com"

Search:

response.xpath(‘//title’)