在settings文件中添加下面一段
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
就算在爬虫代码中增加了请求头，也要在设置中增加
headers = {
'Host': 'channel.chinanews.com',
'Referer': 'https://www.chinanews.com/',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
}
yield scrapy.Request(url=url, headers=headers, callback=self.parse)
网友评论