scrapy 爬取 w3school

scrapy是一个从网站中提取结构化数据爬虫框架，简单易用。

1、安装python pip工具

$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python3 get-pip.py 
$ pip -V
pip 20.2.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

2、安装scrapy，参考：https://github.com/scrapy/scrapy

$ pip install scrapy

$ scrapy startproject w3school

创建一个w3school项目，包含以下内容的目录：

w3school/
 scrapy.cfg 
 w3school/ 
   __init__.py
   items.py 
   middlewares.py 
   pipelines.py 
   settings.py 
 spiders/ 
   __init__.py

scrapy.cfg 存放的目录被认为是 项目的根目录 。该文件中包含python模块名的字段定义了项目的设置。

Item 是保存爬取到的数据的容器；其使用方法和python字典类似， 并且提供了额外保护机制来避免拼写错误导致的未定义字段错误。
定义这个字段定义在items.py文件中。

$ cd w3school
$ vim items.py
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy

class W3SchoolItem(scrapy.Item):
   # define the fields for your item here like:
   # name = scrapy.Field()
   title = scrapy.Field()
   link = scrapy.Field()
   desc = scrapy.Field()
   body = scrapy.Field()
   pass

$ cd spiders
$ scrapy genspider w3school_spider w3school.com.cn

编辑刚刚创建的w3school_spider.py文件

$ vim w3school_spider.py 

import scrapy
from scrapy.selector import Selector
from w3school.items import W3SchoolItem
import requests
import base64
import html
from xml.sax.saxutils import unescape

class W3schoolSpiderSpider(scrapy.Spider):
   name = "w3school_spider"
   allowed_domains = ["w3school.com.cn"]
   start_urls = ["https://www.w3school.com.cn/php/index.asp"]

   def __init__(self):
   self.link = ''

   def parse(self, response):
     # 选择器获取页面源码,
     sel = Selector(response)
     # 使用xparh进行筛选,选取所有div中id为navsecond的层所包含的所有div中id为course的ul中ul标签下的,li标签内容,
     sites = sel.xpath('//div[@id="navsecond"]/div[@id="course"]/ul[1]/li')

     # 定义一个items容器
     items = []
     # site的内容包括href为链接,title为标题,
     for site in sites:
       # 成为ie一个item的字典类型
       item = W3SchoolItem()
       # 对每一个site使用xpath抽取出a标签内的text,href,title.
       title = site.xpath('a/text()').extract()
       link = site.xpath('a/@href').extract()
       desc = site.xpath('a/@title').extract()
       self.link = str(link)
       body = self.body()

       item['title'] = [t.encode('utf-8') for t in title]
       item['link'] = [l.encode('utf-8') for l in link]
       item['desc'] = [d.encode('utf-8') for d in desc]
       item['body'] = [b.encode('utf-8') for b in body]
       # 在列表中加入这个字典
       items.append(item)


     return items

  
   def body(self):

     url = "https://www.w3school.com.cn/"+self.link[2:-2]
     headers = {
       'Content-Type': 'text/html;charset=utf-8',
       'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
     }
     reaponse = requests.get(url, headers=headers)

     bodySel = Selector(reaponse)

     # 提取全部数据
     bodyStr = bodySel.xpath('//div[@id="maincontent"]').extract()

     # 过滤数据
     # bodyStr = bodySel.xpath('//div[@id="maincontent"]')
     # bodyStr.xpath('//div[@id="maincontent"]/div[@id="tpn"]').remove()
     # bodyStr.xpath('//div[@id="maincontent"]/div[@id="bpn"]').remove()
     # bodyStr.xpath('//div[@id="maincontent"]/div[@id="foogz"]').remove()
     # bodyStr.xpath('//div[@id="maincontent"]/div/p[@class="tiy"]').remove()
     # bodyStr.xpath('//div[@id="maincontent"]/div/p[contains(string(), "请点击")]').remove()

     return bodyStr

     # return bodyStr.extract()

选择器Selector
scrapy中可以使用xpath和css从html中选择数据，

  

  

编辑pipelines.py脚本，将取到的item储存在文件中。

$ vim pipelines.py 
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter

import codecs
import json
import base64
from w3school.myEncoding import MyEncoding
from xml.sax.saxutils import unescape

class W3SchoolPipeline(object):
   def __init__(self):
     self.file = codecs.open('w3school_utf-8.json','wb',encoding='utf-8')

   def process_item(self, item, spider):
     # item['body'] = unescape(str(item['body']))
     # item['body'] = base64.b64encode(str(item['body']))
     line = json.dumps(dict(item), cls=MyEncoding) + '\n'

     data = json.loads(line)

     data['body'] = unescape(str(data['body'][0]))

     self.file.write(str(data) + '\n')
 
     return item

   def close_spider(self,spider):
     self.file.close()

七、配置
在settings.py文件中加入下面代码（其实文件中已经有了只是被注释了）

$ vim settings.py

ITEM_PIPELINES = { 
  'w3school.pipelines.W3SchoolPipeline': 300, 
}

scrapy list 列出当前项目中所有可用的spider

在项目下运行一下命令

$ scrapy crawl w3school_spider