Scrapy Framework

Scrapy is a Python application framework for crawling websites and extracting structured data. It can be used for data mining, information processing, historical data storage, and more. It was originally designed for web scraping but can also be used to extract data from APIs (Web Services). Scrapy also supports advanced scraping scenarios such as authentication, content analysis, deduplication, and distributed crawling.

Reference: Scrapy documentation

Installation

Scrapy requires Python 3.6+ (CPython or PyPy 7.2.0+).

Linux:

pip install scrapy

Windows:

pip install wheel
# Download the Twisted wheel for your Python version from:
# http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
pip install Twisted-17.1.0-cp35-cp35m-win_amd64.whl
pip install pywin32
pip install scrapy

Basic Usage

# Create a new Scrapy project
scrapy startproject tutorial

# Navigate into the project directory
cd tutorial

# Generate a spider (must be created inside the spiders/ folder)
scrapy genspider spiderName www.example.com

# Run the spider
scrapy crawl spiderName

Recommended settings.py Configuration

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'
ROBOTSTXT_OBEY = False   # disable robots.txt compliance
LOG_LEVEL = 'ERROR'      # suppress verbose logs
CONCURRENT_REQUESTS = 32

Project Structure

myproject/
├── scrapy.cfg
└── myproject/
    ├── items.py          # data model definitions
    ├── middlewares.py    # spider and downloader middlewares
    ├── pipelines.py      # data storage pipeline
    ├── settings.py       # global configuration
    ├── __init__.py
    └── spiders/
        ├── __init__.py
        └── myspider.py   # the actual spider

Spider File Structure (`spiders/bili.py`)

import scrapy

class BiliSpider(scrapy.Spider):
    name = 'bili'              # unique identifier for this spider
    # allowed_domains = ['search.bilibili.com']  # restrict to this domain
    start_urls = [
        'https://search.bilibili.com/all?keyword=dance',
    ]

    def parse(self, response):
        # response is a Scrapy Response object
        li_list = response.xpath('//*[@id="all-list"]/div[1]/div[2]/ul/li')
        all_data = []
        for item in li_list:
            title = item.xpath('./a/@title')[0].extract()
            video_url = 'https:' + item.xpath('./a/@href')[0].extract()
            all_data.append({'title': title, 'url': video_url})
            print({'title': title, 'url': video_url})
        return all_data

Data Persistence

Method 1: Command-line Output

The simplest way — saves the parse() method’s return value directly to a file:

scrapy crawl spiderName -o output.json
scrapy crawl spiderName -o output.csv

Supported formats: json, jsonlines, jl, csv, xml, marshal, pickle.

Pros: simple and fast. Cons: limited flexibility — only saves what parse() returns.

Method 2: Pipeline-based Persistence

The recommended approach for production use:

**Step 1 — Parse data in the spider** and yield `Item` objects instead of returning them: ```python # spiders/myspider.py import scrapy from myproject.items import VideoItem class BiliSpider(scrapy.Spider): name = 'bili' start_urls = ['https://search.bilibili.com/all?keyword=dance'] def parse(self, response): for li in response.xpath('//ul/li'): item = VideoItem() item['title'] = li.xpath('./a/@title').get() item['url'] = 'https:' + li.xpath('./a/@href').get() yield item # submit the item to the pipeline ``` **Step 2 — Define the item fields** in `items.py`: ```python import scrapy class VideoItem(scrapy.Item): title = scrapy.Field() url = scrapy.Field() ``` **Step 3 — Write the pipeline** in `pipelines.py`: ```python class MyProjectPipeline: def open_spider(self, spider): self.f = open('videos.json', 'w', encoding='utf-8') def process_item(self, item, spider): import json line = json.dumps(dict(item), ensure_ascii=False) + '\n' self.f.write(line) return item # pass the item to the next pipeline (if any) def close_spider(self, spider): self.f.close() ``` **Step 4 — Enable the pipeline** in `settings.py`: ```python ITEM_PIPELINES = { 'myproject.pipelines.MyProjectPipeline': 300, # 'myproject.pipelines.MysqlPipeline': 301, # priority 301 runs after 300 } ```

The number (300, 301, etc.) is the pipeline priority — lower numbers run first. Each pipeline’s process_item must return item to pass the item to the next pipeline in the chain.

When to use multiple pipelines: when you want to store a copy of the data in multiple destinations (e.g., both a JSON file and a MySQL database), define one pipeline class per storage target.

Last updated on 2026-06-23

Data Parsing Selenium