scrapy-redis 使 redis 不止保存 url（例如：json）

最新推荐文章于 2024-05-22 16:20:08 发布

「已注销」

最新推荐文章于 2024-05-22 16:20:08 发布

阅读量507

点赞数

CC 4.0 BY-SA版权

文章标签： redis scrapy json 数据库缓存

全网优质文章转载收藏，均不代表本人立场！

本文链接：https://blog.csdn.net/lyshark_lyshark/article/details/125848387

本文介绍了如何使用scrapy-redis通过Redis存储不仅仅是URL，还包含JSON数据。通过两种方法实现：1. 重写`make_request_from_data`方法，将JSON解析并转化为请求；2. 重写`next_requests`方法。提供了相关教程链接和源码分析。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scrapy-redis 使 redis 不止保存 url：https://my.oschina.net/u/4382439/blog/3712637

Scrapy-redis 和 Scrapyd 用法详解：https://zhuanlan.zhihu.com/p/44564597

Scrapy-redis GitHub 地址：https://github.com/rmax/scrapy-redis

scrapy-redis中url队列类型的控制(zset、list)：https://blog.csdn.net/ryuhfxz/article/details/85782467

先看 scrapy-redis 源码 ( spider.py ):

from scrapy import signals
from scrapy.exceptions import DontCloseSpider
from scrapy.spiders import Spider, CrawlSpider

from . import connection, defaults
from .utils import bytes_to_str


class RedisMixin(object):
    """Mixin class to implement reading urls from a redis queue."""
    redis_key = None
    redis_batch_size = None
    redis_encoding = None

    # Redis client placeholder.
    server = None

    def start_requests(self):
        """Returns a batch of start requests from redis."""
        return self.next_requests()

    def setup_redis(self, crawler=None):
        """Setup redis connection and idle signal.

        This should be called after the spider has set its crawler object.
        """
        if self.server is not None:
            return

        if crawler is None:
            # We allow optional crawler argument to keep backwards
            # compatibility.
            # XXX: Raise a deprecation warning.
            crawler = getattr(self, 'crawler', None)

        if crawler is None:
            raise ValueError("crawler is required")

        settings = crawler.settings

        if self.redis_key is None:
            self.redis_key = settings.get(
                'REDIS_START_URLS_KEY', defaults.START_URLS_KEY,
            )

        self.redis_key = self.redis_key % {'name': self.name}

        if not self.redis_key.strip():
            raise ValueError("redis_key must not be empty")

        if self.redis_batch_size is None:
            # TODO: Deprecate this setting (REDIS_START_URLS_BATCH_SIZE).
            self.redis_batch_size = settings.getint(
                'REDIS_START_URLS_BATCH_SIZE',
                settings.getint('CONCURRENT_REQUESTS'),
            )

        try:
            self.redis_batch_size = int(self.redis_batch_size)
        except (TypeError, ValueError):
            raise ValueError("redis_batch_size must be an integer")

        if self.redis_encoding is None:
            self.redis_encoding = settings.get('REDIS_ENCODING', defaults.REDIS_ENCODING)

        self.logger.info("Reading start URLs from redis key '%(redis_key)s' "
                         "(batch size: %(redis_batch_size)s, encoding: %(redis_encoding)s",
                         self.__dict__)

        self.server = connection.from_settings(crawler.settings)
        # The idle signal is called when the spider has no requests left,
        # that's when we will schedule new requests from redis queue
        crawler.signals.connect(self.spider_idle, signal=signals.spider_idle)

    def next_requests(self):
        """Returns a request to be scheduled or none."""
        use_set = self.settings.getbool('REDIS_START_URLS_AS_SET', defaults.START_URLS_AS_SET)
        fetch_one = self.server.spop if use_set else self.server.lpop
        # XXX: Do we need to use a timeout here?
        found = 0
        # TODO: Use redis pipeline execution.
        while found < self.redis_batch_size:
            data = fetch_one(self.redis_key)
            if not data:
                # Queue empty.
                pri