Scrapy执行工程下所有爬虫

最新推荐文章于 2024-10-08 12:05:21 发布

别None了

最新推荐文章于 2024-10-08 12:05:21 发布

阅读量915

点赞数

CC 4.0 BY-SA版权

分类专栏：笔记文章标签：爬虫 python

本文链接：https://blog.csdn.net/Leetengteng/article/details/121284991

笔记专栏收录该内容

6 篇文章

订阅专栏

当scrapy项目有多个爬虫时，run.py可以同时启动所有爬虫。

# @Time    : 25/12/2018 5:35 PM
# @Site    :
# @File    : run.py
# @Software: PyCharm

from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess

def main():
    setting = get_project_settings()
    process = CrawlerProcess(setting)
    didntWorkSpider = ['sample']

    for spider_name in process.spiders.list():
        print("Running spider %s" % (spider_name))
        process.crawl(spider_name)
    process.start()