python 定时执行爬虫模块_【Python】定时执行网站爬虫

最新推荐文章于 2024-04-03 15:19:37 发布

原创最新推荐文章于 2024-04-03 15:19:37 发布 · 233 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#python 定时执行爬虫模块

本文介绍如何使用Python与BeautifulSoup爬取Yahoo股市数据，并通过pandas处理数据，利用SQLite存储结果，最后借助crontab实现定时任务。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天我们额讨论如何使用Python，SQLite数据库与crontab工具将爬虫程序部署到服务器上并实现定时爬取存储

编写爬虫代码

编写一个爬虫程序，使用requests与beautifulsoup4包爬取和解析Yahoo!股市-上市成交价排行与Yahoo!股市-上柜成交价排行的资料，再利用pandas包将解析后的展示出来。

import datetime

import requests

from bs4 import BeautifulSoup

import pandas as pd

def get_price_ranks():

current_dt = datetime.datetime.now().strftime("%Y-%m-%d %X")

current_dts = [current_dt for _ in range(200)]

stock_types = ["tse", "otc"]

price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]

tickers = []

stocks = []

prices = []

volumes = []

mkt_values = []

ttl_steps = 10*100

each_step = 10

for pr_url in price_rank_urls:

r = requests.get(pr_url)

soup = BeautifulSoup(r.text, 'html.parser')

ticker = [i.text.split()[0] for i in soup.select(".name a")]

tickers += ticker

stock = [i.text.split()[1] for i in soup.select(".name a")]

stocks += stock

price = [float(soup.find_all("td")[2].find_all("td")[i].text) for i in range(5, 5+ttl_steps, each_step)]

prices += price

volume = [int(soup.find_all("td")[2].find_all("td")[i].text.replace(",", "")) for i in range(11, 11+ttl_steps, each_step)]

volumes += volume

mkt_value = [float(soup.find_all("td")[2].find_all("td")[i].text)*100000000 for i in range(12, 12+ttl_steps, each_step)]

mkt_values += mkt_value

types = ["上市" for _ in range(100)] + ["上柜" for _ in range(100)]

ky_registered = [True if "KY" in st else False for st in stocks]

df = pd.DataFrame()

df["scrapingTime"] = current_dts

df["type"] = types

df["kyRegistered"] = ky_registered

df["ticker"] = tickers

df["stock"] = stocks

df["price"] = prices

df["volume"] = volumes

df["mktValue"] = mkt_values

return df

price_ranks = get_price_ranks()

print(price_ranks.shape)

这个的结果展示为

## (200, 8)

接下来我们利用pandas进行前几行展示

price_ranks.head()

price_ranks.tail()

接下来我们就开始往服务器上部署

对于服务器的选择，环境配置不在本课的讨论范围之内，我们主要是要讲一下怎么去设置定时任务。

接下来我们改造一下代码，改造成结果有sqlite存储。