实战项目五：抓取简书文章信息

最新推荐文章于 2025-05-18 20:45:56 发布

K同学啊

最新推荐文章于 2025-05-18 20:45:56 发布

阅读量483

点赞数 1

CC 4.0 BY-SA版权

分类专栏：爬虫 python 文章标签： python 爬虫简书

本文链接：https://blog.csdn.net/qq_38251616/article/details/86819881

python 同时被 2 个专栏收录

77 篇文章

订阅专栏

爬虫

18 篇文章

订阅专栏

本文介绍了一种使用Python的fake_useragent和lxml库抓取简书网站文章信息的方法，包括文章标题、作者昵称、评论数和点赞数。通过分析网页结构，利用XPath表达式定位所需数据，实现了自动化信息抓取。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

源码：

from fake_useragent 			import UserAgent
from lxml 						import etree
import lxml,requests


url="https://www.jianshu.com/c/qqfxgN?utm_campaign=haruki&utm_content=note&utm_medium=reader_share&utm_source=qq"

def getHtml(url):
	'''
	获取网页源码
	return html
	'''
	headers = {
		"Host": "www.jianshu.com",
		"Referer": "https://www.jianshu.com/",
		"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"
	}

	req = requests.get(url, headers=headers)

	html = etree.HTML(req.text)

	return html

def parse(html):
	'''
	解析网页
	'''
	nodes = html.xpath("//ul[@class='note-list']/li//div[@class='content']")

	for node in nodes:

		title = node.xpath(".//a[@class='title']/text()")[0]
		nickname = node.xpath(".//div[@class='meta']/a/text()")[0]
		comment = node.xpath(".//div[@class='meta']/a//text()")[2].strip()
		like = node.xpath(".//div[@class='meta']/span/text()")[0].strip()

		essay = {
			"title" : title,
			"nickname" : nickname,
			"comment" : comment,
			"like" : like
		}

		print("文章信息：{}".format(essay))

def main():
	html = getHtml(url)
	parse(html)

if __name__ == '__main__':
	main()