【菜鸟学Python】使用面向对象以及Xpath爬取猫眼电影排行100

最新推荐文章于 2023-03-06 18:06:32 发布

weixin_30611509

最新推荐文章于 2023-03-06 18:06:32 发布

阅读量404

点赞数

CC 4.0 BY-SA版权

文章标签： python json

原文链接：http://www.cnblogs.com/liduo0413/p/11513092.html

本文介绍了一个使用Python的面向对象方法及Xpath技术爬取猫眼电影排行榜前100名电影信息的实例。通过requests库获取网页数据，lxml.etree解析HTML，最后将结果以JSON格式保存到本地文件。此教程详细展示了如何构造请求头、发送请求、解析响应并存储数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

使用面向对象以及Xpath爬取猫眼电影排行100

import time
import json
import requests
from lxml import etree


class MaoYanTop100(object):
    def __init__(self):
        self.url = 'https://maoyan.com/board/4'
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
        }

    def get_data(self, url, param):
        response = requests.get(url, params=param, headers=self.headers, verify=False)
        print(response.request.url)
        return response.text

    def parse_content(self, text):
        dom = etree.HTML(text)
        return dom

    def parse(self, dom):
        nodes = dom.xpath('//dd')
        for node in nodes:
            yield {
                'title': node.xpath('.//p/a/@title')[0],
                'actor': node.xpath('.//p[@class="star"]/text()')[0].strip(),
                'time': node.xpath('.//p[@class="releasetime"]/text()')[0]
            }

    def save(self, f, item):
        f.write(json.dumps(item, indent=2, ensure_ascii=False) + ',\n')

    def run(self):
        page = int(input('请输入页码：'))
        with open('maoyan.json', 'w') as f_obj:
            for i in range(page):
                param = {
                    'start': i * 10
                }
                text = self.get_data(self.url, param)
                time.sleep(1)
                dom = self.parse_content(text)
                my_generator = self.parse(dom)
                while True:
                    try:
                        # item_list = []
                        item = next(my_generator)
                        # item_list.append(item)
                        self.save(f_obj, item)
                    except:
                        break
                print(f'第{i + 1}页数据已保存完成')


if __name__ == '__main__':
    maoyan = MaoYanTop100()
    maoyan.run()