Tor服务器实现动态Ip的切换访问豆瓣电影

最新推荐文章于 2024-07-01 04:04:03 发布

yitian1585531

最新推荐文章于 2024-07-01 04:04:03 发布

阅读量949

点赞数

CC 4.0 BY-SA版权

分类专栏：爬虫

本文链接：https://blog.csdn.net/yitian1585531/article/details/84342425

爬虫专栏收录该内容

17 篇文章

订阅专栏

本文介绍了一种利用Tor浏览器动态切换IP的方法，通过Python编程实现对豆瓣电影Top250榜单的爬取。该过程涉及了Tor控制器的配置、SOCKS代理设置及网页数据抓取，展示了如何在保护隐私的同时高效地获取目标网站数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

需安装Tor浏览器，动态切换IP

import socket,socks,requests
from stem import Signal
from stem.control import Controller
import time
from lxml import etree

controller = Controller.from_port(port = 9151)
controller.authenticate()
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket

urls = ['https://movie.douban.com/top250?start={}&filter='.format(str(i)) for i in range(0, 250, 25)]
headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36'}
for url in urls:
	a = requests.get("http://checkip.amazonaws.com").text
	print("第n次更新的IP：", a)

	time1 = time.time()
	html = requests.get(url, headers=headers)
	selector = etree.HTML(html.text)
	movie_hrefs = selector.xpath('//div[@class="hd"]/a/@href')
	print(movie_hrefs)
	time2=time.time()
	print('花费的时间：',time2-time1)

	controller.signal(Signal.NEWNYM)  # 跟新ip