用Python写网络爬虫 学习总结和代码.zip

# 下载缓存 上篇文章，我们学习了如何提取网页中的数据，以及将提取结果存到表格中。如果我们还想提取另一字段，则需要重新再下载整个网页，这对我们这个小型的示例网站问题不大，但对于数百万个网页的网站而言来说就要消耗几个星期的时间。所以，我们可以先对网页进行缓存，就使得每个网页只下载一次。 # 1为链接爬虫添加缓存支持 - 我们将downloader重构一类，这样参数只需在构造方法中设置一次，就能在后续多次复用，在URL下载之前进行缓存检查，并把限速功能移到函数内部。 - 在Downloader类的__call__特殊方法实现了下载前先检查缓存，如果已经定义该URL缓存则再检查下载中是否遇到了服务端错误，如果都没问题表明缓存结果可用，否则都需要正常下载该URL存到缓存中。 - downloader方法返回添加了HTTP状态码，以便缓存中存储错误机校验。如果不需要限速或缓存的话，你可以直接调用该方法，这样就不会通过__call__方法调用了。 ```python class Downloader: def __init__(self, delay=5, user_agent='Wu_Being', proxies=None, num_retries=1, cache=None): self.throttle = Throttle(delay) self.user_agent = user_agent self.proxies = proxies self.num_retries = num_retries self.cache = cache def __call__(self, url): result = None if self.cache: try: result = self.cache[url] except KeyError: # url is not available in cache pass else: if self.num_retries > 0 and 500 <= result['code'] < 600: # server error so ignore result from cache and re-download result = None if result is None: # result was not loaded from cache so still need to download self.throttle.wait(url) proxy = random.choice(self.proxies) if self.proxies else None headers = {'User-agent': self.user_agent} result = self.download(url, headers, proxy=proxy, num_retries=self.num_retries) if self.cache: # save result to cache self.cache[url] = result return result['html'] def download(self, url, headers, proxy, num_retries, data=None): print 'Downloading:', url ... return {'html': html, 'code': code} class Throttle: def __init__(self, delay): ... def wait(self, url): ... ``` 为了支持缓存功能，链接爬虫代码也需用一些微调，包括添加cache参数、移除限速以及将download函数替换为新的类。 ```python from downloader import Downloader def link_crawler(... cache=None): crawl_queue = [seed_url] seen = {seed_url: 0} # track how many URL's have been downloaded num_urls = 0 rp = get_robots(seed_url) #cache.clear() ############################### D = Downloader(delay=delay, user_agent=user_agent, proxies=proxies, num_retries=num_retries, cache=cache) while crawl_queue: url = crawl_queue.pop() depth = seen[url] # check url passes robots.txt restrictions if rp.can_fetch(user_agent, url): html = D(url) ###def __call__(self, url): links = [] ... def normalize(seed_url, link): ... def same_domain(url1, url2): ... def get_robots(url): ... def get_links(html): ... """ if __name__ == '__main__': link_crawler('http://example.webscraping.com', '/(index|view)', delay=0, num_retries=1, user_agent='BadCrawler') link_crawler('http://example.webscraping.com', '/(index|view)', delay=0, num_retries=1, max_depth=1, user_agent='GoodCrawler') """ ``` 现在，这个支持缓存的网络爬虫的基本架构已经准备好了，下面就要开始构建实际的缓存功能了。 # 2磁盘缓存 |操作系统|文件系统|非法文件名字符|文件名最大长度| |:------:|:------:|:------:|:------:| |Linux|Ext3/Ext4|`/`和`\0`|255个字节| |OS X|HFS Plus|`:`和`\0`|255个UTF-16编码单元| |Windows|NTFS|`\`、`/`、`?`、`:`、`*`、`"`、`>`、`<`和`|`|255个字节| 为了保证在不同文件系统中，我们的文件路径都是安全的，就需要把除数字、字母和基本符号的其他字符替换为下划线。 ```python >>> import re >>> url="http://example.webscraping.com/default/view/australia-1" >>> re.sub('[^/0-9a-zA-Z\-,.;_ ]','_',url) 'http_//example.webscraping.com/default/view/australia-1' ``` 此外，文件名及其目录长度需要限制在255个字符以内。 ```python >>> filename=re.sub('[^/0-9a-zA-Z\-,.;_ ]','_',url) >>> filename='/'.join(segment[:255] for segment in filename.split('/')) >>> print filename http_//example.webscraping.com/default/view/australia-1 >>> print '#'.join(segment[:5] for segment in filename.split('/')) http_##examp#defau#view#austr >>> ``` 还有一种边界情况，就是URL以斜杠结尾。这样分割URL后就会造成一个非法的文件名。例如： - http://example.webscraping.com/index/ - http://example.webscraping.com/index/1 对于第一个URL可以在后面添加index.html作为文件名，所以可以把index作为目录名，1为子目录名，index.html为文件名。 ```python >>> import urlparse >>> components=urlparse.urlsplit('http://exmaple.scraping.com/index/') >>> print components SplitResult(scheme='http', netloc='exmaple.scraping.com', path='/index/', query='', fragment='') >>> print components.path /index/ >>> path=components.path >>> if not path: ... path='/index.html' ... elif path.endswith('/'): ... path+='index.html' ... >>> filename=components.netloc+path+components.query >>> filename 'exmaple.scraping.com/index/index.html' >>> ``` ## 2.1用磁盘缓存的实现 现在可以把URL到目录和文件名完整映射逻辑结合起来，就形成了磁盘缓存的主要部分。该构造方法传入了用于设定缓存位置的参数，然后在url_to_path方法中应用了前面讨论的文件名限制。 ```python from link_crawler import link_crawler class DiskCache: def __init__(self, cache_dir='cache', ...): """ cache_dir: the root level folder for the cache """ self.cache_dir = cache_dir ... def url_to_path(self, url): """Create file system path for this URL """ components = urlparse.urlsplit(url) # when empty path set to /index.html path = components.path if not path: path = '/index.html' elif path.endswith('/'): path += 'index.html' filename = components.netloc + path + components.query # replace invalid characters filename = re.sub('[^/0-9a-zA-Z\-.,;_ ]', '_', filename) # restrict maximum number of characters filename = '/'.join(segment[:255] for segment in filename.split('/')) return os.path.join(self.cache_dir, filename) #拼接当前目录和文件名为完整目录 def __getitem__(self, url): ... def __setitem__(self, url, result): ... def __delitem__(self, url): ... def has_expired(self, timestamp): ... def clear(self): ... if __name__ == '__main__': link_crawler('http://example.webscraping.com/', '/(index|view)', cache=DiskCache()) ``` 现在我们还缺少根据文件名存取数据的方法，就是Downloader类`result=cache[url]`和`cache[url]=result`的接口方法：`__getitem__()`和`__setitem__()`两个特殊方法。 ```python import pickle class DiskCache: def __init__(self, cache_dir='cache', expires=timedelta(days=30), compress=True): ... def url_to_path(self, url): ... def __getitem__(self, url): ... def __setitem__(self, url, result): """Save data to disk for this url """ path = self.url_to_path(url) folder = os.path.dirname(path) if not os.path