«Scrapy internals» Александр Сибиряков, Scrapinghub
1 like453 views
- Scrapy is a framework for web scraping that allows for extraction of structured data from HTML/XML through selectors like CSS and XPath. It provides features like an interactive shell, feed exports, encoding support, and more. - Scrapy is built on top of the Twisted asynchronous networking framework, which provides an event loop and deferreds. It handles protocols and transports like TCP, HTTP, and more across platforms. - Scrapy architecture includes components like the downloader, scraper, and item pipelines that communicate internally. Flow control is needed between these to limit memory usage and scheduling through techniques like concurrent item limits, memory limits, and delays between calls.
![Twisted event loop
https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work
https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html
events:
[e1: Event, e2: Event, … eN]
Event:
func, args, desired_time](https://image.slidesharecdn.com/205-scrapyinternals-170724105004/85/Scrapy-internals-Scrapinghub-33-320.jpg)
![Twisted event loop
https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work
https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html
events:
[e1: Event, e2: Event, … eN]
Event:
func, args, desired_time min: O(1)](https://image.slidesharecdn.com/205-scrapyinternals-170724105004/85/Scrapy-internals-Scrapinghub-34-320.jpg)
«Scrapy internals» Александр Сибиряков, Scrapinghub
- 1. Scrapy internals Alexander Sibiryakov, 16-17 July 2017, PyConRU 2017 [email protected] made by
- 2. Talk scope
- 3. Talk scope • Design of complex asynchronous application,
- 4. Talk scope • Design of complex asynchronous application, • Flow-control issues,
- 5. Talk scope • Design of complex asynchronous application, • Flow-control issues, • open source life.
- 7. Scrapy: web scraping • extraction of structured data,
- 8. Scrapy: web scraping • extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel
- 9. Scrapy: web scraping • extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell,
- 10. Scrapy: web scraping • extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell, • Feed exports in JSON, CSV, XML and storing in FTP, S3, local fs,
- 11. Scrapy: web scraping • extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell, • Feed exports in JSON, CSV, XML and storing in FTP, S3, local fs, • Robust encoding support and auto-detection,
- 12. Main features
- 13. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines,
- 14. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines, Telnet console
- 15. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines, Form submission Telnet console
- 16. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines, COOKIES Form submission Telnet console
- 17. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines, COOKIES Form submission Telnet console Graceful shutdown by signal
- 18. Main features • Extensible: spider, signals, middlewares, extensions, and pipelines, COOKIES Robots.txt Form submission Telnet console Graceful shutdown by signal
- 20. Twisted
- 21. Twisted • Event-driven network programming framework
- 22. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания»)
- 23. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания») • Protocols and transport:
- 24. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets
- 25. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC
- 26. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC • Cross platform
- 27. Twisted • Event-driven network programming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC • Cross platform
- 29. Glyph Lefkowitz Creator of Twisted
- 30. –Twisted source code self._nameResolver = _SimpleResolverComplexifier(resolver)
- 33. Twisted event loop https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html events: [e1: Event, e2: Event, … eN] Event: func, args, desired_time
- 34. Twisted event loop https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html events: [e1: Event, e2: Event, … eN] Event: func, args, desired_time min: O(1)
- 35. x86 time sources
- 36. x86 time sources • Real Time Clock - absolute time, 1 sec. precision,
- 37. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously,
- 38. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz
- 39. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode,
- 40. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode,
- 41. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target
- 42. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target • RDTSC/RDTSCP - CPU clock cycles
- 43. x86 time sources • Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target • RDTSC/RDTSCP - CPU clock cycles • Proprietary timers
- 44. Twisted.Deferred
- 47. Twisted.Deferred • callback • errback • addCallback, addErrback
- 48. Twisted.Deferred • callback • errback • addCallback, addErrback • cancel
- 49. Twisted.Deferred • callback • errback • addCallback, addErrback • cancel • addTimeout
- 50. Twisted.Deferred • callback • errback • addCallback, addErrback • cancel • addTimeout • pause/unpause
- 54. PROBLEMS
- 56. Throttling between internal components • Downloader,
- 57. Throttling between internal components • Downloader, • Scraper
- 58. Throttling between internal components • Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..)
- 59. Throttling between internal components • Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..) • Feed exports (serialization + disk/network IO)
- 60. Throttling between internal components • Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..) • Feed exports (serialization + disk/network IO) • ?
- 63. Flow control: memory • Unlimited downloading -> unlimited items growth from cascading feed pages.
- 64. Flow control: memory • Unlimited downloading -> unlimited items growth from cascading feed pages. • maintain limit per amount of memory used for Responses in queue (~5Mb)
- 65. Flow control: CPU spending more time on than > reactor.callLater( 0.1 , d.errback, _failure) an artificial delay in 100ms Callbacks-> CPU io
- 66. Summarizing
- 67. Summarizing • concurrent items limits,
- 68. Summarizing • concurrent items limits, • memory consumption limits,
- 69. Summarizing • concurrent items limits, • memory consumption limits, • scheduling of new calls with delays.
- 70. Summarizing • concurrent items limits, • memory consumption limits, • scheduling of new calls with delays. if limit is reached ->
- 71. Summarizing • concurrent items limits, • memory consumption limits, • scheduling of new calls with delays. if limit is reached -> don’t pickup new request from scheduler
- 72. It just stopped…
- 73. It just stopped… • Why?
- 74. It just stopped… • Why? • Some Deferred was lost?
- 75. It just stopped… • Why? • Some Deferred was lost? • Where?
- 76. It just stopped… • Why? • Some Deferred was lost? • Where? • How to debug?
- 77. It just stopped… • Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet.
- 78. It just stopped… • Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet. > self.heartbeat = task.LoopingCall(nextcall.schedule)
- 79. It just stopped… • Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet. > self.heartbeat = task.LoopingCall(nextcall.schedule) + extensive logging
- 81. Design your async application well
- 82. Design your async application well Iterations
- 83. Design your async application well Iterations State diagrams