Oleg Natalushko. Drupal server anatomy. DrupalCamp Kyiv 2011
1 like•1,441 views
The document is a presentation by IT Patrol Inc. discussing the anatomy of a Drupal server, covering topics such as choosing hosting providers, dedicated servers, and potential overload reasons including bot indexing and DDoS attacks. It also lists various bots and offers guidelines for server selection, including CPU, memory, and RAID types. Additionally, it provides insights into performance issues related to Drupal modules and themes.
![Bots Access Log Example
*** [11/Apr/2011:03:39:09 +0400] ******* Googlebot/2.1
*** [11/Apr/2011:03:39:10 +0400] ******* bingbot/2.0
*** [11/Apr/2011:03:39:10 +0400] ******* Googlebot/2.1
*** [11/Apr/2011:03:39:10 +0400] ******* YandexBot/3.0
*** [11/Apr/2011:03:39:10 +0400] ******* msnbot/2.0b
*** [11/Apr/2011:03:39:11 +0400] ******* YandexBlogs/0.99
*** [11/Apr/2011:03:39:12 +0400] ******* Yahoo! Slurp
*** [11/Apr/2011:03:39:12 +0400] ******* YandexAntivirus/2.0](https://image.slidesharecdn.com/olegnatalushko-drupalserveranatomy-drupalcampkyiv2011-110609095745-phpapp01/85/Oleg-Natalushko-Drupal-server-anatomy-DrupalCamp-Kyiv-2011-6-320.jpg)
![List of Bots that is NOT important
Casper Bot Search Alexibot Art-Online asterias BackDoorbot Black.Hole BlackWidow BlowFish
botALot BuiltbotTough Bullseye BunnySlippers Cegbfeieh Cheesebot CherryPicker ChinaClaw
CopyRightCheck cosmos Crescent Custo DISCo DittoSpyder DownloadsDemon eCatch
EirGrabber EmailCollector EmailSiphon EmailWolf EroCrawler ExpresssWebPictures
ExtractorPro EyeNetIE FlashGet Foobot FrontPage GetRight GetWeb! Go-Ahead-Got-It Go!
Zilla GrabNet Grafula Harvest hloader HMView httplib HTTrack humanlinks ImagesStripper
ImagesSucker IndysLibrary InfonaviRobot InterGET Internet Ninja Jennybot JetCar JOC Web
Spider Kenjin.Spider Keyword.Density larbin LeechFTP Lexibot libWeb/clsHTTP
LinkextractorPro LinkScan/8.1a.Unix LinkWalker lwp-trivial Mass Downloader Mata.Hari
Microsoft.URL MIDown tool MIIxpc Mister.PiX Mister PiX moget Mozilla/3.Mozilla/2.01
Mozilla.*NEWT Navroad NearSite NetAnts NetMechanic NetSpider Net Vampire NetZIP
NICErsPRO NPbot Octopus Offline.Explorer Offline Explorer Offline Navigator Openfind
Pagerabber Papa Foto pavuk pcBrowser Program Shareware 1 ProPowerbot/2.14
ProWebWalker ProWebWalker psbot/0.1 QueryN.Metasearch ReGet RepoMonkey RMA
SiteSnagger SlySearch SmartDownload Spankbot spanner Superbot SuperHTTP Surfbot
suzuran Szukacz/1.4 tAkeOut Teleport Teleport Pro Telesoft The.Intraformant TheNomad
TightTwatbot Titan toCrawl/UrlDispatcher toCrawl/UrlDispatcher True_Robot turingos
Turnitinbot/1.5 URLy.Warning VCI VoidEYE WebAuto WebBandit WebCopier WebEMailExtrac.*
WebEnhancer WebFetch WebGo IS Web.Image.Collector Web Image Collector WebLeacher
WebmasterWorldForumbot WebReaper WebSauger Website eXtractor Website.Quester Website
Quester Webster.Pro WebStripper Web Sucker WebWhacker WebZip Wget Widow
[Ww]eb[Bb]andit WWW-Collector-E WWWOFFLE Xaldon WebSpider Xenu's Zeus](https://image.slidesharecdn.com/olegnatalushko-drupalserveranatomy-drupalcampkyiv2011-110609095745-phpapp01/85/Oleg-Natalushko-Drupal-server-anatomy-DrupalCamp-Kyiv-2011-7-320.jpg)
![Bots Blocking Rule
in .htaccess
#block bots
RewriteCond %{HTTP_USER_AGENT} Aport [OR]
RewriteCond %{HTTP_USER_AGENT} Ezooms [OR]
RewriteCond %{HTTP_USER_AGENT} Yahoo [OR]
RewriteCond %{HTTP_USER_AGENT} TwitterFeed [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} rss [OR]
RewriteCond %{HTTP_USER_AGENT} YandexBot [OR]
RewriteCond %{HTTP_USER_AGENT} bingbot
RewriteRule ^.* - [F,L]](https://image.slidesharecdn.com/olegnatalushko-drupalserveranatomy-drupalcampkyiv2011-110609095745-phpapp01/85/Oleg-Natalushko-Drupal-server-anatomy-DrupalCamp-Kyiv-2011-8-320.jpg)
Oleg Natalushko. Drupal server anatomy. DrupalCamp Kyiv 2011
- 1. Drupal Server Anatomy Presented by IT Patrol inc.
- 2. Gold Sponsor of DrupalCamp Kyiv 2011
- 3. Silver Sponsors of DrupalCamp Kyiv 2011
- 4. Plan • Should I look for another hosting company or it is time for a dedicated server • How to choose hosting company for dedicated server • How to choose dedicated server for your needs • What next ?
- 5. Possible “Overload” Reasons • Bots indexing your website • DDOS attack • Habr-effect • Drupal modules with known performance issue • The problem from theme part of drupal • Your website actually grew up and need more resources
- 6. Bots Access Log Example *** [11/Apr/2011:03:39:09 +0400] ******* Googlebot/2.1 *** [11/Apr/2011:03:39:10 +0400] ******* bingbot/2.0 *** [11/Apr/2011:03:39:10 +0400] ******* Googlebot/2.1 *** [11/Apr/2011:03:39:10 +0400] ******* YandexBot/3.0 *** [11/Apr/2011:03:39:10 +0400] ******* msnbot/2.0b *** [11/Apr/2011:03:39:11 +0400] ******* YandexBlogs/0.99 *** [11/Apr/2011:03:39:12 +0400] ******* Yahoo! Slurp *** [11/Apr/2011:03:39:12 +0400] ******* YandexAntivirus/2.0
- 7. List of Bots that is NOT important Casper Bot Search Alexibot Art-Online asterias BackDoorbot Black.Hole BlackWidow BlowFish botALot BuiltbotTough Bullseye BunnySlippers Cegbfeieh Cheesebot CherryPicker ChinaClaw CopyRightCheck cosmos Crescent Custo DISCo DittoSpyder DownloadsDemon eCatch EirGrabber EmailCollector EmailSiphon EmailWolf EroCrawler ExpresssWebPictures ExtractorPro EyeNetIE FlashGet Foobot FrontPage GetRight GetWeb! Go-Ahead-Got-It Go! Zilla GrabNet Grafula Harvest hloader HMView httplib HTTrack humanlinks ImagesStripper ImagesSucker IndysLibrary InfonaviRobot InterGET Internet Ninja Jennybot JetCar JOC Web Spider Kenjin.Spider Keyword.Density larbin LeechFTP Lexibot libWeb/clsHTTP LinkextractorPro LinkScan/8.1a.Unix LinkWalker lwp-trivial Mass Downloader Mata.Hari Microsoft.URL MIDown tool MIIxpc Mister.PiX Mister PiX moget Mozilla/3.Mozilla/2.01 Mozilla.*NEWT Navroad NearSite NetAnts NetMechanic NetSpider Net Vampire NetZIP NICErsPRO NPbot Octopus Offline.Explorer Offline Explorer Offline Navigator Openfind Pagerabber Papa Foto pavuk pcBrowser Program Shareware 1 ProPowerbot/2.14 ProWebWalker ProWebWalker psbot/0.1 QueryN.Metasearch ReGet RepoMonkey RMA SiteSnagger SlySearch SmartDownload Spankbot spanner Superbot SuperHTTP Surfbot suzuran Szukacz/1.4 tAkeOut Teleport Teleport Pro Telesoft The.Intraformant TheNomad TightTwatbot Titan toCrawl/UrlDispatcher toCrawl/UrlDispatcher True_Robot turingos Turnitinbot/1.5 URLy.Warning VCI VoidEYE WebAuto WebBandit WebCopier WebEMailExtrac.* WebEnhancer WebFetch WebGo IS Web.Image.Collector Web Image Collector WebLeacher WebmasterWorldForumbot WebReaper WebSauger Website eXtractor Website.Quester Website Quester Webster.Pro WebStripper Web Sucker WebWhacker WebZip Wget Widow [Ww]eb[Bb]andit WWW-Collector-E WWWOFFLE Xaldon WebSpider Xenu's Zeus
- 8. Bots Blocking Rule in .htaccess #block bots RewriteCond %{HTTP_USER_AGENT} Aport [OR] RewriteCond %{HTTP_USER_AGENT} Ezooms [OR] RewriteCond %{HTTP_USER_AGENT} Yahoo [OR] RewriteCond %{HTTP_USER_AGENT} TwitterFeed [OR] RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR] RewriteCond %{HTTP_USER_AGENT} rss [OR] RewriteCond %{HTTP_USER_AGENT} YandexBot [OR] RewriteCond %{HTTP_USER_AGENT} bingbot RewriteRule ^.* - [F,L]
- 9. DDOS Attack
- 12. Bad Code Example
- 13. ?
- 14. Should I look for other hosting provider? < 1000 visits/day < 5 req/min, < 1000 nodes < 30 req/min, < 1000 nodes, 90% bots
- 15. ?
- 16. Important Parameters of the Data Center Network infrastructure, speed guarantees, UPS, backup power, diesel generators, Guarantees replacement time, The qualification of the technical support.
- 17. ?
- 18. Checklist for Choosing Didicated Server CPU Memory Hard drive Bandwidth
- 19. HDD Types SATA II Velociraptor SAS SCSI SSD
- 20. RAID Types RAID0 RAID1 RAID5 RAID10
- 21. Architecture of Our HDD Subsystem RAID 1 RAID 5 300 GB SAS 1 TB 1 TB SATA II SATA II 300 GB 1 TB 1 TB SAS SATA II SATA II
- 22. We are using
- 23. NGINX and Apache interrelation diagram
- 25. How “opcode cache” works
- 26. “OpCode cacher” performance diagram
- 29. Any questions? Corporate website: http://it-patrol.ru Drupal Hosting: http://dh.it-patrol.ru Contact email: [email protected]