Experience In Building Scalable Web Sites Through Infrastructure's View
- 1. Experience in Building Scalable Web Sites through Infrastructure's Aspect Barcamp Bangkok 3 May 23-24, 2009 Sripathum University
- 2. Gen#1 Web + DB Cost effective for startup
- 3. Gen#2 Web After boom!! DB
- 4. Gen#3 Web More boom!!, add more web servers but DB is the problem DB Web Web
- 5. Gen#4 Web Replication could save your life for a while DB Web Web DB Replication WRITE Query READ ONLY
- 6. Gen#... Memcached: reduce READ query at DB Reverse Proxy: serves static contents DB Web Reverse Proxy DB Replication Memcached Memcached Web Reverse Proxy WRITE Query READ ONLY
- 7. Finally: Horizontal Scaling masterA Reverse Proxy Replication Reverse Proxy Web (dynamic) Web (dynamic) masterB Global Database - Multimaster Slave 1 Slave 2 UserDB1 UserDB2 UserDB3 UserDB..n Memcached
- 8. Behind the Scene Hardware (LB, Server)
- 10. OS
- 11. Web Server
- 12. Caching
- 13. DB
- 14. Infra: dns, network, storage, firewall, security process, log, backup, ntp, logging, source control, monitoring, ...
- 15. Good to Know 80% of the end-user response time is spent on the front-end Ref: developer.yahoo.com
- 16. YAHOO's “Best Practices for Speeding Up Your Web Site” Minimize HTTP requests
- 17. Enable GZIP
- 18. Add “Cache-Control” HTTP header
- 19. Put stylesheet at the top
- 20. Put scripts at the bottom
- 21. ............... more at developer.yahoo.com
- 22. TIPS Enable “keep-alive” at all termination hops
- 23. TIPS Split STATIC contents server Set up new top level domain to reduce cookie in HTTP header
- 24. TIPS No more cookie in HTTP request
- 25. Lot of Things to find out Hardware (LB, Server)
- 27. OS
- 28. Web Server
- 29. Caching
- 30. DB
- 31. Infra: dns, network, storage, firewall, security process, log, backup, ntp, logging, source control, monitoring, ...
- 32. Hardware: Server Any class can be used (blade, rack, local brand)
- 33. DB Server: Raid 1+0, SAS, SCSI, FC (15K)
- 34. Prefer 64 bits
- 35. Battery-backed write cache could improve disk I/O performance
- 36. We have HP BL20p, BL25p, BL460c, DL380 G3 G5, DL360 G3 G5 … Dell PE, Dell R300
- 37. Out of band management is required
- 38. Hardware: Load Balancer Evaluate first (L4 - L7)
- 39. Foundry ServerIron GTE, G4 (now Brocade)
- 40. We tried, Citrix Netscalr
- 41. Radware Modern box comes with HTTP accelerate Crescendo
- 42. OS. 64 Bits
- 43. Choose any os. that you can fully control it
- 44. Monitor your servers CPU, load average, mem, disk, network
- 45. Web Server Apache 2.2 Still use prefork
- 47. Hardening php.ini Lighttpd, Nginx are ok Be careful about FastCGI IIS for some applications
- 48. Web Server (cont.) Keep monitor
- 49. Database MySQL 5.x Use memory table for unimportant data
- 51. InnoDB, difficult to backup PostgreSQL is coming Oracle just acquired SUN (April 2009)
- 53. Database (cont.) Examine all queries (if you can)
- 54. Index hitting for SELECT ?
- 55. Query monitoring
- 56. Database (cont.) Avoid full text searching Use Sphinx
- 57. Use Solr Use EXPLAIN
- 59. Caching Reverse Proxy Squid (designed as Forward proxy)
- 60. Apache Mod_Proxy (never tried)
- 61. Varnish (good response time, only UTF8)
- 62. Storage NAS is painful for scalable engineer
- 63. SAN is not for sharing across servers
- 64. Solutions Put the scripts at local disk (rsync)
- 65. Put only shared folder at NAS Benchmark by iozone
- 66. Caching Caching & Caching
- 67. Clustering Storage [1] Lustre: for HPC
- 69. No redundant Gluster Mountable in userspace
- 70. Slow
- 72. Clustering Storage [2] MogileFS: good performance
- 73. Build on commodity hardware
- 74. Acces via API (php, python, perl) Hadoop Includes Map/Reduce, HDFS, HBase
- 75. May suitable for logging application
- 76. Good for batch processing
- 77. Security Hardening at all components
- 78. Perform security audit if you can
- 80. Try network scan for ports changed
- 82. KB Train the new programmer Never knew about indexing
- 84. Never knew reverse proxy Internal KB (recommend MediaWiki, TWIKI)
- 85. Other Use version control
- 86. Find “one click deployment system”
- 87. All servers must sync their clock
- 88. Centralize log server + Analyzer
- 89. Backup & restore
- 90. Perform URL monitoring from outside & inside
- 91. Try ITIL V3 to improve the quality of IT services