Apache Mesos Distributed Computing Talk
- 1. Apache Mesos A romantic stroll into Distributed Computing Brandon Gulla - Praxis Engineering
- 2. ● Beer. ● Why distributed-computing is hard? *** ● How Mesos helps. ● Mesos in the wild. ● DCOS-Demo. ● Questions? ● Repeat Step 1 This is a shortened version of the talk I normally give, feel free to contact me offline for a deeper dive. Agenda
- 3. $ > whoami Brandon Gulla Distributed DevOps Engineer [email protected] github/bgulla
- 4. Mesosphere ● Commercial company supporting Apache Mesos based in SF and Hamburg, Germany. ● Founded by the Apache Mesos Founder Ben Hindman, Former AirBnB CTO Tobi Knapp and previous Twitter Engineer Florian Leibert. ● Provides full stack Mesos solutions running Apache Mesos as well as their corporate product: DCOS (Datacenter Operating System). ● Keith McClellan will be presenting a demo as soon as you’re tired of me.
- 6. ● Servers, Memory and Disks have gotten cheaper so we buy more. These often go under-utilized. ● Traditionally, clusters have been provisioned statically. ● Puppet, SaltStack, Ansible, **pdsh** :( ● Not Multi-tenant friendly. ● DevOps Nightmare!!! ● Doesn’t Scale :( “Just Network them together!”
- 7. Personal Gateway To Mesos ● Customer had a 100+ node Hadoop cluster with legacy dependency requirements. ● Wanted to utilize current MapReduce libraries without breaking ingest requirements. ● More importantly: Customer wanted a way to manage their resources on demand...
- 8. Solution:
- 9. ● Open-source two-level Resource Manager. ● Written in C++ wrapping around libprocess. ● Scalable - Runs on 10,000s of servers. ● Fault-Tolerant! Multiple masters. Load Balancing! ● Solve Bigger Problems. ● I/O Isolation via Linux cgroups. ● Distributed Resource Manager: ● RAM ● Disks ● Ports ● GPUs ● Network (Yes- Software Defined Networking) Welcome to Apache Mesos
- 10. Mesos: Turn Your Datacenter into One Large Computer ● Where did we install MySQL? Who cares! Mesos + Service Discovery = Dynamic Awesomeness. ● Separates dependencies between Ops and Development. Ops wants to take a rack down, who cares, just move the processes from that rack somewhere else. Ops can do this as part of their work. Developers shouldn't know/care. ● Fault Tolerant, if something dies, Mesos will automatically spin another one up.
- 11. History
- 12. Shared Resources = Multi-Tenancy
- 13. Mesos Explained
- 15. Resource Offers and Launching A Task
- 16. Writing your own Mesos Framework ● Feature Complete API using: ● JVM Languages (Java, Scala, Groovy) ● C++ ● Python ● Go ● Write a framework in minutes using Mesos- Compose. ● YML interface ● Similar to Docker-Compose ● Unit Testing with Mini-Mesos
- 17. Don’t want to write your own Mesos Framework? No problemo, Amigo.
- 18. 0 to Mesos in seconds with Marathon.
- 19. Containers: VMs but awesomer. ● Shares Kernel with Host (no hypervisor) ● Boots in milliseconds. ● Portable. docker pull python:27 ● Run CentOS on Debian, Run Gentoo on Alpine. ● Dependency Friendly
- 20. ● If you’re replacing a once statically installed, non-distributed application: Use Marathon. ● Apache, Nginx, Jenkins..etc ● If your application requires fine grain control and orchestration of tasks: Write your own Framework. ● Apache Accumulo ● Hadoop Yarn (Shameless Apache Myriad Plug) ● Spark Framework Development: My Experiences
- 21. Mesos in the Wild ● Netflix: Mantis+Fenzo ● Apple: Jarvis (Siri) ● Microsoft Azure Container Service.
- 22. Mesos: Full Stack Replacement Current Mesos Equivalent Systemd Marathon Cron Chronos DNS Mesos-DNS Chroot / BSD Jail Docker NFS Gluster, CEF, HDFS
- 25. Mesos Frameworks
- 26. Demo Time To Infinity and Beyond.