DockerCon US 2016 - Scaling Open Source operations
4 likes3,305 views
This document discusses scaling open source operations at Docker. It covers three main areas: the people involved in open source projects including users, contributors and maintainers; the processes for code reviews, design decisions, and managing documentation; and the tooling for measuring activity and automating processes. Maintainers play a key role in reviewing contributions and improving infrastructure. Culture is important for a healthy community. Processes aim to balance contributor experience and code quality. Metrics and automation tools like webhooks help scale projects.
DockerCon US 2016 - Scaling Open Source operations
- 1. Scaling Open Source Operations Arnaud Porterie Sr. Engineering Manager, Docker
- 2. The people Three groups One culture Agenda The processes Maintainership Code & design The tooling Measure Automate
- 3. The people
- 4. Who are the people in open source? The people Users Contributors Maintainers
- 5. Who are the people in open source? The people Users Contributors Maintainers ● Influence the roadmap and priorities ● Report usability issues and bugs (+1!) ● Want to learn about what is coming next
- 6. Who are the people in open source? The people Users Contributors Maintainers ● Contribute something (code, docs, tests, …) ● Come to improve a project they use (hobbyist) ● Come to influence the project (professional)
- 7. Who are the people in open source? The people Users Contributors Maintainers ● Organize the project ● Filter and review contributors input ● Influence by defining the roadmap
- 8. Contributors (~800+ / 6M) Who are the people in open source? The people Users (~6000+ / 6M) ● These groups aren’t mutually exclusive ● These groups aren’t mapped to employers ● These groups are very different in size! Mtnr. (~60)
- 9. Props to (some of!) the Docker maintainers The people
- 10. The secret sauce for a healthy project ● Culture defines your open source community ● Of all things, culture is the one that scales best ○ Old habits die hard ○ Newcomers tend to follow community’s codes ○ It takes very few people to show the example to many ● Behave like you want your community to behave Open source culture
- 11. The processes
- 12. The role of maintainers ● A maintainer is not a super contributor ○ Typically know the project best and happen to also be top contributors ● To the contrary, the maintainers role is to reach 0 open contributions ○ By interacting with community: coaching contributors, reviewing code ○ By improving the project infrastructure: guides, testing infra, tooling ● To maintainers, unreviewed contributions are a liability Becoming a maintainer
- 13. How to identify those key community members? ● Golden rule: anybody should be able to become maintainer ○ The hobbyist dedicating a few hours every week ○ The professional paid by their employer to participate ● Docker’s approach: reward regular activity over extended period of time ○ Decision is vote-based (⅔ of maintainers + BDFL to approve) ○ But how to measure activity? Becoming a maintainer
- 14. How to identify those key community members? ● Number of contributions is not the right metric to identify maintainers ● Our solution: number of issues & pull requests interacted with ○ Opening a pull request and commenting 10 times on it counts as 1 ○ Commenting on someone else’s pull request also counts as 1 ● Promotes people showing broader interest than their own contributions Becoming a maintainer
- 15. ● Different groups ● Different expectations ● Conflicting priorities What and who are you optimizing for? Defining the process Features Quality Users Contributors Maintainers Project pace
- 16. ● Quality first ● Review fast ● Minimize feature creep Docker project tradeoffs Defining the process Features Quality Project pace
- 17. What and who are you optimizing for? ● Docker open source processes mostly favor contributors ○ We don’t delay merges during code freeze ○ We carry patches that can’t be merged as is ○ We work hard to “reach a yes” (80% of contributions are merged) ● A happy contributor may eventually become a helpful maintainers Defining the process
- 18. The goals of the process ● Minimizing frustration for the contributor... ○ By processing contributions in a timely manner ○ By “failing early” for contributions which won’t get merged ● … While preserving quality ○ Good automated testing and coverage ○ Two maintainers are required to merge Example: code review process
- 19. Our approach: 4 steps workflow using GitHub labels ● Easy to tell what’s expected next ○ Is it a contribution we want? ○ Is it properly written, tested, and safe? ○ Is it documented? ○ Profit! ● Plus a few special labels for problematic contributions ○ Tests aren’t passing ○ Not progressing Example: code review process
- 20. The pitfalls of “design by committee” ● Design decisions don’t scale (“too many cooks in the kitchen”) ● Golden rule: “No is temporary, yes is forever” ○ The default is “no” unless consensus can be reached ○ Reaching consensus gets harder as the group grows ● Divide and conquer: smaller groups on better delimited problems Scaling design
- 21. Manage your processes as code ● Processes are just another tool ○ Document them as text files in the repository ○ Adjust the processes through pull requests ● For examples, see project/ subdirectory in github.com/docker/docker Processes best practices
- 22. The tooling Measure, automate, repeat
- 23. Metrics, metrics, metrics! Measuring open source activity ● Many new questions will arise as a project scales ○ Who are the active members of the community? ○ Who is contributing to the project? ○ How fast do maintainers process contributions? ○ What are the most active repositories? ● We failed to find our ideal tool for that, so we built it
- 24. icecrime/vossibility-stack Metrics, metrics, metrics! Measuring open source activity
- 25. Automation ● GitHub webhooks are essential for open source automation ○ Docker uses NSQ to persist and fan-out messages ○ Allows for multiple listeners (queues) for a single repository (topic) docker/leeroy Automate actions on GitHub events docker/gordon-bot IRC bot to interact with Jenkins CI server icecrime/poule Mass interact with pull requests
- 26. What’s next? ● Come learn how to contribute, or meet the maintainers! Day 1 2:00pm Contribute 101: Engine / Swarm / Containerd 2.55pm Meet the maintainers: Engine / Swarm / Containerd Day 2 1:30pm Contribute 101: Compose / Kitematic / Machine 2:25pm Meet the maintainers: Compose / Kitematic / Machine