Building software: the lessons from open source
3 likes2,256 views
This document summarizes Arnaud Porterie's presentation on lessons from open source for building software. Some key points include: - Open source projects follow patterns of "wise crowds" like having a strong mission, free entry for contributors, transparency, and fair authority to scale effectively. - These patterns can help improve closed source software development by fostering greater collective intelligence within companies. - Applying concepts like inner source allows companies to benefit from open source practices internally by opening codebases and encouraging cross-team contributions under shared maintainers. - A practical checklist is proposed for implementing inner source based on principles like giving all employees access to code, having documented missions and contribution processes, and enforcing consistent processes.
Building software: the lessons from open source
- 1. Arnaud Porterie - @icecrime Building software: the lessons from Open Source Arnaud Porterie - @icecrime VP Engineering - Vente Privée
- 2. Open source matters Nobody can afford to ignore the open source movement Virtually every company uses open source infrastructure software “Software is eating the world, open source is eating software”
- 3. Kubernetes # of commits per month - 6 months moving average
- 4. Kafka # of commits per month - 6 months moving average
- 5. Linux # of commits per month - 6 months moving average
- 6. The open source world seems to be doing something right
- 7. Leveraging open source code in your company Contributing to open source as a company Open sourcing company’s code What this talk is not about
- 8. What this talk is about Taking inspiration from open source in the way we build software in-house software
- 9. Alternative title How my role as VP Engineering today is influenced by my previous experience managing a large scale open source project (they said it was too long ¯_(ツ)_/¯)
- 10. Arnaud Porterie - @icecrime A primer on open source project organization
- 11. Users ● File issues, report bugs, ask for features ● Influence the project by voicing their opinions Participants in an Open Source project
- 12. Contributors ● Send patches, fix bugs, write documentation ● Often are also users of the project ● Influence the project by submitting modifications Participants in an Open Source project
- 13. Maintainers ● Guardians of the temple ○ Responsible for the overall health of the project ○ Quality of the code ○ Project scalability ○ Project testing and release cycle ○ Culture of the community ● Typically are also contributors of the project ● Influence the project by deciding what goes in Participants in an Open Source project
- 14. ● Read the contributing guide (usually CONTRIBUTING.md) ● Discuss with maintainers ○ Explain the need ○ Explain how you intend to implement it ● Produce the contribution ● Submit a Pull Request (GitHub) ● Patch gets merged when two maintainers give their LGTM Typical contribution process
- 16. In Open Source, your dependencies (upstreams) don’t work for you ● What you need to be done you contribute yourself As a maintainer, you play nice with those who depend on you (downstreams) ● When they come in to file an issue or contribute a patch, be helpful Managing dependencies
- 17. Arnaud Porterie - @icecrime The elements of healthy open source
- 18. Healthy open source An open codebase with identified maintainers is not enough ● A healthy open source community fosters collective intelligence ● Not a coincidence but a deliberate act of designing a community Under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them. The Wisdom of Crowds, James Surowiecki
- 19. Patterns of a wise crowd Culture & Empire - Pieter Hintjens ISBN-13: 978-1492999775
- 20. Patterns of a wise crowd
- 21. Patterns of a wise crowd ● Strong mission: a formulation of the single main problem to solve ● Free-entry: strangers may walk in and get involved ● Strong protocols: a set of rules to collaborate properly ● Fair authority: a scalable authority to enforce the rules ● Transparency: all activity takes place in the open ● …
- 22. Patterns of a wise crowd Successful online communities expose those patterns For an open source project, it’s a matter of survival
- 23. Impact of wise crowds patterns Ignoring wise crowds patterns Applying wise crowds patterns Open source software Pushes the community away Will inevitably die Scales to thousands contributors Maintains quality and velocity Has extreme resilience
- 24. Impact of wise crowds patterns Ignoring wise crowds patterns Applying wise crowds patterns Open source software Pushes the community away Will inevitably die Scales to thousands contributors Maintains quality and velocity Has extreme resilience Closed source software ? ?
- 25. Arnaud Porterie - @icecrime Looking back at the enterprise
- 26. Is the enterprise enabling wise crowds? ● No universal answer: there are as many organizations as there are companies ● The traditional enterprise optimizes for control and predictability ○ Contrary to open source, applying the patterns is not a matter of survival ● Some of the common organizational pitfalls are clear wise crowds antipatterns with immediate impact on the software it produces
- 27. “Application teams” commonly yield ● Lack of a strong mission expressed in user terms ● Arcane undocumented maintenance knowledge Wise crowds antipatterns Wise crowds patterns Strong mission Measurable success Free entry High scoring Transparency Decentralization Full remixability Free workspaces Strong protocols Smooth learning Fair authority Regular structure Non-tribalism Positivity Self-organization Sense of humor Tolerance Minimalism
- 28. Organizational silos commonly yield ● Strong sense of ownership ● Membership model / “us and them” situations ● Heterogeneous structures between departments Wise crowds antipatterns Wise crowds patterns Strong mission Measurable success Free entry High scoring Transparency Decentralization Full remixability Free workspaces Strong protocols Smooth learning Fair authority Regular structure Non-tribalism Positivity Self-organization Sense of humor Tolerance Minimalism
- 29. Top-down decision making commonly yields ● Complexity (decision decoupled from constraints) ● Suboptimal, insufficiently challenged solutions ● Centralization and an inability to scale ● Reluctance to express conflicting opinions Wise crowds antipatterns Wise crowds patterns Strong mission Measurable success Free entry High scoring Transparency Decentralization Full remixability Free workspaces Strong protocols Smooth learning Fair authority Regular structure Non-tribalism Positivity Self-organization Sense of humor Tolerance Minimalism
- 30. Impact of wise crowds patterns Ignoring wise crowds patterns Applying wise crowds patterns Open source software Pushes the community away Will inevitably die Scales to thousands contributors Maintains quality and velocity Has extreme resilience Closed source software Relies on tribal knowledge Produces lower quality output Tends to build castles over cities ?
- 31. Can we raise the “collective intelligence score” of an enterprise by importing patterns of the open source world?
- 32. Arnaud Porterie - @icecrime Inner source & practical implementation
- 33. Inner source https://en.wikipedia.org/wiki/Inner_source Inner source is the use of open source software development best practices and the establishment of an open source-like culture within organizations. The organization may still develop proprietary software, but internally opens up its development. The term was coined by Tim O'Reilly in 2000.
- 34. Impact on teams dynamics Distinguishing maintainer and contributor roles ● Alters the sense of ownership ○ Maintainers “own” the quality, coherence, and sustainability of the project ○ The organization collectively “owns” the code ● Contributors becomes an open group (free entry principle) ○ Anyone can walk in and participate ● Maintainers also becomes an open group! (fair authority principle) ○ Constructive participation to the project must be rewarded
- 35. Impact on teams dynamics Encouraging cross-teams contributions ● Puts the collective goal above individual ownership considerations ● Raises the bar for best practices, particularly on documentation and testing ● Gives individual teams more control of their destiny ● Keeps the development effort where the need exists versus where its implemented
- 36. ADOPTION Other engineering teams start depending on the project CONTRIBUTIONS Downstreams participate in the project’s development PROJECT’S TEAM GROWTH The project promotes maintainers to follow and sustain a healthy growth Grow software, not teams
- 37. Impact on teams dynamics More importantly ● Creates a community where people collaborate and learn from each other ● Creates a environment where engineers aren’t confined to a single codebase
- 38. Inspired by Joel Spolsky’s “The Joel Test: 12 Steps to Better Code” https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ A practical checklist for implementing inner source
- 39. ● Pick a single source manager for the entire organization ○ Preferably choose a stack-agnostic solution ○ Preferably choose a solution that supports managing and publishing documentation ○ Typically GitHub or GitLab (GitHub may be better if you plan on open sourcing later) ● Potentially a cultural barrier to cross 1. Does everyone in the organization have access to all of the codebase by default?
- 40. ● Openness has little value without discoverability ● Choose a physical organization the team is familiar with ○ Don’t follow the organization chart, it will likely change over time ○ Ideally follow a reasonably stable functional layout (e.g., domain / product / component) ● Allows individual pieces to become “stepping stones” for others to build upon 2. Is the codebase structured or documented in a way that makes it easy to explore?
- 41. ● Think of the single main problem your project is solving for the users ○ This is crucial to constrain the scope ○ NB: “continuing to exist” is not a mission ● Typically formulated in a README.md at the root of the repository 3. Do individual projects have a documented mission?
- 42. ● Low barrier of entry is key for a welcoming project ○ Anyone should be able to build, test, run with very few steps ○ Future hires to the team will thank you ● Typically formulated in a CONTRIBUTING.md at the root of the repository 4. Do individual projects have a documented and straightforward contribution process?
- 43. ● If there’s two processes, you can be sure that one of them is obsolete ○ Spoiler: it’s the one used by people external to the project ● Applies to ○ Coding guidelines ○ Contributions expectation ○ SLA for patch review (recommandation: use FIFO is most cases) ○ Testing (are maintainers relying on a dedicated QA team?) ● Don’t blame external contributors for not knowing your undocumented rules 5. Are all contributors treated equally (i.e., teams / maintainers follow a different process)?
- 44. ● Organise your company communications for transparency & discoverability ○ Encourage conversations to be public by default ○ Make the channels easy to discover (e.g., use a common format for teams) ● Document design and architecture decisions ○ Can be markdown files in git, or simply issue comments 6. Is there an open discussion platform where decisions are taken and can be challenged?
- 45. ● Good rules ○ Are collectively defined and managed as code ○ Start small, and grow in scope as we learn from experience ○ Foster collaboration while leaving space for self-organization ■ For example: to enforce the previously mentioned items of the checklist ● Who is the “fair authority” inside your organization? ○ Ideally a group of highly respected technical leaders (preferrably IC over managers) 7. Is there a set of commonly agreed upon rules enforced by a fair authority?
- 46. Inner source: a checklist 1. Does everyone in the organization have access to all of the codebase by default? 2. Is the codebase structured or documented in a way that makes it easy to explore? 3. Do individual projects have a documented mission? 4. Do individual projects have a documented and straightforward contribution process? 5. Are all contributors treated equally (i.e., teams / maintainers follow a different process)? 6. Is there an open discussion platform where decisions are taken and can be challenged? 7. Is there a set of commonly agreed upon rules enforced by a fair authority?
- 47. Final words Ignoring wise crowds patterns Applying wise crowds patterns Open source software Pushes the community away Will inevitably die Scales to thousands contributors Maintains quality and velocity Has extreme resilience Closed source software Relies on tribal knowledge Produces lower quality output Tends to build castles over cities Allows to scale wisely Puts group’s interest over teams Raises the bar for best practices
- 48. Final words Open source communities have repeatedly shown a capacity to produce high quality sustainable software by carefully designing for collective intelligence These recipes apply to the enterprise, but it’s a matter of culture ● Requiring full support from leadership ● Immune to whatever tool you may throw at it
- 49. Arnaud Porterie - @icecrime Thank you!