How we made jenkins community
Download as PPTX, PDF•4 likes•1,206 views
- Jenkins is an open source automation server with over 53,000 installations and 600+ plugins created by 500 contributors. It uses a modular architecture and extensibility model to enable large-scale distributed development. - Jenkins succeeded by focusing on extensibility through well-defined extension points, treating extensions as first-class citizens, enabling data and code extensibility, and ensuring extensions are themselves extensible. This reduced communication needs and lowered barriers to contribution. - An update center and shared resources like IRC provided a "center of gravity" to connect contributors and ensure the long-term maintenance of plugins. This self-reinforcing community allowed Jenkins to scale development through extensibility.
How we made jenkins community
- 1. ©2010 CloudBees, Inc. All Rights Reserved ©2010 CloudBees, Inc. All Rights Reserved How we made the Jenkins community Kohsuke Kawaguchi Creator of Jenkins / Architect, CloudBees, Inc.
- 2. ©2010 CloudBees, Inc. All Rights Reserved 2©2010 CloudBees, Inc. All Rights Reserved What is Jenkins? • Java web application • For builds, tests, and broader automation – Called “Continuous Integration server” • MIT-licensed, 8 years old jenkins-ci.org
- 3. ©2010 CloudBees, Inc. All Rights Reserved 3©2010 CloudBees, Inc. All Rights Reserved Where we are today • 53,000+ installations around the world • 600+ plugins by 500 contributors • #1 CI server in the market
- 4. ©2010 CloudBees, Inc. All Rights Reserved 4©2010 CloudBees, Inc. All Rights Reserved How did we come this far? • All the usual things – Software that works – Being at the right place at the right time • But I‟d like to think there‟s more to it – And it‟s something I think others can replicate
- 5. ©2010 CloudBees, Inc. All Rights Reserved 5©2010 CloudBees, Inc. All Rights Reserved Communication is inherently painful • Ask if the project likes the change I‟m thinking • Work on the change • Submit the patch and ask for approval • Defend the patch and revise it • Docs, tests, IP assignment • Done
- 6. ©2010 CloudBees, Inc. All Rights Reserved 6©2010 CloudBees, Inc. All Rights Reserved Programmers are picky bunch • All sorts of things to quarrel about – Should „{‟ be on the same line as “if”? – Different taste for design trade-off – Different anticipation for future expansions • Hardly bearable even when you know them personally
- 7. ©2010 CloudBees, Inc. All Rights Reserved 7©2010 CloudBees, Inc. All Rights Reserved More so in FLOSS • We don‟t know each other • No relationship outside coding • No common goals • No command structure • Time zone / language barrier
- 8. ©2010 CloudBees, Inc. All Rights Reserved 8©2010 CloudBees, Inc. All Rights Reserved Communication is root of all evil • Less, the better • Get rid of the need for it – Don‟t stand in the way of people and their itches – Innovations happen everywhere. Don‟t kill them before it materializes
- 9. ©2010 CloudBees, Inc. All Rights Reserved 9©2010 CloudBees, Inc. All Rights Reserved The Question
- 10. ©2010 CloudBees, Inc. All Rights Reserved 10©2010 CloudBees, Inc. All Rights Reserved It starts with modularity • It means: – Large software, divided into small silos – Module interaction controlled by contracts – Code/build/test at module level • Modularity enables extensibility
- 11. ©2010 CloudBees, Inc. All Rights Reserved 11©2010 CloudBees, Inc. All Rights Reserved Extensibility! • Open up to the rest of the world • The only thing that really matters – It enables all kinds of social/technical tricks • It‟s not free, but it‟s worth the effort
- 12. ©2010 CloudBees, Inc. All Rights Reserved 12©2010 CloudBees, Inc. All Rights Reserved #0: Programmability / Scriptability • Every software should be a library – Let other programs talk to your program – There‟s always a layer above you • “App” people tend to forget this – ImageMagick vs Gimp
- 13. ©2010 CloudBees, Inc. All Rights Reserved 13©2010 CloudBees, Inc. All Rights Reserved #1: Code loading at runtime • Define extension points – Contracts to be followed by extensions • Discovering implementations – Java, etc. requires some work • Provide isolation
- 14. ©2010 CloudBees, Inc. All Rights Reserved 14©2010 CloudBees, Inc. All Rights Reserved #2: Treat extensions as 1st class citizen • If core can do something that extensions can‟t, that‟s a bug • If users can tell apart extensions from core, you aren‟t doing enough
- 15. ©2010 CloudBees, Inc. All Rights Reserved 15©2010 CloudBees, Inc. All Rights Reserved #3: Data model extensibility • Extensions need to be able to store stuff – By adding stuff to core domain objects – By adding more domain objects – For Jenkins, XML really helped • Schema versioning becomes non-linear
- 16. ©2010 CloudBees, Inc. All Rights Reserved 16©2010 CloudBees, Inc. All Rights Reserved #4: Embrace heterogeneousness • Good contract enables multiple impls – Take something, slightly change something, and you have a new extension impl – You see a list, I see extensibility • And inheritance • This is surprisingly tricky in some context – In webapp, for example
- 17. ©2010 CloudBees, Inc. All Rights Reserved 17©2010 CloudBees, Inc. All Rights Reserved #5: Turtles all the way down • Extensibility needs to be recursive – Extensions should define its own extension points • In Jenkins, we provide an idiom • Partly culture • Partly refactoring
- 18. ©2010 CloudBees, Inc. All Rights Reserved 18©2010 CloudBees, Inc. All Rights Reserved Symptom of fake modularity/extensibility • Division of labor is not modularity • Modules are tailor-fit to their neighbors • Contracts aren‟t general enough • A fixed set of modules become the product Mosaic I - M.C. Escher
- 19. ©2010 CloudBees, Inc. All Rights Reserved 19©2010 CloudBees, Inc. All Rights Reserved Extensibility in Jenkins • Core – Creates structure – Includes Modules (plugins in disguise) • 600+ Plugins – Actually provide useful CI functionalities
- 20. ©2010 CloudBees, Inc. All Rights Reserved 20©2010 CloudBees, Inc. All Rights Reserved The Question
- 21. ©2010 CloudBees, Inc. All Rights Reserved 21©2010 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/jvuokko/4566270758/
- 22. ©2010 CloudBees, Inc. All Rights Reserved 22©2010 CloudBees, Inc. All Rights Reserved Extensible silo is a good thing • It forces contracts at boundaries – Principles of “continuous X”: do it often enough and it becomes painless • Small code is easier to hack – Looking at 5 files feel easier than 500 files – Faster to build, easier to maintain • Special use case does not clutter general- purpose code/UI – Example: Exclusion in Jenkins Subversion plugin • Niche features are welcome
- 23. ©2010 CloudBees, Inc. All Rights Reserved 23©2010 CloudBees, Inc. All Rights Reserved Extensible silo requires less communication • Everyone gets his own sandbox to hack – No need to discuss designs/features/schedules upfront • No more conflicting development model – Conference-driven development vs test- driven development • No need to see someone else‟s ugly code – No need to feel ashamed of it
- 24. ©2010 CloudBees, Inc. All Rights Reserved 24©2010 CloudBees, Inc. All Rights Reserved Extensible silo fosters innovations • Every whacky idea gets a shot – No need to sell ideas – No need to turn down ideas • Communicate in code, not in English • Some of them will turn out great – No need to drag failed experiments
- 25. ©2010 CloudBees, Inc. All Rights Reserved 25©2010 CloudBees, Inc. All Rights Reserved Extensibility highlights higher-order problems • With enough silos, core devs become silo devs • What core does directly benefits silos – Simplified build mechanism – IDE support – Error checker for common mistakes • It accelerates dev scalability
- 26. ©2010 CloudBees, Inc. All Rights Reserved 26©2010 CloudBees, Inc. All Rights Reserved Lower barrier to entry, enabled by silo • In Jenkins, everyone gets to be a committer – Just ask, and it‟ll be given – We even have an IRC bot that does this • This works, because… – People aren‟t collaborating – Any damage done is compartmentalized – VCS enables rollback so long as it‟s not released • After all, unless you are Linus Torvalds, “too many committers” is not your problem
- 27. ©2010 CloudBees, Inc. All Rights Reserved 27©2010 CloudBees, Inc. All Rights Reserved Lower barrier to entry: observations • People are cautious when they have power – Very few incidents • Works well when people move on – Abandoned plugins get new maintainers • It encourages small contributions • You can never predict who becomes significant contributors
- 28. ©2010 CloudBees, Inc. All Rights Reserved 28©2010 CloudBees, Inc. All Rights Reserved But silo alone doesn‟t create a community Image: NASA/JPL-Caltech
- 29. ©2010 CloudBees, Inc. All Rights Reserved 29©2010 CloudBees, Inc. All Rights Reserved You need a center of gravity • Something that: – Drag everything together – Pulls in people • Shared source tree repo doesn‟t have to be “it” Image: NASA/JPL-University of Arizona
- 30. ©2010 CloudBees, Inc. All Rights Reserved 30©2010 CloudBees, Inc. All Rights Reserved Center of Gravity in Jenkins: IRC • Closest thing to the water cooler in OSS – It is a place people can come, unlike Twitter – You can be present & absent at the same time • It builds human relationships
- 31. ©2010 CloudBees, Inc. All Rights Reserved 31©2010 CloudBees, Inc. All Rights Reserved Center of Gravity in Jenkins: outreach • Connecting people – Meet-ups, Hackathon, Events • Giving the project a voice – Twitter, Blog Mad props to @agentdero
- 32. ©2010 CloudBees, Inc. All Rights Reserved 32©2010 CloudBees, Inc. All Rights Reserved But above all: Update Center • UI built into Jenkins to let users install plugins • To be listed, your plugins need to be co- located in GitHub org • Commit access covers all the repos • Everyone can add others to the project
- 33. ©2010 CloudBees, Inc. All Rights Reserved 33©2010 CloudBees, Inc. All Rights Reserved Observations • Creates an incentive for devs to come • When people move on, – Keeping plugin alive is easier • A structure helps new contributors – Right thing is now the easiest thing • Providing pin-point technical help is easier
- 34. ©2010 CloudBees, Inc. All Rights Reserved 34©2010 CloudBees, Inc. All Rights Reserved Center of Gravity = Self-reinforcing • As we acquire more plugins… – We attract more users – UC becomes more attractive to devs • Machinery gets oiled – As a result of solving higher-order problem
- 35. ©2010 CloudBees, Inc. All Rights Reserved 35©2010 CloudBees, Inc. All Rights Reserved Emerging Challenges • Subversion → Git changed dynamics – Repository no longer projects enough gravity • GitHub and pull requests – It‟s great that contributions are easier – But it also creates “throw the stuff over the wall” mentality – It lets people contribute without joining, reducing the inflow
- 36. ©2010 CloudBees, Inc. All Rights Reserved 36©2010 CloudBees, Inc. All Rights Reserved Conclusions • Scale development with extensibility • Reduce communication with extensibility • Let people innovate with extensibility • I think I made Steve proud Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility! Extensibility!
Editor's Notes
- #4: I suppose these numbers are nothing to brag about when we have folks from all the big projects in this conference, but for a project that somebody started without a corporate backing, with relatively niche user base, I think this is pretty good.
- #5: There’s all the usual things, but there’s something more, too. That’s what I wanted to talk about.
- #6: Lead in: so what are those key ingredients that we put into the Jenkins project? It starts with the realization that…
- #8: I mean, it takes some special mindset to love hacking code for free.
- #17: Heterogeneousness example:Bugs in bugzillaMedia objects in iTunesDocument types in Google DocsAnywhere where you see a list
- #21: Remember the question I asked earlier; now we are ready to answer this question
- #23: While silo is often talked as a bad example of everyone heads down into their little corner of the problem without a broad picture, but …
- #26: Those obstacles for more scalability become far more evident when you work on multiple modules, because they are no longer one-off problems. Because core developers also must work on modules, module developers’ pain point becomes core developers’ pain point.
- #29: But silo alone does not form a community. If everyone minds his own business and if that’s all they do, you don’t end up accumulating critical mass anywhere.A good example of this is custom Ant tasks. It’s done all over the places, and not much sharing and talking going on.
- #31: Someone please do IRC 2.0! Or maybe the barrier to entry is serving as a signal/noise filter. I dunno.
- #34: Expectation