Project-Builder.org presentation
1 like•627 views
The document discusses Project Builder, a GPL continuous packaging solution developed by Bruno Cornec at HP, designed to streamline the packaging process within software development. It emphasizes the importance of integrating packaging into the project lifecycle and proposes a cloud-based continuous packaging approach that is agnostic to various configuration management systems and Linux distributions. The main goal is to facilitate continuous packaging while minimizing overhead and enhancing collaboration, reproducibility, and deployment efficiency.
![Commands
[cms|sbx]2build: Create tar files for the project under your CMS. Parameters are
packages to build if not using default list
build2pkg: Create packages for your running distribution
build2ssh: Send the tar files to a SSH host
pkg2ssh: Send the packages built to a SSH host
build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and
send those packages to a SSH host once built
launchvm: Launch one virtual machine
script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it
test2vm|ve|rm: Test inside a VM/VE/RM
newvm|ve: Create a new VM/VE/RM
setupvm|ve|rm: Setup a VM/VE/RM for pb usage
newver: Create a new version of the project derived from the current one
newproj: Create a new project and a template set of configuration files under
pbconf
17 HP / OSSI / Bruno Cornec](https://image.slidesharecdn.com/pb-presentation-130411191933-phpapp02/85/Project-Builder-org-presentation-15-320.jpg)
Project-Builder.org presentation
- 1. Project Builder: A GPL continuous packaging solution Bruno Cornec HP EMEA Open Source Profession Lead V2.0 April 2013
- 2. Introducing Myself ● Software engineering and Unices since 1988 ● Mostly Configuration Management Systems (CMS), Build systems, quality tools, on multiple commercial Unix systems ● Discover Open Source & Linux (OSL) & first contributions in 1993 ● Full time on OSL since 1995, first as HP reseller then @HP ● Currently: ● Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble ● OSL HP Advocate ● EMEA OSL HP Profession Lead ● Solutions Linux Conference and OWF board member ● MondoRescue, Dploy.org, Project-Builder.org project lead ● LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor ● FOSSBazaar and OSL Governance enthusiast ● Mandriva, Mageia, Fedora packager 2 HP / OSSI / Bruno Cornec
- 3. What is the problem ? Users / Sysadmins environment: >= 90% < 10% Users: easyness (GUI, CLI) Admins: distribution compliance, < 1% smooth integration – reduce admin load 3 HP / OSSI / Bruno Cornec
- 4. What is the problem ? Developers environment: Sourceforge Flat Files ftp, web Project CMS/VCS autotools ? CVS Install proc ? Soft. Eng. ? SVN Fear of distro ? GIT, ... 4 HP / OSSI / Bruno Cornec
- 5. What is the problem ? Packagers environment: Distro ftp, web Flat Files Package CMS/VCS Does s/he only exists ? CVS On time ? Lag behind ? rpmlint ? Distro coverage (ver, #) SVN lintian ? Beta avail ? GIT, ... 5 HP / OSSI / Bruno Cornec
- 6. My answer with my various hats I'm a sysadmin so I want to deal with packages only (deployment server integration) I'm a developer so I want to ease my tests without screwing up my system and testing as a std user/admin I'm a software engineer and want no duplication of code nor information (metadata) I'm a a packager so I want to streamline that work for new projects I'm interested in packaging I'm the marketing department of my own project and use packages as a “competitive advantage” Founding no tool existing to help me doing all that (2005), so I started writing one to scratch my own itch 6 HP / OSSI / Bruno Cornec
- 7. What is the solution ? Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects New mantra: “Package early, package always” Use packaging as your only way of delivery (not a dream) Overhead minimal (not much longer than providing a tgz), maximum benefit (consistancy, reproduceability, team work improvement, distribution integration, improved deployment) Packaging is also a marketing activity based on a technical content. It's an easy way to extend your user base, and improve your community. Makes tests easy. THE SOLUTION IS CONTINUOUS PACKAGING 7 HP / OSSI / Bruno Cornec
- 8. Cloud based Continuous Packaging Local Build Server (may host VMs and/or VEs) Packagers VM or VE Build Repository Build + Server metadata Local build Project RM Build Farm (may host VMs) Remote build Developers 9 HP / OSSI / Bruno Cornec
- 9. Provisioning the Continuous Packaging Cloud Local Build Server (may host VMs and/or VEs) VM or VE Build Build + metadata Local build Repository Server Project RM Build Farm (may host VMs) Remote build 10 HP / OSSI / Bruno Cornec
- 10. Goals Project-builder main goal is to help you package continuously being agnostic: CMS agnostic: no CMS but guys it's 21st century now, SVN, CVS, Mercurial, GIT, SVK.... OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 120+ distro tuples at the moment – repositories for yum, urpmi, apt. Solaris pkg. HP-UX sd in progress Build environment agnostic: local, VM (QEMU, KVM...), VE (rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm) No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project. 11 HP / OSSI / Bruno Cornec
- 11. Goodies Project-builder provides additional goodies: Easy VMs/VEs/RMs management Macro system with perl variables to avoid duplication Skeleton generation to help starting Manages delivery up to your repository (ssh based) Manages announces on mailing lists Integration of tests in the process Manages patches/additional sources when not upstream Checks validity of packages built (lintian, rpmlint) Easy creation of new versions for upstream management 13 HP / OSSI / Bruno Cornec
- 12. VM, VE & RM Management VE Creation: debootstrap for .deb based ones. rpmbootstrap (modeled after debootstrap): create chroot for multiple RPM based distributions (Mageia, Fedora, Mandriva, CentOS, OpenSuSE) VM|RM Creation: KVM or QEMU iso install possibilities. LinuxCOE/OpenStack as provisioning tool for your VM|RM VM|VE|RM setup: adds pb tools and build account VM|VE Snapshot feature Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg). ssh communication for VM/RM 14 HP / OSSI / Bruno Cornec
- 13. Metadata Management Packaging process independent of the project and external Metadata isolated - No duplication 15 HP / OSSI / Bruno Cornec
- 14. Configuration file usage Macro example: ./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar to debootstrap for RPM based distributions ./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY ./rpmbootstrap/deb/control:Description: PBSUMMARY ./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY" Instantiation possible from distro-ver-arch, distro-ver, distro, distro- family, distro-type, os Similar approach for all other parameters (Cf: man pb.conf) 16 HP / OSSI / Bruno Cornec
- 15. Commands [cms|sbx]2build: Create tar files for the project under your CMS. Parameters are packages to build if not using default list build2pkg: Create packages for your running distribution build2ssh: Send the tar files to a SSH host pkg2ssh: Send the packages built to a SSH host build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and send those packages to a SSH host once built launchvm: Launch one virtual machine script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it test2vm|ve|rm: Test inside a VM/VE/RM newvm|ve: Create a new VM/VE/RM setupvm|ve|rm: Setup a VM/VE/RM for pb usage newver: Create a new version of the project derived from the current one newproj: Create a new project and a template set of configuration files under pbconf 17 HP / OSSI / Bruno Cornec
- 16. The Big picture Packagers newver / newproj newvm|ve / setupvm|ve|rm Packaging build2vm|ve|rm cms|sbx2build Project Building Packages build2pkg pkg2ssh build2ssh Developers 18 HP / OSSI / Bruno Cornec
- 17. Multi platform life cycle User Requirements Validation Functional Integration tests Specification Design Unit tests Specification 19 Coding HP / OSSI / Bruno Cornec
- 18. HP+Intel Solution Brief 21 HP / OSSI / Bruno Cornec
- 19. Multi platform packaging cycle VMs/VEs/RMs Developers UUWL Packages VM/VE/RM 23 Packagers HP / OSSI / Bruno Cornec
- 20. Q & A up to now ? 24 HP / OSSI / Bruno Cornec
- 21. DEMONSTRATION 25 HP / OSSI / Bruno Cornec
- 22. To be done LSB chroot support. (or not) LXC support Support for libvirt, virsh, openstack API Other CMS (Bazaar, …) only when/if needed Other VMs (VMWare, Xen, ...) only when/if needed Multiple delivery means Look at interactions with Buildbot Add non-interactive signature support for .deb Config-Model for configuration file management REST API and Web Interface 26 HP / OSSI / Bruno Cornec
- 23. Learn Project-Builder.org Start with the Lab (63 pages) Use man (pb, pb.conf + 8 ProjectBuilder::* man pages) Use the mailing-list pb-announce and pb-devel at http://www.mondorescue.org/sympa Use examples from http://trac.project-builder.org/browser/projects/ 27 HP / OSSI / Bruno Cornec
- 24. Web Resources Project-Builder Web site / Trac / Wiki: ● http://www.project-builder.org ● http://trac.project-builder.org Projects using project-builder.org: ● http://www.project-builder.org (of course :-) ● http://www.mondorescue.org ● Http://www.linuxcoe.org ● http://www.fossology.org ● http://kde-apps.org/content/show.php/Meganizer?content=129907 28 HP / OSSI / Bruno Cornec
- 25. Related tools Project-Builder is mostly suited for upstream projects wanting to package their applications Distributions provide each their build tools ● SuSE: Open Build Service (Multi distro, Web based, BaaS) ● Fedora (Koji) ● Mandriva/Mageia (Youri, mdvsys, mgarepo...) Other complementary tools: ● Buildbot ● KVM/QEMU ● rpmbootstrap, rinse, mock ● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail Project-Builder.org is one of the tool of the vcs-pkg.org initiative. 29 HP / OSSI / Bruno Cornec
- 26. [email protected] Contact (Linux Solution Consultant in the HP/Intel Solution Center) http://www.hp.com/linux ”Changes are never easy to make. There is comfort and safety in tradition, but change must come, no matter how painful or expensive it may be.” Bill Hewlett Linus Torvalds, Richard Stallman, Eric Raymond, Nat Makarevitch, René Cougnenc, Eric Dumas, Rémy Card, Bdale Thanks Garbee, Bryan Gartner, Gallig Renaud, Joachim Langenbach among others, for their work and devotion to the Open Source Software cause... and my family Open Source at HP for his patience :-) http://opensource.hp.com 30 HP / OSSI / Bruno Cornec