![DIA Agency, Inc.
that around and implies the POM dependencies from the MANIFEST.MF just as
PDE does for JDT
However, for build plug-ins that do not map to internal Eclipse builders, m2e uses an
embedded maven3 engine to run maven plug-ins as required. For example, to do a
clean build of the org.eclipse.papyrus.infra.constraints[.edit[or]] plug-ins as
reconfigured in Gerrit review 41167 :2
•the Maven builder provided by m2e generates the sources from the EMF genmodel
into the src-gen/ folder
•if one deletes the generated sources, it takes two clean re-builds to return to a
correctly compiled project: the first one generates the sources and the second one
re-compiles custom code that depends on the generated code
•this is easily fixed: it is only needed to move the Maven Project Builder ahead
of the other builders in the .project file
The Maven lifecycle mappings for a typical Xtend plug-in project
See https://git.eclipse.org/r/#/c/41167/2
Maven Nature in Papyrus Source Projects of9 13](https://image.slidesharecdn.com/adoptingmavennatureinpapyrussourceprojects-150218210110-conversion-gate01/85/Adopting-the-Maven-Nature-in-Papyrus-Source-Projects-10-320.jpg)
Adopting the Maven Nature in Papyrus Source Projects
- 1. DIA Agency, Inc. Maven Nature in Papyrus Source projects Prepared for: The Eclipse Papyrus Project Committer Team Prepared by: Christian W. Damus 18 February, 2015
- 2. DIA Agency, Inc. Background Motivation It has been proposed to convert the source projects (primarily plug-in projects and feature projects) in the Papyrus git repository to “m2e projects”, more formally adding the Maven Nature to these projects and enabling the Maven Project Builder. Reasons include but may not be limited to: •Alignment with the mechanisms of the Hudson build from which the Papyrus integration and release builds are published, which in the Mars release are based on the Tycho plug-ins for Maven •Integration of new build processes/steps, such as generation of code from EMF generator models , which have already been implemented as Maven plug-ins and1 would otherwise need to be re-implemented as Eclipse builders (perhaps tied to additional new project natures) Concerns As it constitutes a significant change to the management and behaviour of the Papyrus source projects in every developer’s workbench, there are naturally concerns raised about potential impact on developer workflows and productivity. Also, the historical experience of individual developers with the technologies involved (Maven, Eclipse m2e) raises questions. Most concerns relate to performance of Eclipse and scalability of the implementation in workspaces that import most or all of the Papyrus source projects: See, for example, the Gerrit change implementing EMF model generation: https://git.eclipse.org/r/#/c/41167/1 Maven Nature in Papyrus Source Projects of1 13
- 3. DIA Agency, Inc. •Off-line work: does a workspace build with Maven projects need to be on-line to check remote repositories and download artifacts? •The Papyrus builds currently perform very poorly in the project dependencies resolution phase. Will workspace builds manifest similar performance issues? •Do Maven projects actually run maven builds to accomplish a workspace build? •Does this scale to the several hundred projects that comprise the complete Papyrus source base? Analysis To answer the questions and concerns raised above, it is prudent to actually try an experiment to prove one way or another the performance and usability of a large Papyrus workspace when all of the source projects have the Maven Nature enabled. This section records observations from just such an experiment, exercising various scenarios of a typical Papyrus developer’s activities on a modern computer system: a mid 2012 Retina MacBook Pro with a 750GB SSD, quad-core Intel Core i7, 16 MB RAM running OS X 10.10.2. Initial Import The experiment started with a fresh workspace into which all of the Papyrus “main” and “main tests” projects were imported using the Oomph Setup model and built in the normal way, from the git commit ff639eed9c4371bfa80f3bc987229a9efe7deb3a (18 Feb 2015 09:15:23 EST) on the master branch. Next, certain Maven preferences were configured to ensure that the experiment was well isolated: Maven Nature in Papyrus Source Projects of2 13
- 4. DIA Agency, Inc. Ensuring that Maven does not update anything from the Internet unnecessarily Using a dedicated local Maven repository stored in a mounted disk image for ease of clean-up Next, all projects in the Project Explorer were selected and converted to m2e projects using the Configure → Convert to Maven Project context menu action. This initiated a Maven Nature in Papyrus Source Projects of3 13
- 5. DIA Agency, Inc. lot of automatic work, pegging the CPU for more than 20 minutes (the timing is unknown due to absence from the office) but it did complete normally, with the Eclipse workbench settling down to idleness without any error dialogs. However, the result was not ready to use, as none of the projects yet understood the Tycho plug-in required for successful build: Project build configuration errors caused by missing lifecycle mappings for Tycho plug-ins This is remedied by installing the Tycho connector for m2e and restarting Eclipse: The m2e connector discovery automatically finds the Tycho connector and then proceeding with the installation: Maven Nature in Papyrus Source Projects of4 13
- 6. DIA Agency, Inc. Installing the Tycho connector for m2e This connector should be added to the Papyrus Setup model before any conversion of the Papyrus projects to Maven Nature is implemented, so that developers do not have to go through the connector discovery process themselves. On restarting Eclipse after completing this installation, all of the source projects could now be built. The resulting impact on the git workspace is considerable: Converting to Maven projects results in 881 changes in git: new, changed, and deleted metadata files Maven Nature in Papyrus Source Projects of5 13
- 7. DIA Agency, Inc. But already this exercise has been illuminating because it highlights some potentially significant problems, including: Some projects that should be in the Papyrus release clearly aren’t being built by Hudson Other projects still have inconsistent build metadata despite recent efforts to clean that up Maven Nature in Papyrus Source Projects of6 13
- 8. DIA Agency, Inc. Developer Workflows The initial import and configuration of projects discussed above is a one-time cost that will be absorbed by whichever committer does the Maven conversion. It should not be something that any other Papyrus developer deals with. This section looks at typical development scenarios that Papyrus developers do see every day. Workspace Re-build From time to time, Eclipse needs to re-build a significant subset of the projects in the workspace (or even all of them) when some build configuration changes. That may be default compiler settings or the dependencies declared in a bundle manifest. In the test workspace, out of 441 projects, most depend on the org.eclipse.papyrus.infra.core.emf plug-in either directly or indirectly. Changing the dependencies in this plug-in’s manifest triggers a rebuild (because ‘build automatically’ is on) of this project and all of its dependents. This re-build: •took 2 minutes and 10 seconds the first time. Subsequent re-builds of a similar nature would typically only take about a minute •this compares to about 45 seconds currently in the regular PDE projects •resulted in 1800+ new compile errors in the Class Diagram plug-in (that first time) •conversion to Maven project updated the plug-in classpath, exposing a disagreement between the required execution environment (BREE) and its compiler settings. These are the compile errors pictured above. This is a good result and easily fixed •resulted in two errors on MANIFEST.MF in a test fragment because the Maven conversion removed source path entries from the .classpath file (probably because the build.properties file omitted the necessary source path specification) •on fixing this, a dependency on the wrong Java SE version was revealed in the same fragment So, in fact, the performance of re-build is comparable to the status quo and the Maven project nature highlights further build inconsistencies that were not revealed by the Maven Nature in Papyrus Source Projects of7 13
- 9. DIA Agency, Inc. Hudson build set-up. Notably, re-builds triggered by changes in API signatures that are used by other projects are practically instantaneous. These are far more common than the clean re-builds required by classpath changes. Xtend Projects Sometimes the Java sources generated from Xtend drift a bit out of sync and we force a re-build by cleaning the project via the Project → Clean… menu action. In the Mavenized org.eclipse.papyrus.uml.profile.assistants.generator projects this takes on the order of a few seconds, comparable to the status quo. It is worth noting that this is an Xtend project in which the generated Java sources are not committed to the git repository. Run-time Workbenches A lot of a developer’s testing is done in a run-time workbench launched from the development instance. This shows no problems, working as usual. In particular, the plug-in configuration of the run-time instance is assembled from the PDE target and the projects in the workspace. Likewise JUnit plug-in test launches. Maven Projects Under the Hood It is evident that m2e actually does not perform a maven build at all when building most projects. Its primary function is to configure the project’s classpath based on the dependencies specified in the pom.xml for the benefit of the regular JDT build. For the most part, all build lifecycle phases are mapped to the internal builders provided by Eclipse such as the JDT/PDE builders, Xtext builder, etc.: •if one deletes the Java and Xtext builders from an Xtend project, then invoking Project → Clean… on the project has no effect at all. Nothing cleaned and nothing built •however, one can run an external Maven build, if necessary, from the context menu Run As → Maven Xyz action. But there would almost never be a reason to •the main function that m2e provides is to give JDT a classpath container that resolves the classpath from the POM dependencies, but the Tycho connector turns Maven Nature in Papyrus Source Projects of8 13
- 10. DIA Agency, Inc. that around and implies the POM dependencies from the MANIFEST.MF just as PDE does for JDT However, for build plug-ins that do not map to internal Eclipse builders, m2e uses an embedded maven3 engine to run maven plug-ins as required. For example, to do a clean build of the org.eclipse.papyrus.infra.constraints[.edit[or]] plug-ins as reconfigured in Gerrit review 41167 :2 •the Maven builder provided by m2e generates the sources from the EMF genmodel into the src-gen/ folder •if one deletes the generated sources, it takes two clean re-builds to return to a correctly compiled project: the first one generates the sources and the second one re-compiles custom code that depends on the generated code •this is easily fixed: it is only needed to move the Maven Project Builder ahead of the other builders in the .project file The Maven lifecycle mappings for a typical Xtend plug-in project See https://git.eclipse.org/r/#/c/41167/2 Maven Nature in Papyrus Source Projects of9 13
- 11. DIA Agency, Inc. Running an external Maven build, which is never necessary The lifecycle mappings for custom Maven plug-ins that the Maven builder runs during project build Maven Nature in Papyrus Source Projects of10 13
- 12. DIA Agency, Inc. Additional Build Scenarios Hopefully, most of the time, re-building the Eclipse workspace is an incremental process. Only projects and/or files particularly affected by some specific change need to be rebuilt. JDT manages this well enough, and as we have seen, in Maven projects JDT is still in charge. However, there is still the need occasionally for a clean re-build of all projects in the workspace (for example after pulling changes from a remote git repository), which even without Maven tends to the tedious. Full Clean Build A clean re-build of the entire workspace, after first deleting the entire contents of the local Maven repository: •takes about 65 seconds for the clean phase, including downloading core Maven plug-ins, Tycho plug-ins, Xtext plug-ins required for cleaning and their dependencies •takes about 3 minutes and 55 seconds for the entire build (including the aforementioned cleaning phase). The build •uses the PDE target: EMF plug-ins are only downloaded that are needed to support the Papyrus EMF model generator plug-in •includes downloading the rest of the maven plug-ins required for build (but, again, not the target platform, which is entirely provided by PDE) •this compares to 1 minute 30 seconds for a full clean build in the pre-Maven workspace (in which the clean step takes a mere 10 seconds) The point about the PDE target, above, implies that we need make no changes to the Papyrus Oomph Setup model, because we still need it to provision the PDE target, except that we want to add the Tycho m2e connector. In particular, there is (under normal circumstances) no duplication of target dependencies between the PDE Target as configured by Oomph and the local maven repository, unless the developer also uses the same repository to perform external (that is, outside of Eclipse) maven builds of the Papyrus source projects. Maven Nature in Papyrus Source Projects of11 13
- 13. DIA Agency, Inc. Initial Build An earlier section detailed the initial build experience of the committer that implements the Maven nature in all of the Papyrus source projects. Every other Papyrus developer will not have that same experience, but will still have an initial build when either •pulling the commit(s) that implement(s) the Mavenization of Papyrus, or •cloning the Papyrus git repository to set up a new development workspace The experiment focuses on the latter scenario, emulated by checking in all changes described previously to implement the Maven nature on a new branch, deleting all projects from the Eclipse workspace, and doing a $ git clean -d -x -f to clean all derived and otherwise untracked files from the git workspace. Subsequently importing all of the “main” and “main test” projects back into the Eclipse workspace: •Oomph import (without any p2 or targlet tasks, just the project import and working-set tasks) takes about 50 seconds •total time to import, including the Oomph import and initial build ('build automatically' is on): 4 minutes 15 seconds. The build finishes with •many projects having build errors. The vast majority of these are of the "project cannot be built until its prerequisite xyz is built" kind, with a only a handful of projects being the roots of these dependency graphs •most of these trace to an erroneous bin/ library entry in the classpath of the org.eclipse.papyrus.view.properties project. Fixing that kicked off a workspace re-build that took 15 seconds to clear up almost all problems. Fixing the same misconfiguration in a few other projects (notably Model Explorer related and the properties customization plug-ins) fixes all remaining build problems. So, implementing the Maven nature should also make these fixes to ensure a clean first build experience for developers Maven Nature in Papyrus Source Projects of12 13
- 14. DIA Agency, Inc. •is comparable to the roughly 3 minutes and 30 seconds time to import the same projects without the Maven nature (as currently) Conclusions This initial experiment seems to demonstrate clearly that converting the Papyrus source projects to the Maven nature will not adversely affect the performance of Eclipse for the Papyrus developer community. Moreover, it will have the benefit of unifying the integration of EMF model generation and possibly other custom build steps in both the developer and Hudson environments. Other benefits remain unclear (the process of converting projects uncovers additional project configuration problems, but this may be a one-time thing). Most importantly, the majority of developer workflows are not perturbed: •Oomph setup and incremental updates, for example to pull in a new nightly Papyrus build under a minimalist source workspace •incremental workspace builds; bundle manifest and other metadata changes; source code editing •testing with run-time workbench and JUnit launch configurations That said, it is perhaps rather late in the Mars release cycle to introduce such a significant change in the configuration of the Papyrus codebase to the developer community. Doing this conversion in the first milestone post Mars for the next annual release (on the master branch, not the Mars maintenance branch) may be advisable. Maven Nature in Papyrus Source Projects of13 13