Staging and Promotion

Staging
and promotion is designed to allow projects to be deployed to independent, temporary repositories. The contents of this repository is then evaluated and eventually either promoted or deleted. When
promoted, it is merged in with a target repository and all appropriate metadata is updated. This implementation of Staging and Promotion requires no changes on the Maven side other than updating your distributionManagementUrl.

Staging works by defining a new url that is used for all deployments. Profiles can be created in Nexus to determine how artifacts are staged. The profile specifies name and id templates for the temporary staging repo, which groups will automatically contain the repository, and who to notify when the staging is complete.

When a new deployment occurs, Nexus selects the profile based on the artifact type (snapshot or release) and the repository target (regex on the artifact path). Nexus then notes the user and ip doing the deployment and makes a new staging repository instance to hold the artifacts from this build. This allows multiple parallel deployments to occur simultaneously and the artifacts are placed in independent repositories. When the staging is finished, the user goes to the Nexus UI and closes the repository to new artifacts. An authorized user can then later decide to promote or drop this repository. When the promotion is selected, the user selects a destination repository, and Nexus then merges the contents automatically, updating the metadata along the way.

Future enhancements include a maven side plugin to automatically start and stop staging based on a unique build id, workflow extensions for automatic handling of staging promotion and verification (ie tie into CI and other metrics), and vote based decisions on promotion.

Procurement

Procurement is designed to create a secured, hosted repository that contains all artifacts needed for a build that is controlled to allow only approved artifacts. Enterprises currently typically operate in one
of two modes:
1) Developers have free access to repositories, but QA, CI and official builds are performed against a manually populated hosted repository.
2) Developers are completely blocked from external repos and only have access to a manually populated hosted repository. We’ll call this “Ask First”

In either mode, the artifacts must be manually loaded into the sanctioned repository. This is tedious and error prone. Many of the artifacts are required only for the build itself (org.apache.maven.*) yet they must still be manually populated.

The Procurement repository is a new type of repository. The repository is associated with another internal target, either a repository or a group. An authorized user is able to browse the internal target and selectively approve artifacts. They may be approved by groupId with wildcard, group and artifact, and group, artifact and version. When a request is made against the repository the current
contents are served. If the requested artifact doesn’t exist, it will be retrieved from the target only if the rules explicitly allow it. Allowing the wildcard solves the problem of constantly having to approve build only artifacts like org.apache.maven.*

In a normal scenario, the developers may be configured to build against a group containing their required repositories. The qa and/or the CI system would be configured against a Procurement repo using the developer repository group as the internal target. This would allow developers easy access to the artifacts required, but still provide control over the artifacts used in official builds. If desired, the developers may also be shielded by the Procurement repository, which would put them in “Ask first” mode where things must be pre-approved before they can use them.

The Procurement repository is treated like a hosted repository in a way that excludes it from scheduled tasks that may remove artifacts accidentally. This is done to maintain a higher level of integrity and reproducibility.

Future enhancements include the ability to automatically approve based one license, the ability to approve an artifact and all transitive dependencies in a single operation, and taking a physical snapshot of the state of a Procurement repo for later reproducibility and tracing.

LDAP Authentication & Authorization

The LDAP functionality supports two modes:

1) User/Pass is stored in LDAP only. This means that the normal user to role association is preserved and we only hit ldap for authentication.

2)User/Pass and group is stored in LDAP. The groups are automatically to a Nexus role by id.

P2 Repository Proxying
Nexus has the ability to host P2 proxy repositories. This allows traditional caching and proxying of P2 artifacts to save bandwidth and provide internal stability when provisioning P2 based artifacts such as Eclipse.

Future enhancements include P2 Repository grouping and Hosting.

We are wrapping up the testing of these new Pro plugins along with Nexus 1.2. The Professional plugins will be released to a handful of selected Beta clients first for testing and user story validation. Once that is successful, we will begin a much wider Beta program prior to the GA release, which we anticipate would occur in the Nexus 1.3 timeframe of Mid-January ‘09.

If you are interested in partaking in the Nexus Pro Beta program, please sign up here.

The following scripts are in Ruby and they use the REXML XML processor and the Net:HTTP library. Both of these libraries are available in the Ruby standard library. I tested these Ruby scripts using the latest JRuby release. If you want to try these scripts out on your own, they will run in any standard Ruby interpreter without requiring any extra RubyGems.

The Nexus REST API

The UI of Nexus is written in ExtJS and it makes heavy use of AJAX callbacks to REST services. In fact, if you load up Nexus in a tool like Firebug, an extension for Firefox which allows you to trace all network activity, you will see that almost every action in Nexus triggers a call to a REST service. We’re going to focus on just two services in this post: list repositories and searching for artifacts, but you should know that there are hundreds of things you can do to Nexus via the various REST interfaces that are available. In fact, you could create a custom UI tailored to your own needs that interfaces with Nexus via this REST backend. In addition to the REST services in the core Nexus installation, you can extend Nexus and add in your own REST services via a Nexus plugin. The Nexus UI is completely decoupled from the set of services Nexus provides, and the services which Nexus provides can be extended.

The Nexus REST Services are documented here. In this post, we’re going to be writing scripts that hit the public instance of Sonatype Nexus at http://repository.sonatype.org, our REST URLs are going to look something like this: http://repository.sonatype.org/service/local/data_index?q=maven. If you were attempting to hit the services on your local Nexus installation, your REST URL would look more like this: http://localhost:8081/nexus/service/local/data_index?cn=HibernateDaoSupport. These REST URLs follow the pattern: “http://{host}:{port}/{context}/service/{instance}/{service}?{query_params}” where the {instance} will almost always be “local”.

If you would like to download the sample scripts in this blog post, you can download this ZIP file which contains four Ruby scripts: http://books.sonatype.com/tutorial-files/nexus-rest-ruby.zip

Listing Repositories

The first script simply lists all of the repositories in a Nexus installation. Here it is:

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/repositories'
resp = Net::HTTP.get_response( URI.parse( url ) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//repositories-item" ) do |r| puts "#{r.elements["name"].text}(#{r.elements["id"].text})" puts "\t" + r.elements["resourceURI"].text + "\n\n"
end

This script sets the pattern for the scripts to follow. We construct a URL in the script (a more general script would read this from a config file), then we make the request using Net::HTTP and we parse the results with REXML by passing it to the Document constructor. At this point, I use XPath to select all of the repositories and I print out the name, id, and the URL of the repository as served by Nexus.

This post is simply a pointer to the service and a quick demonstration so I’m not going to dive into the meaning of every single element in the XML document returned by the repositories service. If you are interested in see the full set of elements that are available, load up the results of this service in a web browser by click on this: http://repository.sonatype.org/service/local/repositories.

Performing a “Quick Search”

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/data_index?q='
resp = Net::HTTP.get_response( URI.parse( url + ARGV[0]) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r| puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This script follows the pattern of the previous script to list all repositories with the exception that it reads an argument from the command line. This script performs a quick search by hitting the data_index service and passing in the q parameter. This script simply prints out the groupId:artifactId:version of all the artifacts located.

If you would like to see an example of the XML that this service produces click here: http://repository.sonatype.org/service/local/data_index?q=activemq. In the full results, you’ll see more information such as the number of search hits available, the resource URL for each artifact found, and the context (or repository) in which the artifact is available.

Searching by Class Name

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/data_index?cn='
resp = Net::HTTP.get_response( URI.parse( url + ARGV[0]) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r| puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This script is almost exactly the same as the prior script that performed a quick search. The difference in this script is that instead of passing the q parameter, this script passes the cn parameter. Passing the cn parameter causes Nexus to search for artifacts which contain classes that match the given value. The results are going to look the same as the quick search query script. To see the XML yourself, search for all artifacts which contain a class named HibernateDaoSupport: http://repository.sonatype.org/service/local/data_index?cn=HibernateDaoSupport.

Performing a GAV Search

require 'net/http'
require 'rexml/document'
include REXML
 
url = "http://repository.sonatype.org/service/local/data_index?g=#{ARGV[0]}&a=#\
{ARGV[1]}&v=#{ARGV[2]}"
resp = Net::HTTP.get_response( URI.parse( url ) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r| puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This final script takes three command line arguments: groupId, artifactId, version, and it performs a GAV (groupId, artifactId, version) coordinate search over the repository.

NOTE: Maximize Video to Full Screen Mode to see Details

If you would like to manage your own instance of Nexus and index your own Hosted repositories, you can download an install Nexus from http://nexus.sonatype.org. If you manage your own instance of Nexus, you can proxy other Maven repositories (including Central) and you will have a place to publish your own organization’s artifacts.

When you host a Nexus instance and you proxy a remote repository, Nexus downloads a Nexus repository index, this index is what enables Nexus to quickly search the contents of a Maven repository. Downloading this small repository index is all Nexus needs to do to search the entire repository. For more information about Nexus, see http://nexus.sonatype.org.

Then type in part of an artifactId or a groupId, and you should see a result screen that finds all matching artifacts across several repositories:

This is a quick way for you to locate the available versions for a particular version and find out what the appropriate groupId, artifactId, and version is for a dependency. If you need more detail, you can click on the POM link to the right of each entry to view the POM for that paricular artifact, and if you would like to download the artifact, click on the artifact link.

If you would like to browse the repositories, click on the browse repositories link on the left-hand side of the window. In the list of repositories, you will see both groups and repositories. Nexus groups are collections of repositories exposed from a single URL.

Looking for artifacts? Go to http://repository.sonatype.org.

In the next episode… search for JARs by a hash or an MD5 fingerprint… and searching for class names.

Hi,

About a month ago I did a talk about core Maven things here at our local JUG in Stockholm, Sweden. m2eclipse wasn’t the main focus of the talk, but I did use it for some examples. Mainly what was used was the pom editor and the Nexus indexes.

Even though I think the actual topics of my talk are highly interesting, most of the questions afterwards were regarding “the plugins I used in Eclipse”. Not only did some people think I hade created my own best of breed mixture, but also were a lot of people very impressed of the actual Maven support. I spoke to some people who had looked at Eclipse support some time ago (6 months - a year) but had then found it very poor. These people said that they would have a look at m2eclipse again now after my talk.

So, thanks and keep up the great job!

/Anders

Thanks for the great feedback Anders. I think it’s good to see users’ sharing their experiences about m2eclipse and Nexus because it does make the choice for the next set of users much easier.

ModelProperties are the building blocks and consist of a URI and a value. If you want to create a hierarchy (or a node), you would specify something like:

mp("http://myorg.org/model/foo", null)
mp("http://myorg.org/model/foo/bar", "fooey")

In XML, this would correspond to:

 <foo> <bar>fooey</bar> </foo>

If two of the same properties are stacked, say “http://myorg.org/model/foo/bar”, then the top most property value will be used. This allows for a simple linearized inheritance, where the most specialized model overrides values of its parent model. But what happens to collections, like dependencies, that need to be merged as we move up the hierarchy of parent poms?

That’s where the concept of the ModelContainer comes in. The first thing to do is to specify #collection in the URI: http://apache.org/maven/project/dependencies#collection.

This tells the ModelTransformer implementation instance not to override the property, as it is part of a collection. Instead, the framework just to adds the property to the general collection. Obviously, the framework doesn’t know anything about how to handle the specifics of these URI collections, as that is a model specific rule. Thus the framework must delegate the decision, calling all the ModelContainerFactories that is has been initialized with, asking each if they can provide a container to handle the property.

 ModelDataSource source = new DefaultModelDataSource(); source.init( props, Arrays.asList( new ArtifactModelContainerFactory(), new IdModelContainerFactory() ) );

A ModelContainer implementation is rather simple. The framework just passes in all like model containers and asks what it should do. A ModelContainer is required to implement the following method:

 ModelContainerAction containerAction( ModelContainer modelContainer );

A ModelContainer implementation instance will know its own model properties (say version, groupId and artifactId) and decides what action to take in relation to the specified modelContainer, which is passed to it by the framework. The returned ModelContainerAction is an enum that specifies an action: no operation, delete or join of containers.

For example, if either the groupIds or versions are different, the container would return a NOP, telling maven-shared-model to leave the specified container in the collection. If the artifactIds, versions and groupIds are all the same, then the container would tell the framework to join the containers. That’s where the implementation of ModelDataSource comes in. It has join, delete and query methods for handling the operations on ModelContainers. If the ModelContainer implementation told the framework to join model containers, the framework would then take both model containers and invoke: ModelDataSource.join(ModelContainer a, ModelContainer b)

Out of the box, you can use the ModelTransformer for transforming XML to a list of model properties, but nothing would stop someone from plugging in their own transformer for virtually any other format. Thus you can used maven-shared-model for inheriting any data format according to a general rule set that you can customize.

Clearly the component name, maven-shared-model, is not appropriate, as it can be used independently of Maven and does not dictate the use of the Maven model. After the first alpha release of Maven 3.0, I’ll be creating a separate project, independent of trunk, for maven-shared-model.

http://weblog.infoworld.com/openresource/archives/2008/11/marketing_maven.html

I am really excited to be working with Mark. More good news to come!

Oleg needed a version of project builder that didn’t require a bunch of dependencies from the Maven trunk, as he designed Mercury to be a standalone module, resolving of any type of artifact, not just Maven specific ones. The bulk of the project builder work was already standalone and isolated into maven-shared-model, a component that handles inheritance and interpolation of generic models (including XML ones). So I stripped out some of the interfaces and a few classes from the maven-project component, which had unneeded tie-ins with the rest of Maven core.

Next came the implementation. Mercury has an interface called DependencyProcessor, with the following method that I needed to implement (MavenDependencyProcessor):

public List getDependencies(ArtifactBasicMetadata bmd, MetadataReader mdReader)

The point of this method is to build a full project model, including inheritance, interpolation and profiles, returning the list of ArtifactBasicMetadata dependencies of the processed model to Mercury.

The ArtifactBasicMetadata is a simple value class, giving the MavenDependencyProcessor instance information like version, groupId, artifactId of the artifact for which to process the returned dependency list. The MetadataReader allows the MavenDependencyProcessor instance a way to call back into Mercury to get the parent of the project.

Since the project builder uses basic property lists to store all Maven properties, converting from dependencies to ArtifactBasicMetadata couldn’t be simpler: The core of the conversion is:

List modelProperties = container.getProperties();
ArtifactBasicMetadata metadata = new ArtifactBasicMetadata();
for ( ModelProperty mp : modelProperties )
{ if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.groupId)) { metadata.setGroupId(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.artifactId)) { metadata.setArtifactId(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.version)) { metadata.setVersion(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.classifier)) { metadata.setClassifier(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.scope)) { metadata.setScope(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.type)) { metadata.setType(mp.getValue()); } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.optional)) { metadata.setOptional(mp.getValue()); }
}

One problem we did encounter was how to handle passing in system and user properties for interpolation and profile activation to the MavenDependencyProcessor, as these properties are currently passed by the MavenEmbedder to the project builder. This requires that Maven core explicitly initialize the MavenDependencyProcessor, passing in all properties through the MavenDependencyProcessor constructor.

Mercury integration is pretty exciting because it will allow us to hemorrhage a lot of resolving code which we have to use in the core, as well as providing a deterministic resolver. Full Mercury integration is not slated for the Maven-3.0-alpha1 release and will likely be a core focus of the alpha-2 release. Initial integration looks promising, as Mercury is already able to process classpaths and dependencies from POMs.

After starting, I found there was little documentation on how to use the command line to build Android apps, leading me to dissect the build.xml file to learn how the command line options worked.

After that, it was just a matter of creating a custom lifecycle for the packaging type ‘android:apk’. The plugins largely consisted of translating POM parameters to the command line, similar to Apache NMaven, undergoing incubation at the ASF.

A sample POM looks like:

<project> <modelVersion>4.0.0</modelVersion> <groupId>org.jvending.masa</groupId> <artifactId>maven-test</artifactId> <version>1.0.1</version> <packaging>android:apk</packaging> <name>maven-test</name> <dependencies> <dependency> <groupId>android</groupId> <artifactId>android</artifactId> <version>1.0</version> </dependency> </dependencies> <build> <sourceDirectory>src</sourceDirectory> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> <plugin> <groupId>org.jvending.masa.plugins</groupId> <artifactId>maven-dx-plugin</artifactId> <extensions>true</extensions> </plugin> </plugins> </build>
</project>

So by specifying the packaging type as ‘android:apk’ and the sourceDirectory as ’src’, you can build your android project with Maven. If you type ‘mvn install -Dmasa.debug’ the plugins will also deploy the apk to the G1 device over USB or to the emulator.

For the latest SDK, I included the packaging type: ‘android:apk:platformTest’
that will deploy the apk to the device and run the unit tests on the target device.

At the time, the Android source was not open-source, so MASA provides a build script for importing the android.jar into the local repo. Now that Android is open-source, I’ll be looking to deploy the android.jar into the Maven repo to avoid this additional step and then getting a release of MASA deployed as well.

One area that I still need to work on is the delayed signing of apks, so expect to see future updates.

• A section in the installation chapter about upgrading from Maven 1 to Maven 2. I’m constantly getting questions from people about customizing a Maven 2 build by string goal names together on the command-line so I put some language into the installation chapter that points people at the Maven Lifecycle. The Maven Lifecycle really is the thing to learn first in Maven, if you figure out how the Maven Lifecycle works and if you become familiar with the Default lifecycle, Maven 2 will make more sense. This new section is small, but it points people toward the Maven One plugin and configuring a legacy repository. If you are upgrading from Maven 1 to Maven 2, let me know if this section is what you needed.
• Arun Kumar submitted a list of typos and grammar errors throughout the Introduction. A number of typos were fixed in the introduction.
• A number of typos were fixed in the Assemblies chapter.
• A number of typos were fixed in the Profiles chapter.
• A number of typos were fixed in the Writing Plugins chapter.

To transform from an XML format to the property format, you can take any stream containing data in XML and pass it the static method ModelMarshaller.marshallXmlToModelProperties. In the case of the Visual Studio’s C# project file, it would look something like:

("http://microsoft.com/Project", "null")
("http://microsoft.com/Project/ItemGroup", "null")
("http://microsoft.com/Project/ItemGroup/Reference", "null")
("http://microsoft.com/Project/ItemGroup/Reference#property/Include", "NGenerics, Version=1.3.0.0, Culture=neutral, processorArchitecture=MSIL")
("http://microsoft.com/Project/ItemGroup/Reference/SpecificVersion", "False")
("http://microsoft.com/Project/ItemGroup/Reference/HintPath", "..libNGenerics.dll")
("http://microsoft.com/Project/ItemGroup/Reference", "")
("http://microsoft.com/Project/ItemGroup/Reference#property/Include", "System")
("http://microsoft.com/Project/Import", "")
("http://microsoft.com/Project/Import#property/Project", "$(MSBuildBinPath)Microsoft.CSharp.targets")

The next step is to transform from the above properties to the canonical format that Maven understands. This requires creating rules that rename (and possibly reorder) the URI elements of each property. For example, to create the dependency section of the pom we would transform assembly references from the C# project file to a Maven project dependency:

List p = new ArrayList();
p.add(new ModelProperty(ProjectUri.xUri, null));
p.add(new ModelProperty(ProjectUri.Dependencies.xUri, null));
for(ModelProperty mp : modelProperties) { if(mp.getUri().equals(CSharpUri.ItemGroup.Reference.include)) { p.add(new ModelProperty(ProjectUri.Dependencies.Dependency.xUri, null)); p.add(new ModelProperty(ProjectUri.Dependencies.Dependency.groupId, mp.getValue())); p.add(new ModelProperty(ProjectUri.Dependencies.Dependency.artifactId, mp.getValue())); }
}

When the transformed list of properties above are fed back into ModelMarshaller.unmarshalModelPropertiesToXml, we would see an XML snippet similar to

<dependencies> <dependency> <groupId>System</groupId> <artifactId>System</artifactId> </dependency>
</dependencies>

In this way, we can construct an entire Maven pom from a C# project file or vice versa. You could even translate a hierarchy of C# project files into a multi-module Maven build and have the Maven project builder process the inheritance and interpolation of the C# project files.

Maven handles transforms similar to a messaging bus. Say you build the bidirectional transforms from VS 2005 project files to the Maven canonical format. Now say someone else builds the transforms for VS 2008 (or 2003) project files. You can now do transforms to and from VS 2005 and 2008 project files with no additional work.

1. Removed the references to the Geronimo implementations of servlet-api and jsp-api. There was once a time when this was very relevant. Sun wasn’t distributing the servlet-api JARs from central and Geronimo was the only implementation in the repository. Ever since the Glassfish team started publishing the javax.servlet:servlet-api JAR there is no longer a need to tell new users to substitute the Geronimo JARs for the Servlet and JSP spec JARs. I’ve kept the appendix which lists alternatives, but I don’t think it makes sense (especially for beginners) to spend value intro time talking about the Geronimo implementations.
2. I added a section at the end of Chapter 4 which shows you how to configure the assembly:attached goal using the executions elememtn in your plugin config. String goal names along on the command-line is an acceptable way to run a plugin goal, but it ignore a critical part of Maven 2. I’ve had many users ask me questions that suggest that they don’t know how to bind a goal to a phase in a POM. I think that the executions element is important to introduce earlier in the book. In general, I’m seeing that many users haven’t grok’d the Maven Lifecycle. This is my fault, and we’re going to emphasize more Lifecycle explanation both on this blog and in future versions of the book.
3. In Chapter 6, I had some inconsistent references to the groupId. I forgot to modify references to the com.sonatype.maven groupId to org.sonatype.mavenbook.ch06 - apologies for the oversight.

PS: Anders Hammar: I see those reports. Thanks for commenting on them. Commenting on an issue that you think has fallen to the bottom of the stack puts it on the top.

I have looked into building Neoclipse using Maven, but at the moment I’m not convinced there are more advantages then disadvantages in this case. Building Eclipse plugins and products has quite many quirks. With increasing use of product configuration options, the number of build quirks increases as well.

There seems to be no really automated build system for Eclipse plugins/products, even when using maven you have to install and update the dependencies manually (the Eclipse artifacts are not deployed to a maven repository).

Sonatype is busily working to address these gaps, and we already have some solutions you can use to start building your Eclipse plugins in Maven. Igor Fedorenko has been working on Tycho and Maven to create a system to facilitate OSGi and Eclipse development in Maven, and m2eclipse is currently built using Tycho. To prove that Tycho is a viable path for build the Neoclipse plugin, I took some time yesterday to generate Maven pom.xml files with Tycho and build an update site for Neoclipse.

A Basic Summary of Tycho

The general idea of Tycho is that it acts as a bridge between Maven and OSGi. It supports a number of usage scenarios and is under active development. It can be used in a manfiest-first mode which builds a Maven POM from an OSGi manifest and it can also be used in a pom-first mode in which Tycho is responsible for generating an OSGi manifest from a Maven POM. Future plans include the ability to build a project usin Maven/Tycho without requiring the presence of a Maven POM, and there is active work under way to allow Nexus Maven repository manager to interact with Eclipse update sites and p2 repositories, that would allow Tycho to consume artifacts from those repositories. If you are looking for a more in depth explanation of Tycho, check out the Tycho Project Overview.

If you are looking for a solution to automate Eclipse builds with Maven, look no further, Tycho is your answer. In this post, I’m going to walk you through the process of configuring a Maven build for a real Eclipse plugin. This process involves the following, high-level steps:

1. Configuring Eclipse
2. Downloading Tycho
3. Checking out Eclipse Plugin Source from Subversion
4. Generating Maven POMs using Tycho
5. Packaging Your Eclipse Plugin and Generating an Update Site with Maven

At this point in the development of Tycho, you’ll have to point the system at an installation of Eclipse, your targetPlatform. Let’s get started.

Configuring Eclipse

Tycho currently builds Maven pom.xml using a local installation of Eclipse which should contain the Plugin Development Environment (PDE) and other Eclipse components your specific plugin depends on. Future versions of the Tycho product will likely interrogate remote p2 repositories and Eclipse update sites, but for now we have to configure a local Eclipse Target Platform with the necessary components. For this particular example, you’ll need to install Eclipse 3.4 and download a few plugins that the Neoclipse plugin depends upon. For more information about what Neoclipse requires, see the Neo4J Wiki Page for Neoclipse. These steps are also needed when you need to work with Neoeclipse code using Eclipse PDE. You need to:

1. Download and install Eclipse 3.4 if you don’t already have it installed.
2. Neoclipse has a graphical editor component which relies on GEF. To build the plugin, you’ll need some of the Draw2D components. The easiest way to install this particular plugin is not via the standard Eclipse update manager. You should download GEF ALL distribution for Eclipse 3.4 from the GEF Download page. The direct link for the GEF-ALL distribution is here.
3. To install the GEF-ALL distribution, unpack the ZIP file to a directory and then copy that directory to your Eclipse installation. This will install all the necessary features and plugins that neoclipse depends on.
4. As usual, I would recommend installing both the m2eclipse plugin and the Subclipse plugin. These are two plugins I almost always need:
   1. m2eclipse update site: http://m2eclipse.sonatype.org/update (or if you want to use the latest stable DEV build http://m2eclipse.sonatype.org/update-dev)
   2. Subclipse Update site: http://subclipse.tigris.org/update_1.4.x

I’m on a Mac, so I usually install my eclipse distribution in /Applications/eclipse. For the remainder of this document, if you see /Application/eclipse, you should substitute your own eclipse installation directory.

Downloading and Configuring Tycho

At this point, you have Eclipse installed and configured with the necessary plugins to build the Neoclipse plugin. Now you need to download and configure Tycho. Tycho contains a development version of Maven 3.0 that has been extended to allow for Tycho to interact with the the OSGi components and format of the plugins directory of your Eclipse plugin. In the future versions you could use regular Maven 3.0 and use Tycho as any other Maven plugin.

1. Download Tycho from http://docs.codehaus.org/display/M2ECLIPSE/Tycho+builds. I downloaded the latest DEV build, and I installed it in ~/programs/tycho - Tycho includes a development SNAPSHOT of Maven.
2. Set M2_HOME to point to ~/programs/tycho. If you don’t do this and you already have Maven 2 installed, you are going to run into errors and issues if you try to run either Maven or Tycho. I usually set M2_HOME to point to my Tycho installation by running “export M2_HOME=~/programs/tycho” in bash.

Once Tycho is configured, you are ready to generate Maven POMs from your Eclipse plugin.

Generating Maven POMs with Tycho and Building an Update Site

The demo script for Tycho uses a contrived a demo project called tychodemo.zip which contains a simple Eclipse plugin. I wanted to demonstrate that Tycho is ready to be used today with a real Eclipse project. This project is the Neoclipse plugin for Eclipse that allows users to interact with a Neo4J database in the Eclipse IDE.

1. Change directories to a working directory. I usually do all of my coding in a ~/svnw directory which is segmented by focus. So, I check out Open Source projects like neo4j in a ~/svnw/os directory: cd ~/svnw/os
2. Checkout the source code for the neoclipse Eclipse plugin with Subversion: svn co https://svn.neo4j.org/components/neoclipse/trunk neoclipse
3. Change directories into the neoclipse directory: cd neoclipse
4. Generate a set of Maven POMs from the Neoclipse source using Tycho: ~/programs/tycho/bin/mvn -X org.codehaus.tycho:maven-tycho-plugin:generate-poms -DgroupId=tycho.demo -Dtycho.targetPlatform=/Applications/eclipse

At this point, you’ll have pom.xml files in all of your plugins and at the top level. Tycho has generated a multi-module Maven project that references components in your targetPlatform, specified using “tycho.targetPlatform” property on the command line. Tycho doesn’t need Eclipse to be running, all it needs is access to the Eclipse directory so that it can resolve components such as the Eclipse Plugin SDK at the GEF plugin installed in the previous section.

To build the update site, run: ~/programs/tycho/bin/mvn package -Dtycho.targetPlatform=/Applications/eclipse You should then have an update site in ./org.neo4j.neoclipse.updatesite/target/site

To avoid specifying tycho.targetPlatform property every time, it could be added to your settings.xml. For example:

... <profiles> <profile> <id>tycho.default</id> <activation> <activeByDefault>true</activeByDefault> </activation> <properties> <tycho.targetPlatform>/Applications/eclipse</tycho.targetPlatform> </properties> </profile> </profiles>
...

UPDATE (9:21 AM CST): Eugene Kuleshov just sent in a bunch of edits and improvements which were incorporated. Removed the section about adding the repository as it should already be in the public group hosted on Sonatype’s Nexus repository.

Back in May, Jason started talking to me about adding in mixin support for the pom, and it became clear that we needed to reconsider the Maven 2.0 design of project builder so that we could support such features. Maven 2.0 took the approach of merging trees of information (XML nodes), a perfectly reasonable approach for single inheritance. For things like mixins and multiple pom inheritance, linearizing the information proves a much simpler approach. In this post I discuss the ramifications of linearizing POM information to support multiple inheritance for POMs.

To linearize, of course, means that we have to flatten the XML nodes. I did this by creating a list of model properties, consisting of a URI and a value. So for example, the version would be modelProperty(”http://apache.org/maven/project/version”, “1.0″). The Maven 3.0 project builder creates a rather large list of model properties, stacking them, the child model properties being on top, while each of the parent model properties are placed lower in the list. Inheritance is now just a matter of sorting and removing model properties according to a set of rules.

To see how easy mixins is under linearized inheritance, let’s say we have project A and project B, where project B is the child. Under single inheritance, Maven applies the sorting rules. For a mixin, we just place its model properties below project B’s properties in the list and above those of project A. Maven applies the same sorting rules as single inheritance and doesn’t care whether it’s dealing with a parent model, a mixin or even a second parent model.

I’ve created a detailed specification of how Maven builds the project model, which I will be releasing shortly, giving all the precise rules Maven 3.0 follows for project construction.

What’s the best option when deploying neo4j code examples
using Maven? It’s really not of any interest to deploy the artifact
itself, it’s the sources (if possible with some exclusion of resources)
that we want.

Anders’ requirements: create a project which packages itself, wire the appropriate goal into the lifecycle. He’s headed in the right direction, the Maven Assembly Plugin, but he might not know about the pre-defined assembly descriptors as they are somewhat hidden on the Apache Maven site. Since other projects might have similar requirements, here’s a quick explanation of all the moving parts…

What is the Maven Assembly Plugin?

The Maven Assembly Plugin allows you to create an archive with an arbitrary structure. If you have a project which needs to produce a ZIP file with a bin/ directory an a set of shell scripts, or if you want to create an archive which contains a particular project’s bytecode and all the runtime dependencies, Maven Assembly Plugin is the place to look. To generate an archive, you’ll need to do a few things:

• Include the Maven Assembly Plugin in your project’s pom.xml file.
• Configure an assembly by referencing a built-in assembly or define your own assembly.
• (optional) configure the lifecycle to bind the assembly goal to the package phase of the lifecycle.

What is an Assembly?

The Maven Assembly Plugin produces an assembly of files and folders using an Assembly descriptor which describes the contents of a directory. You can use one of several predefined assembly descriptors which are built into the Maven Assembly plugin, or you can define your own assembly by configuring your assembly in an assembly descriptor. An assembly descriptor is an XML file which selects files and directories to include in the final, assembled artifact.

Just what sort of “assemblies” can you create on your own with an Assembly descriptor? Just about anything, a collection of .class files and all of your runtime dependency JARs? Sure. The assembly descriptor is flexible enough to describe just about anything. If you want a full explanation of the power of Maven Assemblies, read the Assemblies Chapter of the Free Maven Book or Chapter 4 which introduces a built-in assembly descriptor.

A Project Which Assembles Itself

Back to Anders’ requirements, he wants a project that can package itself up as an example project. He also wants this assembly to be created when he runs mvn deploy. To do this, he can use the built-in project descriptor. Adding the following plugin configuration to any project will configure this built-in project descriptor to create an assembly of a project itself (a self-replicating project) and the executions section will attach the “assembly:attached” goal to the package phase of the lifecycle.

 <build> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>project</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>examples</id> <phase>package</phase> <goals> <goal>attached</goal> </goals> </execution> </executions> </plugin> </plugins> </build>

Go ahead, try it. Run mvn package or mvn deploy and you’ll end up with an assembly in ${basedir}/target. If you want to customize the name of the final product, use the finalName element in the build section of your POM.

The Lifecycle

My next post will likely be on the lifecycle, I’ve had a few questions from people over the past few weeks about running specific goals directly from the command-line. It appears that many people haven’t fully grok’d the Maven Lifecycle. Many people grew familiar with Maven 1, the way that people customized a build in Maven 1 was to string along a number of goals. If you needed to run a specific goal, you would just craft a long command line. In Maven 2, you should shy away from running specific goals on the command line, and you should configure executions (see above). This is the more natural way to customize a build, think of it as hooking your own project’s customized goals to the default lifecycle. If you find yourself running “mvn deploy assembly:assembly” often, you should take a look at the executions element in your plugin configuration.

For more information about the Maven 2 Lifecycle, read Chapter 10: Lifecycles.

The War has been successfully tested on Resin, Tomcat and Jetty but is considered Beta. It is currently known to not work on Glassfish due to that container’s conflict with Log4j. You can find more information regarding the war support at Nexus-58 and Nexus-1055. Please report any issues to the nexus-user list or Jira.

Also included are several enhancements aimed at improving the user experience:

Index Browsing

Previously the browse repository screens only included items already cached by Nexus. The new screen allows you to browse the tree view for a remote repository when the remote index has been downloaded:

Log Viewing

Tail functionality has been added to the log screen:

Admin can directly set a user password

It is now possible to set a password directly without requiring an SMTP server:

Many other improvements and bug fixes are included. The full list can be retrieved from Jira.

Download your copy of Nexus here.

• Added a reference from the Lifecycle Chapter to the Writing Plugins Chapter. I had incorrectly stated that we were going to illustrate the process of creating a custom lifecycle in this chapter. Custom lifecycles and packaging types are not introduced until the Writing Plugins Chapter. An XREF has been inserted into the Lifecycle chapter.
• Changed the installation directory to /usr/local/apache-maven-2.0.9 from /usr/local/maven-2.0.9. This might seem like a small change, but it brings the documentation more in line with what the archive will produce when unpacked. I have not changed the book’s installation instructions to match the install directory on the Maven site. The Maven site suggests that the correct directory for Maven is /usr/local/apache-maven/apache-maven-2.0.9… I’d rather install multiple versions in /usr/local and then use symlinks.
• Added instructions for the installation of Maven 2.0.9 using MacPorts.
• Removed some confusing language in the Settings Details Appendix that talked about “two different kinds of artifacts”
• Added a note to Chapter 8 that references the Optimization chapter. Some people, already familiar with the dependencyManagement element, were getting a bit disturbed to see a top-level POM without this element. I’m leaving the dependencyManagement element to the Optimization chapter to illustrate the progression from a simple POM to a highly-optimized POM.
• Changing the Version Number of the Book

Many of these changes may seem trivial: fixing dangling references, small typos, version numbers. I tend to think that these small changes are critical to creating a book that doesn’t create the sort of friction the community has grown accustomed to. Too often, you’ll be reading documentation for an open source project and it will refer to an as yet unfinished or yet to be realized document, this can be one of the most frustrating experiences…. “For more information about X see the (unwritten) Document Y…”

Thanks again to all of the people paying attention and submiting errata through both the O’Reilly production staff and GetSatisfaction.

To check it out, we installed Nginx in parallel to httpd and once everything looked good, swapped the ports to put it into production. How did it work out? Simply amazing. Since the swap, the loads are averging around .30. This is a quad core machine, so that means that it went from being 28x overloaded to using a third of ONE cpu. Being suspicious, I checked the bandwidth logs and response times. The average response time is 8x faster (no doubt because there’s 3 cpus sipping Mai Tais) and the total throughput shows no discernable change. So we’re now serving the same bandwidth that was swamping the cpus under httpd with next to no cpu with Nginx.

During the ongoing process of tracking the load on Central, we found a surprising number of large organizations out there with a very high connection usage. Please, if you’re using Maven in an enterprise, do us both a favor and install a repository manager such as Nexus. The entire community will benefit because the Central repo will be able to serve the needed artifacts in a timely fashion. You will benefit because it will give you centralized control over your artifacts, less external bandwidth use, less dependency on an external connection, indexing, searching and storage of your internal artifacts, and a ton of other functionality.

• Maven
• m2eclipse
• Nexus
• Hudson

This course is intended for folks who are already somewhat familiar with Maven as the goal of the course is to show you how to setup your development environment where Maven is working optimally for a team setup, your developers are productive with Maven in Eclipse with m2eclipse, where your team artifacts are managed and secured by Nexus, and your build automation is humming along in Hudson.

We don’t have a lot of room available as this was a spontaneous decision given our training material is coming along faster then we expected. If you are interested in participating please email as at training at sonatype dot com and we’ll get you signed up. This is on a first come, first serve basis so if you’re interested I suggest you sign up quickly!

Everyone is welcome to sign up for the training. If you happen to be at ApacheCon or if you live in the area and are interested in Maven related technologies then by all means sign up!

m2eclipse has some compelling features that allow you to quickly import existing projects from Subversion. If you point m2eclipse at a Subversion repository that contains a Maven multi-module project, m2eclipse can checkout all of the source code, scan the code for Maven projects and create Eclipse projects from the information stored in the project POMs.

You can also “materialize” a project from a POM. The screencast demonstrates the materialization of commons-beanutils and then the materialization and import of one of the dependencies of commons-beanutils.

The Sonatype I joined is a loose confederation of top open source developers, tackling really hard problems, validated by some of the largest enterprise software development teams in the world. Jason Van Zyl and Brian Fox have managed to get this team focused, and they are delivering iterations on Maven, Nexus and M2Eclipse with better predictability than most more traditional teams I have worked with. And the users I talk to are excited about and keen to use these products.

So what should I do as the new CEO of this company? Many would choose to impose structure and bring in an executive team than will predict revenues, define corporate messaging, create multi-year product schedules, international expansion and more.

See what Bechtolsheim has to say in this New York Times article: “One mistake a lot of start-ups make with the encouragement of venture capitalists is to hire the whole management team upfront, You have a lot of people twiddling their thumbs and spending money.”

It resonates with me. Customers are already expressing interest, so rather than invest in aggressive marketing and sales, let me make it easy for these customers to transact with us. Rather than having product managers offering R&D their own biased opinions, let me further facilitate the conversation between users and our engineers. And rather than drive to an ‘industry standard’ percentage for R&D costs, G&A, Marketing and Sales, let me minimize all the none-core staffing.

A venture backed company like Sonatype needs to impress investors, and high-profile executives coming from top enterprise software companies are a standard way to do so. I suggest that the VC’s should be impressed instead by our focus on open source to drive product excellence and corporate communications.

The Javadoc is Already in the Maven Repository

If people are looking for JavaDoc, point them at the Maven Repository - they can use m2eclipse to easily browse the Javadoc. You’ve already published the Javadoc and the Source code for Wicket 1.3.5 alongside the Wicket JARs. If you want to point people toward the easiest way to read JavaDoc, point people at this 5 minute video screencast. This screencast demonstrates:

1. Creating a Wicket project using the Maven Archetype
2. Configuring Eclipse to download the Source and JavaDoc archives
3. Browsing Wicket JavaDoc using Eclipse
4. Browsing Wicket Source using Eclipse
5. Starting Jetty using m2eclipse

m2eclipse makes most of this trivial, and it should reduce the pressure on you to have to jump through all the hoops necessary to create aggregate Javadoc in the Wicket distribution. “You want Javadoc? We’ve published all the Javadoc in the Maven repository.”

…but if you really need to generate aggregate Javadoc…

The more direct answer to Martjin’s core problem - generating Javadoc to include in the Wicket distribution: I understand his frustration with the Maven Site plugin, I share it and have gone to great lengths to avoid using it. But, it should be easy to generate aggregate JavaDoc for Wicket. My advice would be to forget the Maven Site plugin and:

• Bind javadoc:aggregate to a phase in the top-level project.
• Include the resulting ${basedir}/target/site/apidocs in your Wicket distribution assembly

I ran javadoc:aggregate for a while to generate aggregate Wicket Javadoc on Minotaur, but I’ve stopped doing this in favor of just configuring m2eclipse to automatically download sources in m2eclipse. I understand that such an answer isn’t going to satisfy people that don’t use Eclipse as an IDE, but nothing stops others from developing tools that make use of the Javadoc and Source code available in the Maven repository. Anyway, hope this helps.

• Decouple major Maven components, making them available as stand-alone building blocks rather then having Maven as as one big monolith, not usable outside of its environment.
  • Artifact - clearly separate an Artifact from its metadata.
  • Repository - convert a repository into active component. It used to give back just pathOf(), now it accepts GAV collections and gives back either metadata or full blown Artifacts.
  • Transport - an API in development.
  • DependencyTreeBuilder main API for dependency graph creation and conflict resolution.
• Decouple container, so that these components are just plain pojos.
• Introduce Jetty-based HTTP/HTTPS and WebDAV transactional transport layer.
  • Asynchronous downloads and uploads. One of the few successful usages of Java NIO in OSS.
  • Transactional operations - all-or-nothing for file sets.
  • Move integrity control into transport layer, upper level components should not care about these details.
• Abstract out metadata cache, provide at least one implementation there.
• Integrate these changes back to Maven 3.x to make it even better platform than it is right now (if it is possible to be better )

Currently Mercury can already be used for accessing repositories, and conflict resolution is under testing.

• my daughter was born,
• we released Nexus 1.0 to general public.

The events were originally planned to be in the opposite order, but Brian was delaying Nexus with his nitty-gritty wishes — you know him –, and my daughter decided to take over. All this happened before almost 8 months. So, sorry for this late report.

But since then, my daughter was contributing a lot to quality, design and implementation of Nexus MRM. Almost at the start, she jumped in the role of “lead gaffer” (it resembles the “superman” pose slightly):

Also, she insisted on using the “thinking hat” very much, when solving almost impossible tasks requested by Brian:

Examined all the tools she had at disposal to solve problems:

She had a lot of “developer to developer”-like consultations with some open-source gurus:

And attended the “How to write server applications to suite all Brian’s needs, 2008″ conference held in Prague (no, she was not having beer, rather just “light refreshing discussion”):

—

Oh, and why was it a “late event”? Since we have a huge shift in timezones (I am GMT+2, Brian is GMT-4 and mr. Cupertino is usually GMT-7, if not at home), and I am the “minority” — at least in head-count. So, the common sense dictated that I should live the “vampire life” and follow their work hours, as I did.

And my daughter Petra was patient, she waited for me to finish, and came at the end of my work day, hence, in late hours in local time.

In Mercury we made an attempt to bring some contemporary ideas into Maven, make Maven not a monolithic build system, but rather a lego-like construction set that allows users to create systems they need. One set of building blocks in particular - implementation of the idea that dependencies is a universal commodity, that exists outside of Maven world. And operation of resolving conflicts on those dependencies can apply to a broader range of dependencies, not just tag in the POM file.

For example - it should be relatively easy to plug in an implementation to read OSGi bundle dependencies. Or read a .properties file with dependencies.

Or - my favorite - keep dependencies in a Drools-backed DSL file. I will try to create this implementation after we integrate Maven POM dependency reader into Mercury.

How is it done:

• Dependency trees and manipulated by DependencyTreeBuilder object
• DependencyTreeBuilder instance is created out of a collection of Repositories (to find Artifacts) and and instance of DependencyProcessor.
• DependencyProcessor abstraction is that externalization component for reading/processing dependencies, making the builder independent of actual dependency information storage

Exapmple code

1
2
3
4
5
6