So Grails 1.1 Beta 2 is out. Rejoice! There are many new features that are detailed in the release notes. However, one of the main ones in this beta is the new support for Maven….Regular readers of my blog will probably be aware of my long history as one who, ahem, is not particularily fond of Maven. Granted I am still not [particularly] fond of Maven, but it is the Christmas period and in the spirit of “why can’t we all just get [along]” I am proud to say that Grails integrates nicely with Maven now

Merry Christmas, indeed.

In the same spirit of “why can’t we all just get along”, I’ve had a similar shift in my opinion of Groovy over the years, I used to talk about Groovy as the wrong choice when compared to something like Jython or JRuby, but after having used Groovy in a number of projects, I ready to admit defeat.  While I’m still interested in Java inoperability with Ruby and Python, it is often much more straightforward to use Groovy than it is to deal with the “impedance mismatch” between something like JRuby and Java.  Groovy has turned into a mature and valuable language that offers tight integration with the JVM thanks to the continued efforts of Graeme and Guillame at G2One (recently acquired by SpringSource).

GMaven provides a great option for people interested in writing Maven Plugins in Groovy and also as a quick way to extend a Maven build with a Groovy script.

Where We Left Off…

As a refresher, here’s what the load looked like at the end.   To summarize, Central was experience load problems on httpd, we subsequently moved to Nginx which fixed the load problem but caused us to regularly saturate a 100 Mbps pipe.   After a few weeks of investigation, we discovered that much of the traffic was due to a misconfigured product that was repeatedly downloading the Nexus index.   We notified the responsible project and blocked access to the index until the problem was fixed.   The end product of all of that work was a Central Maven Repository that had fewer saturation events and which was operating in a consistent 60-80 Mbps range in the middle of a work week.

We still experienced significant amounts of traffic on Monday morning as users fired up M2Eclipse and downloaded the updated weekly index. We tried shifting the index creation from Sunday night to Friday afternoon, to help smooth out some traffic over the weekend, but ultimately this didn’t help much. We ended up having to QOS the transfers of the zip down to something like 30 kbps to keep the rest of the repository available during these loads. This wasn’t a great solution as it significantly increased the time for users to get at the index.

Considering Amazon S3

One of the ideas we had been persuing was to use Amazon S3 to host central. For those of you unfamiliar with S3, it is a Cloud based system for storing and serving data and it is a part of the Amazon Web Services products (EC2, S3, Cloudfront).  The nearly unlimited bandwidth and option to use Cloudfront to bring the data physically closer to the users is a definite bonus.   It was clear to us that we need something that could scale indefinitely as we continue to see increased adoption of tools that rely on the Central Repository.  The drawback with S3, is that we don’t have a direct ability to monitor the traffic and uncover any abuse or misconfigured tools like we do now.   We all agreed that moving to something like S3 was the future.

Despite our frequent pleas to use Repository Managers and not scrape the repo, we continue to find people doing just that nearly every day. The bandwidth on S3 is not free and opening it up without the ability to protect the bottom line from abusers is not a great idea.    Once you start designing systems on an Internet Scale you start to realize that bandwidth isn’t free.

Moving to “The Cloud”: Amazon S3

We decided first to take baby steps and see if S3 could help us with a very specific problem: The index downloads.   Instead of downloading multiple GB of repository data and creating an index locally, we think it makes more sense for people and tools to download a repository index once a week.   This index weighs in at 30 MB, and while that might not seem to be very large at first glance… multiply 30 MB by 50,000 downloads, and you’ll quickly start to serve TB of data.  This is exactly what was happening, and it seemed like an easy target to offload to Amazon S3.

We found a handy Ruby script called S3Sync that we use to synchronize the maven2/.index folder over to a “repo1.maven.org” bucket on S3. Nginx is then configured to send temporary redirects (302) for all /.index/ requests over to S3. By doing this, we are still able to manage traffic to the Index, yet offload the bulk of the data transfer to the S3 network.

So how did it work? Unbelievably well. Take a look at the week prior to the shift to S3 and week after the shift to S3:

Take a look at the Weekly traffic before the switch to S3, realizing that the weekly traffic numbers mask the periodic 100 Mbps saturation that we were seeing in the hourly graphs. A new index was published, and we saw a spike of download activity during the morning of December 1st.   After that our weekly traffic gradually diminishes to a background level of between 40 Mbps and 80 Mbps on an average weekday.     Again, note that even those good days had periods of complete saturation, after blocking the offending tools we saw an improvment, but we wanted to offload the bulk of our index downloads to S3 to gain further improvements.

Now, take a look at the weekly traffic graph after moving the index to S3.   Unless you note the difference in Y Axis scale, this might not seem as impressive.    The number we’re most interested in decreasing is the 95th percentile value, it describes the 5-minute average bandwidth which 95% of our traffic falls within.      If our 95th percentile number is very close to 100 MBps it means that we’re likely to saturate the 100 MBps quite often.   If our 95th percentile is down around 20 MBps, we’re much more likely to have a stable and available repository.  After moving the index to S3, we have a 5x decrease in the 95th percentile from 98.7 MBps to 12.3 MBps.   We went from a weekly bandwith average of 49.28 MBps to 8.24 MBps, and our total transfer for the week went from 3.81 TB to 629.7 GB.

Before moving the index to S3, we were suffering from slow response times and an index download which would take a few minutes during peak traffic.   After we moved the index to S3, response times improved by 300%, and the index download now takes a few seconds to complete.   Just moving the index to S3 yielded dramatic improvements for the Central Maven Repository.

The Result: Greater Speed, Higher Availability

In the 2 weeks since we moved the index to S3 it has served over 4TB of data and 730,000 requests! Even though the bandwidth bill for the S3 service is significant, we estimate it to be half of what we save on the Central connection traffic.    In other words, we’ve increase availability and reduced the overhead costs associated with the Central Maven Repository.

A note of caution: To protect the system from abuse, we may have to change the urls on S3 from time to time, so don’t point directly at the S3 url, continue to request the data from Central and you’ll be ok.

We won’t stop here in finding ways to optimize the repository experience for users. One thing that will be rolled out shortly is Incremental Index support. This will enable tools that use the Nexus Indexer API to grab only chunks of the index that have changed. This should have a significant impact again on the amount of traffic. We also continue to investigate the possibility of leveraging the cloud to host the entire repository so stay tuned.

Note: If you liked the screenshots used in these past two entries, check out Jing Project. Jing is a great free and easy to use tool for capturing screen images or videos and marking them up. Both OSX and Windows versions are available.

We [had] a major crash on Archiva 1.1.3 after our Linux SLES VMWARE partition has been upgraded.  We couldn’t get it to work with Sun JDK 1.5 / 1.6 (seems to be a major problem with Linux/Sun JDK/VMWare) and Archiva didn’t support IBM JDK. I had to switch to Nexus 1.2.0.2 and after 4 hours of works, it just works great. Nexus WAR 1.2.0.2, Tomcat 6.0.18 and IBM JDK 6.0 and everything works now fluently. A big thanks to all Nexus developers and thanks also to made it available as a simple webapp.

While we think it takes less than 4 hours to install Nexus, Henri was probably taking his time to preserve data from his old Archiva installation.   If you are interested in Sonatype Nexus, you can download the 1.2.0.2 release and get started today.

The first change occurred in August, when we moved Central to a new 100 Mbps connection. This temporarily solved most of our availability problems, but the load continued to increase and Central started running out of httpd worker threads. For a while we played a cat and mouse game of bumping up the workers.  Read on to find out how we ultimately solved this problem and made Central more stable and available for the world of developers it serves.

Moving to Nginx

As I previously reported, increasing the number of workers to the required level caused the CPU load on the machine to climb to well over 200 (on a 4 processor machine, this means it was 50x overloaded). On a tip, we investigated and migrated to Nginx, a remarkably fast and efficient http server optimized for static content. This immediately solved our load issue, dropping the load to an average of .05 instead of 200 during high traffic.   You can’t get much better than that.

The next issue we faced was that Nginx was so efficient at serving the content, that the connection was getting saturated, causing packet loss and connection timeouts.  We were easily filling up our 100 Mbps pipe as traffic to Central continued to climb from August 2008 to November 2008.   This next weekly graph hides the problem, it shows the “average” bandwidth used over an hour, but it conveys the trend. 

Isolating the Hourly “Spikes”

If you zoom into this graph you’ll see that 5-minute averages showed a saturated 100 Mbps pipe every hour on the hour. The problem needed an immediate solution, a saturated pipe for 10-15 minutes every hour translates to refused connections, slow transfers, and unhappy users.

These hourly spikes took a long time to isolate, and they were not confined to the regular work week.  They continued around the clock - 7 days a week. On Thanksgiving (between Turkey and Pumpkin Pie), I really dug into the logs. The source was not coming from any single location that was easily identifiable, rather it seemed to be a distributed, coordinated “attack” on the system. We looked closer at the files being accessed and determined that the Nexus Index zip file was a lion’s share of the traffic. Given that it is one of the larger files in the repo at 28mb, it wasn’t really abnormal but was it the target of the hourly spikes? To find out, I moved the index.zip out of the way for 1 minute two separate times during the spike:

A Culprit Identified…

Moving the index.zip decreased the duration of the spike. Finally, we had a target to analyze. Looking closer now at this file, we uncovered many IPs downloading the zip multiple times a day… (one extreme case was download this file every 2 minutes!). All the suspicious requests were using a generic Jakarta user agent so we knew it wasn’t Nexus.  (Nexus defines a User-Agent, you should too.) We started looking into other tools that might use the index and found the culprit. One tool had a bug that caused it to ignore the properties file and the timestamp on the zip and download the full zip every time a request was made. Worse, the default configuration had every instance grabbing the file every 5 hours exactly on the hour even though the file only changes once a week. Account for clock drift and it perfectly explains the hourly spikes from distributed sources.

Now that we identified the source of the problem, we were able to block it from getting the index until the tool was fixed. Implementing the block had an immediate effect:

Problem Solved

The traffic fell off to almost nothing, what you would expect on a holiday weekend. Notice that once an updated index was published, the traffic spike returned for a bit, but the overall ambient traffic was back to ~60-80 Mbps instead of the 98 Mbps that had been the norm over the preceeding months.   This was the impetus for Tim’s earlier blog entries on Efficient Downloads of the Nexus Index and Installing a Repository Manager.    

60-80 Mbps is certainly better than intermittent spikes to 100 Mbps and people suffering from slow downloads, but we continued to crack at the problem to try to increase availability and reduce bandwidth even further.  Tomorrow, I’ll talk about how we have reduced the traffic on Central from 80 Mbps down to 14 Mbps.   The solution might surprise you.

• Jetty Customization: The embedded jetty is now able to consume a full jetty.xml. This means several things, first it allows you to adjust the buffers and timeouts (this is needed to work around windows bugs if you have nexus on windows). More importantly it means that proxying via https/ssl is now possible. A link to the info is available from the FAQ. Examples are also included in the bundle at conf/examples. I recommend you still review the docs.
• Incremental Index Support: Built in to 1.2 for remote repositories. Shortly we will start publishing incremental indexes on Central and Nexus will be able to download only the pieces that have changed. It is not yet publishing the incremental for downstream, but that will come shortly.
• “Clear Cache” is now called “Expire Cache”: to avoid confusion. All this does is mark files as being expired so Nexus will recheck the remote for updates. Nothing is actually cleared and the files would only be redownloaded if the timestamp on the remote is newer.
• Clean/Repair Metadata Files: Nexus is now able to rebuild Maven Metadata files to remove/fixup bad information.
• Command-line Utilities: We also have some command line tools available to separate mixed snapshot/release repositories and convert local repository formats for importing into Nexus.

See the full list of improvements and fixes here.

This 1.2.0.2 release is also the core used for the Nexus Pro 1.2 Beta. The Pro version packages several plugins around the OSS core. These plugins were discussed in a previous post. The Pro Beta is shipping today to those users that have signed up for the beta program.

Please check this posting first. It contains the latest information on the subject.

<br/> <br/>

What exactly is provided:

• Configuration
  • named authentication elements to be used by repository or proxy authentication
  • supports both name/password or certificate pointer (file or URL) plus optional password
  • named repository configurations
  • local repositories (not limited to one)
  • remote repositories
  • stream verification per repository - separate for reading and writing
  • currently supported SHA-1 and PGP (.asc)
• named dependency sets
  • lists dependencies and optionally - their Maven scopes
  • standardized on a one-string naming convention
  • supports OSGi-syntax version ranges
• ant path creation/alteration
  • a task that either creates a new ant path or adds to existing one
  • optionally allows to specify Maven-like scope for the path resolution
• repository writes
  • allows to write named files into repositories
  • optional classifiers, types
  • signature generation is configured per repository
  • SHA1, PGP

Where to get mercury-ant-tasks?

All the development releases will be deployed to this site; grab the latest snapshot and:

• put it into ${user.home}/.ant/lib
• if you plan to use PGP signatures - also put there bcpg-jdk15-140.jar and bcprov-jdk15-140.jar from this central repository location
• if interested - my test build.xml contains all the unit test scripts

mercury-ant-tasks howto-by-example

This snippet shows how to define mercury namespace in the ant project markup

<project name="mercury-ant-test" default="compile" 
xmlns:merc="antlib:org.apache.maven.mercury.ant.tasks" > ... </project>

This snippet shows how to define two repositories - one remote, one local. You don’t have to always define a local repository, but in this case binaries would reside in temp. location and will be reloaded from the remote on each invocation of the merc:resolve task

<merc:config id="conf">
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"/>
</merc:config>

This snippet shows how to define authenticated repository and hide the password in the external file, so that each OS user can have their own credentials

<property file="~/.secret/secret.properties"/>
 
<merc:config id="conf">
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
             authid="repo-auth"/>
</merc:config>

This snippet shows how to configure a repository for PGP signature generation and checking; here secret keyring password is also kept in external file. Key ID always is a 16-digit hex number - use GnuPG GUI to look it up (and also generate/manipulate your keys). The verifier will accept any key from pubring.gpg for the signature

<property file="~/.secret/secret.properties"/>
 
<merc:config id="conf">
 
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
 
  <merc:repo id="localRepo" dir="/maven.repo"/>
 
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
                    authid="repo-auth">
 
    <merc:verifywrite type="pgp">
      <property name="keyring" value="/home/me/pgp/secring.gpg"/>
      <property name="pass" value="${secret.keyring.pass}"/>
      <property name="key" value="0EDB5D91141BC4F2"/>
    </merc:verifywrite>
 
    <merc:verifyread type="pgp">
      <property name="keyring" value="/home/me/pgp/pubring.gpg"/>
    </merc:verifyread>
 
  </merc:repo>
 
</merc:config>

This snippet shows how to define authenticated repository and hide the password in the external file, so that each OS user can have their own credentials

<property file="~/.secret/secret.properties"/>
<merc:config id="conf">
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
                    authid="repo-auth"/>
</merc:config>

This snippet shows how to define a dependency set. Junit will only be added to paths that define scope=”test”

<merc:dep id="my-libs">
  <merc:dependency name="asm:asm:[2.0,3.0)"/>
  <merc:dependency name="junit:junit:[4.0.1,):::test"/>
  </merc:dep>

This snippet shows how to resolve a dependency set and add the resulting binaries to “compile-path” path. This path will not include JUnit from previous snippet

<merc:resolve pathid="compile-path" depid="my-libs" configid="conf" />
<javac srcdir="${src}" destdir="${target}" classpathref="compile-path" />

This snippet shows how to resolve a dependency set in test scope and add the resulting binaries to “test-compile-path” path. This path will have JUnit

<merc:resolve pathid="test-compile-path"
                      depid="my-libs"
                      configid="conf"
                      scope="test"
/>

This snippet shows how to write a jar file and it’s sources to a repository

<merc:write repoid="remoteRepo"
                   name="t:t:1.0"
                   file="${basedir}/target/t.jar"
/>
<merc:write repoid="remoteRepo"
                   name="t:t:1.0:sources"
                   file="${basedir}/target/t-src.jar"
/>

mercury-ant-tasks reference

Now, let’s have a closer look at ant mercury markup. To make it available to ant, provide a namespace definition on the project level, for example: <project name="test" default="compile" xmlns:merc="antlib:org.apache.maven.mercury.ant.tasks">

Everywhere here I mention named - named configuration, named set. It means named in ant sense: something that has an id attribute value, and then we can reference that particular object (configuration, set, etc) by this value.

#1: Populate User-Agent Headers

If your build tool or repository manager interacts with the Central Maven Repository, you need to start setting a reasonable User Agent.  Nexus, Artifactory, Archiva, Maven, and Ivy all identify themselves in the User-Agent header.  This is incredibly important for the health of the Central repository, if there is a bug in a release of one of these tools which pegs the bandwidth max of Central, this can jeopardize the availability of this central resource for others.   Appropriate User-Agent headers help the repository maintainers quickly identify problems so that we can make sure that Central remains available for the majority of users.

In general, if your tool ever interacts with the Central Maven repository it is a good idea to maintain contact with at least one member of the Apache Maven PMC, and there are a few working for Sonatype including Brian, Jason, and John.

If you decide to write a new build tool, that’s great.  Before you distribute it to tens of thousands of developers, make sure your client sets the User-Agent.   We’ve had cases recently that involved misconfigured tools that lacked an identifying User-Agent.  If the tool in question had a meaningful User-Agent header it would have taken all of five minutes to find the problem and identify the project in question.   Instead it took members of the team multiple weeks of effort.

#2: Don’t Scrape Central, Don’t Walk the Repo

There are a few services which have decided to scrape the entire contents of the central repository into another copy and then to operate on this copy. While there are different ways to do this (rsync, getting it from a mirror), we often see people using a tool like wget with a modified User-agent header field to constantly scan the entire repository. This creates a storm of requests against Central, and wastes bandwidth.  It is also another way to crowd out other people trying to use the Central Repository.

Again, this is a case of communicating with the Maven PMC. If you are building a service that is going to be consuming Gigabytes of bandwidth on Central, you need to get in touch with the PMC as they are the body that oversees and supports the Central Maven repository. If you want to do this, you can, but people are going to likely point you at a mirror.  Even with a mirror, you need to make sure that the operators of that particular mirror don’t mind you siphoning off a couple of Gigs every month.  Bandwidth isn’t free, and the first priority is always availability.

#3: Don’t use a 404 as a Search Tool

There are a few tools out there that haven’t figured out how to use the Nexus index from the Maven repository  (Maven’s one of them). Going forward, most tools should start consulting the Nexus index to test for the presence or absence of an artifact. 404 requests are not a problem in and of themselves, but we’re trying to encourage people to minimize the number of remote interactions with Central so we can maximize availability.  We could like serve tens of thousands of 404 requests a second, but we’d like to think that people want to minimize remote interaction when possible.

If you are writing a tool and you want to know how to interact with the Nexus Index.  The code to do so is licensed under an Eclipse Public License and is available from http://nexus.sonatype.com.

At the time I started Byldan, I had been working on NMaven for more than a year and had encountered a lot of issues of getting Maven to play nice with NMaven. I wanted to see what a build system might look like in pure .NET.

The first thing I noticed was the very significant speed enhancements of building a project with Byldan versus NMaven. I also needed to get the reactor build order straightened out and after searching around found NGenerics, which allowed me to do a topological sort on project dependencies. The .NET open-source community may not be as large as Java, but there is some good stuff out there.

The final thing I noticed was I couldn’t find a good library for creating tar files (or even zip) to package the distribution, as any existing tools were either commercial or GPL licensed. So I implemented my own: Archivist.

At the time, I knew that getting full support for the pom, with interpolation and inheritance of all the elements would be a major task, so I settled on a simplified version of the pom for Byldan. But now, with the new project builder coming out soon in Maven 3.0-alpha1, I can tackle this problem head on. I recently converted maven-shared-model and maven-project-builder to .NET assemblies using ikvmc. Since these projects are self contained, only relying on JDK dependencies, this was a simple conversion from Java byte-code to a .NET assembly. Now I can use the same project builder in Byldan as I do in Maven. This code looks like:

javaUtil::List domainModels = new javaUtil::ArrayList();
domainModels.add(new ByldanProjectDomainModel(new FileInputStream(projectFileInfo.FullName)));
 
ModelTransformerContext ctx = new ModelTransformerContext(
javaUtil::Arrays.asList( new ArtifactModelContainerFactory(), new IdModelContainerFactory() ) );
 
ByldanProjectDomainModel model = (ByldanProjectDomainModel) ctx.transform(domainModels,
    new PomTransformer(new PomDomainModelFactory()),
    new PomTransformer(new PomDomainModelFactory()));

The code in C# is almost exactly the same as Java, I just need to qualify some of the namespaces, like java.util. We are using the same libraries to handle pom constructing, whether its Maven/NMaven, Mercury or Byldan, whether it’s Java or .NET.

Now that Byldan project files are in line with the Maven ones, I’m currently looking for other areas to align, like repository format and artifact types. The assembly to support IKVM generated assemblies comes prepackaged with Mono but not Microsoft .NET, so I also have some work there. Expect an official 0.5 release sometime over the next week.

Regarding the long-term plans, Maven 3.0-alpha2 is going to be focused on full Mercury integration with Maven, allowing us to dump the old artifact resolver code. This has interesting implications for both NMaven and Byldan. I’ll go down to the Sonatype offices next month and meet with Oleg to discuss how we can port Mercury to .NET for use in Byldan. We will also discuss toolchain support and how to use Maven toolchains to customize SAT rules, which we will eventually need to handle platform aware resolution of Java artifacts and .NET assemblies.

While this continued growth is great, we’re starting to have some issues with edge-case usage patterns of the Central repository. In order to defend the availability of this essential public resource the team that maintains the Central Maven Repository strong suggests that you…

Start Using a Repository Manager

Clearly, we think Nexus is the best option out there, but there are others including Artifactory and Archiva.  If you haven’t downloaded a repository manager, and you are using Maven (or Ivy), we strongly suggest that you download a repository manager to help us protect Central.  Central can handle the current load, and we’re not in any danger of exceeding the limits of the systems that serve Central, but bandwidth translates to real energy use, and we’d like to think that people are interested in conservation.    We’re at 30 TB and growing in 2008, time will only tell how quickly that figure grows.  Installing a repository manager in your organization will help reduce congestion and make sure that we’re using our resources efficiently.

Our statistics suggest that there are single organizations with more than 50,000 hits per day for the same 100 files. In other words, either someone has configured an automated build to wipe out the local repository, or there are hundreds of developers at a single company being told to constantly wipe out ~/.m2/repository.   Again, if you are making 50,000 requests to the Central Maven Repository, it is time to download Nexus and start taking some control over your own builds.  If you are making 50,000 requests to Central, installing a local repository manager should save you some serious build time.   When Maven is configured to hit a local Nexus instance, it will likely be using a fast network connection to a private resource.  When Maven is configured to hit Central, you are competing with the millions of developers that use Maven.

If you work at an organization which uses Maven, you need to start using a Repository Manager today. By our calculations, if every organization that used Maven, used a repository manager. The traffic and usage profile of the Central Maven repository would be down dramatically. Repository managers like Nexus are easy to setup and freely available.