NuGet Server in pure Java

There are many implementations of NuGet Feed. Most of those implementations are .NET Based. To host them you need to have .NET Framework, IIS, Windows. Recently I saw posts promoting java enterprise-level implementations of NuGet Feed from Artifactory and Nexus.

For some reason one may need to have micro-to-small NuGet Feed server running on some old/small hardware probably under Linux.

You may know that TeamCity 7.0 provides a NuGet Feed as well. Sources of the NuGet support are Apache 2.0 licensed and stored under GitHub. The idea was to reuse the source-base to provide standalone NuGet Feed in pure Java.

The work resulted in some refactorings performed in a new branch in the repository. My goal is to use the branch as a base for further TeamCity plugin development. Some refactorings are still necessary and will be done in the future. I also have an idea (let me know if you need it!) to provide .war package for hosting the feed under Java application servers like Tomcat, Jetty and others.

Note. To run the server you need to have Java Runtime Environment 1.6. For more details, see tools/README.txt

Feel free to download it and get it a try. I need your feedback

Sources of the server are found under GitHub.

NuGet tool package with java code: http://nuget.org/packages/NuGet.Java.Server

You may download it from: http://packages.nuget.org/api/v1/package/NuGet.Java.Server/0.7.13.5, change the extension of the file to .zip.

TeamCity and Mercurial Mq Extensions

In this post I show how to setup Mq patches with TeamCity build.

First of all, let's start with basics. I decided to use Mq extension to deal with patches to a opensource library with sources under Mercurial. The goal was to have a build configuration that fetches sources from the library repository, applies patches from patches repository, builds, tests and packages the library.

Mq Basics

For local checkout on local machine to apply patches you need the following:

• enable Mq extension for your Mercurial
• hg clone remote-repo local-folder
• hg clone pathces-remote-repo local-folder/.hg/patches
• hg qpull -a

At the end you see all your patches applied or you see a patch application error. For more details, please refer to original Mq docs.

Setting up TeamCity Checkout

First we need to select checkout on agent for the build configuration as we need to call Mercurial commands to apply patches. All hg clone commands are replaces with TeamCity VCS root and checkout rules. TeamCity will take care of checkout better.

I add one VCS root that checks out the library repository with no checkout rules and the one VCS root for patches repository with checkout rules +: . => .hg/patches to checkout patches repository to patches repository location.

You need to set Clean all files before build to make TeamCity revert all changes before the next build.

To make Mercurial checkout on agent work, check you have Mercurial installed locally.

Setting up the Patch

Next step is to enable Mq extension (nice that it's embedded into default Mercurial install) and apply the patch:

I use Command Line build runner to call the script and put it as the first build step. The command succeeds only if Mercurial was able to apply patches and thus the build continues

Use

I used the described approach for patches to OData4j library: https://code.google.com/p/odata4j-patches. I set up TeamCity Build Configuration for that

TeamCity Support for NuGet 1.6

Today I added tests for compatibility of NuGet plugin for TeamCity with NuGet 1.6

It's nice to say that my plugin for NuGet commandline that I wrote for NuGet 1.4 is still working. All tests are green.

I created an issue for NuGet.org v2 feed to provide more easy-to-fetch information about packages in the feed.

If you use NuGet, please make sure you on the latest build of the plugin.

Download and Info:

For more information, see NuGet plugin home page at:
NuGet support is included into TeamCity 7.0
http://confluence.jetbrains.net/display/TW/NuGet+support
or download a latest build from TeamCity here

NuGet.config Support

There was an issue for our NuGet in TeamCity plugin support: TW-18927

As it turned out, it's possible to make NuGet use custom folder to store all packages. To do it you need to create NuGet.config file near your .sln file with the following content:

Starting from recent builds, NuGet plugin for TeamCity (versions 0.5.x and bundled) will be using this file to replace default packages location. The plugin uses repository.config file under packages folder to get the full list of projects that uses NuGet dependencies.

Download and Info:

For more information, see NuGet plugin home page at:
NuGet support is included into TeamCity 7.0
http://confluence.jetbrains.net/display/TW/NuGet+support
or download a latest build from TeamCity here

Playing Git

In this post I describe my pattern of Git usage.

My project (NuGet support for TeamCity) contains two main branches called 'v0.5-bugfix' and 'v0.6-bugfix'. 'v0.5-bugfix' branch is targeted for TeamCity 6.5.x. 'v0.6-bugfix' is targeted for TeamCity 7.0 and contains more features.

To start a new feature development I first decide if I want this feature to be delivered for 0.5 or 0.6 version. Let's suppose I decided to create a feature for both versions. I create a branch 'task' for it from 0.5-bugfix by calling:

 git checkout -b task v0.5-bugfix

As work is done I merge branch 'task' onto 0.5-bugfix via:

 git checkout v0.5-bugfix
 git merge task

Main work is done. New feature is implemented in branch 'v0.5-bugfix'. Now it's time to merge it onto v0.6-bugfix. For it I create a branch 'task6' from v0.6-bugfix. Than I merge branch 'task' into 'task6'.

  git checkout -b task6 v0.6-bugfix
  git merge task
  git checkout v0.6-bugfix
  git merge task6

And in the rest I remove obsolete branches by:

  git branch -d task task6

That was it. The feature is ready for both branches. It's time to push.

Another trick I found useful is to try to create branch from the lowest possible version even if you do not need it. In future that will let you simply merge the feature to older version.

To update from remote repository I use the following:

  git fetch origin
  git merge origin/branch

This way of updates let's you avoid issues if remote branch has new commits and your local branch has another commits. I do not like rebase because it requires force push (-f) with possible history overwrite if you decide to push it once more after rebase.

All recent fixed to NuGet plugin for TeamCity I did using this practice. Current public version of the plugin contains all fixes and features that are applicable to the version. It was easy to take two branches in sync. This is a VCS commits graph I saw today:

You may find the source of the graph on GitHub

On-the-fly Code Generation with .NET Expression Trees

Recently I came to the task to fill a bunch of C# object properties from a Dictionary. There were 10+ fields and I decided to use reflection to avoid writing and supporting dummy code on every object change.

So I come up with code like:

 typeof(DataObject).GetProperties().Where(...).Select(x=>x.SetValue(instance, GetValue(x), null));

Of course this code was not fast. I used Reflection and LINQ. The task was to make it work faster. First I decided to drop all reflection code and write all bindings explicitly for every field. Than I come up with another trick!

Why not to make .NET LINQ Expressions runtime generate this code for me at runtime. It was easy to implement and performance was close to hand-written. First, I moved all initialization of Expressions outside of binding code. So the binding turned out to be an array iteration and delegates call.

Let's see how to make Expressions generate a code for property binding.

Using this code I generate a code that sets property value without reflection!

The fun is that one may make a next step to include all parsing and validation logic into this code and thus have a generated code for it. This is really handy that C# allows to write the following code:

This is a simplest way to get Expression tree for simple lambdas.

TeamCity.ServiceMessages Write API

Today I published TeamCity.ServiceMessages v2.0 package which provides API for generating TeamCity specific service messages. Service messages are described in the documentation page at http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity

Introduction

Do you know what service messages are? Service Messages are strings in the following format:

##teamcity[hello message='teamcity']

TeamCity uses Service Messages to provide simple API for a build scripts integration, i.e. build status reporting, artifacts publishing, tests reporting and more.

For precise description of the format and service messages usage in TeamCity, take a loot at Build Script Interaction with TeamCity article. Service Messages introduce a communication protocol for map-like data exchange between processes using stream, socket, HTTP or pipe.

Basic read-write API for service messages was implemented in TeamCity.ServiceMessages library v1.0. There are blog posts describing it here and here.

Reporting Tests

...

public void ReportTests(...) {
  //Create a TeamCity message logger. 
  using (var writer = new TeamCityServiceMessages().CreateWriter())
  {
    using (var block = writer.OpenBlock("Big log from TeamCity Service Messages block"))
    {
      //Let's open a test suite
      using(var suite = block.OpenTestSuite("MyFavoriteTests.dll"))
      {
        //Let's report ignored test
        using (var test = suite.OpenTest("This.Would.Log.Ignored.Test"))
        {       
          test.WriteIgnored();
        }
        //Let's report failed test
        using (var test = suite.OpenTest("This.Would.Log.Failed.Test"))
        {
          test.WriteFailed("Assert failed", "Assert failed stacktrace and more details");
        }
        //Let's report successful test with some output
        using (var test = suite.OpenTest("This.Would.Log.Successful.Test"))
        {
          test.WriteStdOutput("This is a test output to be shown in TeamCity UI");
        }
      } 
    }
  }
}

Thanks to C# using construction that asserts service messages write API is used in the right way. As you open a block you may only use created object to continue logging until you dispose it. TeamCity.ServiceMessages library contains assertions to avoid generating invalid service message sequences.

Reporting Dynamic Build Artifact

public void ReportCustomArtifact(string path) {
  using (var writer = new TeamCityServiceMessages().CreateWriter(x => builder.AppendLine(x)))
  {
     writer.PublishArtifact(path);
  }
} 

Reporting Custom Build Number

public void ReportNewBuildNumber(string buildNumber) {
  using (var writer = new TeamCityServiceMessages().CreateWriter(x => builder.AppendLine(x)))
  {
     writer.WriteBuildNumber(buildNumber);
  }
} 

What's Next

There is a big number of other TeamCity service message that are supported by the implementation. To find more information, see JetBrains.TeamCity.ServiceMessages.Write.Special.ITeamCityWriter and it's parent interfaces. Here are some of supported messages:

• Set build number
• Set build parameter
• Write message, warning or error
• Publish build artifacts
• Publish build statistics value
• Open/Close block
• Open/Close test suite
• Open/Close test
• Open/Close compilation block
• and more...

API Changes since v1.0

I moved IServiceMessage interface from JetBrains.TeamCity.ServiceMessages.Read namespace to JetBrains.TeamCity.ServiceMessages namespace.

Usage

The library is compiled for .NET 3.5. It is available as NuGet package:

PM> Install-Package TeamCity.ServiceMessages

Sources are available at GitHub. I setup a build configuration at TeamCity.CodeBetter.Com.

Thanks to TeamCity support for NuGet, it automatically publishes the library as NuGet package to NuGet.org.