TeamCity #2: Using Simian with MSBuild

Part #1 of this series I discussed a 5,000 foot elevation of MSBuild and TeamCity and rambled on about Continuous Integration

TeamCity #1: Creating an MSBuild File and talks on Continuous Integration

Round #2

I have mentioned before, I dig anything that can give me greater insight into the code I am writing! Simian, as simple of a concept that it is, does this well! It is also one of those lesser known gems out there!

Simian is yet another one of the tools I like to wear on my belt… Basically, simian will scan you code and find similarities which may come from sloppy coding, or that nasty cut and paste virus that seems to attack many programmers. Sometimes, you don’t even realize that you are doing it, *bad programmer, bad!* but you write code that may, without deeply analyzing your efforts, practically duplicates something you wrote in the application 3 months ago. Hate it when that happens…

Anyway Simian is a great tool, and I highly recommend it. So, in my continued efforts to adopt TeamCity as my CI server of choice it was critical for me to have the ability to incorporate Simian into my build in much of the same fashion that CI Factory does.

Execute command line

The basic command line execution of Simian looks like this;

simian.exe [options] [files]

Here are some of the more important options you can use;

MSBuild Tip #2 : Copying files… http://www.sedodream.com/PermaLink,guid,8f6bbb30-ec22-4fc6-90a1-c32715598d23.aspx MS White Papers and here by Ayende:

What does our MSBuild file look like?

Incorporating this into my MSBuild file we use the Exec task.

How do we get the outputted xml into the TeamCity interface???

TeamCity, when creating Build Report tabs from external program output files only seems to like html, not xml with a xsl file. So, I did what any geek would do, since I didn’t know any better, I rolled my own little down and dirty XML to HTML converter using the XslCompiledTransform class object… It (the converter console app) would only be like three lines of code if I didn’t validate the huevos out of the arguments passed in…

public class Program

   {

       private const string MissingExtension = "Please enter a valid {0} file, this is missing the extension.";

       private const string InvalidExtension = "Please enter a valid {0} file, the file provided has an invalid extension.";

       public static void Main(string[] args)

       {

           if (args.Length < 2)

           {

               System.Console.WriteLine("Please enter a xsl file and xml file full path.");

               return;

           }

           var xslFile = args[0];

           var xmlFile = args[1];

           if (!CheckFileNameFormat(xslFile, false))

               return;

           if (!CheckFileNameFormat(xmlFile, true))

               return;

           var transform = new XslCompiledTransform();

           // Load the XSL stylesheet.

           transform.Load(xslFile);

           // Transform xml file into an html using the xsl file provided.

           transform.Transform(xmlFile, xmlFile.Replace("xml", "html"));

       }

       private static bool CheckFileNameFormat(string fileName, bool isXmlFile)

       {

           var extension = isXmlFile ? "xml" : "xsl";

           // valida that the argument has a period

           if (fileName.IndexOf(".") < 1)

           {

               System.Console.WriteLine(string.Format(MissingExtension, extension));

               return false;

           }

           var filePieces = fileName.Split('.');

           // split on the period and make sure the extension is valid

           if (filePieces[filePieces.GetUpperBound(0)] != extension)

           {

               System.Console.WriteLine(string.Format(InvalidExtension, extension));

               return false;

           }

           return true;

       }

   }

Then I incorporated it into the Simian target as a new Exec Task like so;

<?xml version="1.0" encoding="utf-8"?>

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="RunSimian">

    <Message Text="##teamcity[progressStart '***************** Simian...']" />

    <Exec IgnoreExitCode="true" Command="&quot;$(ProgramFiles32)\Simian\bin\simian-2.2.24.exe&quot; -formatter=xml:$(MSBuildProjectDirectory)\..\Build\Artifacts\simian.xml -language=cs -excludes=$(MSBuildProjectDirectory)\..\Product\Production\**\*.Designer.cs $(MSBuildProjectDirectory)\Production\**\*.cs" >

    </Exec>

    <Exec IgnoreExitCode="true" Command="&quot;$(MSBuildProjectDirectory)\..\Build\Packages\XmlToHtmlConverter.exe&quot; $(MSBuildProjectDirectory)\..\Build\Packages\Simian\simian.xsl $(MSBuildProjectDirectory)\..\Build\Artifacts\simian.xml">

    </Exec>

    <Message Text="##teamcity[progressFinish '***************** Simian...']" />

  </Target>

</Project>

One of the important things is to let TeamCity know that the build left a little artifact for it to use in the TeamCity build report. To accomplish this, you need to set the simain.html outputted file up in TeamCity for the build.

Under the General Settings section of the Build configuration in TeamCity there is a section title Artifact paths. JetBrains once again documents out the topic rather well here. 

Basically, TeamCity needs to know that there is something for it to pick up and display. Additionally, there are two other things you need to do.

1. Set your Build Runner to acknowledge the Simian Build Target
2. Amend the main-config.xml file in the Build Server file directory

First, on the Build Runner page of your TeamCity project, there is a handy dandy little textbox for setting up the Targets in your build file that your Build Runner should run;

As you can see here, we have there Targets set to fire in our MSBuild file; BuildSolution, and RunSimian. TeamCity will fire off these Targets when the Build Runner executes.

Secondly, you need to add a line of code to your BuildServer main configuration file typically located here;

C:\BuildServer\config\main-config.xml

What this does is add a new Report tab to your build report. This line should be added with the <server> tags;

<report-tab title="Simian" basePath="" startPage="simian.html" />

I changed up the default simian.xsl file a little to look a little more, ummm… TeamCity like… The output from Simian is placed into our Artifacts directory for our build (see my previous post about folder structure), and TeamCity picks this up and displays it nicely in the report;

Of course, I have no code yet in my projects, so there isn’t any duplication, but I am sure I will rack up a little dupe action once I start coding if I am not paying attention…

So here is the solution up to this point for your reference;

Download

I hope this round shines some light on expanding TeamCity and incorporating external tools into the mix using MSBuild.

Enjoy…

UPDATE 5/20/2009: It was brought up why wouldn't you just use the built in Duplicate Finder Build Runner. For me, I like things to be all inclusive, all analysis under one Build Runner via nAnt or MSBuild. There is a work item (http://jetbrains.net/tracker/issue2/TW-4209) to allow for this functionality on The TeamCity site. But what would this do for personal/ local build outside of TeamCity? It would have to be a component that your MSBuild file could call, and be included with your Third Party dll's in your build harness.

Continue Reading »

An article by RyanAnderson 3 Comments

TeamCity #1: Creating an MSBuild File and talks on Continuous Integration

Round #1: Setting things up

Now, TeamCity is unbelievably simply to set up. No I really mean it! Just watch Rob Conery tell it like it is… Or even more detail with Sebastien Lachance… My point here is, I am not going to reinvent the wheel. There have been several blog posts, video’s and tutorials put together to show you how to do it. Where I want to come in is at the point of external tools that perform code analysis, unit testing, etc to allow you to gain better insight and attain a higher level of accountability over our code…

Starting at MSBuild…

MSBuild allows you to compile your code without the use of an IDE like Visual Studio. It is basically a build platform utility executed from the command line within the .NET Framework. Now, MSBuild can be used purely at the command line using arguments/ switches but its true power comes from an external XML file based on a well-defined XML schema which you can create to perform several tasks.

The build file is made up of the following, and as you’ll find, acts much like a class;

ItemGroups – These are basically user defined collections of, well, fields like in a class. The Include attribute specifies the field value.

<?xml version="1.0" encoding="utf-8"?>

<Project ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <ItemGroup>

    <Compile Include="Customer.cs"></Compile>

    <Compile Include="Order.cs"></Compile>

  </ItemGroup>

  <Target Name="BuildSolution">

    <MSBuild Targets="Rebuild"

             Properties="Configuration=Release;Platform=Any CPU">

      <Csc Sources="$(Compile)"></Csc>

    </MSBuild>           

  </Target>

</Project>

Within the build file you can use these collection items in task inputs/ parameters which can clean things up very nicely, and organized.

Properties - Key/value pairs that can be used to configure builds. Properties are useful similar to ItemGroup collections in that you can use them throughout the build file to clean things up… A little code reuse if you will.

<?xml version="1.0" encoding="utf-8"?>

<Project ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>

    <ProjectToBuild Include="$(MSBuildProjectDirectory)\Lulu.sln"></ProjectToBuild>

  </PropertyGroup>

  <Target Name="RebuildSolution">

    <MSBuild Projects="$(ProjectToBuild)"

             Targets="Rebuild"

             Properties="Configuration=$(Configuration);Platform=Any CPU"/>

  </Target>

</Project>

Tasks - A unit of executable code used by MSBuild to perform procedures within the build. A Task is nested within a Target, and a target can consist of one or more Tasks as stated above. Below we see an example of a Target consisting of a built in MSBuild task called Copy, which will do what its name implies.

<Target Name="CopyFilesToStage">

    <Copy

        SourceFiles="@(SourceFiles)"

        DestinationFolder="@(StageFolder)">

        <Output

            TaskParameter="CopiedFiles"

            ItemName="SuccessfullyCopiedFiles"/>

     </Copy>

</Target>

Targets - Groups Tasks together in a particular order and allow sections of the build process to be called from the command line. The Targets allow you to organize tasks into more logical groups.

Solutions with CI ‘Stands Alone’!!!

On the DMZ in South Korea there is a US Army camp called Camp Greaves where the 1st Battalion, 506th Infantry is stationed. It is stationed, well, on the DMZ and is quite the lonely place… There is a small one way bridge called the Freedom Bridge that is heavily fortified, wired with explosives to blow, and maintained by the 1/506th. They are there in case all hell breaks loose. The yell ‘Stands Alone’ and solute each time a vehicle crosses the bridge. A buddy of mine used to always reply, ‘Sleeps Together’ when we crossed, but that is another story all in itself… They are completely prepared to be isolated and operate on their own. This, in my opinion, is the epitome of a proper CI solution. They need nothing!They have all they need to operate on their own! 

To me, one of the most important things about successful Continuous Integration is the ability for any developer to walk into the project at any give point and be able to work… What I mean by that is they can;

1. Check out the solution from source control
2. Perform a successful build with all whistles and bells…

This means that any and all dependencies that the build has are included within the source, external build and analysis tools included! There should be no need to download and install any third party tools whatsoever.

Think about your solution. All dependencies are referenced and included within the solution file under the Visual Studio umbrella. When it comes to Continuous Integration, think on a broader scope, think outside of the Visual Studio box, think about the build in all its entirety. Test harness exe’s/ dlls, code analysis exe’s/ dll’s, and any other items needed to perform the build!

One of the things I have learned from Jay Flowers and his CI Factory endeavors (great CI solution by the way, love it) is organization in this arena. I do love the directory structure that he has put together; I follow it and strongly recommend it. Here is an example for the “Lulu” solution;

Here you can see the structure I am talking about. The directory tree is fairly straight forward. Starting at the Trunk folder, this enables you to house “branches” as well, you have three primary directories;

• Build - This folder contains all external packages to include any code analysis tools, and your build artifacts. Your build file can also live here.
• Product - Here we store your solution file and two sub directories;
  • Production - Your solution projects in their own sub directory.
  • Unit Tests – With each Test project also in their own sub directories.
• Third Party - Here, all external solution dependencies are stored to include third party components that your solution depends on and unit testing dll's.

In essence, what this structure does for me is give a clear and organized structure to all that is solution, build tools included… Now, granted, this may put extra weight on our repository, maybe 10MB and even more in some instance depending on our external code analysis tools, yet we can achieve our goal of checking the solution out of source control and being able to perform, a proper build!

Jay Flowers, as I stated earlier, is responsible for my adoption of this model. This is the primary model of CI Factory I do, however, would like to point out that it may even be from Mr.CI himself Martin Fowler, better put the God of CI, but for now I am going to go ahead and give props to Jay.

Back to the build file…

As I stated, before I digressed slightly, MSBuild is based on a well defined XML schema. The code required to compile your project can be very straight forward. Here is a brief example;

<?xml version="1.0" encoding="utf-8"?>

<Project ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <ItemGroup>

    <ProjectToBuild Include="$(MSBuildProjectDirectory)\Lulu.sln"></ProjectToBuild>

  </ItemGroup>

  <Target Name="BuildSolution">

    <MSBuild Projects="@(ProjectToBuild)"

             Targets="Rebuild"

             Properties="Configuration=Release;Platform=Any CPU"/>           

  </Target>

</Project>

The Parent/ Root tag is the Project tag. There are a few attributes here worth pointing out. Walking through this code, the first thing that should be noted is the xml namespace reference. Comes in very handy when working in Visual Studio to get intellisense operating on your MSBuild file.

Additionally, the ToolsVersion attribute specifies what version of the .NET Framework to build your solution under.

Full Circle, Back to TeamCity

Hopefully I have given a decent enough overview of the basics of MSBuild to get you up to speed to continue along back to TeamCity. From here, TeamCity makes it simple. The Build Runner page of your Build Configuration select MSBuild;

Set the Build File Path to the location of your build file. This is in respects to your source control root directory. As we discussed earlier, our root is the Trunk folder and, in this instance, the build file is located in our Product folder. Everything else is straight forward here, you are not required to fill out any additional info.

There are some interesting items of great value here which I will dive into in the next post of the series. JetBrains documentation is pretty string here and I recommend checking it out.

Once we configure the create our MSBuild file and configure the Build Runner, we are in business. At this point we should see a nice build report for the project with a successful build.

More to come, enjoy!

You can download the demo for this series here;

Download Lulu CI Demo Solution

Continue Reading »

An article by RyanAnderson No Comments

TeamCity (4.x) & .Net Integration Series with MSBuild

It has been the buzz for quite some time now, Continuous Integration (CI). If you haven’t heard of it, I will be surprised if you are reading this. But nonetheless, good for you!

The benefits of CI are bountiful, and transcends successful software development methodologies. The best of the best development shops implement some sort of CI system, some barebones and some implementing a plethora of tools and programs to monitor and analyze progress, quality, and stability.

As far as myself and CI, I have been using CI Factory for some time now as my Continuous Integration Server, over a year now, and have truly enjoyed it. Still, as always learning. Talk about gaining perspective on your development efforts. This appreciation rises on multiple levels. Integration with Subversion, NCover code coverage analysis, NDepend dependency analysis, Simian similarity analyzer, and several other great code analysis plug-ins.

A little over a month ago I decided it was time to saddle up and hop on the JetBrains TeamCity CI server train and check out what it had to offer my development efforts. There has been a lot of buzz around the dev community about TeamCity, and there are several articles/ posts out there about integration visa-vi NCover, NDepend, and Simian. Problem there is, many of these blog posts and articles are outdated. One example done well was Laurent Kempé’s series on his TeamCity implementation for his Tech Head Brothers Portal. It was only a year ago, but since then, several products have had updated releases, for example, NCover to version 3.0 which has so many breaking changes it is unbelievable.

So, all that being said, I wanted to spend some time whaling through my experiences with the TeamCity Continuous Integration server, and configuring your typical unit testing, code coverage, and code analysis tools I have benefited from using. Over the course of the next month, I will be writing a series on TeamCity and how I leveraged the tool to meet my needs and how you can too!

It is very easy to implement, and with little to no experience with MSBuild you can be up and running in no time. I do however recommend a cool precursor to my series and check out Rob Conery’s recent post on CI and Unit Testing using TeamCity and xUnit.net. He does a great 10,000 level elevation tour of TeamCity and getting it …There are several others out there who also do a splendid job implementing TeamCity as well, and I recommend checking them out. Also, JetBrains does a very good job of documenting their stuff as well.

Here is a run down of the series objectives;

1. Simple Configuring MSBuild for TeamCity
2. Getting MBUnit to play nice with TeamCity
3. Code Coverage (with NCover 3.0, but the fact that $650 bones is steep for many, suggestions welcome! Looking at PartCover)
4. TeamCity NDepend integration
5. Simian similarity analysis TeamCity Integration
6. FxCop Code Conformance Analysis

So, i hope you enjoy my series…

Continue Reading »

An article by RyanAnderson 1 Comment

SQL CTE with PIVOT example: Ugly or Sweet?

Last week I was working on a report of Groundwater Quality Standards and Exceedances.

Took me some time to work out the 1st & 3rd Upper Quartiles. It was probably the most difficult part but SQL Server has a T-SQL function called NTILE, which did the job with a little bit of ugly nested queries. I Imagine there is a much better way of doing this. Main thing is I got it to work!

What do you think, ugly or pretty???

WITH Exceedance(Chemical, Unit, QualityStandard, SampleDate, WellId, Result, MaxValue) AS 

    (

        SELECT 

            a.[Name], a.[Abbreviation], t.[Name], s.[SampleDate], s.[WellId], sr.[ConvertedResultValue], q.[MaxValue]

            --,NTILE(4) OVER(PARTITION BY a.[Name] ORDER BY sr.[ConvertedResultValue] DESC) AS 'Quartile'

            FROM [dbo].[Sample] AS s 

                INNER JOIN [dbo].[SampleResult] AS sr ON s.[SampleId] = sr.[SampleId]

                INNER JOIN [dbo].[vwAnalytes] AS a ON sr.[AnalyteId] = a.[AnalyteId]

                INNER JOIN [dbo].[WaterQualityStandard] AS q ON a.[AnalyteId] = q.[AnalyteId]

                INNER JOIN [dbo].[WaterQualityStardardType] AS t ON q.[WaterQualityStandardTypeId] = t.[WaterQualityStandardTypeId]

                INNER JOIN [dbo].[CSVToTable](@@Wells) AS p ON s.WellId = p.Id

            WHERE

                -- Omit CA DLR (California Detection Limit for purposes of Reporting)

                t.[WaterQualityStandardTypeId] <> 5 AND

                s.[SampleDate] BETWEEN @@Start AND @@End        

    )

    -- Now we PIVOT the CTE on the QualityStandard

    SELECT    ExPivot.Chemical,

            ExPivot.Unit, 

            [Primary EPA MCL]  = [Primary EPA MCL], 

            [Secondary EPA MCL] = [Secondary EPA MCL], 

            [Primary CA MCL] = [Primary CA MCL], 

            [Secondary CA MCL] = [Secondary CA MCL],

            [CA NL] = [CA NL],

            MIN(Result) AS [Min],

            (SELECT MAX(q1.Result) 

                    FROM (

                            SELECT e.Result, 

                                    NTILE(4) OVER(PARTITION BY e.Chemical ORDER BY e.Result) AS Quartile

                                FROM  Exceedance AS e

                                WHERE e.Chemical = ExPivot.Chemical

                                        AND e.Result IS NOT NULL

                            ) AS q1 

                    WHERE q1.Quartile = 1

                             )    

            AS [1st Quartile],

            (SELECT MAX(q.Result) FROM (SELECT TOP 50 PERCENT e.Result 

                                            FROM Exceedance AS e 

                                            WHERE e.Chemical = ExPivot.Chemical 

                                                    AND e.Result IS NOT NULL

                                            ORDER BY e.Result) AS q) AS [Median],

            (SELECT MAX(q1.Result) 

                    FROM (

                            SELECT e.Result, 

                                    NTILE(4) OVER(PARTITION BY e.Chemical ORDER BY e.Result) AS Quartile

                                FROM  Exceedance AS e

                                WHERE e.Chemical = ExPivot.Chemical

                                        AND e.Result IS NOT NULL

                            ) AS q1 

                    WHERE q1.Quartile = 3

                             )

            AS [3rd Quartile],

            MAX(Result) AS [Max],

            AVG(Result) AS Average,

            COUNT(*) AS [Total Number of Samples],

            (SELECT COUNT(DISTINCT e.WellId) 

                FROM Exceedance AS e

                WHERE e.Chemical = ExPivot.Chemical) AS [Number of Wells Sampled],

            (SELECT COUNT(DISTINCT e.WellId) 

                FROM Exceedance AS e

                WHERE e.Chemical = ExPivot.Chemical

                    AND e.Result IS NOT NULL) AS [Number of Wells with Detects],

            (SELECT COUNT(DISTINCT e.WellId) 

                FROM Exceedance AS e

                WHERE e.Chemical = ExPivot.Chemical

                    AND e.Result IS NOT NULL

                    AND e.MaxValue IS NOT NULL

                    AND e.Result > e.MaxValue) AS [Number of Wells with Exceedances]

        FROM Exceedance 

        PIVOT (

            MAX(MaxValue)

             FOR QualityStandard IN ([Primary EPA MCL], [Secondary EPA MCL], [Primary CA MCL], [Secondary CA MCL], [CA NL])

       ) AS ExPivot

       GROUP BY     ExPivot.Chemical,

                    ExPivot.Unit,

                    [Primary EPA MCL], 

                    [Secondary EPA MCL], 

                    [Primary CA MCL], 

                    [Secondary CA MCL], 

                    [CA NL]

        ORDER BY ExPivot.Chemical

You be the judge….

Continue Reading »

An article by RyanAnderson No Comments