Tuesday, December 29, 2009

SolrNet 0.2.3 released

I just released SolrNet 0.2.3. There aren't any significant changes from beta1, I just filled in some missing documentation bits and a couple of tests.

For the next release I'll focus on performance improvements and making multi-core access easier.


Wednesday, December 9, 2009

Boo web console

Here's another embeddable web console: a Boo interpreter. You enter some code, hit the Execute button and it compiles and runs the code. You get access to all of the assemblies in your app.

I use this to run short ad-hoc "queries" against a running web app, like:

  • Is this thing in the cache?
    print (context.Cache["pepe"] == null)
  • Do I have all httpmodules correctly registered?
    import System.Web
    for s in context.ApplicationInstance.Container.ResolveAll(typeof(IHttpModule)):
      print s
  • Checking performance counters:
    import System.Diagnostics
    pc = PerformanceCounterCategory("ASP.NET")
    for counter in pc.GetCounters():
      print counter.CounterName, counter.NextValue()
  • Or even running some NHibernate query (although I'd prefer using the NHibernate web console):
    import NHibernate 
    sf = context.ApplicationInstance.Container.Resolve(typeof(ISessionFactory)) 
    using s = sf.OpenSession(): 
      users = s.CreateQuery("from User order by newid()").SetMaxResults(5).List()
      for u in users: 
        print u.Id 

Why Boo? Because it's pretty much the perfect .net scripting language to embed: small (this console with boo merged in is about 1.9MB), type inference, duck typing, low-ceremony syntax.


  • Put BooWebConsole.dll in your app's bin directory
  • Add it to your web.config's <httpHandlers> section:
    <add verb="*" path="boo/*" validate="false" type="BooWebConsole.ControllerFactory, BooWebConsole"/>
  • Visit /boo/index.ashx


  • Unlike booish, this console is stateless. Each http request creates a new interpreter, it doesn't remember anything you executed in previous requests.
  • Since the code is compiled and run within the main AppDomain, this will leak memory. Not a big deal in a dev environment though. But putting this in a production site is pretty much suicidal.
  • In case it isn't obvious: this is not meant to replace proper testing.

Code is here.

UPDATE: it's now also available on NuGet

Saturday, November 21, 2009

NHibernate web console

When developing web apps, I sometimes need to poke into the app's database, or try out some query. When it's a SQL Server database, I have Management Studio. But what if it's SQLite? Or Firebird? Even when it's SQL Server, most of my apps use NHibernate and I'm really spoiled by HQL. And if it's a local embedded database, you have to browse and open the db file. Boring. I already have the app URL, that should be enough.

So I built a NHibernate web console. It's kind of like NHibernate Query Analyzer, but more web-oriented. Here's a screencast showing its features:

A couple of features that are not evident from the screencast:

Yeah, I know it's butt-ugly and a little rough around the edges, but still it's pretty usable (at least for me).

DISCLAIMER / WARNING: it's up to you to secure this if you put it in a production website! Also, there is no query cancellation, so if you issue a 1M-results query you will effectively kill your app!

Code is here.

UPDATE: just added RSS support for query results.

Sunday, November 8, 2009

Windsor-managed HttpModules

Another interesting question from stackoverflow:

I have a custom HTTP Module. I would like to inject the logger using my IoC framework, so I can log errors in the module. However, of course I don't get a constructor, so can't inject it into that. What's the best way to go about this?

First thing we need to recognize is that there are actually two problems here:

  1. Taking control of IHttpModule instantiation in order to inject it services, and
  2. Managing the lifecycle of the IHttpModule

Let's start with the lifecycle. Ayende has the best summary of the lifecycle of HttpApplication and its related IHttpModules I've found. If you're not familiar with this, go read it now, I'll wait.

Back so soon? Ok, we can implement this in Windsor with a pluggable lifestyle. Lifestyle managers dictate when it's necessary to create a new instance of the component, but not how. They don't get to actually create the instance, that's the responsibility of the component activator. Here's the full component creation flow reference.

So we need to write a new lifestyle manager that allows at most one component instance per HttpApplication instance. I won't bother you with the implementation details since it's very similar to the per-request lifestyle: it's composed of the LifestyleManager itself and a HttpModule as a helper to store component instances.

Now we need a way to manage the IHttpModule instantiation. Unsurprisingly, we can do that with another IHttpModule, which I'll call WindsorHttpModule.
This WindsorHttpModule will be responsible for the initialization of the "user-level", Windsor-managed IHttpModules.

So, to summarize:

  1. Register PerHttpApplicationLifestyleModule and WindsorHttpModule:
        <add name="PerHttpApplicationLifestyleModule" type="HttpModuleInjection.PerHttpApplicationLifestyleModule, HttpModuleInjection"/>
        <add name="WindsorModule" type="HttpModuleInjection.WindsorHttpModule, HttpModuleInjection"/>
  2. Write your http modules using normal dependency injection style, e.g.:
    public class Service {
        public DateTime Now {
            get { return DateTime.Now; }
    public class UserHttpModule : IHttpModule {
        private readonly Service s;
        public UserHttpModule(Service s) {
            this.s = s;
        public void Init(HttpApplication context) {
            context.BeginRequest += context_BeginRequest;
        private void context_BeginRequest(object sender, EventArgs e) {
            var app = (HttpApplication) sender;
        public void Dispose() {}
  3. Make your HttpApplication implement IContainerAccessor
  4. Register your http modules in Windsor, using the custom lifestyle:

    Note that UserHttpModule is not registered in the <httpModules> section of web.config, since it's managed by Windsor now.

You can also combine this with Rashid's BaseHttpModule to make your modules more testable.

Full source code is here.

Tuesday, November 3, 2009

Powered by SolrNet

Some websites and products that use SolrNet:

I'll keep an updated list on the project's wiki. If you have a public website using SolrNet, let me know!

Saturday, October 31, 2009

Visualizing Windsor components dependencies

When working with Windsor on complex applications, it's pretty common to have hundreds or even thousands of components managed by the container. The dependencies can get quite intricate and sometimes you need to see the big picture. Static analyzers like NDepend don't cut it since components are wired by Windsor at runtime.

So here's a little guide to output a PNG of a container's components and dependencies, using QuickGraph and GLEE.

First we need to model the components and dependencies as a regular Dictionary. The key of this dictionary will be a description of the component (a regular string) and the value of this dictionary will be a list of dependencies.

var container = new WindsorContainer();

// ...add components...

var dependencyDict = container.Kernel.GraphNodes
    .Distinct(new EqComparer<GraphNode>((a, b) => a.Describe() == b.Describe(), n => n.Describe().GetHashCode()))
    .ToDictionary(n => n.Describe(), n => n.Dependents.Select(a => a.Describe()));

EqComparer is just a functional IEqualityComparer. Distinct() is needed because when using forwarded types you'd get duplicate dictionary keys.

Now we dump this into a QuickGraph digraph:

var graph = dependencyDict.ToVertexAndEdgeListGraph(kv => kv.Value.Select(n => new SEquatableEdge<string>(kv.Key, n)));

and finally we render the digraph to a PNG using GLEE:


Here's a sample output from SolrNet's inner components (click to zoom):



Only problem with this is that it doesn't take custom subdependency resolvers (like an ArrayResolver) into account, since they can't be modeled in the kernel's GraphNodes.

Here's what it looks like after some ad-hoc fixes:


Here's the full code.

Related articles:


Sunday, October 18, 2009

Git filter-branch with GitSharp

I'm currently working on migrating the Castle project Subversion repository to git. There are already a couple of svn mirrors on github, but this migration is intended to eventually replace svn as the official repository. With over 6000 commits in 5 years of history, it's not a trivial migration.

One of the issues is that all subprojects are currently being split from the main trunk to make them more independent. I'll leave that for another post.

Another issue is the committer mapping. Each svn username needs to be mapped to a github account (name + email). Roelof Blom kindly provided this map, so I was set to import with git-svn. Two days later git-svn finished and I pushed the repository to github.
Much to my dismay, I found that some committers on my repository weren't matching their github accounts. The Ken Egozi on my repo wasn't the same Ken Egozi on github!

I had two options: either fix the user mappings and re-run git-svn or change the committers with git filter-branch. I went with filter-branch as described in the Pro Git book. It was processing about 1 commit/s so I left it working and went to sleep.

The next morning I went to see and it had segfaulted halfway through. Now that did it. I updated my GitSharp fork and wrote this kind of specific filter-branch:

internal class Program {
    private static void Main(string[] args) {
        var committerMap = new Dictionary<string, string> {
            {"Ken Egozi", "a@b.com"},
            {"Krzysztof Ko┼║mic", "c@d.com"},
        var repo = Repository.Open(args[0]);
        var refs = repo.getAllRefs()
            .Where(x => x.Key.StartsWith("refs/heads") || x.Key.StartsWith("refs/tags"))
            .ToDictionary(x => x.Key, x => x.Value);

        var commitMap = new Dictionary<string, string>();

        foreach (var r in refs) {
            Console.WriteLine("Processing ref {0}", r.Key);
            var startCommit = repo.MapCommit(r.Value.ObjectId);
            var newHead = commitMap.ContainsKey(r.Value.ObjectId.Name) ? 
                repo.MapCommit(commitMap[r.Value.ObjectId.Name]) : 
                Rewrite(repo, startCommit, commitMap, committerMap);
            var newRef = repo.UpdateRef(r.Value.Name);
            newRef.NewObjectId = newHead.CommitId;
            newRef.IsForceUpdate = true;

    public static Commit Rewrite(Repository repo, Commit startCommit, Dictionary<string, string> commitMap, Dictionary<string, string> committerMap) {
        Commit lastCommit = null;
        var walker = new RevWalk(repo);
        foreach (var rcommit in walker.iterator().AsEnumerable()) {
            var commit = rcommit.AsCommit(walker);
            if (commitMap.ContainsKey(commit.CommitId.Name)) {
                lastCommit = repo.MapCommit(commitMap[commit.CommitId.Name]);
                Console.WriteLine("{0} already visited, skipping", commit.CommitId.Name);
            if (committerMap.ContainsKey(commit.Author.Name))
                commit.Author = new PersonIdent(commit.Author.Name, committerMap[commit.Author.Name], commit.Author.When, commit.Author.TimeZoneOffset);
            var newCommit = new Commit(repo) {
                TreeId = commit.TreeId,
                Author = commit.Author,
                Committer = commit.Author,
                Message = commit.Message,
                ParentIds = commit.ParentIds.Select(x => repo.MapCommit(commitMap[x.Name]).CommitId).ToArray(),
            commitMap[commit.CommitId.Name] = newCommit.CommitId.Name;
            lastCommit = newCommit;
        return lastCommit;

It took this little code 3 minutes to rewrite the whole repository. That's 34 commits/s !

When time allows, I'll try and clean this up, then merge it into the new GitSharp.CLI project. Instead of using a bash script to define transformations like the original filter-branch, this could use an embedded Boo or IronPython script!

DISCLAIMER: The code shown here is completely throwaway quality. It does not intend to be a reference GitSharp app or anything like that. It does not intend to be a general filter-branch replacement. It works on my machine, etc. Do not run this code on your repositories unless you know what you're doing. It will rewrite your whole repository! You have been warned.

Monday, October 12, 2009

Untangling the mess: Solr, SolrNet, NHibernate, Lucene

I've recently received several questions about the relationship between Solr, SolrNet, NHibernate, Lucene, Lucene.Net, etc, how they fit together, how they should be used, what features does each provide. Here's an attempt at elucidating the topic:

Let's start from the bottom up:

  • RDBMS: every programmer knows what these are. Oracle, SQL Server, MySQL, etc. Everyone uses them, to the point that it's often used as a Golden Hammer. RDBMS can be stand-alone programs (client-server architecture) or embedded (running within your application).
  • Lucene was written to do full-text indexing and searching. The most known example of full-text searching is Google. You throw words at it and it returns a ranked set of documents that match those words.
    In terms of data structures, Lucene at its core implements an inverted index, while relational databases use B-tree variants. Fundamentally different beasts.
    Lucene is a Java library, this means that it's not a stand-alone application but instead embedded in your program.
  • Full-text functions in relational databases: nowadays almost all major RDBMS offer some full-text capabilities: MySQL, SQL Server, Oracle, etc. As far as I know, they are all behind Lucene in terms of performance and features. They can be easier to use at first, but they're proprietary. If you ever need some advanced feature, switching to Lucene could be a PITA.
  • Lucene.Net is a port of Java Lucene to the .Net platform. Nothing more, nothing less. It aims to be fully API compatible so all docs on Java Lucene can be applied to Lucene.Net with minimal translation effort. Index format is also the same, so indices created with Java Lucene can be used by Lucene.Net and vice versa.
  • NHibernate is a port of Java Hibernate to the .Net platform. It's an ORM (object-relational mapper), which basically means that it talks to relational databases and maps your query results as objects for easier consumption in object-oriented languages.
  • NHibernate.Search is a NHibernate contrib project that integrates NHibernate with Lucene.Net. It's a port of the Java Hibernate Search project. It keeps a Lucene index in sync with a relational database and hides some of the complexity of raw Lucene, making it easier to index and query.
    This article explains its basic usage.
  • Solr is a search server. It's a stand-alone Java application that uses Lucene to provide full-text indexing and searching through a XML/HTTP interface. This means that it can be used from any platform/language. It can be embedded in your own Java programs, but it's not its primary design purpose.
    While very flexible, it's easier to use than raw Lucene and provides features commonly used in search applications, like faceted search and hit highlighting. It also handles caching, replication, sharding, and has a nice web admin interface.
    This article is a very good tour of Solr's basic features.
  • SolrNet is a library to talk to a Solr instance from a .Net application. It provides an object-oriented interface to Solr's operations. It also acts as an object-Solr mapper: query results are mapped to POCOs.
    The latest version also includes Solr-NHibernate integration. This is similar to NHibernate.Search: it keeps a Solr index in sync with a relational database and lets you query Solr from the NHibernate interface.
    Unlike NHibernate and NHibernate.Search, which can respectively create a DB schema and a Lucene index, SolrNet can't automatically create the Solr schema. Solr does not have this capability yet. You have to manually configure Solr and set up its schema.

In case this wasn't totally clear, here's a diagram depicting a possible NHibernate-SolrNet architecture:

Diagram made with gliffy!

Tuesday, September 29, 2009

Browsing open source code history

There is a killer characteristic of open source that seems to be often overlooked: the ability to browse the source code history.

This will sound trivial to many, but it's something that I've seen a lot in questions asked on stackoverflow.com and other forums. People ask: "why has feature X changed?" "Why am I getting an exception P since I upgraded project Q to version R?"

The first thing you need to do is assume there is a good reason for that change and set out to find it, instead of considering it a bug in the OSS. Keep yourself humble, remember that the people working on OSS are generally very smart and really know what they're doing.

And the really generic answer to such questions would be (without any intention of being harsh): you can usually find out yourself.

An example: this stackoverflow question (here summarized):

In the previous version of NHibernate (2.0.1) the following property will validate:

internal virtual BusinessObject Parent
  get { /*code*/ }

However, in 2.1 it errors saying that the types should be 'public/protected virtual' or 'protected internal virtual'. Why is this requirement now there?

Now I've been using NHibernate since 0.83 but I'm no expert in NHibernate internals. Despite that, I was able to answer the question by doing this:

  1. Check-out NHibernate's source.
  2. Grep the source code for "public/protected virtual". Only result is NHibernate.Proxy.DynProxyTypeValidator.
  3. Go to NHibernate's Fisheye, browse to DynProxyTypeValidator.
  4. Browse the file's history, starting from the latest diff (at the time of writing it's this one), looking for changes in the proxy validation exception message.
  5. Only two diffs back I find the relevant commit.
  6. The commit message says:
    - Fix NH-1515 (BREAKING CHANGE)
    - Minor refactoring of proxy validation stuff
  7. Go to NHibernate's JIRA. Browse to NH-1515. Read the issue description.

That's it. No special knowledge needed. This process could be further simplified by using Fisheye's search instead of locally grepping (but I never seem to get good results from Fisheye) or getting NHibernate from github instead of svn and then grepping with git-grep or gitk. Browsing history with svn is so painfully slow that I prefer to do it with Fisheye.

By the way, this is the same approach I use on closed code at work when my boss asks me "Hey, this feature used to behave differently! When and why did we change it?" (and this happens a lot)

This is where best practices like specific commit messages, atomic commits and good issue tracking really pay off.

Documentation? I save that for stuff that this approach can't possibly handle.

Saturday, September 26, 2009

Testing IIRF rules with MbUnit

If you're running IIS 6 like me, you know there aren't many options if you want extensionless URLs. I already had a custom URL rewriting engine in place with Windsor integration and stuff, but it couldn't handle extensionless URLs. The 404 solution seems kinda kludgy to me so after some pondering I decided to go with IIRF.

Ok, now let's see how do we test IIRF rules. IIRF is a regular Win32 DLL written in C, so there are no managed hook points that we can use from .Net. Fortunately, it comes with a testing tool called TestDriver, which is a standard .exe that takes a file with source and destination URLs and runs all source URLs against the rules, asserting that each result matches the destination.

All we need now is some glue code to integrate this to our regular tests so if any routes fail it also makes the whole build fail. I also want to see on my TeamCity build log exactly which route failed, and also be able to easily add new route tests. We can do all this with MbUnit's [Factory] and some stdout parsing. Here's the code:

public class IIRFTests {
    public void IIRFTest(string orig, string dest) {
        File.WriteAllText(@"SampleUrls.txt", string.Format("{0}\t{1}", orig, dest));
        var r = RunProcess(@"TestDriver.exe", @"-d .");
        if (r.ErrorLevel != 0) {
            var actual = Regex.Replace(r.Output.Replace("\r\n", " "), @".*actual\((.*)\).*", "$1");
            Assert.AreEqual(dest, actual);

    public IEnumerable<object[]> IIRFTestFactory() {
        yield return new object[] { "/questions/167586/visual-studio-database-project-designers", "/Question/Index.aspx?id=167586" };
        yield return new object[] { "/users/21239/mauricio-scheffer", "/User/Index.aspx?id=21239" };

    public ProcessOutput RunProcess(string fileName, string arguments) {
        var p = Process.Start(new ProcessStartInfo(fileName, arguments) {
            CreateNoWindow = true,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
        return new ProcessOutput(p.ExitCode, p.StandardOutput.ReadToEnd() + p.StandardError.ReadToEnd());

    public class ProcessOutput {
        public int ErrorLevel { get; private set; }
        public string Output { get; private set; }

        public ProcessOutput(int errorLevel, string output) {
            ErrorLevel = errorLevel;
            Output = output;

Monday, September 14, 2009

SolrNet 0.2.3 beta1

Just released SolrNet 0.2.3 beta1. Here's the changelog:

  • Fixed minor date parsing bug
  • Added support for field collapsing
  • Added support for date-faceting
  • Upgraded to Ninject trunk
  • Upgraded sample app's Solr to nightly
  • Added StatsComponent support
  • Added index-time document boosting
  • Added query-time document boosting
  • Bugfix: response parsing was not fully culture-independent
  • All exceptions are now serializable
  • Fixed potential timeout issue
  • NHibernate integration
  • Fixed Not() query operator returning wrong type

These are the interesting new features:

Field collapsing

This is a very cool feature that isn't even included in the Solr trunk. It's currently only available as a patch, but hopefully it will make its way to trunk soon. It allows you to filter query results based on a document field, thus making a flexible duplicate detection.


This one is a Solr 1.4 feature (currently only available from trunk or nightly builds). Like the name says, it gives you statistics about your numeric fields within your query results. The statistics are: min, max, sum, count, missing (i.e. no value), sum of squares, mean, standard deviation. The cool thing about this is that you can facet it, thus getting separate stats for each value of the field.

Date faceting

This allows you to trigger faceting based on date ranges, i.e. you can create a facet for each day from 8/1/2009 to 9/1/2009.

NHibernate integration

This is similar to the NHibernate.Search project. It synchronizes a database with Solr (if the Solr document fields are similar to a NHibernate entity fields) and it allows you to issue Solr queries from a regular NHibernate ISession (well, actually a ISession wrapper). You can see more details about its usage in the wiki.

Contributors to this release: Derek Watson, Matt Mondok, Juuso Kosonen.

Get it here:

I'll probably call this a GA release in a couple of weeks if there aren't any serious bugs and once I get the wiki updated.

P.S.: I'll take this opportunity to clear some things up about the project. SolrNet started, like many open source projects, as a way to scratch my own itch. But in the last few months, it has grown beyond that. As a result, I don't have a use for many new features so I am not so motivated to implement them and I don't have any chance to test them in the wild to iron out any bugs and to make sure they are release-quality. This means that I need help from the community (yes, that includes you! ;-) in the form of:

  • patches for new features, bugfixes, documentation, code samples
  • bug reports
  • feature requests
  • general suggestions (e.g. "it would be cooler to do x like this instead of how it's currently done")
  • voting for issues you consider important/useful in http://code.google.com/p/solrnet/issues/list might boost their priority.
  • general usage feedback (e.g. "we've been using SolrNet for 3 months now at www.example.com. The features we especially use are: facets, ninject module, more like this")

Trunk is very stable, right now there are 368 tests that cover around 80% of the code. I strongly encourage you to get new builds from the build server (see artifacts links) and let me know how it works out for you (both positive and negative constructive feedback are useful).

Finally, I could offer some basic commercial support if you need some feature urgently and don't have the resources to code it.

Thursday, August 27, 2009

OpenX API bindings for .Net

I've been working for the last month on a library that implements API bindings for OpenX in .Net. The OpenX API is quite big and I recently finished mapping all available methods so I decided to call it an early alpha release.

With this you can do:

  • CRUD operations on all OpenX entities (Advertiser, Banner, Campaign, Manager, Publisher, User, Zone)
  • Configure banner targeting
  • Link/unlink zones with banners and campaigns
  • Get various stats with strongly typed results

It relies on Charles Cook's XML-RPC.NET library (which is the de-facto standard xml-rpc library for .Net) which I had to modify slightly due to some irregularities in the OpenX XML-RPC server. Hopefully Charles will make xml-rpc.net more tolerant to these glitches soon so I can kill off my fork and use the standard xml-rpc.net.

Anyway here's a sample snippet querying for an advertiser's daily stats:

public void AdvertiserDailyStats() {
    using (ISession session = new SessionImpl("user", "password", "http://localhost:10002/openx/api/v2/xmlrpc/")) {
        DailyStats[] r = session.GetAdvertiserDailyStatistics(1, DateTime.Now.AddYears(-1), DateTime.Now);
        Console.WriteLine("got {0} stats", r.Length);
        foreach (DailyStats stat in r) {
            Console.WriteLine("Stats for day {0}:", stat.Day.ToShortDateString());
            Console.WriteLine("Impressions: {0}", stat.Impressions);
            Console.WriteLine("Clicks: {0}", stat.Clicks);
            Console.WriteLine("Requests: {0}", stat.Requests);
            Console.WriteLine("Revenue: {0}", ((decimal) stat.Revenue).ToString("C"));

I wanted to release this as as early as possible because it's feature-complete but barely tested so I need people to try it and help me stabilize it by reporting any bugs and hopefully also contribute patches!

You can browse the code and fork it on github.

Binaries are available here.

PS: It seems that both Roger Dickey and I have been working independently on this at the same time, a situation that always sucks. I pinged him when I discovered this and he agreed to work together with me on this one.

Thursday, August 20, 2009

Poor man's profiler with Windsor

I generally advise to keep the IoC container out of the tests, but sometimes it is a convenient tool to have. Take for example the auto-mocking container concept. It's just too useful so I make an exception to the previous rule sometimes.

Here's another use for a container in a test: a profiler. All IoC containers have some proxying abilities (or rather, they delegate to one of the proxying libraries), which can be leveraged to build a basic profiler. A profiler is especially useful in integration tests where there's something that's taking too much time but you can't put your finger on the culprit simply because there are too many components involved.

A real-world example: an application which uses SolrNet was taking too long when getting lots of documents from Solr. There was nothing weird in Solr's log so it was a client issue. Let's profile it!

Setting up the profiler

We create a ProfilingContainer and add all the components we want to profile:

IWindsorContainer container = new ProfilingContainer(); 
container.AddComponent<ISolrDocumentResponseParser<Document>, SolrDocumentResponseParser<Document>>(); 
container.AddComponent<ISolrResponseParser<Document>, ResultsResponseParser<Document>>("resultsParser"); 
container.AddComponent<IReadOnlyMappingManager, AttributesMappingManager>(); 
    .ServiceOverrides(ServiceOverride.ForKey("parsers").Eq(new[] { "resultsParser" }))); 
container.AddComponent<ISolrFieldParser, DefaultFieldParser>(); 
container.AddComponent<ISolrDocumentPropertyVisitor, DefaultDocumentVisitor>();

Adding code to profile

We add the code that exercises the components:

var parser = container.Resolve<ISolrQueryResultParser<Document>>(); 

for (int i = 0; i < 1000; i++) { 

Getting the actual profile data

Finally, we get the profile data:

Node<KeyValuePair<MethodInfo, TimeSpan>> rawProfile = container.GetProfile();

This returns a tree of method calls, each with its logged duration. We just want a list of aggregated durations, so first we flatten the tree:

IEnumerable<KeyValuePair<MethodInfo, TimeSpan>> profile = Flatten(rawProfile);

And now we can query and write the profile:

var q = from n in profile 
        group n.Value by n.Key into x 
        let kv = new { method = x.Key, count = x.Count(), total = x.Sum(t => t.TotalMilliseconds)} 
        orderby kv.total descending 
        select kv; 

foreach (var i in q) 
    Console.WriteLine("{0} {1}: {2} executions, {3}ms", i.method.DeclaringType, i.method, i.count, i.total); 

Which prints something like this (prettified):

Method Executions Total own time
AttributesMappingManager.GetFields(Type) 21000 2803.23ms
DefaultDocumentVisitor.Visit(Object, String, XmlNode) 10000 432.19ms
DefaultFieldParser.Parse(XmlNode, Type) 10000 122.77ms
SolrQueryResultParser<Document>.Parse(String) 1000 93.52ms
DefaultFieldParser.CanHandleType(Type) 10000 44.71ms
SolrDocumentResponseParser<Document>.ParseResults(XmlNode) 1000 43.11ms
ResultsResponseParser<Document>.Parse(XmlDocument, SolrQueryResults<Document>) 1000 30.28ms
DefaultFieldParser.CanHandleSolrType(String) 10000 23.78ms


Analyzing the profile

Well, it's obvious that AttributesMappingManager.GetFields() is to be blamed here, it's taking a lot of time... which is only logical since it uses a lot of reflection...
But wait a minute, I already thought about that when I wrote that MemoizingMappingManager (which is a memoizer decorator for the AttributesMappingManager)... only I forgot to add it to the container!

Fixing and re-running the profile

After adding the MemoizingMappingManager, the profile comes out like this:

Method Executions Total own time
DefaultDocumentVisitor.Visit(Object, String, XmlNode) 10000 358.64ms
SolrQueryResultParser<Document>.Parse(String) 1000 166.64ms
DefaultFieldParser.Parse(XmlNode, Type) 10000 87.13ms
MemoizingMappingManager.GetFields(Type) 21000 50.5ms
DefaultFieldParser.CanHandleType(Type) 10000 42.03ms
SolrDocumentResponseParser<Document>.ParseResults(XmlNode) 1000 34.93ms
ResultsResponseParser<Document>.Parse(XmlDocument, SolrQueryResults<Document>) 1000 30.83ms
DefaultFieldParser.CanHandleSolrType(String) 10000 22.22ms
AttributesMappingManager.GetFields(Type) 1 19.82ms



This cheap profiler has a number of caveats:

  • Since it works by adding a DynamicProxy interceptor to everything, it can only profile interceptable classes and methods managed by Windsor.
  • Times should be taken with a grain of salt. Based on a couple of unscientific tests, it can introduce an error of 10ms (on my machine), rendering a 10ms measured time useless. I'd only use these times for a qualitative comparison. There's a reason why there's a specific .Net profiling API.
  • Multithreading: each thread will get its own interceptor, so each thread gets its own profile.
  • What I just described is not really a test. There is nothing asserted, it can't pass or fail. The "test" here is just a means to run the profile. It should be [Ignore]d.

All in all, I found this to be quite useful. ProfilingContainer is just a thin wrapper around WindsorContainer that adds a ProfilerFacility, which in turn sets up a ProfilingInterceptor. So you could, for example, add the ProfilingInterceptor to your application container with a per-web-request lifestyle and code a HttpModule to profile an entire http request.

Code is here. Sample profile shown in this post is here.

Sunday, July 19, 2009

The current status of Git on Windows

So I finally hopped on the Git wagon, and I gotta say it's awesome! It takes some learning, but once you grok it, it's quite a simple model really.

I'm currently using Git via git-svn at work at the moment since I don't want to disturb the rest of the team just yet. Git-svn allows me to take most of the advantages of DVCS and when I'm done I commit to svn to publish my changes. Well, I'm not going to do a tutorial here, you can easily find lots of them on the net. I'll just say that if you're considering using a DVCS, go for it. Don't listen to those who say it's more complex or it has many more commands. Just go for it, it takes some learning but it's totally worth it. But please, do it with an open-minded approach. It's a different paradigm, try not to judge it from centralized version control concepts.

In my opinion, within three years from now almost everyone will be using a DVCS. Even the svn team is considering implementing an "hybrid distributed/centralized model" so if you stick with svn you'll probably use decentralized features in a couple of years.

The adoption rate of DVCS is pretty fast. Take a look at these graphs of Git Survey 2008 responses, and GitHub traffic:

 git_survey_responses (1)


And that's just Git. Mercurial and Bazaar usage are growing too.

However, it isn't all roses with Git on Windows. Linux people are used to the command line so it's not much of a problem for them to use the git CLI, but we Windows users are too spoiled by the ubiquitous GUI, and the excellent TortoiseSVN, AnkhSVN and VisualSVN. I think I never had to drop down to the command line to do something in svn. Even when I had to script something that involved svn, I just used SharpSvn and Boo.

With Git, I'm currently using the following mix of interfaces:

  • gitk for history browsing and deleting branches.
  • git-gui or TortoiseGit for committing.
  • TortoiseGit for rebasing and conflict resolution.
  • git-gui for merging, creating branches, pushing, fetching.
  • gitk or git-gui for checkout.
  • GitExtensions for single-file history within Visual Studio.
  • git bash (command line) for git-svn and sometimes pulling, fetching and merging.

There are even more GUIs, like git-cola, qgit and others. So the problem isn't really a lack of GUIs. The problem is that none of them gets everything right, like TortoiseSVN does. Or at least I haven't found one. So I have to keep switching between them, which obviously isn't optimal. Now I don't mean to bash them, to their credit, GUIs and Windows support have improved hugely in the last 12 months or so. In particular, TortoiseGit has gone from nil to very usable in about 6 months by taking advantage of the TortoiseSVN codebase.

Another issue is the speed. Everyone says git is fast. Not so much on Windows, because Windows doesn't have fork() so it has to be emulated (which is slow) and exec() is also much slower than Linux. Git-svn is particularly slow because of this.

Now I don't like to rant aimlessly. I think the way to truly address these issues is to first build a solid foundation, a solid git library for Windows. And that foundation is GitSharp, which is a port of the Java jgit. The port is currently only 50% complete so it needs as much help as possible. Are you interested? Fork away! ;-)

Friday, July 3, 2009

Redundancy + Dependency injection

= Reliability.

Today, one of the biggest payment gateways, Authorize.Net, went down because of a big power outage caused by a fire. Everyone is twittering like crazy, ranting and asking customers to order by phone.

Authorize.Net is also our payment gateway at work, but this wasn't a major issue for us. We just had to swap the payment processing implementation from Authorize.Net with the one that uses a Verisign Payflow Pro account, and that's it. We even didn't have to shut down the site because we have this set up via dynamic.web.config.

Really, you gotta love dependency injection in times like these!

Tuesday, June 9, 2009

Analyzing SolrNet with NDepend

I've been using NDepend off and on for the last couple of years. Mainly to diagnose build issues and help me write build scripts for complex legacy applications, since legacy applications with no automated build often fall into cyclic dependencies between different solutions. NDepend's assembly build order feature is great for this. But I've never had the time to really dive into the myriad other features this product offers.

A couple of weeks ago Patrick Smacchia kindly sent me a Pro license for NDepend (thanks Patrick!), so I thought it would be a great opportunity to use it to analyze SolrNet. Now, SolrNet is by all measures a tiny project (ohloh stats are inflated due to Sandcastle, SHFB, etc being in the trunk), but that doesn't mean it can't benefit from NDepend.

First of all, if you're analyzing a library, I recommend that you include your tests in the analysis, so that NDepend can see how the library is actually used. Otherwise, NDepend will suggest that you mark some things as private when it shouldn't. Don't worry about the tests raising warnings in the analysis, you can filter them out as I'll explain later. Plus, having the ability to analyze the tests can be pretty handy too.

For example, we can easily issue some CQL queries to get the code/test ratio:

SELECT ASSEMBLIES WHERE NameLike "Tests" 1972 LOC (as defined here)

So the code:test ratio is 1:1.27, more LOC of tests than code! However, keep in mind that this metric alone doesn't imply a correct coverage.

Browsing the default queries, I find that "Fields that could be declared internal" caught a

public readonly string query;

Oops, fixing right away!

Under "Purity / Immutability / Side-Effects", "Fields should be marked as ReadOnly when possible" showed 29 results, which seemed strange since I always try to make my objects immutable. 24 of these were <>l__initialThreadId fields, which is one of the fields of the iterator that the C# compiler builds when you use yield return. This also happened with the "Methods too big", "Methods too complex" and "Potentially unused methods" metrics.

Of course, you can edit or delete the default CQL queries. For example, the "Potentially unused methods" is defined by default as:

// <Name>Potentially unused methods</Name>
 MethodCa == 0 AND            // Ca=0 -> No Afferent Coupling -> The method is not used in the context of this application.
 !IsPublic AND                // Public methods might be used by client applications of your assemblies.
 !IsEntryPoint AND            // Main() method is not used by-design.
 !IsExplicitInterfaceImpl AND // The IL code never explicitely calls explicit interface methods implementation.
 !IsClassConstructor AND      // The IL code never explicitely calls class constructors.
 !IsFinalizer                 // The IL code never explicitely calls finalizers.

We can easily add another condition so that these methods don't bother us: AND !FullNameLike "__"

One of the most useful features of NDepend is comparing two versions of a project. I compared the latest release of SolrNet against trunk. Here's a chart of added methods (83 in total, in blue):

83 methods added

34 methods changed:


API breaking changes (no graphic here, just a list):


Don't worry, these are all internal breaking changes, they won't affect the library consumer...

From these graphics you can immediately see that there aren't really many changes, and they aren't focused. The reason is that most of the changes are minor bugfixes and a couple of minor added features.

I only scratched the surface of what's possible with NDepend, but as you can see, small projects can also profit from it, so check it out!

Sunday, May 31, 2009

Duck typing extensions for Windsor

Whenever you have to use some piece of legacy code that's badly designed (i.e. static, no coding to interfaces, god classes), your own code gets tainted with some unwanted dependencies or it gets harder to test because of that unmockable legacy code.

The usual cure for this is to isolate the legacy code with adapters, factories, etc. But what if the legacy code isn't so bad? What if we only need to extract an interface? The resulting adapter would be just trivially forwarding calls to the real object... which is boring code. Can't we do better?

After my last post I've been playing a bit more with David Meyer's Duck Typing Project (aka DeftTech) and thought it would be nice to have the ability to expose a registered Windsor component under one or more duck-typed interfaces. Kind of like forwarded types, but without having the service actually implement those interfaces.

So I came up with a couple of simple extension methods that allow this:

public class Duck {
    public bool HasQuacked { get; private set; }
    public bool HasSwum { get; private set;}

    public void Quack() {
        HasQuacked = true;

    public void Swim() {
        HasSwum = true;

public interface IQuack {
    void Quack();

public interface ISwimmer {
    void Swim();

public void WindsorDuckTyping() {
    var container = new WindsorContainer();
    DuckComponentExtensions.Kernel = container.Kernel;
    ISwimmer swimmer = container.Resolve<ISwimmer>();
    IQuack quack = container.Resolve<IQuack>();
    Duck duck = container.Resolve<Duck>();

So the Duck can be resolved as IQuack even though it doesn't implement IQuack. Here's the code that enables this:

public static class DuckComponentExtensions {
    public static IKernel Kernel { get; set; }

    public static ComponentRegistration<object> Duck<TDuck>(this ComponentRegistration<object> reg) {
        var targetType = reg.Implementation ?? reg.ServiceType;
        if (!DuckTyping.CanCast<TDuck>(targetType))
            throw new ApplicationException(string.Format("Can't duck type '{0}' to type '{1}'", targetType, typeof(TDuck)));
                            .UsingFactoryMethod(k => DuckTyping.Cast<TDuck>(k.Resolve(reg.Name, reg.ServiceType))));
        return reg;

Full source code with tests available here.
Apparently the proxies generated by DeftTech can't be proxied by DynamicProxy... so don't try to define any interceptors for these components! Again, the optimal solution would be to implement proper duck-typing with DynamicProxy...

Monday, May 25, 2009

Duck typing with Castle DynamicProxy

I was trying to come up with some duck typing solution using C# 4.0, based on this, and this, when I realized that DynamicProxy could do this without any new language features:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Castle.Core.Interceptor;
using Castle.DynamicProxy;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DuckTypingTests {
    public class Duck {
        public void Quack() {
            Console.WriteLine("Quack Quack!");

        public void Swim() {

    public interface IQuack {
        void Quack();

    public interface ISwimmer {
        void Swim();

    public class DuckTypingTests {
        public void DuckTyping() {
            var duck = new Duck();

    public static class DuckTypingExtensions {
        private static readonly ProxyGenerator generator = new ProxyGenerator();

        public static T As<T>(this object o) {
            return generator.CreateInterfaceProxyWithoutTarget<T>(new DuckTypingInterceptor(o));

    public class DuckTypingInterceptor : IInterceptor {
        private readonly object target;

        public DuckTypingInterceptor(object target) {
            this.target = target;

        public void Intercept(IInvocation invocation) {
            var methods = target.GetType().GetMethods()
                .Where(m => m.Name == invocation.Method.Name)
                .Where(m => m.GetParameters().Length == invocation.Arguments.Length)
            if (methods.Count > 1)
                throw new ApplicationException(string.Format("Ambiguous method match for '{0}'", invocation.Method.Name));
            if (methods.Count == 0)
                throw new ApplicationException(string.Format("No method '{0}' found", invocation.Method.Name));
            var method = methods[0];
            if (invocation.GenericArguments != null && invocation.GenericArguments.Length > 0)
                method = method.MakeGenericMethod(invocation.GenericArguments);
            invocation.ReturnValue = method.Invoke(target, invocation.Arguments);

Again, this is just a spike, it only matches methods, not properties, and even the method matching in the interceptor will break with method overloading. And let's not mention performance.

It would be interesting to somehow merge David Meyer's Duck Typing Project into Castle DynamicProxy, as it is the most complete duck-typing solution for .net that I know...

Testing private methods with C# 4.0

Here's another practical use for dynamic in C# 4: testing private methods. I won't discuss here whether this should or shouldn't be done, that's been argued to death for years. Instead, I'll show how it can be done easier with C# 4, without resorting to reflection (at least not directly) or stubs, while also reducing some of the friction associated with doing TDD in statically-typed languages like C#.

The answer is very simple: we just walk away from static typing using C# dynamic:

public class Service {
    private int Step1() {
        return 1;

public class TransparentObjectTests {
    public void PrivateMethod() {
        dynamic s = new Service().AsTransparentObject();
        Assert.AreEqual(1, s.Step1());

When going dynamic, it's kind of like turning off the compiler, so, for example, it won't bother you at compile-time if a method isn't present. Instead, you'll just get an exception when you run your test, which is more like the dynamic language way of testing things.

Here's the code that enables this. Just like my last post, it's just a spike, it's incomplete and I didn't test it properly, but it could serve as a base for a more complete implementation:

using System;
using System.Dynamic;
using System.Linq;
using System.Reflection;
using Microsoft.CSharp.RuntimeBinder;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DynamicReflection {
    public class Service {
        private int Step1() {
            return 1;

        protected int Step2() {
            return 2;

        internal int Step3() {
            return 3;

        private int Prop { get; set; }

        public int Execute(int a) {
            return Step1() + Step2() + Step3() + a;

        private string ExecuteGeneric<T>(T a) {
            return a.ToString();

    public class TransparentObjectTests {
        public void PublicMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(10, s.Execute(4));

        public void PrivateMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(1, s.Step1());

        public void ProtectedMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(2, s.Step2());

        public void InternalMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(3, s.Step3());

        public void PrivateProperty() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual(0, s.Prop);

        public void GenericPrivateMethod_type_mismatch() {
            dynamic s = new Service().AsTransparentObject();
            s.ExecuteGeneric<int>("lalal"); // type param mismatch

        public void GenericPrivateMethod() {
            dynamic s = new Service().AsTransparentObject();
            Assert.AreEqual("1", s.ExecuteGeneric<int>(1));

    public static class TransparentObjectExtensions {
        public static dynamic AsTransparentObject<T>(this T o) {
            return new TransparentObject<T>(o);

    public class TransparentObject<T> : DynamicObject {
        private readonly T target;

        public TransparentObject(T target) {
            this.target = target;

        public override bool TryGetMember(GetMemberBinder binder, out object result) {
            var members = typeof(T).GetMember(binder.Name, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
            var member = members[0];
            if (member is PropertyInfo) {
                result = (member as PropertyInfo).GetValue(target, null);
                return true;
            if (member is FieldInfo) {
                result = (member as FieldInfo).GetValue(target);
                return true;
            result = null;
            return false;

        public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {
            var csBinder = binder as CSharpInvokeMemberBinder;
            var method = typeof(T).GetMethod(binder.Name, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
            if (method == null)
                throw new ApplicationException(string.Format("Method '{0}' not found for type '{1}'", binder.Name, typeof(T)));
            if (csBinder.TypeArguments.Count > 0)
                method = method.MakeGenericMethod(csBinder.TypeArguments.ToArray());
            result = method.Invoke(target, args);
            return true;

UPDATE 5/17/2010: Igor Ostrovsky recently got the same idea outlined here and has written a proper implementation of this.

Saturday, May 23, 2009

Rails-like finders for NHibernate with C# 4.0

Now that VS2010 Beta 1 is out, I figured I'd take another look and try to actually do something with it. So I spiked a little proof of concept for what I mentioned about six months ago: Rails-like finders for NHibernate. In RoR's ActiveRecord you can write:


without really having a find_by_name method, thanks to Ruby's method_missing. Now, with a few lines of code, you can write this in C# 4.0, thanks to DynamicObject:

ISession session = ...
Person person = session.AsDynamic().GetPersonByName("pepe");

which behind the scenes will parse the method name "GetPersonByName" (using a Pascal-Casing convention) into a DetachedCriteria and execute it.

Note that I wrote "Person person = ..." instead of "var person = ...", that's because the result of GetPersonByName() is a dynamic so it has to be cast to its proper type to jump back to the strongly-typed world.

As I said before, this is only a proof of concept. It doesn't support finders that return collections, or operators (as in FindPersonByNameAndCity("Pepe", "Boulogne-sur-Mer")), etc. It would certainly be interesting to really implement this, but I wonder if people would use it, with IQueryable being more strongly-typed, way more flexible, and with better language integration.

Anyway, here's the code that makes this possible. Pay no attention to Castle ActiveRecord, I just used it because... well, geographical convenience, really.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Dynamic;
using System.Linq;
using System.Text.RegularExpressions;
using Castle.ActiveRecord;
using Castle.ActiveRecord.Framework.Config;
using log4net.Config;
using NHibernate;
using NHibernate.Criterion;
using NUnit.Framework;

namespace NHDynamicTests {
    public class DynamicTests {
        public void DynamicGetPersonByName() {
            using (ISession s = ActiveRecordMediator.GetSessionFactoryHolder().GetSessionFactory(typeof(object)).OpenSession()) {
                dynamic ds = s.AsDynamic();
                Person person = ds.GetPersonByName("pepe");

        public class Person {
            public int Id { get; set; }

            public string Name { get; set; }

        public void FixtureSetup() {
            var arConfig = new InPlaceConfigurationSource();
            var properties = new Dictionary<string, string> {
                {"connection.driver_class", "NHibernate.Driver.SQLite20Driver"},
                {"dialect", "NHibernate.Dialect.SQLiteDialect"},
                {"connection.provider", "NHibernate.Connection.DriverConnectionProvider"},
                {"connection.connection_string", "Data Source=test.db;Version=3;New=True;"},

            arConfig.Add(typeof(ActiveRecordBase), properties);
            var arTypes = GetType().GetNestedTypes()
                .Where(t => t.GetCustomAttributes(typeof(ActiveRecordAttribute), true).Length > 0)
            ActiveRecordStarter.Initialize(arConfig, arTypes);

    public static class ISessionExtensions {
        public static dynamic AsDynamic(this ISession session) {
            return new DynamicSession(session);

    public class DynamicSession : DynamicObject {
        private readonly ISession session;

        public DynamicSession(ISession session) {
            this.session = session;

        public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {
            result = null;
            if (!binder.Name.StartsWith("Get"))
                return false;
            var tokens = Regex.Replace(binder.Name, "([A-Z])", " $1").Split(' ').Skip(1).ToArray();
            if (tokens.Length < 4 || args.Length < 1)
                return false; // some parameter is missing
            var typeName = tokens[1];
            var type = session.SessionFactory.GetAllClassMetadata()
                .Select(e => e.Key).Cast<Type>()
                .FirstOrDefault(t => t.Name == typeName);
            if (type == null)
                throw new ApplicationException(string.Format("Type '{0}' is not mapped in NHibernate", typeName));
            var fieldName = tokens[3];
            var criteria = DetachedCriteria.For(type).Add(Restrictions.Eq(fieldName, args[0]));
            result = criteria.GetExecutableCriteria(session).UniqueResult();
            return true;

BTW: I just can't stand WPF's font rendering, it's so blurry! I don't care if it uses Ideal Width or Compatible Width or whatever you want to call it, it just looks very bad. Please vote for this issue so they fix it as soon as possible!

Tuesday, May 12, 2009

Abusing using to prioritize threads

Welcome to another chapter of "Abusing Using"! IDisposable has been used or abused (depending on who you ask) to create scopes/contexts of all sorts, from database testing to HTML generation to clipboard overriding. Here's another little (ab)use: thread priority setting, for all those background tasks that eat our CPUs:

public static class ThreadSettings {
    private class DisposableThreadPriority: IDisposable {
        private readonly ThreadPriority originalPriority;

        public DisposableThreadPriority(ThreadPriority p) {
            originalPriority = Thread.CurrentThread.Priority;
            Thread.CurrentThread.Priority = p;

        public void Dispose() {
            Thread.CurrentThread.Priority = originalPriority;

    public static IDisposable LowestPriority {
        get {
            return new DisposableThreadPriority(ThreadPriority.Lowest);

I like to keep this in an interceptor so I can easily apply it to different components just by adding a couple of lines in my web.config:

public class LowPriorityInterceptor : IInterceptor {
    public void Intercept(IInvocation invocation) {
        using (ThreadSettings.LowestPriority) {