SolrNet 0.2.3 beta1

Just released SolrNet 0.2.3 beta1. Here's the changelog:

• Fixed minor date parsing bug
• Added support for field collapsing
• Added support for date-faceting
• Upgraded to Ninject trunk
• Upgraded sample app's Solr to nightly
• Added StatsComponent support
• Added index-time document boosting
• Added query-time document boosting
• Bugfix: response parsing was not fully culture-independent
• All exceptions are now serializable
• Fixed potential timeout issue
• NHibernate integration
• Fixed Not() query operator returning wrong type

These are the interesting new features:

Field collapsing

This is a very cool feature that isn't even included in the Solr trunk. It's currently only available as a patch, but hopefully it will make its way to trunk soon. It allows you to filter query results based on a document field, thus making a flexible duplicate detection.

StatsComponent

This one is a Solr 1.4 feature (currently only available from trunk or nightly builds). Like the name says, it gives you statistics about your numeric fields within your query results. The statistics are: min, max, sum, count, missing (i.e. no value), sum of squares, mean, standard deviation. The cool thing about this is that you can facet it, thus getting separate stats for each value of the field.

Date faceting

This allows you to trigger faceting based on date ranges, i.e. you can create a facet for each day from 8/1/2009 to 9/1/2009.

NHibernate integration

This is similar to the NHibernate.Search project. It synchronizes a database with Solr (if the Solr document fields are similar to a NHibernate entity fields) and it allows you to issue Solr queries from a regular NHibernate ISession (well, actually a ISession wrapper). You can see more details about its usage in the wiki.

Contributors to this release: Derek Watson, Matt Mondok, Juuso Kosonen.

Get it here:

• Binaries
• Sample app

I'll probably call this a GA release in a couple of weeks if there aren't any serious bugs and once I get the wiki updated.

P.S.: I'll take this opportunity to clear some things up about the project. SolrNet started, like many open source projects, as a way to scratch my own itch. But in the last few months, it has grown beyond that. As a result, I don't have a use for many new features so I am not so motivated to implement them and I don't have any chance to test them in the wild to iron out any bugs and to make sure they are release-quality. This means that I need help from the community (yes, that includes you! ;-) in the form of:

• patches for new features, bugfixes, documentation, code samples
• bug reports
• feature requests
• general suggestions (e.g. "it would be cooler to do x like this instead of how it's currently done")
• voting for issues you consider important/useful in http://code.google.com/p/solrnet/issues/list might boost their priority.
• general usage feedback (e.g. "we've been using SolrNet for 3 months now at www.example.com. The features we especially use are: facets, ninject module, more like this")

Trunk is very stable, right now there are 368 tests that cover around 80% of the code. I strongly encourage you to get new builds from the build server (see artifacts links) and let me know how it works out for you (both positive and negative constructive feedback are useful).

Finally, I could offer some basic commercial support if you need some feature urgently and don't have the resources to code it.

OpenX API bindings for .Net

I've been working for the last month on a library that implements API bindings for OpenX in .Net. The OpenX API is quite big and I recently finished mapping all available methods so I decided to call it an early alpha release.

With this you can do:

• CRUD operations on all OpenX entities (Advertiser, Banner, Campaign, Manager, Publisher, User, Zone)
• Configure banner targeting
• Link/unlink zones with banners and campaigns
• Get various stats with strongly typed results

It relies on Charles Cook's XML-RPC.NET library (which is the de-facto standard xml-rpc library for .Net) which I had to modify slightly due to some irregularities in the OpenX XML-RPC server. Hopefully Charles will make xml-rpc.net more tolerant to these glitches soon so I can kill off my fork and use the standard xml-rpc.net.

Anyway here's a sample snippet querying for an advertiser's daily stats:

[Test]
public void AdvertiserDailyStats() {
    using (ISession session = new SessionImpl("user", "password", "http://localhost:10002/openx/api/v2/xmlrpc/")) {
        DailyStats[] r = session.GetAdvertiserDailyStatistics(1, DateTime.Now.AddYears(-1), DateTime.Now);
        Console.WriteLine("got {0} stats", r.Length);
        foreach (DailyStats stat in r) {
            Console.WriteLine("Stats for day {0}:", stat.Day.ToShortDateString());
            Console.WriteLine("Impressions: {0}", stat.Impressions);
            Console.WriteLine("Clicks: {0}", stat.Clicks);
            Console.WriteLine("Requests: {0}", stat.Requests);
            Console.WriteLine("Revenue: {0}", ((decimal) stat.Revenue).ToString("C"));
        }
    }
}

I wanted to release this as as early as possible because it's feature-complete but barely tested so I need people to try it and help me stabilize it by reporting any bugs and hopefully also contribute patches!

You can browse the code and fork it on github.

Binaries are available here.

PS: It seems that both Roger Dickey and I have been working independently on this at the same time, a situation that always sucks. I pinged him when I discovered this and he agreed to work together with me on this one.

Poor man's profiler with Windsor

I generally advise to keep the IoC container out of the tests, but sometimes it is a convenient tool to have. Take for example the auto-mocking container concept. It's just too useful so I make an exception to the previous rule sometimes.

Here's another use for a container in a test: a profiler. All IoC containers have some proxying abilities (or rather, they delegate to one of the proxying libraries), which can be leveraged to build a basic profiler. A profiler is especially useful in integration tests where there's something that's taking too much time but you can't put your finger on the culprit simply because there are too many components involved.

A real-world example: an application which uses SolrNet was taking too long when getting lots of documents from Solr. There was nothing weird in Solr's log so it was a client issue. Let's profile it!

Setting up the profiler

We create a ProfilingContainer and add all the components we want to profile:

IWindsorContainer container = new ProfilingContainer(); 
container.AddComponent<ISolrDocumentResponseParser<Document>, SolrDocumentResponseParser<Document>>(); 
container.AddComponent<ISolrResponseParser<Document>, ResultsResponseParser<Document>>("resultsParser"); 
container.AddComponent<IReadOnlyMappingManager, AttributesMappingManager>(); 
container.Register(Component.For<ISolrQueryResultParser<Document>>().ImplementedBy<SolrQueryResultParser<Document>>() 
    .ServiceOverrides(ServiceOverride.ForKey("parsers").Eq(new[] { "resultsParser" }))); 
container.AddComponent<ISolrFieldParser, DefaultFieldParser>(); 
container.AddComponent<ISolrDocumentPropertyVisitor, DefaultDocumentVisitor>();

Adding code to profile

We add the code that exercises the components:

var parser = container.Resolve<ISolrQueryResultParser<Document>>(); 

for (int i = 0; i < 1000; i++) { 
    parser.Parse(responseXMLWithArrays); 
} 

Getting the actual profile data

Finally, we get the profile data:

Node<KeyValuePair<MethodInfo, TimeSpan>> rawProfile = container.GetProfile();

This returns a tree of method calls, each with its logged duration. We just want a list of aggregated durations, so first we flatten the tree:

IEnumerable<KeyValuePair<MethodInfo, TimeSpan>> profile = Flatten(rawProfile);

And now we can query and write the profile:

var q = from n in profile 
        group n.Value by n.Key into x 
        let kv = new { method = x.Key, count = x.Count(), total = x.Sum(t => t.TotalMilliseconds)} 
        orderby kv.total descending 
        select kv; 

foreach (var i in q) 
    Console.WriteLine("{0} {1}: {2} executions, {3}ms", i.method.DeclaringType, i.method, i.count, i.total); 

Which prints something like this (prettified):

Method	Executions	Total own time
AttributesMappingManager.GetFields(Type)	21000	2803.23ms
DefaultDocumentVisitor.Visit(Object, String, XmlNode)	10000	432.19ms
DefaultFieldParser.Parse(XmlNode, Type)	10000	122.77ms
SolrQueryResultParser<Document>.Parse(String)	1000	93.52ms
DefaultFieldParser.CanHandleType(Type)	10000	44.71ms
SolrDocumentResponseParser<Document>.ParseResults(XmlNode)	1000	43.11ms
ResultsResponseParser<Document>.Parse(XmlDocument, SolrQueryResults<Document>)	1000	30.28ms
DefaultFieldParser.CanHandleSolrType(String)	10000	23.78ms

 

Analyzing the profile

Well, it's obvious that AttributesMappingManager.GetFields() is to be blamed here, it's taking a lot of time... which is only logical since it uses a lot of reflection...
But wait a minute, I already thought about that when I wrote that MemoizingMappingManager (which is a memoizer decorator for the AttributesMappingManager)... only I forgot to add it to the container!

Fixing and re-running the profile

After adding the MemoizingMappingManager, the profile comes out like this:

Method	Executions	Total own time
DefaultDocumentVisitor.Visit(Object, String, XmlNode)	10000	358.64ms
SolrQueryResultParser<Document>.Parse(String)	1000	166.64ms
DefaultFieldParser.Parse(XmlNode, Type)	10000	87.13ms
MemoizingMappingManager.GetFields(Type)	21000	50.5ms
DefaultFieldParser.CanHandleType(Type)	10000	42.03ms
SolrDocumentResponseParser<Document>.ParseResults(XmlNode)	1000	34.93ms
ResultsResponseParser<Document>.Parse(XmlDocument, SolrQueryResults<Document>)	1000	30.83ms
DefaultFieldParser.CanHandleSolrType(String)	10000	22.22ms
AttributesMappingManager.GetFields(Type)	1	19.82ms

 

Conclusion

This cheap profiler has a number of caveats:

• Since it works by adding a DynamicProxy interceptor to everything, it can only profile interceptable classes and methods managed by Windsor.
• Times should be taken with a grain of salt. Based on a couple of unscientific tests, it can introduce an error of 10ms (on my machine), rendering a 10ms measured time useless. I'd only use these times for a qualitative comparison. There's a reason why there's a specific .Net profiling API.
• Multithreading: each thread will get its own interceptor, so each thread gets its own profile.
• What I just described is not really a test. There is nothing asserted, it can't pass or fail. The "test" here is just a means to run the profile. It should be [Ignore]d.

All in all, I found this to be quite useful. ProfilingContainer is just a thin wrapper around WindsorContainer that adds a ProfilerFacility, which in turn sets up a ProfilingInterceptor. So you could, for example, add the ProfilingInterceptor to your application container with a per-web-request lifestyle and code a HttpModule to profile an entire http request.

Code is here. Sample profile shown in this post is here.

The current status of Git on Windows

So I finally hopped on the Git wagon, and I gotta say it's awesome! It takes some learning, but once you grok it, it's quite a simple model really.

I'm currently using Git via git-svn at work at the moment since I don't want to disturb the rest of the team just yet. Git-svn allows me to take most of the advantages of DVCS and when I'm done I commit to svn to publish my changes. Well, I'm not going to do a tutorial here, you can easily find lots of them on the net. I'll just say that if you're considering using a DVCS, go for it. Don't listen to those who say it's more complex or it has many more commands. Just go for it, it takes some learning but it's totally worth it. But please, do it with an open-minded approach. It's a different paradigm, try not to judge it from centralized version control concepts.

In my opinion, within three years from now almost everyone will be using a DVCS. Even the svn team is considering implementing an "hybrid distributed/centralized model" so if you stick with svn you'll probably use decentralized features in a couple of years.

The adoption rate of DVCS is pretty fast. Take a look at these graphs of Git Survey 2008 responses, and GitHub traffic:

 

And that's just Git. Mercurial and Bazaar usage are growing too.

However, it isn't all roses with Git on Windows. Linux people are used to the command line so it's not much of a problem for them to use the git CLI, but we Windows users are too spoiled by the ubiquitous GUI, and the excellent TortoiseSVN, AnkhSVN and VisualSVN. I think I never had to drop down to the command line to do something in svn. Even when I had to script something that involved svn, I just used SharpSvn and Boo.

With Git, I'm currently using the following mix of interfaces:

• gitk for history browsing and deleting branches.
• git-gui or TortoiseGit for committing.
• TortoiseGit for rebasing and conflict resolution.
• git-gui for merging, creating branches, pushing, fetching.
• gitk or git-gui for checkout.
• GitExtensions for single-file history within Visual Studio.
• git bash (command line) for git-svn and sometimes pulling, fetching and merging.

There are even more GUIs, like git-cola, qgit and others. So the problem isn't really a lack of GUIs. The problem is that none of them gets everything right, like TortoiseSVN does. Or at least I haven't found one. So I have to keep switching between them, which obviously isn't optimal. Now I don't mean to bash them, to their credit, GUIs and Windows support have improved hugely in the last 12 months or so. In particular, TortoiseGit has gone from nil to very usable in about 6 months by taking advantage of the TortoiseSVN codebase.

Another issue is the speed. Everyone says git is fast. Not so much on Windows, because Windows doesn't have fork() so it has to be emulated (which is slow) and exec() is also much slower than Linux. Git-svn is particularly slow because of this.

Now I don't like to rant aimlessly. I think the way to truly address these issues is to first build a solid foundation, a solid git library for Windows. And that foundation is GitSharp, which is a port of the Java jgit. The port is currently only 50% complete so it needs as much help as possible. Are you interested? Fork away! ;-)

Redundancy + Dependency injection

= Reliability.

Today, one of the biggest payment gateways, Authorize.Net, went down because of a big power outage caused by a fire. Everyone is twittering like crazy, ranting and asking customers to order by phone.

Authorize.Net is also our payment gateway at work, but this wasn't a major issue for us. We just had to swap the payment processing implementation from Authorize.Net with the one that uses a Verisign Payflow Pro account, and that's it. We even didn't have to shut down the site because we have this set up via dynamic.web.config.

Really, you gotta love dependency injection in times like these!