The solution is simply to add some hooks to your Mercurial set up as described here and here.

To summarise:

To act on changesets of a push/pull/unbundle, put this in .hg/hgrc:

[hooks]
changegroup = hg update >&2

This goes in .hg/hgrc on the remote repository. Output has to be redirected to stderr (or /dev/null), because stdout is used for the data stream.

As I’ve spent much of my career using Visual Studio and its various incarnations, I’ve become quite used to using all the keyboard shortcuts to reduce the time I spend using the mouse.

As result of keyboard shortcuts, there are a lot of menu bars and windows on the screen that I just don’t need.  The image in this post is of Internet Explorer.  Web browsers have suffered from this problem for a long time as well:  if you look at the image, you will see that almost a third of the image is taken up by search bars and menus that are practically unused.

The same is true in Visual Studio.  The default settings are to give you just about every window possible (Output, Solution Explorer, etc.), in addition to lots of menu bars:  ‘Edit’ – is there a developer out there that doesn’t know Ctrl+C, Ctrl+V, Ctrl+X, Ctrl+Y, Ctrl+Z?  The amount of area for writing code seems to be 50% or less of the IDE window.

My settings are now:

• No menu bars except for a custom one that changes build configuration (release/debug), and has ‘build’ and ‘rebuild’ solution.
• One docked vertical window that contains the ‘Solution Explorer’.
• One minimized docked horizontal window with tabs for Output and Find.
• And one large window for editing code, as nature intended.

It isn’t just Visual Studio that suffers from this, take a look at Eclipse:

Once again, the code editor takes up about 50% of the available space.

There are benefits and drawbacks to increasing your editor size.  The drawbacks are that, perhaps, some of the windows and menu bars you are doing without, you need more than you realise. With more screen real estate for your editor you can view more of your code with less scrolling.  The other choice is that you can now increase the font size, making the coding easier on the eye (quite literally less strain whilst looking at your monitor).

Since I work on Windows, it is easy to get the current application’s name and version (provided you have set it and maintain it) from the resource file, either in .NET or Win32. So straight away you have your application name, and your current application version.   The next thing is to store the new version away somewhere. Going back to my comment about the simplest solution, parsing HTML is, to say the least, difficult. Putting updates on the web demand a web page, and hence HTML.

HTML can be tackled in a variety of ways, the worst of which is probably a regular expression, up to using something like http://htmlagilitypack.codeplex.com/. Either way, in HTML, you end up with a solution not dissimilar to:

<span class='name'> My App </span>
<span class='version'> My Version </span>

and parsing it relies on you trusting whoever edits the HTML to preserve the tags, classes and/or ids.

So back once again to ‘simple’. My solution is this: a text file containing comma separated values. Why? It is easy, it is human readable, and it is pretty near impossible to get wrong. When scanning it, you can just skip everything in there except your application information. It could contain a header, plus the information:

Name,Version,Installer
My App, 1.0, http://localhost/my_installer_1.0.msi

At this point someone reading this (yes, you!) will say: “but what about the web page?” Well, that is now fairly trivial with modern Javascript libraries. A little bit of jquery, and a little bit more, and you’re most of the way there. Making the installer a hyperlink is left as an exercise. Or you could simply then just maintain a static HTML page to go with the text file.

For C#, the code to get the file is remarkably trivial. For a blocking single-threaded web request:

System.Net.WebClient webClient = new System.Net.WebClient();
//  blocking...
webClient.DownloadFile(new System.Uri("http://localhost/updates.csv"), @"C:\updates.csv"));

Put this into the appropriate try catch block, and download to a file of your choosing.

On Version Naming.

What’s the difference between version 3.2 and version 3.12? For one thing, you might think 3.12 is later than version 3.2, but you would be wrong. A naive implementation of an auto-update system that checks versions would do this:

if (new_version > old_version) {  run_installer(); }

Except for two things:

1. You might have chosen new_version and old_version to be doubles. Then 3.2 is actually 3.20 which IS greater than 3.12 and your installer never gets run.

2. You might have been smarter and chosen new_version and old_version to be strings, so you could have 3.2a and 3.12b, but you would still be in trouble because a string comparison compares characters at a time, and once it gets to the ’1′ in 3.12, and the ’2′ in 3.2, it once again decides that 3.2 is greater that 3.12, and once again your installer doesn’t start.

So your run_installer clause ends up being:

if (new_version != old_version) { run_installer(); }

which has the side effect of allowing rollbacks to prior versions if you so choose (assuming of course that the MSI installers don’t prevent you from doing that).

• Get a dictionary of keys and values, where the keys are dates that are parsable by dateutil.parser.
• Turn the dictionary into two arrays; one containing dates sorted into order, the other containing the matching values (put into appropriate order for the dates).
• Pass the two arrays into the plotting function that calls matplotlib, which then saves the graph.
• Lastly, use the system verb ‘open’ to open the graph image in the default image viewer.

You’ll need Python 2.6, http://matplotlib.sourceforge.net/ and http://numpy.scipy.org/

import matplotlib
import matplotlib.pyplot as plt
import pylab
import dateutil
import os

#  Install matplotlib and numpy to get this to work      

def dict_to_dates_and_values(data):
    dates = []
    values = []   

    #  ensure the data is sorted by date, i.e. dict key
    sorted_keys = data.keys()
    sorted_keys.sort()
    for key in sorted_keys:
        #  parse date to system date object
        dates.append(dateutil.parser.parse(key))
        values.append(int(data[key]))

    return (dates, values)

def plot_graph(xtitle, ytitle, dates, values, filename):
    plt.plot_date(pylab.date2num(dates), values, linestyle = '-')
    plt.ylabel(ytitle, fontsize='xx-small')
    plt.xlabel(xtitle, fontsize='xx-small')
    plt.xticks(fontsize='xx-small')
    plt.yticks(fontsize='xx-small')
    current_figure = plt.gcf()
    current_axes = plt.gca()
    #  Set number of x axis ticks
    current_axes.xaxis.set_major_locator(matplotlib.ticker.MaxNLocator(4))
    #  Set figure size
    current_figure.set_size_inches(5,3)
    #  Save a png
    current_figure.savefig(filename, bbox_inches='tight')

data = dict()
data["2010-01-01"] = 1
data["2010-01-02"] = 2
data["2010-01-03"] = 4
data["2010-01-04"] = 8
data["2010-01-05"] = 16
data["2010-01-06"] = 32

(dates, values) = dict_to_dates_and_values(data)

xtitle = "Dates"
ytitle = "Values"
filename = "test.png"

plot_graph(xtitle, ytitle, dates, values, filename)

#  Use system verb and open the png in the default viewer.
os.startfile(filename, 'open')

London, as geotagged on flickr

Found this great site which has a series of geotagged maps of major cities, by the photos taken on flickr.   Interesting to see the usual hotspots around London are popular:  the West End, the City, Docklands, Greenwich, and then the few in the periphery, such as Wimbledon and Croydon.

Version Control

Distributed version control systems (DVCS) have been around for quite a while, and I came across  Joel’s post and thought I ought to investigate.

I had never looked at any DVCS before, but I have to admit, the article had the desired effect, and I finally bothered to download a DVCS tool, in this case, mercurial.  Since mercurial is written on top of python it is nicely portable across platforms, and seems to present the same feature set on all platforms (Windows and linux), including the built-in websever (with nice branching graphs).

So what is the fuss all about?  If you are used to ‘normal’ version control systems such as subversion then the most interesting difference is best explained in VCS terms:  when you ‘checkout’ from the (main DVCS) repository you are always checking out a new branch.  That’s it really.

In normal VCS you would check out a copy of the repository, and your commits would go back onto the trunk/head of that repository, unless you created a branch, and checked that out.

In DVCS you always clone the entire repository onto your machine, and your commits go against your own clone of repository (that you cloned).  In effect, this is forced branching for the old VCS crowd.  Consequently, merging in DVCS systems is faster and better than in normal VCS systems (especially the older ones) as it is a core feature of DVCS.

The DVCS approach is great for most projects, e.g. linux (which famously uses git).  However in large businesses control is key, and they still like to have a centrally hosted and administered repository, where all manner of control is applied to all manner of things (commit, branch, tagging rights, browsing etc.).   Branches are (generally) publicly viewable to anybody with access to the repository.

Any repository clones that you make (and branches off of the clone) are entirely private to your local machine, and do not pollute the main/master repository with dead-end branches.

One minor drawback of DVCS is that until you push your changes back to the remote repository your changes are entirely local to your machine.  This is a drawback in the sense that in some businesses you have

• high turn over of staff
  • the developer doesn’t push back to the remote repository before they leave and their work is lost
• or, and it does happen, unstable developer machines (e.g. user allowed to install software, old hardware that fails etc.).
  • machine dies and work is lost

These are, frankly, minor issues.  If you push and pull frequently enough, you can always get your repository back from another developer.

One other minor issue with DVCS is that you are always cloning an entire repository.  For large repositories you are going to start getting performance issues based on, in no particular order:  bandwidth, main repository speed, and local disc performance.  This is equally true of VCS repositories.  However you can do sparse checkouts, which can help.

The general lesson to learn here for any type of VCS is to be quite ruthless with your repository management:  is it too large, are there too many projects (KDE is a classic example)?   If you use a CI tool as part of your build process (and you should), you are probably aware that the CI tool will often check out your entire repository (depending on set up), and therefore the repository size becomes a time issue for rebuilds.  For DVCS the overall main repository size (not the checked out repository) is going to be considerably smaller than a standard VCS repository as it will have had less commits made to it (and a higher percentage of useful ones).

DVCS have the benefit of being built from scratch based on several decades of experience and usage of previous version control systems and where they went wrong.  You can commit whenever you like, without having to worry about meeting ‘commit requirements’ such as test suites passing (until you commit to the master repository).  The performance is invariably much better than traditional VCS.  As mentioned by Linus Torvalds here (if you get the chance, watch the full one hour talk, but be objective about what is said), KDE imported into git takes about 1.3Gb, in Subversion it takes over 4Gb, in other words DVCS efficiently stores data.

There seems to be a slight amount of mania regarding DVCS.  If you use an existing VCS and works for you, don’t worry! If you’re starting a new project, consider DVCS.  If you’re starting a new project that is cross platform, consider Mercurial first.

Update: 05 June 2010:  Git also produces the same graphs that mercurial does via ‘hg server’, but from the command line.  See here for details.

I came across a manifest issue recently that I think is worth describing here precisely due to the lack of information elsewhere.

Disclaimer:  most of the opinions on OS behaviour are based on non-exhaustive observations of our particular problem, especially in the absence of any particularly obvious documentation.  They may not necessarily be correct.

The Technical Problem.

Quite simply, we have a C++/native comand line process (an exe and lots of dlls) built under Visual Studio 2005 on XP and Vista.  The release build works correctly as expected.  However, on a machine that does not have VS2005 installed the application fails to work:  it fails silently on the command line.  We ship the VC runtime redistributable dlls in the same folder as our exe along with their manifests.

Common wisdom on the web, which is basically solving the problem without actually understanding what the problem is, is to “just install the VC8 redistributable runtime on your machine”.  Personally, I cannot abide ‘fixing’ something, without understanding what I’ve fixed and why it needed fixing, and you almost always find you reuse the knowledge you have learnt again in the future.

The Non-Technical Problem.

This is a work problem.  At my company, applications are automatically distributed to locked down machines that have no admin access.  Rolling out the redistrbutable patch, for purely bureaucractic reasons, is not a solution.  Also, I only have one test machine, and have to therefore be very careful with what I install on it, otherwise I have to wait for a long time for the machine to be rebuilt.

Preamble

Since it is obvious that installing VS2005 would solve the problem, and most likely the redistributable package, in order to start diagnosing the problem, we need to put the minimal set of tools on a machine, in this case Windows XP.

At this point, I wasn’t aware of what the problem actually was, so the first stop was our good old friend depends.exe (http://www.dependencywalker.com).

Hint 1:  I would suggest to anyone working in a corporate or otherwise locked down environment to either ship a set of tools with your application or making them available on an internal network drive.

Once I had loaded the application into depends, it appeared that everything was fine.  Later, however, it was obvious that this was not the case.  Buried in one of the collapsed nodes was a problem that I did not see.

The next step was to fire up the venerable DrWatson.  Straight away it was clear that the application was doing the Windows equivalent of a core dump.  The dump itself was apparently pointing to a empty (non-pure) virtual function that returned a void.  At this point I was still under the misguided idea that the problem was caused by a problem that could be diagnosed by a debug or core dump analysis.

After some head scratching, I decided to get windbg installed on the test machine.  This would have a low impact on the machine configuration as it would not install lots of unnecessary junk in order to work.  It did, of course, require admin rights:  exactly what you don’t need in a corporate environment.

Firing up the Windows debugger and attaching it to our process, it was clear on the very first run that the problem lay after a particular dll load event.  In this instance the debugger was also pointing to a different dll and a different part of the code.

Hint 2:  ship .map and .pdb files with your release build, and keep a copy of the build available: source code, pdb, objects etc.

I now started to investigate the loading of the dlls.  Next stop was psmon from the sysinsternals/Microsoft website.  Running this against my process then confirmed the loading that I saw in the Windows debugger with a little bit more interesting, including the attempted load of ‘our.dll.2.manifest’.  Whilst this would not appear to be an error, it is what first attracted my attention back to manifest files.

Back to depends.exe;  I now tried to load ‘our.dll’ into depends.exe and received the message:

“The side-by-side configuration information for ‘our.dll’ contains errors.  This application has failed to start because the application configuration is incorrect.  Reinstalling the application may fix the problem.”

How is that for a helpful message?  At least it is a start.  So the problem with the silent failure of our application is now firmly identified to lie withing ‘our.dll’.  There were also additional system log messages in the event viewer, including the usual ‘the command completed successfully’ in a reported error.

Debugging manifests

The next stop was google.  What does the error mean, and how do I debug it for more information?  I then found this “classic” (for all the wrong reasons).

The summary is that you can debug the manifest/sxs but only on Vista.

Yes, that’s right, Windows XP has manifests, but you can’t get a dump of the manifest files.  If you know otherwise, leave a comment below.

As luck would have it, I have a Vista box.  Unfortunately, it has Visual Studio 2005 installed, and therefore, the offending application works on that machine.  Giving sxstrace a go, doesn’t reveal anything I didn’t already expect to see.  Running it was a little strange:  start a console window, and run
sxstrace trace -logfile:sxstrace.etl

In a second console:
myapp.exe

Wait for it to finish, back to console 1, hit enter to stop the trace, then:
sxstrace parse -logfile:sxstrace.etl -outfile:sxstrace.txt

to make it human readable.  This then gives you a summary of the manifests that the dll depends on.  As this gave no information as it was running on a machine where the app worked, I went back to depends.exe.

Hint 3:  In depends.exe expand all nodes, and switch on the full path option for the dlls.

At this point I noticed that part of the dll in question was referencing another dll that was in turn referencing an out of date VC8 runtime.

This is where the manifest madness begins.  As far as I am aware, under Windows 2000 onwards, the search behaviour for a dll named ‘widget.dll’ (for example), is to search the local folder first, except in the presence of manifests or redirection. The behaviour with manifests is to load dlls via their manifest descriptions first.

Hint 4:  One way to see the manifest of a dll is to just drag into Visual Studio, and look at the second resource ordinal.

Consequently our app appeared to be loading two versions of the VC80 runtime.  There are two workarounds to this.  The first is to rebuild the offending dll against a newer version of the VC runtime.

If you cannot do that, for example, it is a third party dll, then you can redirect the loading by creating an ‘ourapp.exe.config’ file.  Of course, this didn’t work for us, but we were lucky in that we could recompile the dll.  However, the extra manifest entry still appeared to be there.

The reason why the app.exe.config doesn’t work is also interesting.  As far as I can find there is no documentation that states this explicitly:  application config files can only do binding redirects if the dlls they are redirecting are installed in the WinSxS folder.  The behaviour I was hoping for (which may not be correct, since this mechanism covers the .NET world as well), is that the redirect occurs on and to a manifest file and dll in the current working directory of your application.

Back to the problem;  the recompiled dll still showed that the second version of the vc runtime was in the manifest.  At this point depends was telling us that nothing directly depended on the oldest vc runtime.

So the question then was “where is this extra manifest entry coming from?”  In the land of Unix:
find . | xargs grep 50608

will find all files that contain the string ’50608′ which is part of the version number of the offending vc runtime.  Without using such a scatter gun approach, we know that the value can only be coming from a limited number of places, the source code, or something that is linked in.  We knew it wasn’t the source code, so that left some static libs.

Dragging each one into notepad finally revealed that on of the static libs that we use was referencing the old vc runtime.  This therefore meant that that particular static lib was built at some point by VS2005 RTM, and not VS2005 SP1 which the rest of the application was being built with.  By some good fortune it was our (not my!) source code, and we resurrected it, rebuilt it, and guess what?

Problem solved!

Summary

After this somewhat lengthy article, the summary is brief:  I located the problem to be in ‘our.dll’, and that the problem was in turn due to an old lib built by an older compiler that was being linked in.

It is really disappointing that on Windows XP, at least, there does not appear to be a tool for further diagnosing dll problems such as this, other than a random scattering of blog postings, and poor documentation.

At the very least there should be a tool that list the manifests of each dll, and where they were found.  The manifest tool, in verbose mode, does not do this.

Final note: if you choose to leave a comment, please be helpful and polite. If you have questions about manifests, it’s unlikely I’ll be able to help beyond what I learnt whilst writing this article.

Troubleshooting manifests

These are a list of links that I gradually worked through to figure out what on earth was going on:

http://channel9.msdn.com/forums/TechOff/22266-Side-by-side-screwup/?CommentID=272900

http://blogs.msdn.com/jreddy/archive/2005/12/23/troubleshooting-c-c-isolated-applications-and-side-by-side-assemblies-scenario-based-with-solutions.aspx

http://blogs.msdn.com/dsvc/archive/2008/08/07/part-2-troubleshooting-vc-side-by-side-problems.aspx