Category Archives: Uncategorized

Introduction to Kafka @ ALT.NET

On Tuesday night I gave a presentation on Apache Kafka at the Sydney ALT.NET meet up.

What is Kafka?

Kafka is publish-subscribe messaging rethought as a distributed commit log. Originally developed at LinkedIn in 2011 it has been adopted by some big companies since such as Twitter, Netflix and Microsoft to provide high throughput, low latency messaging.

Most common use cases for Kafka include:

  • Pub/Sub
  • Activity tracking
  • Metrics
  • Log aggregation
  • Stream processing
  • Event sourcing
  • Commit logs

How do I use it?

There are a bunch of libraries available for a lot of languages. Unfortunately the .NET ones are quite immature. Recently a few of the boffins at Microsoft open-sourced the library they’ve used internally called CSharpClient-for-Kafka. This library is targeted at version 0.8, but should soon support 0.9. It is fast and with some work will be quite reliable.

However, it does not support the new consumer protocol that was released with v0.9. If you want something that does, you’ll have to go for ah-/RDKafka-DotNet. This is a wrapper around the librdkafka c library that’s the most commonly used outside of Java-land. The downside of this is that due to interop it’s not quite as performant as the native .NET TCP communication of the MS library. In my tests I’ve found publishing with CSharpClient-for-Kafka to be ~.1ms per message/batch and consuming ~20ms. RDKafka-DotNet performs these at ~2ms and ~150ms with similar parameters.

Also, if you think that TCP is too hard core, you can give the Confluent REST Proxy or Kafka Pixy a go… These are services that allow you to publish to and subscribe from Kafka via HTTP APIs. Not the most performant option, but definitely easier to debug…

If you’re looking for samples of using these clients you can check out a couple of my repositories –

  • kafka-basic – a wrapper for CSharpClient-for-Kafka that provides simple producer/consumer abstractions
  • rdkafka-tests – a test console for RDKafka-DotNet

Where do I learn more?

The Apache Kafka site has a lot of great information to get you up and running. Confluent are the commercial entity providing supported distributions and whose founders were originally responsible for the development of Kafka at LinkedIn.

My presentation is available from Tuesday night on slideshare with a few funky animations to help visualise message delivery and rebalances.


Saying goodbye to Readify

After 8 years at Readify I’ve decided to hang up my consulting hat. Over the course of the next two weeks I’m finalising my hand over and a few fun tasks before, well, riding off into the sunset.

At our quarterly company info night the other evening I had the chance to formally say goodbye. In my  time here the company has grown from 40 people to over 200 and seen such a phenomenal change in the size and number of simultaneous projects we’re taking on. We continue to win awards and more importantly the respect of our customers all the time.

It’s really been a privilege and an honour to be part of such a fantastic group of people, many of whom I know will remain friends for a long time. I have grown so much and owe a lot of it to the inspiring and challenging environment that Readify creates. 

As for what I’m doing next… Well, that’s undecided. I’m looking for an opportunity to really test myself and grow further. I have no doubt I’ll find something that pique’s my interest soon, but if that takes a while and I end up repainting and laying some floors in my apartment or spending a little time at the snow with the kids then I won’t be disappointed.

Using Web API Validation with jQuery Validate

Building on my last post about validating your model with Web API, if you’re calling a Web API controller from JavaScript you may need to parse the validation result and display it on the screen.

Most people using MVC would be using the jQuery Validate plugin that’s been included with the default template for quite a while now. While most validations are performed using JavaScript adapters, some are only performed server side. As a result, the regular unobtrusive JavaScript adapters will not catch this before the post occurs. This means that you if you are using JavaScript requests with Web API to handle data manipulation you will need to somehow manually handle the validation errors that will be returned.

Plugging into jQuery Validation is actually quite easy… To validate a form, simply select the form using jQuery and call .validate() on it – e.g.

var validator = $('.main-content form').validate();

This will return a validator object will a few handy methods on it. Two of which are valid() and showErrors(). The valid method will return a Boolean value indicating whether the form is valid or not and the showErrors method will show any validation errors on the current form. The showErrors method also accepts an object that defines any additional error messages you wish to display – e.g. to display the message “The title is incorrect” for a property named Title:

validator.showErrors({ Title: 'The title is incorrect.' });

Now, assuming I a view with the following mark-up inside the form, I should see a validation error:

<div class="editor-label">@Html.LabelFor(model => model.Title)</div>
<div class="editor-field">
    @Html.TextBoxFor(model => model.Title)
    @Html.ValidationMessageFor(model => model.Title)

But how do we connect this to Web API…? Well, if you’ve read my previous post you’ll recall that calling a Web API controller’s PUT action that’s decorated with the ValidateFilter attribute I created will return a collection of validation errors if the model is not valid. To test this, I’ll modify my TodoApiController from the previous post as follows:

public void Put(int id, TodoItem value)
    if (value.Title == "hi there")
        ModelState.AddModelError("Title", "The title is incorrect.");

    if (!ModelState.IsValid) return;

    db.Entry(value).State = EntityState.Modified;

I should now receive a validation error whenever I try to update an item with the title “hi there”. Let’s write some jQuery to submit my form:

function updateItem(form, url) {
    var validator = form.validate(),
        serialized = form.serializeArray()
        data = { };

    if (!validator.valid()) { return; }

    // turn the array of form properties into a regular JavaScript object
    for (var i = 0; i < serialized.length; i++) {
        data[serialized[i].name] = serialized[i].value;

        type: 'PUT', // Update Action
        url: url, // API Url e.g. http://localhost:9999/api/TodoApi/1
        data: data, // e.g. { TodoItemId: 1, Title: 'hi there', IsDone: false }
        dataType: 'JSON',
        success: function () { alert('success'); },
        error: function (jqXhr) { extractErrors(jqXhr, validator); }


Now let’s look at extractErrors:

function extractErrors(jqXhr, validator) {

    var data = JSON.parse(jqXhr.responseText), // parse the response into a JavaScript object
        errors = { };

    for (var i = 0; i < data.length; i++) { // add each error to the errors object
        errors[data[i].key] = data[i].value;

    validator.showErrors(errors); // show the errors using the validator object

Lastly, attaching to the form’s submit event will call this whenever the Enter key is hit or the Submit button is clicked:

$('.main-content form').submit(function () {
    updateItem($(this), '/api/TodoApi/' + $('#TodoItemId').val());

Project Liike (aka. Microsoft Patterns & Practices do Mobile Web)

Microsoft have started a new Patterns & Practices project name Liike (LEEE-keh) with the aim of delivering guidance for building mobile web solutions.

How do they plan to do this…?

Well they’ve got a few FTE’s who are creating a mobile version of a previous project as a reference implementation so they can make the mistakes we’ve all made in the past. They’ve also taken on an advisory board consisting of a few members of the community (including myself) for some ideas and extra guidance.

One thing that’s been made clear so far is that reusable code is not a major deliverable. There are many people out there who’ve done a bang-up job of building UI frameworks and plugins for mobile web, so they don’t want to re-invent the wheel – they just want to reduce the squeak in yours by giving you the oil you need to run smoothly.

So, how can you help…?

1. There’s a uservoice site available for you to throw your ideas for what you think are the most important challenges:

2. Start a conversation with myself or anyone else on the advisory board! I’m always available on twitter and am happy to take comments on my blog. If you want to get me on email, just leave a comment on this post and I’ll get back to you.

Using the System Tray to show progress in Windows Phone 7 Mango

Before Mango it was a little difficult to put a generic status message at the top of your page. The Mango tools have made this a bit easier by giving developers (more) access to the System Tray. That’s this part:

You are still a bit limited in what you can do (i.e. you can’t actually add or remove icons), but there’s a lot more flexibility than before.

You can access the system tray either in code or XAML using the Microsoft.Shell.SystemTray class. This class has the following dependency properties:

  • IsVisible
  • Opacity
  • BackgroundColor
  • ForegroundColor
  • ProgressIndicator

While it’s quite exciting to be able to style the status bar, I cried out of joy when I saw the ProgressIndicator property. Why…? Because now I can add progress information to the top of my pages quickly and easily! J

So, I want to add a downloading message and progress bar to the top of my page (above the application title) as follows:

Using XAML, I can add text and a progress bar to a page by dropping the following into the page:

    <shell:ProgressIndicator IsIndeterminate="true" IsVisible="True" Text="Click me..." />

Or, using code, I can accomplish the same thing by dropping this into my code:

ProgressIndicator progress = new ProgressIndicator
    IsVisible = true,
    IsIndeterminate = true,
    Text = "Downloading details..."
SystemTray.SetProgressIndicator(this, progress);

Feel free to download my sample application showing how to manipulate the System Tray:


Pushing to Windows Phone 7 devices using MetroPimp

Recently I released a little open source project with the aim of making it easier for developers to write services that push information to Windows Phone 7 devices called MetroPimp.

This library is available via nuget and on bitbucket.

Based on a blog post I wrote when WP7 was still in beta, MetroPimp provides a simple and consistent API that allows you to push Toast, Tile (single and double sided) and Raw push notifications to WP7 devices using the Microsoft Push Notification Service (MPNS).

As developers who have tried this already may know, there is quite a bit of work that goes into setting up push notifications. First you have to register for notifications on the device by opening a channel. Then you have to get the URI provided by MPNS and send it to a service of your own. Lastly, using this URI you need to form XML packets and craft HTTP requests with the right headers to push the data to the device.

MetroPimp simplifies the last step of this process by providing a model that represents the messages you want to send to devices, a single method call to handle serializing the message and sending it up the wire and a simple result which, in the case of failure, identifies the reason for the failure and provides informative message.

Enough talking… show us the code!!!

The main entry point of MetroPimp is the Pusher class (which can be mocked via the IPusher interface). It has a single method named Send that accepts a Message and returns a SendMessageResponse. To create a Pusher, simply new it up (or inject it using your preferred IOC container).

var pusher = new Pusher();

The following sections will define the classes you can use to push various notification formats to a device. For more information on message formats, please refer to Sending Push Notifications for Windows Phone on MSDN.


To send raw data to the phone (i.e. a notification that is not a toast or tile update), create an instance of Raw and send it using the pusher.

var rawMessage = new Raw {
    Uri = /* insert MPNS uri here */,
    Content = "hello world!"

MPNS does not limit you to sending strings in the raw message. However, seeing as just about anything can be represented as a string, I decided this may be more universal. Under the covers, MetroPimp simply converts the string to its UTF8 representation and drops it into the request body.

For more information on sending and receiving raw notifications, refer to How to: Send and Receive Raw Notifications for Windows Phone on MSDN.


Sending a toast message is similar to sending a raw message in that you simply create an instance of a Toast object and pass it into Send. Toast has 3 fields that differ it from a raw:

  • Text1 – The
    first bit of bold text that appears in the toast (required)
  • Text2 – The second bit of text that appears in the toast (optional)
  • Param – A parameter string to be passed to the application if the user taps the notification (optional)
var toastMessage = new Toast {
    Uri = /* insert MPNS uri here */,
    Text1 = "Hello World!",
    Text2 = "This is a toast.",
    Param = "helloWorldToasted"

For more information on sending and receiving toast notifications, refer to How to: Send and Receive Toast Notifications for Windows Phone on MSDN.


Tile notifications can be specified to be either single or double sided tiles. As double sided tiles are only supported by WP7 Mango, the functionality has been split into two classes. Sending a single sided tile is done by specifying properties available on the Tile class.

  • BackgroundImageUri – The URI to the background image for the tile. Can be either embedded or remote. (required)
  • Title – The text to use as the title of the tile. (required)
  • Count – The number to show in the blue bubble. Not defining or using 0 will clear the bubble. (optional)
var tileMessage= new Tile {
    Uri = /* insert MPNS uri here */,
    BackgroundImageUri = "background.png",
    Title = "Hello World",
    Count = 9

Double sided tiles are sent by specifying properties on the DoubleSidedTile class. This class has the same set of properties as Tile for the front side of the tile, along with the following properties for the back side:

  • BackBackgroundImageUri – The URI to the background image for the back of the tile.
  • BackTitle – The text to use as the title for the back of the tile.
  • BackContent – The text to display over the top of the background image on the back of the tile.
var doubleSidedTileMessage = new DoubleSidedTile {
    Uri = /* insert MPNS uri here */,
    BackgroundImageUri = "background.png", 
    Title = "Hello World", 
    Count = 9, 
    BackBackgroundImageUri = "back_background.png",
    BackTitle = "Hello Back",
    BackContent = "This is the back"

For more information on sending and receiving toast notifications, refer to How to: Send and Receive Tile Notifications for Windows Phone on MSDN.

Auxiliary Properties

Raw, Toast, Tile and DoubleSidedTile all inherit the base Message class. This class provides the following optional properties to give you greater control over your notification messages:

  • Id – a GUID value representing the message identifier.
  • DeliveryInterval – Defines when the message should be delivered to the device. This can either be immediate (default), within 450 seconds or within 900 seconds.


As soon as you’ve started trying to send notifications to devices, you’ll want to know if it gets through and/or why it failed. To handle this, Pusher.Send returns a SendMessageResponse object with the following properties:

  • ErrorMessage – The message returned from MPNS on the HTTP response.
  • MessageId – The
    identifier of the message. Only included if specified when sending.
  • HttpStatusCode – The HTTP status code (e.g. 200, 400, 401, etc.) of the HTTP response returned from MPNS.
  • HttpStatusDescription – The description of the HTTP status code returned from MPNS.
  • StatusCode – The actual status of the notification’s delivery returned from MPNS (i.e. Received, Queue Full, Suppressed or Dropped).
  • DeviceConnectionStatusCode – The status of the device’s connection to MPNS (i.e. Connected, Temporarily Disconnected, Inactive or Disconnected).
  • SubscriptionStatusCode – The status of the device’s subscription to MPNS (i.e. Active or Expired).
  • DetailsStatusDescription – The detailed description of the notification message’s status given the HTTP status code, MPNS status code, connection and subscription status according to the MSDN documentation.

For more information on response codes, refer to Push Notification Service Response Codes for Windows Phone on MSDN.

XPS 16 Mini-Review

At the request of a colleague, I’m writing a little review on my precious baby… 🙂

Before I start boring most, I’m going to go out there and say I agree with *almost* everything this guy says, so I’ll just rehash a few things and give my opinion.

Firstly, some quick specs:

  • Processor: Intel Core i7 820QM (8MB 1.73GHz – 3.066GHz)
  • Memory: 8GB – 2DIMM 1333MHz DDR3
  • HDD: 128 GB G.Skill Falcon SSD (aftermarket – orignially with 500GB 7200RPM)
  • Graphics: 1GB ATI Mobility Radeon HD 4670
  • Display: 16.0" 1080p Full HD RGBLED LCD with 2.0 MP Webcam
  • Optical Drive: 4X Blu-ray Disc Combo Drive (DVD/CD +/- RW +BD Read)
  • OS: Windows 7 Ultimate 64-bit
  • Wireless: Intel WiFi Link 5300 (802.11a/g/n)
  • Battery: 9-cell
  • Dimensions: 2.5-5cm x 38cm x 26cm with 9-cell battery (H x W x D) 

Build and Design


Just like the rest of the (Studio) XPS range, this laptop just looks sexy. It’s sleek, shiny, dressed in a little leather, and has all the shiny touch-sensitive media buttons to match. The backlit keyboard is an absolute pleasure to use with its big spaced keys. The unit has very little flex in it at all. The 16” frameless glossy RGBLED screen is so gorgeous it makes staring at code even more fun. All the ports are in the right place (security lock, VGA, Ethernet, DisplayPort, HDMI, 2x USB, 2x headphone, microphone, USB/eSATA combo, FireWire, ExpressCard and card reader). And the sound is awesome…


The shininess of the plastic means that fingerprints really stand out. The Synaptics multi-touch track pad is hard to use and can get very annoying when trying to do simple things like crop using in a picture editor. The gloss on the screen makes it pretty hard to use outside on a bright day. The slot-loader drive is not my favourite as it tends to be a bit noisy (compared to tray-loaders) and quite slow. Also, the battery doesn’t quite line up with the rest of the unit so there is a bit of a gap constantly staring me in the face as I’m typing.

Most importantly… THERE’S NO SEPARATE RAM AND HDD COVER ON THE MAIN PANEL!!! This really got on my nerves when I switched in my SSD, because it meant what is usually a 2 minute job on most laptops became substantially more due to the funny screw placements and fear of “screwing” something up.


I ran a few benchmarks in an earlier post, and found my PCMarks at 9018 and WEI at 6.7. The review on notebookreview had the PCMarks at 6303. It also had the WPrime 32M result at 31.827s, which I bested at 15.257s. I think it’s safe to say that this baby ain’t no slug… 🙂

However, there is usually one downfall to all this power that not even the XPS 16 could surpass… HEAT! This thing gets sooo hot that putting it to good use means it can’t be used as a “laptop”. Noise was not too bad, but probably because the fan isn’t actually doing much.

Also, the battery life is a bit too short for my liking. My 9-cell only gets me about 2.5hrs, which some might think is quite high given what it’s running, but I think could be improved.


I love  it. It’s sexy and fast. It falls down on the build quality a bit like most Dell laptops and can be used to help you get through the winters, but the bang for buck this machine offers more than makes up for it in my eyes.

Unboxing My New Laptop – Dell XPS 16

I finally received my new laptop… A Dell XPS 16. Specs:

Processor Intel Core i7 802QM (1.73GHz, 3.06GHz turbo)
RAM 8GB 1333MHz DDR3
Video 1GB ATI Mobility Radeon HD 4670
Optical Drive Slot Load  Blu-Ray BDROM, DVD +/- RW combo
Monitor 16″ Full HD RGBLED

So, first thing’s first… unbox it!





What next…? pull it apart of course!

Using a VHD to Store stuff in Windows 7

Firstly, I’m not taking the credit for this one. It was Paul Stovell’s idea, but because he’s currently blogless I’m going to post it on mine… 🙂

The other day Paul was talking about a way of utilising the new VHD features of Windows 7 to keep all his documents and important stuff in a single location so that he can back them all up by copying one file. He had created a VHD and written a script that mounts it as a drive at start up. I thought this was a great idea, so I reproduced it and am now sharing it.

Firstly, create the VHD either using Virtual PC or the Disk Management console Action > Create VHD. Attach the VHD in Disk Manager then initialise and format it. Detach it and we can start scripting the attach process.

Create a text file named “Attach VHD.txt” with the following contents in %windir%\System32\GroupPolicy\User\Scripts\Logon entering the location of the VHD:

SELECT VDISK file="<Location of VHD>"

NOTE: I’ve used U as the drive letter. You can change this if it will cause a conflict or you just don’t like it.

In the same location, create a batch file named “Attach VHD.cmd” with the following contents:

DISKPART /s "Attach VHD.txt"

Open the group policy editor (Hit Start then type “group policy”) and drill down to User Configuration > Windows Settings > Scripts (Logon).


Open the the Logon script properties and add the batch file you just created (it should open the location you created the files in by default).


Now, log out and on again. It may take a few seconds, but the autorun screen should pop up when the drive is attached. Open Windows Explorer and you will see a new drive in your tree.


That’s it!

I’ve actually now moved all my user folders (i.e. Documents, Music, etc.) to the VHD, but if you don’t want to do that you can just use the libraries to include a folder and set it as the save location.

My Visual Studio Colour scheme

Ok, so here’s the current version of my colour scheme

What’s it look like?


Hope you like it. 🙂