Running Karma Tests for TypeScript Node.js Code in WebStorm 7

After getting up and running with TypeScript Node.js development in WebStorm, I was pretty confident that I had everything set up and ready to go for my next project, I wanted to write some tests and realized I had no idea how to do that. I've just worked out a nice solution and my initial googling didn't come up with anything so I've decided it's worth a post.

My searches started out well, I found this blog post which goes through testing JavaScript using Karma and Jasmine and shows off some neat WebStorm integration, but this was written with "vanilla" JavaScript in mind, so even though it works with transpiled TypeScript, it doesn't work with Node.js modules. The reason is because it's running in a browser, so you can't pull in node modules using CommonJS-compiled TypeScript code. I briefly experimented with using AMD as the module system and getting the browser to use RequireJS, but that would have required me to have two different transpilations of my TypeScript code, and I couldn't get it to work anyway.

After a bit of poking around, I'm almost certain that "the way" to run jasmine Node.js tests is to use jasmine-node in place of the built-in Karma support in WebStorm. I figured I would just run jasmine-node from the terminal and deal with an un-integrated test runner, but I really wanted to be able to debug through my code while the tests were running. The way I've done this before was to use node-inspector, so I started searching around for a way to get that working with jasmine-node, and I found this StackOverflow question. The answer points out that jasmine-node is just an alias for a normal Node.js app, so I figured I would just add that as a Node.js run/debug configuration, and surprise surprise everything worked perfectly.

For posterity, here are all the steps I took:

1. Create a new empty WebStorm project.
2. Make a simple TypeScript file that we can test:

// File: Roster.ts
export class Person {
 constructor(public name:string) {}
}



3. Add the TypeScript file watcher, fix it by adding "--module commonjs" to the arguments.
4. Add karma-jasmine as an external library as shown in the JetBrains blog post video so that autocomplete and code generation work for the test spec files.
5. Get the jasmine TypeScript definitions from DefinitelyTyped.
6. Make a simple test, referencing the TypeScript code and jasmine definitions:

// File: test/PersonSpec.ts
/// 
/// 

import p = require("../Person");

describe("suite", () => {
 it("should be true", () => {
  expect(true).toBe(true);
  var person = new p.Person("Joe");
  expect(person.name).toBe("Joe");
 });
});



7. Make a new Node.js run configuration to run the tests, use the node_modules\jasmine-node\lib\jasmine-node\cli.js file as the JavaScript file (this can be either local or globally installed via npm), and the test folder as the application parameters.
8. I also suggest making another run configuration that is identical except for having an additional "--autotest" application parameter before the test folder. This will run in the background and continually re-run tests whenever a change is detected in the source code.

Now when you want to run tests or debug through the test, you can just use one of these new run configurations.

Following Along with the WebStorm 7 Demonstration Video

As I mentioned in my last post, I was thinking about starting a new project using Node.js, and this video for the new features in WebStorm 7.0 caught my eye:

It looks like a very slick IDE and I was looking forward to trying it out, but I ran into a whole lot of little problems when I tried to follow along. I'm pretty sure I've figured most of them out now, so I thought I'd try to go through the entire video and explain how to follow along with it.

First things first - making Roster.ts (0:00 - 0:30). This demonstrates the automatic compilation of TypeScript, so you'll first need to make sure TypeScript is properly set up. The file is created using the alt-insert shortcut, which lets you type in the type of file to create followed by the name of the file. When you first create a TypeScript file, you'll get a prompt about adding a File Watcher, which is what will automatically compile your TypeScript into JavaScript (along with a map file so you can debug through the TypeScript code). The first problem I encountered was that I had installed TypeScript 0.8 via an executable a while ago, so even though I installed the latest version (0.9.1-1) using npm, the file watcher was using the 0.8 version that was stored in a different location. Make sure you check that the watcher is pointing to the right file by running the compiler with the -v flag which will print out the version number. The second problem I encountered was that I kept getting the error "error TS5037: Cannot compile external modules unless the '--module' flag is provided." After a bit of module confusion as I mentioned in my last post, I realized that since we are writing Node.js code, we should specify that tsc should use the CommonJS module system, which you can do by editing the file watcher and adding --module "commonjs" to the arguments. With that confusing diversion dealt with, I was able to write up the Roster.ts file with the autocomplete working as it shows in the video. One small comment is that in the video they use a capital-cased String rather than a lower-case string, but that is easily changed.

Next is server.ts (0:30 - 1:43). This file uses some Node.js packages, so there are two "hidden" things to set up here. Firstly, you'll need to make a package.json file that sets up dependencies for the app. This example only uses express and cors, and we don't need to be strict about the version, so you can use the following file:

{
    "name": "application-name",
    "version": "0.0.1",
    "dependencies": { "express": "*", "cors": "*" }
}

Now you just need to run "npm install" from your project directory to pull in the required libraries. There are a few other ways to do this, you could just npm install the packages individually or use the Node.js and NPM settings, but keeping it in the package.json file means that we will be able to install the project in other places easily.

The second piece of setup is DefinitelyTyped, which is a set of files that contains the definitions for many different npm packages, so that we can compilate and use code completion for those packages. This can be downloaded or added as a git submodule using their github repo, or we can pull in individual definitions by going to the JavaScript -> Library settings and Downloading from the TypeScript community stubs section. I prefer to use a git submodule so that it's easy to update the definitions. I do like the idea of the community stubs, but as far as I can tell you need to copy and paste the file (renaming it in the process) into your projects main directory structure before you can use them properly, and that's too much work for me.

With this setup done, the creation of the server.ts file goes about as well as it shows in the video but there are some hidden tricks. I'm not entirely sure, but I think the magic that translates "ref" into "/// <reference path=""/>" is a custom Live Template, which I was able to add by making "ref" an abbreviation for "/// <reference path="$END$"/>". The extra "find relative references" magic can be conjured by using ctrl-space (possibly twice), which will bring up the list of relevant files. The next trick which automatically inserts a variable for "new jb.Roster()" is called Extract Variable, and can be accessed using ctrl-alt-v. One thing I couldn't get working was the explicit typing for ExpressApp/ExpressServerRequest etc., which I guess has had a change in name since the video was made. The IDE is smart enough to infer the type from the context so it's not a big deal. And for server.ts's final trick ... I guess it's some more Live Templates that are converting "get!" and "post!" to some pre-built function skeletons.

"Debug Node with ease" (1:43 - 1:58) really is as easy as they show (as long as you have the right npm packages installed), but "Test REST services" (1:58 - 2:30) is a bit trickier. The first thing to note is the "Enter action or option name" dialog which you can summon using ctrl-shift-a and use to find just about anything - in this case "test rest" opens up the REST client. Before it works as shown in the video you'll probably need to change the port and path and ensure that you've added the "content-type: application/json" header. Once that is sorted out though, take some time to play around with the pain-free TypeScript debugging, it really is pretty amazing when you think about it.

Now we get to the client side, the file app.ts starts out as a pretty standard AngularJS app, written with the assistance of the auto-completion and templates we've seen before (2:30 - 3:25). AngularJS is an awesome framework for client-side code but I'm not going to go into it here, if you want more information check out their website. The only note I have for this section is that my version of app.ts had a compilation error "The property 'message' does not exist on value of type 'AppCtrl'", which was easily fixed by adding message as a public variable on the class (public message:string;).

The next file that is created is a HTML file, but one that is created using "Emmet live templates" (3:25 - 3:40). This is something I have never seen before, but it looks like a bunch of shortcuts that expand into full HTML, nifty! It looks like it takes the best part of Jade - not needing to write HTML - with the added benefit of producing HTML rather than the sometimes confusing Jade markup. These are configured as Live Templates, so the usage and configuration is the same as the "ref" template we saw earlier. Find relative references also works inside script tags, so that is a few more keystrokes you can save here. But wait, what is this bower_components folder and where does it come from? This is another technology I've never seen before, but my minimal research leads me to believe that it is basically npm but for client side libraries. You create a bower.json file, specify the libraries you want to install, then run "bower install" to pull the files into your project (after installing it using "npm install -g bower"). This example only uses angular and angular-resource, so we can use the following (strangely familiar) file:

{
    "name": "application-name-client",
    "version": "0.0.1",
    "dependencies": { "angular": "*", "angular-resource": "*" }
}

Now the files can be found, and we can start putting some AngularJS into the HTML (3:40 - 3:55). As hinted at in the video, there is auto-complete for Angular directives, but it requires a plugin. To install it, go to the Plugins section of the settings, browse the repositories, and search for and install the AngularJS plugin. After restarting, the auto-completion should be working as shown in the video. I'm not sure what is converting "bi" to "{{}}" but I'm just going to assume its another custom template and move on.

The next part demonstrates how to start a debugger on the client as well as the server (3:55 - 4:25), which is pretty easy to follow along with but you might need to install the ExtensionJetBrains IDE Support Chrome extension. After a small change, we realize that the app dependencies aren't loaded correctly, so there is a bit of a detour to go an install that package, which is again easy to follow (4:25 - 4:45).

Next is some demonstrations on how to use browserify, which is yet again something I had to look up. A high level outline is that it looks like something that lets us write CommonJS/Node.js style require/export code in client-side code. Browserify can parse this code and use it to generate a bundled javascript file that can be included in the html. I'm not sure why this would be preferred over compiling TypeScript using the AMD module and using RequireJS to load the dependencies, but it does look pretty easy. The video shows a few different ways to call browserify (4:45 - 5:40), you can do it by running an interative terminal (alt-minus), or apparently through a "simpler command window", but I couldn't find that. You can also set up a file watcher that will detect when a change is made to app.js and automatically run browserify on it, similar to how JavaScript is created whenever you modify a TypeScript file. This is also easy to follow as long as you pause the video quite a bit so you can actually see what's being clicked on. With the bundle generated, we can now run and inspect our code.

The rest of the video is some code that you'll need to know a bit about AngularJS to understand, but it all works as shown (5:40 - 6:50), and that pretty much wraps up the video! Two things I still don't understand are how to get the camel-case expansion to work (e.g. RR => RosterResource), and what that "jump-to" shortcut is or what it's doing, but I'm sure I'll figure that out eventually.

Now to work on a real project in WebStorm!

Making Sense of Modules in TypeScript, Node.js/CommonJS, and AMD/RequireJS

After a little while in the land of pure JavaScript I've decided that I want to have another go at Node.js in TypeScript. I haven't looked into TypeScript development in about 9 months, and in that time a lot has changed. At version 0.9.1-1, the language has almost fully matured to that magical version 1.0, Visual Studio has added some new features that make some of the hacks that I described in my early blog posts unneccesary, and IDEs like WebStorm have done a great job of making it easy to work with TypeScript and Node.js, as they demonstrate in this video. I'm hoping to write a full run-down on exactly what's happening in that video, but as I tried to follow along with it, I ran into some confusion about how exactly modules work TypeScript. Since explaining is a good way of learning, today I'm going to try to go through what I've learnt as I tried to make sense of it all. Fair warning here, I'm not an expert, so I can't guarantee that everything I say is absolutely correct, but I can at least promise that it works.

Instead of going straight into the TypeScript implementation, it's best to start at JavaScript, which has two main module management systems, the AMD (Asynchronous Module Definition) API which is implemented by RequireJS and jQuery (among others), and CommonJS's module specification which is implemented by Node.js. Both of these systems attempt to solve the problem of compartmentalizing sections of code without polluting the global scope.

In AMD, modules are defined using the define keyword, which takes a name if you don't accept the default of a file-based module name (which in general you should), a list of dependencies which are specified in the array argument, and a function within which the dependencies are scoped to a variable. Only the object returned from the function is the public interface of the module, so variables and functions can be hidden within the module definition. Modules can be referenced using the require function, which acts similar to the define function in regards to its dependency management, but does not allow a module definition to be returned. The AMD implementation can see a module's dependencies and load them asynchronously on-demand, so this form of dependency management is usually used on the client-side where bandwidth and speed is important. An example of some JavaScript code that uses the AMD API is shown below:

// File: subdir/dependency1.js
define(function () {
    return function () {
        console.log("Hi from dep1!");
    };
});

// File: dependency2.js
define('dependency2', ['subdir/dependency1'], function (dep1) {
    var private = "Private";
    return {
        public: "Public",
        func: function () {
            console.log("Hi from dep2! " + private);
            dep1();
        }
    };
});

// File: main.js
require(["subdir/dependency1", "dependency2"], function(dep1, dep2) {
    dep1();
    dep2.func();
    console.log(dep2.public);
    console.log(dep2.private);
});

// Output
Hi from dep1!
Hi from dep2! Private
Hi from dep1! 
Public
undefined 

CommonJS is similar but does not use the same scoping system, each file is a module who's public interface is defined by any property added to an "exports" variable (or anything that is assigned to the module.exports variable), and you pull in a module and assign it to a variable by using a require command. Besides syntax, the main difference between CommonJS and AMD is that in CommonJS all modules are loaded on startup, so it is mostly used in server-side code. CommonJS code that produces identical behavior as the code above is shown below:

// File: subdir/dependency1.js
module.exports = function () {
    console.log("Hi from dep1!");
};

// File: dependency2.js
var dep1 = require('./subdir/dependency1');
var private = "Private";
exports.public = "Public",
exports.func = function () {
    console.log("Hi from dep2! " + private);
    dep1();
};

// File: main.js
var dep1 = require('./subdir/dependency1');
var dep2 = require('./dependency2');
dep1();
dep2.func();
console.log(dep2.public);
console.log(dep2.private);

// Output
Hi from dep1!
Hi from dep2! Private
Hi from dep1! 
Public
undefined 

And now we get to TypeScript, which has its own module syntax. Modules are defined using the "module" keyword, and their public interfaces are defined by exporting things from inside the module. Functions and variables can also be exported in the file, without them needing to be wrapped in a module. Modules and exported variables are imported using syntax similar to CommonJS, but with the import keyword in place of var. An example is shown below:

// File: externalModule.ts
export module ExternalModule {
 export function public () {
  console.log("ExternalModule.public");
 };
 function private () {
  console.log("ExternalModule.private");
 }
}

export function ExportedFunction() {
 console.log("ExportedFunction");
}

// File: main.ts
module InternalModule {
 export function public () {
  console.log("InternalModule.public");
 };
 function private () {
  console.log("InternalModule.private");
 }
}
InternalModule.public();
//InternalModule.private(); // Does not compile

import externalModule = require("externalModule");
externalModule.ExportedFunction();
externalModule.ExternalModule.public();
//externalModule.ExternalModule.private(); // Does not compile

// Output:
InternalModule.public
ExportedFunction
ExternalModule.public

In my opinion, since this syntax is more explicit, it is cleaner and easier to understand than AMD or CommonJS modules, but here's where things get confusing. Since TypeScript compiles into JavaScript, you can actually compile this code into either AMD or CommonJS by using the --module flag on the tsc command.

For example, when you run tsc externalModule.ts --module "amd", you get:

define(["require", "exports"], function(require, exports) {
    (function (ExternalModule) {
        function public() {
            console.log("ExternalModule.public");
        }
        ExternalModule.public = public;
        ;
        function private() {
            console.log("ExternalModule.private");
        }
    })(exports.ExternalModule || (exports.ExternalModule = {}));
    var ExternalModule = exports.ExternalModule;

    function ExportedFunction() {
        console.log("ExportedFunction");
    }
    exports.ExportedFunction = ExportedFunction;
});

When you run tsc externalModule.ts --module "commonjs" you get:

(function (ExternalModule) {
    function public() {
        console.log("ExternalModule.public");
    }
    ExternalModule.public = public;
    ;
    function private() {
        console.log("ExternalModule.private");
    }
})(exports.ExternalModule || (exports.ExternalModule = {}));
var ExternalModule = exports.ExternalModule;

function ExportedFunction() {
    console.log("ExportedFunction");
}
exports.ExportedFunction = ExportedFunction;

To further add to the confusion, since all valid JavaScript is also valid TypeScript, there is nothing stopping you from mixing and matching TypeScript modules, AMD modules, and CommonJS modules (as long as you have an implementation of the module loader e.g. RequireJS for AMD and Node.js for CommonJS). Given the recent confusion I experienced I would recommend you just stick with the TypeScript syntax and have that compile into CommonJS for Node.js server-side code and AMD for client-side code.

One last thing I want to mention is the reference path syntax in TypeScript, because that can add to the module confusion a bit. As an example, let's make a simple class:

// File: MyClass.ts
export module MyModule {
    export class MyClass {
        constructor(public str:String) {}
        public func() {
            console.log(str);
        }
    } 
}

We can use this class in another file by referencing it:

// File: main.ts
/// 
var myclass = new MyModule.MyClass("test");
myclass.func();

So why does this work without us needing to import any modules? Because MyModule.MyClass is compiled into a variable, and TypeScript doesn't know how you're going to load your JavaScript files. You could easily have a HTML file that includes both of these files in script tags and it would work fine. What the reference tag does is tell the compiler where to find definitions, so it can tell that the MyClass class is within the MyModule module and it has a func function, so the code in main.ts is valid. When you're writing Node.js however, modules need to be loaded using CommonJS, so you need to use import/require commands in addition to referencing the TypeScript files or definitions (which are still required for syntax checks).


Hope this helps someone, here is some recommended reading/watching:

• TypeScript Modules – Part 4
• TypeScript Modules Demystified : Internal, AMD with RequireJS, CommonJS with NodeJS
• Relation between CommonJS, AMD and RequireJS?

Steam Trash Bot

Quick post today for a quick project. After getting sick of seeing 4 beta invites for a free-to-play game that was released a long time ago in my inventory I thought I'd make a Steam bot that would automatically accept trade requests so I could finally get rid of the crap in my inventory. I also figured that I might as well make it so anyone can take anything of use out of the bot's inventory, since after all one person's trash is another person's treasure. Thanks in large part to the amazing steam and steam-trade Node libraries, this ended up being really easy to do, and the code is a bit unspectacular. The one interesting challenge I had to solve was how to make it easy to request an item from the chat bot. After thinking for a while and looking at the available API, I decided that the best way was to have a user paste a link to the item into the trade window's chat area. This link contained all the information needed to find the item in the bot's inventory in an easily parseable format, so that worked quite well.

If you like the sound of this, check out the code on GitHub or the bot itself here!

Online Steam Group Chat App

I've just finished off another project, so it's time for another blog post. This time my main goal was to get some experience with AngularJS, something I have been reading a lot about which seems like a pretty cool bit of tech. When I was trying to think of something to make, I remembered that in the early days of my Steam chat bot, I made a little website that would allow people to send a message to the chat room through the bot. It was just a bit of fun at the time but I thought I could expand on that and make a web interface into a group chat room, something that Steam has not yet provided for some reason, even though you can do user-to-user chat through their site. This would also give me an opportunity to use Socket.IO, something I'd used previously but not really well.

This took me a bit longer than I was expecting for a "simple AngularJS project" but at the end of the project I've learnt a lot and I've made a nice website that at least a few people should find useful. It looks and acts pretty much exactly like the Steam client which I'm pretty happy about, and thanks to Foundation it handles mobile/tablet resolutions quite nicely. I've put the code up over at GitHub, it should be fairly easy to set up for any Steam group so go and have a look if you're interested. Here is a blurry view of what it ended up looking like on the desktop (if the screen gets small enough, the chat member section gets hidden and you can click a button to switch between the chat view and the member view):

Instead of going into implementation details I'll just quickly go through the different technologies I used and explain them one by one.

The first thing I should mention is that I again made use of the excellent node-steam node library, which is what allowed me to mirror the chat messages and send messages to the chat room. I had considered using my chat bot, but in the end I really didn't need any of the features it provided and I decided to just use the vanilla library. There isn't much to go into for my usage of the library, I basically just used the provided functions to get and send messages, and to find out when user statuses changed so that I could update the list of users in the chat room.

For client-side code I used AngularJS, which as I mentioned was the main reason for starting work on this project. I was very impressed with Angular, whenever anything changed on the Steam side (a new message, a user joined or left, or a user's state changed), all I needed to do was update my model and the UI was magically updated. I also really liked the separation of controller/filter/directive/services. Although I didn't really make use of any services (except a pre-made Socket.IO one), it was great to be able to make a filter that built a URL for a Steam profile image given a set of hex numbers, or a directive that represented the display of a users information, and have that completely separated out from my main app code so I could just concentrate on the task at hand. I feel like I still have a lot to learn about Angular, and I'm pretty sure a lot of what I did wasn't the "correct" Angular way of doing things, but even just scratching the surface of what Angular has to offer has been very interesting.

For server-side code I used the express framework, which I've used for a few different projects before. Since I was working on a single-page app without too much back-end logic (besides things I'll mention separately), there isn't too much to go into with my usage of express. One thing I'll say about the server-side implementation in general is that I really should have used TypeScript. The reason I didn't was because I wanted to use plain JavaScript for the front-end, and I thought it might get a bit confusing to switch between the two. It was good to get a bit more experience with JavaScript, but I really feel like some sort of type checking is crucial to coding, and TypeScript would have been a good safety net to use.

The communication between server and client was handled mainly by Socket.IO. When someone visits the site, they have a socket set up for them which goes through some authentication. To be able to chat they need to be signed in through Steam, so the Socket.IO authentication has access to the user's Steam ID. I use this to make sure that the user should be able to see the chat room, and if they are I set up a bunch of socket listeners so that I can notify the client of any chat room changes, or other state changes such as notifications that the chat has been muted or that the proxy user has disconnected for some reason. One thing I noticed was that a lot of this code was just boilerplate, since I had a model of sorts on the server side and an Angular model on the client side, and most of the Socket.IO code was used to synchronize these two models. I was going to try to think of a way to build some sort of system that would automatically listen for any changes on the server side and send that down to the client side via sockets, but it was looking like it was going to take too much time. Now that I've finished the project I think I may revisit that, as it seems like something that would be very useful for anyone making an Angular app that needs to be notified of state changes from the server side.

For the website layout I decided to use Foundation. I was originally going to use Bootstrap, but I wasn't going to use any of their styles and I had heard that Foundation was better as a "clean slate" solution. Also admittedly I just read about Foundation and it looked like something new and interesting. In the end I'm pretty happy with the way it worked out, it is a bit simpler than Bootstrap and it had a few features that came in very handy, particularly Top Bar which handled my menu needs really well, and the Reveal modal dialog which was great to use as a state messaging system since I needed something that would block the normal use of the site and provide a message in a very obvious way. One thing I wasn't quite able to work out was a non-hacky way of providing a mobile and full-screen interface to the chat room. For normal website usage I wanted the chat room and member details to be shown side-by-side and for mobile use I wanted the user to be able to switch between the two by pressing a button. In the end I solved this by having two completely separate divs, one that is shown for desktop use and one that is shown for mobile use. This wasn't too bad since I made use of Angular scope and directives to separate out code that would otherwise have been duplicated, but I could tell that there must be a better way of doing it. Of course, this isn't a problem with Foundation, just my lack of experience with it.

One final thing I'll mention is the authentication solution I came up with. This is one thing that went through a few different iterations but I think the solution I ended up with works really well. The first part of authentication was simple enough, Steam provides an Open ID login library in their Web API, so all I needed to do was use an Open ID node library and I could get access to a user's Steam ID without any risks. The second part of authentication was trying to figure out if a particular Steam ID should be able to see the chat room. I had originally thought of having a whitelist, so a user would have to request access and I'd have to go in look at the user's profile page and approve if it looked OK, but I decided that was too much work. After this I thought I'd just make it a rule that a user would have to be in the chat room the first time they used the site, and that would automatically add them to the whitelist so they'd be able to log in without being in chat from then on, but that seemed a bit too confusing for users. After a bit more digging around in Steam's web API, I found out that they provided a way to see the groups a particular steam ID belongs to, so I decided to just use that. Any time a user logs in, I check to make sure they are part of the group, and allow them access if they are. This is completely transparent to the users and it keeps out randoms from the site. The only problem is that someone can be kicked/banned from the chat room and they'd still be able to use the site, but that rarely if ever happens in the steam group I'm using the site for so it's not a big concern.

So that's another project and rambling blog post done. Hope either of them come in handy for someone else at some point!

Building a Module Factory in Node.js

As part of my Steam Chat Bot I was trying to figure out a good way to make a factory in Node.js that could keep a registry of a bunch of subclasses and construct them based on the name of the class. I also wanted  to be able to save the details of a set of these subclasses as JSON so that they could be stored and loaded, and I wanted to have some way of loading the subclasses dynamically, so that I could just create a new file, drop it in the right folder, and be able to create an instance of the new subclass object without restarting my app. I'm sure a good instructional blog post on how to do this exists somewhere but I couldn't find it, and since I figured out a solution I thought I'd go through it.

To keep things simple, I'll go through the creation of a factory that creates different types of ducks, which will be simple objects that just have a single "quack" function that returns a string based on the type of duck and an attribute of the class. Whenever we call a create function on the factory, it will scan a folder to load all the different types of duck, and create a new duck object of a particular type. This way, we can add or modify files in the "ducks" folder and we'll be able to get a new object with those changes without taking down our app.

So first, here is our factory object, stored at ./duckFactory.js:

var fs = require('fs');

var DuckFactory = function() {
 this.duckTypes = {};
};

// Go through all files in the ducks directory and register the 
// create function to the duckType
DuckFactory.prototype.loadModules = function() {
 var files = fs.readdirSync('./ducks/');
 this.duckTypes = {};

 for (var i=0; i < files.length; i++) {
  var moduleName = './ducks/' + files[i];

  // Clear the module cache so module updates can be loaded
  delete require.cache[require.resolve(moduleName)];
  var module = require(moduleName);

  if (module.duckType && module.create) {
   this.duckTypes[module.duckType] = module.create;
  }
 }
}

// Create a duck with the specified type
DuckFactory.prototype.createDuck = function(type, name, details, refresh) {
 if (refresh) {
  this.loadModules();
 }

 if (type in this.duckTypes) {
  return this.duckTypes[type](name, details);
 }
 return null;
}

exports.DuckFactory = DuckFactory;

Whenever a duck is requested, we scan through a folder and load in each file as a module. We check to make sure that the module has a duckType string and a create function in its exports, and then tie the string to the function. Since we are clearing the module cache before loading the module using "delete require.cache[require.resolve(moduleName)];", we will always get the latest version of the module as well as any new module files that have been added to the folder. Deleting the cache isn't usually recommended since it can cause dependency cycles, but I trust myself to not do anything silly like that. Note that this factory leaves open the option to refresh the modules on creation, or externally by the client as needed.

Now that we know that a module just needs a duckType string and a create function, we can write a simple base duck class (./ducks/baseDuck.js):

var BaseDuck = function(type, name, details) {
 this.type = type;
 this.name = name;
 this.details = details || {};
};

exports.BaseDuck = BaseDuck;
var type = "BaseDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new BaseDuck(type, name, details);
};

BaseDuck.prototype.quack = function() {
 throw "abstract function";
}

There isn't much to say about this, it's a pretty standard base class, except for the fact that most constructor parameters are specified as a map instead of separate variables so that it can be a standard signature across classes with different options. I could have left this file out of the ducks folder, so that it was not possible to create a BaseDuck object, but I thought it was a bit neater to leave it there. It also leaves the option open to create a BaseDuck and override the quack function on that object, in case that is ever useful.

Now lets create a concrete subclass, MallardDuck (./ducks/mallardDuck.js):

var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var MallardDuck = function() {
 MallardDuck.super_.apply(this, arguments);
};

util.inherits(MallardDuck, BaseDuck);

var type = "MallardDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new MallardDuck(type, name, details);
};

MallardDuck.prototype.quack = function() {
 return this.details.attitude + " quack!";
}

The most important thing to note about this file is that it uses the inherits function from the utils module. This behaves similar to inheritance in object-oriented languages, but is implemented in a different way. It simply copies all the base classes prototype functions into the subclass, as well as copying the base classes constructor as a new function called _super. Although in our case we don't have any useful prototype functions in our base class, in real life we could put any shared duck functionality in the base class and automatically have access to it in our subclass. We do use the _super function though, passing it the arguments object so that we could conceivably change the signature of the base classes constructor without needing to change all the subclasses.

Let's also create a RubberDuck class for some variety (./ducks/rubberDuck.js):

var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var RubberDuck = function() {
 RubberDuck.super_.apply(this, arguments);
};

util.inherits(RubberDuck, BaseDuck);

var type = "RubberDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new RubberDuck(type, name, details);
};

RubberDuck.prototype.quack = function() {
 return "squeak in the " + this.details.speciality;
}

Now we'll make a fairly simple client that can demonstrate the use of the factory (./client.js):

var sys = require("sys");
var DuckFactory  = require('./duckFactory.js').DuckFactory;

var duckFactory = new DuckFactory();
var stdin = process.openStdin();
var ducks = {};

stdin.addListener("data", function(input) {
 input = input.toString().trim();
 var words = input.split(" ");
 if (words.length > 1) {
  var command = words[0];
  var params = input.replace(command, "").trim();

  if (command == 'create') {
   var paramMap = JSON.parse(params);
   createDuck(paramMap.type, paramMap.name, paramMap.details);
  }
  else if (command == 'quack') {
   quackDuck(params);
  }
 }
});

function createDuck(type, name, details) {
 var duck = duckFactory.createDuck(type, name, details, true);
 if (duck) {
  console.log("> Created " + duck.name + " of type " + duck.type);
  ducks[duck.name] = duck;
 }
 else {
  console.log("> No class found with name " + type);
 }
}

function quackDuck(name) {
 var duck = ducks[name];
 if (duck) {
  console.log("> " + duck.name + " the " + duck.type + 
   " says '" + duck.quack() + "'");
 }
 else {
  console.log("> No duck found with name " + name);
 }
}

We can use this client to create a MallardDuck and a RubberDuck, and we'll also try to create a RoboDuck which we haven't written yet.

node client
create { "name": "Adam", "type": "MallardDuck", "details": { "color": "brown", "attitude": "feisty" } }
> Created Adam of type MallardDuck
create { "name": "Bob", "type": "RubberDuck", "details": { "size": "small", "speciality": "bath" } }
> Created Bob of type RubberDuck
create { "name": "Carl", "type": "RoboDuck", "details": { "primeDirective": "DESTROY" } }
> No class found with name RoboDuck
quack Adam
> Adam the MallardDuck says 'feisty quack!'
quack Bob
> Bob the RubberDuck says 'squeak in the bath'
quack Carl
> No duck found with name Carl

Without quitting the app, we can make a change to MallardDuck (just removing the exclamation mark from the quack function) and add a new RoboDuck class file to the ducks folder (./ducks/roboDuck.js):

var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var RoboDuck = function() {
 RoboDuck.super_.apply(this, arguments);
};

var type = "RoboDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new RoboDuck(type, name, details);
};

util.inherits(RoboDuck, BaseDuck);

RoboDuck.prototype.quack = function() {
 return "I WILL " + this.details.primeDirective + " YOU!!!";
}

Now the magic happens, we can make a RoboDuck and an updated MallardDuck and see the results in our still-running client.

create { "name": "Carl", "type": "RoboDuck", "details": { "primeDirective": "DESTROY" } }
> Created Carl of type RoboDuck
create { "name": "Doug", "type": "MallardDuck", "details": { "color": "black", "attitude": "emo" } }
> Created Doug of type MallardDuck
quack Adam
> Adam the MallardDuck says 'feisty quack!'
quack Bob
> Bob the RubberDuck says 'squeak in the bath'
quack Carl
> Carl the RoboDuck says 'I WILL DESTROY YOU!!!'
quack Doug
> Doug the MallardDuck says 'emo quack'

There is still a bit of scaffolding code required in the subclasses, but I think this is a pretty neat solution. I've put the code up at github, and if anyone knows of a better way of implementing this type of factory, I'd love to hear it.

Steam Chat Bot

I'm going to keep this ball rolling and actually write a blog post about a project before moving on to my next one. I just finished this one yesterday so it's still very fresh in my mind, but I don't think there is too much to say about it so this post will be shorter than my last one.

For this project I made a chat bot for a Steam group chat room that I've been going to for a while. I got the idea to make this because some sort of bot started joining the chat room a while ago, I'm still not sure what its purpose is, but almost every day it will come into the chat room, say "hi" and then leave a few minutes later. After seeing that happen for a while, I was linked to another bot called Dota 2 Dispenser which was made as a way for people to share the Dota 2 invites that hang around uselessly in their inventory. Seeing that the Dota bot uses a Node.js library, I thought back to days I used to go to a few IRC chat rooms and I remembered that there were a few pretty hilarious bots around so I figured that making a chat bot for Steam would be a neat little project to work on. It started as just a bot that says a few in-jokes randomly, but it has expanded a bit as I found a few really useful Node.js modules. Right now it automatically responds to various messages, it is an interface into Cleverbot so that it can communicate in some sort of intelligent way, it searches YouTube or Wolfram Alpha and spits out the result to the chat room, and it automatically posts links to a Tumblr page.

The code is up on GitHub and the package is published on npm, so take a look if you're interested. The basic outline of the code is that there is a bot class that handles the connection to Steam and intercepts the messages that are provided by the steam node package. This bot has a set of trigger objects, which are notified of things like chat messages and they can choose to respond by calling a "send message" function on the bot. Each of the triggers had its own set of challenges, but many of them are basically just a wrapper around another node package, e.g. cleverbot-node, wolfram, tumblr.js, and youtube-feeds. I think the most interesting part of the implementation is the factory I made to create the triggers, but I'll leave the details of that to another post since I couldn't find another good description of how to do this online (although I'm sure it's been done before).

I decided to use plain JavaScript and Sublime Text as an editor for this project, and the main lesson I learnt was that I much prefer the TypeScript & Visual Studio combination. I was writing code a bit faster since I didn't need to worry about syntax as much, but this came at the expense of having annoying bugs pop up. After I had spent an hour trying to work out why something wasn't working, only to discover that I had misspelled a variable name, I started wishing I was writing TypeScript again. I did like using Sublime Text, but its code completion functionality can't compete with Visual Studio's Intellisense.

So that's another project done! It feels pretty good to actually be finishing projects now rather than spending months on them and then leaving them unfinished.

ScrobbleAlong

Once again I've moved on to a new project and stopped thinking about my old one, but now that I've wrapped up the new one I guess I really should come back and write something about the old one. This is mainly going to be an "after action report" so that I have an opportunity to go through the technology choices I made and think about what worked and what didn't. I'll go into a bit of technical details but I'm not really 100% happy with the code so I'm not going to put that up anywhere. But first ...

What Is It?

ScrobbleAlong is a website I made which does two (closely related) things. It's a background process that continuously polls the "now playing" feeds of a few radio stations I like and scrobbles all the tracks to last.fm.  It also has a frontend that lets other last.fm users select a radio station that they are listening to so that the songs can also be scrobbled to their account. It's a nifty little tool for people who like to scrobble absolutely everything they listen to, and it's also a nice way to see what various radio stations are playing. If I ever want to see what the latest popular tracks are back home in Australia, I can load up Triple J's last.fm page and check the most played tracks for the last few weeks.

The tool actually has a pretty long history, it started a few years ago as a Python script that just did the radio station scrobbling, but I was looking for something to do as a Node.js project, so I decided to update it and add the "scrobble along" functionality. I'm pretty happy with the way it turned out, if I did it again I would probably do a few things differently, but as a first "proper" Node.js project I think it worked out pretty well.

How Does It Work?

Scrobbling for the stations is handled by a task that runs every 15 seconds, which calls an update function for every station. This update function does a HTTP request for a URL that contains the details for the currently playing song. A parser takes the body of the request and extracts out the artist and song names, and this is compared against the last time the URL was queried. If the song changed since the last request, and it was playing for a long enough time (>30 seconds), the last song is scrobbled. If the new song is valid (e.g. something is actually playing), a "now playing" request is sent to last.fm. One fiddly thing here is that there is no way to tell how long the song will play for, so we tell last.fm that the song is 40 seconds long, and update it again in 30 seconds if it's still playing. 

Scrobbling for users (scrobbling along) is handled mainly using a nice last.fm Node.js module which supports callbacks that are fired when a song has just been scrobbled or a song has started playing. Using this it is fairly easy to have a list of users attached to each station which can be updated whenever the station is updated. Again, some cleverness needs to be applied to avoid problems related to an unknown song length, otherwise only a single now playing notice is sent, so it looks like the user only listens to each song for 30 seconds, then stops listening until the next song starts. For the scrobble along case, this is handled using an interval that fires every 30 seconds and updates the now playing details, which is killed when the song is eventually scrobbled.

The front-end only needs to update the backend storage (MongoDB in my case), which contains a list of users and the station that they are listening to. The stations are presented in a pretty nice way thanks to the Isotope library which works pretty well with Bootstrap to make a nice responsive design. I also chucked a little bit of socket.io in there so that each station "block" automatically updates its display of the latest played tracks.

Lessons Learnt

I used a whole lot of technologies that I'd never used before when writing Scrobble Along, so I'll quickly go through them with some short impressions on what I thought of them.

TypeScript

I've now written a pretty significant project in both TypeScript and plain JavaScript, and I've got to say I really like the safety net and cleaner code that TypeScript provides. I can be a bit annoying sometimes, especially when I was using a definition file that wasn't quite right and I had to keep updating the definition before the app would compile without errors. I also wasted a lot of time writing definitions for modules that didn't have any, which I probably wouldn't bother with again since the time spent was not really worth it. I think in the future I'll continue to use TypeScript but I'll make liberal use of the "any" keyword for modules that don't have any definitions.

Visual Studio

Visual Studio and TypeScript are a great combination. The Intellisense and code navigation options are brilliant. It's a shame that it's not possible to use the debugging tools, but I'm pretty certain I'm going to be using VS for all my TypeScript projects.

Node.js

This was my first attempt at using Node.js and I was very impressed. There are pre-made modules for basically anything you can think of, it reminds me of Python development in that sense. It's also a very rapid development process which is always a good thing. Once I got over my irrational hatred of everything JavaScript and realized that it's not all bad, I really started to enjoy it.

Jade

Using Jade instead of HTML is one thing I'm not sure I'll do again. On one hand, it is more compact and it's nice to be able to do things like looping and conditionals, but on the other hand it's a layer of abstraction around HTML that can be a bit confusing. Maybe my only problem was that Visual Studio doesn't handle Jade files nearly as well as it handles HTML, so I might stick with it but try using Sublime Text to edit it rather than Visual Studio.

MongoDB

I used MongoDB for my data storage, and it's been a great first NoDB experience. It's really easy to use, easy to set up locally, and powerful enough for my needs while still being simple to use. There is also a site called MongoLab that has a generous free tier that I'm using for my production data, so I'm keeping my operational costs at the best price point of $0.

Socket.io

I must admit that I really didn't use Socket.io the best way, mainly due to my inexperience and the fact that by the time I got up to using it I just wanted to get the project finished as soon as I could. Even though I used it really badly, it still worked pretty well for me, and I'm sure I'll find more uses for it now that I almost understand how to use it.

Isotope

This worked out really well for me, but only because I was doing something that was exactly what it was designed for. If I am ever making something that is a grid of things that needs to be able to be rearranged in a responsive way I'm sure I'll use it again.

Bootstrap

I have a love-hate relationship with Bootstrap, it's great when it works but it seems to force pages to be designed in a particular way that doesn't allow much flexibility. I'm sure this is because I'm not a designer and I'm just sticking with the defaults, but I still think I'll try to look into alternatives next time I'm designing a site. I must admit it is really easy to use if you just stick to the defaults though.

The End

So that's another project done! It's a bit buggy, but it works and at least a few other people are using it, so I'm counting it as a success.

Caption Captain

Before I write a post about my most recent project, I thought it would be a good idea to write about the thing I did before that. It had a much smaller scope, and I made it so long ago now that I'm in danger of forgetting absolutely everything what I did.

Before I start the explanation, I should just provide a link to the site - http://captioncapta.in. Glad that's out of the way! So Caption Captain is a game that picks one image and three titles at random from reddit, and the goal is to pick the title that the image was submitted with. With a subreddit like Reaction GIFs this is a pretty fun game and an entertaining way of browsing the latest submissions. The game also works with other subreddits, but obviously the ones that are not image-heavy will not work too well, and the captions are sometimes pretty obvious for subreddits like funny or gifs.

The main reason I wanted to make this was both to have a bit of practice with Dojo and Bootstrap, and to see what I could do with a site when I limited myself to having a minimal infrastructure - I wanted to implement everything in pure HTML and JavaScript and have no server-side logic. That means no database, no server to access remote URLs, and no URL routing. The site could be served as a single HTML file, but I split it up a bit for clarity. It's currently running on Amazon S3 so it has minimal cost and maintenance.

This post will probably be light on the code, but I've put up all the source on GitHub so you can follow along by going there.

The first problem I encountered was due to my technology choices. Bootstrap (the JavaScript part at least) is generally written to be used alongside jQuery instead of Dojo. I would also prefer to use jQuery, but I knew I had to use Dojo for another thing I was working on, so I wanted to get at least a little bit of practice with it. Thankfully, some kind soul has done the hard work and re-written Bootstrap's JavaScript modules in a format that works well with Dojo and they creatively named it Dojo Bootstrap. All I wanted to use was Bootstrap's Modal functionality, and with Dojo Bootstrap this was as easy as require-ing the Modal module and putting the right tags into the HTML.

The next problem to solve was thinking about how to get subreddit data. Usually I'd just write some code into the server to query reddit, then return that as part of a site or ajax request, but since I didn't want to actually use a server this was not possible. Luckily, reddit's JSON API supports the usage of JSONP, which means we can call it on the client side and use the response inside a JavaScript function. Using this I made a function that would attempt to get the next image and set of captions, and if there were not left it used JSONP to get some more images and captions from reddit before returning one of them. All of the image loading logic is in imageLoader.js but to summarize - getNextImage is the main function, this will return an image if we have it, otherwise it calls _loadImages. _loadImages calls a set of query functions asynchronously, one for a few different sort orders of the subreddit, and that function (_getDataFromReddit) is a fairly generic JSONP call. Once all of these queries finish, the results are filtered to make sure that the link is an image and that its score isn't too low, then it's separated into a set of images and captions that are ready for use.

The rest of the code and HTML (mainly in index.html and controller.js) is pretty standard scaffolding code so I won't go through most of it, but the last thing I wanted to mention was the solution to the problem of URL routing when not using a server. I wanted to support the selection of different subreddits, but the usual solution of having a unique URL for each subreddit and using URL routes to detect the subreddit would not work in a plain HTML page. However, JavaScript does have access to the full URL, including query string, so all we need to do is redirect to index.html with e.g. ?subreddit=reactiongifs, detect that on page load, and the problem is solved.

So that's it, the main thing I learnt is that I don't really like using Dojo as much as jQuery, at least for the "utility" functionality. I still think it's pretty good for UI work, and the rest of it has improved a lot recently, but it's hard to beat jQuery for things like AJAX and async calls. I also quite enjoyed the challenge of making a single-HTML page, and if I ever think of something else that would work in that form I'll have another go at it. It's just lovely to be able to shove something on S3 and forget about it while it keeps running happily in the background.

Code Jam 2013

Having recently finished my hobby project (more on that later ... probably), I was looking for something else to start working on. Coincidentally, the Code Jam was starting, so my choice was basically made for me. I had a go at the competition back in 2010, mainly to keep my knowledge of Python up to date, but for the next 2 years, it was scheduled in times that meant I couldn't dedicate enough time for it, but I had no such excuses this year!

Given my recent work in TypeScript and Node.js, I figured that would be an ideal choice as a platform, but I left it a bit late and didn't even start on the template code that I could use for all the problems until a few hours after the competition started. In the end, I managed to finish full solutions for 2 of the questions, a partial solution for 1, and an important learning experience for the last question. I'll quickly go through my experiences and some things I learned.

First - the template. In case anyone is unfamiliar with the competition, Code Jam is a set of coding questions that are presented as a set of textual inputs, resulting in a set of definite results. You are given an explanation of the problem, a small sample set of inputs, and the expected outputs for those inputs. After you've confirmed that your solution works for that sample set, you can download a small set of other inputs, and upload your solutions for them, and after that you get download a larger set of inputs and upload your solution for those. The input files are usually just a set of numbers or characters, one set for each test case, and the output format is standard, so it is very helpful to have some "framework" code put together to handle the input and output of the problems, so that you can quickly jump in and write the code specific to the problem without needing to reinvent the wheel every time. My template code is fairly boilerplate, so I'll mainly leave it to the code to tell the story, but I will mention one unexpected benefit of the choice to use Node.js - asynchronicity. When I was using Python, I was running tests one at a time, but using the lovely async package I was able to very easily tell node to run as many tests as possible at the same time. The basic idea of my template is that I read in the input file, convert it to an array of test cases, then instruct async to "map" that array through a solver function to a results array, and output those results when all the processing has been finished. I'm not really even sure if this ended up being necessary, but it still made me fall in love node just a bit more.

The first problem - tic-tac-toe-tomek - is understandably fairly easy to solve. The idea is that we have to have a look at the state of a game similar to tic-tac-toe, and say if someone won, if it was a draw, or if the game was not yet finished. The differences between the game and tic-tac-toe are that it's played on a 4x4 board, before players start placing Xs and Os a single T can be randomly placed on the board, and to win, a row or column or diagonal must contain all Xs and Ts, or all Os and Ts. Given that the board never got larger than 4x4, I chose to do this in a brute force-ish way and just iterate through all the rows, columns, and diagonals for each test case and check for the existence of dots, Xs, and Os (ignoring Ts completely). A dot meant that there was no winner in that row/column/diagonal, otherwise no Xs meant that O won and no Os meant that X won. My solution only took a few seconds to churn through the large dataset, so I was off to a good start.

The second problem - lawnmower - was a bit trickier, I was given a set of heights of square sections of grass, and needed to figure out if those heights could be created by the usage of a lawnmower with an adjustable trim length that must move across or up and down the whole yard. Thinking about this for a little while, and given that the set of heights was constrained to a maximum of 100 different lengths, I figured that this could be efficiently calculated by working through the heights one at a time from shortest to tallest. If we find any rows or columns that are completely made of up height 1, we can mark that row or column as "safe". Moving up the lengths, we can treat a row or column as safe if it either contains that height, or the exceptional row or column is also "safe". Figuring out the data model for this was another interesting challenge, but I think that the code explains that better than words could. I'm not exactly happy with all of the code, but it was getting late and it solved the large dataset in milliseconds so I called it a day.

The third problem - fair-and-square - was one that I didn't get a chance to complete fully. The problem involved finding all numbers between two limits that satisfied some constraints. The number must be a palindrome (same forwards and backwards), and it must be the square of another number that is a palindrome. I decided to take the easy way out for this one and I picked a solution that I knew would work for the first large dataset (where the largest possible limit was 10^14) but would definitely not work for the second large dataset (largest limit of 10^100). The easy way out was to just pre-calculate all possible numbers that satisfied the constraints, then iterate through that list for each test case, considering only the numbers that are between each test cases's limits. Since the number must be a square, we only need to go through each number up to the square root of the largest limit, which in the case of 10^14 is "only" 10^7, so as long as we can check 10 million numbers fairly efficiently, we are good to go. In the end this only took about a second and I'll leave it to the code to explain the details. When I had a look at the set of valid fair-and-square numbers, there was a pretty obvious pattern that I'm sure I could have extrapolated to the larger case if I tried hard enough, but I never quite got around to it.

The fourth and final problem - treasure - taught me a lesson that I had strangely enough been reminded of a few days ago - "your strongest intuitions are the ones that you should question first". The problem was that, given some keys of various types, and a set of treasure chests (along with information about the type of key that is needed to open the chest, and the keys that are inside the chest), we needed to work out if it was possible to open all the chests. I realized that this could be modeled as a graph traversal problem, with the nodes being the set of keys in hand and chests left unopened, and the edges being the act of using a key to open a chest. Since I was having a love affair with node's asynchronicity, my intuition told me that I could us an inefficient solution, as long as I paralellized the investigation of each set of edges. This, of course, turned out to be completely incorrect, but I only realized this after spending way too long trying to get my heard around how to use the async library, which often happens whenever I do anything tricky and try to do it aysnchronously. When I realized that my solution wasn't going to work, and I only had about an hour before the competition was over, I decided to throw in the towel. I still thought it was a pretty fun problem to work on, and I look forward to being able to see the solution and whatever the trick was to minimizing the number of graph traversals that needed to be made. I'm guessing there is some sort of smart look-ahead shortcut that can be used but it's still a bit of a mystery to me.

And that's it - I ended up placing at about 3500 out of 20,000 entrants and qualifying for the next round. Not a great result, but not too bad considering the fact that I went into it "blind", without a framework set up and without knowing too much about the libraries I would be using.

Secondary sort in dgrid

In my never ending attempt to harvest google search clicks help people in need, today I'm going to explain a way to get a secondary sort working in dgrid, something my google searches could not find a good solution for. I needed this because I was making a grid that could be sorted by several headings as well as being filtered. Everything worked fine for small data sets, but I guess due to the way that dgrid's in-memory store works, sometimes I would apply a filter and the order of the rows would get switched up a bit. The results were still sorted correctly, but the order of the rows that all contained identical values for the column being sorted changed randomly. For example, suppose I'm sorting a movie list by genre:

Then I apply a filter for movies made in Nigeria, the results would still be sorted by genre, but the order of all the movies in the same genre got mixed up:

You can see that in the original list, the Nigerian Action movies showed up in the order 43-8-4-13, but after the results were filtered, the order was 4-8-13-43. The best solution I thought to fix this would be to sort primarily by genre, and secondarily by ID. Since the ID is unique across all the rows, the order of movies that contain identical genres will stay the same.

Let's start with the some HTML and JavaScript to create the filterable grid, there are several tutorials online about how to get this far, so I won't go into any details.

<html>
<head>

<title>dgrid secondary filtering</title>
<script data-dojo-config="async: true" src="js/dojo/dojo.js"></script>
<script>
    require([
        "dgrid/OnDemandGrid", 
        "dojo/store/Memory", 
        "dojo/on", 
        "dojo/dom", 
        "dojo/domReady!"
        ], function(Grid, Memory, On, Dom) {

        var genres = ['Comedy','Action','Drama', 'Musical','Doco'];
        var countries = ['Australia','USA','Japan','England','Nigeria'];
        var data = [];
        for (var i=1; i<50; i++) {
            var title = "Movie " + i;
            var genre = genres[Math.floor(Math.random() * genres.length)];
            var country = countries[Math.floor(Math.random() * 
                countries.length)];
            data.push({id:i, title:title, genre:genre, country:country});
        }

        store = new Memory({ data: data });         
        grid = new Grid({
            store: store,
            query: filterData,
            columns: {id:"ID",title:"Title",genre:"Genre",country:"Country"}
        }, "grid");

        grid.set('sort', [ 
            { attribute : 'genre', descending : false },
            { attribute : 'id', descending : false }
        ]); 
        
        On(Dom.byId("country-select"), "change", function(evt) {
            grid.refresh();
        });
        On(Dom.byId("genre-select"), "change", function(evt) {
            grid.refresh();
        });

        function filterData(item, index, items) {
            var countrySel = Dom.byId("country-select");
            var country = countrySel.options[countrySel.selectedIndex].value;

            var genreSel = Dom.byId("genre-select");
            var genre = genreSel.options[genreSel.selectedIndex].value;

            return (!country || country == item.country) &&
                    (!genre || genre == item.genre);
        }
    });
</script>

<style>
    body { font-family: sans-serif; font-size: 11px; }
    .claro .ui-state-default.dgrid-row-odd { background: #EEE; }
</style>
</head>
<body class="claro">
<div>
    <label for="genre-select">Genre:</label>
    <select id="genre-select">
        <option value="">All</option>
        <option value="Comedy">Comedy</option>
        <option value="Action">Action</option>
        <option value="Drama">Drama</option>
        <option value="Musical">Musical</option>
        <option value="Doco">Doco</option>
    </select>
    <label for="country-select">Country:</label>
    <select id="country-select">
        <option value="">All</option>
        <option value="Australia">Australia</option>
        <option value="England">England</option>
        <option value="India">India</option>
        <option value="Japan">Japan</option>
        <option value="Nigeria">Nigeria</option>
        <option value="USA">USA</option>
    </select>
</div>
<div id="grid">
</div>
</body>
</html>

Although it's not too easy to find in the API, it is possible to sort a grid by multiple columns. For example, we could add this code to set up a default sort which would use a secondary sort on ID to retain the order of rows after filtering:

grid.set('sort', [ 
    { attribute : 'genre', descending : false },
    { attribute : 'id', descending : false }
]);

This works for the default sort, but if we try to sort by country and filter by genre, we run into the same problems. Luckily, dgrid allows you to intercept the sort event and override it with your own implementation by using:

grid.on("dgrid-sort", onSort);

We can then prevent the usual sort for happening by calling preventDefault/stopPropagation on the event passed to our sort function, and then use the sort information from the event, but add our own secondary sort option:

function onSort(event) {
    // Stop the normal sort event/bubbling
    event.preventDefault();
    event.stopPropagation();

    updateSort(event.sort[0]);
}

function updateSort(sortType) {
    // Always sort secondarily by ID so that a definite order is kept 
    grid.set('sort', [
        sortType,
        { attribute : 'id', descending : false }
    ]); 
}

And there you go, sort order maintained, mission accomplished!

Here is the full code:

<html>
<head>


<title>dgrid secondary filtering</title>
<script data-dojo-config="async: true" src="js/dojo/dojo.js"></script>
<script>
    require([
        "dgrid/OnDemandGrid", 
        "dojo/store/Memory", 
        "dojo/on", 
        "dojo/dom", 
        "dojo/domReady!"
        ], function(Grid, Memory, On, Dom) {

        var genres = ['Comedy','Action','Drama', 'Musical','Doco'];
        var countries = ['Australia','USA','Japan','England','Nigeria'];
        var data = [];
        for (var i=1; i<50; i++) {
            var title = "Movie " + i;
            var genre = genres[Math.floor(Math.random() * genres.length)];
            var country = countries[Math.floor(Math.random() * 
                countries.length)];
            data.push({id:i, title:title, genre:genre, country:country});
        }

        store = new Memory({ data: data });         
        grid = new Grid({
            store: store,
            query: filterData,
            columns: {id:"ID",title:"Title",genre:"Genre",country:"Country"}
        }, "grid");

        grid.on("dgrid-sort", onSort);
        updateSort({ attribute : 'genre', descending : false });
        
        On(Dom.byId("country-select"), "change", function(evt) {
            grid.refresh();
        });
        On(Dom.byId("genre-select"), "change", function(evt) {
            grid.refresh();
        });

        function filterData(item, index, items) {
            var countrySel = Dom.byId("country-select");
            var country = countrySel.options[countrySel.selectedIndex].value;

            var genreSel = Dom.byId("genre-select");
            var genre = genreSel.options[genreSel.selectedIndex].value;

            return (!country || country == item.country) &&
                    (!genre || genre == item.genre);
        }

        function onSort(event) {
            // Stop the normal sort event/bubbling
            event.preventDefault();
            event.stopPropagation();

            updateSort(event.sort[0]);
        }

        function updateSort(sortType) {
            // Always sort secondarily by ID so that a definite order is kept 
            grid.set('sort', [
                sortType,
                { attribute : 'id', descending : false }
            ]); 
        }
    });
</script>

<style>
    body { font-family: sans-serif; font-size: 11px; }
    .claro .ui-state-default.dgrid-row-odd { background: #EEE; }
</style>
</head>
<body class="claro">
<div>
    <label for="genre-select">Genre:</label>
    <select id="genre-select">
        <option value="">All</option>
        <option value="Comedy">Comedy</option>
        <option value="Action">Action</option>
        <option value="Drama">Drama</option>
        <option value="Musical">Musical</option>
        <option value="Doco">Doco</option>
    </select>
    <label for="country-select">Country:</label>
    <select id="country-select">
        <option value="">All</option>
        <option value="Australia">Australia</option>
        <option value="England">England</option>
        <option value="India">India</option>
        <option value="Japan">Japan</option>
        <option value="Nigeria">Nigeria</option>
        <option value="USA">USA</option>
    </select>
</div>
<div id="grid">
</div>
</body>
</html>

Hosted MongoDB connection problems

Just a quick story/solution today so that I don't completely neglect this blog. I recently changed a little project from using Azure to using AppFog, and everything was going fine until I started having some random connection issues with MongoDB. After deployment everything would work fine for a while, but then a few minutes later I wouldn't be able to connect the the database and I just got a random error or timeout.

After scratching my head for a while, Googling around for a solution, and changing my DB layer implementation a few times, I was out of ideas. Then I started to look into my configuration. I have my DB hosted at mongolab and I hadn't updated the hosting location - it was still hosted on Azure, but my AppFog site was hosted on AWS. I didn't think this would be a problem, but I cloned my DB and hosted it on AWS, and then everything started working perfectly.

So if anyone happens to have this problem, and you somehow manage to end up at this site - just make sure that your DB is hosted at the same place as your app.