Sunday, June 16, 2013

Building a Module Factory in Node.js

As part of my Steam Chat Bot I was trying to figure out a good way to make a factory in Node.js that could keep a registry of a bunch of subclasses and construct them based on the name of the class. I also wanted  to be able to save the details of a set of these subclasses as JSON so that they could be stored and loaded, and I wanted to have some way of loading the subclasses dynamically, so that I could just create a new file, drop it in the right folder, and be able to create an instance of the new subclass object without restarting my app. I'm sure a good instructional blog post on how to do this exists somewhere but I couldn't find it, and since I figured out a solution I thought I'd go through it.

To keep things simple, I'll go through the creation of a factory that creates different types of ducks, which will be simple objects that just have a single "quack" function that returns a string based on the type of duck and an attribute of the class. Whenever we call a create function on the factory, it will scan a folder to load all the different types of duck, and create a new duck object of a particular type. This way, we can add or modify files in the "ducks" folder and we'll be able to get a new object with those changes without taking down our app.

So first, here is our factory object, stored at ./duckFactory.js:
var fs = require('fs');

var DuckFactory = function() {
 this.duckTypes = {};
};

// Go through all files in the ducks directory and register the 
// create function to the duckType
DuckFactory.prototype.loadModules = function() {
 var files = fs.readdirSync('./ducks/');
 this.duckTypes = {};

 for (var i=0; i < files.length; i++) {
  var moduleName = './ducks/' + files[i];

  // Clear the module cache so module updates can be loaded
  delete require.cache[require.resolve(moduleName)];
  var module = require(moduleName);

  if (module.duckType && module.create) {
   this.duckTypes[module.duckType] = module.create;
  }
 }
}

// Create a duck with the specified type
DuckFactory.prototype.createDuck = function(type, name, details, refresh) {
 if (refresh) {
  this.loadModules();
 }

 if (type in this.duckTypes) {
  return this.duckTypes[type](name, details);
 }
 return null;
}

exports.DuckFactory = DuckFactory;

Whenever a duck is requested, we scan through a folder and load in each file as a module. We check to make sure that the module has a duckType string and a create function in its exports, and then tie the string to the function. Since we are clearing the module cache before loading the module using "delete require.cache[require.resolve(moduleName)];", we will always get the latest version of the module as well as any new module files that have been added to the folder. Deleting the cache isn't usually recommended since it can cause dependency cycles, but I trust myself to not do anything silly like that. Note that this factory leaves open the option to refresh the modules on creation, or externally by the client as needed.

Now that we know that a module just needs a duckType string and a create function, we can write a simple base duck class (./ducks/baseDuck.js):
var BaseDuck = function(type, name, details) {
 this.type = type;
 this.name = name;
 this.details = details || {};
};

exports.BaseDuck = BaseDuck;
var type = "BaseDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new BaseDuck(type, name, details);
};

BaseDuck.prototype.quack = function() {
 throw "abstract function";
}

There isn't much to say about this, it's a pretty standard base class, except for the fact that most constructor parameters are specified as a map instead of separate variables so that it can be a standard signature across classes with different options. I could have left this file out of the ducks folder, so that it was not possible to create a BaseDuck object, but I thought it was a bit neater to leave it there. It also leaves the option open to create a BaseDuck and override the quack function on that object, in case that is ever useful.

Now lets create a concrete subclass, MallardDuck (./ducks/mallardDuck.js):
var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var MallardDuck = function() {
 MallardDuck.super_.apply(this, arguments);
};

util.inherits(MallardDuck, BaseDuck);

var type = "MallardDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new MallardDuck(type, name, details);
};

MallardDuck.prototype.quack = function() {
 return this.details.attitude + " quack!";
}

The most important thing to note about this file is that it uses the inherits function from the utils module. This behaves similar to inheritance in object-oriented languages, but is implemented in a different way. It simply copies all the base classes prototype functions into the subclass, as well as copying the base classes constructor as a new function called _super. Although in our case we don't have any useful prototype functions in our base class, in real life we could put any shared duck functionality in the base class and automatically have access to it in our subclass. We do use the _super function though, passing it the arguments object so that we could conceivably change the signature of the base classes constructor without needing to change all the subclasses.

Let's also create a RubberDuck class for some variety (./ducks/rubberDuck.js):
var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var RubberDuck = function() {
 RubberDuck.super_.apply(this, arguments);
};

util.inherits(RubberDuck, BaseDuck);

var type = "RubberDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new RubberDuck(type, name, details);
};

RubberDuck.prototype.quack = function() {
 return "squeak in the " + this.details.speciality;
}

Now we'll make a fairly simple client that can demonstrate the use of the factory (./client.js):
var sys = require("sys");
var DuckFactory  = require('./duckFactory.js').DuckFactory;

var duckFactory = new DuckFactory();
var stdin = process.openStdin();
var ducks = {};

stdin.addListener("data", function(input) {
 input = input.toString().trim();
 var words = input.split(" ");
 if (words.length > 1) {
  var command = words[0];
  var params = input.replace(command, "").trim();

  if (command == 'create') {
   var paramMap = JSON.parse(params);
   createDuck(paramMap.type, paramMap.name, paramMap.details);
  }
  else if (command == 'quack') {
   quackDuck(params);
  }
 }
});

function createDuck(type, name, details) {
 var duck = duckFactory.createDuck(type, name, details, true);
 if (duck) {
  console.log("> Created " + duck.name + " of type " + duck.type);
  ducks[duck.name] = duck;
 }
 else {
  console.log("> No class found with name " + type);
 }
}

function quackDuck(name) {
 var duck = ducks[name];
 if (duck) {
  console.log("> " + duck.name + " the " + duck.type + 
   " says '" + duck.quack() + "'");
 }
 else {
  console.log("> No duck found with name " + name);
 }
}

We can use this client to create a MallardDuck and a RubberDuck, and we'll also try to create a RoboDuck which we haven't written yet.
node client
create { "name": "Adam", "type": "MallardDuck", "details": { "color": "brown", "attitude": "feisty" } }
> Created Adam of type MallardDuck
create { "name": "Bob", "type": "RubberDuck", "details": { "size": "small", "speciality": "bath" } }
> Created Bob of type RubberDuck
create { "name": "Carl", "type": "RoboDuck", "details": { "primeDirective": "DESTROY" } }
> No class found with name RoboDuck
quack Adam
> Adam the MallardDuck says 'feisty quack!'
quack Bob
> Bob the RubberDuck says 'squeak in the bath'
quack Carl
> No duck found with name Carl

Without quitting the app, we can make a change to MallardDuck (just removing the exclamation mark from the quack function) and add a new RoboDuck class file to the ducks folder (./ducks/roboDuck.js):
var util = require('util');
var BaseDuck = require('./baseDuck.js').BaseDuck;

var RoboDuck = function() {
 RoboDuck.super_.apply(this, arguments);
};

var type = "RoboDuck";
exports.duckType = type;
exports.create = function(name, details) {
 return new RoboDuck(type, name, details);
};

util.inherits(RoboDuck, BaseDuck);

RoboDuck.prototype.quack = function() {
 return "I WILL " + this.details.primeDirective + " YOU!!!";
}

Now the magic happens, we can make a RoboDuck and an updated MallardDuck and see the results in our still-running client.
create { "name": "Carl", "type": "RoboDuck", "details": { "primeDirective": "DESTROY" } }
> Created Carl of type RoboDuck
create { "name": "Doug", "type": "MallardDuck", "details": { "color": "black", "attitude": "emo" } }
> Created Doug of type MallardDuck
quack Adam
> Adam the MallardDuck says 'feisty quack!'
quack Bob
> Bob the RubberDuck says 'squeak in the bath'
quack Carl
> Carl the RoboDuck says 'I WILL DESTROY YOU!!!'
quack Doug
> Doug the MallardDuck says 'emo quack'

There is still a bit of scaffolding code required in the subclasses, but I think this is a pretty neat solution. I've put the code up at github, and if anyone knows of a better way of implementing this type of factory, I'd love to hear it.

Sunday, June 9, 2013

Steam Chat Bot

I'm going to keep this ball rolling and actually write a blog post about a project before moving on to my next one. I just finished this one yesterday so it's still very fresh in my mind, but I don't think there is too much to say about it so this post will be shorter than my last one.

For this project I made a chat bot for a Steam group chat room that I've been going to for a while. I got the idea to make this because some sort of bot started joining the chat room a while ago, I'm still not sure what its purpose is, but almost every day it will come into the chat room, say "hi" and then leave a few minutes later. After seeing that happen for a while, I was linked to another bot called Dota 2 Dispenser which was made as a way for people to share the Dota 2 invites that hang around uselessly in their inventory. Seeing that the Dota bot uses a Node.js library, I thought back to days I used to go to a few IRC chat rooms and I remembered that there were a few pretty hilarious bots around so I figured that making a chat bot for Steam would be a neat little project to work on. It started as just a bot that says a few in-jokes randomly, but it has expanded a bit as I found a few really useful Node.js modules. Right now it automatically responds to various messages, it is an interface into Cleverbot so that it can communicate in some sort of intelligent way, it searches YouTube or Wolfram Alpha and spits out the result to the chat room, and it automatically posts links to a Tumblr page.

The code is up on GitHub and the package is published on npm, so take a look if you're interested. The basic outline of the code is that there is a bot class that handles the connection to Steam and intercepts the messages that are provided by the steam node package. This bot has a set of trigger objects, which are notified of things like chat messages and they can choose to respond by calling a "send message" function on the bot. Each of the triggers had its own set of challenges, but many of them are basically just a wrapper around another node package, e.g. cleverbot-node, wolfram, tumblr.js, and youtube-feeds. I think the most interesting part of the implementation is the factory I made to create the triggers, but I'll leave the details of that to another post since I couldn't find another good description of how to do this online (although I'm sure it's been done before).

I decided to use plain JavaScript and Sublime Text as an editor for this project, and the main lesson I learnt was that I much prefer the TypeScript & Visual Studio combination. I was writing code a bit faster since I didn't need to worry about syntax as much, but this came at the expense of having annoying bugs pop up. After I had spent an hour trying to work out why something wasn't working, only to discover that I had misspelled a variable name, I started wishing I was writing TypeScript again. I did like using Sublime Text, but its code completion functionality can't compete with Visual Studio's Intellisense.

So that's another project done! It feels pretty good to actually be finishing projects now rather than spending months on them and then leaving them unfinished.

Saturday, June 8, 2013

ScrobbleAlong

Once again I've moved on to a new project and stopped thinking about my old one, but now that I've wrapped up the new one I guess I really should come back and write something about the old one. This is mainly going to be an "after action report" so that I have an opportunity to go through the technology choices I made and think about what worked and what didn't. I'll go into a bit of technical details but I'm not really 100% happy with the code so I'm not going to put that up anywhere. But first ...


What Is It?


ScrobbleAlong is a website I made which does two (closely related) things. It's a background process that continuously polls the "now playing" feeds of a few radio stations I like and scrobbles all the tracks to last.fm.  It also has a frontend that lets other last.fm users select a radio station that they are listening to so that the songs can also be scrobbled to their account. It's a nifty little tool for people who like to scrobble absolutely everything they listen to, and it's also a nice way to see what various radio stations are playing. If I ever want to see what the latest popular tracks are back home in Australia, I can load up Triple J's last.fm page and check the most played tracks for the last few weeks.

The tool actually has a pretty long history, it started a few years ago as a Python script that just did the radio station scrobbling, but I was looking for something to do as a Node.js project, so I decided to update it and add the "scrobble along" functionality. I'm pretty happy with the way it turned out, if I did it again I would probably do a few things differently, but as a first "proper" Node.js project I think it worked out pretty well.


How Does It Work?


Scrobbling for the stations is handled by a task that runs every 15 seconds, which calls an update function for every station. This update function does a HTTP request for a URL that contains the details for the currently playing song. A parser takes the body of the request and extracts out the artist and song names, and this is compared against the last time the URL was queried. If the song changed since the last request, and it was playing for a long enough time (>30 seconds), the last song is scrobbled. If the new song is valid (e.g. something is actually playing), a "now playing" request is sent to last.fm. One fiddly thing here is that there is no way to tell how long the song will play for, so we tell last.fm that the song is 40 seconds long, and update it again in 30 seconds if it's still playing. 

Scrobbling for users (scrobbling along) is handled mainly using a nice last.fm Node.js module which supports callbacks that are fired when a song has just been scrobbled or a song has started playing. Using this it is fairly easy to have a list of users attached to each station which can be updated whenever the station is updated. Again, some cleverness needs to be applied to avoid problems related to an unknown song length, otherwise only a single now playing notice is sent, so it looks like the user only listens to each song for 30 seconds, then stops listening until the next song starts. For the scrobble along case, this is handled using an interval that fires every 30 seconds and updates the now playing details, which is killed when the song is eventually scrobbled.

The front-end only needs to update the backend storage (MongoDB in my case), which contains a list of users and the station that they are listening to. The stations are presented in a pretty nice way thanks to the Isotope library which works pretty well with Bootstrap to make a nice responsive design. I also chucked a little bit of socket.io in there so that each station "block" automatically updates its display of the latest played tracks.


Lessons Learnt


I used a whole lot of technologies that I'd never used before when writing Scrobble Along, so I'll quickly go through them with some short impressions on what I thought of them.


TypeScript


I've now written a pretty significant project in both TypeScript and plain JavaScript, and I've got to say I really like the safety net and cleaner code that TypeScript provides. I can be a bit annoying sometimes, especially when I was using a definition file that wasn't quite right and I had to keep updating the definition before the app would compile without errors. I also wasted a lot of time writing definitions for modules that didn't have any, which I probably wouldn't bother with again since the time spent was not really worth it. I think in the future I'll continue to use TypeScript but I'll make liberal use of the "any" keyword for modules that don't have any definitions.


Visual Studio


Visual Studio and TypeScript are a great combination. The Intellisense and code navigation options are brilliant. It's a shame that it's not possible to use the debugging tools, but I'm pretty certain I'm going to be using VS for all my TypeScript projects.


Node.js


This was my first attempt at using Node.js and I was very impressed. There are pre-made modules for basically anything you can think of, it reminds me of Python development in that sense. It's also a very rapid development process which is always a good thing. Once I got over my irrational hatred of everything JavaScript and realized that it's not all bad, I really started to enjoy it.


Jade


Using Jade instead of HTML is one thing I'm not sure I'll do again. On one hand, it is more compact and it's nice to be able to do things like looping and conditionals, but on the other hand it's a layer of abstraction around HTML that can be a bit confusing. Maybe my only problem was that Visual Studio doesn't handle Jade files nearly as well as it handles HTML, so I might stick with it but try using Sublime Text to edit it rather than Visual Studio.


MongoDB


I used MongoDB for my data storage, and it's been a great first NoDB experience. It's really easy to use, easy to set up locally, and powerful enough for my needs while still being simple to use. There is also a site called MongoLab that has a generous free tier that I'm using for my production data, so I'm keeping my operational costs at the best price point of $0.


Socket.io


I must admit that I really didn't use Socket.io the best way, mainly due to my inexperience and the fact that by the time I got up to using it I just wanted to get the project finished as soon as I could. Even though I used it really badly, it still worked pretty well for me, and I'm sure I'll find more uses for it now that I almost understand how to use it.


Isotope


This worked out really well for me, but only because I was doing something that was exactly what it was designed for. If I am ever making something that is a grid of things that needs to be able to be rearranged in a responsive way I'm sure I'll use it again.


Bootstrap


I have a love-hate relationship with Bootstrap, it's great when it works but it seems to force pages to be designed in a particular way that doesn't allow much flexibility. I'm sure this is because I'm not a designer and I'm just sticking with the defaults, but I still think I'll try to look into alternatives next time I'm designing a site. I must admit it is really easy to use if you just stick to the defaults though.

The End


So that's another project done! It's a bit buggy, but it works and at least a few other people are using it, so I'm counting it as a success.