ScrobbleAlong

Once again I've moved on to a new project and stopped thinking about my old one, but now that I've wrapped up the new one I guess I really should come back and write something about the old one. This is mainly going to be an "after action report" so that I have an opportunity to go through the technology choices I made and think about what worked and what didn't. I'll go into a bit of technical details but I'm not really 100% happy with the code so I'm not going to put that up anywhere. But first ...

What Is It?

ScrobbleAlong is a website I made which does two (closely related) things. It's a background process that continuously polls the "now playing" feeds of a few radio stations I like and scrobbles all the tracks to last.fm.  It also has a frontend that lets other last.fm users select a radio station that they are listening to so that the songs can also be scrobbled to their account. It's a nifty little tool for people who like to scrobble absolutely everything they listen to, and it's also a nice way to see what various radio stations are playing. If I ever want to see what the latest popular tracks are back home in Australia, I can load up Triple J's last.fm page and check the most played tracks for the last few weeks.

The tool actually has a pretty long history, it started a few years ago as a Python script that just did the radio station scrobbling, but I was looking for something to do as a Node.js project, so I decided to update it and add the "scrobble along" functionality. I'm pretty happy with the way it turned out, if I did it again I would probably do a few things differently, but as a first "proper" Node.js project I think it worked out pretty well.

How Does It Work?

Scrobbling for the stations is handled by a task that runs every 15 seconds, which calls an update function for every station. This update function does a HTTP request for a URL that contains the details for the currently playing song. A parser takes the body of the request and extracts out the artist and song names, and this is compared against the last time the URL was queried. If the song changed since the last request, and it was playing for a long enough time (>30 seconds), the last song is scrobbled. If the new song is valid (e.g. something is actually playing), a "now playing" request is sent to last.fm. One fiddly thing here is that there is no way to tell how long the song will play for, so we tell last.fm that the song is 40 seconds long, and update it again in 30 seconds if it's still playing. 

Scrobbling for users (scrobbling along) is handled mainly using a nice last.fm Node.js module which supports callbacks that are fired when a song has just been scrobbled or a song has started playing. Using this it is fairly easy to have a list of users attached to each station which can be updated whenever the station is updated. Again, some cleverness needs to be applied to avoid problems related to an unknown song length, otherwise only a single now playing notice is sent, so it looks like the user only listens to each song for 30 seconds, then stops listening until the next song starts. For the scrobble along case, this is handled using an interval that fires every 30 seconds and updates the now playing details, which is killed when the song is eventually scrobbled.

The front-end only needs to update the backend storage (MongoDB in my case), which contains a list of users and the station that they are listening to. The stations are presented in a pretty nice way thanks to the Isotope library which works pretty well with Bootstrap to make a nice responsive design. I also chucked a little bit of socket.io in there so that each station "block" automatically updates its display of the latest played tracks.

Lessons Learnt

I used a whole lot of technologies that I'd never used before when writing Scrobble Along, so I'll quickly go through them with some short impressions on what I thought of them.

TypeScript

I've now written a pretty significant project in both TypeScript and plain JavaScript, and I've got to say I really like the safety net and cleaner code that TypeScript provides. I can be a bit annoying sometimes, especially when I was using a definition file that wasn't quite right and I had to keep updating the definition before the app would compile without errors. I also wasted a lot of time writing definitions for modules that didn't have any, which I probably wouldn't bother with again since the time spent was not really worth it. I think in the future I'll continue to use TypeScript but I'll make liberal use of the "any" keyword for modules that don't have any definitions.

Visual Studio

Visual Studio and TypeScript are a great combination. The Intellisense and code navigation options are brilliant. It's a shame that it's not possible to use the debugging tools, but I'm pretty certain I'm going to be using VS for all my TypeScript projects.

Node.js

This was my first attempt at using Node.js and I was very impressed. There are pre-made modules for basically anything you can think of, it reminds me of Python development in that sense. It's also a very rapid development process which is always a good thing. Once I got over my irrational hatred of everything JavaScript and realized that it's not all bad, I really started to enjoy it.

Jade

Using Jade instead of HTML is one thing I'm not sure I'll do again. On one hand, it is more compact and it's nice to be able to do things like looping and conditionals, but on the other hand it's a layer of abstraction around HTML that can be a bit confusing. Maybe my only problem was that Visual Studio doesn't handle Jade files nearly as well as it handles HTML, so I might stick with it but try using Sublime Text to edit it rather than Visual Studio.

MongoDB

I used MongoDB for my data storage, and it's been a great first NoDB experience. It's really easy to use, easy to set up locally, and powerful enough for my needs while still being simple to use. There is also a site called MongoLab that has a generous free tier that I'm using for my production data, so I'm keeping my operational costs at the best price point of $0.

Socket.io

I must admit that I really didn't use Socket.io the best way, mainly due to my inexperience and the fact that by the time I got up to using it I just wanted to get the project finished as soon as I could. Even though I used it really badly, it still worked pretty well for me, and I'm sure I'll find more uses for it now that I almost understand how to use it.

Isotope

This worked out really well for me, but only because I was doing something that was exactly what it was designed for. If I am ever making something that is a grid of things that needs to be able to be rearranged in a responsive way I'm sure I'll use it again.

Bootstrap

I have a love-hate relationship with Bootstrap, it's great when it works but it seems to force pages to be designed in a particular way that doesn't allow much flexibility. I'm sure this is because I'm not a designer and I'm just sticking with the defaults, but I still think I'll try to look into alternatives next time I'm designing a site. I must admit it is really easy to use if you just stick to the defaults though.

The End

So that's another project done! It's a bit buggy, but it works and at least a few other people are using it, so I'm counting it as a success.

TypeScript Node.js Development Part 4 - More Packages

In this post I'll go through how to add some useful tools to your Node.js TypeScript app - Underscore for handy utility functions, MongoDB for persistence, and Socket.IO for real-time client-server communication.

Since this is Node, nice people have already made packages for these (underscore, mongodb, socket.io) and adding them is simply done by updating the package.json file:

{
  "name": "package-name"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express": "2.5.8"
    , "ejs": ">= 0.8.3"
    , "jade": ">= 0.27.7"
    , "underscore": ">= 1.4.2"
    , "mongodb": ">= 1.1.11"
    , "socket.io": ">= 0.9.11"
  }
}

And since these packages are already included in soywiz's definitions, getting them working in TypeScript is as easy as adding the definitions to the app.d.ts file:

/// <reference path="./node-definitions/node.d.ts" />
/// <reference path="./node-definitions/express.d.ts" />
/// <reference path="./node-definitions/mocha.d.ts" />
/// <reference path="./node-definitions/underscore.d.ts" />
/// <reference path="./node-definitions/mongodb.d.ts" />
/// <reference path="./node-definitions/socket.io.d.ts" />

To test that underscore works, we can add some code to one of our test routes:

import _ = module("underscore")
app.get('/testUrl', function(req, res) {
    console.log('test url ' + req.query['testQS']);

    // Test underscore
    var underscoreTest: string = "";
    var array:string[] = ["a", "b", "c"];
    _.each(array, (item) => { underscoreTest += item });

    res.send(underscoreTest + " env = " + app.settings.env, 200);
});

If you type "_" in to Visual Studio, you'll notice that the TypeScript definitions have given us Intellisense for Underscore, nice!

Socket.IO is also easy to test out, using some of their sample code. We can add this code to our app (again with full Intellisense):

import io = module("socket.io");
var sio = io.listen(app);
sio.sockets.on('connection', function (socket) {
    socket.emit('news', { hello: 'world' });
    socket.on('my other event', function (data) {
        console.log(data);
    });
});

And this code to our index.jade file:

script(src='/socket.io/socket.io.js')
script(type='text/javascript')
  var socket = io.connect();
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });

npm install, then run the app, and we see some console output on the server side logs and the browser, indicating that everything is all going peachy.

MongoDB is a bit more complicated. First, we need to get a local server running so that we can do some testing locally and luckily there is a good tutorial for that over on the MongoDB site. After installation, run mongod.exe and we are ready to write some code to utilize the server.

Here's something I prepared earlier (based on the node package docs), some code that allows addition and querying of a key-value pair to the database (model.ts):

///<reference path='app.d.ts' />

import mongodb = module('mongodb');

interface TestDoc {
    key: string;
    value: string;
}

export class Model {
 private server: mongodb.Server;
 private client: mongodb.Db;

 constructor () {
        var dbname: string = 'testdbsdf';
        var host: string = 'localhost';
        var port: number = 27017;

     this.server = new mongodb.Server(host, port, { auto_reconnect: true });

     this.client = new mongodb.Db(dbname, this.server, { safe: true });
        this.client.open((error) => {
            if(error) { console.error(error); return; }
        });
 }

 putTestDoc(doc: TestDoc, callback: (doc: TestDoc) => void, errorcb: (error: any) => void): void {
     this.client.collection('TestDoc', function (error, docs) {
         if (error) { console.error(error); errorcb(error); return }

         docs.insert(doc, function (error, object) {
             if (error) { console.error(error); errorcb(error); return; }
             callback(object)
         });
     });
 }

 getTestDoc(key: string, callback: (doc: TestDoc) => void, errorcb: (error: any) => void): void {
     this.client.collection('TestDoc', function (error, docs) {
         if (error) { console.error(error); errorcb(error); return; }
            
            docs.findOne({'key': key}, function(error, doc) {
               if(error) { console.error(error); errorcb(error); return; }
               callback(doc);
            });
     });
 }

 getAllTestDocs(callback: (docs: TestDoc[]) => void, errorcb: (error: any) => void): void {
     this.client.collection('TestDoc', function (error, docs) {
         if (error) { console.error(error); errorcb(error); return; }
            
         docs.find({}, { limit: 100 }).toArray(function (err, docs) {
             if (error) { console.error(error); errorcb(error); return; }
             callback(docs);
         });
     });
 }
} 

We can now add some routes that use this class:

import model = module("./model")
app.get('/testMongo/:key/:value', function(req, res) {
    // Add the key/value pair to the DB and then fetch and return it
    database.putTestDoc({ key: req.params.key, value: req.params.value }, function () {
        database.getTestDoc(req.params.key, function (doc) {
            res.send("key = [" + doc.key + "] value = [" + doc.value + "]", 200);
        }, function (err) { res.send(err, 200); });
    }, function (err) { res.send(err, 200); });
});

app.get('/testMongo', function(req, res) {
    // return a list of all pairs
    var response: string = "";
    database.getAllTestDocs(function (docs) {
        _.each(docs, (doc) => { response += ("key = [" + doc.key + "] value = [" + doc.value + "]"); });
        res.send(response, 200);
    }, function (err) { res.send(err, 200); });
});

This is very simplistic - for example, it doesn't check for duplicate keys, but it's good enough to confirm that the database is set up correctly.

So that's all set up locally, but what about ~The Cloud~? Thankfully, The Cloud is generous, and there is a free service to set up cloud-based MongoDB servers called MongoLab. Sign up for an account, create a database, and we are nearly ready to go. Thanks, The Cloud! Before we can use this, we need to modify our model.ts file a bit, so that it can access MongoLab's database. We also now need to authenticate with our username and password. Since we all know that we should never put sensitive information like this in code, we will do this using Azure's app settings. First, we need to update our Model's constructor:

 constructor () {
        var dbname: string = process.env['MONGO_NODE_DRIVER_DBNAME'] || 'testdb';
        var host: string = process.env['MONGO_NODE_DRIVER_HOST'] || 'localhost';
        var port: number = parseInt(process.env['MONGO_NODE_DRIVER_PORT']) || 27017;

     this.server = new mongodb.Server(host, port, { auto_reconnect: true });

     this.client = new mongodb.Db(dbname, this.server, { safe: true });
        this.client.open((error) => {
            if(error) { console.error(error); return; }

            var username: string = process.env['MONGO_NODE_DRIVER_USER'];
            var password: string = process.env['MONGO_NODE_DRIVER_PASSWORD'];
            
            if (username && password) {
                this.client.authenticate(username, password, function (error) {
                    if (error) { console.error(error); return; }
                });
            }
        });
 }

Then we need to put those new settings into our Azure app settings section (in the dashboard, under Configure.

Save those, upload the new code to Azure, and we're up and running all over The Cloud. That's it for now, in the next post, I'll go over testing and debugging your app. As always, here is a Visual Studio template up to this point.

Previously: TypeScript Node.js Development Part 3 - Twitter Bootstrap, next: TypeScript Node.js Development Part 5 - Unit Tests and Debugging

TypeScript Node.js Development Part 2 - Standard Packages and Azure Deployment

In Part 1 I went through the process of making a basic Node.js app using TypeScript in Visual Studio. In this post, I'll make the app a bit less basic, and deploy it to Azure (a.k.a. "THE CLOUD").

To make this server a bit easier to work with, I'll use three Node.js packages - express for routing etc, ejs for templating, and jade for confusing-looking HTML that seems like it's an interesting thing to try out. Since we're using Node.js, adding these packages is easy, just create a package.json file alongside your app.ts file:

{
    "name": "package-name"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express": "2.5.8"
    , "ejs": ">= 0.5.0"
    , "jade": ">= 0.0.1"
  }
}

Then run "npm install" in the app folder.

This pulls in the packages, but TypeScript doesn't know how the express code is defined until we use the definition provided by node-definitions (which we covered in part 1). You could just add "/// <reference path="./node-definitions/express.d.ts" />" to the top of your app.ts file, along with the node.d.ts reference, but a nicer way of doing this is to separate all the definition references to another file, app.d.ts:

/// <reference path="./node-definitions/node.d.ts" />
/// <reference path="./node-definitions/express.d.ts" />

And reference "app.d.ts" at the top of all your TypeScript files. This way, you can add references to your app.d.ts file as required, and the definitions will be available in all your source code files.

I won't go into details about express, ejs, or jade, firstly because I don't really understand it in-depth yet, and secondly because I think the code is pretty self-explanatory. The new app.ts, with some sample get and post URLs, looks like this:

///<reference path='app.d.ts' />

import http = module("http")
import url = module("url")
import routes = module("./routes/index")
import express = module("express")

var app = express.createServer();
var port = process.env.PORT || 1337;

// Configuration
app.configure(function(){
  app.set('views', __dirname + '/views');
  app.set('view engine', 'jade');
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(express.static(__dirname + '/public'));
});

app.configure('development', function(){
  app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
});

app.configure('production', function(){
  app.use(express.errorHandler());
});


// Routes

app.get('/', routes.index);

app.get('/testUrl', function(req, res) {
    console.log('test url ' + req.query['testQS']);
    res.send('ok', 200);
});

app.get('/testUrl/:folder', function(req, res) {
    console.log('testing folder ' + req.params.folder);
    res.send('ok', 200);
});

app.post('/testPost/:userid/newboard', function(req, res) {
    console.log('testing post ' + req.params.userid + ', ' + req.param('postdata'));
});

app.listen(port, function(){
    console.log("Express server listening on port %d in %s mode", port, app.settings.env);
});

export var App = app;

We also have a separated route file in routes/index.ts - using routes specified inline or in external files is a matter of preference but I've included both just so you can see how it works:

///<reference path='app.d.ts' />
import express = module("express")
export function index(req: express.ExpressServerRequest, res: express.ExpressServerResponse){
    res.render('index', { title: 'Page Title', testArray: ["1", "2", "3", "4"] })
};

And our jade "HTML" looks like this:

views/index.jade

h1= title
p Welcome to #{title}

ul
- each test in testArray
  li
    a(href= "/user/"+test)
      b= test

views/layout.jade

!!!
html
  head
    title PageTitle
    link(rel='stylesheet', href='/css/style.css')
  body
    #header
      script(src='http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.min.js')
    #container!= body

Finally we have some basic css (public/css/style.css) that I must have gotten from some example code somewhere:

html { background-color: #f9f9f9; margin: 0; padding: 0; }
body { margin: 0 auto; padding: 0; font-family:"Segoe UI","HelveticaNeue-Light", sans-serif; font-weight:200;}
h1, h2, p, summary, footer, li { line-height: 170%; }
h1, h2 { border-bottom: 1px solid #aaa; font-family:"Segoe UI Light","HelveticaNeue-UltraLight", sans-serif; font-weight:100; }
h2 { font-size: 16pt; }
p { margin: 1em 20px 0 20px; }
ul { margin-top: 1em; }
#footer { font-style: italic; color: #999; text-align: center; padding: 1em 0 2em 0; margin-top: 1em; font-size: 80%; }
em { letter-spacing: 1px; }
li { margin-left: 1em; }

After all of this code, you should now be able to build the project and run "node app", then test out the various URLs and check out how the jade template is translated to HTML.

Now we can move onto something pretty awesome, deploying this code to Azure - which has a free tier that allows you to run up to 10 web sites for free. The Azure team has a really good tutorial that goes through setting up a new Azure Node.js website and getting it ready for git deployment, and they describe the process a lot better that I can, so go ahead and read through that, but instead of using their sample Node.js code, use the code that we have just written.

If you're back, hopefully you've managed to deploy this code to Azure, if so well done! The only thing I want to add is that if you're like me you don't like the idea of publishing a whole lot of files that are not needed for the web server. To avoid this, you can make a .gitignore file that includes things like the node definitions and useless dll files, as well as local copies of the node modules (since Azure will automatically npm install your app once deployed):

bin/
obj/
node_modules/
node-definitions/
[appname].csproj
[appname].csproj.user
[appname].sln
[appname].v11.suo

So there you have it, Node.js code, written in TypeScript, built using Visual Basic, running on Azure. This post was mainly just code or links and not much instructions, but hopefully it is useful for someone. Since this is starting to involve a bit of code, I've made a Visual Studio template so you can quickly make a new project with everything set up for you. However, things like node-definitions and possibly other packages will be a bit out of date and may need to be updated.

The next part will cover getting a basic front-end framework up and running.

Previously: TypeScript Node.js Development Part 1 - Getting Started, next: TypeScript Node.js Development Part 3 - Twitter Bootstrap

TypeScript Node.js Development Part 1 - Getting Started

Note: This information is now a bit outdated, see this blog post for a rundown of the most recent Node.js tools in Visual Studio. Also check out my blog post about getting started with Node.js using TypeScript in WebStorm

I've recently started doing some development in a new environment - coding Node.js apps in TypeScript using Visual Studio, and deploying to "The Cloud". Despite a few small roadblocks, and my personal hate for the term "The Cloud", I've found it to be quite a rapid development path, and for small projects at least I think it's my favourite environment to work with so far. When I first started out, I couldn't find many good introductory tutorials about how to get this environment going, so in this small series of posts I'll try to go through what I did to get started.

The end result here will be a basic Node.js app along with all the "good stuff" of modern software development including type checking, unit tests, debugging, and easy deployment. I'll also include some handy additions like connecting to a MongoDB server and using Twitter Bootstrap as a front-end framework. And best of all, everything I use is free! Perfect for the people like me who are doing this for entertainment rather than income.

As a first step, I'll go for the most basic Node.js Hello World app, built using Visual Studio, running as a local server. I'm going to assume you know what TypeScript is, if not you can read about it here. The first thing you'll need to do is quite obvious, download and install Visual Studio Express 2012 for Web. After that you're going to need to download the TypeScript addon, which will allow you to compile TypeScript files and make new ones that are automatically "attached" to the compiled JavaScript file. You need to close Visual Studio to install this, but I think I also had to restart my comptuer, so if the next step doesn't work for you, try turning it off and on again. Now you should be able to create a new project from the "HTML Application with TypeScript" template under C#. This gets you a basic project with some client-side TypeScript code, which can be compiled into JavaScript using Build Project. For some reason it also creates a dll file, which is something that you'll just have to ignore.

But we are not doing this for client-side TypeScript, we want a Node.js app! So go ahead and delete the code in the project, and add a new "TypeScript File" item called "app.ts". We can try to make this file the TypeScript version of the usual basic Node.js app:

import http = module('http')
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');

But if you try to compile that, you'll get a whole lot of errors and an invalid JS file. This is happening because the TypeScript file doesn't know about the Node.js codebase yet, and can't find the http module. Thankfully, a lovely gentleman has made a github project which provides TypeScript definition files for many common Node.js packages (including definitions for Node itself). So go ahead and clone or download that, and put it in the same folder as your app.ts file. You can then add:

/// <reference path="./node-definitions/node.d.ts" />

to the top of your app.ts file. Take a moment now to bask in the glory that is auto-complete for Node.js. Type in "http." or "console." and witness the beauty:

Translating this from TypeScript into English, it means: createServer is a function, the first (and only) parameter, requestListener, is optional, and is a callback which takes 2 parameters - request is of the type ServerRequest in the http module, and response is of the type ServerResponse in the http module. The callback does not return anything, but the createServer function returns an object of the type Server in the http module. And you found out all this without even having to look at any API docs! After you get over your gobsmacked amazement, build the project and run the app using:

node app.js

Then browse to http://127.0.0.1:1337/ and you should see the two words every programmer loves seeing:

I'll leave it at that for now, but soon I'll make another post which will go through the process of using some packages to make this more of a normal web server, and publishing the app to Azure.

Next: TypeScript Node.js Development Part 2 - Standard Packages and Azure Deployment.