ScrobbleAlong

Once again I've moved on to a new project and stopped thinking about my old one, but now that I've wrapped up the new one I guess I really should come back and write something about the old one. This is mainly going to be an "after action report" so that I have an opportunity to go through the technology choices I made and think about what worked and what didn't. I'll go into a bit of technical details but I'm not really 100% happy with the code so I'm not going to put that up anywhere. But first ...

What Is It?

ScrobbleAlong is a website I made which does two (closely related) things. It's a background process that continuously polls the "now playing" feeds of a few radio stations I like and scrobbles all the tracks to last.fm.  It also has a frontend that lets other last.fm users select a radio station that they are listening to so that the songs can also be scrobbled to their account. It's a nifty little tool for people who like to scrobble absolutely everything they listen to, and it's also a nice way to see what various radio stations are playing. If I ever want to see what the latest popular tracks are back home in Australia, I can load up Triple J's last.fm page and check the most played tracks for the last few weeks.

The tool actually has a pretty long history, it started a few years ago as a Python script that just did the radio station scrobbling, but I was looking for something to do as a Node.js project, so I decided to update it and add the "scrobble along" functionality. I'm pretty happy with the way it turned out, if I did it again I would probably do a few things differently, but as a first "proper" Node.js project I think it worked out pretty well.

How Does It Work?

Scrobbling for the stations is handled by a task that runs every 15 seconds, which calls an update function for every station. This update function does a HTTP request for a URL that contains the details for the currently playing song. A parser takes the body of the request and extracts out the artist and song names, and this is compared against the last time the URL was queried. If the song changed since the last request, and it was playing for a long enough time (>30 seconds), the last song is scrobbled. If the new song is valid (e.g. something is actually playing), a "now playing" request is sent to last.fm. One fiddly thing here is that there is no way to tell how long the song will play for, so we tell last.fm that the song is 40 seconds long, and update it again in 30 seconds if it's still playing. 

Scrobbling for users (scrobbling along) is handled mainly using a nice last.fm Node.js module which supports callbacks that are fired when a song has just been scrobbled or a song has started playing. Using this it is fairly easy to have a list of users attached to each station which can be updated whenever the station is updated. Again, some cleverness needs to be applied to avoid problems related to an unknown song length, otherwise only a single now playing notice is sent, so it looks like the user only listens to each song for 30 seconds, then stops listening until the next song starts. For the scrobble along case, this is handled using an interval that fires every 30 seconds and updates the now playing details, which is killed when the song is eventually scrobbled.

The front-end only needs to update the backend storage (MongoDB in my case), which contains a list of users and the station that they are listening to. The stations are presented in a pretty nice way thanks to the Isotope library which works pretty well with Bootstrap to make a nice responsive design. I also chucked a little bit of socket.io in there so that each station "block" automatically updates its display of the latest played tracks.

Lessons Learnt

I used a whole lot of technologies that I'd never used before when writing Scrobble Along, so I'll quickly go through them with some short impressions on what I thought of them.

TypeScript

I've now written a pretty significant project in both TypeScript and plain JavaScript, and I've got to say I really like the safety net and cleaner code that TypeScript provides. I can be a bit annoying sometimes, especially when I was using a definition file that wasn't quite right and I had to keep updating the definition before the app would compile without errors. I also wasted a lot of time writing definitions for modules that didn't have any, which I probably wouldn't bother with again since the time spent was not really worth it. I think in the future I'll continue to use TypeScript but I'll make liberal use of the "any" keyword for modules that don't have any definitions.

Visual Studio

Visual Studio and TypeScript are a great combination. The Intellisense and code navigation options are brilliant. It's a shame that it's not possible to use the debugging tools, but I'm pretty certain I'm going to be using VS for all my TypeScript projects.

Node.js

This was my first attempt at using Node.js and I was very impressed. There are pre-made modules for basically anything you can think of, it reminds me of Python development in that sense. It's also a very rapid development process which is always a good thing. Once I got over my irrational hatred of everything JavaScript and realized that it's not all bad, I really started to enjoy it.

Jade

Using Jade instead of HTML is one thing I'm not sure I'll do again. On one hand, it is more compact and it's nice to be able to do things like looping and conditionals, but on the other hand it's a layer of abstraction around HTML that can be a bit confusing. Maybe my only problem was that Visual Studio doesn't handle Jade files nearly as well as it handles HTML, so I might stick with it but try using Sublime Text to edit it rather than Visual Studio.

MongoDB

I used MongoDB for my data storage, and it's been a great first NoDB experience. It's really easy to use, easy to set up locally, and powerful enough for my needs while still being simple to use. There is also a site called MongoLab that has a generous free tier that I'm using for my production data, so I'm keeping my operational costs at the best price point of $0.

Socket.io

I must admit that I really didn't use Socket.io the best way, mainly due to my inexperience and the fact that by the time I got up to using it I just wanted to get the project finished as soon as I could. Even though I used it really badly, it still worked pretty well for me, and I'm sure I'll find more uses for it now that I almost understand how to use it.

Isotope

This worked out really well for me, but only because I was doing something that was exactly what it was designed for. If I am ever making something that is a grid of things that needs to be able to be rearranged in a responsive way I'm sure I'll use it again.

Bootstrap

I have a love-hate relationship with Bootstrap, it's great when it works but it seems to force pages to be designed in a particular way that doesn't allow much flexibility. I'm sure this is because I'm not a designer and I'm just sticking with the defaults, but I still think I'll try to look into alternatives next time I'm designing a site. I must admit it is really easy to use if you just stick to the defaults though.

The End

So that's another project done! It's a bit buggy, but it works and at least a few other people are using it, so I'm counting it as a success.

Caption Captain

Before I write a post about my most recent project, I thought it would be a good idea to write about the thing I did before that. It had a much smaller scope, and I made it so long ago now that I'm in danger of forgetting absolutely everything what I did.

Before I start the explanation, I should just provide a link to the site - http://captioncapta.in. Glad that's out of the way! So Caption Captain is a game that picks one image and three titles at random from reddit, and the goal is to pick the title that the image was submitted with. With a subreddit like Reaction GIFs this is a pretty fun game and an entertaining way of browsing the latest submissions. The game also works with other subreddits, but obviously the ones that are not image-heavy will not work too well, and the captions are sometimes pretty obvious for subreddits like funny or gifs.

The main reason I wanted to make this was both to have a bit of practice with Dojo and Bootstrap, and to see what I could do with a site when I limited myself to having a minimal infrastructure - I wanted to implement everything in pure HTML and JavaScript and have no server-side logic. That means no database, no server to access remote URLs, and no URL routing. The site could be served as a single HTML file, but I split it up a bit for clarity. It's currently running on Amazon S3 so it has minimal cost and maintenance.

This post will probably be light on the code, but I've put up all the source on GitHub so you can follow along by going there.

The first problem I encountered was due to my technology choices. Bootstrap (the JavaScript part at least) is generally written to be used alongside jQuery instead of Dojo. I would also prefer to use jQuery, but I knew I had to use Dojo for another thing I was working on, so I wanted to get at least a little bit of practice with it. Thankfully, some kind soul has done the hard work and re-written Bootstrap's JavaScript modules in a format that works well with Dojo and they creatively named it Dojo Bootstrap. All I wanted to use was Bootstrap's Modal functionality, and with Dojo Bootstrap this was as easy as require-ing the Modal module and putting the right tags into the HTML.

The next problem to solve was thinking about how to get subreddit data. Usually I'd just write some code into the server to query reddit, then return that as part of a site or ajax request, but since I didn't want to actually use a server this was not possible. Luckily, reddit's JSON API supports the usage of JSONP, which means we can call it on the client side and use the response inside a JavaScript function. Using this I made a function that would attempt to get the next image and set of captions, and if there were not left it used JSONP to get some more images and captions from reddit before returning one of them. All of the image loading logic is in imageLoader.js but to summarize - getNextImage is the main function, this will return an image if we have it, otherwise it calls _loadImages. _loadImages calls a set of query functions asynchronously, one for a few different sort orders of the subreddit, and that function (_getDataFromReddit) is a fairly generic JSONP call. Once all of these queries finish, the results are filtered to make sure that the link is an image and that its score isn't too low, then it's separated into a set of images and captions that are ready for use.

The rest of the code and HTML (mainly in index.html and controller.js) is pretty standard scaffolding code so I won't go through most of it, but the last thing I wanted to mention was the solution to the problem of URL routing when not using a server. I wanted to support the selection of different subreddits, but the usual solution of having a unique URL for each subreddit and using URL routes to detect the subreddit would not work in a plain HTML page. However, JavaScript does have access to the full URL, including query string, so all we need to do is redirect to index.html with e.g. ?subreddit=reactiongifs, detect that on page load, and the problem is solved.

So that's it, the main thing I learnt is that I don't really like using Dojo as much as jQuery, at least for the "utility" functionality. I still think it's pretty good for UI work, and the rest of it has improved a lot recently, but it's hard to beat jQuery for things like AJAX and async calls. I also quite enjoyed the challenge of making a single-HTML page, and if I ever think of something else that would work in that form I'll have another go at it. It's just lovely to be able to shove something on S3 and forget about it while it keeps running happily in the background.

TypeScript Node.js Development Part 3 - Twitter Bootstrap

This is going to be a quick post since I only want to cover how to get a jade layout working with Twitter Bootstrap. Bootstrap is a front-end framework that includes a nice standard UI, an easy way to make a neat layout that updates depending on the size of the window, and a bunch of neat Javascript features like popups, dropdowns, and alerts. It's a great way to start off a site, especially for people like me who are horrible at graphic design.

To get this started, first I'm going to get rid of the original CSS code, but leave the file there in case I want to add anything else later. Getting the site "bootstrap enabled" is a simple matter of referencing the bootstrap CSS and JS files, and I'm also going to include jQuery since the Bootstrap Javascript code requires it, and also because it's something that is pretty useful on its own. With these changes, the layout.jade file now looks like:

!!!
html
  head
    title Page Title
    link(rel='stylesheet', href='//netdna.bootstrapcdn.com/twitter-bootstrap/2.2.1/css/bootstrap-combined.min.css')
    link(rel='stylesheet', href='/css/style.css')

  body
    #header
      script(src='http://ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js')
      script(src='http://netdna.bootstrapcdn.com/twitter-bootstrap/2.2.1/js/bootstrap.min.js')

    #container!= body

The basic way to layout a Bootstrap page is to use row divs to separate vertical parts of the page and column spans with optional offsets to separate horizontal parts of rows. The Bootstrap docs explain this in more detail, but for my simple index page, I'm going to enclose all my content in one row which is 4 columns wide and offsetted 2 columns from the left, and in jade that looks like this:

.row
  .offset2.span4
    [content goes here]

My content is going to be the title and test array from part 2, then some code which tests that jQuery and Bootstrap are both hooked up correctly and working. The jQuery test is a simple accessor that modifies some HTML. The Bootstrap test is a bit more involved - it's a button that when clicked will open up a modal dialog. Translating some jQuery code and sample Bootstrap code to Jade, we get this:

    h1= title
    p Welcome to #{title}

    ul
    - each test in testArray
      li
        a(href= "/user/"+test)
          b= test

    #jquery-test

    script(type='text/javascript')
      $("div#jquery-test").html("jQuery works!")

    a#bootstrap-test.btn(href="#",role="button") 
      i.icon-thumbs-up
      | Bootstrap Modal

    #myModal.modal.hide.fade(role="dialog")
      .modal-header
        button.close(type="button", data-dismiss="modal") x
        h3 Modal Test

    script(type='text/javascript')
      $('#bootstrap-test').click(function() { $('#myModal').modal() });

Load that up, and we see the reassuring "jQuery works!" and a button which, as expected, opens up a modal dialog.

The final step I wanted to include was to add some Google Analytics tracking code, since that is something that I want in all of my sites. This gets added to the layout file, and in Jade it looks like:

    script(type='text/javascript')
      var _gaq = _gaq || [];
      _gaq.push(['_setAccount', 'ANALYTICS-ID']);
      _gaq.push(['_trackPageview']);
      (function () {
        var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
        ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
        var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
      })();

And that's all I wanted to cover in this post. As with part 2, I've made a Visual Studio template which includes these changes.

Previously: TypeScript Node.js Development Part 2 - Standard Packages and Azure Deployment, next: TypeScript Node.js Development Part 4 - More Packages.