Shaving Last.FM Yaks

Long-time readers might remember that I once had a bit of an obsession with aggregating web feeds on sites that I called “planets”. I wrote Perlanet to make this job easier and I registered the domain theplanetarium.org to host these planets.

The planets I built were of varying levels of usefulness – but of all of them, planet davorg was the vanity project. It was simply a way to aggregate all the web feeds that I produced. There were feeds from various blogs along with things like Flickr, Twitter and CPAN.

One of the things I liked about planets was that they were self-maintaining. Once you’ve configured a planet, it will just keep on running (well, as long as the cron job is running). If the web feeds they are aggregating have new content, the planet will have new content. And many of the feeds that powered planet davorg were still running.

But last weekend I found a couple of  problems with it. Firstly, it looked like it was designed by an idiot. Which, to be fair, it was. Web design was never my strong point. But we have Bootstrap now, so there’s no excuse for web sites to look that bad. So that’s how I spent the first hour or  so – slapping a bit of Bootstrap paint onto the site. I think it now looks acceptable.

The second problem was that not all of the feeds were still running some of them (Delicious, for example) were just dead. I can’t remember the last time I posted anything to Delicious – can you? So I spent some time tweaking and fixing the feeds (replacing CPAN with MetaCPAN, for example). Most of this was easy.

However, one feed was a problem. My Last.fm feed was dead. For over ten years I’ve been “scrobbling” ever song I’ve listened to and one of the feeds I was aggregating was that list. According to this page on their web site, my feed is supposed to be at http://ws.audioscrobbler.com/1.0/user/davorg/recenttracks.rss – and that was the URL in my Perlanet configuration. But it doesn’t work. It returns a 404 error.

I tried to contact someone at Last.fm to find out what was going on, but I haven’t got any kind of response. It looks like they’ve been running on a skeleton staff since CBS took them over and they don’t seem to have the time to support their users (not, I suspect, a recipe for long-term success!)

But there was one possibility. You can get the same data through their API. And some quick experimentation, revealed that their API hasn’t been turned off.

And CPAN has Net::LastFM which will make the API calls for me. Ok, so it hasn’t been updated since 2009, but it still works (I’ve just noticed that there’s also Net::LastFMAPI which is a little more recent).

So it just took a small amount of work to write a little program which grabs uses the Last.fm API to get some JSON that contains the information that I want and convert it to an Atom feed. In case this is useful to anyone else, I’ve put the code on Github. Please let me know if you do anything interesting with it.

And if anyone from Last.fm reads this. Please either turn the web feeds back on or remove the documentation that still claims they exist.

Learning About Traits

I’ve been teaching basic Moose in my training courses for several years now. And, as I’ve mentioned before, I’ve been slowly converting some of my CPAN modules to use Moose. But there are still bits of Moose that I haven’t really needed to get to grips with.

One such area is Moose’s support for Traits. Oh, I knew vaguely what they were and I understood why you might use them. But I’d never implemented a system using traits, so my knowledge about how you’d actually use them was a bit shaky to say the least.

But over the last few days I’ve learned quite a lot about how to use traits. And I’ve had to learn it pretty quickly.

It all started a few weeks ago when I got a github pull request from Oliver Charles. Oliver had taken a fork of my Perlanet repository and had massively refactored the codebase so that all the clever bits were implemented as traits.

What this means is that the core Perlanet code is pretty dumb. In order to do anything really useful with it to need to add in some traits. There are traits to read the configuration from a YAML file, traits to carry out various kinds of cleaning of the input and a trait to produce the output using the Template Toolkit. There are also traits to handle caching and OPML generation. All in all it makes the code far nicer to work on.

Oh, and there’s a Perlanet::Simple module which uses all of the traits required to implement the Perlanet behaviour that users currently expect.

There were a few problems with Oliver’s initial version. Some of the dependencies weren’t quite right. But we soon fixed that and last week I finally released Perlanet 0.47 which implemented these changes.

Then I installed it on the server which hosts most of my planets. And everything broke.

So I’ve spent a lot of the weekend fixing these issues. Part of it was that Oliver’s changes assumed some configuration file changes that I hadn’t implemented. I changed his changes so that they worked with the existing configuration settings (we don’t want users having to change configuration files unnecessarily). Other changes were harder to track down. I particularly enjoyed one where no feeds were fetched unless the user turned on OPML support. I learned a lot about how traits worked by tracking that one down.

But it all seems to be fixed now. Perlanet version 0.51 is available on CPAN and it will hopefully be a lot easier to customise to your needs. I hope we’ll see a number of other Perlanet traits appearing over the next few months.

And, most importantly, I’ve learnt a lot about traits. I think I understand them now.

If you want to learn traits, I can highly recommend asking someone to implement traits in one of your projects. And it’s even better if they do it in a slightly broken way so that you need to debug it.

Ironman and XML::Feed

Sam Graham complains that since the Ironman feed switched to using Perlanet, the entries have been “mangled”. By that he means that in some cases any HTML in feed entries is lost.

I think they’re running up against this bug in XML::Feed (which is one of the modules that Perlanet uses to process the feeds it subscribes to). There’s a patch in the bug report that I’ve applied to my local installation of XML::Feed and it seems to have fixed the problem.

Until the author releases a new version of XML::Feed that includes this patch, I recommend that anyone using XML::Feed (and that includes everyone using Perlanet) applies the patch for themselves.

Perlanet Improvements

I can be a bit of a lazy open source author at times. I love it when someone improves my code and then just emails me patches. I love it even more now that I’m using Github so that people can just fork my code and send me pull requests.

That happened over the weekend. I got a pull request from cycles saying that he’d been refactoring Perlanet and asking if I’d be interested in merging his changes. These refactorings had come out of a mini-hackathon that had been run by the North West England Perl Mongers. I knew that they had been interested in using Perlanet to power the Iron Man aggregator, but that the monolithic nature of  the code made it hard for them to subclass the bits that they needed to change.

So what cycles has done is to take my code and break it down into a number of smaller functions (and a couple more classes) so that people can subclass it and override methods where they want different behaviour to the defaults that I use. I can see this leading to a little ecosystem of Perlanet subclasses where people release their favourite tweaks to CPAN. I see this as a good thing.

I liked cycles’ changes so I merged them into my repository. They’re available there now if you want to look. I haven’t yet released a new version to CPAN as there are a few things I want to check out first.

Firstly, this new update seems to have increased the version of Moose required. With the version of Moose that I was running on my system (0.88), a lot of the tests were failing. Updating to 0.93 solved that problem. I need to work out exactly what the problem was and update the “requires” line in Build.PL appropriately.

Secondly, the newer version of Moose gave a deprecation warning when used with CHI. Updating to the latest version of CHI fixed that (but that, in turn, meant upgrading a few modules in the Log::Any family). This is all starting to get a bit too close to the bleeding edge of CPAN for a module that I want as many people as possible to use.

Finally, cycles had started to use TryCatch in the module. Not, of course, that I object to high quality exception handling in my code, but this is another module that isn’t yet in general use. It’s something that you won’t find in a “standard” (whatever that means) Perl installation.

I’m in the process of building RPMs of all of the missing modules (or later versions for the modules where the Fedora/Centos build is just lagging CPAN a bit). They’ll be available from rpm.mag-sol.com in the next few days.

Currently I’m leaning towards just releasing the new version and hoping that the people who want to use it will have enough enthusiasm that they won’t complain about the updated and new modules that are required. But I thought it would be interesting to ask for your opinions too.

As a user of CPAN modules, how do you decide when a module is too cutting edge for you to use? Do you just install newer versions of modules automatically when an installation asks you too? Or are you a little more careful than that? Would the constraints in this latest version of Perlanet prevent you from using it?

And as a writer of CPAN modules, how cutting edge do you allow yourself to be? Are you happy to release stuff that only works with the very latest versions of Moose or other fast-moving modules? Or do you like to ensure that your stuff is usable by people who might be a little behind the curve?

I should make it clear that I’m very grateful for the work that cycles did and I’m not disparaging his efforts at all. I’m just dithering a bit about how cutting edge I want to be.