Shaving Last.FM Yaks

Long-time readers might remember that I once had a bit of an obsession with aggregating web feeds on sites that I called “planets”. I wrote Perlanet to make this job easier and I registered the domain theplanetarium.org to host these planets.

The planets I built were of varying levels of usefulness – but of all of them, planet davorg was the vanity project. It was simply a way to aggregate all the web feeds that I produced. There were feeds from various blogs along with things like Flickr, Twitter and CPAN.

One of the things I liked about planets was that they were self-maintaining. Once you’ve configured a planet, it will just keep on running (well, as long as the cron job is running). If the web feeds they are aggregating have new content, the planet will have new content. And many of the feeds that powered planet davorg were still running.

But last weekend I found a couple of  problems with it. Firstly, it looked like it was designed by an idiot. Which, to be fair, it was. Web design was never my strong point. But we have Bootstrap now, so there’s no excuse for web sites to look that bad. So that’s how I spent the first hour or  so – slapping a bit of Bootstrap paint onto the site. I think it now looks acceptable.

The second problem was that not all of the feeds were still running some of them (Delicious, for example) were just dead. I can’t remember the last time I posted anything to Delicious – can you? So I spent some time tweaking and fixing the feeds (replacing CPAN with MetaCPAN, for example). Most of this was easy.

However, one feed was a problem. My Last.fm feed was dead. For over ten years I’ve been “scrobbling” ever song I’ve listened to and one of the feeds I was aggregating was that list. According to this page on their web site, my feed is supposed to be at http://ws.audioscrobbler.com/1.0/user/davorg/recenttracks.rss – and that was the URL in my Perlanet configuration. But it doesn’t work. It returns a 404 error.

I tried to contact someone at Last.fm to find out what was going on, but I haven’t got any kind of response. It looks like they’ve been running on a skeleton staff since CBS took them over and they don’t seem to have the time to support their users (not, I suspect, a recipe for long-term success!)

But there was one possibility. You can get the same data through their API. And some quick experimentation, revealed that their API hasn’t been turned off.

And CPAN has Net::LastFM which will make the API calls for me. Ok, so it hasn’t been updated since 2009, but it still works (I’ve just noticed that there’s also Net::LastFMAPI which is a little more recent).

So it just took a small amount of work to write a little program which grabs uses the Last.fm API to get some JSON that contains the information that I want and convert it to an Atom feed. In case this is useful to anyone else, I’ve put the code on Github. Please let me know if you do anything interesting with it.

And if anyone from Last.fm reads this. Please either turn the web feeds back on or remove the documentation that still claims they exist.

Version Numbers

Last week I mentioned how I had uploaded a new version of Symbol::Approx::Sub. Because there were pretty major changes to the inner workings of the module (although the interface still looked the same) I decided that I would move it from version 2.07 to version 3. At the same time, I decided that I would switch to a semantic versioning scheme.

Later in the week, I released minor updates to a few more of my modules. And I decided to apply semantic versioning to those as well. But as I was only making minor packaging fixes to these modules, I didn’t increment the major version number. For example, Array::Compare went from 12.2 to 12.2.1.

It turns out that was a mistake.

Well, I don’t really think it was a mistake. I think it was the right thing to do. But it appears that my opinion is at odds with what some parts of the Perl toolchain think.

Last night I got this bug report. It seems that by switching to three-part semantic versions, the version number can (in some quite common circumstances) appear to decrease.

To my mind, a version number is a dot-separated sequence of numbers. So 12.2 is smaller than 12.2.1. Any sane version number comparison will separate the two strings on dots and compare the individual components. Any missing components (12.2 is, for example, one component shorter than 12.2.1) should be assumed to be zero.

But that’s not what the Perl toolchain does. Observe:

When the version number with two components (2.12) is split into components, the second component is bizarrely treated as a three-digit number so it becomes 12o instead of 12 and when it is compared with the second component of the three-component version, 120 is obviously larger than 12 and any tool which relies on this behaviour to work out which version of a module is the most recent will get the wrong answer.

This leads to other “interesting” effects. In my head, versions 1.1, 1.01 and 1.001 are all the same version. The leading zeroes mean nothing. But under this scheme, they are very different version numbers.

I know that versioning isn’t as easy as it should be and I know that some people use bizarre versioning systems. And I’m pretty sure that no matter how bizarre a versioning system is, you’ll almost certainly find an example of it on CPAN. So I suppose that this behaviour was a “least worse” scenario that was chosen to make the most sense given CPAN’s wide range of versioning schemes.

Personally, I see it as a bug in version.pm. But I’m not going to report it as such as I’m sure the Perl toolchain gang know what they’re doing and have very good reasons for adopting this seemingly broken behaviour.

I just need to remember to be more careful when switching my modules to semantic versioning. Using a minor or patch level version change when switching to semantic versioning is likely to lead to confusion and bug reports. Only a major level change (as I did with Symbol::Approx::Sub) is guaranteed to work.

And, I suppose, I’ll need to release Array::Compare 3.0.0 to CPAN pretty soon.

Hacking Symbol::Approx::Sub

In October, for (I think) the second year, Digital Ocean ran Hacktoberfest – a campaign encouraging people to submit pull requests to Github repos in exchange for free t-shirts.

A few of us thought that this might be a good way to do a small bit of easy Perl advocacy, so we tagged some issues on Perl repos with “hacktoberfest” and waited to see what would happen.

I created a few issues on some of my repos. But the one I concentrated on most was symbol-approx-sub. This is a very silly CPAN module that allows you to make errors in the names of your subroutines. I wrote it many years ago and there’s an article I wrote for The Perl Journal explaining why (and how) I did it.

Long-time readers might remember that in 2014 I wrote an article for the Perl Advent Calendar about Perl::Metrics::Simple. I used Symbol::Approx::Sub as the example module in the article and it showed me that the module had some depressingly high complexity scores and I planned to get round to doing something about that. Of course, real life got in the way and Symbol::Approx::Sub isn’t exactly high on my list of things to do, so nothing happened. Until this October.

Over the month, a lot of changes were made to the module. I probably did about half of it and the rest was pull requests from other people. The fixes include:

  • Better tests (and better test coverage – it’s now at 100%)
  • Using Module::Load to load module
  • Using real exceptions to report errors
  • Updating the code to remove unnecessary ampersands on subroutine calls
  • Fixed a couple of long-term bugs (that were found by the improved tests)
  • Breaking monolithic subroutines down

And I’m pretty happy with how it all went. The work was mostly completed in October and this morning I finally got round to doing the last couple of admin-y bits and version 3.0.0 of Symbol::Approx::Sub is now on the way to CPAN. You still shouldn’t use it in production code though!

Thanks to everyone who submitted a pull request. I hope you did enough to earn a free t-shirt.

If you want to get involved in fixing or improving other people’s code, there’s the 24 Pull Request Challenge taking place over Advent. Or for more Perl-specific code, there’s the CPAN Pull Request Challenge.

p.s. In the Advert Calendar article, I linked to the HTML version of the results. For comparison, I’ve also put the new results online. It’s a pretty good improvement.

Updating Ogg::Vorbis::Header

Last night, I uploaded a new version of Ogg::Vorbis::Header – a CPAN module that hasn’t been updated since 2003 and which I strongly suspect no-one at all uses any more. You might be interested to hear what I did or why I did it.

About a year ago, I wrote about the dashboard I had written for my CPAN modules. It’s a simple page that pulls together information about all of my modules and, among other things, shows me how they are doing on Travis CI and Coveralls.

One of the aims of the dashboard was to encourage me to do more work ensuring that my CPAN modules were working well and had good test coverage. The idea was that if I’m constantly looking at a page which shows how rubbish the test coverage for a module is, then I’ll be more motivated to fix it. Of course, that only works if I’m constantly looking at the dashboard and, to be honest, over the year since I built it I really haven’t taken much notice of it.

But recently, I was reminded of its existence as I updated it to remove some modules that I’ve handed over to other people and to add a couple of new modules I had released. And, in doing that, I took a closer look at it and my attention was drawn to AudioFile::Info::Ogg::Vorbis::Header. This is the only one of my modules which doesn’t even build on Travis. Clearly, more investigation was needed. But, before we get into that, it’s probably worth making a brief digression to explain what the module does.

Some of you will be too young to remember this, but there was a time back in the early middle ages of internet history (so, perhaps, fifteen years ago) when not everyone listened to music as MP3s. Back then, one of the biggest sources of digital music for many people was ripping their existing CDs (ask your parents – they might still have a CD or two they can show you). And when ripping music from CDs we had a choice of formats. Most people (even then) were using MP3, but some of us took the ideologically superior option of ripping to Ogg Vorbis. The main reason was that MP3 format was patented, but Ogg Vorbis was completely free.

All of this meant that in the first five or six years of the 21st century, I ended up with hundreds (maybe thousands) of Ogg Vorbis files on my hard disk. This immediately gave me problems as it dramatically limited the devices that I could play my music on. For example, it probably explains why I’ve never owned an iPod.

But I also… er… acquired a number of MP3s over the same time. And, being a geek, at times I wanted to write programs that gathered information about all of my music, no matter what format it was stored in. There were modules on CPAN for dealing with MP3s and there were modules on CPAN for dealing with Ogg Vorbis files. But (as is so often the way with these things) all of these modules worked in completely different ways.

And that’s I wrote the AudioFile::Info set of modules. They acted as a wrapper around the various modules for dealing with the different audio formats and gave them all the same interface. It meant that I could write programs that got information from any of my audio files and I didn’t need to care what format they were in. Think of them a bit like a DBI for audio file formats.

Of course, no-one else ever had any use for them. And soon afterwards MP3 became the de-facto standard for digital audio and Ogg Vorbis was relegated to the same (virtual) drawer as Betamax. I’d say that no-one uses it any more – but I suspect there are actually about eight users left and they would all write comments telling me that I was wrong.

None of the AudioFile::Info modules have been updated for a very long time, because no-one uses them any more and no-one cares about them. I’d remove them from CPAN, but that goes against my pack-rat nature.

All of which leaves me annoyed by the failure of AudioFile::Info::Ogg::Vorbis::Header to build on Travis. So a couple of weeks ago, I investigated further. And, to my delight, I found that it wasn’t my fault. Actually, it was the underlying module (Ogg::Vorbis::Header) that had the problem. That module no longer built successfully on modern Perls. And that failure prevented my module from building on top of it.

The problem is described in this RT ticket. Basically, there was some very funky syntax in a test. Syntax that became a fatal error through some parser fixes in Perl 5.22. The test looked like this:

When it should have looked like this:

In the RT ticket, H. Merijn Brand gives a good explanation of how the test ever passed – but try working it out for yourself before looking.

So, anyway, I knew what the problem was and I knew how to fix it. My next step was to pass this information on to the author of the module. I emailed him a couple of weeks ago, offering to make the fixes myself if he was too busy (or too uninterested) to do it himself. I got no reply, so at the end of last week I emailed the CPAN Powers That Be explaining the situation and asking for co-maintenance rights on the module so that I could fix the problem. They granted my request – which is why the new version was released yesterday. I can already see that the tests for this version look a lot healthier than the ones for the previous version.

Healthier, but still not as healthy as I’d like them. Within an hour or so of my release hitting CPAN, this issue was raised. The Makefile.PL uses Inline::MakeMaker and I can’t work out how to make that work, given that the “use” statement is executed long before any of the configuration code that tells the build tools what modules are required. If you have any suggestions, please let me know (or send me a pull request). I’m a bit out of my depth when it comes to Inline-based modules.

There are a few other things that I might fix. It’s an old-style distribution where there are no /lib or /t directories. It’s all in the top-level directory. I’m very tempted to move all that stuff around.

But really, I should get back to ensuring that my module builds successfully now.

Update: On Sunday, I released another version of Ogg::Vorbis::Header which fixed the packaging problems. But it still hasn’t solved my Travis-CI woes and there are still a disappointing number of failures on CPAN testers (actually they are “unknown” results rather than real failures – because there are still cases where the module won’t even build).

The problem is the underlying C libraries. Ogg::Vorbis::Header relies on the libogg and libvorbis libraries. And a large number of people aren’t going to have those libraries installed (the Travis-CI environment certainly doesn’t). Trying to build the module on a system that doesn’t have those libraries is doomed to failure.

The solution is, I suspect, to build Alien modules for these two libraries. But that’s something that I know very little about. I doubt I’ll have the time to learn a whole new area of CPAN packaging until after YAPC Europe at the earliest. Of course, if some kind person who knows more about this area than me was to offer to help (or even to produce the Alien modules for me) then that would make me very happy 🙂

Taming QMail

I run my own email server. It uses QMail. I realise there are at least two problems there – all of the cool kids have been using Gmail for their email since approximately forever, and who the hell uses QMail anyway?

Like most of these situations, it’s that way for historical reasons. And, of course, I’d love to change the current situation but it works well enough most of the time that the occasional pain it gives me isn’t worth the pain of changing the set-up.

So there I am. One of about three people in the world who are still running an email server using QMail. And that occasionally gives me problems.

One problem is the management of the mailing queues. The tools that QMail gives you for checking the contents of the queues and deleting any unnecessary messages are rather primitive. It seems that everyone uses third-party tools. A couple of years ago, I had a WordPress installation that was compromised and was used to send thousands of email messages to random people. The outgoing queue was full of spam and the incoming queue was full of bounce messages. I stopped QMail and started looking for a way to cleanly delete all the bad messages while retaining the tiny number of legitimate ones.

I discovered a program called qmHandle which did exactly what I wanted. It enabled me to remove messages from both queues that matched various criteria and before very long at all, I was back in business (having also cleaned up the WordPress installation and tightening its security).

The qmHandle program was written in Perl. And I always had it in the back of my mind that at some point I’d revisit the program and give something back by fixing it or improving it in some way. A few months ago I found time to do that.

I started by looking at the code. And realised that it had pretty much been written to be as hard to maintain as possible. Ok, that’s probably not true, but it certainly wasn’t written to be particularly easy to understand. What the original author, Michele Beltrame, created was really useful, but it seemed to me that he had made it far harder for himself than he needed to.

So I had found my project. I wanted to hack on qmHandle. But before  I could do that, I needed to rewrite it so that it was easier to work on. That became something that I’d hack on in quite moments over a few weeks. The new version is on Github. I started by importing the original version, so it’s interesting to read the commit history to trace the changes that I made. I think there were three main areas where I improved things.

  1. Splitting most of the logic out into a module. I say “most”, but it’s actually all. The command-line program is now a pleasingly simple:
  2. Improving (by which I mainly mean simplifying) the logic and the syntax. I moved a few variable declarations around (so their scope was smaller) and renamed some so their meaning was more obvious. Oh, and I added a couple of useful CPAN modules – Term::ANSIColor and Getopt::Std.
  3. Using Moose. Switching to an OO approach was a big win in general and Moose made this far easier than it would otherwise have been. At some point in the future, I might consider moving from Moose to Moo, for performance reasons.

For a few weeks now, I’ve been using the revised version on my email server and it seems to be working pretty much the same as the original version. So it’s time to set it loose on the wider world. This afternoon, I released it to CPAN. I’ve said above that the number of people using QMail to handle their email is tiny. But if you’re in that group and you want a more powerful way to manage your mail queues, then the new version of qmHandle might well be useful to you.

There are a few things that I know I need to do.

  1. More tests. The main point of moving most of the code into a module was to make it easier to test. Now it’s time to prove that. The current test suite is tiny. I need to improve that.
  2. Configuration. Currently, the configuration is all hard-coded. And different systems might well need different configuration (for example, the queues might be stored in a different directory). There needs to be a simple way to configure that.
  3. Bug fixes and improvements. This was, after all, why I started doing this. I don’t know what those might be, but I’m sure there are ways to improve the program.

I hope at least someone finds this useful.