Two Weekend Projects

It’s far too long since I’ve posted anything here. I’ve no excuse really. Following the end of my contract in Canary Wharf, I was off work for seven weeks. OK, I was on holiday for two of those weeks, but that still leaves five weeks when I could have been doing something constructive, but actually just spent a lot of time watching Netflix.

But there were a couple of things I did. Neither of them took more than a few hours, but I thought it was worth writing them up – if only to give an example of a couple of really useful (to me, at least) things that I was able to build really quickly with Perl.

Cooking Vinyl

If you were a music fan in the 1990s, then there’s a good chance that you own at least one album released on Cooking Vinyl Records. At times, it seemed like pretty much every album I bought was released by them. Back in 2005, I wrote a blog post where I tried to explain how much they meant to me.

In particular, they produced a series of compilation albums that introduced me to so many of my favourite acts. Ten years ago, I tried to find a definitive list of all of the songs and artists which appeared on those compilation albums. As I failed to find one, I created it myself. At the time, it was a static list of albums which listed the tracks and artists on each of the albums. For ten years I’ve had it in the back of my head to do something more interesting with the data. A few weeks ago, I finally got round to it.

As I said, the original page just had a list of albums with artists and song titles. That’s useful, but it would be more interesting to be able to cross-reference the data in various ways – list all of the albums that an artist appeared on, for example. And for that, we need a database.

If you’ve come on any of my database training course over the last ten years, you’ll know that I use a CD database example. The model that I use is pretty simple and, in particular, it assumes that all tracks on a given CD are by the same artist. As I say in the class “various artists compilations don’t exist in this simplified universe”. Obviously, that’s not going to work in this example. So I needed to come up with another database model.

Compilation album data model
Compilation album data model

Here’s the data model I designed. You’ll see that it all hinges on the track table. A track is an instance of a particular song, recorded by a particular album appearing on a particular album. The only extra data on the track table is the “number” column which allows us to declare the order in which tracks appear on an album.

Advanced students will have spotted an omission from the data model. An artist might well have different versions of a song. There could be the original version, an edited single version and many live or remixed versions. So actually, we could add a “recording” table and it’s the recording that appears on an album. That’s, perhaps, an enhancement for the future.

Having designed the database the rest of the code just falls out really. I already had a data file so it was just a case of parsing that and inserting the data into an SQLite database. DBIx:Class (and, particularly the find_or_create method) makes this trivial. I then wrote another program that generated the web site using the Template Toolkit. Nothing complex there at all.

The site is at http://cookingvinyl.dave.org.uk/. And all of the code is on Github. It could do with being made a bit prettier – perhaps I can add some pictures.

Why not have a look. And check out some Cooking Vinyl recordings.

Tower Bridge

I’ve lived in London for thirty-five years. And in all that time I have never seen Tower Bridge opening. Oh, I’ve seen it when it’s  open, but I’ve never been in the right place at the right time to see it actually opening. As a Londoner, that’s a matter of supreme embarrassment to me.

But the office I’m working in currently is three minutes walk from Tower Bridge. All I need is a way to get a notification a few minutes before the bridge lifts. Surely, there must be a way to get that?

Sadly, no. The Tower Bridge web site has a page listing the upcoming lifts, but no service that would send any kind of notification. So, once again, it was up to me to provide one. I asked the London Perl Mongers on IRC what would be a good way to get notifications of upcoming events on an Android phone and Ilmari pointed out that the obvious method was to create a calendar that could be read by the calendar app on my phone.

So that’s what I’ve done. I use Web::Query to scrape the data from the Tower Bridge web site (doing some over-complicated madness to account for the fact that they are missing the year from their dates) and then create a .ics file using Date::ICal and Data::ICal. I also create a JSON version of the data in case it’s useful to anyone (if it is, please let me know).

The site is at http://towerbridge.dave.org.uk/ and (of course) the code is on Github.


So, there you are. Two (hopefully) little projects that I threw togther in a very small amount of time using the power of Perl. Please let me know if you find either of them useful.

Version Numbers

Last week I mentioned how I had uploaded a new version of Symbol::Approx::Sub. Because there were pretty major changes to the inner workings of the module (although the interface still looked the same) I decided that I would move it from version 2.07 to version 3. At the same time, I decided that I would switch to a semantic versioning scheme.

Later in the week, I released minor updates to a few more of my modules. And I decided to apply semantic versioning to those as well. But as I was only making minor packaging fixes to these modules, I didn’t increment the major version number. For example, Array::Compare went from 12.2 to 12.2.1.

It turns out that was a mistake.

Well, I don’t really think it was a mistake. I think it was the right thing to do. But it appears that my opinion is at odds with what some parts of the Perl toolchain think.

Last night I got this bug report. It seems that by switching to three-part semantic versions, the version number can (in some quite common circumstances) appear to decrease.

To my mind, a version number is a dot-separated sequence of numbers. So 12.2 is smaller than 12.2.1. Any sane version number comparison will separate the two strings on dots and compare the individual components. Any missing components (12.2 is, for example, one component shorter than 12.2.1) should be assumed to be zero.

But that’s not what the Perl toolchain does. Observe:

When the version number with two components (2.12) is split into components, the second component is bizarrely treated as a three-digit number so it becomes 12o instead of 12 and when it is compared with the second component of the three-component version, 120 is obviously larger than 12 and any tool which relies on this behaviour to work out which version of a module is the most recent will get the wrong answer.

This leads to other “interesting” effects. In my head, versions 1.1, 1.01 and 1.001 are all the same version. The leading zeroes mean nothing. But under this scheme, they are very different version numbers.

I know that versioning isn’t as easy as it should be and I know that some people use bizarre versioning systems. And I’m pretty sure that no matter how bizarre a versioning system is, you’ll almost certainly find an example of it on CPAN. So I suppose that this behaviour was a “least worse” scenario that was chosen to make the most sense given CPAN’s wide range of versioning schemes.

Personally, I see it as a bug in version.pm. But I’m not going to report it as such as I’m sure the Perl toolchain gang know what they’re doing and have very good reasons for adopting this seemingly broken behaviour.

I just need to remember to be more careful when switching my modules to semantic versioning. Using a minor or patch level version change when switching to semantic versioning is likely to lead to confusion and bug reports. Only a major level change (as I did with Symbol::Approx::Sub) is guaranteed to work.

And, I suppose, I’ll need to release Array::Compare 3.0.0 to CPAN pretty soon.

Hacking Symbol::Approx::Sub

In October, for (I think) the second year, Digital Ocean ran Hacktoberfest – a campaign encouraging people to submit pull requests to Github repos in exchange for free t-shirts.

A few of us thought that this might be a good way to do a small bit of easy Perl advocacy, so we tagged some issues on Perl repos with “hacktoberfest” and waited to see what would happen.

I created a few issues on some of my repos. But the one I concentrated on most was symbol-approx-sub. This is a very silly CPAN module that allows you to make errors in the names of your subroutines. I wrote it many years ago and there’s an article I wrote for The Perl Journal explaining why (and how) I did it.

Long-time readers might remember that in 2014 I wrote an article for the Perl Advent Calendar about Perl::Metrics::Simple. I used Symbol::Approx::Sub as the example module in the article and it showed me that the module had some depressingly high complexity scores and I planned to get round to doing something about that. Of course, real life got in the way and Symbol::Approx::Sub isn’t exactly high on my list of things to do, so nothing happened. Until this October.

Over the month, a lot of changes were made to the module. I probably did about half of it and the rest was pull requests from other people. The fixes include:

  • Better tests (and better test coverage – it’s now at 100%)
  • Using Module::Load to load module
  • Using real exceptions to report errors
  • Updating the code to remove unnecessary ampersands on subroutine calls
  • Fixed a couple of long-term bugs (that were found by the improved tests)
  • Breaking monolithic subroutines down

And I’m pretty happy with how it all went. The work was mostly completed in October and this morning I finally got round to doing the last couple of admin-y bits and version 3.0.0 of Symbol::Approx::Sub is now on the way to CPAN. You still shouldn’t use it in production code though!

Thanks to everyone who submitted a pull request. I hope you did enough to earn a free t-shirt.

If you want to get involved in fixing or improving other people’s code, there’s the 24 Pull Request Challenge taking place over Advent. Or for more Perl-specific code, there’s the CPAN Pull Request Challenge.

p.s. In the Advert Calendar article, I linked to the HTML version of the results. For comparison, I’ve also put the new results online. It’s a pretty good improvement.

The Fragility of Contracting

I’ve been rather quiet for a few months. That’s because I’ve been working for a large investment bank in Canary Wharf. It’s no so much that the work takes up more of my time than other contracts I’ve had, but more that the incredibly restrictive firewalls banks have around their networks have meant that I have far less ability to keep in touch with things during the working day. I understand security is so important to them but, wow, it’s hard having to live with it.

Working in the finance sector is lucrative, but not much fun (which, I suppose, might explain why they make it so lucrative).

But all that is about to change. On Wednesday, the project leader told me that the bank were letting all of the contractors in the group go. It had come as a complete surprise to him too – he had just received an email telling him to let us know. That’s the way things work in the banking sector.

I’m not sure if it was a slow reaction to Brexit or an extremely quick reaction to Trump or something else completely. But we’ll all be leaving at the end of this month.

Which means that I’m looking for a new contract. So if you’re reading this and you know of a team who are looking for a contractor then please let me know and we might be able to work something out.

Because of way this was timed, I think I’ll probably be looking for something to start at the beginning of next year. I’m going to South Africa for a couple of weeks at the end of the year and it seems pretty pointless to do a couple of weeks at a new job before going away for a while and forgetting everything I’ve learned.

But it would be nice if I didn’t spend all of the first two weeks of December watching Netflix. So there are a few possibilities I’m considering:

  • Could I find magazines or web sites that would pay me to write articles for them?
  • Could I go into a company for a few days of consultancy (perhaps an architectural review or something like that)?
  • Could I do a code review for some of your companies codebase?

Or, the most likely option:

  • Do you have colleagues who could benefit from a few days Perl training? Have you been vaguely thinking “you, know it might be nice to get Dave in to run some in-house training”? If that’s the case, then the first couple of weeks of December would be a great time to get more serious about this.

In fact, if there’s any way that you think I could be of use to your company for a few days in December or on a longer-term basis from January, then please get in touch.

Feedback

During the week, Barbie sent out the results from the feedback survey that he ran after YAPC Europe. The general results will be published later, but all of the speakers will have received an email containing the feedback from their talks. That feedback is private, but I’m happy to share mine with the world.

The feedback survey takes the form of five questions. People are asked to answer these questions with a rating from 1 to 10. The questions are:

  • Q1: Your prior knowledge of subject?
  • Q2: Speaker’s knowledge of subject?
  • Q3: Speaker’s presentation of subject?
  • Q4: Quality of presentation materials?
  • Q5: Overall presentation rating?

There is also an opportunity for people to write in more detailed comments if they want.

I gave two talks at the conference. A lightning talk called “Medium Perl” which introduced the idea of the Cultured Perl blog and a longer talk called “Error(s) Free Programming” which talked about Damian Conway’s module Lingua::EN::Inflexion.

Eight people gave feedback on “Medium Perl”.

Qu 1 2 3 4 5 6 7 8 9 10 Avg
Q1 1 1 2 1 2 1 4.5
Q2 1 1 1 5 9.25
Q3 1 1 1 5 8.875
Q4 1 1 1 5 8.875
Q5 2 1 1 4 8.875

What aspects of the tutorial or presentation worked really well?

  • I always enjoyed Dave’s humor.
  • History and goal are clear
  • Excellent presentation, as you always do. Funny and surprising.

How could the tutorial or presentation be improved?

  • Make Medium use a readable font or have them stop forcing me to use serif fonts. As long as the articles are presented as they are, I won’t read them at all. Period. (let alone open the possibility that I would post any material myself)

I’m not really sure how I’m supposed to make Medium change their fonts. I suppose I could suggest that they make other fonts available as an option. But then, so could the person who made that comment.

Four people gave feedback on “Error(s) Free Programming”.

Qu 1 2 3 4 5 6 7 8 9 10 Avg
Q1 1 1 1 1 4
Q2 3 1 9.25
Q3 2 2 9.5
Q4 1 3 9.75
Q5 2 2 9.5

What aspects of the tutorial or presentation worked really well?

  • Just about everything, an excellent presentation. Congrats.
  • Damianware!

How could the tutorial or presentation be improved?

  • I misunderstood the topic, and I thought it was a talk about programming without errors instead of how to solve localization of messages.

I can only suggest that the last people reads the talk description, not just the title in future.

I also got feedback about the “Modern Web Programming with Perl and Dancer” course that I ran before the conference. The feedback here is in a slightly different format as it’s a form that I made up myself. I got feedback from 11 people.

1 2 3 4 5 6 7 8 9 10 Avg
On a scale of 1 to 10, how do you rate your Perl ability?
1 1 2 2 3 1 1  7.09
On a scale of 1 to 10, how useful did you find the course?
2 1 6 1 1  7.45
On a scale of 1 to 10, how much did you enjoy the course?
1 5 2 3  8.54
On a scale of 1 to 10, how do you rate the instructor’s knowledge of the subject?
2 1 6 2  8.72
On a scale of 1 to 10, how well did the instructor teach the subject matter?
1 2 2 4 2  8.36
On a scale of 1 to 10, please rate the amount of material covered
1 1 3 1 1 2 2  6.27

That last question is always tricky. The form is clear that if you think it was just right, to score 5. But I always get some people choosing 10 and I think I’d know if people thought I was covering stuff far too quickly. That 1 is a bit of a worry though.

So, all in all, not bad scores. And generally people saying nice things. Which is always nice to see.

Now I need to start thinking about the London Perl Workshop.