A Cautionary Tale

I can never remember exactly how Time::Piece works. But that’s ok because I have documentation.

$ perldoc Time::Piece
No documentation found for "Time::Piece".


$perl -v
This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-thread-multi

$ corelist Time::Piece
Time::Piece was first released with perl v5.9.5

$ perl -MTime::Piece -E'say $Time::Piece::VERSION'
Can't locate Time/Piece.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
BEGIN failed--compilation aborted.

So Time::Piece has been in the Perl core since 5.9.5. I’m running Perl 5.14.2 but I don’t have Time::Piece installed.

After ten minutes or so of head-scratching it came to me.

$ sudo yum install perl-core
Loaded plugins: langpacks, local, presto, refresh-packagekit
[ stuff snipped ]
---> Package perl-Time-Piece.x86_64 0:1.20.1-212.fc17 will be installed
[ more stuff snipped]

I’m running Fedora. The Fedora packagers have decided that they don’t need to install the whole standard Perl distribution as part of their standard installation. I don’t have a problem with that. I do have a problem with their naming conventions.

The minimal Perl installation that they include by default is in an RPM called “perl”. The full RPM that includes everything that a Perl developer would expect to see is called “perl-core”. Surely it’s obvious that those names are the wrong way round?

Isn’t there some way that the Perl 5 Porters can object to  this renaming of Perl?

I know I should be installing my own Perl with perlbrew. But I generally find that the system Perl works for everything that I need. There’s just this one thing that is guaranteed to trip me up every time I work on a new Fedora installation.

This is a public service blog post. Perhaps someone will come across it and be saved a couple of hours of confusion.

CGI.pm vs Templates

I’ve just been involved in a discussion on LinkedIn that I thought deserved a wider audience (I have no idea how well that link works if you’re not a member or or logged in to LinkedIn).

A couple of days ago, someone asked for advice on the best way to include HTML in a Perl program. They got a lot of good advice (basically – don’t do that, use a templating system instead). And then this morning, someone came in and said:

use CGI module, u don’t need to write a single HTML tag..just use CGI Methods…

I did a bit of a double take at that point. I thought we’d all stopped using CGI.pm’s HTML generation methods back in the last millennium. I replied saying that, but this chap was adamant that CGI.pm worked for him. He asked me to explain why I was so against his approach.

This was my reply:

I never said that it didn’t work. I just think that it’s a really limited way to build things and you’d be better off taking a more flexible approach right from the start.

My main objection is the separation of concerns. The logic of your application is separate to the display layer. It’s quite possible that the display of the application will need to change in the future. And that all becomes easier if the HTML is stored separately from code.

This leads to other problems too:

As I hinted at above, CGI.pm only has pretty basic support for HTML. It’s really hard to create a great-looking web application using CGI.pm. In fact, I wouldn’t be surprised if CGI.pm was largely responsible for the terrible web sites that the Perl community has been building for itself for most of the last twenty years.

Your project might well have HTML experts who know how to create good looking web pages. Expecting them to do it by editing Perl code is a bad idea. Front end developers need to know HTML, CSS and Javascript. Why force them to understand Perl as well?

Even if you’re the only developer on a project (as I often am for my personal projects) there are advantages to separating your work into “front-end task” and “back-end tasks”. I find it really helps me switch my mindset to the appropriate skills if I switch from editing a Perl source file to editing an HTML template.

It makes your Perl code more complicated than it needs to be. Seriously, try moving the HTML out of the code and into a template. See how much simpler your code becomes.

It’s unlikely that your CGI-generated pages will make up all of the web site. There will almost certainly be static pages too. If you use a tool like the Template Toolkit then you can generate static pages from the same set of templates that your CGI programs use. That way it’s easier to maintain a consistent look and feel across all of the pages on the site.

I’m not saying “don’t use CGI.pm” (well, I am a bit – but that’s a different story). The CGI bits of CGI.pm (for example the bits that give you access to the incoming parameters or let you set the outgoing headers) are great. It’s just the HTML generation stuff that I strongly recommend that you don’t use.

I’ve just remembered an article that I wrote in 2001 that might also help. The first half is about cookies, but the second half demonstrates using Templates instead of the HTML generation methods. I hope it shows how much simpler it makes things

See http://mag-sol.com/articles/cgi3.html

Does that help at all?

I think that pretty much sums up my objections. Did I miss anything obvious? Or am I completely wrong and many of you are still using the HTML generation methods in CGI.pm to do this?

The Perl community on LinkedIn is a fascinating place. I should really write a blog post about it.

What is Modern Perl?

I wrote an article for Josette called “What is Modern Perl?” In it, I talk about the different things that people might mean when they talk about Modern Perl and why it’s well worth buying a copy of the new edition of the camel book.

After a gap of twelve years, a new edition of Programming Perl (affectionately known as “the Camel book” to its many fans) was published earlier this year. That came as quite a surprise to many people who had given up on seeing a new edition.  What has changed in the new edition? Does the book now cover Modern Perl? The answer depends on what you mean by the term “Modern Perl”.

Of course it’s not a complete coincidence that the subject ties in nicely with my forthcoming Modern Perl for Non-Perl Programmers course.

Update: There’s a discussion of the article on Hacker News. It seems to be taking a predictable path.

How Well Can You Read Documentation?

(I was going to call this post “How well do you understand context?” but I think this title is more accurate).

I just saw someone recommending this code:

Looks sensible enough, doesn’t it? But it isn’t. What’s the hidden inefficiency?

“You Must Hate Version Control Systems”

I’ve been an independent consultant for a long time now. Over the last seventeen years I’ve worked for dozens of different clients. In that time it’s been interesting to watch how good practices have slowly permeated the industry. These days, when I start working with a new client there’s about a 50% chance that they will have some kind of Continuous Integration environment in place. Over the next couple of years that percentage will, no doubt, increase and CI will just become part of the standard software development toolkit. Those of you thinking “but it’s already part of the standard software development toolkit” should realise that not everyone is as leading edge as you are.

For example, before CI was on the scene, unit testing was the big new idea. Over the last ten years, the percentage of clients where I seen unit tests being used has gone from about 10% to pretty much 100%. For most software developers, the idea that you would develop any large project without unit tests seems ridiculous. But it wasn’t always like that.

Before that, it was source code control. When I first started out in my career I had a number of clients where I spent a lot of time persuading people of the benefits of source code control. At one large bank in the City of London I was charged with getting all of the development teams in one department to use source code control. It was probably SCCS or RCS – either of which is just barely better than nothing. One of the development team leaders was particularly hard to persuade. At one point he told me:

I understand exactly what source code control is for. But it solves a problem that my team just doesn’t have.

I didn’t really understand that. He had a team of three people. They all worked on the same codebase. How was it possible that source code control wouldn’t make their lives easier? Later I worked more closely with that team and came to understand their working techniques as I found a directory of tarballs with datestamps in their names[1].

Of course, this is all ancient history now. In these enlightened times we can laugh at stories like this because we all know how important source code control is.

But look at this job description which was posted on jobs.perl.org a couple of weeks ago. There are a number of things in this advert which worry me – “raw perl (no modules)” – but I think the thing that scared me most was where it says:

You must hate version control systems, we won’t be using any.

I’m not sure what’s the most surprising thing here – the fact that there are people who still think like this or the fact that they admit it in a job advert as they think it will encourage people to apply for their job.

All in all I don’t think that Holophrastic Enterprises[2] sounds like the kind of place that I’d like to work. You might disagree. You might think that cutting through all this “best practices” nonsense and just getting on with coding sounds like your perfect job.

If you apply, please let us know how you get on.

Me, I’ll be sticking with version control.

[1] It can’t be coincidence that this was also the team leader who complained the most when the infrastucture and deployment team I worked in took the decision to remove developer access to the production servers.

[2] And if anyone can tell me why they need all that Javascript on their single static page web site, I’d love to hear it.

Being Helpful

I like to help people who know less Perl than I do. I like to help them to improve their standard of Perl. I particularly like to help to improve the standard of Perl that is found on random sites on the web. This is because if I find your nasty Perl code on the internet then someone trying to learn Perl might also find your nasty Perl code and not realise just how nasty it is.

I used to do a lot more of this, but I’ve really cut down. Mainly I have a lot less free time now, but also it used to sometimes get me in trouble. People aren’t always as grateful for help as you’d like to think they would be. Those of you who have known me for ten years or so might remember some amusing scrapes from the support forum for a particular beginners’ Perl/CGI book. The phrase “use strict is gay!” still brings a smile to the face of the older London Perl Mongers.

A few months ago I saw this site. The site is owned by a MySQL consultant who posts some really quite complex programs that he has written to interact with MySQL in various ways. Most of his programs are written in Perl. But it’s clear that the author is not really familiar with Perl – he makes that point explicitly in his sidebar. This means that the Perl really isn’t very good.

Now I know that we’re happy for people to use “baby-talk Perl”. And I know that a correct Perl program is one that gets the job done. But I also think that if you’re sharing code with other people then you should make a bit of an effort to make it as well-written as possible.

So soon after I found the site I dived in with some suggestions. “Try using strict and warnings”, “these days we like to recommend lexical filehandles and three-argument open”, “you know you can avoid those leaning toothpicks” – that kind of thing. I didn’t get much response and the Perl didn’t get any better so I lost interest and drifted away.

Yesterday I got an email from the site saying that someone had added a comment to one of the entries that I had commented on. Reading the new comment I saw my contributions described as “obnoxious”. Following my experiences of ten years ago I’m a little sensitive about accusations like that. I grew up on Usenet and I know that my language can sometimes come across as more robust than I intend. But reading back what I wrote, I don’t think that’s the case here. I tried to explain but was told that I was just making my critic’s point even more valid.

I don’t think I’ll be going back to that site. They don’t seem to be interested in my help. But I left them a parting gift – a link to my version of the program. I doubt they’ll say thank you.

Programming Like It’s 1999

This article was published yesterday. It shows a way to extract data about a film from IMDB and put it into a local database. Actually, it doesn’t even do that. It produces SQL that you can then run to insert the data.

It’s all rather nasty stuff and indicative of the fact that most people using Perl are still using idioms that would have made us shudder ten years ago.

There were a few things that I didn’t like about the code. The use of curl to grab data from the web site, the indirect object syntax when creating the XML::Simple object and, in particular, the huge amount of repetitive code used to create the SQL statements.

So I did what anyone would do in my situation. I rewrote it. Here’s my version.

I haven’t actually changed that much. I’ve tidied up a bit. Switched to using LWP::Simple, removed some unnecessary escaping, things like that. I have made two pretty big changes. I’ve got rid of all of the nasty variables containing data about a film. A film is a single object and therefore should be stored in a single variable. And, happily enough, the $data you get back from XMLin contains a hash that does the trick perfectly.

The second change I made was to rejig the way that the SQL is created. By using an array that contains the names of all of the columns in the table, I can generate the SQL programmatically without all of that repetitive code. I’ve even made the SQL a little safer by explicitly listing the columns that we are inserting data into (this has the side effect of no longer needing to insert a NULL into the id column).

Of course, this would just be a first step. The whole idea of generating SQL to run against the database is ugly. You’d really want to use DBIx::Class (or, at the very least, DBI) to insert the data directly into the database. And why mess around with raw XML when you can us something like IMDB::Film to do it?

At that point in my thought process I had an epiphany. You don’t need the database at all. The IMDB data changes all the time. Why take a local copy? Why not just use the web service directly with IMDB::Film (or perhaps WebService::IMDB – I haven’t used either of them so I have no strong opinions on this).

In general, I think that the original code was too complicated. Which made it hard to maintain. My version is better (but I am, of course, biased) but it can be made even better by using more from CPAN.

CPAN is Perl’s killer app. If you’re not using CPAN then you’re not using half the power of Perl.

What do you think? How would you write this program?

Update: A few people have mentioned the fact that I’m directly interpolating random data into my SQL statements – which is generally seen as a bad thing as it opens the door to SQL injection attacks. In my defence, I’d like to make a couple of points.

Firstly, the data I’m using isn’t just any old data. It’s data that is returned from the IMDB API. So it would be hard to use this for a malicious attack on the system (at least until Hollywood makes a film about the life of Bobby Tables).

Secondly, I am cleaning the data before using it. I’m escaping any single quotes in the input data. I think that removes the possibility of attack. I could be wrong though, if that’s the case, please let me know what I’m missing.

But, in general, I agree that this approach is dangerous. This is one of the major advantages of using DBI. By using bound parameters in your SQL statements you can remove possibility of SQL injection attacks.

Update 2: You can, of course, rely on Zefram to point out the issues here. His comment is well worth reading.

Other people (on IRC) raised the potential of other Unicode characters that databases treat as quote characters but that aren’t covered by my substitution.

Update 3: Here’s a local copy of the original code.

Perl Search Engine

Often on sites like StackOverflow you’ll see questions that people could have answered for themselves if they had just searched the right web sites (usually perldoc or CPAN). But instead, they just went straight for Google and ended up with some dodgy, out of date information that just left them confused.

In order to get round that, I’ve created a Google Custom Search Engine which searches known Perl web sites. You can try it out here.

If you want to use this search engine on your site, the code is below.


This generated a lot of discussion in a training course that I ran this week so I thought it was worth sharing more widely.

I think you can say that you understand the concept of context in Perl if you know what these four statements will print and (more importantly) can explain why they don’t all produce the same thing.

Too Easy or Too Hard

We hear a lot of people complaining that programming in Perl is too difficult, but I think that a lot of these problems stem from people making the opposite assumption – that writing Perl is easier than it actually is. Let me share a couple of examples. I’ve lightly disguised the companies in question – so if you think you recognise a place where we’ve worked together you’re probably wrong.

In the first, I’m working for a global organisation. Most of their bespoke software is written in Java, but (like pretty much every company) there are a few Perl programs running vital parts of the infrastructure. This company have also outsourced a lot of their maintenance work to another company in India. This Indian company has dozens of people who are dedicated to working on my client’s systems. But they are Java programmers and every once in while they need to do some work on a Perl program.

This company has an internal IRC network and there’s a Perl channel which I keep half an eye on whilst getting on with my work (which is maintaining a huge Perl program). A few times a day one of these Java programmers turns up on the Perl channel explaining a problem they have to solve in a Perl program and asking for help. In pretty much every case, the problem boils down to the Java programmer having major misconceptions about how Perl works. Perl programmers on the channel try their hardest to to help, but it’s often a frustrating experience. Usually the correct answer is “go away and read Learning Perl and Intermediate Perl, then you will understand how the code works”. But these programmers don’t have the time to do that. They need this task finished in an hour. Often it’s a task that could be easily achieved in an hour – but only after you’ve done the groundwork of understanding how Perl works.

The problem this company had was that Perl wasn’t seen as being as important as it actually was. People in the Indian company saw it as a “scripting language” that any of their (no doubt highly qualified) Java programmers would be able to use without any preparation or training. That’s clearly not the case.

But, I hear you say, that’s not fair. Perl and Java are really different languages. I’m glad you pointed that out as it’s a nice link to my second example.

In this example, I’m working for a dotcom company London. Like so many dotcom companies they have a Perl codebase that was thrown together five years ago in a couple of months by people who didn’t really know what they were doing. This company has the added problem that they are finding it hard to recruit people with Perl experience. So they start recruiting people with experience of languages like Python and Ruby in the belief that these languages are so similar that the programming skills are completely interchangeable.

And, of course, Perl, Ruby and Python are all a lot closer to each other than they are to languages like Java. But there are differences. Important differences that aren’t just syntactic. I can read Python and Ruby just fine. But if I got a job where I had to write a lot of either of those languages, I wouldn’t assume that I could just pick it up by osmosis, I’d get a book and read about the language. If I was going permanent at the company, I might even suggest that they send me on a training course.

Conversations with these non-Perl programmers often took a predictable course. They’d start complaining that the code was hard to read. This was hard to counter as a lot of the code was horrible. But if you pointed out some of the newer and better code, they wouldn’t see the improvement. They would insist that somehow Ruby or Python code was inherently easier to read than Perl code. I’d try to point out that Perl programmers find Perl code as easy to read as Python (or Ruby) programmers find Python (or Ruby).

So there seems to be this perception that Perl should be as easy to read as Random Programmer’s favourite language. And I don’t understand why that is. Just because I’ve been programming for almost thirty years, I don’t expect to be able to program in any language I happen to look at – well, certainly not well enough to be paid for doing it.

I’m not sure what we can do to counter this misconception. I think it probably stems from the late 90s when everyone was writing Perl. And if everyone is doing something, then it must be really easy.Of course, most people were writing really horrible Perl because Perl isn’t as easy as they thought it was.

Not sure it’s possible to sum this up in a simple marketing slogan. “Perl is Easy (but not as easy as you think)”.