Category Archives: Programming

Give Me MetaCPAN

Ever since MetaCPAN launched I’ve been getting increasingly irritated with people who still use links to search.cpan.org. Isn’t it obvious that MetaCPAN is better? Why do people still insist on sharing links to the older site?

Of course they do it for various reasons. Perhaps they aren’t as in touch with the modern Perl world as I am. Perhaps they are wary about changing to use the new shiny toys because they know that a newer shinier one will be along soon. Perhaps I’m reading a web page from five years ago and they can be forgiven for not linking to a site that didn’t exist at the time.

Eventually I realised that there was no point in getting annoyed. I had a computer. Surely I could do something that would fix this problem.

My first idea was to write a GreaseMonkey script. Then I realised that would be hard. MetaCPAN and CPAN have slightly different URL schemes. I’d need to recognise a CPAN URL and convert it to the equivalent MetaCPAN URL. Not an impossible task at all, but not something I could knock up in an hour or so. Especially not in Javascript.

So I asked on the #metacpan IRC channel. Surely I couldn’t be the first person to have this problem. And someone there introduced me to mcpan.org. There’s no point in clicking on that link. You’ll just end up on MetaCPAN. Because that’s what mcpan does. It’s a URL rewriting service. You give it a CPAN URL (with the cpan.org changed to mcpan.org) and it redirects you to the equivalent page on MetaCPAN.

That’s the hard bit of the problem. The bit I didn’t want to write. And someone else has already written it. But they seem to have kept it very quiet. This deserves more publicity. I wish I could remember who wrote it. If you know, please leave a comment.

So we’re now most of the way there. Now if I click on a CPAN link, I can just edit the location bar to add an ‘m’ and I’ll be redirected to the right place. But we can do better than that. I’d like to be automatically redirected. That’s when I discovered the Redirector extension for Firefox. Once it’s installed you can configure it to redirect certain URLs to other ones. I have it configured so that http://search.cpan.org* redirects to http://mcpan.org$1. See how you can use wildcards to match URLs and then use whatever they matched in the replacement URL. It’s a lot like regexes in Perl.

And we’re done. Now whenever I click on an old-style CPAN link, I’m automatically redirected to mcpan, And that, in turn, redirects me to MetaCPAN. And, best of all, I didn’t have to write any code. It was just a case of putting together tools that already existed.

I did all this a few months ago. I meant to write this blog post at the time, but I forgot. I was reminded this morning when Chisel mentioned a GreaseMonkey script he had written to do all of this. See, Chisel isn’t afraid of parsing URLs in Javascript like I was. He just went ahead and did it.

But having alternative solutions to the problem is good, right?

Update: I’ve just been talking about this on the #metacpan IRC channel and it seems that I was rather misunderstanding what was going on here. Here are the details.

Firstly, it was dpetrov who told me about mcpan.org. It’s his domain. I asked him for the code, and he pointed out that there is no code. mcpan.org just redirects everything to a domain called sco.metacpan.org which is where the magic happens. He just got tired of editing search.cpan.org to sco.metacpan.org so he registered another, simpler, domain.

So the actual cleverness happens over on sco.metacpan.org. And that’s really just a list of rewrite rules (the code is on github). I say “just”, but I still wouldn’t want to write them myself.

All this means that mcpan,org is only a convenient tool for when you’re manually editing your location bar. When you’re using the Redirector extension for Firefox you can miss out the middle man and redirect straight to sco.metacpan.org. So I’ve updated my Redirector rule appropriately.

Why Corporates Hate Perl

This is a reprint of an old blog post.

A few years ago I was writing blog posts (semi-)regularly for O’Reilly. This is the one that probably got the most feedback. I’m reprinting it now because a) it’s pretty hard to find on the O’Reilly site and b) it’s relevant to a couple of conversations that I’ve had over the last few days.

Last week I was in Copenhagen for YAPC::Europe. One of the announcements at the conference was the location of next year’s conference which will be in Lisbon. The theme of next year’s conference will be “Corporate Perl”. And that (along with a couple of conversations last night) got me thinking about a talk that I’ll submit to next year’s conference which might well be entitled “Why Corporates Hate Perl”.

It’s not true, of course. There are a still large number of large companies who love Perl. I could probably work through to my retirement enhancing and extending systems that are written in Perl at many of the big banks in the City of London. There are, however, also many companies who are moving away from Perl for a number of reasons. Here’s one of the reasons that will be included in my talk.

I was talking to people from one such company last night. The Powers That Be at this company have announced that Perl is no longer their language of choice for web systems and that over time (probably a lot of time) systems will be rewritten in a combination of Java and PHP. Management have started to refer to Perl-based systems as “legacy” and to generally disparage it. This attitude has seeped through to non-technical business users who have started to worry if developers mention a system that is written in Perl. Business users, of course, don’t want nasty old, broken Perl code. They want the shiny new technologies.

And so, in a matter of months, the technical managers at this company have created a business environment where Perl is seen as the cause of most of the problems with the current systems. It’s an impressive piece of social engineering.

It’s also, of course, completely unfair. I don’t deny at all that this company (like many others) has a large amount of badly written and hard to maintain Perl code. But I maintain that this isn’t directly due to the code being written in Perl. It’s because the Perl code has developed piecemeal over the last ten or so years in an environment where there was no design authority which encouraged developers to think beyond getting their immediate task done. Many of these systems date back to this company’s first steps onto the internet and were made by separate departments who had no interaction with each other. It’s not really a surprise that the systems don’t interact well and a lot of the code is hard to maintain.

There are, on the other hand, a number of newer systems which are also written in Perl which follow current best practices in Perl development and are far easier to to maintain and enhance – as easy, I would contend, as anything written in the new approved languages.

It’s certainly true that this company has a large number of systems that need to be rewritten over the next few years. But throwing away all of the company’s accumulated Perl expertise and moving to new languages seems to be a step too far. Management are blaming Perl for the problems when really they should be blaming the management and design procedures that were in place (or, more likely, weren’t in place) when the code was originally written.

Many organisations are in the same situation, with large amounts of unwieldy Perl code. Ten or twelve years ago everyone was writing web systems in Perl and we were all making mistakes. We all have to deal with those mistakes but we’ve  hopefully, learned from them and can rewrite our systems to take account of everything that we’ve learned in the last ten years.

It’s too late for the company I’ve been talking about in this article. The anti-Perl social engineering has probably insinuated itself too deeply into the culture. It’s unlikely that Perl’s reputation can be rescued.

But if you have similar problems in your own company, then please try to ensure that blame is apportioned correctly and that you don’t use Perl as a scapegoat.

A couple of updates to the post. I did propose the talk to the next YAPC, but the proposal wasn’t accepted. And the company I talk about in the article is still employing a lot of Perl programmers – four years after this post was written.

 

DBIC vs DBI

Three times in the last few months I’ve had the “DBIC or raw DBI” discussion. People have told me that they don’t use DBIC because raw DBI is better. And each time, the person promoting DBI in the discussion has used an argument that boils down to “DBIC is probably useful for people who don’t know SQL very well”.

I find that argument really puzzling. Not least because I like to think that I know more than a little about SQL. SQL is a skill that that has run through my career for longer than Perl. I’ve been using SQL since I left college in 1988. I only started using Perl in about 1996. And yet, although I still consider myself a bit of an expert in SQL, I use DBIC for pretty much all of the database work I do these days – and have done for about five years.

I use DBIC not because I don’t understand SQL. I use it because it makes my life easier. I use it because it frees up some of the time I used to spend dealing with the minutia of database communication so that I can spend it working on other, more interesting, parts of my project.

When I’m running training courses that introduce DBIC I have slide that is entitled “SQL is Boring”. It’s a joke of course but, like all the best jokes, it gets a laugh because there’s more than a little truth in it. There are, of course, many interesting SQL problems. I’ve spent many an enjoyable (if slightly frustrating at the time) hour trying to coax the right data out of complex query with correlated subqueries, outer joins and aggregated functions. But that’s the exception rather than the rule.

The vast majority of the SQL I write for the applications I work on is incredibly boring. It’s  boring because it’s all so similar. You get the data to present a list of objects to the user. The user selects the object they’re interested in, so you select all the data about that object. You might select some data about related objects. The user changes some of that data, so you update that row in the database. On a good day, you might delete an object from the database. Or insert a new one. Most of the SQL you need is like that. It’s boring.

We have computers to do the boring work for us. So let the computer generate all that boring SQL. Free up your time to work on the gnarly and interesting problems.

But that’s not the only advantage of using DBIC (or some other ORM). Think about the data that you get back from the database. The data you get back from a DBI call is an array. Or perhaps a hash. Or maybe a multi-dimension data structure if you’re using one of DBI’x more complex fetch() methods. But it’s still a dumb variable. From DBIC, I get an object. An intelligent variable. A variable that knows how to react to various messages. A variable that will save any changes back to the database automatically without me having to worry about where it came from and making sure that I’m writing it back to the right place.

I’m not saying for a second that there’s no place for DBI any more. For a start, DBI underlies DBIC so it’s still a vital part of our toolkit. And of course I still use it for quick one-off scripts. But when those scripts are still hanging round being used and maintained three months later (as they always are) I’ll rewrite them to use DBIC.

If you want to write large applications that are going to be easy to maintain and extend, then you should really be using DBIC (or something similar). I don’t care how well you know SQL. DBIC will make your life easier.

I’ve just mentioned a couple of reasons why I think that DBIC makes my life easier. I’m sure I’ve missed important stuff. What do you think? Why do you use DBIC instead of DBI?

Learning from Bad Code

I’ve written before about Linux Format’s habit of sharing badly written Perl code. I thought things were improving, but in the new edition (November 2012, issue 163) they’re back to their old tricks.

This time it’s a tutorial called “Starfield: Learn new languages”. In this tutorial Mike Saunders writes similar starfield simulation code in C, Python and Perl. Mike’s loyalties are made perfectly clear when these three sections are entitled “Low Level”, “High Level” and “Unusual Level” respectively, so I wasn’t expecting much. But here’s what I got.

#!/usr/bin/perl

$numstars = 100;

use Time::HiRes qw(usleep);
use Curses;
$screen = new Curses;
noecho;
curs_set(0);

for ($i = 0; $i < $numstars ; $i++) {
  $star_x[$i] = rand(80);
  $star_y[$i] = rand(24);
  $star_s[$i] = rand(4) + 1;
}

while (1) {
  $screen->clear;

  for ($i = 0; $i < $numstars ; $i++) {
    $star_x[$i] -= $star_s[$i];
    if ($star_x[$i] < 0) {
      $star_x[$i] = 80;
    }

    $screen->addch($star_y[$i], $star_x[$i], ".");
  }

  $screen->refresh;
  usleep 50000;
}

Let’s be clear here. This code works exactly as Mike intended it to. But if you’re writing sample code for other people to learn from, I think you should be setting the bar a little higher than “works for me”. I think you should should be aiming to show people good quality code that can be easily mainitained. And this code falls well short of that aim.

Let’s look at some of the problems in more detail:

  • No use strict or use warnings. Let’s be generous and assume the editor removed them to save space.
  • Undeclared variables. Of course. if the code contained use strict then the variables would all have had to be declared. Not declaring them means that we’re using package variables rather than lexical variables. In code this simple it doesn’t make a difference. But it’s a bad habit to get into.
  • Indirect object notation. When creating the Curses object the tutorial uses the syntax $screen = new Curses. Again, not a problem in this program, but a really bad habit to be encouraging. In the article’s defence, the documentation for the Curses module only includes this flawed syntax.
  • Split data structures. The author of the article says “instead of having an array of stars containing coordinates and speeds for each one (i.e. an array of arrays), to make things simpler we’ve just set up three arrays.” I read this to mean “I’ve never been able to work out how to make arrays of arrays work in Perl so I’ve taken the easy way out.” This is, of course, a terrible idea. Linked data items should be stored in the same data structure.
  • C-style for loops. The mark of a C programmer who never really got to grips with Perl. The C-style for loop is rarely used in Perl code. The foreach loop almost always leads to more readable code.
  • Magic numbers. The size of the screen and the maximum speed of the stars appear as numbers in the code. Even if you’re not a Perl programmer, surely you would know that it’s good practice to move those into variables or constants.

With all that in mind, here’s my version of the program.

#!/usr/bin/perl

use strict;
use warnings;

use Time::HiRes qw(usleep);
use Curses;

my $numstars = 100;
my $screen_x = 80;
my $screen_y = 24;
my $max_speed = 4;

my $screen = Curses->new;
noecho;
curs_set(0);

my @stars;

foreach my $i (1 .. $numstars) {
  push @stars, {
    x => rand($screen_x),
    y => rand($screen_y),
    s => rand($max_speed) + 1,
  };
}

while (1) {
  $screen->clear;

  foreach my $star (@stars) {
    $star->{x} -= $star->{s};
    $star->{x} = $screen_x if $star->{x} < 0;

    $screen->addch($star->{y}, $star->{x}, '.');
  }

  $screen->refresh;
  usleep 50000;
}

What do you think? Is it more readable? Easier to maintain? Are there any problems or improvements that I’ve missed?

A Cautionary Tale

I can never remember exactly how Time::Piece works. But that’s ok because I have documentation.

$ perldoc Time::Piece
No documentation found for "Time::Piece".

Huh?

$perl -v
This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-thread-multi
...

$ corelist Time::Piece
Time::Piece was first released with perl v5.9.5

$ perl -MTime::Piece -E'say $Time::Piece::VERSION'
Can't locate Time/Piece.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
BEGIN failed--compilation aborted.

So Time::Piece has been in the Perl core since 5.9.5. I’m running Perl 5.14.2 but I don’t have Time::Piece installed.

After ten minutes or so of head-scratching it came to me.

$ sudo yum install perl-core
Loaded plugins: langpacks, local, presto, refresh-packagekit
[ stuff snipped ]
---> Package perl-Time-Piece.x86_64 0:1.20.1-212.fc17 will be installed
[ more stuff snipped]

I’m running Fedora. The Fedora packagers have decided that they don’t need to install the whole standard Perl distribution as part of their standard installation. I don’t have a problem with that. I do have a problem with their naming conventions.

The minimal Perl installation that they include by default is in an RPM called “perl”. The full RPM that includes everything that a Perl developer would expect to see is called “perl-core”. Surely it’s obvious that those names are the wrong way round?

Isn’t there some way that the Perl 5 Porters can object to  this renaming of Perl?

I know I should be installing my own Perl with perlbrew. But I generally find that the system Perl works for everything that I need. There’s just this one thing that is guaranteed to trip me up every time I work on a new Fedora installation.

This is a public service blog post. Perhaps someone will come across it and be saved a couple of hours of confusion.

CGI.pm vs Templates

I’ve just been involved in a discussion on LinkedIn that I thought deserved a wider audience (I have no idea how well that link works if you’re not a member or or logged in to LinkedIn).

A couple of days ago, someone asked for advice on the best way to include HTML in a Perl program. They got a lot of good advice (basically – don’t do that, use a templating system instead). And then this morning, someone came in and said:

use CGI module, u don’t need to write a single HTML tag..just use CGI Methods…

I did a bit of a double take at that point. I thought we’d all stopped using CGI.pm’s HTML generation methods back in the last millennium. I replied saying that, but this chap was adamant that CGI.pm worked for him. He asked me to explain why I was so against his approach.

This was my reply:

I never said that it didn’t work. I just think that it’s a really limited way to build things and you’d be better off taking a more flexible approach right from the start.

My main objection is the separation of concerns. The logic of your application is separate to the display layer. It’s quite possible that the display of the application will need to change in the future. And that all becomes easier if the HTML is stored separately from code.

This leads to other problems too:

As I hinted at above, CGI.pm only has pretty basic support for HTML. It’s really hard to create a great-looking web application using CGI.pm. In fact, I wouldn’t be surprised if CGI.pm was largely responsible for the terrible web sites that the Perl community has been building for itself for most of the last twenty years.

Your project might well have HTML experts who know how to create good looking web pages. Expecting them to do it by editing Perl code is a bad idea. Front end developers need to know HTML, CSS and Javascript. Why force them to understand Perl as well?

Even if you’re the only developer on a project (as I often am for my personal projects) there are advantages to separating your work into “front-end task” and “back-end tasks”. I find it really helps me switch my mindset to the appropriate skills if I switch from editing a Perl source file to editing an HTML template.

It makes your Perl code more complicated than it needs to be. Seriously, try moving the HTML out of the code and into a template. See how much simpler your code becomes.

It’s unlikely that your CGI-generated pages will make up all of the web site. There will almost certainly be static pages too. If you use a tool like the Template Toolkit then you can generate static pages from the same set of templates that your CGI programs use. That way it’s easier to maintain a consistent look and feel across all of the pages on the site.

I’m not saying “don’t use CGI.pm” (well, I am a bit – but that’s a different story). The CGI bits of CGI.pm (for example the bits that give you access to the incoming parameters or let you set the outgoing headers) are great. It’s just the HTML generation stuff that I strongly recommend that you don’t use.

I’ve just remembered an article that I wrote in 2001 that might also help. The first half is about cookies, but the second half demonstrates using Templates instead of the HTML generation methods. I hope it shows how much simpler it makes things

See http://mag-sol.com/articles/cgi3.html

Does that help at all?

I think that pretty much sums up my objections. Did I miss anything obvious? Or am I completely wrong and many of you are still using the HTML generation methods in CGI.pm to do this?

The Perl community on LinkedIn is a fascinating place. I should really write a blog post about it.

What is Modern Perl?

I wrote an article for Josette called “What is Modern Perl?” In it, I talk about the different things that people might mean when they talk about Modern Perl and why it’s well worth buying a copy of the new edition of the camel book.

After a gap of twelve years, a new edition of Programming Perl (affectionately known as “the Camel book” to its many fans) was published earlier this year. That came as quite a surprise to many people who had given up on seeing a new edition.  What has changed in the new edition? Does the book now cover Modern Perl? The answer depends on what you mean by the term “Modern Perl”.

Of course it’s not a complete coincidence that the subject ties in nicely with my forthcoming Modern Perl for Non-Perl Programmers course.

Update: There’s a discussion of the article on Hacker News. It seems to be taking a predictable path.

“You Must Hate Version Control Systems”

I’ve been an independent consultant for a long time now. Over the last seventeen years I’ve worked for dozens of different clients. In that time it’s been interesting to watch how good practices have slowly permeated the industry. These days, when I start working with a new client there’s about a 50% chance that they will have some kind of Continuous Integration environment in place. Over the next couple of years that percentage will, no doubt, increase and CI will just become part of the standard software development toolkit. Those of you thinking “but it’s already part of the standard software development toolkit” should realise that not everyone is as leading edge as you are.

For example, before CI was on the scene, unit testing was the big new idea. Over the last ten years, the percentage of clients where I seen unit tests being used has gone from about 10% to pretty much 100%. For most software developers, the idea that you would develop any large project without unit tests seems ridiculous. But it wasn’t always like that.

Before that, it was source code control. When I first started out in my career I had a number of clients where I spent a lot of time persuading people of the benefits of source code control. At one large bank in the City of London I was charged with getting all of the development teams in one department to use source code control. It was probably SCCS or RCS – either of which is just barely better than nothing. One of the development team leaders was particularly hard to persuade. At one point he told me:

I understand exactly what source code control is for. But it solves a problem that my team just doesn’t have.

I didn’t really understand that. He had a team of three people. They all worked on the same codebase. How was it possible that source code control wouldn’t make their lives easier? Later I worked more closely with that team and came to understand their working techniques as I found a directory of tarballs with datestamps in their names[1].

Of course, this is all ancient history now. In these enlightened times we can laugh at stories like this because we all know how important source code control is.

But look at this job description which was posted on jobs.perl.org a couple of weeks ago. There are a number of things in this advert which worry me – “raw perl (no modules)” – but I think the thing that scared me most was where it says:

You must hate version control systems, we won’t be using any.

I’m not sure what’s the most surprising thing here – the fact that there are people who still think like this or the fact that they admit it in a job advert as they think it will encourage people to apply for their job.

All in all I don’t think that Holophrastic Enterprises[2] sounds like the kind of place that I’d like to work. You might disagree. You might think that cutting through all this “best practices” nonsense and just getting on with coding sounds like your perfect job.

If you apply, please let us know how you get on.

Me, I’ll be sticking with version control.

[1] It can’t be coincidence that this was also the team leader who complained the most when the infrastucture and deployment team I worked in took the decision to remove developer access to the production servers.

[2] And if anyone can tell me why they need all that Javascript on their single static page web site, I’d love to hear it.

Being Helpful

I like to help people who know less Perl than I do. I like to help them to improve their standard of Perl. I particularly like to help to improve the standard of Perl that is found on random sites on the web. This is because if I find your nasty Perl code on the internet then someone trying to learn Perl might also find your nasty Perl code and not realise just how nasty it is.

I used to do a lot more of this, but I’ve really cut down. Mainly I have a lot less free time now, but also it used to sometimes get me in trouble. People aren’t always as grateful for help as you’d like to think they would be. Those of you who have known me for ten years or so might remember some amusing scrapes from the support forum for a particular beginners’ Perl/CGI book. The phrase “use strict is gay!” still brings a smile to the face of the older London Perl Mongers.

A few months ago I saw this site. The site is owned by a MySQL consultant who posts some really quite complex programs that he has written to interact with MySQL in various ways. Most of his programs are written in Perl. But it’s clear that the author is not really familiar with Perl – he makes that point explicitly in his sidebar. This means that the Perl really isn’t very good.

Now I know that we’re happy for people to use “baby-talk Perl”. And I know that a correct Perl program is one that gets the job done. But I also think that if you’re sharing code with other people then you should make a bit of an effort to make it as well-written as possible.

So soon after I found the site I dived in with some suggestions. “Try using strict and warnings”, “these days we like to recommend lexical filehandles and three-argument open”, “you know you can avoid those leaning toothpicks” – that kind of thing. I didn’t get much response and the Perl didn’t get any better so I lost interest and drifted away.

Yesterday I got an email from the site saying that someone had added a comment to one of the entries that I had commented on. Reading the new comment I saw my contributions described as “obnoxious”. Following my experiences of ten years ago I’m a little sensitive about accusations like that. I grew up on Usenet and I know that my language can sometimes come across as more robust than I intend. But reading back what I wrote, I don’t think that’s the case here. I tried to explain but was told that I was just making my critic’s point even more valid.

I don’t think I’ll be going back to that site. They don’t seem to be interested in my help. But I left them a parting gift – a link to my version of the program. I doubt they’ll say thank you.