Tag Archives: perl

Dots and Perl

I was running a training course this week, and a conversation I had with the class reminded me that I have been planning to write this article for many months.

There are a number of operators in Perl that are made up of nothing but dots. How many of them can you name?

There are actually five dot operators in Perl. If the people on my training courses are any guide, most Perl programmers can only name two or three of them. Here’s a list. It’s in approximate order of how well-known I think the operators are (which is, coincidentally, also the order of increasing length).

One Dot

Everyone knows the one dot operator, right? It’s the concatenation operator. It converts its two operands to strings and concatenates them.

# $string contains 'one stringanother string'
my $string = 'one string' . 'another string';

It’s also sometimes useful to use it to force scalar context onto an expression. Consider the difference between these two statements.

say "Time: ", localtime;
say "Time: " . localtime;

The difference between the two statements is tiny, but the output is very different.

There’s not much more to say about the single dot.

Two Dots

Things start to get a little more interesting when we look at two dots. The two dot operator is actually two different operators depending on how it is used. Most people know that it can be used to generate a range of values.

my @numbers = (1 .. 100);
my @letters = ('a' .. 'z');

You can also use it in something like a for loop. This is an easy way to execute an operation a number of times.

do_something() for 1 .. 3;

In older versions of Perl, this could potentially eat up a lot of memory as a temporary array was created to contain the range and therefore something like

do_something() for 1 .. 1000000;

could be a problem. But in all modern versions of Perl, that temporary array is not created and the expression causes no problems.

Two dots acts as a range operator when it is used in list context. In scalar context, its behaviour is different. It becomes a different operator – the flip-flop operator.

The flip-flop operator is so-called because it flip-flops between two states. It starts as returning false and continues to do so until something “flips” it into its true state. It then continues to return true until something else “flops” it back into a false state. The cycle then repeats.

So what causes it to flip-flop between its two states? It’s the evaluation of its left and right operands. Imagine you are processing a file that contains text that you are interested in and other text that you can ignore. The start of a block that you want to process is marked with a line containing “START” and the end of the block is marked with “END”. Between and “END” marker and the next “START”, there can be lots of text that you want to ignore.

A naive way to process this would involve some kind of “process this” flag.

my $process_this = 0;
while (<$file>) {
  $process_this = 1 if /START/;
  $process_this = 0 if /END/;
  process_this_line($_) if $process_this;
}

I’ve often seen code that looks like this. The author of that code didn’t know about the flip-flop operator. The flip-flop operator encapsulates all of that logic for you. Using the flip-flop operator, the previous code becomes this:

while (<$file>) {
  process_this_line($_) if /START/ .. /END/;
}

The flip-flop operator returns false until its left operand (/START/) evaluates as true. It then returns true until its right operand (/END/) evaluates as true. And then the cycle repeats.

The flip-flop operator has one more trick up its sleeve. One common requirement is to only process certain line numbers in a file (perhaps we just want to process lines 20 to 40). If one of the operands is a constant number, then it is compared the the current record number (in $.) and the operand is fired if the line number matches. So processing lines 20 to 40 of a file is a simple as:

while (<$file>) {
  process_this_line($_) if 20 .. 40;
}

Three Dots

It was the three dot operator that triggered the conversation which reminded me to write this article. A three dot operator was added in Perl 5.12. It’s called the “yada-yada” operator. It is used to stand for unimplemented code.

sub reverse_polarity {
  # TODO: Must write this
  ...
}

Programmers have been leaving “TODO” comments in their code for decades. And they’ve been using ellipsis (a.k.a. three dots) to signify unimplemented code for just as long. But now you can actually put three dots in your source code and it will compile and run. So what’s the benefit of this over just leaving a TODO comment? Well, what happens if you call a function that just contains a comment? The function executes and does nothing. You might not realise that you haven’t implemented that function yet. With the yada-yada operator standing in for the unimplemented code, Perl throws an “unimplemented” error and you are reminded that you need to write that code before calling it.

But the yada-yada operator wasn’t the first three dot operator in Perl. There has been another one in the language for a very long time (you might say since the year dot!) And I bet very few of you know what it is.

The original three dot operator is another flip-flop operator. And the difference between it and the two dot version is subtle. It’s all to do with how many tests can be run against the same line. With the two dot version, when the operator flips to true it also checks the right-hand operand in the same iteration – meaning that it can flip from false to true and back to false again as part of the same iteration. If you use the three dot version, then once the operator flips to true, then it won’t check the right-hand operand until the next iteration – meaning that the operator can only flip once per iteration.

When does this matter? Well if our text file sometimes contains empty records that contain START and END on the same line, the difference between the two and three dot flip-flops will determine exactly which lines are processed.

If the two dot version encounters one of these empty records, it flips to true (because it matches /START/) and then flops back to false (because it matches (/END/). However, the flop to false doesn’t happen until after the expression returns a value (which is true). The net effect, therefore is that the line is printed, but the flip-flop is left in the false state and the following lines won’t be printed until one contains a START.

If the three dot version encounters one of these empty records, it also flips to true (because it matches /START/) but then doesn’t check the right-hand operand. So the line gets printed and the flip-flop remains in its true state so the following lines will continue to be printed until one contains an end.

Which is correct? Well, of course, it depends on your requirements. In this case, I expect that the two dot version gives the results that most people would expect. But the three dot version is also provided for the cases when its behaviour is required. As always, Perl gives you the flexibility to do exactly what you want.

But I suspect that the relatively small number of people who seem to know about the three dot flip-flip would indicate that its behaviour isn’t needed very often.

So, there you have them. Perl’s five dot operators – concatenation, range, flip-flop, yada-yada and another flip-flop. I hope that helps you impress people in your next Perl job interview.

Update: dakkar points out that yada-yada isn’t actually an operator. He’s absolutely right, of course, it stands in place of a statement and has no operands. But being slightly loose with our terminology here makes for a more succinct article.

Perl APIs

For a lot of programmers out there, Perl has become largely invisible. They just never come across it. That might seem strange to you as you sit inside the Perl community echo chamber reading the Perl Ironman or p5p, but try this simple experiment.

Think of a web site that you use and that supplies an API. Now go to that API’s documentation and look at the example code. What languages are the examples written in? PHP? Almost certainly. Ruby? Probably. Python? Probably. C#? Quite possibly. Perl? Almost certainly not[1].

Perl has fallen so far off the radar of most people that when web sites write example code for these APIs, they very rarely consider Perl as a language worth including. And because they don’t bother including Perl then any random programmer coming to that API will assume that the API doesn’t support Perl, or (at the very least) that the lack of examples will make using Perl harder than it would be with plenty of examples to copy from.

This then becomes a self-fulfilling prophecy as Github fills with more and more projects using other languages to talk to these APIs. And the chance of anyone who isn’t already a Perl user ever trying to interact with these APIs using Perl falls and falls.

Of course, this is all completely wrong. These APIs are just going to be a series of HTTP requests using REST or XML-RPC (or, if you’re really unlucky, SOAP). Perl has good support for all of that. You might need to use something like OAuth to get access to the API – well Perl does that too.

Of course, in some cases good Perl support exists already – Net::Twitter is a good example. And to be fair to Twitter, their API documentation doesn’t seem to give any examples in specific languages – so Perl isn’t excluded here. But in many other cases, the Perl version languishes unnoticed on CPAN while other languages get mentioned on the API page.

I think that we can try to address this in 2014. And I’d like to ask you to help me. I’ve set up a mailing list called perl-api-squad where we can discuss this. In a nutshell, I think that the plan should be something like this:

  1. Identify useful APIs where there is either no Perl API or no Perl examples
  2. Write CPAN API wrappers where they are missing
  3. Approve API owners and offer them Perl examples to add to their web site

That doesn’t sound too complicated to me. And I think (or, perhaps, hope) that most API owners will be grateful to add more examples of API usage to their site – particularly if it involves next to no effort on their part.

I also expect that the Perl API Squad will produce a web site that lists Perl API support. We might even move towards producing a framework that makes it easy to write a basic Perl wrapper around any new API.

What do you think? Is this a worthwhile project? Who’s interested in joining in?

[1] Yes, I know there are exceptions. But they are just that – exceptions.

Perl in Banks

An email has flooded in:

I came across your presentation ‘Perl in the Enterprise’ and happen to have a burning concern closely related to one of the bullets ‘Banks see <Perl> as a competitive advantage’.

I am consulting at a major bank in South Africa, and our team have been using Perl very productively to develop and maintain a customer loyalty rewards  application. This application has now suddenly caught the attention of the ‘Architecture Board’ who are questioning whether Perl (or any other scripting language) is appropriate for production use. A presentation will have to be made to this board to present a business case.

Whatever the basis for this concern, some quotable and referencable stories from other banks, such as the list included in your clients on the website, would be gold for us. I have searched extensively on the web and cannot find anything in this light which is also recent, as in the last 5 years. Other than Nvidia and Booking.com I can only find anecdotal, 2nd hand evidence of Perl use for medium-to-large applications in well-known corporates.

Do you by any chance have anything or any contacts in those banks that would be willing to respond to an email to confirm their use of Perl and perhaps repeat the competitive advantage claim?

The presentation he’s talking about is Perl in the Enterprise, which I gave at Linux World Expo in 2005. The particular line he’s interested in is on slide 6 where I say that banks see using Perl as a competitive advantage. I’m pretty sure that I was paraphrasing Phillip Moore of Morgan Stanley who had said something similar in a keynote at OSCON in about 2001.

Obviously I know that banks in the City of London still use Perl. But it’s been several years since I worked at one, so my personal experience is slightly out of date. What my correspondent needs is people who are currently working in banks who are happy to go public and say “we use Perl in our production systems”. It would be even more helpful if they could add “and here’s why…”.

Can anyone out there help?

Perl Search Revisited

A couple of years ago I wrote a blog post about a Google Custom Search that I had set up to create a specialised search engine for Perl.

Recently I’ve revisited this idea. I’ve given the search engine its own subdomain and I’ve added some new sites to the list of sites that it covers. I’ve also given it a simplified look (thanks Bootstrap) and it’s now being hosted on Github pages.

It lives at http://search.perlhacks.com/. Please give it a try and let me know what you think.

Just Build Something

The Political Web

About a month ago, JT Smith suggested that we should all stop talking about Perl and just build something. And, purely coincidentally, over the last few weeks I resurrected a project that I have been poking at for about five years and have finally turned it into something that I’m happy to show the world.

The Political Web is a site which aggregates all of the information I can find on the web about individual British MPs. I say “all of the information”, but that’s obviously a bit of a work in progress. But I think that what I already have is useful and interesting – well, for people who are interested in British politics. I have plans to bring in more information in the future.

Although I’ve been working on the site for five years, I pretty much rebuilt it from scratch when I recently returned to it. Actually getting something useful up and running took about four hours. That’s because I was building it using Perl and, more specifically, Dancer.

Removing Modules from Core

I was on holiday last week and missed most of the discussions about removing Module::Build and CGI from the Perl core in the next few years. I hope you won’t mind if I chime in a little late with my thoughts.

Module::Build

I’m a little bemused by the Module::Build story. Well, perhaps “bemused” isn’t quite the right word. Perhaps I should say “embarrassed”. Embarrassed because the discussions around Module::Build just show how out of touch I am with important parts of the Perl ecosystem.

I remember seeing Schwern’s MakeMaker Is DOOMED! talk ten years ago. What he said made a lot of sense to me and I switched all of my CPAN modules to Module::Build over the next few years. And then I rather took my eye off the ball. I was vaguely aware that people were complaining about Module::Build and were switching to things like Module::Install and Dist::Zilla. But I was happy with Module::Build and just stuck with it.

Now I’m hearing people saying that Module::Build is fundamentally flawed. Someone whose opinion I value mentioned that he lowers his estimate of the IQ of any CPAN author who uses Module::Build. That, presumably, includes me. But when I ask people what the problems are with Module::Build, I can’t get anyone to give me a detailed and dispassionate answer.

I’m quite prepared to believe that there are problems with Module::Build. Given the calibre of the people who are telling me that these problems exist, it would be astonishing if it wasn’t true. But it would be nice if there was a good explanation available somewhere of just what these problems are.

And I think that this is indicative of a deeper problem that the Perl community has. I like to think that I’m pretty much in touch with what is going on in the Perl community. I read the blogs. I’m subscribed to (too many) mailing lists. But it appears I’m not following discussions of the CPAN toolchain closely enough. I guess it’s something that I just expect to work. And I guess I expect that if there are important discussions going on, then I’ll hear about them through other channels – blog posts, for example. But that doesn’t seem to have happened here. Of course I can subscribe myself to another mailing list or two to correct that, but I don’t think I’m unusual in not trying to follow every single Perl discussion. There are surely many people like me who would like information like this to be disseminated better. This is part of the reason why Leo and I set up the Perl News site. But we have a tiny number of editors and we can’t know everything that is worth publishing. If you think we’ve missed an interesting story then please let us know about it.

So I’m not going to object to Module::Build leaving the core. I’m sure there are good reasons, I just wish I knew what they are. I am, however, slightly disappointed to find that Schwern was wrong ten years ago and that ExtUtils::MakeMaker wasn’t doomed.

CGI

CGI.pm and I go back a long way. In fact I think CGI.pm was added to the Perl core at about the same time as I started using Perl.

It’s obvious why it was added to the core. Back in 1997, Perl and CGI were tightly bound together in many people’s minds. I had people telling me that Perl could only be used to write CGI programs and other people telling me that CGI programs could only be written in Perl. Both sets of people were, of course, wrong; but it’s easy to see how they reached both of those conclusions.

Before CGI.pm was added to the Perl core, people used to parse CGI parameters using horribly broken code copied from Matt Wright (code which he had originally copied from Reuven Lerner). Of course, adding the module to the Perl core didn’t magically rewrite all those horrible CGI programs to use CGI::params, but at least it meant that the option was there.

This became important to me when I started the nms project. This project rewrote Matt Wright’s scripts using a better standard of Perl code. The problem was that many people started by installing one of Matt’s scripts on their server and later, when they wanted to move on from using other people’s code, they would use Matt’s code as a template for their own first steps into Perl. This is partly what made Perl so popular. But it’s also what lead to the preponderance of terrible Perl code that you still find on the web today.

We wrote the nms programs to work, as far as possible, in the same environment as Matt’s scripts. Matt said that his scripts would work fine on Perl 4 (back then, there were still many cheap hosting plans that came with Perl 4). We decided not to put ourselves through that pain and to target Perl 5.004 – because that was the first version to include CGI.pm. Remember, we were targeting Matt Wright’s users. And they were people who used cheap hosting plans with basic FTP access and little or no chance of installing anything from CPAN. We had to rely on core modules. If CGI.pm hadn’t been in the Perl core, then the nms programs would have been far harder to write and the project may have never been started.

The nms project was started over ten years ago. What is the situation like now? Those cheap hosting plans still exist. People still use them for simple web sites. Some of those people still use Matt’s scripts or the nms alternatives. But the numbers of people using those programs are far smaller than they used to be. These days, people wanting to add simple dynamic functionality to a web site are far more likely to use a PHP program. And those cheap web hosts are far more likely to want to support PHP.

Of course, these days, no serious Perl developer uses CGI.pm to write web programs. We’d be far more likely to use something based on PSGI. And even if we did want to use CGI.pm, we’d know how to get it installed from CPAN. The only people who are going to suffer from the removal of CGI.pm are the people using something like the nms programs on a cheap hosting plan. I worry a little about the influx of support email we will get when some cheap hosting company updates from Perl 5.18 to Perl 5.21 and suddenly all the nms programs stop working. It’s slightly galling to realise that Matt’s scripts will continue to work at that point. Of course, the kinds of companies that we’re talking about tend to lag well behind the bleeding edge of Perl versions, so this might not happen for eight or more years. So I think it might well be someone else’s problem.

All in all, I think it will be sad to see CGI.pm removed from the core. It will effectively see the end of even half-decent Perl code being used on bottom of the range hosting plans. But Perl has been losing that market for years and, speaking as someone who spent a lot of time helping out on support forums for basic Perl/CGI code, I’m far from convinced that I’ll miss those users. So my sadness at seeing CGI,pm go, will be purely historical.

But I’d really like to see some cut-down version of PSGI take its place!

What New(ish) Perl features Do You Use?

Over on LinkedIn, someone asked me “What core PERL[sic] features do you use regularly that are new since 95?” It’s hard to be sure as the perldelta files only seem to go back to 1997 (for example, when were qw(...), q(...) and qq(...) added?), but here’s a quick list off the top of my head.

  • my was, of course, added in 5.0. But 5.004 added the ability to use it in control expressions – while (my $foo = <>) – and in foreach loops – foreach my $foo (@foos)
  • use VERSION
  • Regex extensions – (?<=RE) and similar. Oh, and qr/.../
  • Data::Dumper (added in 5.005)
  • Unicode support – first added in 5.6.0 and improved in every release since
  • our
  • Three-argument open
  • Omission of intermediate arrows in data structure lookups – $foo[$x][$y] instead of $foo[$x]->[$y]
  • use warnings
  • Memoize
  • Test::More and Test::Simple
  • say
  • defined-or
  • use base (or, more recently, use parent)
  • yada-yada operator

Have I missed anything obvious? What new Perl features do you use most?

Training Debrief

I’ve spent a lot of the last seven days running training courses. It might be interesting to share some thoughts about how they went.

Last Saturday was Perl School 4. A week before the course I was a little worried about ticket sales, but I did a bit of marketing early last week and managed to more than double sales in a few days. In the end I had 27 people signed up.

Perl School is always enjoyable. I think that people often turn up with quite low expectations as it’s so cheap. So it’s fun to overturn those expectations and give them a day of high quality training. People obviously recognise that as I’m getting a lot of repeat business – at least one person has come along to three of the four courses so far.

Many of the courses I give are overviews of Perl at various levels. This one was just about DBIx::Class so it was great to be able to go into a lot more depth on a single topic. Of course, DBIx::Class is a great subject to cover and it was fun explaining its more powerful corners to a room of people who don’t know much about it.

I thought it went well. But don’t just take my word for it. I’ve been asking attendees to fill in feedback forms about all the Perl School courses and I’ve published a page summarising that feedback.

Then this week has been two two-day courses for flossUK. Two day courses give us time to include practical sessions so that people go home having actually tried out the techniques that I’ve taught – which nicely reinforces the lessons. I really enjoy those sessions as you really see lightbulb moments as people see how easy it is to use these tools. This afternoon, for example, it was great to see people getting a simple Catalyst application up and running in less than an hour. An hour later people were really impressed as I introduced them to Plack::Middleware::Debug and showed them how I could get detailed DBIC_TRACE output on the web page by making tiny changes to the application code. At least one person went away determined to reimplement a number of key applications in Catalyst as soon as possible.

And that, to me, is the joy of running training courses. It’s great to open people’s eyes to the possibilities that these new tools give them. I love to see them leave filled with renewed enthusiasm for the language.

Perl Books

The Perl community on LinkedIn is fascinating. It’s a great way to see how Perl is perceived and used outside of the echo chamber. And that’s a real eye-opener.

Here’s an example. Every few weeks (it seems) someone asks for advice on Perl books.At that point a few people will jump in with sensible suggestions. But for every reasonable suggestion, you’ll get three or four people suggesting something from this list:

  • Something horribly out of date. A lot of publishers stopped updating their Perl books about ten years ago. The most recent version of Perl Black Book that I can find is from 2001. Teach Yourself Perl in 21 Days was last updated in 2002. Perl By Example dates from 2007. I haven’t read any of these books, so I can’t comment on their quality, but I’d be really wary of suggesting that someone learns Perl from a book that is so out of date. Before the days of Amazon, these books would quietly disappear from bookshops, but now they always seem to be available.
  • Something out of date, but that used to be great. There are plenty of books that I would have recommended when they were current, but that I wouldn’t really want to recommend now. A lot of these are O’Reilly books that haven’t been updated recently – Advanced Perl Programming and The Perl Cookbook, for example. I’m sure there’s lots of good stuff in those books. But I really wouldn’t want anyone to read them if they didn’t already have enough experience to update the Perl to current best practices. I sometimes recommend these books myself, but only in specific circumstances. Whenever I run a course on OO Perl I mention Damian’s book. But I also point out that there have been a lot of advances in the area since the book was published. And, I have to confess, my own Data Munging with Perl fits firmly in this category.
  • Something of dubious provenance.  Most of these discussions will, at some point, attract a link to some dubious web site in Eastern Europe that contains the full text of O’Reilly’s various CD bookshelves. I know that these sites exist and I know that there’s nothing that O’Reilly can do about them, But I’d rather not see them mentioned on a professional site like LinkedIn. And quite apart from the copyright issues, there’s the fact that most of the books on these sites fall firmly into the first category above. They’re all over ten years old.
  • Some great tutorial on the internet. I’ve talked about the problems of old and dodgy tutorials before. And things definitely seem to be looking up. We are getting more good tutorials out there. But people still insist on sharing links to the tutorial that they learned from. Even if it’s appalling.

In a recent discussion someone said (and I’m paraphrasing) “just go to Amazon and look for the highest rated books”. I think there are two problems with that:

  1. The people who are rating beginner’s programming books on Amazon are usually the least qualified people to do that. Sure, they can tell how easy the book was to read and how well they picked up what the author taught them. But they have noway of knowing whether what they learned was accurate or useful.
  2. Amazon ratings last forever. But, as noted above, the quality of a technical book in fast-moving subject like programming falls over time. Perhaps Amazon ratings on programming books should have a half-life.

Here are some examples of things that you’re going to miss out on by using outdated Perl references.

  • say – We all love say, don’t we?
  • Lexical filehandles – Storing filehandles in lexical variables is great. I no longer have to worry about bareword filehandles being reused elsewhere in the code.
  • Defined or operator – There’s now no excuse for the $val ||= $default bug.
  • given/when – Perl has a switch statement. And it’s better than anyone else’s :)
  • Unicode support – Perl’s Unicode support is second to none. Except, if you’re using an old version of Perl in which case it’s a bit rubbish. And if you’re reading about Perl in an old book, then you won’t know about it.
  • State variables – Ok, I don’t use those every day. But when I need them, they make my code cleaner.

And then there are all the great CPAN modules that aren’t covered in books that were written before they were released. Would you really want to introduce someone to OO Perl without mentioning Moose?

My rules for recommending books are pretty simple. They should be books that I’ve found useful and they should have been published in the last few years. And given the falling numbers of Perl books that are published each year, that’s now a rather small number of books. Perhaps a dozen or so.

Am I being too harsh? Can beginners get something useful out of older Perl books? How do you decide whether to recommend a book to a colleague?