Monthly Archives: January 2014

The Return of blogs.perl.org

About an hour ago we turned blogs.perl.org back on. There’s also a blog post where we explain what happened in a lot more detail.

If you have an account on the site then you will have received an email explaining what you need to do now. Basically, we’ve invalidated all of the passwords so you’ll need to ask the system for a new one.

Sorry again for the inconvenience. And huge thanks to the rest of the blogs.perl.org team (particularly Aaron Crane) for fixing this.

blogs.perl.org

It seems that last night blogs.perl.org was hacked. I first became aware of it when someone pointed me at this story a few hours ago. As you’ll see, the contents of the mt_author table have been made public.

We’re still investigating the extent of the hack. But, as a precaution, we have configured the site so that all dynamic pages return a 404 response. This will, unfortunately, prevent you from logging on to the site.

We will publish more information when we have it.

Apologies for the inconvenience.

Update:

  • As I said, the mt_author table was leaked
  • This contains both your username and password
  • The password is salted and encrypted (with crypt)
  • If you use your blogs.perl.org password elsewhere, we strongly recommend that you change it

Update 2:
Here’s a cut-down version of the published data that includes only the name columns. Hopefully you can use this to work out whether or not you have an account on the system.

Dots and Perl

I was running a training course this week, and a conversation I had with the class reminded me that I have been planning to write this article for many months.

There are a number of operators in Perl that are made up of nothing but dots. How many of them can you name?

There are actually five dot operators in Perl. If the people on my training courses are any guide, most Perl programmers can only name two or three of them. Here’s a list. It’s in approximate order of how well-known I think the operators are (which is, coincidentally, also the order of increasing length).

One Dot

Everyone knows the one dot operator, right? It’s the concatenation operator. It converts its two operands to strings and concatenates them.

# $string contains 'one stringanother string'
my $string = 'one string' . 'another string';

It’s also sometimes useful to use it to force scalar context onto an expression. Consider the difference between these two statements.

say "Time: ", localtime;
say "Time: " . localtime;

The difference between the two statements is tiny, but the output is very different.

There’s not much more to say about the single dot.

Two Dots

Things start to get a little more interesting when we look at two dots. The two dot operator is actually two different operators depending on how it is used. Most people know that it can be used to generate a range of values.

my @numbers = (1 .. 100);
my @letters = ('a' .. 'z');

You can also use it in something like a for loop. This is an easy way to execute an operation a number of times.

do_something() for 1 .. 3;

In older versions of Perl, this could potentially eat up a lot of memory as a temporary array was created to contain the range and therefore something like

do_something() for 1 .. 1000000;

could be a problem. But in all modern versions of Perl, that temporary array is not created and the expression causes no problems.

Two dots acts as a range operator when it is used in list context. In scalar context, its behaviour is different. It becomes a different operator – the flip-flop operator.

The flip-flop operator is so-called because it flip-flops between two states. It starts as returning false and continues to do so until something “flips” it into its true state. It then continues to return true until something else “flops” it back into a false state. The cycle then repeats.

So what causes it to flip-flop between its two states? It’s the evaluation of its left and right operands. Imagine you are processing a file that contains text that you are interested in and other text that you can ignore. The start of a block that you want to process is marked with a line containing “START” and the end of the block is marked with “END”. Between and “END” marker and the next “START”, there can be lots of text that you want to ignore.

A naive way to process this would involve some kind of “process this” flag.

my $process_this = 0;
while (<$file>) {
  $process_this = 1 if /START/;
  $process_this = 0 if /END/;
  process_this_line($_) if $process_this;
}

I’ve often seen code that looks like this. The author of that code didn’t know about the flip-flop operator. The flip-flop operator encapsulates all of that logic for you. Using the flip-flop operator, the previous code becomes this:

while (<$file>) {
  process_this_line($_) if /START/ .. /END/;
}

The flip-flop operator returns false until its left operand (/START/) evaluates as true. It then returns true until its right operand (/END/) evaluates as true. And then the cycle repeats.

The flip-flop operator has one more trick up its sleeve. One common requirement is to only process certain line numbers in a file (perhaps we just want to process lines 20 to 40). If one of the operands is a constant number, then it is compared the the current record number (in $.) and the operand is fired if the line number matches. So processing lines 20 to 40 of a file is a simple as:

while (<$file>) {
  process_this_line($_) if 20 .. 40;
}

Three Dots

It was the three dot operator that triggered the conversation which reminded me to write this article. A three dot operator was added in Perl 5.12. It’s called the “yada-yada” operator. It is used to stand for unimplemented code.

sub reverse_polarity {
  # TODO: Must write this
  ...
}

Programmers have been leaving “TODO” comments in their code for decades. And they’ve been using ellipsis (a.k.a. three dots) to signify unimplemented code for just as long. But now you can actually put three dots in your source code and it will compile and run. So what’s the benefit of this over just leaving a TODO comment? Well, what happens if you call a function that just contains a comment? The function executes and does nothing. You might not realise that you haven’t implemented that function yet. With the yada-yada operator standing in for the unimplemented code, Perl throws an “unimplemented” error and you are reminded that you need to write that code before calling it.

But the yada-yada operator wasn’t the first three dot operator in Perl. There has been another one in the language for a very long time (you might say since the year dot!) And I bet very few of you know what it is.

The original three dot operator is another flip-flop operator. And the difference between it and the two dot version is subtle. It’s all to do with how many tests can be run against the same line. With the two dot version, when the operator flips to true it also checks the right-hand operand in the same iteration – meaning that it can flip from false to true and back to false again as part of the same iteration. If you use the three dot version, then once the operator flips to true, then it won’t check the right-hand operand until the next iteration – meaning that the operator can only flip once per iteration.

When does this matter? Well if our text file sometimes contains empty records that contain START and END on the same line, the difference between the two and three dot flip-flops will determine exactly which lines are processed.

If the two dot version encounters one of these empty records, it flips to true (because it matches /START/) and then flops back to false (because it matches (/END/). However, the flop to false doesn’t happen until after the expression returns a value (which is true). The net effect, therefore is that the line is printed, but the flip-flop is left in the false state and the following lines won’t be printed until one contains a START.

If the three dot version encounters one of these empty records, it also flips to true (because it matches /START/) but then doesn’t check the right-hand operand. So the line gets printed and the flip-flop remains in its true state so the following lines will continue to be printed until one contains an end.

Which is correct? Well, of course, it depends on your requirements. In this case, I expect that the two dot version gives the results that most people would expect. But the three dot version is also provided for the cases when its behaviour is required. As always, Perl gives you the flexibility to do exactly what you want.

But I suspect that the relatively small number of people who seem to know about the three dot flip-flip would indicate that its behaviour isn’t needed very often.

So, there you have them. Perl’s five dot operators – concatenation, range, flip-flop, yada-yada and another flip-flop. I hope that helps you impress people in your next Perl job interview.

Update: dakkar points out that yada-yada isn’t actually an operator. He’s absolutely right, of course, it stands in place of a statement and has no operands. But being slightly loose with our terminology here makes for a more succinct article.

Perl APIs

For a lot of programmers out there, Perl has become largely invisible. They just never come across it. That might seem strange to you as you sit inside the Perl community echo chamber reading the Perl Ironman or p5p, but try this simple experiment.

Think of a web site that you use and that supplies an API. Now go to that API’s documentation and look at the example code. What languages are the examples written in? PHP? Almost certainly. Ruby? Probably. Python? Probably. C#? Quite possibly. Perl? Almost certainly not[1].

Perl has fallen so far off the radar of most people that when web sites write example code for these APIs, they very rarely consider Perl as a language worth including. And because they don’t bother including Perl then any random programmer coming to that API will assume that the API doesn’t support Perl, or (at the very least) that the lack of examples will make using Perl harder than it would be with plenty of examples to copy from.

This then becomes a self-fulfilling prophecy as Github fills with more and more projects using other languages to talk to these APIs. And the chance of anyone who isn’t already a Perl user ever trying to interact with these APIs using Perl falls and falls.

Of course, this is all completely wrong. These APIs are just going to be a series of HTTP requests using REST or XML-RPC (or, if you’re really unlucky, SOAP). Perl has good support for all of that. You might need to use something like OAuth to get access to the API – well Perl does that too.

Of course, in some cases good Perl support exists already – Net::Twitter is a good example. And to be fair to Twitter, their API documentation doesn’t seem to give any examples in specific languages – so Perl isn’t excluded here. But in many other cases, the Perl version languishes unnoticed on CPAN while other languages get mentioned on the API page.

I think that we can try to address this in 2014. And I’d like to ask you to help me. I’ve set up a mailing list called perl-api-squad where we can discuss this. In a nutshell, I think that the plan should be something like this:

  1. Identify useful APIs where there is either no Perl API or no Perl examples
  2. Write CPAN API wrappers where they are missing
  3. Approve API owners and offer them Perl examples to add to their web site

That doesn’t sound too complicated to me. And I think (or, perhaps, hope) that most API owners will be grateful to add more examples of API usage to their site – particularly if it involves next to no effort on their part.

I also expect that the Perl API Squad will produce a web site that lists Perl API support. We might even move towards producing a framework that makes it easy to write a basic Perl wrapper around any new API.

What do you think? Is this a worthwhile project? Who’s interested in joining in?

[1] Yes, I know there are exceptions. But they are just that – exceptions.