Misunderstanding Context

Over the last few days I’ve been involved in a discussion on LinkedIn[1]. It has been interesting as it shows how many people still misunderstand many of the intricacies of context and, in particular, how it ties in with the values returned from subroutines.

The original question asked why these two pieces of code acted differently:

The first one gives the first index where the element equals $variable, the second one gives the number of indexes where the element equals $variable.

The answer, of course, comes down to context.  And most people who answered seemed to understand that, but many of their explanations were still way off the mark. Here’s the first answer:

grep returns an array – so the first one will return the value of the first match, but 2nd one in scalar context will return the size of the array, the number of matches. I believe.

The first statement here – “grep returns an array” – is wrong, so any explanation built on that fact is going to be fundamentally flawed.

After a couple of similar answers, I jumped in and pointed out that the only way to know how grep will work in different contexts is to read the documentation; which says:

returns the list value consisting of those elements for which the expression evaluated to true. In scalar context, returns the number of times the expression was true.

At that point it got a bit weird. People started telling me all sorts of strange things in order to show that the earlier answers were better than mine. I was told that lists and arrays are the same thing, that there was something called “array context” and that it was possible for function to return arrays.

I think I’ve worked out what most of the misconceptions in the discussion were. Here’s a list.

1. Arrays are not lists

I know this is a very common misunderstanding. I come across it all the time. People use the terms “list” and “array” interchangeably and end up thinking that they are the same thing. They aren’t. A list is a data value and an array is a variable. They act the same way a lot of the time but unless you understand the difference, you will make mistakes.

Mike Friedman wrote a great blog post that explains the difference in considerable detail. But the core difference is this – arrays are persistent (at least while they are in scope) data structures; lists are ephemeral.

On training courses, I tell people that if they understand the difference between an array and a list they’ll be in the top 20% of Perl programmers. In my experience, that’s pretty close to the truth.

2. Subroutines return lists

A subroutine can only ever return a list. Never an array. It can be an empty list. It can be a list with only one item. But it’s always a list.

There’s one small “gotcha” here. I know this bit me a few times in the past. If you read the documentation for return, it says this:

Evaluation of EXPR may be in list, scalar, or void context, depending on how the return value will be used

This means that when you have code like return @array, it’s @array that is evaluated in the context of the subroutine call. So, depending on  the context of the call, this will return either a list consisting of all the elements in @array, or a list with one element which is the number of elements in @array.

3. There is no “array context”

When you confuse lists and arrays, then it’s not surprising that you might also decide that you can also invent a new context called “array context”. There are only list context, scalar context and void context. Ok, actually there are a few more specialised contexts that you can use (see the Want module for details) but most of the time you’ll only be dealing with those three.

Of course, the name of the wantarray function doesn’t help at all here. But it’s worth noting that the documentation for wantarray ends by saying:

This function should have been named wantlist() instead.

4. You  can’t guess contextual behaviour

One common argument I got when pointing this out on LinkedIn ran along the lines of “but grep acts like it returns an array, so it’s a useful mental model – it helps people to understand context”.

The first part of this is true – grep does act like it returns an array. It returns a list (which you might often store in an array) in list context and it returns the length of that list in scalar context. I can see how you would mistake that for returning an array. But  see my point 2 above. Subroutines do not ever return arrays.

But is it a useful mental model? Does it help people understand how subroutines work in different contexts?

No. It helps people to understand how this particular function works. And there are several other Perl functions that work the same way (keys is one example). But there are plenty of other functions  that don’t work that way. The canonical example is localtime. In list context, it returns a list of nine values; in scalar context it returns a single value (which isn’t the number 9).  Another good example is caller. In list context, it returns a list of items (which can contain either three or eleven elements); in scalar context it returns the first item of that list. There are many more examples.

So the problem with a mental model that assumes that functions work as though they return arrays is that it only works for a subset of functions. And you’d need to waste effort remembering which functions it works for and which ones it doesn’t work for. You’d be far better off (in my opinion) realising that there is no rule that works for all functions and just remembering how each function works (or, more practically, remembering to check the documentation whenever you need to know).

If you’ve seen me giving a lightning talk at a conference this  year, you’ll know that I’ve been encouraging more people from the Perl community to get involved in discussions in places outside the echo chamber, places like LinkedIn. It’s interesting because you see how people outside of the community see Perl. You see the misunderstandings that they work with and you might see how we can improve the Perl documentation to help them get around those misunderstandings.

This morning, this comment was posted to the LinkedIn discussion that I’ve been talking about:

This stuff should be included in a default perl tutorial to avoid common mistake.

And, you know, I think he’s probably right.

[1] Because of the way LinkedIn works, you won’t be able to see this discussion unless you’re a member of LinkedIn and, probably, a member of the Perl group as well.

Two Books

I’ve recently received review copies of a couple of new books. Here are the reviews of those books that I have submitted to Amazon.

Designing the Internet of Things – Adrian McEwen & Hakim Cassimally (Wiley)

I’ve been hearing people talking about “the internet of things” for a few years now. And I’ve always meant to find out what they meant by the term. Well now, thanks to this book, I have a really good idea. I might even think about building something for the internet of things myself (you know, in my copious free time!)

Adrian and Hakim obviously know what they are talking about. They both have experience of working on these kinds of projects. And, crucially, they also have the writing skills to pass their expertise on to the reader.

Some of the stuff in the earlier chapters, I already knew. But things like an introduction to TCP/IP and networking will be useful to many people who don’t have such a technical background. Software, I can do – it was the chapters about hardware that I found most useful. I now know far more than I did about the different toolkits that are available for building internet-connected devices.

Part I is about prototyping your device and part 2 is about building it. I’m sure that other books cover these topics – although, perhaps, not in the focussed way that this book does. But it’s part 3 where this book really shines. These three chapters cover business models, scaling your manufacturing process and some of the ethical issues that these devices raise. This section really makes the book a “one-stop shop” for finding all the information you’ll need to take your vague idea to a complete product and (hopefully) a profitable company.

Perl One-Liners – Peteris Krumins (No Starch)

Perl one-liners are an important part of its power and flexibility. The ability to process a file quickly without having to write a program is often really useful. Any Perl programmers should take the time to get to know the command like switches that make this possible. This book is a pretty good introduction to this way of using Perl.

So why only three stars? Well, I have a couple of reservations about the book. Firstly, there are a few technical errors which the editors should have caught. For example, a few times the author refers to “array context” where he means “list context”. The difference between arrays and lists is often difficult for beginners to master and it doesn’t help when books blur the distinction.

My other reservation is with the programs themselves. The book boasts “130 programs that get things done”. But I think they have had to stretch things a bit to get to that number. One program might be “print lines that match a pattern”. Then the next program will be “print lines that don’t match a pattern”. I’m not sure that inverting the logic in a one-liner is enough of a difference to justify counting it separately. Sometimes you’ll come across two or three pages of examples all of which are only tiny permutations of each other.

But it’s good to see publishers bringing out books on Perl. And this is certainly an area of Perl that hasn’t received much coverage before. I just think it’s a rather thin concept to spin out to a book. Even this stretched, it’s a rather thin book (140 pages – 50 of which are appendices). It might have been better as a cheap Kindle-only publication.

Perl in Banks

An email has flooded in:

I came across your presentation ‘Perl in the Enterprise’ and happen to have a burning concern closely related to one of the bullets ‘Banks see as a competitive advantage’.

I am consulting at a major bank in South Africa, and our team have been using Perl very productively to develop and maintain a customer loyalty rewards  application. This application has now suddenly caught the attention of the ‘Architecture Board’ who are questioning whether Perl (or any other scripting language) is appropriate for production use. A presentation will have to be made to this board to present a business case.

Whatever the basis for this concern, some quotable and referencable stories from other banks, such as the list included in your clients on the website, would be gold for us. I have searched extensively on the web and cannot find anything in this light which is also recent, as in the last 5 years. Other than Nvidia and Booking.com I can only find anecdotal, 2nd hand evidence of Perl use for medium-to-large applications in well-known corporates.

Do you by any chance have anything or any contacts in those banks that would be willing to respond to an email to confirm their use of Perl and perhaps repeat the competitive advantage claim?

The presentation he’s talking about is Perl in the Enterprise, which I gave at Linux World Expo in 2005. The particular line he’s interested in is on slide 6 where I say that banks see using Perl as a competitive advantage. I’m pretty sure that I was paraphrasing Phillip Moore of Morgan Stanley who had said something similar in a keynote at OSCON in about 2001.

Obviously I know that banks in the City of London still use Perl. But it’s been several years since I worked at one, so my personal experience is slightly out of date. What my correspondent needs is people who are currently working in banks who are happy to go public and say “we use Perl in our production systems”. It would be even more helpful if they could add “and here’s why…”.

Can anyone out there help?

Perl Search Revisited

A couple of years ago I wrote a blog post about a Google Custom Search that I had set up to create a specialised search engine for Perl.

Recently I’ve revisited this idea. I’ve given the search engine its own subdomain and I’ve added some new sites to the list of sites that it covers. I’ve also given it a simplified look (thanks Bootstrap) and it’s now being hosted on Github pages.

It lives at http://search.perlhacks.com/. Please give it a try and let me know what you think.

Talks from Kiev

It has only been a few weeks since YAPC::Europe in Kiev and already all of the videos are available on YouTube. Here are the recordings of my three talks.

On the first day I spoke about “25 Years of Perl”.

Later that day I was one of the lightning talk speakers. My talk starts at about 52 minutes.

Then on the second day I spoke about “Matt’s PSGI Archive”.

Parallel Universe Perl 6

Last night was the monthly London Perl Mongers social meeting. I hadn’t been for far too long, but I went last night and enjoyed myself.

The talk was as varied as it always is, but one conversation in particular got me thinking. We were talking about YAPC Europe and someone asked if I had seen the Future Perl Versioning Panel. I said I had and that I was slightly disappointed with the make-up of the panel. In my opinion having three people on the panel who were all strong advocates for Perl 6 remaining Perl 6 didn’t really lead to much of a discussion.

In the end, though, any discussion on this subject is pretty pointless. Larry’s word is law and he has made it very clear that he wants things to remain the way they are. And, of course, any discussion of what might have happened differently if Perl 6 had been given a different name or any of the other alternatives is all completely hypothetical.

But hypothetical discussions can be fun.

So lets turn the discussion round and look at it from a slightly different angle.

Imagine you’re in an alternative universe. One where Jon Orwant never threw those coffee cups and the Perl 6 project was never announced. But also imagine that Perl 5 development in this universe had proceeded along the same lines as it has in our universe (I know that’s unlikely as a lot of Perl 5 development in the last ten years has come out of people wanting to implement ideas from Perl 6 – but let’s ignore that inconvenient fact).

My question to you, then is this…

In this parallel universe, at which point in the last thirteen years of Perl 5 development should we have changed the major version number to 6?

I have an answer to the question, but I’d like to hear some other opinions before sharing it.


If you’ve been reading my blog for a while, you’ll know that I have an interest in packaging CPAN modules as RPMs for Linux distributions based on Red Hat.

For a few years, I’ve been (infrequently) building spreadsheets which list the various modules that are available as RPMs from the better know repositories (and my own small repository). Over the weekend, I thought that those spreadsheets might be more useful if they were turned into web pages. So that’s what I did.

You can see the lists for Fedora and Centos. Over the next few days, I plan to set up cron jobs so that they are rebuilt daily.

Please let me know if you find these lists useful.

Unicode and Perl

Over the last couple of days I’ve been involved in a couple of discussions where it is clear that other people don’t understand how Perl deals with Unicode. The documentation is clear and detailed (there’s even a good tutorial) but for some reason people still persist in misunderstanding it.

Here’s a quick quiz. Can you explain (in detail) what is going on with all of these four command-line programs? And for bonus points, which one should we be emulating in our code?

In all cases, assume that my locale is set to en_US.UTF-8.

I’ll post explanations in a few days time.

Update: Coincidentally, Miyagawa posted something very similar on his blog.

A WordPress Critter

Perl News - Powered by WordPress
Perl News – Powered by WordPress

If you were at YAPC::Europe this week you might well have seen Richard Jelinek’s talk about how to increase Perl’s popularity (update: the slides are here). As part of that talk he suggested that the Perl community needed to run more of its infrastructure using Perl and (amongst other examples) he mentioned a discussion he had with an unnamed Perl News administrator about why Perl News is run using WordPress (which is written in PHP).

I’m happy to admit that I’m the unnamed Perl News administrator. But I think that Richard’s report of our conversation omits a lot of the detail in the points I tried to make to him. So I’d like to take the time to clarify my thoughts on this. There are three points in particular that I’d like to make.

1/ Multitasking

I have a lot of projects on the go. I should probably cut back a bit. Yes, I could start a project to create a WordPress clone in Perl, but that would mean that I would need to shelve a few other projects for some considerable time. The alternative is to quickly build Perl News using an existing tool. You know which option I chose. Of course, you’re free to disagree with me.

2/ Network Effects

Even if there was a capable WordPress replacement written in Perl I probably wouldn’t use it. You see, WordPress isn’t just the software. There’s also a huge community behind it. And that means that there are a huge number of themes and plugins available – with more being released all the time. Every time I want to add a feature to WordPress site, I just find the appropriate plugin and install it. Without that huge community, I would have to implement lots of stuff myself. Which would mean that either I’d be working on Perl News full-time or Perl News would be missing lots of features (for example the social networking hooks).

Don’t forget that I’m also involved in running blogs.perl.org. That runs on Movable Type (which is written in Perl). And have you seen the list of issues that we have with that site?

3/ Patches Welcome

Leo and I built Perl News because we thought it was a useful site for the Perl community to have. You can probably tell from the frequency of updates on the site that it’s not exactly a top priority for either of us. Personally, I’d be very happy if someone else took responsibility for it. So if you think that you can do better, or if you have a Perl system that you think could be used in place of WordPress without any removal of functionality, them please let us know. I’d be really happy to give you a dump of our database (so that you have all the existing stories) and update the DNS to point the domain at your server.

If it matters to you to have Perl web sites running on Perl code, then just go ahead and do it. I would be happy to see it happen. I just don’t have the time to do it myself.

YAPC Europe Approaches

This time next week YAPC Europe will have started and many of us will be enjoying ourselves in Kiev. The organisers published the schedule over the weekend and it looks like it’s going to be a great conference.

I spent some time trying to work out which talks I want to see and I’ve written some suggestions in an article over at Josetteorama. Let me know if I’ve missed something that you’re planning to see.