Misunderstanding Context

Over the last few days I’ve been involved in a discussion on LinkedIn[1]. It has been interesting as it shows how many people still misunderstand many of the intricacies of context and, in particular, how it ties in with the values returned from subroutines.

The original question asked why these two pieces of code acted differently:

The first one gives the first index where the element equals $variable, the second one gives the number of indexes where the element equals $variable.

The answer, of course, comes down to context.  And most people who answered seemed to understand that, but many of their explanations were still way off the mark. Here’s the first answer:

grep returns an array – so the first one will return the value of the first match, but 2nd one in scalar context will return the size of the array, the number of matches. I believe.

The first statement here – “grep returns an array” – is wrong, so any explanation built on that fact is going to be fundamentally flawed.

After a couple of similar answers, I jumped in and pointed out that the only way to know how grep will work in different contexts is to read the documentation; which says:

returns the list value consisting of those elements for which the expression evaluated to true. In scalar context, returns the number of times the expression was true.

At that point it got a bit weird. People started telling me all sorts of strange things in order to show that the earlier answers were better than mine. I was told that lists and arrays are the same thing, that there was something called “array context” and that it was possible for function to return arrays.

I think I’ve worked out what most of the misconceptions in the discussion were. Here’s a list.

1. Arrays are not lists

I know this is a very common misunderstanding. I come across it all the time. People use the terms “list” and “array” interchangeably and end up thinking that they are the same thing. They aren’t. A list is a data value and an array is a variable. They act the same way a lot of the time but unless you understand the difference, you will make mistakes.

Mike Friedman wrote a great blog post that explains the difference in considerable detail. But the core difference is this – arrays are persistent (at least while they are in scope) data structures; lists are ephemeral.

On training courses, I tell people that if they understand the difference between an array and a list they’ll be in the top 20% of Perl programmers. In my experience, that’s pretty close to the truth.

2. Subroutines return lists

A subroutine can only ever return a list. Never an array. It can be an empty list. It can be a list with only one item. But it’s always a list.

There’s one small “gotcha” here. I know this bit me a few times in the past. If you read the documentation for return, it says this:

Evaluation of EXPR may be in list, scalar, or void context, depending on how the return value will be used

This means that when you have code like return @array, it’s @array that is evaluated in the context of the subroutine call. So, depending on  the context of the call, this will return either a list consisting of all the elements in @array, or a list with one element which is the number of elements in @array.

3. There is no “array context”

When you confuse lists and arrays, then it’s not surprising that you might also decide that you can also invent a new context called “array context”. There are only list context, scalar context and void context. Ok, actually there are a few more specialised contexts that you can use (see the Want module for details) but most of the time you’ll only be dealing with those three.

Of course, the name of the wantarray function doesn’t help at all here. But it’s worth noting that the documentation for wantarray ends by saying:

This function should have been named wantlist() instead.

4. You  can’t guess contextual behaviour

One common argument I got when pointing this out on LinkedIn ran along the lines of “but grep acts like it returns an array, so it’s a useful mental model – it helps people to understand context”.

The first part of this is true – grep does act like it returns an array. It returns a list (which you might often store in an array) in list context and it returns the length of that list in scalar context. I can see how you would mistake that for returning an array. But  see my point 2 above. Subroutines do not ever return arrays.

But is it a useful mental model? Does it help people understand how subroutines work in different contexts?

No. It helps people to understand how this particular function works. And there are several other Perl functions that work the same way (keys is one example). But there are plenty of other functions  that don’t work that way. The canonical example is localtime. In list context, it returns a list of nine values; in scalar context it returns a single value (which isn’t the number 9).  Another good example is caller. In list context, it returns a list of items (which can contain either three or eleven elements); in scalar context it returns the first item of that list. There are many more examples.

So the problem with a mental model that assumes that functions work as though they return arrays is that it only works for a subset of functions. And you’d need to waste effort remembering which functions it works for and which ones it doesn’t work for. You’d be far better off (in my opinion) realising that there is no rule that works for all functions and just remembering how each function works (or, more practically, remembering to check the documentation whenever you need to know).

If you’ve seen me giving a lightning talk at a conference this  year, you’ll know that I’ve been encouraging more people from the Perl community to get involved in discussions in places outside the echo chamber, places like LinkedIn. It’s interesting because you see how people outside of the community see Perl. You see the misunderstandings that they work with and you might see how we can improve the Perl documentation to help them get around those misunderstandings.

This morning, this comment was posted to the LinkedIn discussion that I’ve been talking about:

This stuff should be included in a default perl tutorial to avoid common mistake.

And, you know, I think he’s probably right.

[1] Because of the way LinkedIn works, you won’t be able to see this discussion unless you’re a member of LinkedIn and, probably, a member of the Perl group as well.

Parallel Universe Perl 6

Last night was the monthly London Perl Mongers social meeting. I hadn’t been for far too long, but I went last night and enjoyed myself.

The talk was as varied as it always is, but one conversation in particular got me thinking. We were talking about YAPC Europe and someone asked if I had seen the Future Perl Versioning Panel. I said I had and that I was slightly disappointed with the make-up of the panel. In my opinion having three people on the panel who were all strong advocates for Perl 6 remaining Perl 6 didn’t really lead to much of a discussion.

In the end, though, any discussion on this subject is pretty pointless. Larry’s word is law and he has made it very clear that he wants things to remain the way they are. And, of course, any discussion of what might have happened differently if Perl 6 had been given a different name or any of the other alternatives is all completely hypothetical.

But hypothetical discussions can be fun.

So lets turn the discussion round and look at it from a slightly different angle.

Imagine you’re in an alternative universe. One where Jon Orwant never threw those coffee cups and the Perl 6 project was never announced. But also imagine that Perl 5 development in this universe had proceeded along the same lines as it has in our universe (I know that’s unlikely as a lot of Perl 5 development in the last ten years has come out of people wanting to implement ideas from Perl 6 – but let’s ignore that inconvenient fact).

My question to you, then is this…

In this parallel universe, at which point in the last thirteen years of Perl 5 development should we have changed the major version number to 6?

I have an answer to the question, but I’d like to hear some other opinions before sharing it.

Unicode and Perl

Over the last couple of days I’ve been involved in a couple of discussions where it is clear that other people don’t understand how Perl deals with Unicode. The documentation is clear and detailed (there’s even a good tutorial) but for some reason people still persist in misunderstanding it.

Here’s a quick quiz. Can you explain (in detail) what is going on with all of these four command-line programs? And for bonus points, which one should we be emulating in our code?

In all cases, assume that my locale is set to en_US.UTF-8.

I’ll post explanations in a few days time.

Update: Coincidentally, Miyagawa posted something very similar on his blog.

Just Build Something

The Political Web

About a month ago, JT Smith suggested that we should all stop talking about Perl and just build something. And, purely coincidentally, over the last few weeks I resurrected a project that I have been poking at for about five years and have finally turned it into something that I’m happy to show the world.

The Political Web is a site which aggregates all of the information I can find on the web about individual British MPs. I say “all of the information”, but that’s obviously a bit of a work in progress. But I think that what I already have is useful and interesting – well, for people who are interested in British politics. I have plans to bring in more information in the future.

Although I’ve been working on the site for five years, I pretty much rebuilt it from scratch when I recently returned to it. Actually getting something useful up and running took about four hours. That’s because I was building it using Perl and, more specifically, Dancer.

Pebble and Perl

I’ve been wearing a Pebble watch for a couple of months now. I really like it but, to be honest, it’s the potential that has me most excited. The number of apps currently available is a bit disappointing and the API is taking its time appearing.

But even when the API is published, I wonder if I’ll have the time to learn all the necessary technologies in order to write Pebble-aware apps. All I want is to have some way to send a notification that pops up on the phone. Surely there must be an easy way to achieve that. Preferably one that I can use from Perl.

Of course, it turns out that there is. The secret is an app called Pushover. Pushover is a web service that sends notifications to your Android (or iOS if you’re that way inclined) device. There’s an app that you install on your device (it’s not free – I think it cost me £3.30) and you need to sign up for a free account. Then you can send notifications to your device either from their web site or using their API. The API is a simple HTTP request-based system. There’s an example on the Pushover web site that uses LWP::UserAgent, but you can make it even simpler using WebService::Pushover.

That’s all well and good. We can now send arbitrary notification from to our phone. How do we get from there to the watch?

The standard Pebble Android app is currently a little disappointing. In particular, it only supports pushing notifications from a tiny number of apps from the phone to the watch. But there’s another alternative. There’s an app called Pebble Notifier which will forward notifications from any app on the phone to the watch. When you install Pebble Notifier you can choose which apps you want to forward notifications for.

So, in summary, sign up for a Pushover account and install Pushover and Pebble Notifier on your phone. Then install WebService::Pushover on your computer. Then you can write code like this:

And that will send a message to your Pebble.

Now all I need is something useful to do with it.

Update: I’ve just noticed that there’s also an IFTTT channel for Pushover. It has nothing to do with Perl, obviously, but would be an easy way to trigger Pebble notifications for certain triggers.