Tag Archives: programming

“I Do Not Want To Use Any Modules”

Almost every day on the Perl groups on LinkedIn (or Facebook, or StackOverflow, or somewhere like that) I see a question that includes the restriction “I do not want to use any modules”.

There was one on LinkedIn yesterday. He wanted to create a MIME message to pass to sendmail, but he didn’t want to install any modules. Because “getting a module installed will have to go though a long long process of approvals”.

And I understand that. I really do. We’ve all seen places where getting new software installed is a problem. But I see that problem as a bug in the development process. A bug that needs to be fixed before anything can get done in a reasonable manner. Here’s what I’ve just written in reply:

Of course it can be achieved without modules. Just create an email in the correct format and pass it to sendmail.

Ah, but what’s the right format? Well, that is (of course) the tricky bit. I have no idea what the correct format is. Oh, I could Google a bit and come up with some ideas. I might even find the RFC that defines the MIME format. And then I’d be able to knock up some code that created something that looked like it would work. But would I be sure that it works? In every case? With all the weird corner-cases that people might throw at it?

This is where CPAN modules come in handy. You’re using someone else’s knowledge. Someone who is (hopefully) an expert in the field. And because modules are used by lots of people, bugs get found and fixed.

A lot of modern Perl programming is about choosing the right set of CPAN modules and plumbing them together. That’s what makes Perl so powerful. That’s what makes Perl programmers so efficient. We’re standing on the shoulders of giants and re-using other people’s code.

If you’re not going to use CPAN then you might as well use shell-scripting or awk.

If you’re in a situation where getting CPAN modules installed is hard, then fixing that problem should be your first priority. Because that’s a big impediment to your Perl programming. And investing time in fixing that will be massively beneficial to you in a very short amount of time.

The obvious solution is to install your own module tree (alongside your own Perl) as part of your application. But that might be overkill in some situations, so you could also consider using the system Perl and asking your sysadmin to install packages from your distribution’s repositories. Of course, that might need a change in process. But it’s a change that is well worth making; a change that will improve your (programming) life immensely.

Programming Language Usage

Back in May, I spent an afternoon at Silicon MilkRoundabout. Silicon MilkRoundabout is a recruitment fair for techies. It’s specifically aimed at people who want to work for start-ups around the Old Street area (although they aren’t particularly stringent about sticking to that – for example, the BBC were there).

We were given a booklet containing details of all of the companies who were recruiting. Those details usually included information about the tech stack that the companies used.

Over the weekend, I went through that booklet and listed the programming languages mentioned by the companies. The results speak for themselves.

There were 135 companies at the event. About twenty of them unhelpfully listed their tech stack as “ask us for details”.

Here’s the graph:Usage of Programming Languages by Companies at Silicon MilkRoundabout

Usage of Programming Languages by Companies at Silicon MilkRoundabout

I’ll obviously have some more to say about this over the next few days. But I wanted to get the raw data out there as soon as possible.

Perl APIs

For a lot of programmers out there, Perl has become largely invisible. They just never come across it. That might seem strange to you as you sit inside the Perl community echo chamber reading the Perl Ironman or p5p, but try this simple experiment.

Think of a web site that you use and that supplies an API. Now go to that API’s documentation and look at the example code. What languages are the examples written in? PHP? Almost certainly. Ruby? Probably. Python? Probably. C#? Quite possibly. Perl? Almost certainly not[1].

Perl has fallen so far off the radar of most people that when web sites write example code for these APIs, they very rarely consider Perl as a language worth including. And because they don’t bother including Perl then any random programmer coming to that API will assume that the API doesn’t support Perl, or (at the very least) that the lack of examples will make using Perl harder than it would be with plenty of examples to copy from.

This then becomes a self-fulfilling prophecy as Github fills with more and more projects using other languages to talk to these APIs. And the chance of anyone who isn’t already a Perl user ever trying to interact with these APIs using Perl falls and falls.

Of course, this is all completely wrong. These APIs are just going to be a series of HTTP requests using REST or XML-RPC (or, if you’re really unlucky, SOAP). Perl has good support for all of that. You might need to use something like OAuth to get access to the API – well Perl does that too.

Of course, in some cases good Perl support exists already – Net::Twitter is a good example. And to be fair to Twitter, their API documentation doesn’t seem to give any examples in specific languages – so Perl isn’t excluded here. But in many other cases, the Perl version languishes unnoticed on CPAN while other languages get mentioned on the API page.

I think that we can try to address this in 2014. And I’d like to ask you to help me. I’ve set up a mailing list called perl-api-squad where we can discuss this. In a nutshell, I think that the plan should be something like this:

  1. Identify useful APIs where there is either no Perl API or no Perl examples
  2. Write CPAN API wrappers where they are missing
  3. Approve API owners and offer them Perl examples to add to their web site

That doesn’t sound too complicated to me. And I think (or, perhaps, hope) that most API owners will be grateful to add more examples of API usage to their site – particularly if it involves next to no effort on their part.

I also expect that the Perl API Squad will produce a web site that lists Perl API support. We might even move towards producing a framework that makes it easy to write a basic Perl wrapper around any new API.

What do you think? Is this a worthwhile project? Who’s interested in joining in?

[1] Yes, I know there are exceptions. But they are just that – exceptions.

Misunderstanding Context

Over the last few days I’ve been involved in a discussion on LinkedIn[1]. It has been interesting as it shows how many people still misunderstand many of the intricacies of context and, in particular, how it ties in with the values returned from subroutines.

The original question asked why these two pieces of code acted differently:

my ($index) = grep {
  $array[$_] == $variable
} 0 .. $#array;
my $index = grep {
  $array[$_] == $variable
} 0 .. $#array;

The first one gives the first index where the element equals $variable, the second one gives the number of indexes where the element equals $variable.

The answer, of course, comes down to context.  And most people who answered seemed to understand that, but many of their explanations were still way off the mark. Here’s the first answer:

grep returns an array – so the first one will return the value of the first match, but 2nd one in scalar context will return the size of the array, the number of matches. I believe.

The first statement here – “grep returns an array” – is wrong, so any explanation built on that fact is going to be fundamentally flawed.

After a couple of similar answers, I jumped in and pointed out that the only way to know how grep will work in different contexts is to read the documentation; which says:

returns the list value consisting of those elements for which the expression evaluated to true. In scalar context, returns the number of times the expression was true.

At that point it got a bit weird. People started telling me all sorts of strange things in order to show that the earlier answers were better than mine. I was told that lists and arrays are the same thing, that there was something called “array context” and that it was possible for function to return arrays.

I think I’ve worked out what most of the misconceptions in the discussion were. Here’s a list.

1. Arrays are not lists

I know this is a very common misunderstanding. I come across it all the time. People use the terms “list” and “array” interchangeably and end up thinking that they are the same thing. They aren’t. A list is a data value and an array is a variable. They act the same way a lot of the time but unless you understand the difference, you will make mistakes.

Mike Friedman wrote a great blog post that explains the difference in considerable detail. But the core difference is this – arrays are persistent (at least while they are in scope) data structures; lists are ephemeral.

On training courses, I tell people that if they understand the difference between an array and a list they’ll be in the top 20% of Perl programmers. In my experience, that’s pretty close to the truth.

2. Subroutines return lists

A subroutine can only ever return a list. Never an array. It can be an empty list. It can be a list with only one item. But it’s always a list.

There’s one small “gotcha” here. I know this bit me a few times in the past. If you read the documentation for return, it says this:

Evaluation of EXPR may be in list, scalar, or void context, depending on how the return value will be used

This means that when you have code like return @array, it’s @array that is evaluated in the context of the subroutine call. So, depending on  the context of the call, this will return either a list consisting of all the elements in @array, or a list with one element which is the number of elements in @array.

3. There is no “array context”

When you confuse lists and arrays, then it’s not surprising that you might also decide that you can also invent a new context called “array context”. There are only list context, scalar context and void context. Ok, actually there are a few more specialised contexts that you can use (see the Want module for details) but most of the time you’ll only be dealing with those three.

Of course, the name of the wantarray function doesn’t help at all here. But it’s worth noting that the documentation for wantarray ends by saying:

This function should have been named wantlist() instead.

4. You  can’t guess contextual behaviour

One common argument I got when pointing this out on LinkedIn ran along the lines of “but grep acts like it returns an array, so it’s a useful mental model – it helps people to understand context”.

The first part of this is true – grep does act like it returns an array. It returns a list (which you might often store in an array) in list context and it returns the length of that list in scalar context. I can see how you would mistake that for returning an array. But  see my point 2 above. Subroutines do not ever return arrays.

But is it a useful mental model? Does it help people understand how subroutines work in different contexts?

No. It helps people to understand how this particular function works. And there are several other Perl functions that work the same way (keys is one example). But there are plenty of other functions  that don’t work that way. The canonical example is localtime. In list context, it returns a list of nine values; in scalar context it returns a single value (which isn’t the number 9).  Another good example is caller. In list context, it returns a list of items (which can contain either three or eleven elements); in scalar context it returns the first item of that list. There are many more examples.

So the problem with a mental model that assumes that functions work as though they return arrays is that it only works for a subset of functions. And you’d need to waste effort remembering which functions it works for and which ones it doesn’t work for. You’d be far better off (in my opinion) realising that there is no rule that works for all functions and just remembering how each function works (or, more practically, remembering to check the documentation whenever you need to know).

 

If you’ve seen me giving a lightning talk at a conference this  year, you’ll know that I’ve been encouraging more people from the Perl community to get involved in discussions in places outside the echo chamber, places like LinkedIn. It’s interesting because you see how people outside of the community see Perl. You see the misunderstandings that they work with and you might see how we can improve the Perl documentation to help them get around those misunderstandings.

This morning, this comment was posted to the LinkedIn discussion that I’ve been talking about:

This stuff should be included in a default perl tutorial to avoid common mistake.

And, you know, I think he’s probably right.

[1] Because of the way LinkedIn works, you won’t be able to see this discussion unless you’re a member of LinkedIn and, probably, a member of the Perl group as well.

Parallel Universe Perl 6

Last night was the monthly London Perl Mongers social meeting. I hadn’t been for far too long, but I went last night and enjoyed myself.

The talk was as varied as it always is, but one conversation in particular got me thinking. We were talking about YAPC Europe and someone asked if I had seen the Future Perl Versioning Panel. I said I had and that I was slightly disappointed with the make-up of the panel. In my opinion having three people on the panel who were all strong advocates for Perl 6 remaining Perl 6 didn’t really lead to much of a discussion.

In the end, though, any discussion on this subject is pretty pointless. Larry’s word is law and he has made it very clear that he wants things to remain the way they are. And, of course, any discussion of what might have happened differently if Perl 6 had been given a different name or any of the other alternatives is all completely hypothetical.

But hypothetical discussions can be fun.

So lets turn the discussion round and look at it from a slightly different angle.

Imagine you’re in an alternative universe. One where Jon Orwant never threw those coffee cups and the Perl 6 project was never announced. But also imagine that Perl 5 development in this universe had proceeded along the same lines as it has in our universe (I know that’s unlikely as a lot of Perl 5 development in the last ten years has come out of people wanting to implement ideas from Perl 6 – but let’s ignore that inconvenient fact).

My question to you, then is this…

In this parallel universe, at which point in the last thirteen years of Perl 5 development should we have changed the major version number to 6?

I have an answer to the question, but I’d like to hear some other opinions before sharing it.

Unicode and Perl

Over the last couple of days I’ve been involved in a couple of discussions where it is clear that other people don’t understand how Perl deals with Unicode. The documentation is clear and detailed (there’s even a good tutorial) but for some reason people still persist in misunderstanding it.

Here’s a quick quiz. Can you explain (in detail) what is going on with all of these four command-line programs? And for bonus points, which one should we be emulating in our code?

$ perl -E'say "£"'
£
$ perl -Mutf8 -E'say "£"'
�
$ perl -C -E'say "£"'
£
$ perl -C -Mutf8 -E'say "£"'
£

In all cases, assume that my locale is set to en_US.UTF-8.

I’ll post explanations in a few days time.

Update: Coincidentally, Miyagawa posted something very similar on his blog.

Just Build Something

The Political Web

About a month ago, JT Smith suggested that we should all stop talking about Perl and just build something. And, purely coincidentally, over the last few weeks I resurrected a project that I have been poking at for about five years and have finally turned it into something that I’m happy to show the world.

The Political Web is a site which aggregates all of the information I can find on the web about individual British MPs. I say “all of the information”, but that’s obviously a bit of a work in progress. But I think that what I already have is useful and interesting – well, for people who are interested in British politics. I have plans to bring in more information in the future.

Although I’ve been working on the site for five years, I pretty much rebuilt it from scratch when I recently returned to it. Actually getting something useful up and running took about four hours. That’s because I was building it using Perl and, more specifically, Dancer.

Give Me MetaCPAN

Ever since MetaCPAN launched I’ve been getting increasingly irritated with people who still use links to search.cpan.org. Isn’t it obvious that MetaCPAN is better? Why do people still insist on sharing links to the older site?

Of course they do it for various reasons. Perhaps they aren’t as in touch with the modern Perl world as I am. Perhaps they are wary about changing to use the new shiny toys because they know that a newer shinier one will be along soon. Perhaps I’m reading a web page from five years ago and they can be forgiven for not linking to a site that didn’t exist at the time.

Eventually I realised that there was no point in getting annoyed. I had a computer. Surely I could do something that would fix this problem.

My first idea was to write a GreaseMonkey script. Then I realised that would be hard. MetaCPAN and CPAN have slightly different URL schemes. I’d need to recognise a CPAN URL and convert it to the equivalent MetaCPAN URL. Not an impossible task at all, but not something I could knock up in an hour or so. Especially not in Javascript.

So I asked on the #metacpan IRC channel. Surely I couldn’t be the first person to have this problem. And someone there introduced me to mcpan.org. There’s no point in clicking on that link. You’ll just end up on MetaCPAN. Because that’s what mcpan does. It’s a URL rewriting service. You give it a CPAN URL (with the cpan.org changed to mcpan.org) and it redirects you to the equivalent page on MetaCPAN.

That’s the hard bit of the problem. The bit I didn’t want to write. And someone else has already written it. But they seem to have kept it very quiet. This deserves more publicity. I wish I could remember who wrote it. If you know, please leave a comment.

So we’re now most of the way there. Now if I click on a CPAN link, I can just edit the location bar to add an ‘m’ and I’ll be redirected to the right place. But we can do better than that. I’d like to be automatically redirected. That’s when I discovered the Redirector extension for Firefox. Once it’s installed you can configure it to redirect certain URLs to other ones. I have it configured so that http://search.cpan.org* redirects to http://mcpan.org$1. See how you can use wildcards to match URLs and then use whatever they matched in the replacement URL. It’s a lot like regexes in Perl.

And we’re done. Now whenever I click on an old-style CPAN link, I’m automatically redirected to mcpan, And that, in turn, redirects me to MetaCPAN. And, best of all, I didn’t have to write any code. It was just a case of putting together tools that already existed.

I did all this a few months ago. I meant to write this blog post at the time, but I forgot. I was reminded this morning when Chisel mentioned a GreaseMonkey script he had written to do all of this. See, Chisel isn’t afraid of parsing URLs in Javascript like I was. He just went ahead and did it.

But having alternative solutions to the problem is good, right?

Update: I’ve just been talking about this on the #metacpan IRC channel and it seems that I was rather misunderstanding what was going on here. Here are the details.

Firstly, it was dpetrov who told me about mcpan.org. It’s his domain. I asked him for the code, and he pointed out that there is no code. mcpan.org just redirects everything to a domain called sco.metacpan.org which is where the magic happens. He just got tired of editing search.cpan.org to sco.metacpan.org so he registered another, simpler, domain.

So the actual cleverness happens over on sco.metacpan.org. And that’s really just a list of rewrite rules (the code is on github). I say “just”, but I still wouldn’t want to write them myself.

All this means that mcpan,org is only a convenient tool for when you’re manually editing your location bar. When you’re using the Redirector extension for Firefox you can miss out the middle man and redirect straight to sco.metacpan.org. So I’ve updated my Redirector rule appropriately.

Why Corporates Hate Perl

This is a reprint of an old blog post.

A few years ago I was writing blog posts (semi-)regularly for O’Reilly. This is the one that probably got the most feedback. I’m reprinting it now because a) it’s pretty hard to find on the O’Reilly site and b) it’s relevant to a couple of conversations that I’ve had over the last few days.

Last week I was in Copenhagen for YAPC::Europe. One of the announcements at the conference was the location of next year’s conference which will be in Lisbon. The theme of next year’s conference will be “Corporate Perl”. And that (along with a couple of conversations last night) got me thinking about a talk that I’ll submit to next year’s conference which might well be entitled “Why Corporates Hate Perl”.

It’s not true, of course. There are a still large number of large companies who love Perl. I could probably work through to my retirement enhancing and extending systems that are written in Perl at many of the big banks in the City of London. There are, however, also many companies who are moving away from Perl for a number of reasons. Here’s one of the reasons that will be included in my talk.

I was talking to people from one such company last night. The Powers That Be at this company have announced that Perl is no longer their language of choice for web systems and that over time (probably a lot of time) systems will be rewritten in a combination of Java and PHP. Management have started to refer to Perl-based systems as “legacy” and to generally disparage it. This attitude has seeped through to non-technical business users who have started to worry if developers mention a system that is written in Perl. Business users, of course, don’t want nasty old, broken Perl code. They want the shiny new technologies.

And so, in a matter of months, the technical managers at this company have created a business environment where Perl is seen as the cause of most of the problems with the current systems. It’s an impressive piece of social engineering.

It’s also, of course, completely unfair. I don’t deny at all that this company (like many others) has a large amount of badly written and hard to maintain Perl code. But I maintain that this isn’t directly due to the code being written in Perl. It’s because the Perl code has developed piecemeal over the last ten or so years in an environment where there was no design authority which encouraged developers to think beyond getting their immediate task done. Many of these systems date back to this company’s first steps onto the internet and were made by separate departments who had no interaction with each other. It’s not really a surprise that the systems don’t interact well and a lot of the code is hard to maintain.

There are, on the other hand, a number of newer systems which are also written in Perl which follow current best practices in Perl development and are far easier to to maintain and enhance – as easy, I would contend, as anything written in the new approved languages.

It’s certainly true that this company has a large number of systems that need to be rewritten over the next few years. But throwing away all of the company’s accumulated Perl expertise and moving to new languages seems to be a step too far. Management are blaming Perl for the problems when really they should be blaming the management and design procedures that were in place (or, more likely, weren’t in place) when the code was originally written.

Many organisations are in the same situation, with large amounts of unwieldy Perl code. Ten or twelve years ago everyone was writing web systems in Perl and we were all making mistakes. We all have to deal with those mistakes but we’ve  hopefully, learned from them and can rewrite our systems to take account of everything that we’ve learned in the last ten years.

It’s too late for the company I’ve been talking about in this article. The anti-Perl social engineering has probably insinuated itself too deeply into the culture. It’s unlikely that Perl’s reputation can be rescued.

But if you have similar problems in your own company, then please try to ensure that blame is apportioned correctly and that you don’t use Perl as a scapegoat.

A couple of updates to the post. I did propose the talk to the next YAPC, but the proposal wasn’t accepted. And the company I talk about in the article is still employing a lot of Perl programmers – four years after this post was written.

 

DBIC vs DBI

Three times in the last few months I’ve had the “DBIC or raw DBI” discussion. People have told me that they don’t use DBIC because raw DBI is better. And each time, the person promoting DBI in the discussion has used an argument that boils down to “DBIC is probably useful for people who don’t know SQL very well”.

I find that argument really puzzling. Not least because I like to think that I know more than a little about SQL. SQL is a skill that that has run through my career for longer than Perl. I’ve been using SQL since I left college in 1988. I only started using Perl in about 1996. And yet, although I still consider myself a bit of an expert in SQL, I use DBIC for pretty much all of the database work I do these days – and have done for about five years.

I use DBIC not because I don’t understand SQL. I use it because it makes my life easier. I use it because it frees up some of the time I used to spend dealing with the minutia of database communication so that I can spend it working on other, more interesting, parts of my project.

When I’m running training courses that introduce DBIC I have slide that is entitled “SQL is Boring”. It’s a joke of course but, like all the best jokes, it gets a laugh because there’s more than a little truth in it. There are, of course, many interesting SQL problems. I’ve spent many an enjoyable (if slightly frustrating at the time) hour trying to coax the right data out of complex query with correlated subqueries, outer joins and aggregated functions. But that’s the exception rather than the rule.

The vast majority of the SQL I write for the applications I work on is incredibly boring. It’s  boring because it’s all so similar. You get the data to present a list of objects to the user. The user selects the object they’re interested in, so you select all the data about that object. You might select some data about related objects. The user changes some of that data, so you update that row in the database. On a good day, you might delete an object from the database. Or insert a new one. Most of the SQL you need is like that. It’s boring.

We have computers to do the boring work for us. So let the computer generate all that boring SQL. Free up your time to work on the gnarly and interesting problems.

But that’s not the only advantage of using DBIC (or some other ORM). Think about the data that you get back from the database. The data you get back from a DBI call is an array. Or perhaps a hash. Or maybe a multi-dimension data structure if you’re using one of DBI’x more complex fetch() methods. But it’s still a dumb variable. From DBIC, I get an object. An intelligent variable. A variable that knows how to react to various messages. A variable that will save any changes back to the database automatically without me having to worry about where it came from and making sure that I’m writing it back to the right place.

I’m not saying for a second that there’s no place for DBI any more. For a start, DBI underlies DBIC so it’s still a vital part of our toolkit. And of course I still use it for quick one-off scripts. But when those scripts are still hanging round being used and maintained three months later (as they always are) I’ll rewrite them to use DBIC.

If you want to write large applications that are going to be easy to maintain and extend, then you should really be using DBIC (or something similar). I don’t care how well you know SQL. DBIC will make your life easier.

I’ve just mentioned a couple of reasons why I think that DBIC makes my life easier. I’m sure I’ve missed important stuff. What do you think? Why do you use DBIC instead of DBI?