Easy PSGI

When I write replies to questions on StackOverflow and places like that recommending that people abandon CGI programs in favour of something that uses PSGI, I often get some push-back from people claiming that PSGI makes things far too complicated.

I don’t believe that’s true. But I think I know why they say it. I think they say it because most of the time when we say “you should really port that code to PSGI” we follow up with links to Dancer, Catalyst or Mojolicious tutorials.

I know why we do that. I know that a web framework is usually going to make writing a web app far simpler. And, yes, I know that in the Plack::Request documentation, Miyagawa explicitly says:

Note that this module is intended to be used by Plack middleware developers and web application framework developers rather than application developers (end users).

Writing your web application directly using Plack::Request is certainly possible but not recommended: it’s like doing so with mod_perl’s Apache::Request: yet too low level.

If you’re writing a web application, not a framework, then you’re encouraged to use one of the web application frameworks that support PSGI (http://plackperl.org/#frameworks), or see modules like HTTP::Engine to provide higher level Request and Response API on top of PSGI.

And, in general, I agree with him wholeheartedly. But I think that when we’re trying to persuade people to switch to PSGI, these suggestions can get in the way. People see switching their grungy old CGI programs to a web framework as a big job. I don’t think it’s as scary as they might think, but I agree it’s often a non-trivial task.

Even without using a web framework, I think that you can get benefits from moving software to PSGI. When I’m running training courses on PSGI, I emphasise three advantages that PSGI gives you over other Perl web development environments.

  1. PSGI applications are easier to debug and test.
  2. PSGI applications can be deployed in any environment you want without changing a line of code.
  3. Plack Middleware

And I think that you can benefit from all of these features pretty easily, without moving to a framework. I’ve been thinking about the best way to do this and I think I’ve come up with a simple plan:

  • Change your shebang line to /usr/bin/plackup (or equivalent)
  • Put all of your code inside my $app = sub { ... }
  • Switch to using Plack::Request to access all of your input parameters
  • Build up your response output in a variable
  • At the end of the code, create and return the required Plack response (either using Plack::Response or just creating the correct array reference).

That’s all you need. You can drop your new program into your cgi-bin directory and it will just start working. You can immediately benefit from easier testing and later on, you can easily deploy your application in a different environment or start adding in middleware.

As an experiment to find how easy this was, I’ve been porting some old CGI programs. Back in 2000, I wrote three articles introducing CGI programming for Linux Format. I’ve gone back to those articles and converted the CGI programs to PSGI (well, so far I’ve done the programs from the first two articles – I’ll finish the last one in the next day or so, I hope).

It’s not the nicest of code. I was still using the CGI’s HTML generation functions back then. I’ve replaced those calls with HTML::Tiny. And they aren’t very complicated programs at all (they were aimed at complete beginners). But I hope they’ll be a useful guide to how easy it is to start using PSGI.

My programs are on Github. Please let me know what you think.

If you’re interested in modern Perl Web Development Techniques, you might find it useful to attend my upcoming two-day course on the subject.

Update: On Twitter, Miyagawa reminds me that you can use CGI::Emulate::PSGI or CGI::PSGI to run CGI programs under PSGI without changing them at all (or, at least, changing them a lot less than I’m suggesting here). And that’s what I’d probably do if I had a large amount of CGI code that I wanted to to move to PSGI quickly. But I still think it’s worth showing people that simple PSGI programs really aren’t any more complicated than simple CGI programs.

The Long Death of CGI.pm

CGI.pm has been removed from the core Perl distribution. From 5.22, it is no longer included in a standard Perl installation.

There are good technical reasons for this. CGI is a dying technology. In 2015, there are far better ways to write web applications in Perl. We don’t want to be seen to encourage the use of a technology which no-one should be using.

This does lead to a small problem for us though. There are plenty of web hosting providers out there who don’t have particularly strong Perl support. They will advertise that they support Perl, but that’s just because they know that Perl comes as a standard part of the operating system that they run on their servers. They won’t do anything to change their installation in any way. Neither you nor I would use a hosting company that works like that – but plenty of people do.

The problem comes when these companies start to deploy an operating system that includes Perl 5.22. All of a sudden, those companies will stop including CGI.pm on their servers. And while we don’t want to encourage people to use CGI.pm (or, indeed, the CGI protocol itself) we need to accept that there are thousands of sites out there that have been happily using software based on CGI.pm for years and the owners of these sites will at some point change hosting providers or upgrade their service plan and end up on a server that has Perl 5.22 and doesn’t have CGI.pm. And their software will break.

I’ve always assumed that this problem is some time in the future. As far as I can see, the only mainstream Linux distribution that currently includes Perl 5.22 is Fedora 23. And you’d need to be pretty stupid to run a web hosting business on any version of Fedora. Fedora is a cutting edge distribution with no long term support. Versions of Fedora are only supported for about a year after their release.

So the problem is in the future, but it is coming. At some point Perl 5.22 or one of its successors will make it into Red Hat Enterprise Linux. And at that point we have a problem.

Or so I thought. But that’s not the case. The problem is here already. Not because of Perl 5.22 (that’s still a year or two in the future for most of these web hosting companies) but because of Red Hat.

Red Hat, like pretty much everyone, include Perl in their standard installation. If you install any Linux distribution based on Red Hat, then the out of the box installation includes an RPM called “perl”. But it’s not really what you would recognise as Perl. It’s a cut down version of Perl. They have stripped out many parts of Perl that they consider non-essential. And those parts include CGI.pm.

This change in the way they package Perl started with RHEL 6 – which comes with Perl 5.10. And remember it’s not just RHEL that is affected. There are plenty of other distributions that use RHEL as a base – Centos, Scientific Linux, Cloud Linux and many, many more.

So if someone uses a server running RHEL 6 or greater (or another OS that is based on RHEL 6 or greater) and the hosting company have not taken appropriate action, then that server will not have CGI.pm installed.

What is the “appropriate action” you ask. Well it’s pretty simple. Red Hat also make another RPM available that contains the whole Perl distribution. So bringing the Perl up to scratch on a RHEL host is as simple as running:

yum install perl-core

That will work on a server running RHEL 6 (which has Perl 5.10) and RHEL 7 (which has Perl 5.16). On a future version of RHEL which includes Perl 5.22 or later, that obviously won’t work as CGI.pm won’t be part of the standard Perl installation and therefore won’t be included in “perl-core”. At that point it will still be a good idea to install “perl-core” (to get the rest of the installation that you are missing) but to get CGI.pm, you’ll need to run:

yum install perl-CGI

So this is a plea to people who are running web hosting services using Red Hat style Linux distributions. Please ensure that your servers are running a complete Perl installation by running the “yum” command above.

All of which brings me to this blog post that Marc Lehmann wrote a couple of days ago. Marc found a web site which no longer worked because it had been moved to a new server which had a newer version of Perl – one that didn’t include CGI.pm. Marc thinks that the Perl 5 Porters have adopted a cavalier approach to backward compatibility and that the removal of CGI.pm is a good example of the problems they are causing. He therefore chose to interpret the problems this site was having as being caused by p5p’s approach to backward compatibility and the removal of CGI.pm.

This sounded unlikely to me. As I said above, it would be surprising if any web hosting company was using 5.22 at this point. So, I did a little digging. I found that the site was hosted by BlackNight solutions and that their web says that their servers run Perl 5.8. At the same time, Lee Johnson, the current maintainer of CGI.pm, got in touch with the web site’s owner who confirmed what I had worked out was correct.

Later yesterday I had a conversation with @BlackNight on Twitter. They told me that their hosts all ran Cloud Linux (which is based on RHEL) and that new servers were being provisioned using Cloud Linux 6 (which is based on RHEL 6).

So it seems clear what has happened here. The site was running on an older server which was running Cloud Linux 5. That includes Perl 5.8 and predates Red Hat removing CGI.pm from the “perl” RPM. It then moved to a new host running Cloud Linux 6 which is based on RHEL 6 and doesn’t include CGI.pm in the default installation. So what the site’s owner said is true, he moved to a new host with a newer version of Perl (that new version of Perl was 5.10!) but it wasn’t the new version of Perl that caused the problems, it was the new version of the operating system or, more specifically, the change in  the way that Red Hat (and its derivatives) packaged Perl.

Marc is right that when Perl 5.22 hits the web hosting industry we’ll lose CGI.pm from a lot a web servers. You can make your own mind up on how important that is and whether or not you share Marc’s other opinions on how p5p is steering Perl. But he’s wrong to assume that, in this instance, the problem was caused by anything that p5p have done. In this instance, the problem was caused by Red Hat’s Perl packaging policy and was compounded by a hosting company who didn’t know that upgrading their servers to Cloud Linux 6 would remove CGI.pm.

RHEL 6 was released five years ago. I suspect it’s pretty mainstream in the web hosting industry by now. So CGI.pm will already have disappeared from a large number of web servers. I wonder why we haven’t seen a tsunami of complaints?

Update: More discussion on Reddit and Hacker News.

Removing Modules from Core

I was on holiday last week and missed most of the discussions about removing Module::Build and CGI from the Perl core in the next few years. I hope you won’t mind if I chime in a little late with my thoughts.

Module::Build

I’m a little bemused by the Module::Build story. Well, perhaps “bemused” isn’t quite the right word. Perhaps I should say “embarrassed”. Embarrassed because the discussions around Module::Build just show how out of touch I am with important parts of the Perl ecosystem.

I remember seeing Schwern’s MakeMaker Is DOOMED! talk ten years ago. What he said made a lot of sense to me and I switched all of my CPAN modules to Module::Build over the next few years. And then I rather took my eye off the ball. I was vaguely aware that people were complaining about Module::Build and were switching to things like Module::Install and Dist::Zilla. But I was happy with Module::Build and just stuck with it.

Now I’m hearing people saying that Module::Build is fundamentally flawed. Someone whose opinion I value mentioned that he lowers his estimate of the IQ of any CPAN author who uses Module::Build. That, presumably, includes me. But when I ask people what the problems are with Module::Build, I can’t get anyone to give me a detailed and dispassionate answer.

I’m quite prepared to believe that there are problems with Module::Build. Given the calibre of the people who are telling me that these problems exist, it would be astonishing if it wasn’t true. But it would be nice if there was a good explanation available somewhere of just what these problems are.

And I think that this is indicative of a deeper problem that the Perl community has. I like to think that I’m pretty much in touch with what is going on in the Perl community. I read the blogs. I’m subscribed to (too many) mailing lists. But it appears I’m not following discussions of the CPAN toolchain closely enough. I guess it’s something that I just expect to work. And I guess I expect that if there are important discussions going on, then I’ll hear about them through other channels – blog posts, for example. But that doesn’t seem to have happened here. Of course I can subscribe myself to another mailing list or two to correct that, but I don’t think I’m unusual in not trying to follow every single Perl discussion. There are surely many people like me who would like information like this to be disseminated better. This is part of the reason why Leo and I set up the Perl News site. But we have a tiny number of editors and we can’t know everything that is worth publishing. If you think we’ve missed an interesting story then please let us know about it.

So I’m not going to object to Module::Build leaving the core. I’m sure there are good reasons, I just wish I knew what they are. I am, however, slightly disappointed to find that Schwern was wrong ten years ago and that ExtUtils::MakeMaker wasn’t doomed.

CGI

CGI.pm and I go back a long way. In fact I think CGI.pm was added to the Perl core at about the same time as I started using Perl.

It’s obvious why it was added to the core. Back in 1997, Perl and CGI were tightly bound together in many people’s minds. I had people telling me that Perl could only be used to write CGI programs and other people telling me that CGI programs could only be written in Perl. Both sets of people were, of course, wrong; but it’s easy to see how they reached both of those conclusions.

Before CGI.pm was added to the Perl core, people used to parse CGI parameters using horribly broken code copied from Matt Wright (code which he had originally copied from Reuven Lerner). Of course, adding the module to the Perl core didn’t magically rewrite all those horrible CGI programs to use CGI::params, but at least it meant that the option was there.

This became important to me when I started the nms project. This project rewrote Matt Wright’s scripts using a better standard of Perl code. The problem was that many people started by installing one of Matt’s scripts on their server and later, when they wanted to move on from using other people’s code, they would use Matt’s code as a template for their own first steps into Perl. This is partly what made Perl so popular. But it’s also what lead to the preponderance of terrible Perl code that you still find on the web today.

We wrote the nms programs to work, as far as possible, in the same environment as Matt’s scripts. Matt said that his scripts would work fine on Perl 4 (back then, there were still many cheap hosting plans that came with Perl 4). We decided not to put ourselves through that pain and to target Perl 5.004 – because that was the first version to include CGI.pm. Remember, we were targeting Matt Wright’s users. And they were people who used cheap hosting plans with basic FTP access and little or no chance of installing anything from CPAN. We had to rely on core modules. If CGI.pm hadn’t been in the Perl core, then the nms programs would have been far harder to write and the project may have never been started.

The nms project was started over ten years ago. What is the situation like now? Those cheap hosting plans still exist. People still use them for simple web sites. Some of those people still use Matt’s scripts or the nms alternatives. But the numbers of people using those programs are far smaller than they used to be. These days, people wanting to add simple dynamic functionality to a web site are far more likely to use a PHP program. And those cheap web hosts are far more likely to want to support PHP.

Of course, these days, no serious Perl developer uses CGI.pm to write web programs. We’d be far more likely to use something based on PSGI. And even if we did want to use CGI.pm, we’d know how to get it installed from CPAN. The only people who are going to suffer from the removal of CGI.pm are the people using something like the nms programs on a cheap hosting plan. I worry a little about the influx of support email we will get when some cheap hosting company updates from Perl 5.18 to Perl 5.22 and suddenly all the nms programs stop working. It’s slightly galling to realise that Matt’s scripts will continue to work at that point. Of course, the kinds of companies that we’re talking about tend to lag well behind the bleeding edge of Perl versions, so this might not happen for eight or more years. So I think it might well be someone else’s problem.

All in all, I think it will be sad to see CGI.pm removed from the core. It will effectively see the end of even half-decent Perl code being used on bottom of the range hosting plans. But Perl has been losing that market for years and, speaking as someone who spent a lot of time helping out on support forums for basic Perl/CGI code, I’m far from convinced that I’ll miss those users. So my sadness at seeing CGI,pm go, will be purely historical.

But I’d really like to see some cut-down version of PSGI take its place!

CGI.pm vs Templates

I’ve just been involved in a discussion on LinkedIn that I thought deserved a wider audience (I have no idea how well that link works if you’re not a member or or logged in to LinkedIn).

A couple of days ago, someone asked for advice on the best way to include HTML in a Perl program. They got a lot of good advice (basically – don’t do that, use a templating system instead). And then this morning, someone came in and said:

use CGI module, u don’t need to write a single HTML tag..just use CGI Methods…

I did a bit of a double take at that point. I thought we’d all stopped using CGI.pm’s HTML generation methods back in the last millennium. I replied saying that, but this chap was adamant that CGI.pm worked for him. He asked me to explain why I was so against his approach.

This was my reply:

I never said that it didn’t work. I just think that it’s a really limited way to build things and you’d be better off taking a more flexible approach right from the start.

My main objection is the separation of concerns. The logic of your application is separate to the display layer. It’s quite possible that the display of the application will need to change in the future. And that all becomes easier if the HTML is stored separately from code.

This leads to other problems too:

As I hinted at above, CGI.pm only has pretty basic support for HTML. It’s really hard to create a great-looking web application using CGI.pm. In fact, I wouldn’t be surprised if CGI.pm was largely responsible for the terrible web sites that the Perl community has been building for itself for most of the last twenty years.

Your project might well have HTML experts who know how to create good looking web pages. Expecting them to do it by editing Perl code is a bad idea. Front end developers need to know HTML, CSS and Javascript. Why force them to understand Perl as well?

Even if you’re the only developer on a project (as I often am for my personal projects) there are advantages to separating your work into “front-end task” and “back-end tasks”. I find it really helps me switch my mindset to the appropriate skills if I switch from editing a Perl source file to editing an HTML template.

It makes your Perl code more complicated than it needs to be. Seriously, try moving the HTML out of the code and into a template. See how much simpler your code becomes.

It’s unlikely that your CGI-generated pages will make up all of the web site. There will almost certainly be static pages too. If you use a tool like the Template Toolkit then you can generate static pages from the same set of templates that your CGI programs use. That way it’s easier to maintain a consistent look and feel across all of the pages on the site.

I’m not saying “don’t use CGI.pm” (well, I am a bit – but that’s a different story). The CGI bits of CGI.pm (for example the bits that give you access to the incoming parameters or let you set the outgoing headers) are great. It’s just the HTML generation stuff that I strongly recommend that you don’t use.

I’ve just remembered an article that I wrote in 2001 that might also help. The first half is about cookies, but the second half demonstrates using Templates instead of the HTML generation methods. I hope it shows how much simpler it makes things

See http://mag-sol.com/articles/cgi3.html

Does that help at all?

I think that pretty much sums up my objections. Did I miss anything obvious? Or am I completely wrong and many of you are still using the HTML generation methods in CGI.pm to do this?

The Perl community on LinkedIn is a fascinating place. I should really write a blog post about it.

Marketing Perl

Sometimes people ask me why Perl marketing is so important. This morning I came across an excellent example of the kind of thing that we’re trying to counter.

In the current issue of Linux Format, there’s an article about building a Twitter client in the bash shell. It’s written by Nick Veitch – who seems to dislike Perl a bit. In the article he wants to URL encode a string and can’t find an easy way to do it in bash. He writes:

However, the compromise I’ve found for this isn’t too bad (props to http://stakface.com/nuggets). It uses Perl and, like everything in that language, it looks like you ought to sacrifice a goat or something before you run it.

perl -p -e 's/([^A-Za-z0-9.-~])/sprintf("%%%02X", ord($1))/seg'

It’s not clear to me which part of the Perl make him want to start sacrificing goats. Is it the s/// syntax that Perl borrowed from sed? Perhaps it’s the regular expression syntax that Perl shares with pretty much any language with regex support. Or maybe it’s the sprintf function that Perl borrowed from C and that many other languages (including bash) support with a pretty similar syntax.

Of course I’m not saying that Perl doesn’t have some grungy corners in its syntax. But the three pieces of syntax used in this code fragment look to me like things that should be understood easily by just about anyone with experience of programming in the Unix environment.

And if this code is so hard to understand, why use it? I don’t know what Nick’s programming language of choice is, but couldn’t you do exactly the same thing in Python, Ruby or PHP? I can’t believe that Perl is the only language that is so powerful. And I’d be very surprised if the same code didn’t look very similar in many other languages.

So why the unnecessary little dig at Perl?

Of course, it’s clear that Perl isn’t a language that Nick is particularly familiar with (neither, indeed, is the person who he took the solution from). Anyone who knew Perl would realise that Perl’s standard distribution includes the CGI module which contains an escape function which does exactly what Nick wanted – without exposing the programmer to all of that scary syntax. I would write Nick’s code something like this:

perl -MCGI=escape -e'print escape "@ARGV"'

There are a lot of people out there with really strange ideas about Perl. People who don’t bother to find out the best ways to do things in Perl. And having widely-read Linux magazines printing snide comments about Perl does no-one any good at all.

This is why I think that Perl marketing is important. We need to reach the people outside of the echo chamber and tell them that Perl isn’t the outdated, hard-to-use language that they are being told that it is.

Update: I should have pointed out that I’ve sent an abbreviated version of this to Linux Format. Hopefully it’ll be published in the next issue. And I’m considering proposing a series of articles on Perl to them.