Categories
Programming

Please Don’t Use CGI.pm

Earlier this week, the Perl magazine site, perl.com, published an article about writing web applications using CGI.pm. That seemed like a bizarre choice to me, but I’ve decided to use it as an excuse to write an article explaining why I think that’s a really bad idea.

It’s important to start by getting some definitions straight – as, often, I see people conflating two or three of these concepts and it always confuses the discussion.

  • The Common Gateway Interface (CGI) is a protocol which defines one way that you can write applications that create dynamic web pages. CGI defines the interface between a web server and a computer program which generates a dynamic page.
  • A CGI program is a computer program that is written in a manner that conforms to the CGI specification. The program optionally reads input from its environment and then prints to STDOUT a stream of data representing a dynamic web page. Such programs can be (and have been!) written in pretty much any programming language.
  • CGI.pm is a CPAN module which makes it easier to write CGI programs in Perl. The module was included in the Perl core distribution from Perl 5.004 (in 1997) until it was removed from Perl 5.22 (in 2015).

A Brief Introduction to CGI.pm

CGI.pm basically contained two sets of functions. One for input and one for output. There was a set for reading data that was passed into the program (the most commonly used one of these was param()) and a set for producing output to send to the browser. Most of these were functions which created HTML elements like <h1> or <p>. By about 2002, most people seemed to have worked out that these HTML creation functions were a bad idea and had switched to using a templating engine instead. One output function that remained useful was header() which gave the programmer an easy way to create the various headers required in an HTTP response – most commonly the “Content-type” header.

For at least the last ten years that I was using CGI.pm, my programs included the line:

as it was only the param() and header() functions that I used.

I should also point out that there are two different “modes” that you can use the module in. There’s an object-oriented mode (create an object with CGI->new and interact with it through methods) and a function-based mode (just call functions that are exported by the module). As I never needed more than one CGI object in a program, I always just used the function-based interface.

Why Shouldn’t We Use CGI.pm Today?

If you’re using CGI.pm in the way I mentioned above (using it as a wrapper around the CGI protocol and ignoring the HTML generation functions), then it’s not actually a terrible way to write simple web applications. There are two problems with it:

  1. CGI programs are slow. They start up a Perl process for each request to the CGI URL. This is, of course, a problem with the CGI protocol itself, not the CGI.pm module. This might not be much of a problem if you have a low-traffic application that you want to put on the web.
  2. CGI.pm gives you no help building more complicated features in a web application. For example, there’s no built-in support for request routing. If your application needs to control a number of URLs, then you either end up with a separate CGI program for each URL or you shoe-horn them all into the same program and set up some far-too-clever mod_rewrite magic. And everyone reinvents the same wheels.

Basically, there are better ways to write web applications in Perl these days. It was removed from the Perl code distribution in 2015 because people didn’t want to encourage people to use an outdated technology.

What are these better methods? Well, anything based on an improved gateway interface specification called the Perl Server Gateway Interface (PSGI). That could be a web framework like Dancer2, Catalyst or Web::Simple or you could even just use raw PSGI (by using the toolkit in the Plack distribution).

Often when I suggest this to people, they think that the PSGI approach is going to be far more complex than just whipping up a quick CGI program. And it’s easy to see why they might think that. All too often, an introduction to PSGI starts by building a relatively powerful (and, therefore, complicated) web application using Catalyst. And while Catalyst is a fine web framework, it’s not the simplest way to write a basic web application.

But it doesn’t need to be that way. You can write PSGI programs in “raw PGSI” without reaching for a framework. Sure, you’ll still have the problems listed in my point two above, but when you want to address that, you can start looking at the various web frameworks. Even so, you’ll have three big benefits from moving to PSGI.

The Benefits of PSGI

As I see it, there are three huge benefits that you get from PSGI.

Software Ecosystem

The standard PSGI toolkit is called Plack. You’ll need to install that. That will give you adapters enabling you to use PSGI programs in pretty much any web deployment environment. It also includes a large number of plugins and extensions (often called “middleware”) for PSGI. All of this software can be added to your application really simply. And any bits of your program that you don’t have to write is always a big advantage.

Testing and Debugging

How do you test your CGI program? Probably, you use something like Selenium (or, perhaps, just LWP) to fire requests at the server and see what results you get back.

And how about debugging any problems that your testing finds? All too often, the debugging that I see is warn() statements written to the web server error log. Actually, when answering questions on StackOverflow, often the poster has no idea where to find the error log and we need to resort to something like use CGI::Carp 'fatalsToBrowser', which isn’t exactly elegant.

A PSGI application is just a subroutine. So it’s trivial for testing tools to call the subroutine with the correct parameters. This makes testing PSGI programs really easy (and all of the tools to do this are part of the Plack distribution I mentioned above). Similarly, there are tools debugging a PSGI program far easier than the equivalent CGI program.

Deployment Flexibility

This, to me, is the big one. I talked earlier about the performance problems that the CGI environment leads to. You have a CGI program that is only used by a few people on your internal network. And that’s fine. The second or so it takes to respond to each request isn’t a problem. But it proves useful and before you know it, many more people start to use it. And then someone suggests publishing it to external users too. The one-second responses stretch to five or ten seconds, or even longer and you start getting complaints about the system. You know you should move it to a persistent environment like FastCGI or mod_perl, but that would require large-scale changes to the code and how are you ever going to find the time for that?

With a PSGI application, things are different. You can start by deploying your PSGI code in a CGI environment if you like (although, to be honest, it seems that very few people do that). Then when you need to make it faster, you can move it to FastCGI or mod_perl. Or you can run it as a standalone web service and configure your web proxy to redirect requests to it. Usually, you’ll be able to use exactly the same code in all of these environments.

In Conclusion

I know why people still write CGI programs. And I know why people still write them using CGI.pm – it’s what people know. It’s seen as the easy option. It’s what twenty-five years of web tutorials are telling them to do.

But in 2018 (and, to be honest, for most of the last ten years) that’s simply not the best approach to take. There are more powerful and more flexible options available.

Please don’t write code using CGI.pm. And please don’t write tutorials encouraging people to do that.

Categories
Programming

Line of Succession

I’m a republican. No… wait… come back! That’s not what I mean.

I’m a long way from being a supporter of the Republican Party. I mean “republican” in its older meaning of “someone who thinks their country should be a republic. That is to say, I’m not a big fan of the British royal family.

But while I believe that the UK should get rid of the royal family, I’m also fascinated by them. In particular, I’m fascinated by the laws that determine the line of succession – that is the list of people in line to take over the throne.

When I was a child I believed that the line of succession was a big list that had every British person’s name on it and that it would only take a single catastrophic event to propel my name to the top of that list. Later on I discovered the Act of Settlement (1701), which is the law which actually defines the line of succession (modulo a few later tweaks). I was disappointed to find that there were only a few thousand people on the list (and that didn’t include me!) and also that a lot of the people on the list weren’t British (largely due to Queen Victoria’s children marrying royalty from all over Europe).

A few months ago, I started to think about building a web site that would allow people to explore the line of succession through time. And over the last few weeks, I have build the site. It’s at lineofsuccession.co.uk. On the main page, you will see the current line of succession. And in the navigation bar is a drop-down menu that allows you to move to a few interesting data (the days that the last four monarchs came to the throne) and a date picker allowing you to choose any random date.

The code is, of course, on Github. The web app is a pretty standard Dancer2 application which really doesn’t do anything clever. Most of the complexity in an application like this is in the data gathering.

Currently I have just over a hundred people in the database. That’s most of the descendants of Edward VII (there a few lines that I haven’t completely filled out yet), but eventually I want to go back to all include all of the descendants of Electress Sophia (the person who the crown was “settled on” in the Act of Settlement). I’ve heard estimates that she has somewhere between 5-6,000 descendants. So I have a bit of work to do there!

Other than more people, I have a few other things I’d like to add to the site:

  • More names and titles People on the line of succession tend to have many titles during their lives. The data model already supports the concept of a name that is only valid on a range of dates (see the current queen described as Princess Elizabeth of York on the day before her father became king,  Princess Elizabeth when he was king, Duchess of Edinburgh when she got married and, finally, Queen Elizabeth II when she became queen. But tracking down and adding all that data is hard work.
  • Excluded people Catholics are excluded from the line of succession. And people who married Catholics were also excluded until recently. Oh, and obviously children born out of wedlock (that’s more common than you might expect in some of the more obscure branches of the modern Windsor family). Of course, people can convert to (and from) Catholicism at any time, so supporting that in the app would mean implementing some kind of “exclusions” data. But it would be good to show these people, perhaps in a dimmer font.
  • Tree views Today I added text to each person showing their relationship to the monarch. That can help a reader to visualise the family tree, but I’d like to make it more explicit. Nested lists could make it easy to see the relationships. And, later, perhaps show the whole tree using SVG.
  • Position changes Sometimes the line of succession feels a bit like the pop charts. Take Prince Harry, for example. He entered the chart at position 3 and stayed there until Prince George bumped him down to number 4 and Princess Charlotte pushed him down to 5. He’s only likely to fall further in the coming years. The same thing happened to Princess Anne, who was number 3 when she was born, but currently languishes down at number 12. I think it would be interesting to plot those changes over time.
  • Make it prettier Bootstrap does the job. It allows design dunces like me to get a reasonable-looking site up and running in no time. But it’s not very regal.

Anyway, there’s my current itch scratched. And, as in so many cases, it’s just given me more itches. But please let me know if you find the site at all interesting or useful.

Categories
Training

Intended Audience

I thought I’d pretty much finished blogging about my upcoming Modern Web Development with Perl and Dancer training course. But a couple of days ago I saw a tweet that reminded me about an aspect that I’d completely forgotten.

And he’s right, of course. I haven’t mentioned that at all. Let’s put that right.

As it happens, yesterday I pretty much finished writing the slides for the course. So that means that I know what I’ll be covering and, therefore, what the attendees will need to know.

What You’ll Need To Know

To start with, I need to make it clear that this is not a “beginning Perl” course. There’s a lot of new topics to cover and if Perl itself was on the list then it would need to be a two or three day course.

So you’ll need to know Perl. But to what level?

If you’ve read Intermediate Perl then you’ll be fine. That means you’ll need to understand how to use modules, packages and references. Probably the most advanced Perl concept we’ll need is subroutine references. But, to be honest, if you’re not completely comfortable with them, that won’t be a problem.

You’ll need to know a bit about how web pages are made – so a little bit of HTML and CSS. We’ll be using Bootstrap to deal with most of our CSS, so you won’t need to do anything at all complicated with CSS. If you understand the difference between a class and an id in CSS terms then you’ll be fine.

We’ll be using quite a lot of Javascript – specifically jQuery with Mustache. I’m no Javascript expert, so it’s likely that many of the people in the class will know more than me. If you’ve never used jQuery, then I recommend that you spend a couple of hours looking into it before coming to the class. You don’t need to know anything about Mustache before the course.

There will be a database at the back-end of the app. I’ll be running MySQL (actually, probably MariaDB), but any of the popular database systems will work – just as long as Perl’s DBI supports it. I’ll supply SQL to set up the database and insert some test data and we’ll be using DBIx::Class which will remove the need to know any SQL. But it would be good if you were familiar with whatever database system you’re using – to the extent that you can run queries against your local database.

What You’ll Need to Bring

You’ll need a laptop. I’m assuming that we’ll have access to WiFi at the training venue, but it would be great if you could install as much as possible of the required software before the day – just so we save a bit of time.

My laptop runs Windows 10, but I do all of my development in a virtual machine running Fedora 24. I’m happy for you to work in Windows or OSX, but the level of support I can provide for people not running Linux will be limited.

You’ll need Perl installed. Linux and OSX will already have a version of Perl installed. For Windows users, I recommend Strawberry Perl. Get the most recent version of Perl that you can install. The current version is 5.24. I think my laptop has 5.22. Anything  earlier than 5.10 is unlikely to be particularly useful.

You’ll need some CPAN modules installed. These are all pretty common modules:

  • Dancer2
  • Dancer2::Plugin::DBIC
  • DBIx::Class
  • DBIx::Class::Schema::Loader
  • DBI
  • DBD::* (for whatever database you are using – e.g. DBD::mysql)
  • Moose
  • MooseX::NonMoose
  • MooseX::MarkAsMethods
  • DateTime
  • DateTime::Format::Strptime
  • Template

You’ll need a database server installed on your laptop. As I mentioned above, any of the popular database engines will work – but I’ll be using MariaDB. Make sure that you know how to start the database server and connect to it using a command line program.

You’ll need a Git client so that you can clone the Git repository that contains the source code for the course. You’ll want to ensure the the repository is cloned to your laptop before turning up to the course. You might even want to glance through some of the code to get a head-start on the rest of the attendees.

You can find the course code at

The CSS and Javascript libraries are all included in the Git repository.

I think that’s about all you need to know. Please let me know if you have any further questions.

I’ve been really pleased with the reaction to this course. We already have a large number of people signed up. So many, in fact, that I need to start thinking about the number of people I have room for. I think we can get another five (perhaps ten) people in. So if you’re thinking of signing up, please do it soon to avoid disappointment (trainers say stuff like this for every course – but this time it’s really true).

Hope to see some of you in Cluj-Napoca.

Categories
Training

Modern Web Development with Perl and Dancer2

Here are some more details of the Modern Web Development with Perl and Dancer2 course that I’ll be running in Cluj-Napoca on the day before YAPC Europe.

The course runs a full day (that’s six hours – in four 90-minute sessions with breaks in between). It’s a hands-on course – you’ll need to bring a laptop and closer to the time I’ll email attendees with details of the software they will need to have installed. Like all of the pre-conference training, the course will take place at Cluj Hub on Tuesday 23rd August.

Over the course of twelve steps, we’ll build a simple Todo list program. We’ll be using a number of modern web development techniques (not just Perl) in order to make the app look really shiny and modern.

The twelve steps we will be taking are as follows:

  1. Set up a basic Dancer2 app
  2. Make it look nicer with the addition of the Bootstrap CSS framework
  3. Use Plack Middleware to serve static content more efficiently
  4. Display some data in our app
  5. Get the data from a database
  6. Return the data as JSON and display it using Mustache
  7. Use jQuery to show/hide completed items
  8. Mark items as completed
  9. Add new tasks to the app
  10. Add user login
  11. Edit and delete tasks
  12. Add tags to tasks and filter the display on those tags

If there’s time left at the end, we’ll discuss other useful enhancements that we might want to make to the app – and perhaps even try adding them.

We’ll be using the following Perl tools:

And the following non-Perl tools:

Usually, a course like this would cost around £300. But because it’s at YAPC and the sponsors are so generous, we can offer it for the heavily discounted price of 100€.

Cluj Hub sounds like a fabulous venue for the training courses and I’m sure that the day will be a lot of fun. Perhaps more importantly, I’m also sure that attendees will come away with some useful skills to add to their CVs.

Tickets are on sale now. Please buy quickly – before they sell out.

Categories
Programming

Easy PSGI

When I write replies to questions on StackOverflow and places like that recommending that people abandon CGI programs in favour of something that uses PSGI, I often get some push-back from people claiming that PSGI makes things far too complicated.

I don’t believe that’s true. But I think I know why they say it. I think they say it because most of the time when we say “you should really port that code to PSGI” we follow up with links to Dancer, Catalyst or Mojolicious tutorials.

I know why we do that. I know that a web framework is usually going to make writing a web app far simpler. And, yes, I know that in the Plack::Request documentation, Miyagawa explicitly says:

Note that this module is intended to be used by Plack middleware developers and web application framework developers rather than application developers (end users).

Writing your web application directly using Plack::Request is certainly possible but not recommended: it’s like doing so with mod_perl’s Apache::Request: yet too low level.

If you’re writing a web application, not a framework, then you’re encouraged to use one of the web application frameworks that support PSGI (http://plackperl.org/#frameworks), or see modules like HTTP::Engine to provide higher level Request and Response API on top of PSGI.

And, in general, I agree with him wholeheartedly. But I think that when we’re trying to persuade people to switch to PSGI, these suggestions can get in the way. People see switching their grungy old CGI programs to a web framework as a big job. I don’t think it’s as scary as they might think, but I agree it’s often a non-trivial task.

Even without using a web framework, I think that you can get benefits from moving software to PSGI. When I’m running training courses on PSGI, I emphasise three advantages that PSGI gives you over other Perl web development environments.

  1. PSGI applications are easier to debug and test.
  2. PSGI applications can be deployed in any environment you want without changing a line of code.
  3. Plack Middleware

And I think that you can benefit from all of these features pretty easily, without moving to a framework. I’ve been thinking about the best way to do this and I think I’ve come up with a simple plan:

  • Change your shebang line to /usr/bin/plackup (or equivalent)
  • Put all of your code inside my $app = sub { ... }
  • Switch to using Plack::Request to access all of your input parameters
  • Build up your response output in a variable
  • At the end of the code, create and return the required Plack response (either using Plack::Response or just creating the correct array reference).

That’s all you need. You can drop your new program into your cgi-bin directory and it will just start working. You can immediately benefit from easier testing and later on, you can easily deploy your application in a different environment or start adding in middleware.

As an experiment to find how easy this was, I’ve been porting some old CGI programs. Back in 2000, I wrote three articles introducing CGI programming for Linux Format. I’ve gone back to those articles and converted the CGI programs to PSGI (well, so far I’ve done the programs from the first two articles – I’ll finish the last one in the next day or so, I hope).

It’s not the nicest of code. I was still using the CGI’s HTML generation functions back then. I’ve replaced those calls with HTML::Tiny. And they aren’t very complicated programs at all (they were aimed at complete beginners). But I hope they’ll be a useful guide to how easy it is to start using PSGI.

My programs are on Github. Please let me know what you think.

If you’re interested in modern Perl Web Development Techniques, you might find it useful to attend my upcoming two-day course on the subject.

Update: On Twitter, Miyagawa reminds me that you can use CGI::Emulate::PSGI or CGI::PSGI to run CGI programs under PSGI without changing them at all (or, at least, changing them a lot less than I’m suggesting here). And that’s what I’d probably do if I had a large amount of CGI code that I wanted to to move to PSGI quickly. But I still think it’s worth showing people that simple PSGI programs really aren’t any more complicated than simple CGI programs.