Perl Hacks

Crufty Old Perl

2010-03-11T18:05:36Z

It's eighteen months since I wrote "Why Corporates Hate Perl" and it's worth pointing out that the company I discussed in that article which was dropping Perl in favour of PHP and Java is still employing as many good Perl programmers as it can find.

I talked in that article about some rather unsubtle social engineering that had been going on at that company. Management had started to talk about Perl as a "legacy language" in an attempt to persuade the business that they really don't want their applications written in Perl. That doesn't seem to have been as successful as I feared it would be.

But there's another kind of social engineering that I've seen going on. And this is happening at the developer level. I've lost count of the number of times I've been sitting in meetings with developers, discussing ways to improve the quality of crufty old Perl code when someone throws in the (more than half-serious) suggestion that we should just rewrite the whole thing using Rails or Django.

There seems to be this idea that switching to a new framework in a new language will act as some time of magic bullet and that suddenly all of our problems will be solved. There are so many ways that this perception is flawed, Here are my two favourites.

The current system is old and crufty not because it's written in Perl, but because it was written by people who didn't understand the business requirements fully, didn't know their tools particularly well or were under pressure to deliver on a timescale that didn't give them time to design the system properly. Often it's a combination of all three. Given the time to rewrite the system from scratch, of course it will be better. But it will be better because the business is better understood and tools and techniques have been improved - not because it's no longer written in Perl.
Frameworks in other languages are not easier to use or more powerful than frameworks in Perl. Anything that you can do with Rails or Django you can do just as easily with Catalyst. It's using a framework that's the big win here, not the particular framework that you use. Sure, if you're a Ruby house then using a Ruby framework will probably match your existing developers' skills more closely but if your current system is written in Perl then, hopefully, you have a team of people with Perl skills and that's going to be the language you'll want to look at.

I'm tired of Perl being seen as a second-class citizen in the dynamic languages world. I freely admit that there's a lot of bad Perl code out there (I'll even admit to writing some of it) but it's not bad Perl code because it's Perl, it's bad Perl code because it's bad code.

This is what the Perl marketing initiative is for. We need people to know that Perl is not the same language that they stopped using ten years ago. Modern Perl can compete with any of the features of any other dynamic language.

By al means, try to get the time to rewrite your crufty old systems. But rewrite them in Modern Perl. You'll enjoy it and you'll be really productive.

p.s. I should point out that I'm not talking about any specific client here. This is based on conversations that I've had at various companies over the last couple of years and also discussions with many developers in many pubs.

Plug Plug Plug

2010-03-01T21:50:34Z

I've got another set of public training courses coming up next month. They will be held at the Imperial Hotel in Russell Square, London.

There are three one-day courses - Introduction to Perl, Intermediate Perl and Advance Perl. They're running on the 13th, 14th and 15th of April.

More details on my training web site. Hope to see some of you there.

p.s. Gabor has been kind enough to advertise the courses on his newly revamped CPAN Forum site - so the least I can do is to repay the favour and recommend that you have a look at the site.

Marketing Perl at FOSDEM

2010-02-19T13:04:15Z

It's two weeks since I went to FOSDEM and I promised to write an article about what happened there. Better do that before I forget everything.

Some time ago, Gabor applied for a Perl stand at this year's FOSDEM. The idea was that we could go along and promote Perl to people who are part of the Open Source community but not part of the Perl community.

When I first arrived at the venue, it took me some time to find the Perl stand. This was largely because I was searching in the wrong building. I forgot that FOSDEM is spread over several buildings at the ULB. I had assumed that we'd be in the main building, but we were actually in another building along with most of the stands.

The Perl Foundation had paid for some stuff for us to give away from the stand. We had some postcards listing Perl events in Europe this year and some round tuits. There were also a few other leaflets promoting particular Perl events.

I think it was unusual for a programming language to have a stand at the conference. Plenty of other projects had stands, but I didn't see any other languages. A lot of the other stands were promoting projects that they were able to demonstrate. I think it's hard to demonstrate a programming language in a situation like that.

We got a lot of people passing by the stand and many of them stopped to talk. The round tuits attracted the most attention, but it was sometimes hard to explain the joke to people whose first language wasn't English. There were at least a couple of times when I just gave up trying.

On Saturday afternoon, Juerd arrived. He brought a project with him and we set that up projecting a hastily assembled slideshow on the wall opposite us. That also drew a lot of attention to the stand. In the future I think it's a good idea to plan something like that in advance.

Just about everyone who we talked to knew about Perl. And most of them had used it at some point. Most of the people I spoke to were still using it to some extent. But very few of them knew about the "Modern Perl" projects that we were promoting (Catalyst, Moose, DBIx::Class, etc) or the huge number of Perl events that take place i Europe every year. I think we got some of them interesting in Modern Perl and I'm hoping that we'll see a few new faces at various Perl events this summer. I promised to buy a drink for some of them if they come along to YAPC::Europe. If they all take me up on it, it might get a bit expensive.

Our presence at the conference was all very experimental. We know that this is something that we want to do more of, but we're just working out the most effective approaches to take. But I think that we can count this attempt as a success and take the lessons learned forward to other non-Perl conferences. The next one on the list is CeBIT.

Other people have also written about this event: Gabor, Claudio, Erik, Salve.

META.yml and Building RPMs

2010-02-10T13:03:50Z

An email has flooded in. It was in response to my piece about Building RPMs from CPAN Distributions and it was from Andreas Koenig. Andreas runs PAUSE, which is the service CPAN authors use to upload stuff to CPAN, so he knows what he's talking about when it comes to CPAN (and many other matters). He says this:

It's not correct that the META.yml contains the exact list of dependencies. The META.yml is not the authoritative source for them. The reason behind is that dependencies do differ across architectures. Exceptions to this rule may declare dynamic_config=0. In order to obtain the real list of dependencies you must run your Makefile.PL or Build.PL. Recent Module::Build provides a MYMETA.yml after Build.PL has run. You could use that instead. MakeMaker always had the dependency as a comment in the Makefile.

He is, of course, right. My previous article skipped blithely over some of the more gnarly corners of this problem. I should point out that Gabor and I discussed some of these over the weekend but it's almost certainly worthwhile going into a little more detail.

It's true that a static META.yml file can't deal with all of the possibilities. Here are a number of examples of areas that need to be looked at in more detail.

Environmental differences
This is the area that Andreas is talking about. And the Padre problem I mentioned on Monday is one example of this. Padre runs on several different platforms. And some dependencies will only be required on certain platforms. For example the Win32::API module is only required if the module is being installed on Windows.

But it's not just different operating systems or architectures that cause problems like this. If you're trying to use Plack on a server with Apache 2 installed, you'll need Apache2::Request. If your server has Apache 1 installed you'll need Apache::Request. In each case, you won't need the request module for the Apache version that you aren't using. As things stand, the META.yml for Plack doesn't list either of the Apache request modules, but a more intelligent system could work out which one of them is required and add that one to the list of dependencies.

"Choose One" requirements
Some modules exist simply as a way of allowing the user to choose between one of a number of implementations of a feature. A good example is JSON::Any. There are (at least) three different JSON modules on CPAN - JSON, JSON::DWIW and JSON::XS). Different systems will have different ones installed. JSON::Any allows a program to use any JSON module and not care which of them is installed. But how do you model that dependency? If you make any (or all) of the supported modules a required dependency, you rather miss the whole point of the module. JSON::Any's META.yml ignores the problem and leaves it to the Makefile.PL to work out what to do. The Fedora RPM for this module takes a weird approach and makes JSON::XS a required dependency. Even if META.yml could support this mode of working, RPM doesn't have this feature.

Added features
Some modules have optional requirements. That is, if certain other modules are installed then the module gains more features. One example is the Template-XML distribution. Template-XML contain a plugin (Template::Plugin::XML) for the Template Toolkit. Template::Plugin::XML is a wrapper around a number of XML processing modules. If a particular module (for example, XML::DOM) is installed then Template::Plugin::XML allows the user to uses XML::DOM for XML processing. It works similarly for XML::LibXML, XML::Parser, XML::RSS, XML::Simple and XML::XPath. None of them are required, different functionality is turned on for each one that is installed. You don't have to configure Template::Plugin::XML at all to work with these modules. It just works if a particular module is installed. If, at a later date, you remove that module then Template::Plugin::XML removes the features supported by that module.

This seems to be somewhere where I have philosophical differences with the Fedora RPM packaging team. I believe that all of these modules should be seen as optional and there for shouldn't be listed as dependencies in the META.yml or the RPM. The Fedora team disagrees. They want each RPM to depend on all of the modules it needs in order to have as many features as possible, The Template-XML RPM therefore requires all of the XML processing modules I listed above. That seems wrong to me.

META.yml supports the concept of "recommended modules". I think that these optional modules should be listed there. But I don't believe that RPM has a similar feature.

So there are a few problems that I see with the META.yml approach. In the face of these issues I should probably back down slightly from my previous position that META.yml is the definitive way to get a list of dependencies. What I now believe is that parsing META.yml will give you a better position to start from than parsing the Perl code and extracting all of the "use" statements.

But I hadn't previously heard of the MYMETA.yml that Andreas mentioned in his email. That's certainly a way to get round the environmental differences I listed above. I don't think it solves the other two issues though.

Are there any other corner cases that I've missed. Does anyone else have any opinions on building RPMs from CPAN distributions?

Useful RPM Stuff

2010-02-09T10:43:47Z

I forgot to mention this yesterday. I've set up a github project (see http://github.com/davorg/rpm_stuff) where I'll dump bits and pieces that I'm writing to make my RPM-building life easier.

The first utility I've uploaded there is called can_rpmbuild. You pass it an RPM spec file and it tells you which of the dependencies are available from the RPM repositories that you use and which of them you'll have build yourself.

I'm sure it can be improved. Patches welcome of course. And please let me know if you find it useful.

Building RPMs from CPAN Distributions

2010-02-08T13:01:02Z

Regular readers will know that in the past I've shown some interest in building RPMs from CPAN distributions. It's been a while since I did much work in this area (although I do still release the occasional module to my RPM repository.

Over the weekend I was at FOSDEM and I attended Gabor's talk on packaging CPAN modules for Linux distributions. This has rekindled my interest in this area and I spent most of the train journey back from Brussels hacking around the area.

There's one thing that has been bothering me in particular recently. The standard RPM building mechanism (or, at least, the way it's configured in Fedora and Centos) does something incredible brain dead when trying to work out what other modules the current module depends on. It does it by parsing the source code and looking for "use" statements. This means that a module that might only be used in really obscure cases is going to be listed as a mandatory requirement for your module.

Gabor and I actually saw an example of this over the weekend when the Fedora packaging team raised a bug against Padre because it requires Win32::API. Padre, of course, only uses Win32::API when being used on Windows. And for that reason Win32::API is not listed as a dependency in its META.yml.

And that's, of course, where the RPM builders should be going to get a list of dependencies. META.yml contains the list of other modules that the author wants the module to depend on. This should be seen as the definitive list. Of course, there might be errors in that list - but that should be addressed by raising a bug against the module.

I've poked at this problem a few times, trying to work out how the RPM system parses the code and trying to replace that with code that looks at META.yml instead. But the RPM system uses a baroque system of interdependent macros and eventually they all lead to a piece of rather clunky Perl code. So each time I've approached this problem, I've backed off again.

The problem became more urgent when I wanted to package Plack for Fedora. Plack supports all sorts of hosting environments and therefore includes "use" statements loading a number of modules that most people will never use. Fedora includes Apache2, so Apache::Request (which is for Apache1) will never be available. It's not listed in META.YML, but it is used by one of the modules. The RPM build system was therefore insisting that it should be present. An impasse was reached.

Then I decided to turn the problem on its head. RPM building has two steps. You create a spec file for the RPM and then you build the RPM using the spec file and your original tarball. I started wondering if I could ensure that the spec had all of the requirements (from the META.yml). Once I'd done that I would only need to find some way to turn off the RPM build system's default behaviour.

People packaging CPAN modules for Fedora (and Centos) use a program called 'cpanspec' to generate spec files. I started digging into the code there in order to find out how to insert the list of correct dependencies.

Only to find that it has already been done. cpanspec is already doing the right thing and generating a list of 'Requires' statements from the data in META.yml.

Then all I needed to do was to see if I could turn off the (broken) default RPM build behaviour which was adding spurious extra dependencies. That proved to be easy too. It's just a case of adding %__perl_requires %{nil} to your .rpmmacros files.

So now all of my RPMs will have only the correct dependencies listed. This makes me very happy.

I suppose I should go back and rebuild all of the older ones too.

Oh, and because I've worked out a really easy way to generate this - here's a spreadsheet listing which CPAN modules are available as RPMs for Fedora. I plan to keep this list up to date (and make it much longer). [Link now fixed]

p.s. More about my trip to FOSDEM and the Perl marketing push there over the next couple of days.

Cultured Perl Blog

2010-02-02T09:29:46Z

A couple of years ago I thought that one thing the Perl community was missing was a network of blog sites about Perl. I'm not talking about the individual blogs that are being shown off to such good effect by the Iron Man project, I'm talking about a set of multi-author blogs that covered particular facets of the Perl world. Something like a Perl-specific version of LifeHacker or BoingBoing. To that end, I registered a number of domains and set about installing Movable Type.

That bit was easy. That bit I can do. The next bit is harder.

The next bit involves getting authors interested in writing for the blogs on a regular basis. That bit I didn't do so well at and none of the blogs florished.

One of them didn't even get going. That was Cultured Perl. The idea behind Cultured Perl was that it would discuss Perl culture. That's all the non-technical bits of the Perl world. Perl Mongers, Perl conferences, things like that. I had a few authors signed up, but nothing ever really happened.

So why am I telling you this? Well, the Cultured Perl domains are up for renewal. And I'm trying to work out whether it's worth keeping them.

Would you be interested in reading a Cultured Perl blog? And would you be interested in writing for it?

FOSDEM

2010-02-01T12:52:31Z

This weekend is the annual FOSDEM conference in Brussels. I really enjoy FOSDEM but, for reasons I don't really understand, this will be the first time I've been since 2005. It will also be one of the rare occasions where I attend a conference without giving a talk - the organisers turned down my proposed talk on Modern Perl.

I like FOSDEM because it's not just a Perl conference. It's about the wider open source movement. In fact Perl is a really small part of of the conference. In many years it has been completely unrepresented. One of the things I mentioned in my "M Word" talk at the London Perl Workshop was that Perl needed to be better represented at non-Perl conferences. With that in mind, the Perl Foundation has booked a stand at the conference and various volunteers (including me) will be there telling people about how wonderful Perl is.

The main driver behind this push to get Perl represented at other conferences has been Gabor Szabo and he'll also be at FOSDEM giving a couple of talks. One is a lightning talk introducing people to Padre. The other is about packaging CPAN modules for Linux distributions. Those of you with long memories might remember me talking about this at YAPC in Copenhagen. I'm hoping that attending Gabor's talk will galvanise me into having another go at my project to automatically build RPMs of many more CPAN modules than are currently available.

So, as you can see, there are plenty of good reasons to be at FOSDEM this weekend. And that's even before considering that it takes place in one of my favourite European cities. I might even treat myself to a Kwak in one of the bars on the Grand Place.

If you're at FOSDEM next weekend, please stop by the Perl stand and say hello.

Perl Web Server

2010-01-22T09:10:32Z

Coming a little late to this game, but I understand there are some concerns in the Perl community that the phrase "Perl Web Server" doesn't find the best site when you search for it in Google. I'm happy to add my vote that "Perl Web Server" should point at the Plack web site.

Market Research

2010-01-12T15:18:13Z

I hope no-one minds if I use this post to do a quick bit of market research.

I'm planning to run some more public Perl training courses in April. Until now when I've run these courses, they have been pretty generic beginners, intermediate and advanced courses. But now I'm thinking that it might be interesting to run more focussed courses.

I'm planning to keep the intermediate course (as that seems to be pretty popular). So that leaves me looking for two more courses to run. Current ideas include:

Testing
OO Perl (largely covering Moose)
Databases (largely covering DBIx::Class)
Web programming (touching on basic CGI but moving on swiftly to things like Catalyst and PSGI)

Does anyone have any opinions? What Perl courses would you like to see in London? What courses would you attend? Or, perhaps more importantly, what courses would your employers pay for you to attend?

Important SpamAssassin Update

2010-01-02T14:23:55Z

Do you run SpamAssassin to remove spam from your mailbox? Have you noticed a fall in the amount of mail you've received in the last couple of days? I mean all mail, not just spam.

You might well be falling foul of this problem. The rules in the latest version of SpamAssassin available from CPAN include this rule.

##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX      Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX      The date is grossly in the future.
##} FH_DATE_PAST_20XX

And here are the scores for that rule:

score FH_DATE_PAST_20XX 2.075 3.384 3.554 3.188 # n=2

Basically, any mail with a date header greater than 2009 is given a score of about 3. Depending on your local configuration and other rules that the mail might trigger, that might well increase the chance of legitimate mail being marked as spam.

So have a look in your spam folder. You might find some good stuff in there.

Of course, I might be the only person who has been caught by this. It's possible that you all knew of the existence of sa-update and you've all got it running in a cronjob so you're using the latest rules and not the eighteen month old set that I have. I'll be setting that up today.

This has been a public service announcement for people who are as stupid as me.

The "Without Whom"s

2009-12-24T08:32:13Z

In The "M" Word, I listed many of the things that have happened in the Perl marketing world over the last year. I wanted to end the year by mentioning a few people without whom this project would not have got off the ground at all.

Curtis "Ovid" Poe started writing about Perl's image issues on his use.perl journal back in July. But he didn't just write about it. He chaired the Marketing BOF that we held in Lisbon and he was the driving force behind setting up the TPF marketing committee. He has also been instrumental in getting the blogs.perl.org site up and runnimg.
Gábor Szabó wasn't very far behind Ovid in starting to blog about marketing. He has also been vociferous in TPF marketing committee discussions. He always has another idea for projects we can take on and often does those projects himself before anyone else has a chance to volunteer. I particularly want to draw attention to the clean-up he did on pm.org (a site full of links to dead Perl Monger groups isn't great advertising) and his work trying to increase Perl's visibility at non-Perl conferences.
Matt Trout and Mark Keating set up the Iron Man Blogging competition. To my mind this has been the single biggest success in marketing Perl this year. I'm still astonished daily by the number of people out there who are blogging about Perl. They have somewhere around two hundred blogs in the competition.
Leo Lapworth has done some amazing work dragging Perl's web sites into the 21st century. Not many weeks go by without Leo announcing that he has redesigned another web site. I'm sure I've forgotten some of his announcements, but for a sample of his work see www.perl.org, learn.perl.org, dbi.perl.org and lists.perl.org. Thanks should also go to his employers, Foxtons, who donated some of his time (and some time from a web designer too).

Of course it's true that the recent resurgence of Perl is due to the new and improved modules that you can find on CPAN. Moose, DBIx::Class, Catalyst and Plack (to name only four) are what enables Perl to retain its place in the forefront of modern programming languages. But it's the people listed above (and dozens of other who also work on marketing projects) who enable Perl news to break out of the echo chamber.

We owe these people our thanks. And, on a more practical level, we owe them beer. If you see them at a Perl Mongers meeting, at a conference or even just on the street, it is your duty to stop them and say the magic words:

"Hi. Thanks for your work. Can I buy you a beer?"

Perlanet Improvements

2009-12-15T08:40:16Z

I can be a bit of a lazy open source author at times. I love it when someone improves my code and then just emails me patches. I love it even more now that I'm using Github so that people can just fork my code and send me pull requests.

That happened over the weekend. I got a pull request from cycles saying that he'd been refactoring Perlanet and asking if I'd be interested in merging his changes. These refactorings had come out of a mini-hackathon that had been run by the North West England Perl Mongers. I knew that they had been interested in using Perlanet to power the Iron Man aggregator, but that the monolithic nature of the code made it hard for them to subclass the bits that they needed to change.

So what cycles has done is to take my code and break it down into a number of smaller functions (and a couple more classes) so that people can subclass it and override methods where they want different behaviour to the defaults that I use. I can see this leading to a little ecosystem of Perlanet subclasses where people release their favourite tweaks to CPAN. I see this as a good thing.

I liked cycles' changes so I merged them into my repository. They're available there now if you want to look. I haven't yet released a new version to CPAN as there are a few things I want to check out first.

Firstly, this new update seems to have increased the version of Moose required. With the version of Moose that I was running on my system (0.88), a lot of the tests were failing. Updating to 0.93 solved that problem. I need to work out exactly what the problem was and update the "requires" line in Build.PL appropriately.

Secondly, the newer version of Moose gave a deprecation warning when used with CHI. Updating to the latest version of CHI fixed that (but that, in turn, meant upgrading a few modules in the Log::Any family). This is all starting to get a bit too close to the bleeding edge of CPAN for a module that I want as many people as possible to use.

Finally, cycles had started to use TryCatch in the module. Not, of course, that I object to high quality exception handling in my code, but this is another module that isn't yet in general use. It's something that you won't find in a "standard" (whatever that means) Perl installation.

I'm in the process of building RPMs of all of the missing modules (or later versions for the modules where the Fedora/Centos build is just lagging CPAN a bit). They'll be available from rpm.mag-sol.com in the next few days.

Currently I'm leaning towards just releasing the new version and hoping that the people who want to use it will have enough enthusiasm that they won't complain about the updated and new modules that are required. But I thought it would be interesting to ask for your opinions too.

As a user of CPAN modules, how do you decide when a module is too cutting edge for you to use? Do you just install newer versions of modules automatically when an installation asks you too? Or are you a little more careful than that? Would the constraints in this latest version of Perlanet prevent you from using it?

And as a writer of CPAN modules, how cutting edge do you allow yourself to be? Are you happy to release stuff that only works with the very latest versions of Moose or other fast-moving modules? Or do you like to ensure that your stuff is usable by people who might be a little behind the curve?

I should make it clear that I'm very grateful for the work that cycles did and I'm not disparaging his efforts at all. I'm just dithering a bit about how cutting edge I want to be.

The "M" Word

2009-12-06T18:52:08Z

Yesterday was the London Perl Workshop. As always it was a fabulous day packed full of great talks about Perl. Thanks to the organisers for all the work they put in.

I gave the keynote speech first thing in the morning. The talk was called The "M" Word and it was an overview of how the Perl community has started to get to grips with the problem of marketing over the last year.

Here are the slides:

The "M" Word

View more presentations from Dave Cross.

Blogs.perl.org Problems

2009-11-30T08:27:08Z

It's ten days since we announced blogs.perl.org and it still doesn't work properly.

Our mistake was to launch it on a really underpowered server. It worked fine when there were half a dozen of us testing it out, but when the entire Perl community descended on it to take a look at the site (and to sign up for accounts) the server collapsed under the strain.

All of the problems that I've been told about so far have been down to memory allocation issues. The server has 512 MB (corrected from KB - it's bad, but not that bad!) of RAM and it really needs four times that.

Things have calmed down over the last few days though. I suspect that's largely because people have lost interest in the site and have given up trying to register. We've also switched all of the MT processes to using FCGI. I see that a few people have registered successfully and are posting entries on the site. I salute their persistence.

We have a plan for fixing the situation, however, and that will be put in place over the next week or so. The site will be moving to a new server with a more appropriate hardware configuration. We hope to minimise the amount of downtime whilst this happens.

I can only apologise, once more, for the problems. I hope that once the new server is up and running, you'll all consider giving blogs.perl.org another try.

I hope it's obvious, but none of these problems should be taken as a reflection on the quality of Movable Type or the work that Six Apart have done for us on the site. The problems all come down to capacity planning on the part of the people running the project.

Which reminds me - if you're ever looking for someone to do capacity planning on your web site, I'm not the right person for the job!