Perl Hacks

Modern Perl at OpenTech

2010-07-06T08:13:51Z

I mentioned a few months ago that I'd be running an "Introduction to Modern Perl" training course at YAPC::Europe this year. But in the interests of speaking outside of the Perl community as much as possible, I'm also going to be giving a slightly different version of that course at the OpenTech conference in London in September.

I say "slightly different", but that's a bit of an understatement. The original training course runs for six hours. The OpenTech talk is twenty minutes. But hopefully that will be long enough to introduce some people to many of the interesting things that are going on in the Perl world.

It you're going to be in London in September, then the OpenTech conference is always a lot of fun. I highly recommend that you come along. It's cheap too - just a fiver on the door.

YAPC::Europe Talks Accepted

2010-06-23T07:49:25Z

The YAPC::Europe organisers said that they would tell speakers which talks had been accepted on July 1st. Well, it seems that the excitement was too much for them and they decided to do it a week earlier. Yesterday I got email telling me that some of my talks had been accepted and the list of accepted talks is now on the web site. As always, it looks like a really interesting conference.

I've had two twenty-minute talks accepted:

The Perl Community
The Perl community is a complex and interesting city. I've been exploring it for almost fifteen years and I'm not sure that I've been to every corner of it.

In this talk I'll attempt to guide you round some of the more interesting and useful parts of the Perl community. I'll point out some ancient monuments, some nice new areas and warn you about some places where you really shouldn't walk alone after dark.

Things I Learned From Having Users‎
When I first started releasing modules to CPAN it was great. I released modules that no-one used. I could release new versions as and when I wanted to.

Then people started using a couple of my modules. I started to get email about them. Suddenly my modules were no longer just for me. I had to deal with users.

In this talk I'll discuss how having users effects the way that you develop and release software. I'll also look at a few ways to keep on top of things.

I'm also doing my first lightning talk for several years:

‎Perl Vogue‎

You might not believe it to look at us, but the Perl community is a deeply fashionable place. If you're not using the currently fashionable modules in your code then people will be sneering at you behind your back.

Join Dave Cross for a quick review of the history of Perl fashion.

See you in Pisa.

YAPC::Europe Talks

2010-06-11T15:05:54Z

The Call for Papers for YAPC::Europe closes in four days. If you're thinking of giving a talk in Pisa then this weekend would be a very good time to give it some serious thought.

I proposed four talks last night. I'm hoping that the organisers won't choose more than two of them, but I like to give them a bit of choice. The titles are as follows:

The Perl Community
Web Services for Fun and Profit
Things I Learned From Having Users
Perlanet Update

I've also submitted a proposal for a lightning talk called "Perl Fashion". It's been several years since I've given a lightning talk, so that'll be an interesting experience.

Don't forget that there are also training courses (including my course on Modern Perl).

Looking forward to the conference very much. Hope to see some of you there.

How Not to Ask a Question

2010-06-10T12:34:34Z

I received this email last week. I often get random email from people I don't know asking for help with Perl problems and I'm happy to help whenever I can (although I always point our that Perl Monks is going to get better answers in less time).

But if you're going to ask random strangers for help, you should probably make a bit more effort than this person did. I've reformatted it and corrected the English.

Hello sir,

I am new developer in perl scripting language using MVC Frame works, my doubt is how to write below query in MVC.
$query = "select belarc_update_dt
           from   Device
           where  belarc_update_dt  >  $expiry_date
           AND    belarc_update_dt <= $current_date
           AND    (scrapped_on > $current_date
                   OR scrapped_on = '0000-00-00')";
the above query how to write in mvc frameork can we please help me sir,

Thanks and Regards

How would you reply to mail like that?

Update: I asked which MVC framework he was using. He replied:

I am using CGI::Carp 'fatalsToBrowser' with linux shell.

At that point I gave up.

Ironman and XML::Feed

2010-04-26T08:02:04Z

Sam Graham complains that since the Ironman feed switched to using Perlanet, the entries have been "mangled". By that he means that in some cases any HTML in feed entries is lost.

I think they're running up against this bug in XML::Feed (which is one of the modules that Perlanet uses to process the feeds it subscribes to). There's a patch in the bug report that I've applied to my local installation of XML::Feed and it seems to have fixed the problem.

Until the author releases a new version of XML::Feed that includes this patch, I recommend that anyone using XML::Feed (and that includes everyone using Perlanet) applies the patch for themselves.

More RPM Stuff

2010-04-25T16:02:17Z

It's been a while since I wrote anything here. if anyone is keeping score I've probably failed the Iron Man challenge of posting something every ten days.

Don't have much to add here either but I thought some of you might be interested in a quick tweak I made to my spreadsheet of CPAN RPMs available for Fedora. It now lists all of the RPMs available across all of the repositories that I use and shows you which version of the module is available. I've also added the current CPAN version for all of these modules.

This gives me the information I need to do a few things that I've wanted to do for a while. In particular I should be able to script the automatic removal of RPMs from my repository when the official Fedora repository catches up with the version I'm carrying. I can also easily identify CPAN modules where the latest Fedora version (from any of the repositories) is lagging behind the CPAN version.

As always, the code is available on Github and patches are very welcome.

Update: And here's another spreadsheet covering CPAN RPMs available for Centos.

Training Update

2010-04-01T10:14:23Z

The training courses for this summer's YAPC in Pisa have been announced. And my course on Modern Perl has been chosen. It's a one-day course on August 2th (just before the conference). It costs € 180. You'll be able to book once the payments system on the conference web site goes live.

Here's the description of the course from the YAPC site:

This course introduces the major building blocks of modern Perl. We'll be looking at a number of CPAN modules that can make your Perl programming life far more productive.

The major tools that we will cover will be:

Template Toolkit

DBIx::Class

Moose

Catalyst

Plack

We'll also look at some other modules including autodie, DateTime and TryCatch.

There are several other good courses running both before and after the conference. I'm sure there'll be something that you'll find interesting.

N.B: This is not an April Fool's joke!

Perl Books

2010-03-30T07:27:49Z

Those of you who have been following my work for some time might remember that many years ago I took some interest in Perl books. I don't mean the kind of books that we all read (or, even write) with animals on the cover. I mean the kinds of terrible Perl books that at one stage seemed to take up over 90% of the space given over to Perl books in most bookshops. They had brightly coloured covers and made unlikely claims about how good you would be with Perl after an unfeasibly short period of time.

They all seemed to be written by people who didn't actually know anything about Perl and they were full of the most atrocious Perl code that I've ever seen. Of course they were all not just about Perl but were about CGI programming with Perl (as that's what everyone wanted to use Perl for ten years ago). It was easy to tell the quality of these books from the index. You just looked for mentions of "use strict", "use warnings" or CGI.pm. Most books ignored them completely. At the time Schwern had a "litmus test" for the quality of Perl books. It was list of bullet points that you could use to easily calculate how bad a Perl book was. I seems to have dropped off the internet at some point in the last ten years.

Last weekend I was in the centre of London and I went into a couple of large book shops that I haven't been to for some time. And it looks to me as though this problem has finally gone away. Looking at the Perl section in Foyles, for example, I saw a very few dodgy Perl books. Of course, there were far fewer Perl books of any quality than there would have been five or six years ago, but the ones that were left were pretty much all decent books.

My initial thought was that this was just another sign that Perl has lost the battle for the low-end web development market. People who want to hack up a web site no longer even consider using CGI programs written in Perl. These days those people would all use PHP. But it seemed to me that there weren't many "Learn PHP in 24 Hours" books on the shelves either. So perhaps the low-end web development market doesn't exist any more. Or, at least, perhaps it's not big enough to make it profitable enough for publishers.

I'll admit that my sample wasn't particularly big. And Foyles isn't exactly your average book shop. It caters for the higher end of the book-buying market. I wanted to test my hypothesis by looking at the computer books section in Borders over the road. But Borders has closed down. Another indication of the diminishing size of the book market.

I'm going to take this investigation further. I'll check out the computer section in some lower-end book shops. But just to give myself some closure I'm going to declare my campaign against bad Perl books to be over. We won in the end. But it may be a slightly Pyrrhic victory.

Training in Pisa

2010-03-21T16:16:44Z

The YAPC::Europe organisers have put out a call for training courses. They want a number of courses to offer to attendees in the couple of days before the conference. The call closed yesterday and I expect they'll be announcing the courses in a couple of weeks.

I've given training courses at the last couple of YAPC::Europes so I've sent in a proposal for a course. I'm hoping to run a new course called "An Introduction to Modern Perl". In this course we'll be looking at some of the tools that form the basis of all modern Perl programming. This will include Template Toolkit, Moose, DBIx::Class, Catalyst and Plack.

I hope that this sounds interesting to some of you and that you'll consider attending the course (if it gets accepted). Please keep an eye on the conference web site to see when the courses are announced.

And please consider holding off booking your travel and hotel until you've seen the selection of courses that will be offered before the conference.

I suppose now I should start thinking about some talks to give at the conference.

Crufty Old Perl

2010-03-11T18:05:36Z

It's eighteen months since I wrote "Why Corporates Hate Perl" and it's worth pointing out that the company I discussed in that article which was dropping Perl in favour of PHP and Java is still employing as many good Perl programmers as it can find.

I talked in that article about some rather unsubtle social engineering that had been going on at that company. Management had started to talk about Perl as a "legacy language" in an attempt to persuade the business that they really don't want their applications written in Perl. That doesn't seem to have been as successful as I feared it would be.

But there's another kind of social engineering that I've seen going on. And this is happening at the developer level. I've lost count of the number of times I've been sitting in meetings with developers, discussing ways to improve the quality of crufty old Perl code when someone throws in the (more than half-serious) suggestion that we should just rewrite the whole thing using Rails or Django.

There seems to be this idea that switching to a new framework in a new language will act as some time of magic bullet and that suddenly all of our problems will be solved. There are so many ways that this perception is flawed, Here are my two favourites.

The current system is old and crufty not because it's written in Perl, but because it was written by people who didn't understand the business requirements fully, didn't know their tools particularly well or were under pressure to deliver on a timescale that didn't give them time to design the system properly. Often it's a combination of all three. Given the time to rewrite the system from scratch, of course it will be better. But it will be better because the business is better understood and tools and techniques have been improved - not because it's no longer written in Perl.
Frameworks in other languages are not easier to use or more powerful than frameworks in Perl. Anything that you can do with Rails or Django you can do just as easily with Catalyst. It's using a framework that's the big win here, not the particular framework that you use. Sure, if you're a Ruby house then using a Ruby framework will probably match your existing developers' skills more closely but if your current system is written in Perl then, hopefully, you have a team of people with Perl skills and that's going to be the language you'll want to look at.

I'm tired of Perl being seen as a second-class citizen in the dynamic languages world. I freely admit that there's a lot of bad Perl code out there (I'll even admit to writing some of it) but it's not bad Perl code because it's Perl, it's bad Perl code because it's bad code.

This is what the Perl marketing initiative is for. We need people to know that Perl is not the same language that they stopped using ten years ago. Modern Perl can compete with any of the features of any other dynamic language.

By al means, try to get the time to rewrite your crufty old systems. But rewrite them in Modern Perl. You'll enjoy it and you'll be really productive.

p.s. I should point out that I'm not talking about any specific client here. This is based on conversations that I've had at various companies over the last couple of years and also discussions with many developers in many pubs.

Plug Plug Plug

2010-03-01T21:50:34Z

I've got another set of public training courses coming up next month. They will be held at the Imperial Hotel in Russell Square, London.

There are three one-day courses - Introduction to Perl, Intermediate Perl and Advance Perl. They're running on the 13th, 14th and 15th of April.

More details on my training web site. Hope to see some of you there.

p.s. Gabor has been kind enough to advertise the courses on his newly revamped CPAN Forum site - so the least I can do is to repay the favour and recommend that you have a look at the site.

Marketing Perl at FOSDEM

2010-02-19T13:04:15Z

It's two weeks since I went to FOSDEM and I promised to write an article about what happened there. Better do that before I forget everything.

Some time ago, Gabor applied for a Perl stand at this year's FOSDEM. The idea was that we could go along and promote Perl to people who are part of the Open Source community but not part of the Perl community.

When I first arrived at the venue, it took me some time to find the Perl stand. This was largely because I was searching in the wrong building. I forgot that FOSDEM is spread over several buildings at the ULB. I had assumed that we'd be in the main building, but we were actually in another building along with most of the stands.

The Perl Foundation had paid for some stuff for us to give away from the stand. We had some postcards listing Perl events in Europe this year and some round tuits. There were also a few other leaflets promoting particular Perl events.

I think it was unusual for a programming language to have a stand at the conference. Plenty of other projects had stands, but I didn't see any other languages. A lot of the other stands were promoting projects that they were able to demonstrate. I think it's hard to demonstrate a programming language in a situation like that.

We got a lot of people passing by the stand and many of them stopped to talk. The round tuits attracted the most attention, but it was sometimes hard to explain the joke to people whose first language wasn't English. There were at least a couple of times when I just gave up trying.

On Saturday afternoon, Juerd arrived. He brought a project with him and we set that up projecting a hastily assembled slideshow on the wall opposite us. That also drew a lot of attention to the stand. In the future I think it's a good idea to plan something like that in advance.

Just about everyone who we talked to knew about Perl. And most of them had used it at some point. Most of the people I spoke to were still using it to some extent. But very few of them knew about the "Modern Perl" projects that we were promoting (Catalyst, Moose, DBIx::Class, etc) or the huge number of Perl events that take place i Europe every year. I think we got some of them interesting in Modern Perl and I'm hoping that we'll see a few new faces at various Perl events this summer. I promised to buy a drink for some of them if they come along to YAPC::Europe. If they all take me up on it, it might get a bit expensive.

Our presence at the conference was all very experimental. We know that this is something that we want to do more of, but we're just working out the most effective approaches to take. But I think that we can count this attempt as a success and take the lessons learned forward to other non-Perl conferences. The next one on the list is CeBIT.

Other people have also written about this event: Gabor, Claudio, Erik, Salve.

META.yml and Building RPMs

2010-02-10T13:03:50Z

An email has flooded in. It was in response to my piece about Building RPMs from CPAN Distributions and it was from Andreas Koenig. Andreas runs PAUSE, which is the service CPAN authors use to upload stuff to CPAN, so he knows what he's talking about when it comes to CPAN (and many other matters). He says this:

It's not correct that the META.yml contains the exact list of dependencies. The META.yml is not the authoritative source for them. The reason behind is that dependencies do differ across architectures. Exceptions to this rule may declare dynamic_config=0. In order to obtain the real list of dependencies you must run your Makefile.PL or Build.PL. Recent Module::Build provides a MYMETA.yml after Build.PL has run. You could use that instead. MakeMaker always had the dependency as a comment in the Makefile.

He is, of course, right. My previous article skipped blithely over some of the more gnarly corners of this problem. I should point out that Gabor and I discussed some of these over the weekend but it's almost certainly worthwhile going into a little more detail.

It's true that a static META.yml file can't deal with all of the possibilities. Here are a number of examples of areas that need to be looked at in more detail.

Environmental differences
This is the area that Andreas is talking about. And the Padre problem I mentioned on Monday is one example of this. Padre runs on several different platforms. And some dependencies will only be required on certain platforms. For example the Win32::API module is only required if the module is being installed on Windows.

But it's not just different operating systems or architectures that cause problems like this. If you're trying to use Plack on a server with Apache 2 installed, you'll need Apache2::Request. If your server has Apache 1 installed you'll need Apache::Request. In each case, you won't need the request module for the Apache version that you aren't using. As things stand, the META.yml for Plack doesn't list either of the Apache request modules, but a more intelligent system could work out which one of them is required and add that one to the list of dependencies.

"Choose One" requirements
Some modules exist simply as a way of allowing the user to choose between one of a number of implementations of a feature. A good example is JSON::Any. There are (at least) three different JSON modules on CPAN - JSON, JSON::DWIW and JSON::XS). Different systems will have different ones installed. JSON::Any allows a program to use any JSON module and not care which of them is installed. But how do you model that dependency? If you make any (or all) of the supported modules a required dependency, you rather miss the whole point of the module. JSON::Any's META.yml ignores the problem and leaves it to the Makefile.PL to work out what to do. The Fedora RPM for this module takes a weird approach and makes JSON::XS a required dependency. Even if META.yml could support this mode of working, RPM doesn't have this feature.

Added features
Some modules have optional requirements. That is, if certain other modules are installed then the module gains more features. One example is the Template-XML distribution. Template-XML contain a plugin (Template::Plugin::XML) for the Template Toolkit. Template::Plugin::XML is a wrapper around a number of XML processing modules. If a particular module (for example, XML::DOM) is installed then Template::Plugin::XML allows the user to uses XML::DOM for XML processing. It works similarly for XML::LibXML, XML::Parser, XML::RSS, XML::Simple and XML::XPath. None of them are required, different functionality is turned on for each one that is installed. You don't have to configure Template::Plugin::XML at all to work with these modules. It just works if a particular module is installed. If, at a later date, you remove that module then Template::Plugin::XML removes the features supported by that module.

This seems to be somewhere where I have philosophical differences with the Fedora RPM packaging team. I believe that all of these modules should be seen as optional and there for shouldn't be listed as dependencies in the META.yml or the RPM. The Fedora team disagrees. They want each RPM to depend on all of the modules it needs in order to have as many features as possible, The Template-XML RPM therefore requires all of the XML processing modules I listed above. That seems wrong to me.

META.yml supports the concept of "recommended modules". I think that these optional modules should be listed there. But I don't believe that RPM has a similar feature.

So there are a few problems that I see with the META.yml approach. In the face of these issues I should probably back down slightly from my previous position that META.yml is the definitive way to get a list of dependencies. What I now believe is that parsing META.yml will give you a better position to start from than parsing the Perl code and extracting all of the "use" statements.

But I hadn't previously heard of the MYMETA.yml that Andreas mentioned in his email. That's certainly a way to get round the environmental differences I listed above. I don't think it solves the other two issues though.

Are there any other corner cases that I've missed. Does anyone else have any opinions on building RPMs from CPAN distributions?

Useful RPM Stuff

2010-02-09T10:43:47Z

I forgot to mention this yesterday. I've set up a github project (see http://github.com/davorg/rpm_stuff) where I'll dump bits and pieces that I'm writing to make my RPM-building life easier.

The first utility I've uploaded there is called can_rpmbuild. You pass it an RPM spec file and it tells you which of the dependencies are available from the RPM repositories that you use and which of them you'll have build yourself.

I'm sure it can be improved. Patches welcome of course. And please let me know if you find it useful.

Building RPMs from CPAN Distributions

2010-02-08T13:01:02Z

Regular readers will know that in the past I've shown some interest in building RPMs from CPAN distributions. It's been a while since I did much work in this area (although I do still release the occasional module to my RPM repository.

Over the weekend I was at FOSDEM and I attended Gabor's talk on packaging CPAN modules for Linux distributions. This has rekindled my interest in this area and I spent most of the train journey back from Brussels hacking around the area.

There's one thing that has been bothering me in particular recently. The standard RPM building mechanism (or, at least, the way it's configured in Fedora and Centos) does something incredible brain dead when trying to work out what other modules the current module depends on. It does it by parsing the source code and looking for "use" statements. This means that a module that might only be used in really obscure cases is going to be listed as a mandatory requirement for your module.

Gabor and I actually saw an example of this over the weekend when the Fedora packaging team raised a bug against Padre because it requires Win32::API. Padre, of course, only uses Win32::API when being used on Windows. And for that reason Win32::API is not listed as a dependency in its META.yml.

And that's, of course, where the RPM builders should be going to get a list of dependencies. META.yml contains the list of other modules that the author wants the module to depend on. This should be seen as the definitive list. Of course, there might be errors in that list - but that should be addressed by raising a bug against the module.

I've poked at this problem a few times, trying to work out how the RPM system parses the code and trying to replace that with code that looks at META.yml instead. But the RPM system uses a baroque system of interdependent macros and eventually they all lead to a piece of rather clunky Perl code. So each time I've approached this problem, I've backed off again.

The problem became more urgent when I wanted to package Plack for Fedora. Plack supports all sorts of hosting environments and therefore includes "use" statements loading a number of modules that most people will never use. Fedora includes Apache2, so Apache::Request (which is for Apache1) will never be available. It's not listed in META.YML, but it is used by one of the modules. The RPM build system was therefore insisting that it should be present. An impasse was reached.

Then I decided to turn the problem on its head. RPM building has two steps. You create a spec file for the RPM and then you build the RPM using the spec file and your original tarball. I started wondering if I could ensure that the spec had all of the requirements (from the META.yml). Once I'd done that I would only need to find some way to turn off the RPM build system's default behaviour.

People packaging CPAN modules for Fedora (and Centos) use a program called 'cpanspec' to generate spec files. I started digging into the code there in order to find out how to insert the list of correct dependencies.

Only to find that it has already been done. cpanspec is already doing the right thing and generating a list of 'Requires' statements from the data in META.yml.

Then all I needed to do was to see if I could turn off the (broken) default RPM build behaviour which was adding spurious extra dependencies. That proved to be easy too. It's just a case of adding %__perl_requires %{nil} to your .rpmmacros files.

So now all of my RPMs will have only the correct dependencies listed. This makes me very happy.

I suppose I should go back and rebuild all of the older ones too.

Oh, and because I've worked out a really easy way to generate this - here's a spreadsheet listing which CPAN modules are available as RPMs for Fedora. I plan to keep this list up to date (and make it much longer). [Link now fixed]

p.s. More about my trip to FOSDEM and the Perl marketing push there over the next couple of days.