Monthly Archives: February 2010

Marketing Perl at FOSDEM

It’s two weeks since I went to FOSDEM and I promised to write an article about what happened there. Better do that before I forget everything.

Some time ago, Gabor applied for a Perl stand at this year’s FOSDEM. The idea was that we could go along and promote Perl to people who are part of the Open Source community but not part of the Perl community.

When I first arrived at the venue, it took me some time to find the Perl stand. This was largely because I was searching in the wrong building. I forgot that FOSDEM is spread over several buildings at the ULB. I had assumed that we’d be in the main building, but we were actually in another building along with most of the stands.

The Perl Foundation had paid for some stuff for us to give away from the stand. We had some postcards listing Perl events in Europe this year and some round tuits. There were also a few other leaflets promoting particular Perl events.

I think it was unusual for a programming language to have a stand at the conference. Plenty of other projects had stands, but I didn’t see any other languages. A lot of the other stands were promoting projects that they were able to demonstrate. I think it’s hard to demonstrate a programming language in a situation like that.

We got a lot of people passing by the stand and many of them stopped to talk. The round tuits attracted the most attention, but it was sometimes hard to explain the joke to people whose first language wasn’t English. There were at least a couple of times when I just gave up trying.

On Saturday afternoon, Juerd arrived. He brought a projector with him and we set that up projecting a hastily assembled slideshow on the wall opposite us. That also drew a lot of attention to the stand. In the future I think it’s a good idea to plan something like that in advance.

Just about everyone who we talked to knew about Perl. And most of them had used it at some point. Most of the people I spoke to were still using it to some extent. But very few of them knew about the “Modern Perl” projects that we were promoting (Catalyst, Moose, DBIx::Class, etc) or the huge number of Perl events that take place i Europe every year. I think we got some of them interesting in Modern Perl and I’m hoping that we’ll see a few new faces at various Perl events this summer. I promised to buy a drink for some of them if they come along to YAPC::Europe. If they all take me up on it, it might get a bit expensive.

Our presence at the conference was all very experimental. We know that this is something that we want to do more of, but we’re just working out the most effective approaches to take. But I think that we can count this attempt as a success and take the lessons learned forward to other non-Perl conferences. The next one on the list is CeBIT.

Other people have also written about this event: Gabor, Claudio, Erik, Salve.

META.yml and Building RPMs

An email has flooded in. It was in response to my piece about Building RPMs from CPAN Distributions and it was from Andreas Koenig. Andreas runs PAUSE, which is the service CPAN authors use to upload stuff to CPAN, so he knows what he’s talking about when it comes to CPAN (and many other matters). He says this:

It’s not correct that the META.yml contains the exact list of dependencies. The META.yml is not the authoritative source for them. The reason behind is that dependencies do differ across architectures. Exceptions to this rule may declare dynamic_config=0. In order to obtain the real list of dependencies you must run your Makefile.PL or Build.PL. Recent Module::Build provides a MYMETA.yml after Build.PL has run. You could use that instead. MakeMaker always had the dependency as a comment in the Makefile.

He is, of course, right. My previous article skipped blithely over some of the more gnarly corners of this problem. I should point out that Gabor and I discussed some of these over the weekend but it’s almost certainly worthwhile going into a little more detail.

It’s true that a static META.yml file can’t deal with all of the possibilities. Here are a number of examples of areas that need to be looked at in more detail.

Environmental differences
This is the area that Andreas is talking about. And the Padre problem I mentioned on Monday is one example of this. Padre runs on several different platforms. And some dependencies will only be required on certain platforms. For example the Win32::API module is only required if the module is being installed on Windows.

But it’s not just different operating systems or architectures that cause problems like this. If you’re trying to use Plack on a server with Apache 2 installed, you’ll need Apache2::Request. If your server has Apache 1 installed you’ll need Apache::Request. In each case, you won’t need the request module for the Apache version that you aren’t using. As things stand, the META.yml for Plack doesn’t list either of the Apache request modules, but a more intelligent system could work out which one of them is required and add that one to the list of dependencies.

“Choose One” requirements
Some modules exist simply as a way of allowing the user to choose between one of a number of implementations of a feature. A good example is JSON::Any. There are (at least) three different JSON modules on CPAN – JSON, JSON::DWIW and JSON::XS). Different systems will have different ones installed. JSON::Any allows a program to use any JSON module and not care which of them is installed. But how do you model that dependency? If you make any (or all) of the supported modules a required dependency, you rather miss the whole point of the module. JSON::Any’s META.yml ignores the problem and leaves it to the Makefile.PL to work out what to do. The Fedora RPM for this module takes a weird approach and makes JSON::XS a required dependency. Even if META.yml could support this mode of working, RPM doesn’t have this feature.

Added features
Some modules have optional requirements. That is, if certain other modules are installed then the module gains more features. One example is the Template-XML distribution. Template-XML contain a plugin (Template::Plugin::XML) for the Template Toolkit. Template::Plugin::XML is a wrapper around a number of XML processing modules. If a particular module (for example, XML::DOM) is installed then Template::Plugin::XML allows the user to uses XML::DOM for XML processing. It works similarly for XML::LibXML, XML::Parser, XML::RSS, XML::Simple and XML::XPath. None of them are required, different functionality is turned on for each one that is installed. You don’t have to configure Template::Plugin::XML at all to work with these modules. It just works if a particular module is installed. If, at a later date, you remove that module then Template::Plugin::XML removes the features supported by that module.

This seems to be somewhere where I have philosophical differences with the Fedora RPM packaging team. I believe that all of these modules should be seen as optional and there for shouldn’t be listed as dependencies in the META.yml or the RPM. The Fedora team disagrees. They want each RPM to depend on all of the modules it needs in order to have as many features as possible, The Template-XML RPM therefore requires all of the XML processing modules I listed above. That seems wrong to me.

META.yml supports the concept of  “recommended modules”. I think that these optional modules should be listed there. But I don’t believe that RPM has a similar feature.

So there are a few problems that I see with the META.yml approach. In the face of these issues I should probably back down slightly from my previous position that META.yml is the definitive way to get a list of dependencies. What I now believe is that parsing META.yml will give you a better position to start from than parsing the Perl code and extracting all of the “use” statements.

But I hadn’t previously heard of the MYMETA.yml that Andreas mentioned in his email. That’s certainly a way to get round the environmental differences I listed above. I don’t think it solves the other two issues though.

Are there any other corner cases that I’ve missed. Does anyone else have any opinions on building RPMs from CPAN distributions?

Useful RPM Stuff

I forgot to mention this yesterday. I’ve set up a github project (see http://github.com/davorg/rpm_stuff) where I’ll dump bits and pieces that I’m writing to make my RPM-building life easier.

The first utility I’ve uploaded there is called can_rpmbuild. You pass it an RPM spec file and it tells you which of the dependencies are available from the RPM repositories that you use and which of them you’ll have build yourself.

I’m sure it can be improved. Patches welcome of course. And please let me know if you find it useful.

Building RPMs from CPAN Distributions

Regular readers will know that in the past I’ve shown some interest in building RPMs from CPAN distributions. It’s been a while since I did much work in this area (although I do still release the occasional module to my RPM repository.

Over the weekend I was at FOSDEM and I attended Gabor’s talk on packaging CPAN modules for Linux distributions. This has rekindled my interest in this area and I spent most of the train journey back from Brussels hacking around the area.

There’s one thing that has been bothering me in particular recently. The standard RPM building mechanism (or, at least, the way it’s configured in Fedora and Centos) does something incredible brain dead when trying to work out what other modules the current module depends on. It does it by parsing the source code and looking for “use” statements. This means that a module that might only be used in really obscure cases is going to be listed as a mandatory requirement for your module.

Gabor and I actually saw an example of this over the weekend when the Fedora packaging team raised a bug against Padre because it requires Win32::API. Padre, of course, only uses Win32::API when being used on Windows. And for that reason Win32::API is not listed as a dependency in its META.yml.

And that’s, of course, where the RPM builders should be going to get a list of dependencies. META.yml contains the list of other modules that the author wants the module to depend on. This should be seen as the definitive list. Of course, there might be errors in that list – but that should be addressed by raising a bug against the module.

I’ve poked at this problem a few times, trying to work out how the RPM system parses the code and trying to replace that with code that looks at META.yml instead. But the RPM system uses a baroque system of interdependent macros and eventually they all lead to a piece of rather clunky Perl code. So each time I’ve approached this problem, I’ve backed off again.

The problem became more urgent when I wanted to package Plack for Fedora. Plack supports all sorts of hosting environments and therefore includes “use” statements loading a number of modules that most people will never use. Fedora includes Apache2, so Apache::Request (which is for Apache1) will never be available. It’s not listed in META.YML, but it is used by one of the modules. The RPM build system was therefore insisting that it should be present. An impasse was reached.

Then I decided to turn the problem on its head. RPM building has two steps. You create a spec file for the RPM and then you build the RPM using the spec file and your original tarball. I started wondering if I could ensure that the spec had all of the requirements (from the META.yml). Once I’d done that I would only need to find some way to turn off the RPM build system’s default behaviour.

People packaging CPAN modules for Fedora (and Centos) use a program called ‘cpanspec’ to generate spec files. I started digging into the code there in order to find out how to insert the list of correct dependencies.

Only to find that it has already been done. cpanspec is already doing the right thing and generating a list of ‘Requires’ statements from the data in META.yml.

Then all I needed to do was to see if I could turn off the (broken) default RPM build behaviour which was adding spurious extra dependencies. That proved to be easy too. It’s just a case of adding %__perl_requires %{nil} to your .rpmmacros files.

So now all of my RPMs will have only the correct dependencies listed. This makes me very happy.

I suppose I should go back and rebuild all of the older ones too.

Oh, and because I’ve worked out a really easy way to generate this – here’s a spreadsheet listing which CPAN modules are available as RPMs for Fedora. I plan to keep this list up to date (and make it much longer). [Link now fixed]

p.s. More about my trip to FOSDEM and the Perl marketing push there over the next couple of days.

Cultured Perl Blog

A couple of years ago I thought that one thing the Perl community was missing was a network of blog sites about Perl. I’m not talking about the individual blogs that are being shown off to such good effect by the Iron Man project, I’m talking about a set of multi-author blogs that covered particular facets of the Perl world. Something like a Perl-specific version of LifeHacker or BoingBoing. To that end, I registered a number of domains and set about installing Movable Type.

That bit was easy. That bit I can do. The next bit is harder.

The next bit involves getting authors interested in writing for the blogs on a regular basis. That bit I didn’t do so well at and none of the blogs florished.

One of them didn’t even get going. That was Cultured Perl. The idea behind Cultured Perl was that it would discuss Perl culture. That’s all the non-technical bits of the Perl world. Perl Mongers, Perl conferences, things like that. I had a few authors signed up, but nothing ever really happened.

So why am I telling you this? Well, the Cultured Perl domains are up for renewal. And I’m trying to work out whether it’s worth keeping them.

Would you be interested in reading a Cultured Perl blog? And would you be interested in writing for it?

FOSDEM

This weekend is the annual FOSDEM conference in Brussels. I really enjoy FOSDEM but, for reasons I don’t really understand, this will be the first time I’ve been since 2005. It will also be one of the rare occasions where I attend a conference without giving a talk – the organisers turned down my proposed talk on Modern Perl.

I like FOSDEM because it’s not just a Perl conference. It’s about the wider open source movement. In fact Perl is a really small part of of the conference. In many years it has been completely unrepresented. One of the things I mentioned in my “M Word” talk at the London Perl Workshop was that Perl needed to be better represented at non-Perl conferences. With that in mind, the Perl Foundation has booked a stand at the conference and various volunteers (including me) will be there telling people about how wonderful Perl is.

The main driver behind this push to get Perl represented at other conferences has been Gabor Szabo and he’ll also be at FOSDEM giving a couple of talks. One is a lightning talk introducing people to Padre. The other is about packaging CPAN modules for Linux distributions. Those of you with long memories might remember me talking about this at YAPC in Copenhagen. I’m hoping that attending Gabor’s talk will galvanise me into having another go at my project to automatically build RPMs of many more CPAN modules than are currently available.

So, as you can see, there are plenty of good reasons to be at FOSDEM this weekend. And that’s even before considering that it takes place in one of my favourite European cities. I might even treat myself to a Kwak in one of the bars on the Grand Place.

If you’re at FOSDEM next weekend, please stop by the Perl stand and say hello.