London Perl Workshop 2015

This time next week we will all be enjoying the London Perl Workshop. I thought it was worth looking at what the day has in store.

As always (well, except that one time when they had no power) the LPW will take place at the Cavendish Campus of the University of Westminster. I’m told there are exams or something like that taking place on the same day, so it’s important to follow the signs when you get there or you might end up in the wrong place being forced to take an exam.

The workshop starts at 9am, but registration queues can be quite long, so I’d recommend getting there half an hour or so earlier than that. If you get lucky and register quickly, then why not look for an organiser and volunteer to help out for a while.

You’ll want to be in the main room for the welcome address at 9am – just in case there’s any important news about the day. But the talks start at 9:10.

My ‎Modern Perl Web Development‎ course starts then. Hopefully it will be in my usual classroom. Alteratively, Andrew Solomon’s Crash course on Perl, the Universe and Everything‎ starts at the same time and goes on much longer. Or you might want to see some shorter courses. If I wasn’t running my training, I’d want to see Tom Hukins talking about ‎Escaping Insanity‎ and Rick Deller on Developing Your Brand – from a job seeker , Business to sole contractor/consultant‎ – he assures me that his slides are no longer the shocking pink he has used in previous years.

At 11:00 there’s a coffee break sponsored by Evozon. My training finishes at that point, so I’m free to see a few talks. Unfortunately, I want to see all of the talks in the next slot. I suspect I’ll end up seeing Neil Bowers’ Boosting community engagement with CPAN‎ and Smylers’ ‎Don’t Do That: Code Interface Mistakes to Avoid‎, but I could well be tempted into Aaron Crane’s Write-once data: writing Perl like Haskell‎ instead. Or, back on the workshop track, there’s Dominic Humphries on From can to can’t: An intro to functional programming. Just before lunch, I think I’ll see Neil Bowers again. This time he’s talking about Dependencies and the River of CPAN.

After lunch there’s another session where I want to see everything. I’d love to see Stevan Little talking about his latest iteration of the p5-mop, but I suspect I’ll end up seeing Leon Brocard on Making your website seem faster‎ followed by Kaitlyn Parkhurst on Project Management For The Solo Developer. Dominic’s functional programming workshop continues after lunch and is joined by John Davies and Martin Berends talking about Parallel Processing Performed Properly in Perl on Pi‎.

The big talk after the next short break is going to be Matt Trout on A decade of dubious decisions‎ but it’s another I’ll miss as I’m talking about Conference Driven Publishing‎ in another room during the second half of it. During the first half I’d recommend Steve Mynott’s Perl 6 Grammars‎.  But, I saw him practice it at a recent London Perl Mongers technical meeting, so I’ll be seeing Andrew Solomon explaining How to grow a Perl team‎. In the workshop stream, Christian Jaeger will be covering Functional Programming on Perl‎.

Then there’s another coffee break (this time sponsored by Perl Careers) and then we’re into the last few sessions. In the first you have a choice between Jeff Goff on From Regular Expressions to Parsing JavaScript: Learn Perl6 Grammars‎ and Theo van Hoesel on ‎Dancer2 REST assured‎. I think I’ll be in Theo’s talk.

These are followed by Jonathan Worthington’s keynote – The end of the beginning‎ and the lightning talks. It will, no doubt, be a great end to a fabulous day.

The London Perl Workshop is always a great day a learning about Perl and catching up with old friends. And because of the brilliant sponsors, it doesn’t cost the attendees a penny.

If you’re going to be near London next weekend and you have any interest in Perl, then why not register and come along?

Here’s a brief video of last year’s workshop.

Training Courses – More Details

Last week I mentioned the public training courses that I’ll be running in London next February. A couple of people got in touch and asked if I had more details of the contents of the courses. That makes sense of course, I don’t expect people to pay £300 for a days training without knowing a bit about the syllabus.

So here are details of the first two courses (the Moose one and the DBIx::Class one). I hope to have details of the others available by next weekend.

Object Oriented Programming with Perl and Moose

  • Introduction to Object Oriented programming
  • Overview of Moose
  • Object Attributes
  • Subclasses
  • Object construction
  • Data types
  • Delegation
  • Roles
  • Meta-programming
  • Further information

Database Programming with Perl and DBIx::Class

  • Brief introduction to relational databases
  • Introduction to databases and Perl
    • DBI
    • ORM
  • Schema Classes
  • Basic DB operations
    • CRUD
  • Advanced queries
    • Ordering, joining, grouping
  • Extending DBIC
  • Further information

If you have any further questions, please either ask them in the comments or email me (I’m dave at this domain).

And if I’ve sold you on the idea of these courses, the booking page is now open.

Public Training in London – February 2016

For several years I’ve been running an annual set of public training courses in London in conjunction with FLOSS UK (formerly known as UKUUG). For various scheduling reasons, we didn’t get round to running any this year, but we have already made plans for next year.

I’ll be running five days of training in central London from 8th – 12th February. The courses will take place at the Ambassador’s Hotel on Upper Woburn Place. Full details are in the process of appearing on the FLOSS UK web site, but the booking page doesn’t seem to be live yet, so I can’t tell you how much it will cost.

We’re doing something a little different this year. In previous years, I’ve been running two generic two-day courses – one on intermediate Perl and one on advanced Perl. This year we’re running a number of shorter but more focussed courses. The complete list is:

  • Object Oriented Programming with Perl and Moose (Mon 8th Feb)
  • Database Programming with Perl and DBIx::Class (Tue 9th Feb)
  • An Introduction to Testing Perl Programs (Wed 10th Feb)
  • Modern Web Programming with Perl (two day course – Thu/Fri 11th/12th Feb)

This new approach came out of some feedback we’ve received from attendees over the last couple of years. I’m hoping that by offering this shorter courses, people will be able to take more of a “mix and match” approach and will select courses that better fit their requirements. Of course, if you’re interested, there’s no reason why you shouldn’t come to all five days.

I’ll update this page when I know how much the courses will cost and how you can book. But please put these dates in your calendar.

Update: And less than 24 hours after publishing this blog post, the booking page has gone live.

Places are £300 a day (so £600 for the two-day course on web programming) and there’s a special offer of £1,320 for the full week.

Prices are cheaper (by £90 a day) for members. And given that an annual individual membership costs £35, that all sounds like a bit of a no-brainer to me.

Build RPMs of CPAN Modules

If you’ve been reading my blog for a while, you probably already know that I have an interest in building RPMs of CPAN modules. I run a small RPM repository where I make available all of the RPMs that I have built for myself. These will either be modules that aren’t available in other RPM repositories or modules where I wanted a newer version than the currently available one.

I’m happy to take requests for my repo, but I don’t often get any. That’s probably because most people very sensibly use the cpanminus/local::lib approach or something along those lines.

But earlier this week, I was sitting on IRC and Ilmari asked if I had a particular module available. When I said that I didn’t, he asked if I had a guide for building an RPM. I didn’t (well there are slides from YAPC 2008 – but they’re a bit dated) but I could see that it was a good suggestion. So here it is. Oh, and I built the missing RPM for him.

Setting Up

In order to build RPMs you’ll need a few things set up. This is stuff you’ll only need to do once. Firstly, you’ll need two new packages installed – cpanspec (which parses a CPAN distribution and produces a spec file) and rpm-build (which takes a spec file and a distribution and turns them into an RPM). They will be available in the standard repos for your distribution (assuming your distribution is something RPM-based like Fedora or Centos) so installing them is as simple as:

If you’re using Fedora 22 or later, “yum” has been replaced with “dnf”.

Next, you’ll need a directory structure in which to build your RPMs. I always have an “rpm” directory in my home directory, but it can be anywhere and called anything you like. Within that directory you will need subdirectories called BUILD, BUILDROOT, RPMS, SOURCES, SPECS and SRPMS. We’ll see what most of those are for a little later.

The final thing you’ll need is a file called “.rpmmacros” in your home directory. At a minimum, it should contain this:

The packager and vendor settings are just to stop you having to type in that information every time you build an RPM. The _topdir setting points to the “rpm” directory that you created a couple of paragraphs up.

I would highly recommend adding the following line as well:

This turns off the default behavior for adding “Requires” data to the RPM. The standard behaviour is to parse the module’s source code looking for every “use” statement. By turning that off, you instead trust the information in the META.yml to be correct. If you’re interesting in hearing more detail about why I think the default behaviour is broken, then ask me in a pub sometime.

Ok. Now we’re all set. We can build our first RPM.

Building an RPM

Building an RPM is simple. You use “cpanspec” to make the spec file and then “rpmbuild” to build the RPM. You can use “cpanspec” in a few different modes. If you have the module tarball, then you can pass that to “cpanspec”.

That will unwrap the tarball, parse the code and create the spec file.

But if you’re building an RPM for a CPAN module, you don’t need to download the tarball first, “cpanspec” will do that for you if you give it a distribution name.

That will connect to CPAN, find the latest version of the distribution, download the right tarball and then do all the unwrapping, parsing and spec creation.

But there’s another, even cleverer way to use “cpanspec” and that’s the one that I use. If you only know the module’s name and you’re not sure which distribution it’s in, then you can just pass the name of the module.

This is the mode that I always use it in.

No matter how you invoke “cpanspec”, you will end up with the distribution tarball and the spec file – which will be called “perl-Some-Module.spec”. You need to copy these files into the correct directories under your rpm building directory. The tarball goes into SOURCES and the spec goes into SPECS. It’s also probably easiest if you change directory into your rpm building directory.

You can now build the RPM with this command:

You’ll see a lot of output as “rpmbuild” goes through the whole CPAN module testing and building process. But hopefully eventually you’ll see some output saying that the build has succeeded and that an RPM has been written under your RPMS directory (in either the “noarch” or “x86_64” subdirectory). You can install that RPM with any of the following commands:

And that should be that. Of course there are a few things that can go wrong. And that’s what the next section is about.

Fixing Problems

There are a number of things that can go wrong when building RPMs. Here are some of the most common, along with suggested fixes.

Missing prerequisites

This is also known as “dependency hell”. The module you are building is likely to require other modules. And you will need to have those installed before “rpmbuild” will let you build the RPM (and, note, they’ll need to be installed as RPMS – the RPM database doesn’t know about modules you have installed with “cpan” or “cpanminus”).

If you have missing prerequisites, the first step is to try to install them using “yum” (or “dnf”). Sometimes you will get lucky, other times the prerequisites won’t exist in the repos that you’re using and you will have to build them yourself. This is the point at which building an RPM for a single module suddenly spirals into three hours of painstaking work as you struggle to keep track of how far down the rabbit-hole you have gone.

I keep thinking that I should build a tool which parses the prerequisites, works out which ones already exist and automatically tries to build the ones that are missing. It would need to work recursively of course. I haven’t summoned the courage yet.

Extra files

Sometimes at the end of an RPM build, you’ll get an error saying that files were found which weren’t listed in the spec file. This usually means that the distribution contains programs that “cpanspec” didn’t find and therefore didn’t add to the spec file. This is a simple fix. Open the spec file in an editor and look for the section labelled ‘%files’. Usually, it will look something like this:

This is a list of the files which will be added to the RPM. See the _mandir entry? That’s the man page for the module that is generated from the module’s Pod (section 3 is where library documentation goes). We just need to add two lines to the bottom of this section:

This says “add any files you find in the binaries directories (and also any man pages you find for those programs)”.

If you add these lines and re-run the “rpmbuild” command, the build should now succeed.

Missing header files

If you’re building an XS module that is a wrapped around a C library then you will also need the C header files for that library in order to compile the XS files. If you get errors about missing definitions, then this is probably the problem. In RedHat-land a C library called “mycoolthing” will live in an RPM called “libmycoolthing” and the headers will be in an RPM library called “libmycoolthing-devel”. You will need both of those installed.

Your users, however, will only need the C library (libmycoolthing) installed. It’s well worth telling the RPM system that this external library is required by adding the following line to the spec file:

That way, when people install your module using “yum” or “dnf”, it will pull in the correct C library too. “cpanspec” will automatically generate “Requires” lines for other Perl RPMs, but it can’t do it for libraries that aren’t declared in the META.yml file.

 

So that’s it. A basic guide to building RPMs from CPAN distributions. There’s a lot more detail that I could cover, but this should be enough to work for 80-90% of the modules that you will want to build.

If you have any questions, then please leave a comment below.

The Joy of Prefetch

If you heard me speak at YAPC or you’ve had any kind of conversation with me over the last few weeks then it’s likely you’ve heard me mention the secret project that I’ve been writing for my wife’s school.

To give you a bit of background, there’s one afternoon a week where the students at the school don’t follow the normal academic timetable. On that afternoon, the teachers all offer classes on wider topics. This year’s topics include Acting, Money Management and Quilt-Making. It’s a wide-ranging selection. Each student chooses one class per term.

This year I offered to write a web app that allowed the students to make their selections. This seemed better than the spreadsheet-based mechanisms that have been used in the past. Each student registers with their school-based email address and then on a given date, they can log in and make their selections.

I wrote the app in Dancer2 (my web framework of choice) and the site started allowing students to make their selections last Thursday morning. In the run-up to the go-live time, Google Analytics showed me that about 180 students were on the site waiting to make their selections. At 7am the selections part of the site went live.

And immediately stopped working. Much to my embarrassment.

It turned out that a disk failed on the server moments after the site went live. It’s the kind of thing that you can’t predict.But it leads to lots of frustrated teenagers and doesn’t give a very good impression.

To give me time to rebuild and stress-test the site we’ve decided to relaunch at 8pm this evening. I’ve spent the weekend rebuilding the app on a new (and more powerful) server.

I’m pretty sure that the timing of the failure was coincidental. I don’t think that my app caused the disk failure. But a failure of this magnitude makes you paranoid, so I spent a lot of yesterday tuning the code.

The area I looked at most closely was the number of database queries that the app was making. There are two main actions that might be slow – the page that builds the list of courses that a student can choose from and the page which saves a student’s selections.

I started with the first of these. I set DBIC_TRACE to 1 and fired up a development copy of the app. I was shocked to see the app run about 120 queries – many of which were identical.

Of course I should have tested this before. And, yes, it’s an idiotic way to build an application. But I’m afraid that using an ORM like DBIx::Class can make it all too easy to write code like this. Fortunately, it makes it easy to fix it too. The secret is “prefetch”.

“Prefetch” is an option you can pass to the the “search” method on a resultset. Here’s an example of the difference that can make.

There are seven year groups in a British secondary school. Most schools call them Year 7 to Year 13 (the earlier years are in primary school). Each year group will have a number of forms. So there’s a one to many relationship between years and forms. In database terms, the form table holds a foreign key to the year table. In DBIC terms, the Year result class has a “has_many” relationship with the Form result class and the Form result class has a “belongs_to” relation with the Year result class.

A naive way to list the years and their associated forms would look like this:

Run code like that with DBIC_TRACE turned on and you’ll see the proliferation of database queries. There’s one query that selects all of the years and then for each year, you get another query to get all of its associated forms.

Of course, if you were writing raw SQL, you wouldn’t do that. You’d write one query that joins the year and form tables and pulls all of the data back at once. And the “prefetch” option gives you a way to do that in DBIC as well.

All we have done here is to interpose a call to “search” which adds the “prefetch” option. If you run this code with DBIC_TRACE turned on, then you’ll see that there’s only one database query and it’ll be very similar to the raw SQL that you would have written – it brings back the data from both of the tables at the same time.

But that’s not all of the cleverness of the “prefetch” option. You might be wondering what the difference is between “prefetch” and the rather similar-sounding “join” option. Well, with “join” the columns from the joined table would be added to your main table’s result set. This would, for example, create some kind of mutant Year resultset object that you could ask for Form data using calls like “get_column(‘forms.name’)”. [Update: I was trying to simplify this explanation and I ended up over-simplifying to the point of complete inaccuracy – joined columns only get added to your result set if you use the “columns” or “select/as” attributes. And the argument to “get_column()” needs to be the column name that you have defined using those options.] And that’s useful sometimes, but often I find it easier to use “prefetch” as that uses the data from the form table to build Form result objects which look exactly as they would if you pulled them directly from the database.

So that’s the kind of change that I made in my code. By prefetching a lot of associated tables I was able to drastically cut down the number of queries made to build that course selection page. Originally, it was about 120 queries. I got it down to three. Of course, each of those queries is a lot larger and is doing far more work. But there’s a lot less time spent compiling SQL and pulling data from the database.

The other page I looked at – the one that saves a student’s selections – wasn’t quite so impressive. Originally it was about twenty queries and I got it down to six.

Reducing the number of database queries is a really useful way to make your applications more efficient and DBIC’s “prefetch” option is a great tool for enabling that. I recommend that you take a close look at it.

After crowing about my success on Twitter I got a reply from a colleague pointing me at Test::DBIC::ExpectedQueries which looks like a great tool for monitoring the number of queries in your app.