YAPC Europe 2016

I’ve been back from Cluj-Napoca for almost a week, so I should really write down what I remember about YAPC Europe before it’s all forgotten.

Day -1

I arrived in Cluj-Napoca on Sunday evening and got to my hotel quickly. There was just time for a quick meal before bed.

Monday was the day that I was going to get to grips with the city. After meeting a few Perl Mongers at breakfast, my wife and I set off to explore. My first target was to find Cluj Hub, the venue where I was running a training course the following day. That was simple and took less than fifteen minutes. We then explored both the Orthodox and Catholic cathedrals before settling into a bar on the main square called “Guevara” for a coffee. After that we decided that we needed to pick up some supplies and whilst on that hunt we bumped into Max Maischein who recommended a visit to the botanical gardens.

On returning to the hotel with our supplies, we met Curtis Poe and invited him to join us for lunch. Wandering at random we found a really good restaurant called Livada and enjoyed a very pleasant meal.

After lunch we spent a very enjoyable couple of hours in the botanical gardens and only just failed to get back to the hotel before it really started raining. That evening we ate in restaurant really close to the hotel called the Crying Monkey (but in Romanian).

Day 0

Tuesday was my “Modern Web Development with Perl and Dancer” training course. This was by far the most successful training course that I’ve ever run at a YAPC. I’ll write more about it when I get the feedback results, but I think that the attendees enjoyed it. I know I had great fun giving it. Cluj Hub was a great venue and Andra Gligor and her small team looked after us all really well.

That evening, the traditional pre-conference meet-up was held on the roof of Evozon’s offices. As always, it was lovely to catch-up with old friends that I only get to see once or twice a year.

Day 1

On Wednesday, I set off in plenty of time to find the venue. It turned out that our hotel was really well located for both sight-seeing and the conference and I got there in ten minutes or so. The registration queues seemed shorter than usual and before long I had my name-tag and bag of conference swag.

As always, there were far too many good talks and it was impossible to see everything. I’ll just talk about the talks that I saw. Everything was videoed, so it will all be online soon.

The day began with Amalia welcoming us to the conference. Then the YAPC Europe Foundation announced that next year’s conference will be in Amsterdam. This is the first time that the conference has returned to a previous city (the second YAPC Europe was held in Amsterdam back in 2001) and I’m looking forward to going.

The first day’s keynote was from Curtis Poe. It was a wide-ranging talk covering the history and future of both Perl and the Perl community. After that I went into one of the small rooms to see H. Merijn Brand talking about his recent improvements to Perl’s CSV parser followed by Alex Muntada on how the Debian project packages CPAN modules. I then went back to the main room to see Mickey Nasriachi talking about PONAPI, which is a Perl implementation of JSONAPI.

Lunch suffered slightly from the inevitable queues, but it was worth the wait as the quality of the food (as it was throughout the conference) was very high.

After lunch I saw Lee Johnson giving some Git tips, Sawyer talking about the XS guts of Ref::Util and Jose Luis Martinez talking about PAWS (the Perl interface to Amazon Web Services). I saw Jose Luis talking about PAWS last year in Granada but really wanted to see how the project is progressing. I think this has the potential to be a great advocacy tool for Perl.

A quick coffee break and then I saw Thomas Klausner give his opinions on writing API endpoints and Tina Müller talking about App::Spec which looks like a great tool for easily writing command line applications.

Then it was was lightning talks. They were the usual combination of the useful, the banal and the ridiculous. I think the highlight for me was Curtis Poe announcing more details of his online game (which is now officially called Tau Station). This was the point at which I announced Cultured Perl – which seems to be going well so far.

That evening was the conference dinner. Which was a buffet party held in the open-air quadrangle at the centre of the Banffy Palace (Cluj’s major art museum). A great time was had by all.

Day 2

Another day, another keynote. This time it was Sawyer X with “The State of the Velociraptor” – an annual round-up of what’s going on in the Perl 5 world. This year Sawyer found a number of volunteers who all gave short talks about their part of the Perl community. This was a great idea which was only slightly marred by the fact that the projector wasn’t at all happy changing laptops – so the switches between presenters weren’t as smooth as they could have been.

After that I saw Max talking about how he uses ElasticSearch on his laptop to give himself a local search engine and Job van Achterberg talking about making web sites more accessible. This was a great talk – particularly the sections where he showed just how bad most web sites appear to screen readers.

Another queue for another great lunch. And also many interesting conversations.

After lunch I saw a former colleague, Mirela Iclodean, talking about how her company have managed to shoe-horn many modern tools and practices into their working day – while still maintaining a nasty monolithic code-base which they are slowly chipping away at. It was a great talk and it made me miss working on that project. I’m hoping that she will repeat this talk at the London Perl Workshop.

Later that afternoon, I gave my “Error(s) Free Programming” talk – in a slot where every speaker was a London.pm leader. The talk seemed to go down well, but somehow I ran considerably short.

After that I saw Albert Hilazo talk about his first few months as a Perl programmer. I found this really interesting as Albert talked in some detail about things that other language communities provide but he found hard to find for Perl. In particular, he would like to see more “war story” blog posts showing how people have solved particular problems using Perl tools.

Then it was Matt Trout celebrating ten years in the Perl community by explaining how his career was largely a series of happy accidents and that a lot of the responsibilities he has taken on were just through being in the wrong place at the wrong time – or something like that.

One talk I couldn’t miss was Andrew Yates talking about the work that his team do at the European Bioinformatics Institute. I couldn’t miss it as I was at least partly responsible for Andrew proposing the talk. I ran some training at the EBI earlier this year and during our email conversation YAPC was mentioned and Andrew asked if people might be interested in hearing about their work. I replied “hell, yes!” and sent him a link to the talk proposal web page.

And then, of course, there another ten or so lightning talks to close the day entertainingly.

Day 3

The keynote speaker on the last day was Larry Wall. His topic was “Strange Consistency”. If you’ve seen Larry speak before, you’ll know what it was like.

I followed that by watching Jason Clifford talk about how his team had written a major new toolset in Perl despite management pressure to use other technologies. The project, of course, ended up being very successful.

One of the most interesting talks was Nicholas Clark’s view of an alternative universe where Jon Orwant never threw those mugs in 2000 and the Perl 6 project was never started. The main lesson appeared to be “what goes around, comes around” and his fictional universe didn’t end up too far away from where we are now.

The afternoon had a curious combination of some time slots where I wanted to see every talk and others where I didn’t really want to see anything. So in some cases I’m eagerly awaiting the videos going online and in others I sat in the back of the room only half-concentrating while giving most of my attention to Twitter or Facebook.

I really enjoyed Sawyer talking about the things that were added in Perl 5.24 (and very carefully not talking about the things that were added in previous versions) and also Jose Luis Perez talking about what he has got out of doing the CPAN Pull Request Challenge.

The final lightning talks were as much fun as they always are. The projectors were still giving the speakers plenty of technical difficulties which led to lots of time for “lightning adverts” between the talks. I think that towards the end the differences between the two rather broke down and on the video at one point I expect you’ll hear Geoff Avery saying “I seem to have lost control of this”.

The conference ended, as it always does, with a brief presentation from the organisers of next year’s conference, a final thank-you to all of the speakers and sponsors and a standing ovation for the organisers.

This was one of the best-organised YAPCs I’ve been to for a very long time. And Cluj-Napoca is a city I would never have considered visiting if it wasn’t for the Perl community there. And already I’m considering a return visit. I had a lovely time in the city and returned to London completely recharged and reinvigorated.

See you all in Amsterdam next year.

Cultured Perl

Back in about 2008, I set up a group blog called “Cultured Perl”. The idea was to have a blog that concentrated on the Perl community rather than the technical aspects that most Perl bloggers write about most of the time. It didn’t last very long though and after a few posts it quietly died. But the name “Cultured Perl” still appeals to my love of bad puns and I knew I would reuse it at some point.

At YAPC Europe 2010 in Pisa, I gave a lightning talk called Perl Vogue. It talked about the way the Perl modules come into fashion and often go out of fashion again very quickly. I suggested an online Perl magazine which would tell people which modules were fashionable each month. It was a joke, of course (not least because Vogue are famously defensive of their brand.

Over the last many years people have suggested that the Perl community needs to get “out of the echo chamber” and talk to people who aren’t part of the community. For example, instead of posting and answering Perl questions on a Perl-specific web site like Perl Monks, it’s better to do it on a general programming site like Stack Overflow.

Hold those three thoughts. “Cultured Perl”, online Perl magazine, getting out of the echo chamber.

Medium is a very popular blogging site. Many people have moved their blogging there and it’s a great community for writing, sharing and recommending long-form writing. I get a “recommended reading” email from Medium every day and it always contains links to several interesting articles.

Medium has two other features that interest me. Firstly, you can tag posts. So if you write a post about web development using Perl and tag it with “web dev” then it will be seen by anyone who is following the web dev tag. That’s breaking out of the echo chamber.

Secondly, Medium has “publications”. That is, you can bring a set of articles together under your own banner. Publication owners can style their front page in various ways to differentiate it from Medium’s default styling. Readers can subscribe to publications and they will then be notified of every article published in that publication. That’s an online magazine.

So I’ve set up a publication on Medium (called “Cultured Perl” – to complete the set of three ideas). My plan is to publish (or republish) top quality Perl articles so we slowly build a brand outside of the echo chamber where people know they can find all that is best in Perl writing.

If you write about Perl, please consider signing up to Medium, becoming a contributor to Cultured Perl and submitting your articles for publication. I’ll publish the best ones (and, hopefully, work with authors to improve the others so they are good enough to publish).

I’m happy to republish stuff from your other blogs. I’m not suggesting that we suddenly move all Perl blogging to Medium. For example, whenever I publish something on Perl Hacks, the post gets mirrored to a Perl Hacks publication that I set up on Medium earlier this year. There’s a WordPress to Medium plugin that does that automatically for me. There may well be similar tools for other blogging platforms (if you can’t find one for your blog – then Medium has an API so you could  write one).

If you are a reader, then please consider subscribing to Cultured Perl. And please recommend (by clicking on the heart symbol) any articles that you enjoy. The more recommendations that an article gets, the more likely it becomes that Medium will recommend it to other readers.

I have no idea how this will go, but over the next few months I hope to start by publishing four or five articles every week. Perhaps you could start by submitting articles about what a great time you had at YAPC Europe.

Oh, and here are the slides from the lightning talk I used to announce this project at YAPC Europe in Cluj-Napoca, Romania yesterday.


Training Courses – More Details

Last week I mentioned the public training courses that I’ll be running in London next February. A couple of people got in touch and asked if I had more details of the contents of the courses. That makes sense of course, I don’t expect people to pay £300 for a days training without knowing a bit about the syllabus.

So here are details of the first two courses (the Moose one and the DBIx::Class one). I hope to have details of the others available by next weekend.

Object Oriented Programming with Perl and Moose

  • Introduction to Object Oriented programming
  • Overview of Moose
  • Object Attributes
  • Subclasses
  • Object construction
  • Data types
  • Delegation
  • Roles
  • Meta-programming
  • Further information

Database Programming with Perl and DBIx::Class

  • Brief introduction to relational databases
  • Introduction to databases and Perl
    • DBI
    • ORM
  • Schema Classes
  • Basic DB operations
    • CRUD
  • Advanced queries
    • Ordering, joining, grouping
  • Extending DBIC
  • Further information

If you have any further questions, please either ask them in the comments or email me (I’m dave at this domain).

And if I’ve sold you on the idea of these courses, the booking page is now open.

Public Training in London – February 2016

For several years I’ve been running an annual set of public training courses in London in conjunction with FLOSS UK (formerly known as UKUUG). For various scheduling reasons, we didn’t get round to running any this year, but we have already made plans for next year.

I’ll be running five days of training in central London from 8th – 12th February. The courses will take place at the Ambassador’s Hotel on Upper Woburn Place. Full details are in the process of appearing on the FLOSS UK web site, but the booking page doesn’t seem to be live yet, so I can’t tell you how much it will cost.

We’re doing something a little different this year. In previous years, I’ve been running two generic two-day courses – one on intermediate Perl and one on advanced Perl. This year we’re running a number of shorter but more focussed courses. The complete list is:

  • Object Oriented Programming with Perl and Moose (Mon 8th Feb)
  • Database Programming with Perl and DBIx::Class (Tue 9th Feb)
  • An Introduction to Testing Perl Programs (Wed 10th Feb)
  • Modern Web Programming with Perl (two day course – Thu/Fri 11th/12th Feb)

This new approach came out of some feedback we’ve received from attendees over the last couple of years. I’m hoping that by offering this shorter courses, people will be able to take more of a “mix and match” approach and will select courses that better fit their requirements. Of course, if you’re interested, there’s no reason why you shouldn’t come to all five days.

I’ll update this page when I know how much the courses will cost and how you can book. But please put these dates in your calendar.

Update: And less than 24 hours after publishing this blog post, the booking page has gone live.

Places are £300 a day (so £600 for the two-day course on web programming) and there’s a special offer of £1,320 for the full week.

Prices are cheaper (by £90 a day) for members. And given that an annual individual membership costs £35, that all sounds like a bit of a no-brainer to me.

The Joy of Prefetch

If you heard me speak at YAPC or you’ve had any kind of conversation with me over the last few weeks then it’s likely you’ve heard me mention the secret project that I’ve been writing for my wife’s school.

To give you a bit of background, there’s one afternoon a week where the students at the school don’t follow the normal academic timetable. On that afternoon, the teachers all offer classes on wider topics. This year’s topics include Acting, Money Management and Quilt-Making. It’s a wide-ranging selection. Each student chooses one class per term.

This year I offered to write a web app that allowed the students to make their selections. This seemed better than the spreadsheet-based mechanisms that have been used in the past. Each student registers with their school-based email address and then on a given date, they can log in and make their selections.

I wrote the app in Dancer2 (my web framework of choice) and the site started allowing students to make their selections last Thursday morning. In the run-up to the go-live time, Google Analytics showed me that about 180 students were on the site waiting to make their selections. At 7am the selections part of the site went live.

And immediately stopped working. Much to my embarrassment.

It turned out that a disk failed on the server moments after the site went live. It’s the kind of thing that you can’t predict.But it leads to lots of frustrated teenagers and doesn’t give a very good impression.

To give me time to rebuild and stress-test the site we’ve decided to relaunch at 8pm this evening. I’ve spent the weekend rebuilding the app on a new (and more powerful) server.

I’m pretty sure that the timing of the failure was coincidental. I don’t think that my app caused the disk failure. But a failure of this magnitude makes you paranoid, so I spent a lot of yesterday tuning the code.

The area I looked at most closely was the number of database queries that the app was making. There are two main actions that might be slow – the page that builds the list of courses that a student can choose from and the page which saves a student’s selections.

I started with the first of these. I set DBIC_TRACE to 1 and fired up a development copy of the app. I was shocked to see the app run about 120 queries – many of which were identical.

Of course I should have tested this before. And, yes, it’s an idiotic way to build an application. But I’m afraid that using an ORM like DBIx::Class can make it all too easy to write code like this. Fortunately, it makes it easy to fix it too. The secret is “prefetch”.

“Prefetch” is an option you can pass to the the “search” method on a resultset. Here’s an example of the difference that can make.

There are seven year groups in a British secondary school. Most schools call them Year 7 to Year 13 (the earlier years are in primary school). Each year group will have a number of forms. So there’s a one to many relationship between years and forms. In database terms, the form table holds a foreign key to the year table. In DBIC terms, the Year result class has a “has_many” relationship with the Form result class and the Form result class has a “belongs_to” relation with the Year result class.

A naive way to list the years and their associated forms would look like this:

Run code like that with DBIC_TRACE turned on and you’ll see the proliferation of database queries. There’s one query that selects all of the years and then for each year, you get another query to get all of its associated forms.

Of course, if you were writing raw SQL, you wouldn’t do that. You’d write one query that joins the year and form tables and pulls all of the data back at once. And the “prefetch” option gives you a way to do that in DBIC as well.

All we have done here is to interpose a call to “search” which adds the “prefetch” option. If you run this code with DBIC_TRACE turned on, then you’ll see that there’s only one database query and it’ll be very similar to the raw SQL that you would have written – it brings back the data from both of the tables at the same time.

But that’s not all of the cleverness of the “prefetch” option. You might be wondering what the difference is between “prefetch” and the rather similar-sounding “join” option. Well, with “join” the columns from the joined table would be added to your main table’s result set. This would, for example, create some kind of mutant Year resultset object that you could ask for Form data using calls like “get_column(‘forms.name’)”. [Update: I was trying to simplify this explanation and I ended up over-simplifying to the point of complete inaccuracy – joined columns only get added to your result set if you use the “columns” or “select/as” attributes. And the argument to “get_column()” needs to be the column name that you have defined using those options.] And that’s useful sometimes, but often I find it easier to use “prefetch” as that uses the data from the form table to build Form result objects which look exactly as they would if you pulled them directly from the database.

So that’s the kind of change that I made in my code. By prefetching a lot of associated tables I was able to drastically cut down the number of queries made to build that course selection page. Originally, it was about 120 queries. I got it down to three. Of course, each of those queries is a lot larger and is doing far more work. But there’s a lot less time spent compiling SQL and pulling data from the database.

The other page I looked at – the one that saves a student’s selections – wasn’t quite so impressive. Originally it was about twenty queries and I got it down to six.

Reducing the number of database queries is a really useful way to make your applications more efficient and DBIC’s “prefetch” option is a great tool for enabling that. I recommend that you take a close look at it.

After crowing about my success on Twitter I got a reply from a colleague pointing me at Test::DBIC::ExpectedQueries which looks like a great tool for monitoring the number of queries in your app.