Categories
Conferences

Perl Conference in Riga

It’s only two weeks until I head to Riga for PerlCon 2019. I thought it was worthwhile posting a quick update confirming that I was going and telling you what I would be doing there.

Firstly, I’ve previously mentioned that I was planning to run my “Modern Web Development with Dancer” workshop on the day before the conference. That’s now not going to happen as we didn’t manage to sell enough tickets to make the workshop economically viable.

But I will be giving two talks at the conference. On day two (Thursday) I’ll be speaking on Monoliths, Balls of Mud and Silver Bullets. That’s at 12:30 in the main room. This is a version of a talk I tried out at a London Perl Mongers technical meeting back in February. It’s a not-entirely-serious look at some of the problems you might encounter when replacing old monolithic code with new, shiny micro-services. Then on day three (Friday) I’ll be giving a longer talk on Measuring the Quality of your Perl Code. That’s, again, at 12:30, but in the second room. This does exactly what its title says. We’ll look at some measurements you can use to determine how good your Perl code is and ways to make those measurements automatic.

I have also submitted a proposal for a lightning talk. It’s about a Twitter bot that I wrote last weekend called Apollo 11 at 50 so, hopefully, you’ll find that interesting if you’re interested in either space or Twitter bots.

I’ll be a tourist in Riga for a few days before the conference. I’m arriving on Saturday 3rd August and leaving a week later on the 10th. Hope to see some of you there.

Categories
Programming

Perl Weekly Challenge – 2019-03-25

I’m not sure that I’ll have time to do these every week, but here are my answers to this week’s two Perl Weekly Challenges.

Challenge #1

Write a script to replace the character ‘e’ with ‘E’ in the string ‘Perl Weekly Challenge’. Also print the number of times the character ‘e’ found in the string.

Nothing really complicated here. We can discuss why I reached for tr/.../.../ rather than s/.../.../. I just think it’s silly to invoke the full power of the regex engine when you’re only changing a few letters.

Challenge #2

Write one-liner to solve FizzBuzz problem and print number 1-20. However, any number divisible by 3 should be replaced by the word fizz and any divisible by 5 by the word buzz. Numbers divisible by both become fizz buzz.

This is a bit more interesting. And I have to say that I think this is deeply dirty code. Over the twenty plus years I’ve been writing Perl I’ve trained myself to write code that is use strict and use warnings clean. This code is very much not that and with warnings turned on, you’d see a lot of warnings!

So how does it work?

The core is the (string)[index] construct.  We all know that you can access an individual element in an array with code like $my_array[$index]. But you can also do this with a list of values. For example:

returns “bar”. I often use this when I only need some of the values returned by localtime().

Here, I’m using something very similar. (fizz)[$_ % 3] is looking up a value in a list that contains only a single value (the string “fizz”). So if the index is zero then we get the first (and only) value in the list. If the index is anything other than zero then we’re looking for an element that’s off the end of the list and Perl gives us undef, which is interpreted as an empty string. And the index expression, $_ % 3, gives us zero if $_ is exactly divisable by 3. There are two things here that would throw warnings under use strict and use warnings. Firstly, “fizz” is a bareword and secondly, we’re using undef in a concatenation.

We then repeat the same logic using “buzz” and 5 and we concatenate together the results of those two expressions. That gives us either “fizz”, “buzz”, “fizzbuzz” or the empty string. Of those values, only the empty string is seen as false, so we can use || $_ to replace that with the original number.

There’s one mystery left. Why have I put that + before the first expression? I think I’m going to leave that as an exercise for the reader. If you work it out, please leave a comment.

And there we have it. A solution that is simultaneously both horribly dirty and yet, I think, rather clever.

Have you come up with your own solution yet?

Update:

It’s been pointed out to me that my solution for challenge #2 prints “fizzbuzz” for 15, when the specification clearly asks for “fizz buzz” (with a space). So here’s a version which fixes that:

Categories
Conferences

Plans for Riga

The European Perl Conference this year is going to be held in Riga in August. That might seem a long way away, but it’s never too early to start thinking about these things. For example the conference web site went live earlier this week, enabling users to register for the conference and buy their tickets.

And people who are planning to speak at the conference or run workshops alongside the conference need to get their act together early so that people who are planning to attend the conference know what is going to be happened. In particular, it’s a good idea to get the workshops organised and announced early so that people booking flights and hotels for the conference know that there are workshops taking place in the days before (and, sometimes, after) the conference and can book travel for the right dates.

So I’ve been thinking about what I want to do in Riga this summer and I think I have a plan.

I’m planning to re-run the “Modern Perl Web Development with Dancer” workshop that I ran in Cluj in 2016. It was easily the most successful YAPC/PerlCon workshops I’ve ever run with a full class of twenty people working through the day to build a simple web application using a number of modern web  development tools. This will be an updated version of the course as things have moved on a bit in the three years since I last ran the workshop.

Nothing is set in stone yet. I’ve submitted a proposal to the organisers and I hope we can get details tied down and tickets on sale as soon as possible. I’ll report on progress as I here what’s going on.

I’ve also come up with a talk that I’ve proposed for the main conference. It’s called “Measuring the Quality of your Perl Code” and it will be a look at was to measure the “quality” of your Perl code – on the basis that only once you start to measure something, can you start to make improvements. Again, it’s currently just a proposal. It hasn’t been accepted (but I’m taking the fact that it’s currently displayed on the front page of the site as a good sign!)

I should come up with a lightning talk too. Currently, I have no idea what that might be.

How about you? Are you planning to come to Riga in August? Will you be giving a talk?

Categories
Programming

A Subtle Bug

Earlier this week, I saw this code being recommended on Stack Overflow. The code contains a nasty, but rather subtle bug. The version I saw has been fixed now, but I thought there were some interesting lessons to learn by looking at the problems in some detail.

Let’s start by working out what the bug is. Here’s the code:

On first glance, it seems fine. It uses the common “open or die” idiom. It uses the modern approach of using a lexical filehandle. It even uses the three-argument version of “open()”. Code like has appeared in huge numbers of Perl programs for years. What can possibly be the problem?

I’ll give you a couple of minutes to have a closer look and work out what you think the problem is.

[ … time passes … ]

So what do you think? Do you see what the problem is?

The problem is that there is no error checking.

“What do you mean, Dave?” I hear you say. “There’s error checking there – I can see it plainly.” Some of you might even be wondering if I’m going senile.

And, yes, it certainly looks like it checks for errors. But the error checking doesn’t work. Let me prove that to you. We can check it with a simple command line program.

You would expect to see the “die” message there. But it doesn’t appear. Ok, perhaps I’m lying. Perhaps I really do have a file called “not.there”. Let’s try another, slightly different, version of the code.

And there we see the error message. That file really doesn’t exist.

So what went wrong with the first version? Of course, a good way to start working that out is to compare the two versions and look at the differences between them. The difference here is that when I put parentheses around the parameters to “open()” it started working. And when you fix things by adding parentheses it’s a pretty sure bet that the problem comes down to precedence.

The order of precedence for Perl operators is listed in perldoc perlop. If you look at that list you’ll see that the “or” operator we used (“||”) is at position 16 on the list. But what other operators are we using in our code? The answer is lurking down at position 21 on the list. When we call a Perl built-in function without using parentheses around the parameters, it’s known as a list operator. And list operators have rather low precedence.

All of which means that our original code is actually parsed as if we had written it like this:

Notice the parentheses that have appeared around $file and (crucially) the whole “or die” clause. That means that the bracketed expression is evaluated and passed to “open()” as its third argument. And when Perl evaluates that expression, it does that clever “Boolean short-circuiting” thing. An expression of “A || B” evaluates A first and if that is true, it returns it. Only if A is false will it go on to evaluate B and return that. In our case, the filename will always be true (well, unless you have a file called “0”) so the second half of the expression (the “or die…” bit) is never evaluated and, effectively, ignored.

Which is why I said, back at the start, that this code has no error checking at all – that’s literally true as the error checking has no effect at all.

So how do we fix it? Well, we’ve already seen one approach – you can explicitly add parentheses around the arguments to “open()”. But Perl programmers don’t like to use unnecessary punctuation and I’m sure I’ve seen this written without parentheses, so how does that work?

If you take another look at the table of operator precedence and look down below the list operators, you’ll see another “or” operator (the one that’s actually the word “or”, rather than punctuation). It’s right at the bottom of the list – at position 24. And that means we can use that version without needing the parentheses around the parameters to “open()”.

And that’s the version that you’ll see in most codebases. But, as we’ve seen, it’s vitally important to use the correct version of the “or” operator.

The worst thing about this bug is that it appears at the worst time. If your file exists and you can open it successfully, then everything works fine. Things only go wrong when… well, when things go wrong. If you can’t open your file for some reason, you won’t know about it. Which is bad.

So it’s important to test that your code works correctly when things go wrong. And that’s why we have modules like Test::Exception. You could write a test program like this:

And it would fail every time. But if you switched to the other “or” operator, it will work.

There’s one other approach you can take. You can use autodie in your code and just forget about adding “or die” to any of your calls to “open()”.

This is an easy bug to introduce into your code and a hard one to track down. Who’s confident that it doesn’t appear in any of their code?

Categories
Programming

Fixing a Bug

I fixed a bug earlier this week. Ok, actually, I introduced a bug and then spent the next few hours tracking it down and fixing it – but that doesn’t sound quite so positive, does it?

I thought it might be interesting to talk you through the bug and the fix. I should point out that this is client code, so all identifiers have been changed to protect the innocent.

We had some code that was being called in a loop while building the response to an API call. Usually, the loop would have a few dozen iterations, but in a few cases it could have something like 6,000 iterations. And in those cases we wanted to speed things up. One of the approaches I took was to eliminate all unnecessary database calls from the code in the loop. So I set DBIC_TRACE and made a request. And saw about a dozen database queries for each iteration. They had to go, so I started looking at the code.

One line I saw, looked like this:

The $widgets variable ends up storing a reference to an array containing details of three commonly used widget types. But they are always the same three widget types. So we can cache this.

This is basically the same code, but we now only run one query against the database. So that’s a big win. There was one other step. It turned out I was building the same cache in two different subroutines, so I moved the cache to a package-level lexical variable.

I made a few more similar changes and then submitted the code for review by some other developers in the team before releasing it. One of them asked why I hadn’t used the existing Widget::Cache module instead of inventing my own cache. The answer was that I hadn’t heard of it but on investigation it seemed to be just what I wanted. As the server starts, this module reads details of all of the widgets and stores them in a hash. It ends up looking like this:

The module also exports helper functions which gave access to the various parts of the cache – things like get_widget_by_code() and get_widget_by_id(). I added a new subroutine called get_common_widgets() and called it from my code like this:

I also removed the package-level lexical variable that I had previously been using to store my widget cache.

At this point, I had introduced the bug. But it wasn’t found until a few hours later when the fix was in QA. The API response that I had been working on was still working fine, but a few other unrelated pages on the site had stopped working. And the error messages all came from functions in the Widget::Cache module. They all complained that the cache variable, $widgets, was being used as a hash reference when it was actually an array reference.

It was clear that I’d broken something. But what? Can you see it?

Eventually, I resorted to checking every piece of code that touched the $widgets widgets variable. And there were two places that set a variable called $widgets. One was the initialisation code that I hadn’t touched and the other was my code where I get the list of common widgets. Remember? It looks like this:

And get_common_widgets() returns an array reference – so that would certainly explain the behaviour I’m seeing. But this variable isn’t the cache variable from Widget::Cache. It’s not even in the same file. Oh. Wait!

Remember I said that Widget::Cache exports some helper functions that give access to parts of the cache? Well, it also exports the $widget variable itself. I guess that’s in case you want to access a bit of it that isn’t exposed by a helper function. But in that situation, you should do what I did – you should add a new helper function. You shouldn’t just go rummaging in the innards of a global variable. In fact you shouldn’t have access to that global variable at all.

So, the fix was simple.

We now have a local, lexical variable and we don’t go trashing a global variable that various parts of the codebase rely on. The next step would be to remove that dangerous export and check that all code that uses it is replaced with code that uses helper functions instead. But it’s a large codebase and I’m not sure I want to open that particular can of worms.

Global variables are a terrible idea. Only use them if it’s absolutely necessary (and, even then, you’re probably wrong about it being absolutely necessary).

So now you all know how I nearly broke a client’s site this week and how the problem was only caught at the last minute in QA. Remember this when I’m next asking you to employ me.

(And, yes, I know I make the codebase sound very “procedural”. Lots of it is, I’m afraid, and we just have to do the best with what we have.)