Skip to main content

Measuring effort is easier than measuring results.

Have you heard the joke about the guy looking for his keys under a street light when he lost them in the parking lot because "that's where the light is"?

This is how I feel about measuring effort rather than results. Effort is where the light is. You have time estimates from planning and you have time spent on a burndown chart. You have lines of code and number of commits. You have green tests and passing builds.You have loads of data that show what effort was made and how long it took. All of that is really easy to measure and that might be why so much effort goes into measuring the easy to measure thing.

Here's the thing. All data would still be there if the only reason why you did the thing was that it was cool and you (or the product owner) wanted to and the feature provided no benefit to the stakeholders.

This is why I am making an effort to write stories that specifically state expected measurable results as part of the acceptance criteria, beyond just normal "it works and does a thing" as a way to increase focus on the value of outcomes of the work rather than just the effort of completing that task in the current sprint.

What I am aiming for are criteria, developed with the product owners; that would show that the work was not only completed, but that the work did a thing that that we wanted to get done in a way that makes sense for the stakeholders. Adding a feature that no one uses, no one wanted or that made some other aspect of the application worse, might be a successful completion of a sprint task. It might even be what the customer wanted. It just isn't a good outcome.

Old Style: 

Story: As a user, it should be possible to add a new payment type to an existing account.
Acceptance: User account with n+1 payment types.

In this case, you have a what not a how, that can be tested in a standard given, when, expect format test. This is very typical and in many cases might be the right approach.

New Style: 

Story: As a user, it should be possible to add a new payment type to an existing account.
Acceptance: User account with n+1 payment types.
Expected results: The number of support tickets related to payment type issues should decline. The number of users with multiple payment types should increase. User retention should increase or stay the same. Users looking at the help screens for payment types or while adding a payment type should decline.

In this case, you also have a what and a set of why criteria that should be true if an only if the change added value in the way the product owner expected based on their view of the users needs and wants.

But what if you're wrong? Then the effort might not have been worth it. Maybe the users didn't what that functionality. Maybe it works just fine but isn't discoverable and they don't know its there. Why the thing that was done didn't result in the thing you wanted might be disappointing, but it is a valuable insight that you only get from putting that bet on the table.

Old Style: 

Story: Improve mobile page load speed while keeping the look and feel.
Acceptance: Page is loaded, responsive and feature complete in X seconds.

Again, perfectly cromulent criteria that focuses on a what and not a how and can be tested in a standard given, when, expect format test.

New Style: 

Story: Improve mobile page load speed while keeping the look and feel.
Acceptance: While keeping the style and branding of the pages consistent, the page should be loaded, responsive and feature complete in X seconds.
Expected results: The number of users that timeout on subsequent page loads should decrease. The median page load time should decrease in proportion. The number of users who bounce while waiting for a page load should decrease.

In this case you are specifying why you think improving mobile page load speed is worth the effort and what you think the results of making the page faster should be. If you reach the load time goal and do not see the expected benefits, you have to go back and reconsider what would make the page seem fast or if page load time is what the users are unhappy with. That's disappointing, but would you know this without making this part of the story?

What if you can't think of measurable outcomes?

Are you sure there is a reason why you are doing this outside of "it's cool and I want to" ?

What if I'm wrong?

Failing faster is better. There is no bigger waste than adding, optimizing or improving an unnecessary or unwanted feature. Anything done to prevent this would be a good thing, even being wrong often.

Wouldn't it take a while to show that the results are there?

In most cases I'm interested in, yes. I expect the features to be A/B tested to see if they yield the expected results. If they don't, I would consider recommending rejecting or canceling the change and reevaluating the goal. 

Another potential advantage here is that the measurable acceptance criteria can be continually tracked in future sprints, either by automated testing of by gathering analytical data from the running application. In cases where a future change breaks a prior positive expected result, it can be properly tracked as a regression error that undid some prior useful work.

A drop in the use of a feature following addition of new functionality might be evidence of changing use cases or evidence that the feature was made less useful. This approach may find cases where some functionality has been "broken" by a subsequent change even when all unit and integration tests continue to pass.

Popular posts from this blog

Capturing text from any Mac Application into Emacs org-mode with Automator and org-protocol

After decades of using vi and Vim I switched to  Spacemacs  which is an amazing vi keystroke emulation layer running on Emacs and configured with an amazing set of preconfigured layers for different tasks. I decided to give it a try after seeing Org-Mode in action and seeing it was a nice taking system with integrations with almost anything imaginable. A few weeks ago I found out about org-protocol and followed this post  by Jethro on using a bookmarket to capture from the Web to Emacs.  This page assumes a few things You use Emacs on a Mac You are using org and understand how to use capture and capture templates. You need to yank text from random apps into Emacs You don't need to be using Spacemacs and this should work with any install of Emacs that supports org, org-capture and org-protocol. Creating Automator Action Start Automator. It's this icon. I'm guessing many people have had this for years and have never used it.  Open it and pick Quick Action Grab the following t

Halloween Candy Distribution Robot Chute

I am not a hardware guy and my Brooklyn apartment lacks true workshop space but we were able to put together a reasonably well done candy chute robot able to deliver candy directly into Trick-or-Treat's candy bags.  My wife wanted the robot to blink lights and wave an arm. I decided to use a servo motor driven by a Raspberry pi pico running MicroPython. The pico and MicroPython were chosen because I had them already from prior projects with my son.  Legos, chopsticks and leftover screws. Only the best. Cardboard, aluminum foil and Tupperware to protect the electronics. Those are the bags of candy and we managed to go through all of the candy by dark. This is what it looked like up on our balcony. How do you get the candy down to the trick-or-treaters? A dryer duct. Last time we used plastic sheeting and zip ties. The $25 to get a duct was worth it. We tested it with fun sized chocolate, smarties, double bubble gum, skittles and m&ms. The bagged candy, skittles and M&Ms didn

Setting up Twitter Cards in Blogger

There are plenty of blog posts explaining how to add Twitter Card integration to your Blogger posts but I haven't seen any that are written for the newer Blogger UI. The code that does the integration hasn't changed since this article in 2013 , but the steps to add it to your theme have.  If you haven't logged onto Blogger, do so. You should see this panel on your left. Pick "Theme" and you will see something that looks like this on the top portion of your screen. First things first, make a backup. Click on the little arrow If you are using the classic themes, it will look like this.  After clicking on the arrow, pick Backup. Download the file. It's the same for classic themes. Pick download. Save the file somewhere just in case you need to recover your theme. While editing your theme will not destroy any of your pages or posts, it can cause problems for the theme you are using if you make a mistake. If you've spent some time editing your blogroll and cust