Estimates and theory building

Faced with a complex situation, hard deadlines and uncertainty we often grasp for straws or anything that can help us sense our environment and make sense of the situation. We seek to navigate an unknown future by guessing about unknowns.

Reasons cited for this exercise, are various but often centred around the need to know, with some accuracy how long something will take or how much can be delivered by a given point in time.

I'm not going to argue for the validity of these needs here. I'll simply assume for the rest of this post that the needs and constraints are real.

What happens in most (software development) groups when such a situation is presented is that hilarity ensue in the form of some planning exercise, often taking the form of a group therapy session where participants yell out numbers and the facilitator proudly cites wisdom of the crowds and wideband delphi as inspirations. The basic idea being that if we slice up the work in small chunks, quickly estimate them and furiously negotiate what number on the magic fibonacci scale to label each chunk with, the end result will be a set of estimates that when tallied will give us the whole picture.

Suffice to say, I'm not a horribly big fan of exercises like this. They're time consuming affairs where after a period people will simply cede judgement to strong voices in order to get out of the room and "get back to work"...

Diagnosing the process as problematic does little to meet the needs, and as many would say the common, almost epidemic failure might just be a case of bad implementation of a fundamentally sound concept. Might. I've started to consider the main problem one of goal displacement, or rather a inappropriate framing or assumption about how to best go about meeting the stated needs. The problem essentially stems from focusing on the estimates, believing that the estimates are the purpose and output from the exercise. As so often is the case, even if that was our goal it's a goal best approached obliquely.

Let me attempt to illustrate an alternative frame and approach to meeting those needs by drawing parallels to a recent experience of mine.

Some weeks back I participated in a 24 hour rogaining (https://en.wikipedia.org/wiki/Rogaining). Teams are given a map with checkpoints that can be taken in any order as long as they're back within the given time budget. In this particular case that meant 24 hours of beautiful Swedish mountains, going up summits and down into swamps. With unpredictable weather conditions aided by compass, altimeter and a simple watch (no GPS or other fancy gadgets allowed).

Essentially a risky and uncertain environment. Tight deadlines and unfamiliar territory.
How do you tackle something like that? Not having a plan, shooting from the hip, seemed for many reasons like a rather horrible idea. But since even the simple act of estimating running speed when you'll be going trackless, scrambling over rocks or blazing new trails won't get you very far either, it's not entirely obvious how to tackle the task in a reasonably manner.

Since penalties for arriving late are very steep we need a plan that ensures we're not late, but we also strive to maximize time spent doing useful work (collecting checkpoints). Resolving this conundrum proved to be a challenge and team building exercise.
This is how we went about it.

1. Assess the whole

   Looking at the broad picture we tried to get a sense for the major decisions to be made. What direction to attempt to plot the course taking into account day/night cycles, type of terrain and how to preserve options for late in the race where they'd carry to most value.

   This generated a few key insights, yielding some parts of the course with few sensible routes and options for adjusting to schedule, they offered very good return on invested time but only if they could be taken in a way that don't jeopardize the whole. Other areas offered seemingly lower payback for invested time and energy but kept options open. One especially jarring insight was that the area that would give the quickest initial points would also leave us with a very tight endgame and essentially zero margins.

2. Sketch out a few routes based upon gut fee

   Having feel for the overall layout, interesting major themes and areas we sketched out a few possible orderings without much regard for viability. Guided by gut feel we mapped out a few rough candidates that covered the different areas. These candidates were all throw aways with the intent to get something concrete to talk about, enabling smaller grained conversations about choices and options in specific areas but overall probably yielding plans that would be either unrealistic or all too pessimistic

3. Building a first model

   Having talked about the broad picture and some select details, we had started to build a mutually coherent mental model of the task and a few concrete decisions. Those conversations had also helped highlight a few of our respective strengths and weaknesses. Knowing what type of terrain to prefer, what type of obstacles to handle at different stages of the race (fatigue contributes a lot here). But we needed something more to check the viability of our plans. Since the checkpoints were just revealed historical data for this terrain and route wasn't available so we had to look elsewhere for some rough bounds of what to expect. Collectively we had a few scarce data points from similar races, sharing a few properties with the one we were about to set out on. Based upon those we could fairly quickly determine some upper bounds with regards to total distance, ascent + descent. Knowing that we also need to make room for navigation, food stops and unknown unknowns that are bound to pop up when each curve on the map is 20 meters you might find a wall or ravine occupying the bit of reality you thought you were going to run through. The main discussion here centered on finding a common agreement on roughly what could make sense. This helped us factor in things like accumulated fatigue, roughly what trade offs would be involved when deciding between extra milage or vertical movement.

4. Use model on first sketches

   Having created a "good enough" model we took our sketches and put them through it to see what would come out. After some tweaking we got numbers that seemed to make sense. This led to additional route alternatives being investigated, cross checking calculations to ensure subsections atleast felt accurate enough to be meaningful.

5. Simulation

   Having a set of alternative routes having been both gut checked and modeled we created a set of simulations to gauge sensitivity of the solutions and use some brute force to perhaps find additional optional routes. Crunching the numbers revealed that the route we've mostly settled on would be a good choice unless our model and base speed was of with a fairly large margin. The problem being, if that would turn out to be true it changed drastically enough that it would be essentially 180 degree off from what we planned. That did little to relieve stress levels. Having done the work of setting it all up we used a few different cost models to get a feel for how the problem would behave. This spawned a few additional tweaks to the plan, mostly ideas on how to preserve options for as late in the game as possible but also to have some mid-race decision points on the form of "if we arrive here before we head for that CP otherwise we go straight towards that one".

6. Putting it all together

   Given the estimates, reference data, model and simulation results we had a plan. Plans can be good. But what was even more important, we had from building all of it, built a solid understanding of the task at hand, a strong anticipation for what would be required and discussed and agreed upon many details on what types of obstacles to prefer, how to pace the race, when and how to eat. We inked our route onto the maps, jotted down the expected times for each checkpoint for both the main planned route and optional extras. Noting both total time and time between checkpoints.

Given all of this we arrived at the starting line armed with a plan, but more importantly we had built a shared theory that we would spend the next 24 hours validating, updating and leaning upon to do the thinking after many hours of strenuous activity.

The knowingly silly precision of the plan (down to the minute for checkpoints) was never a commitment but still something we took very seriously to track progress towards and follow up. Why such discipline when we knew, even before the start, that it couldn't possibly be correct?

The counterintuitive reason for that being that although we knew it to be wrong, getting precise measures on exactly how wrong yields the most information to update the model and continuously learn and adapt. Essentially knowing not only if our guess was high or low, but the exact amount of our wrongness (or rightness) provides a much richer information environment from which to improve.

Another key aspect here is also that we did all of that work, not for someone else but for internal reasons, it was by no means a control mechanism revealed externally or committed to outside of the team. We were responsible to ourselves, to our own learning and performance.
I think there's a deep lesson there too.

Overall, we didn't get a single number right. Our precision was rather horrible but overall accuracy was extremely good, we wound reaching one additional checkpoint and finishing slightly ahead of schedule. Having covered 110km, 6400m of ascent in slightly less than 23 hours I can say I'm happy we estimated but not for the estimate, but for the shared theory and understanding we had.

As they say, the plan is nothing but planning is everything.