Are you a Story Point Expert?

A Scale of Expertise for Story Points

I’m currently working on a presentation that is ostensibly about the Dunning Kruger (DK) effect (note: I’ve linked to the Wiki, but the meme “stupid people don’t know how stupid they really are” is now probably more important than the actual research). My ultimate purpose is to explore how agile transformations are complicated by different levels of understanding – i.e. of how the Dunning Kruger effect hinders open communication and learning.

Because it is easy to get lost in any discussion of DK, (owing, ironically, to the DK effect itself), I wanted to use a practical example that agilists would understand.  I came up with levels of expertise/understanding for story points (as story points are a relatively simple agile concept).  This might not be definitive (I’m bound by my own DK horizon, after all!).

Level 1: Executive Understanding

The Purpose Is To Measure Productivity for Continuous Improvement

Often limited to bullet points on a powerpoint presentation, this level of understanding is primarily WIIFM (what’s in it for the stakeholder or ROI for the business).  This level tends to focus on business outcomes such as productivity through continuous improvement activities.  Just enough sophistication at this level is provided to distinguish story points from ‘how we’ve estimated projects previously’, and with enough business-speak to make it sound smart and become a meme.

People at this level believe that:

  1. Story points are not time estimates, but are indeed relative sizes.
  2. Fibonacci points are cool, and sound way more sophisticated than T-Shirt sizes.
  3. Team/project progress can be measured in story points per sprint, and this is known as “velocity”
  4. Velocity is a measure of productivity that can be tracked, so management will have transparency into team and project performance.
  5. You can target velocity as a continuous improvement goal. Higher velocity is good.

Level 1 is how agile is often sold to executive management and non-involved stakeholders. 

Level 2: Official Agile Position

The Purpose Is To Quickly Estimate Stories for Team Planning

At Level 2, story points are understood as a team-level communication device that allow teams to estimate faster and better.  They are to be used primarily as a team-planning tool.

People at this level believe that:

  1. Scrum teams should ideally meet story point commitments.
  2. Story points help protect teams from schedule micromanagement.
  3. Velocity/story points are a team statistic that shouldn’t be reported up.
  4. Story points are a great way of ensuring stories are ‘small’.
  5. Poker planning is a good, effective way to estimate stories.
  6. Dev Bugs shouldn’t be included in velocity, but bugs for in-the-wild bug fixes should.
  7. Moving averages should be used to calculate velocity
  8. Team velocity variations are used as learning points for SM’s/Coaches.
  9. Triangulate stories regularly to keep the meaning of a story point constant.

Whether it is CSM or PMI-ACP, level 2 is where certification aims for.  More in-depth than a powerpoint, this level encapsulates the mechanical aspects of ‘big A’ Agile expounded by early 21st century authors like Mike Cohn. 

Level 3: SAFe

The Purpose Is To Standardize Stories for Project Planning 

In order to facilitate scaling SAFe holds that as story points are a fast way of estimating work, with some tweaking they can be leveraged to estimate project duration as part of a planning process.  This is just one way in which ‘Scaled Agile’ has to be different from ‘Small Agile’. 

People at this level (are forced to) believe that:

  1. Product Increment Planning is important, so story points should be standardized as much as possible.
  2. Velocity and Story Points are enterprise-level statistics.
  3. Story Points are a standard unit of size/time.
  4. Velocity can be calculated on a per-developer basis, and used as a capacity planning metric across the Release (PI).

Yes, SAFe has its own level!

Level 4: Empirical Pragmatism

The Purpose Is To Have Small Stories for Fast Cadence / Low W.I.P.

A major purpose of story point estimation is to leverage story point estimation to help refine stories into small, uniform units of work, allowing for repetition, failing fast, and predictability.

  1. The purpose of backlog refining/decomposition is to create small stories.
  2. The way to maximize velocity is by keeping developers focused on finishing stories.
  3. Any measurement system (including story points and velocity) can and will be gamed by the team.
  4. Any measurement system (including story points and velocity) can and will be abused by management.
  5. You can’t target velocity as a continuous improvement goal.
  6. For any particular team, points can actually be translated into time (based on past performance) despite what Cohn says.
  7. T-Shirt sizes are actually more sophisticated than using Fibonacci (as they are less open to misuse, and Story Points aren’t supposed to be cardinal).
  8. Story point triangulation confuses more than it clarifies owing to changing technologies and skills.
  9. ‘Epic’ level estimates can’t be decomposed or compared to Story Point estimates for planning purposes, so don’t even try. 
  10. Cross-team velocity variations should be completely ignored – they mean nothing.
  11. Commitments are sometimes useful for capacity planning, but not beyond that.

Level 4 is where the majority of Scrum Masters end up after a year or two in the field.  It is where peer-to-peer group conversations tend to happen in the larger commercial Agile Conferences (as opposed to unconferences – see below).

Level 5: System Thinking

The Purpose Is To Capture System-Throughput Statistics

At this level it is understood that although Scrum and XP are nice, they merely describe processes that are acting as sub-systems within a larger context.  Story points are not particularly useful at any level, as their weaknesses often outweighs their utility.

People at this level believe that:

  1. #noestimates and all that entails  (estimates are waste)
  2. For a functioning team, story estimates have a stable story point average of between 3.5 and 4 (depending on how team treats ‘8’), so counting cards/stories on a groomed backlog (and multiplying by 4) is just as effective as estimating individual stories.
  3. Sprint Commitments are at best unnecessary, at worst, evil as they can distort other statistics.
  4. Velocity captures capacity throughput, which is different from productivity.
  5. Team should aim at 80% max loading (so most times will have slack at the end of the sprint).  This is good.
  6. The purpose of estimation is to have good conversations, not to be accurate, or even to estimate.
  7. Effects of system improvement (and long term effects of team improvement) lead to increased capacity, not increases in velocity and in fact, velocity should be stable over time.
  8. Monte Carlo and other statistical methods can be used to provide forecasts (not estimates) based on exceptionally small samples of empirical data (i.e.
  9. Actual velocity values aren’t enterprise statistics, but trends of velocity can be.
  10. Throughput (velocity) is only one system-level statistic, that may not be the most important/important at all.
  11. Cross-Team variations can in fact be used as system diagnostic tools.
  12. Even bugs-in-the-wild should be excluded from velocity/throughput calculations.
  13. Story points aren’t cardinal (so story points really shouldn’t be added/divided etc.) – even though we sometimes treat them as if they were.

Level 5 is for Agile Coaches that have discovered Reinertsen’s “Product Management Flow”, Snowden’s “Cynefin” framework, and/or even read some Deming.  Experienced ‘Agile Coaches’ (capitals intended) like to hang out at this level, and hold unconferences.  It is where development teams are still relevant to the business, but the scope of enquiry expands to the whole of human experience.

Level 6: Value Thinking / Lean UX

The Purpose Is To Maximize Learning/Business Value

Although the notion of business value is embedded in Agile canon right from Level 1, it is not until level 6 that the insight that value is ‘all that matters’ becomes dominant.

People at this level believe that:

  1. Story points aren’t nearly as important as business value, so should be largely ignored.
  2. Goals of decomposing stories into small development units is often counter to goal of value discovery (i.e. creating an MVP that tests an hypothesis).
  3. Sprint commitments are paramount – but should be at the experiment/MVP level (aka Sprint Goal), not the body of work (story points).  (and yes, this is stipulated in the Scrum Guide at Level 2 – go figure!).

Level 6 is where agilists who have crossed over to ‘Enterprise Agility’ tend to hang out.  As the new trendy, Enterprise Agility is also subject to it’s own DK, which is orthogonal to this DK curve, of course…this isn’t that.

Level 7: [Beyond The DK Horizon]

The DK meme suggests that we are not able to comprehend that levels of understanding exist too much above our own, because we don’t have the intellectual tools to know how much we don’t know.  Now, my comprehension stops at Level 6, though I’m thinking maybe AI and #nocode enter the  fray….  However, perhaps you can see further than me…if so, please let me know, or better yet, write your own blog about Level 7, 8, and 9!

Evaluating Your Personal Level

In order to figure out what level you are at on this scale:

  1. Starting from Level 1, circle all the statements that you believe in.
  2. Some statements are superseded in higher levels, but are restated in even higher levels.  If you truly understand why they are restated, uncircle the lower level selection.
  3. You are at the first level that you have a circle in.

Why the first level that you have a circle in, and not the last, or level where you have the most circles?  Each level supersedes previous levels on the basis of fundamental principles.  Sure, we may intellectually understand a few of the higher level concepts, but that’s not the same as truly grokking the paradigm shift that the higher level represents.  Once we are truly at the higher level, we know that adherence to lower-level concepts are at best, pragmatic, and at worst, just plain wrong!

What do you think?  Did I get the levels about right?  Does this gel with your experience?  Is it useful? What would you add to improve the concept?

Paul Osborn

Leave a Reply

Your email address will not be published. Required fields are marked *