Metrics and More Metrics

Continuing a discussion that was raised in a recent discussion between Tom Grant and Saeed Khan where they (we) made a distinction between metrics about products that Product Managers use to monitor the world, and metrics about Product Managers for promotions and salary reviews. Some additional thoughts of mine, along with a lightweight PM assessment tool…

Metrics About Products

For the most part, metrics track the health of products*. We should be constantly monitoring things like:

• unit sales versus forecast
• average discount from list
• win/loss, close rates, or the portion of bids that result in sales
• customer complaints, support cases or bug counts
• actual development progress versus roadmaps or release plans

Etc. There are, of course, dozens (hundreds) of possible metrics to choose from. Importantly, these may indicate possible problems – but RARELY identify root causes or specific market change. Standard metrics give us a starting point for more questions. “What might be driving customers from the high end to our mid-tier product?” “Why is Europe reporting more installation problems?”
We (product managers) act as clinicians: interpreting the data and digging for real-world reasons why discounts have increased, or why the product mix has shifted, or why sales to retirement communities are suddenly accelerating. The “why” that we derive from product metrics is a combination of clever investigation, good field relationships, experience, and dumb luck.

Chase: X-rays confirm the fluid that almost suffocated her to death was from pulmonary edema.

• Foreman: Means the problem’s either in her heart or lungs.
• Thirteen: Tox screen’s clean for everything except the alcohol, and her B.A.C. was barely .05
• Chase: That means she only had one or two drinks, tops.
• House: And there’s no sign of trauma.
• Chase: How’d you know?
• House: Because if there was, Cuddy wouldn’t have needed me to take the case…

So product metrics help us track the health of our products, and drive us to think more deeply about cause and effect. What product metrics don’t do is help the Director of Product Management evaluate her team. How do we measure good product management?

Metrics About Product Managers

I’ve managed a lot of PMs in my time, but still don’t have any metrics about product managers that I trust. That’s counter-intuitive for someone who lives ‘by the numbers.’ Yet I’m hesitant to create formal metrics that smart PMs can “game,” thereby distracting all of us from the important work at hand.

For instance, we might rank PMs on their timeliness in delivering requirements. This sets up the perverse incentive for PMs to send crappy-but-prompt MRDs over to Engineering instead of taking the time to evaluate market facts. If we believe that product management adds strategic value, then we should avoid simplistic metrics. In a similar vein, engineering managers don’t pay their developers “per line of code written.”
(Steven Kerr’s “Folly of Rewarding A” seems ever more relevant when reading about US bond rating agencies. )

Personally, I’m looking for judgment and organizational savvy and customer intuition and creative problem-solving and technical expertise in a good product manager. That means I apply qualitative criteria to my quant-jock staff. Notice that good diagnostic instincts fit into this essential-but-elusive skill set. We don’t expect doctors to save every patient, but we do expect them to separate root causes from symptoms.

I’d appreciate input/comments from other PM executives about metrics they’ve used to evaluate their staff.

One way to supplement our own judgment is with thoughtful feedback from product management’s stakeholder groups. What do your VP/Director-level peers say about the PMs on your team?

• Does Development see a PM as truly representing markets / customers / requirements? Are your folks seen as product-savvy or technically incompetent?
• Does Sales (or Marketing) have what it needs to correctly describe your product? Do they want your PM in customer meetings, or find excuses to go it alone?
• Do executives consider your PMs as product experts? Is your bottoms-up strategic analysis strong enough to shape top-down planning?

A Product Management Assessment Tool

With encouragement from Scott Gilbert, I built a short assessment tool for product management teams. This is not intended to evaluate any one specific PM, but may be useful for a team-wide situation analysis. It raises some of the same questions, though: how are we doing as a product management group? Regardless of how individual products succeed or fail, are we fulfilling core needs and building the necessary organizational relationships? If you manage a PM team, consider a working session for your group to rate itself.

Sound Byte

Product metrics still requires us to apply our investigative skills to find why things happen. And these product metrics are not very useful in evaluating product managers themselves.

* As always, I talk about “products” in their broadest sense, including services and intangibles and other stuff that we expect someone to pay for.