Measuring Product Managers (in Swedish or English)

I’m back from a week of product management workshops and seminars in Sweden, including a Product Leadership event hosted by Tolpagorni’s Magnus Billgren.

In a half-dozen discussions with the heads of product management groups, I was struck by how familiar their concerns are. We could have been in Sunnyvale rather than in Stockholm. Topics that came up repeatedly:

What metrics do we use for evaluating product managers, and how can we tell if they are doing a good job? Are there PM KPIs*?
Our agile development teams tell us that roadmaps are no longer needed, but our customers and sales teams still demand firm commitments.
How should we organize our product teams? Senior/junior, technology vs. market segment, product owner vs. product manager?
What do career paths look like?

These were discussions about how organizations and real people interact in a technical environment. They reflect what execs running product management groups worry about: boosting effectiveness, building relationships, and relating the business of product management to the company’s overarching business.

Some product geography

Sweden’s tech industry is heavily weighted toward B2B and telecom, with many firms clustered around Ericsson in Stockholm’s northern suburbs. Product managers stay with their companies (and in their current positions) much longer than we’re used to in Silicon Valley. And since telecom customers have long implementation cycles, product releases are methodically timed. Internally agile development teams are faced with un-agile, contract-driven shipment dates. In search of cost reduction and talent, most teams are split geographically. In other words, not very different from B2B tech firms back home.

With limited time in each workshop, I used an old agilist technique of soliciting concerns, then having attendees rank them. Voila! A backlog of product management issues that we

time-boxed and attacked in order. (And a handy topic list for my next few posts.)

Here’s a summary of discussions about the first topic above – metrics for evaluating product managers, or “PM KPIs” – which drew heavily on an assessment tool and post that Scott Gilbert and I created in 2010.

What should we measure?

There’s a lot of confusion between metrics about products, assessments of product management teams, and scorecards for individual PM job performance. Taking each in turn:

[1] Product metrics are an embodiment of our market and financial goals, such as “units shipped” or “incremental revenue” or “gross margin in our reseller channel” or “market share in Asia Pacific.” Results depend by an infinite set of external and internal factors (sales compensation plans, competitive price shifts, PR, on-time release, weather) that are mostly out of a product manager’s control.

Product KPIs are surface indicators that should trigger deeper questions. Why are margins higher in Latin America than Europe? Can we see any impact from our competitor’s social/community efforts? Which segments should we target for next quarter’s whiz-bang features? What internal sales education techniques work the best? As problem solvers, we take these as challenges.

[2] Department-level assessments help us improve the business of getting great products shipped. Success is a cross-functional team sport, so these tend to be a mix of objective and subjective scores. “Are requirements complete and reflecting market needs?” is paired with “does engineering agree that PM is the primary proxy for customers?”. “Customers understand our core benefits/features” goes hand-in-hand with “Sales is eager for product management to meet prospects.”

Again, we use these metrics or assessments for a problem-solving approach. If messaging is lost in the handoff from product to marketing to sales to channel to customer, where are we getting confused? If late-arriving requirements are frustrating our developers, how can we anticipate better (or convince Engineering to relax a little, or keep Sales from promising futures)? KPIs should be weathervanes, not career bludgeons.

[3] Finally, the hard question that my Swedish colleagues wanted answers to: how to quantitatively evaluate individual product managers? Jag vet inte. (That’s Swedish for “Honestly, I don’t know.“)

In my experience, the best product managers throw themselves against unanticipated product-specific market and organizational issues. They practice higher-order problem solving, making progress fiendishly hard to measure. Great PMs fill the organizational gaps, work intramurally, and often hang back when credit is being awarded. Their success can be invisible: the absence of confusion and unhealthy politicking.

I’m hesitant to assign numeric goals to my individual product managers. If I give bonuses for perfect requirements, I’ll get 100-page MRDs (instead of time spent with customers). If I reward pure revenue performance, my folks might spend all of their time on the road (and neglect next quarter’s planning). Etc.

My best indicators of a product manager’s success are subjective:

Does she have a roadmap agreed to by Engineering and Marketing (as a proxy for consensus building)?
Does Engineering consider her a core part of their team (and not a technical lightweight)?
Do Marketing and Sales see her as the source of product facts, strategy, and market intelligence?
Do I trust her judgment, objectivity, whole product thinking, and personal integrity?
Does the team show more focus (and less panic) over time?
Are other departments trying to recruit or borrow her?

Not as scientific an answer as I’d like, but the basis for discussion here and in Stockholm. If you have more quantitative approaches, please share.

Sound Bytes

Measuring product success is easy; gauging departmental success a little harder. I’m still stumped for general, quantitative metrics for individual product managers.

* Key performance indicators (KPIs) are quantitative definitions of success, which are then used to measure progress or score results.