Customization Debt

Richard Mironov -

There’s a pattern that I’ve seen at most B2B/enterprise software companies – and some B2C/mass consumer companies –  that doesn’t seem to have a name or much recognition. 
It’s a (subtle) trade-off between closing major deals now and the long-term scalability/supportability/sustainability of our core products.  Let’s call it customization debt, analogous to architectural debt or design debt or rapidly expanding bug queues.

Examples of Customization or “Specials”?

Typically, these are deal-driven demands from our largest B2B/enterprise prospects:

  • MajorCorp is ready to buy our enterprise MRP application.  But one of their divisions has an old (non-standard) parts database that cleverly encodes manufacturing locations in their part numbers, so that 123-MIL-56a7b8 is made in MILwaukee and 125-GUA-99a7r is made in GUAdalajara.  They need us to write a connector that converts these into our normal “part#” and “plant-location” fields.   How hard could that be?
  • NiftyBank is interested in our turnkey back-office consumer-bank-in-a-box solution.  But they have a special student checking account with fees waivers based on STEM class credits at select universities.  (Very innovative!)  So they need an extension to our account fee API to access this fascinatingly obscure computation.
  • The next generation of our ecommerce search engine (optimistically) ships in about 12 months.  MammothMarketPlace is ready to implement now, though.  So they are holding our contract hostage, demanding a “back port” commitment – we would promise to add every new feature to both the existing and new versions for the next 3+ years so that they don't have to upgrade soon.
  • MassiveInc has 22k employees, and wants to use our AI-driven business analytics package.  All of our other customers are on per-seat-per-month subscriptions, but MassiveInc has armies of Senior Analysts and Junior Analysts, so wants us to create two classes of paid subscribers: full-price Senior users and half-price Junior users.  (If we whip up these new personas/permissions/licenses – and word gets out – there are probably lots of other customers looking to pay us less for per-seat subscriptions.)

So anything we build for just one customer is “custom.”  Any new bit of logic or deviation from standard data handling is a “special.”  Agreeing to generate/send unique daily reports for one buyer is “bespoke.”  Creating a new SKU or licensing model for a new customer is a “one-off.”  These are usually accompanied by evidence-free arguments that dozens of other major customers need exactly the same thing, implemented in exactly the same way.

Alongside an unfounded-but-hopeful assumption that Development has some unassigned people who can slip just one more small ticket into the queue without delaying or sacrificing anything.  (Never been true in the history of the planet.  And see my 2015 tirade on why specials are bad for product businesses.)

What’s happening here? 

This is almost always a trade-off between current revenue (a bird nearly in the hand) and invisible long-term costs borne by the maker organization  (engineering/product/design). We promise to worry about it later, and somehow never get there.  Here’s the B2B organizational context that I see:

  • Every big customer has some asks that aren’t met by the base product.   Most have an archeological stack of old and very old and truly ancient systems – accreted over time into a unique assembly of poorly understood apps and workaround processes.  And some truly unique aspects of their business.  They each have a shopping list for us.
  • There’s intense pressure to agree to their top few demands.  This quarter’s top 10 deals are very visible (even to the Board), and company valuations are driven by current revenue.  Failing to close major deals often means layoffs.
  • Therefore, there is tremendous incentive for everyone to ignore the long-term work/commitments/risks/trade-offs/impact of “specials.”  Or to claim that the entire market wants X, and it should have already been on our current quarter roadmap.  Critics are “not commercial enough” or aren’t team players.  Revenue trumps technical arguments.
  • Specials are treated as ONE-TIME PROJECTS, with limited scope and fixed timeline and implementers who will move onto the next project as soon as this one is done.  (FYI, it doesn’t matter if the work goes to a Tiger Team, implementation consultant, sales engineer, or core Engineering.  Everyone is borrowed from their regular work for this.)  We stamp the work FINISHED and move on.  Minimal testing, zero documentation, and no ongoing support or staff.  Fingers crossed, it will never break or need changes.
  • Even though these happen 5 times every quarter, execs and sales leaders treat these as “very occasional specials” and rare exceptions.  Despite some R&D whining.  On the product/engineering side, we don’t relentlessly call this out as our standard operating approach, so go-to-market leaders aren’t forced to see the pattern or hear about the costs. 

 And so on.  Each of these feels like a small request, and comes with hard-to-argue-in-real-time revenue math: “This is an $800k deal, and it can’t cost more than $20k to build the silly widget that they need.  That’s a 40x return on a couple of weeks of development.”   We tackle these one at a time, and don’t see the bigger picture.  Engineering atherosclerosis. 

Why is this a problem?

Whether or not our biggest customers pay for their one-off/ bespoke/ unique/ custom solutions, they expect us to support them.  For as long as those specials are in production, which may be for decades.  And inevitably, some bug or tiny new requirement or operational issue pops up, long after the tiger team has moved onto other things – or other companies.  (Hint: the hottest of these BigCorp escalations happen on holiday weekends.)   Followed by a mad scramble to figure out/find/remember what was built, the underlying customer context, how it fits into our product architecture, and who might be available for some customization CPR.  Most likely, we pull someone from Engineering off their assigned work for this technical archeology.

Viewed as a once-per-year incident, this isn’t a big deal.  Engineering gets to play hero, and Sales gets to explain how this won’t happen again.  Revenue protected, engineering prowess proven, happy customer.  High fives!

But at scale, it’s a major-but-hidden disrupter to R&D.  With an installed base of hundreds of corporate customers, we’re likely to see a “special” that needs serious repair every single week.  And that can dramatically slow down the technical innovation intended to drive next year’s revenue.  (It also tanks Product/Engineering morale, since our plan for a few truly strategic roadmap items was already optimistic.  We’re punished when anything slips.)

But this Special won’t ever break or need enhancements!

We all know this is a lie, but we desperately want to believe it.  Perhaps if we say it out loud enough times…?  Instead, at some randomly inconvenient time, something will happen:

  • Business requirements change.  MajorCorp opens a new manufacturing plant in HANoi; NiftyBank designs a no-fee checking account for consumers with more than 4 vowels in their last names; MassiveInc adds a mid-level title between Senior and Junior.  New privacy rules limit the data that a customer app is allowed to return.
  • Some part of the software universe changes.  Google updates Chrome.  AWS obsoletes some back-rev API.  A popular application monitoring solution shuts down.  The Federal Reserve adds a new tag for suspicious funds transfers.  We upgrade part of our back end to something more modern. 
  • GiganticCorp is finally ready to migrate from v2.5 to v7.0, but now needs each of their v2.5 specials ported or re-implemented.  No one on either side remembers the details of what we did 5 years ago.
  • Or the dreaded “Special Widget X isn’t working, and we don’t have anyone on our side who understands it.  We pay a huge annual license fee to you for a working system.  You built it, you fix it, and it needs to be running again by Monday morning.”

Yes, but They Paid For It!

Collecting money for our work is great!  But the translation from money-in-the-till to identified-and-prepared-developer-ready-for-a-call almost always falls down.  First, it’s very rare for any of that sweet incoming revenue to actually be allocated and spent on a specific someone with long-term code support responsibilities.  We might fund the initial short-term consulting/integration work (internally or externally) and take our victory laps, but there’s almost never a mechanism to fund ongoing software maintenance for specials.

And we charge the customer a one-time fee (or bury the cost in the overall license).  There’s no separate line item to keep this operational in Year 2 or 3 or 10.  Finance and Sales move on to the next opportunity.  So whether we collect money ignores the essential problem. (And that money almost never gets to Engineering.)

So What Can We Do?

I don’t have any easy fixes.  (Love your suggestions!)  But identifying this as a real issue is one first step.  
Product should spin up a starter list of existing specials and custom items.  Non-trivial, since there may not be any central repository, and various groups may have done this in various ways (e.g. Tiger team, outside consultant, field SE, 2nd level support, open source, GenAI-sourced sample code…)  This won’t be a popular query, and some “special” projects have already been forgotten. 
Then as CPO, I need to bravely raise this as a “pay the piper” exercise.  Imagine that I shame the rest of the exec team, dragging everyone through my incomplete recitation of 25 recent specials.  (“Remember X?  Remember Y?  Remember Z?  Remember ABC?  Remember Project Sisyphus?”  “Okay, please stop.  We get your point.”)   As a product leader,  I feel obligated to connect the ten-meter-high dots. 

Then when?

  • I could assert that we need 5-10% of a named developer to own each of these specials for the long haul, anticipate problems, and make repairs as needed.  So I’d ask for 2 additional developers forever to babysit our known list of 25 commitments.  And another 10% engineering headcount each time we do this again.  So we now have a team of “specials engineers” actually assigned.  And explicitly expanded with each new bit of bespoke work.  (BTW, I’ve never managed to sell this to an exec team.  But it highlights the ongoing costs and risks.)
    And I’d expect it to be tough on Engineering morale and staffing: not a team that our best folks want to be on.  So we’d try to position six-month assignments to these “specials” teams as something other than demotions or punishment duty.
  • Or we create an explicit invoice line item for the customer, naming the special component, and charged every year for as long as it's in production.  That would let us hire an outside consultancy/agency and pay them to understand the tech and be on call.  With 25+ custom bits, this might be an attractive partnering opportunity.  (We shouldn’t keep any margin on this.  We don’t want to create yet another perverse incentive to encourage more specials.)  
  • Or we can gift the solution back to the customer, for them to maintain internally.  “You paid for it, it’s yours now.”  We should still expect their-developer-to-our-developer questions and support issues, but without holiday weekend responsibilities to respond.

An alternate approach that’s mostly theoretical: establish a short list of fully implemented customization options, and refuse the rest.  We might use a third-party reporting platform, and let user/consultants knock themselves out writing custom reports.  We might design a colors/logos/I18N name string architecture that lets customers upload their corporate colors, images, and branding text strings to make their version of our app look more made-to-order.  We might include unused database fields for their unique information... In practice, I don't think we can ever catch up to the randomness and complexity of enterprise systems.

Or finally, we can leave this buried, unaddressed.  One more reason why expecting Engineering to work at full capacity on planned stuff is optimistic.

Sound Byte

Like other kinds of tech debt, customization debt is hard to see; hard to identify as a behavioral pattern; hard to hard to quantify.  But it’s a hidden taxes on our maker teams.  Recognizing it as an unconscious trade-off between short-term revenue wins and long-term maker productivity and plannability.  It may be time to fight this next battle.