The DIY Illusion

I’ve recently talked with several companies that are spending $1M/year or more to build generic software that is very similar to commercially available products. In each case, they’ve ended up with partial solutions, late delivery, disappointed users, and unconscionable costs. I call this the Do It Yourself Illusion: the idea that broadly available commercial products don’t give us exactly what we want, but our internal team can whip up just the right thing.

Recent discussions included an ecommerce company that built its own website performance monitoring tools; a circuit board manufacturer that whipped up a custom ERP application; a training company with a home-grown learning management system; and a security software company hand-crafting its reseller/partner portal. None were essential to their core businesses, but each group convinced itself that its needs were special enough to warrant an in-house project. Do It Yourself.

Let’s imagine how we get there, for instance with the partner portal:

Our business users asked for a standard portal, but with some unique twist. “Some resellers also share vulnerability data with us, so we need custom categories tied to anonymous data submission.”
A quick review of commercial (off-the-shelf) partner/reseller SaaS apps confirms that our needs are truly special, and our unique requirement is too important to ignore. We reject all currently available partner portals.
Our technical lead is tapped for a quick estimate to build a similar application internally, and approve funds (budget) to build it. Some basic code that shows reseller revenue-to-date and provides an anonymous drop box sounds straightforward. How hard could it be?
The business team put in a budget request to cover 2 or 3 technical resources for a couple of quarters: let’s get this approved and started, because resellers are waiting!

We’ve just made a strategic conceptual mistake: comparing an imaginary future outcome with currently available software implementations. We’ve airbrushed out trade-offs, delays, incomplete requirements, technical challenges, unfilled developer chairs, testing, evolving needs, and everything we experience building real software. We’ve sold ourselves on a more perfect result.

Underneath this is a long series of poor assumptions:

Clear understanding of the problem. Rather than do a full needs assessment, we have end users jump into system design. Let’s assuming that our business processes can’t change, and that vendors don’t understand what’s truly unique about our business. We move directly to sketching dialog boxes.
Available resources. IT has no idle developers, and a dozen unfilled positions. Important projects have first dips on new hires. We’re not really going to get started on this until 2Q 2017, and then we’ll only be half-staffed.

The feature iceberg. Most of the work to build a partner portal (or anything else) is below the surface. To support a few visible actions, it will need lots of invisible features like user password reset workflows, support for multiple browser, checks for valid inputs, malware protection, cloud failover, test automation, consistent user experience, privacy controls, time zone conversion, local language support, etc. Internal projects tend to discover these hidden requirements along the way. How many in-house apps only work on Internet Explorer 8 (or 7 or 6) because there’s no plan for Chrome or Firefox or Microsoft Edge?

Evolving needs. We’re only thinking about this month’s requirements. But partner portal vendors compete with each other for the next market-relevant innovation or feature. So commercial customers will probably get deal registration, marketing fund management, lead sharing, contest creation tools, and a Partner Advisory Board invitation template. Or whatever partner managers like ours are asking, as vendors plan for the next new thing.
How big is it really? Neither our users nor development team have ever built a partner portal before. From the outside, it looks pretty simple. All developers are optimists; every feature can be built in a weekend.
Commercial quality and support. Our end users expect solid, mostly bug-free software. And training videos to show partners how it works. And someone to fix problems today, because our reseller revenue is on the line.
One-time project funding. Software is never finished, and new user needs are constantly appearing. It’s rare for anything to naturally reach a pure maintenance state. Yet IT budgets tend to assume that we’ll completely finish our portal on time, and move staff onto the next hot project. History tells us otherwise, especially as we push out required features to hit our dates.
Architectural debt. One-off applications and user-designed workflows are rarely built with a technical architecture in mind. But they stack up, and the back office gets more convoluted over time. Easy things get harder, and “how it works” descriptions get longer. We meander into complexity.

It’s easy to make the discussion personal. “IT is lazy, or stupid, or disinterested, and refuses to consider all of our great new ideas.” “The Business is technically unsophisticated, or ADHD, or unappreciative, and they are welcome to sit at this keyboard to show us just how easy building portal pages is.” But this problem is endemic: it’s can’t be written off as individual failures. This is a strategic mistake.

I see the DIY Illusion as a fundamental misunderstanding at the leadership level about software economics. We repeatedly set aside our own experience, imagine away all of the real costs / challenges of building commercial-quality software, and ignore nearly-complete market offerings.

Let’s run the numbers.

IT will need a full scrum team to build/test/maintain a decent reseller portal and its infrastructure. Maybe two full teams. Forever. That means we’re spending $1M-$2M/year on this project. Forever.

Commercial software companies price their products for a competitive mass market, so no one customer bears the full development cost. (You can’t build your own small business accounting solution for $30/month, but Xero can rent it to you at that price.) Vendors amortize a much larger development team across hundreds of thousands of users. This includes tons of iceberg work: support; user-centered design; testing new browser versions; security audits; cloud backup and availability; language support; international privacy standards. And they are forever chasing (or being chased by) the next market-relevant feature.

So imagine a different opening conversation with between a Business Unit VP, CFO and CIO: “We could license a commercial partner portal for $15k/year, but it won’t let us custom-assign some selected resellers into our vulnerability data sharing program. Or, we could build exactly the portal we need for $1M+/year.” I’d expect to hear some new options:

“Gee. We only have a handful of partners in both categories. Let’s just assign them two separate logins, and they can decide which half of the system they want to use. Probably different users at their end, anyway.”
“I bet other buyers have needed similar kinds of special categories. Let’s offer $75k to the vendor who’s willing to pull this feature forward off their roadmap.”
“Most good SaaS products include APIs for custom functions. Can we write our own partner lookup and plug that back into a supported product? Probably save us 90% of the cost to build the rest of the solution.”

Faced with a realistic assessment of costs, timelines, risks and the need for a strategic IT focus, I’d expect most companies to push commercial software as their default option everywhere. Claims of uniqueness and special needs should be hard to sustain.

And that would free up very scarce, very valuable internal development resources for the very few ongoing projects that are core to the company’s future. That deliver >6x economic value to the company. That truly have to be DIY.

Sound Byte

Commercial software is almost always a huge bargain versus internally built systems, both in time and money. As technical and business leaders, we should closely inspect every request to do it ourselves. And not fall victim to the DIY illusion.