It seems like a rare organization today that doesn’t rely on some sort of software system to market or manage some critical facet of their business. As they grow or their business changes, it’s not uncommon that they need to find and replace these enterprise software systems with something newer or more powerful.

Implementing a new enterprise application is a mountain of work. But just selecting the right replacement solution is an entire project in itself – one that can (and should) includes people from all over the organization. The selection project usually involves requirements gathering, proposals, demonstrations, site-visits, negotiation, and often other more involved steps such as a pilot program. It can take many months.

Every selection project also always includes reference interviews. But never, in my opinion, enough.

In my career, I’ve been involved with hundreds of software select projects – either within an “acquiring” organization or on “selling” side working with the selection teams. Only rarely – perhaps 1% or so – do organization ask for more than 3 references to call.

Yes, this is a (multi)million-dollar purchase decision. But if this is a line of business system, one that you’ll count on for years to run your business, it’s worth much more than that. What happens if your new online order-entry system goes down for 6 weeks? How do you process orders if your new e-commerce site that drives 80% of your business is not available to customers? How do you stay in business if your new billing system can’t generate bills – meaning you can’t collect cash – for months? A million dollar purchase decision can affect all of your organization’s annual revenue. Yet, time and time again, software selection committees just call the 3 customers the software vendor gives them.

Anybody can give you 3 reference accounts.

Here’s what I tell my customers: ask for and interview 20 reference accounts. Ask for names of people in different positions – end-users, I/T, managers, executives, and finance.

One eye-opener is when a software company has difficulty coming up with 20 references for you to interview. If they do provide you with the names, call them and really talk with them. Don’t just ask the basic “how did it go?” question. Really dig into their experience.

Let’s face it, every enterprise software implementation project has issues, especially if it involves configuration, tailoring, or customization. The important thing to find out is how the company reacted. Did they pull out all the stops? Did they work nights and weekends to meet your due dates? Or, did they stop communicating? Did they have trouble allocating resources to solve the issues? Those are the real answers that you want. If you really truly want to be thorough, ask the references if they know of any other customers (that the software company didn’t give you) who have had issues.

Calling and interviewing 20 references is time-consuming, no doubt about it. Twenty in-depth interviews will require you to really engage, focus and not just go through the motions. But it will be worth it in the long run. If this is a mission-critical application, be as thorough as you can possibly be in order to ensure your organization’s success.