System selection reviews

By Andrew Clifford

When you are selecting a new software package, do not rush the preparation of your vendor questionnaire. A well-prepared questionnaire helps you review the impact of each option easily and effectively, and saves you time in the long run.

Many business have a process for selecting new software packages. This might include an invitation to tender (ITT), request for information (RFI) or request for proposal (RFP).

Within your process, you need to investigate how well each option meets business requirements. You need to look at commercial aspects such as cost and vendor viability. You need to investigate whether each vendor can deliver to your plans.

You also need to understand the technical issues of each option. This will identify options that have such severe problems that you should not select them. Whichever option you do finally select, a knowledge of the issues will help you understand what you need to do to implement it effectively.

Technical reviews usually involve sending a questionnaire to each vendor, and reviewing their responses. I have been involved in many of these reviews. They are often frustrating.

• Often you do not get back half of the information you need, because you did not ask the right questions.
• Often you do not get back the sort of answers you expect. You get one-line answers when you expected an explanation, or pages of marketing fluff when you wanted a one-line answer.
• Many questions are pointless. Questions like "Can the data be backed up?" always give the answer "Yes".
• When you come to review the answers, there are arguments about what is good and bad. The review descends into political infighting and which vendor gave the best lunch.

These problems are surprisingly common. They completely undermine the review. They stop you identifying critical technical issues.

These problems are caused by a lack of preparation. A little preparation goes a long way.

• Start with a standard questionniare template so that you know you have covered all the important areas. Run a workshop with all relevant parties (often different technical specialists from the IT department) to refine the questions.
• Think what you are looking for in each answer. Think what a good answer would look like, and an unacceptable one. Think about the limit of acceptability: an answer that is OK, but only just.
• Carefully word each question to ask precisely for the answers and level of detail that you need. Never ask a question to which everyone can answer "yes".
• Agree a weighting and scoring scheme before you send out the questionnaire. One easy way is to take an average of the weightings from different stakeholders.

This approach helps you understand the technical issues with each option. It is effective, fast and cheap. It provides an auditable process to show that you have investigated properly.

This approach is the basis for the system review method that we promote at Metrici, about which I wrote last week. You can use the same approach, and similar templates, to help you during system selection, and to review systems during projects.

It isn't rocket science. But it works.

Subscription