What are the benefits of having a repeatable evaluation process?

Is it a good idea to use a spreadsheet for software evaluation and comparison?

How can we save time and effort in the software evaluation process?

How is software evaluation and selection different in a SOA environment?

Is it a good idea to download and evaluate software packages before an RFP is created?

Criteria- vs. requirements-focused evaluation - What is the better approach?

Do evaluation criteria and requirements mean different things?

Is it a good idea to break down evaluation criteria into features?

What is the difference between feature and requirement?

What are key characteristics of high-quality requirements?

How many requirements can my manager reasonably expect me to author during a typical working day?

What will it cost to write a single high-quality requirement?

What are the benefits of having a repeatable evaluation process?

Without a repeatable process, each evaluation project more or less reinvents the wheel. Planning, estimating, budgeting, and so on, are based on shaky foundations.

As a consequence, it is of great benefit to have a repeatable process. To learn more, please see here.

Is it a good idea to use a spreadsheet for software evaluation and comparison?

While spreadsheets are commonly used to compare software products, some shortcomings are obvious.

It is not that easy to represent a multi-level hierarchy in a spreadsheet. The "criterion-feature-requirement" structure constitutes a three-level hierarchy. Of course, criteria can be nested, which would add more levels to the hierarchy. In addition, high-quality requirement definitions encompass a set of attributes. As a consequence, a spreadsheet is not really adequate to accommodate an Evaluation Specification.

For the above reasons, it is highly recommended to use a solution that has been specifically designed for Evaluation Specification authoring, such as the Evaluator's Application Suite (EAS).

How can we save time and effort in the software evaluation process?

The answer lies in active vendor participation. However, key to active vendor participation are well-written requirements.

Errors which originate in requirements tend to be the most expensive and troublesome to eliminate later. With well-written, complete, and unambiguous requirements, it is easy for potential vendors to respond to your RFP.

Without well-written requirements, you would need to invest a significant amount of time to make sense of vendor responses. If potential vendors were not able to clearly interpret requirements, you would need to provide additional information, and/or the vendor needs to provide information. In other words, you spend extra time. Thus, doing things right from the beginning saves time and effort  

How is software evaluation and selection different in a SOA environment?

In a SOA environment, the focus in on integration. To learn more, please read the article 'Software Evaluation and Selection in a SOA Environment'.

Is it a good idea to download and evaluate software packages before an RFP is created?

In general, while it is very interesting to do an initial hands-on software evaluation, more often than not it proves as a waste of time at the end of the day. Before you have created an Evaluation Specification (which precedes an RFP), you do not exactly know what your requirements are. Hence, you do not have a yardstick that helps you determine whether the software package in question meets your requirements.

Criteria- vs. requirements-focused evaluation - What is the better approach?

By definition, a requirement is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to stakeholders. For example, functional requirements define expected behavior, such as "A user with administrator role shall be able to create a new user account" or "A sufficiently privileged user shall be able to create a new folder". Non-functional requirements define expected quality.

A criterion refers to a standard or test by which individual things or people may be compared and judged. In general, criteria are relatively coarse-grained, such as "Installation", "Document Editor", "Check-in/Check-out", etc.

Requirements-focused evaluation is considered the better approach, since you clearly define what you expect. So, you can tell when things are not as expected.

Criteria-focused evaluation saves time in the early stages of the evaluation process. The number of criteria would typically be much lower than the number of requirements. However, you will spend more time in the later stages. Very often, evaluation teams need to spend unplanned time since time needed for discussion and issue resolution is grossly underestimated. 

Do evaluation criteria and requirements mean different things?

There is no formal definition of evaluation criterion. Typical examples of evaluation criteria are: "Installation", "Team Development", "Utilities", etc. Sometimes, evaluation criteria are even phrased as questions.

Evaluation criteria like the above mentioned real-world examples are relatively coarse-grained and leave ample room for interpretation. A criterion may refer to a collection of features, to a single feature, or to a high-level quality attribute (e.g. "Reliability"). Often, there is no distinction between functional (behavioral) and non-functional (non-behavioral) aspects. As a consequence, it is not easy to determine a score that objectively expresses how well a certain product meets requirements.

In contrast, a requirement refers to a single entity, a single piece of functionality, or to a single quality characteristic. For more information, please please read the article 'Writing High-Quality Requirements'.

Is it a good idea to break down evaluation criteria into features?

Breaking down criteria into features helps strengthen the focus. Vendors can respond as to whether a certain feature exists or not. For example, if you just asked whether the vendor provides an installer, the answer would invariably be "yes". This would not be very helpful though.

In addition, you can outsource the assessment of RFP Responses. For example, you may wish to outsource the assessment of one or more RFP Responses to an IT consulting firm and thus share the workload. If you just specified criteria, the probability would be low that persons involved in the assessment have a common understanding of the precise meaning of criteria. 

What is the difference between feature and requirement?

By common definition, in computing, a feature refers to a beneficial capability of a piece of software.
 
A requirement is a singular documented need of what a particular product or service should be or do. It is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user.

Since both feature and requirement refer to capability, there is no clear distinction. Sometimes, the terms are even used interchangeably.

By experience, multiple requirements support a feature. However, there are rare occasions where feature and requirements seem to be one and the same. Analysis is needed to determine whether this is really the case.

What are key characteristics of high-quality requirements?

Requirement author and vendor representatives must have a mutual understanding of the meaning of requirements and interpret them in the same way. As an analogy, the more precise a question is, the higher the probability that you get a meaningful answer. To learn more, please read the article 'Writing High-Quality Requirements'. 

How many requirements can my manager reasonably expect me to author during a typical working day?

Experience shows that it is reasonable to expect a domain expert  to author 4 high-quality requirements per working hour on average, which amounts to roughly 30 requirements per working day. 

What will it cost to write a single high-quality requirement?

Based on the assumption that a domain expert authors 4 requirements per hour on average, authoring a single high-quality requirement would cost in the region of US$ 20. Of course, key influencing factors are salaries and overhead expenses.