Some thoughts about 'Pilot Programs' for evaluating new IT systems

Recently, I have been involved in a few projects with my clients where the stakeholder teams wanted to perform “pilot programs” as part of the vendor evaluation process.

It goes without saying that a strategy of “try before you buy” as a form of assessing an IT solution can be advantageous to a company before signing up for a long term partnership.

That said, what I have seen is that even a short term, three to five month, pilot can cost over $100,000 to execute successfully and that’s not including the soft costs involved like the resources needed to run the program (especially if a side-by side comparison of two or more systems are being evaluated), the data security and IT life cycle assessments, testing and validation requirements, the strategic sourcing and contracting activities, and so forth.

With the movement to SaaS subscription services prevalent in most organizations, we need to understand that the big dollar line items on a quote or SOW come in the provisioning of the system. The vendor needs to acquire the infrastructure through their cloud provider to create your environment. They need to configure the platform to your requirements and specifications. They may need to load company data into the environment. They need to train your user base to competently utilize the application. In short, there is a considerable amount of time and resources that need to be put forth  by the side to get our pilot up and into production. Once that part is finished, the actual cost of running and maintaining the application is cost beneficial to your compared to hosting on-prem or in a dedicated hosted instance, hence the reason we look towards SaaS solutions for new initiatives.

A mistake that I have seen my clients make is that they are introducing the idea of a pilot program too late into the process of evaluating the suppliers. They may have issued a RFP/RFI, gone through series of demos, and perhaps even asked for an initial quote before bringing to a vendor the request to run a pilot program.

From a negotiation standpoint, bringing the pilot up at the later stages of a vendor evaluation puts you at a disadvantage. A considerable amount of time has been invested in creating, distributing, and analyzing the RFP. Project team members have sat through vendor demos and everyone is anxious to move the project forward. 

Project team leads and their sourcing partners will review the Pilot Program proposals and typically will go through several rounds of discussions and negotiations. While you can leverage the idea the “if the pilot goes well, we’ll move towards a longer term engagement”, and try to keep your requirements as "Out of the Box" as possible, it’s difficult to force the account reps hand on reducing costs when they know they have gotten very close to a sale.

So what are some solutions? Below are a couple of ideas that we should use going forward on these types of engagements:

1. If issuing a RFP/RFI, the question should be asked if the vendor is able to provide a Pilot evaluation of their solution at no cost to your organization.
1. I recommend having this question in the RFP even if you don’t think you would run a pilot program. It’s just another evaluation point for your team to consider.
2. You should ask what the vendor would provide in a pilot program (length of program, amount of custom configurations they will provide, training, support, etc.)
3. If a vendor cannot run a pilot at no cost, you should know up front what they would charge for the short-term engagement. This will help with budgeting, and may influence the decision to even pursue a pilot program with a particular vendor.
4. It goes without saying that having a very strong set of requirements and desired outcomes will go a long way in working through the needs for a pilot, as well as how the team would design the pilot in conjunction with the vendor.
2. If not issuing a RFP, and going straight to demos or direct discussions with vendors, the same set of questions should be asked.

Pilot programs can be a part of a smart and effective strategy for assessing information technology partners. But they can be budget busters also. As with all strategic sourcing initiatives, you want to gain as much insight as early in the process to gain the knowledge needed to make well-informed decisions.

Source One has been providing Strategic Sourcing consulting services for 25 years. We have subject matter experts in all areas of information technology and are ready to help with all of your IT sourcing and procurement challenges.