Trying to apply a standardized set of evaluation criteria in a space experiencing such rapid evolution and maturation is challenging. And trying to come up with an apples-to-apples comparison often results in evaluating against only the “table stakes” criteria – often with the lowest-cost provider being selected. In my experience in the TPRM space, companies that conduct a hands-on evaluation are happier with their decisions and more successful in their programs.
Since that webinar, I’ve received a lot of questions about best practices for PoCs, so I thought I would share some of my answers here.
Q: How do I design an effective PoC?
A: A PoC needs to account for your specific use cases, have measurable criteria, and set appropriate expectations for stakeholder participation according to the following steps:
Document the three to five most important use cases that address the business needs or challenges you’re trying to solve with TPRM technology and what actions you want your stakeholders to be able to execute in the software. Provide that documentation to the vendor as a requirement for the PoC.
Make sure to include technical stakeholder use cases amongst these (i.e. create/modify a questionnaire, create/modify a workflow, create/modify reports & dashboards). This ensures that you’re comfortable not only with the usability of the interface, but also with your team’s ability to own and manage the software without having to be dependent on the vendor. Otherwise, you could end of with a solution that quickly becomes outdated as internal or external requirements change or incur unplanned and unpredictable costs for vendor services.
Have the vendor(s) come onsite and walk through how they’ve implemented your use cases within their system. This validates that everything is working as you requested before they hand the PoC over to you and your team. It also gives you an opportunity to get their guidance and feedback during the walkthrough so that you can evaluate their level of expertise and their ability to understand and respond to your needs.
Set the appropriate expectations with your stakeholders to ensure that they allocate appropriate time to complete their use cases and provide feedback. Their timely participation in the process and buy-in on the value of a TPRM solution are necessary to conduct a valid evaluation. To facilitate involvement, be sure to:
Communicate specific expectations to them well in advance, including how much of their time it will take and the beginning and ending dates.
Set up a cadence of reminders to all stakeholders who have not completed their use cases or provided their feedback.
Before the PoC begins, document the specific, measurable criteria that will be used to evaluate success. For example, you might specify that an acceptable solution must be able to:
Complete all use cases with 0 errors.
Complete all use cases within X hours.
Complete all use cases without the vendor’s assistance.
Collect stakeholder feedback not only on the measurable criteria, but also on their perception of the system. User adoption is critical to the success of any technology implementation, so this evaluation can be sometimes as effective a predictor of success as features and functions. Some things you might ask your stakeholders to rate on a scale of 1 to 10 include:
Ease-of-use (“How easy was the system to use?”)
Their perception of performance (“How ‘fast’ was the system’s performance?”)
Their willingness to adopt (“How willing would you be to use the system?”)
Q: How long should I expect PoC to take?
A: Generally, in my experience, a PoC shouldn’t take more than 2 weeks. I want to emphasize that each stakeholder won’t need to spend 100% of their time on the PoC, so by conducting it over a two-week period, they should have time to complete their use cases in addition to their “day jobs” without too much disruption. We’ve seen a few cases where a PoC has been completed in just 1-2 days because the company committed to getting their entire stakeholder team in a together in a room to focus entirely on the PoC.
Q: What are industry standards around paying for PoCs?
A: As long as the PoCs are kept within a couple of weeks, it’s not usual for a vendor to consider it a “cost of sale.” However, there are exceptions, such as:
If you ask your vendor(s) to travel long distances and stay onsite with your team throughout the PoC, it’s common practice to pay for airfare and accommodations.
When conducting a longer or more complex PoC, the vendor may require a SOW for time & materials in addition to travel expenses to secure their resources for extended periods. Unless you’re considering an intense project that lasts a month or more in duration, a vendor typically doesn’t (or shouldn’t) ask for this kind of investment.
In my experience, a PoC is a better indication of whether a vendor will successfully meet your expectations for a TPRM solution. Feel free to contact me if you have any additional questions about PoCs or would like to see some examples of successful PoCs the Aravo team have participated in.
