Ask the Experts: Risk-Based Monitoring

Q: Since we implemented Risk-Based Monitoring we have been trying to monitor the process to determine how well it is working. We have developed metrics such as whether alerts from Centralized Monitoring are closed on time but are uncertain what to use as a target. How are other organizations approaching this? (Metric Insights, Jan. 2021) 

A: With a new or significantly revised process, you should always be asking the question “How will we know whether it is working?” You need to establish measurements the evaluate the process. We applaud your approach! The Centralized & Site Monitoring Process Metrics Work Group has been discussing this exact topic. The work group developed a high-level process map that includes the centralized monitoring and remote/onsite site monitoring processes. The work group identified a series of questions about the process (aka Key Performance Questions) they wished to answer and are working on defining metrics to answer the questions. This approach ensures that our metric sets are comprised of metrics that provide actionable data.

The measurement that you state in your question is a type of time metric we call an “on time” or “timeliness” metric. The work group defined an alternative time metric that is a “cycle time” metric – instead of measuring if the task is completed within the timeframe you establish as the target, it measures the actual time it takes to complete the task.

The work group explored the pros/cons of each approach and concluded that the industry doesn’t have enough experience with the new process to determine what the target should be. Using a cycle time metric provides the opportunity for the industry to gather data on the new process and, in time, decide whether it is feasible and useful to have a timeliness metric that measures against a target. Additionally, upon review of the results, we may determine that critical risks should be assessed faster than others. Finally, we don’t want to encourage the premature closures of risks (prior to proper investigation) in order to meet timeliness targets without basis in understanding the process. Whenever metrics are defined and used, it is important to think critically about the purpose of the metrics and the behaviors they might drive.

Ask the Experts: Answers to your Metric Questions

With more organizations focusing on metrics, we have received an increase of questions ranging from how to use metrics, why some metrics are better than others, which type of metrics is best to use, as well as questions about specific metrics. This column provides a forum for us to share these questions and answers with you. Submit your question ›

Linda Sullivan

Senior Advisor, Metrics & Performance Management

WCG Avoca

Keith Dorricott

Metrics Subject Matter Expert

WCG Avoca