Control effectiveness is the degree to which a control actually reduces the risk it was put in place to address. It splits into two questions that are routinely collapsed into one.
Design effectiveness asks whether the control, as specified, would reduce the risk if it worked as intended. A quarterly access review is well designed if it covers the right systems, is performed by someone with the knowledge to judge, and results in revocations.
Operating effectiveness asks whether it is actually being performed that way. The review may be happening on schedule and consist of a manager approving a list they did not read.
A control can pass a design assessment and fail entirely in operation. The reverse is rarer and less interesting.
Whether the control is reducing the risk at all.
A control can be well designed and faithfully operated and still be ineffective, because the risk it addresses arrives through a path the control does not cover. Testing confirms the control works. It does not confirm the control matters.
This is why control pass rates and risk indicators need to be read together. A high pass rate alongside a deteriorating KRI means the control set is aimed at the wrong thing.
Coverage of the population, not a sample, wherever the control permits it.
Failure rate over time rather than a point-in-time pass, since the pattern of failure is more informative than the count.
Time to detect a failure, which is often the more actionable number.
Evidence retained for each test, which is what turns an assertion into an assessment.
From the blog