Topics in this domain
- The Security Program: Charter, Scope and Objectives
- Security Team Structure, Roles and Staffing
- Working with Legal, HR, IT, Audit and the Business
- Program Implementation: From Plan to Operating Controls
- Information Asset Identification and Valuation
- Data Classification: Owners, Custodians and Users
- Data Lifecycle: Retention, Archiving and Secure Disposal
- Enterprise Architecture for Security Managers
- Information Security Architecture: Principles and SABSA
- Defence in Depth and Zero Trust
- Cloud Security and Shared Responsibility
- Control Frameworks: NIST SP 800-53, CIS Controls and ISO 27002
- Security Policy Development and Lifecycle
- Security Standards and Baseline Requirements
- Procedures, Guidelines and Acceptable Use
- Policy Enforcement, Exceptions and Review
- Designing Security Metrics: KPIs, KRIs and KCIs
- Collecting, Analysing and Reporting Metrics
- Security Maturity Models and Program Assessment
- Control Types and Categories
- Control Design and Selection Based on Risk and Cost
- Control Testing: Design vs Operating Effectiveness
- Identity and Access Management
- Network and Endpoint Security Controls
- Application Security and the Secure SDLC
- Configuration, Change and Patch Management
- Cryptography for Security Managers
- Security Testing: Scanning, Pen Testing and Tracking Remediation
- Security Awareness Program Design
- Role-Based Security Training
- Phishing Simulations and Social Engineering Defences
- Third-Party Risk Management and Due Diligence
- Vendor Contracts, SLAs and Right to Audit
- Fourth-Party and Supply Chain Risk
- Outsourcing and Managed Security Services
- Security Stakeholder Communication
- Executive Dashboards and Board Reporting
- Continuous Improvement of the Security Program
Control Testing: Design vs Operating Effectiveness
Two different questions
| Design effectiveness | Operating effectiveness | |
|---|---|---|
| Question | If performed as intended, would it meet its objective? | Was it actually performed consistently, as designed, over the period? |
| Tested by | Walkthrough and design review | Sampling occurrences over the period |
| Evidence | One good example can confirm the design | Evidence for each sampled occurrence |
| Example | The access review covers all privileged accounts | All four quarterly reviews were done, and removals actioned |
Test design first. If the design is wrong, testing whether it operated is wasted effort.
Testing methods
| Method | What it involves | Strength |
|---|---|---|
| Inquiry | Asking the owner how it works | Weakest; never enough alone |
| Inspection | Reviewing records, logs and sign-offs | Moderate |
| Observation | Watching it being performed | Moderate; only shows that moment |
| Re-performance | Doing it again independently and comparing results | Strongest |
Combine methods for important controls.
Who tests
| Tester | Strength | Limitation |
|---|---|---|
| Control owner (self-assessment) | Cheap, frequent | Not independent |
| Security / second line | Expertise, some independence | Still part of management |
| Internal audit | Independent | Periodic, limited coverage |
| Automated monitoring | Continuous and complete | Only for technical controls |
Example. The quarterly privileged access review is tested. Design: it covers all admin accounts and requires the system owner to approve each one, so it’s well designed. Operation: of four quarters, one was skipped, and in another, removals approved by owners weren’t actioned for two months. The control is rated as an operating deficiency, and the fix is a workflow tool with escalation.
Continuous control monitoring
- Automated checks of technical controls, daily or continuously: MFA coverage, backup success, configuration compliance.
- Exceptions raise alerts or tickets immediately.
- This frees people to test complex manual controls.
Results from all testing feed the risk register, metrics and the improvement plan.
How the exam asks about it
| If the question asks… | The answer is usually… |
|---|---|
| Would the control meet its objective if performed? | Design effectiveness |
| Was the control performed consistently over time? | Operating effectiveness |
| Strongest testing method | Re-performance |
| Weakest testing method | Inquiry |
| What to test first | Design |
In short
- Design effectiveness asks whether a control, if performed as intended, would meet its objective.
- Operating effectiveness asks whether it was actually performed consistently, as designed, over the period.
- Test design first; a badly designed control isn't worth testing for operation.
- Testing methods, from weakest to strongest: inquiry, inspection, observation, re-performance.
- Continuous control monitoring automates testing for technical controls and catches failures quickly.
- Control testing and evaluation is a named CISM topic; results feed risk ratings, metrics and improvement.
Exam traps
- Inquiry alone is never sufficient evidence.
- A control that operated perfectly but is badly designed still leaves the risk.
- One successful example proves design, not operation over a period.
- Control owners testing their own controls lack independence; use second- or third-line testing for key controls.
Slides
Click the slide and use the arrow keys. F toggles full screen.
Two different questions
Testing methods
1 / 9