So you're interested in moving forward with a cyber risk quantification project. Well, this article is just the thing for you. This blog post summarizes some of the key questions to ask when you are planning a cyber risk quantification project. It is applicable to both in-house CRQ projects, and also to those considering procuring a specialist CRQ tool/vendor.
We've put this article together based on our experiences delivering cyber risk quantification projects. It factors in some of the lessons learned and good practices that we have discovered so that you can be sure that when you are taking forward your CRQ project, you are going to have the greatest chance of success.
We're going to cover the following critical success factors:
- The audience and the scope of the quantification;
- Who are the people who need to be involved;
- The information needed for the quantification, and where to get it from;
- How do you ensure that you get repeatable and consistent results over time;
- Other useful things to think about if you're considering using external tools for cyber risk quantification.
Understand Why You Are Quantifying
Who will be making decisions with the CRQ results?
The first question to ask when planning a cyber risk quantification should be: who is the audience? Typically there are two audiences for CRQ: top management/the Board and technical experts. The target audience will drive the requirements for the project overall.
The Top Management audience will want to look at risks from a “big picture” perspective, and focus on those risks whose potential financial losses are large. Cyber risk assessment and reports for the board need to highlight the most significant risks, i.e. those that may have the greatest impact on the organization as a whole, and that can be compared alongside the other non-cyber risks a company has to manage.
On the other hand, technologists responsible for the building, maintaining and security of information systems within a company can benefit from risk reporting containing more technically detailed information. This could include granular breakdowns of vulnerabilities and threat intelligence. It may also need to support exceptions for local risk decisions.
What does this mean for your CRQ project?
- If your target audience is Top Management, then you will need to ensure that the outputs are described in terms of the business consequences, explainable in non-technical language, and aligned with other enterprise risk metrics.
- If your target audience is technologists, then you may benefit from more focus on the threats and causes of risks and a narrower scope of assessment.
If you don’t already have a clear view of the audience, then ask yourself: am I looking to steer strategic decision making, or to justify operational actions? Does my audience need to know more about the technical details of a risk or on the business consequences?
What decisions do I want CRQ to help with?
A cyber risk quantification will give you some numbers. To make these numbers useful you need to put these numbers into the context of a decision-making story. And in order to get the context you need to be clear what decisions you are looking to steer with CRQ.
I’ve listed some typical risk management decisions below which can be helped by CRQ:
- how does our current risk level across the company compare to our risk appetite and tolerance?
- how much should we invest in improving security controls? and which controls should we prioritise?
- to what extent will our information security strategy reduce our current risk?
- which actions outside the security team could reduce the financial impact of an incident?
Different CRQ approaches and tools provide different features for visualising and answering these questions.
What am I going to quantify?
Having a clear view of what to quantify will ensure that stakeholder questions can be answered and time is not wasted “boiling the ocean” and trying to collect data on irrelevant topics. This final scoping question is closely related to the target audience and use cases. It is also crucial to understand when looking at potential CRQ tools as different vendors have different strengths and weaknesses for different assessment scopes as set out in the table below. (see our Top Down vs Bottom Up article for more details).
Who is involved in a CRQ project?
It is important to understand that a successful Cyber Risk Quantification is a process, requiring human interaction, communication, and ongoing commitment. It is not simply a case of buying a tool and getting a number. Understanding the types of people who contribute to a quantification, the types of skills needed and the relations between them will help lead to a successful project.
The quantification project usually consists of a “Core Team”, responsible for leading and managing the quantification efforts (usually the information security team), and an “Extended Team”, who may hold the necessary information needed for the quantification.
An understated benefit of CRQ is that it often enables security teams to forge new relationships outside the IT department and get a better understanding of the business.
The Core Team
The Core Team is responsible to deliver the successful CRQ project. Typically, CRQ will be driven from the information security/cyber security team. The size of the team will vary depending on the scope of the project; a CRQ can be delivered by a single person, but we would recommend at least two to allow for redundancy and a spread of skills.
The Core Team should possess the following knowledge and skills:
- Risk assessment knowledge - good understanding of risk management in general, and the specific CRQ method being used in particular. Different vendors use different CRQ methodologies; if you are procuring an external solution you should ask the vendor how they support knowledge transfer to your team.
- Project management skills - conducting a CRQ requires bringing together people and data from across the company. It may also include procurement work and IT integration work. Having someone on the core team to coordinate and keep track of this work is very helpful, particularly for the first quantification project.
- Communication and stakeholder management skills - Organizational silos are a frequent challenge for CRQ projects, and having someone on the core team who can explain the CRQ project to other stakeholders across the company and ask them to share data that may not have been shared before is a strong indicator for success. Having a senior sponsor for the project also helps here (easier for a Top Management driven CRQ than an IT driven CRQ).
An external vendor can often provide each of these skills as part of the service delivery, or as top up consulting services. Using these consulting services can speed up delivery, but means that you may be dependent on the provider for ongoing changes and repeat quantifications. A balanced approach could be for the vendor to lead the first quantification while training internal members of the Core Team so that they can lead future quantifications.
The Extended Team
In order to put financial figures on cyber risk, the Core Team needs to obtain information from across the company. The Extended Team is the name given to those stakeholders who will need to be contacted for this information. The exact list of stakeholder will vary depending on the project scope, but usually includes the following:
- Finance: the finance team should be able to provide guidance as to the most profitable business areas, helping to target the scope, and also help with putting monetary figures on expenses that occur following an incident.
- Data privacy: a breach of personal data is usually one of the scenarios to quantify, so understanding the concerns of the Data Protection Officer, and getting their view on volumes and locations of personal data, is essential.
- Operations / business continuity: business continuity specialists may already have conducted analysis about risk scenarios, and have a view about the activities that will take place in response. This can speed up scoping and identifying the cost drivers to quantify.
- Threat intelligence / SOC: understanding the relevant threats to your organization, and historic trends relating to these, can be useful for Bottom Up CRQ in particular. Some vendors include historic threat and loss information within their service, so an in-house view is not always needed.
- Information security: it will be necessary to understand the contols that are in place to protect the scope of the assessment, and how effective these are. If this information is not already known to the Core Team, then expect to speak to other members of the information security function for this.
People Summary
The success of a CRQ project depends as much on the people involved as on the technology used. Understand that the Core Team is there to drive and coordinate the project, and the Extended Team is there to provide the necessary data. Both the Core Team and the Extended Team need to have the knowledge and skills to execute the project effectively.
External vendors can support the Core Team, or take on that role entirely, but the Extended Team will always come from within the company being assessed.
What information is needed for a successful CRQ?
The discussion of the Extended Team above gives an indication of the different types of information needed for a cyber risk quantification. The key observation is that much of this information will come from outside of the information security team, and as such the Core Team needs to be able to network within the company to break down information silos and negotiate access to the necessary information.
The types of data needed can be broadly grouped into three categories:
Financial information
This includes both overarching information at the company level needed to quantify financial impact on sales, profitability etc. as well as more specific information about the costs of certain services that the company will use in an incident, for example legal fees, forensic fees, cyber insurance limits and deductibles.
Business impact scenario information
This includes defining concretely what exactly is being quantified. It is important when selecting scenarios to avoid overlap and double counting. This is easier in a Top Down CRQ approach, which will usually define scenarios based on the overall business consequences. However, it needs particular attention in a Bottom Up CRQ approach so that aggregating different low level risk quantifications doesn’t double count.
While the list of threat scenarios can be open-ended, typically the consequences are more concrete. For example, many different threats can lead to the consequences of either loss of availability, breach of sensitive data, or financial theft/fraud. The focus on consequences makes things easier to explain to non-technical stakeholders who are more interested in the business impact than the technical cause, while a focus on threats may be more useful for a defender looking to understand what might cause an incident and how to mitigate it.
CRQ vendors can help shape business impact assessment by providing relevant questions and industry specific suggestions of relevant scenarios.
Information security information
The information security information required for a CRQ usually consists of threat information, and control information.
Some Bottom Up CRQ methodologies require the estimation of threat parameters, such as the expected frequency of each attack type, the skills of adversarial groups and whether these skills are sufficiently advanced to overcome the defensive controls. Estimating such parameters can be difficult, and leaves the results open to challenge if different stakeholders have different opinions.
Companies with sophisticated in-house security or SOC teams may have sufficient data to inform these threat estimates; most companies will need to look to publicly or commercially available data sets to help inform this. Squalify incorporates historic loss data within our service offering, so subscribers do not need to make these threat estimates.
The other category of information security information required for a quantification relates to the control environment. A Top Down CRQ approach will look across the organization as a whole, and typically require a maturity assessment against an industry standard controls frameworks, such as the NIST Cybersecurity Framework, or ISO27001. A Bottom Up CRQ approach will look more closely at the specific technical controls in place for the in-scope system.
Integrating a CRQ tool into other systems?
A question often asked is whether it is necessary to integrate a CRQ tool into other systems such as vulnerability management platforms, threat intelligence platforms or other operational security tools.
There can be benefits to making an integration:
- It can help to accelerate the collection of information security information in certain circumstances;
- It may be useful in particular if you are performing a Bottom Up CRQ to get technical telemetry data;
- It allows you to “re-use” data from other tools to inform risk management.
However, there are also downsides of integration and it should not be seen as a CRQ silver-bullet:
- It is unlikely to replace the hugely valuable business engagement part of the CRQ process;
- Including more technical data may shift the focus away from Top Management audiences and make it more difficult to explain the outcomes to non-technical stakeholders;
- It gives an external vendor access to your sensitive operational security systems; and
- Planning and building integrations can add scope and extend the timelines of a CRQ project.
Information Summary
The information needed for a successful CRQ will include financial information, business impact scenario information, and information security information. Some vendors may include datasets for part of the information security information, but the other categories will need to come from inside the company.
Integrating a CRQ tool into other security systems may help to accelerate information gathering for the information security information category, and this can be useful for Bottom Up technical CRQ assessments. However, integrations will not provide all the information needed for a CRQ, the information will likely shift the focus of a CRQ away from a Top Management audience and may add risk and delay to the project.
Making CRQ repeatable and consistent
Just like any risk assessment process, cyber risk quantification is not a one-off activity. It is important to plan for regular re-assessments, and it is crucial that your initial planning sets you up to be able to perform future CRQs repeatably and consistently. Here are some of the key considerations to ensure ongoing success.
When to quantify depends on the CRQ goals
The frequency at which quantifications should be reviewed and re-assessed will be driven by the scope. It is important to consider the quantification frequency as part of the overall information security governance approach.
- Annual quantifications can be used to inform company business plans, corporate risk disclosures and reporting, aligning business and cybersecurity strategies, and cyber insurance purchasing/renewal decisions.
- Quarterly quantifications may be driven by quarterly reporting cycles either to the Top Management/the Board or other reporting committees. Quarterly updates can also be used by a CISO to support their governance and risk management practices, for example requiring subsidiary entities in a global group to report on updates to their security maturity and tracking progress in risk reduction.
- More frequent quantifications (e.g. monthly) could be used if circumstances are rapidly changing within a company, however we find that typically the inputs needed for CRQ are usually stable over at least a quarter.
- One-off quantifications can be used for special projects, such as a merger and acquisition, to get a quantified view of risk for a specific scope.
Changes to quantification inputs and outputs need to be explainable
Over time the inputs for a risk quantification will change. This may result from organizational restructurings changing the scope of the quantification, operational or business changes altering the most profitable parts of the business or the business continuity scenarios, or changes to the information security threat and control environment.
The people involved in cyber risk quantification will also change over time, and this may lead to different interpretations of CRQ inputs, and ultimately inconsistent CRQ execution. Having a stable and enduring Core Team helps ensure consistency of CRQ results over time.
A CRQ process needs to be able to take these changes into account and be flexible in performing quantifications from one time to the next. This is where having a Core Team to own the process comes into its own. One benefit of having a stable Core Team is that future quantifications can be performed more quickly as the team are skilled and have the internal network to source the updates to the needed information. The Core Team can also stay in touch with the stakeholders in the Extended Team to monitor changes, and through this be able to explain how variations in the outputs have manifested from the relevant changes in the input.
For example, if the risk quantified has reduced by 5% from one year to the next, it is important to be able to explain whether this has come from improvements in the control inputs, changes in the organization reducing the scope being quantified, or improvements in the threat landscape.
When considering an external CRQ tool, it is important to consider how the features of this tool can help you in this story-telling; putting the changing numbers into the business context is where the true value of CRQ lies.
Consistency in the quantification model itself
The second key factor to ensuring repeatable results is the stability of the quantification model itself. It is beneficial to continually improve a quantification model, so that it takes into account for example:
- New methodologies identified that better quantified the cost drivers of incidents;
- Regulatory drivers introducing new use cases or risk management goals;
- New features introduced by CRQ vendors.
However, if the changes to the model are not understood and well-managed, then this can lead to outputs that are not easily compared from one quantification to the next.
For in-house CRQ models, change control procedures should be used so that changes to the model are understood and introduced in consultation with the Core Team.
For vendor provided CRQ tools, you should ask the vendor how the model is updated, how frequently updates are expected, and how the vendor ensures that quantifications can be consistently compared before and after model updates.
Repeatability summary
In order to ensure not just a one-off CRQ delivery, but long-term success, it is important to plan for ongoing quantification as a process. When identifying the initial scope consider how frequently re-assessments should be performed to answer the ongoing questions of the target audience. The differences in CRQ outputs from one quantification to the next will be closely scrutinised and the Core Team needs to be able to explain where these differences have come from, whether from changes to inputs or the model itself. A team that can tell a long term story with quantification results over time will have truly unlocked the value of CRQ.
Conclusion
Quantifying cyber risk is achievable, and with this guide you now have the tactics needed to successully implement CRQ within your organization.
Before starting CRQ, it is crucial to have a clear picture of the scope. This includes the target audience for the quantification results, and whether this is targeting Top Management for strategic decision making, or more technical stakeholders for operational decision making. It also includes the scope of what to quantify, which could start with a high level picture across the comany as a whole and then explore further into organizational units. A clear view of scope is crucial before starting looking for a CRQ vendor.
It is important to understand the people who will be involved in the CRQ process. We recommend having a Core Team to own and lead the process, and identifying the Extended Team, who will be the stakeholders providing the necessary data for quantification. A vendor can support, or even take on the role of the Core Team, but for longer term stability it is recommended that this team is kept in-house. The Extended Team will always be in-house.
Information security threat and control information is just one category of information needed for a successful CRQ. Some vendors can provide historical data to speed up collection of threat data. However, you will additionally need financial information and information about the business impact scenarios being quantified, which will need to come from in-house. Integrating a CRQ tool to other security tools can also accelerate security data collection, but introduces both security risks and delivery risks.
To get long term benefits from CRQ, you need to plan for repeatable and consistent quantifications. It is important to be able to explain changes in the quantification model itself, as well as the changes to the organization and wider environment that lead to changes in the inputs and consequently the outputs.
