Managing Information in the Digital Economy: Issues & Solutions 227

Information Security Risk Assessment: The Qualitative Versus Quantitative Dilemma

Adrian Munteanu, Alexandru Ioan Cuza University, Iasi, Romania, adrian.munteanu@feaa.uaic.ro Abstract This paper presents main security risk assessment methodologies used in information technology. The author starts from [Sherer and Alter, 2004] and [Ma and Pearson, 2005] research, bringing realworld examples as to underline limitations of the two risk assessment models. After a critical review of standards that reveal lack of rigour, a practical comparison of the quantitative information security risk assessment models with the qualitative models shows that we can introduce two new factors which have an impact on risk assessment: time constraint and moral hazard of the analyst. Information technology managers know that in information systems long-term security is an ideal situation and that financial impact of poor information security policies, procedures and standards are in most cases very difficult to be calculated. These calculations rarely will be accurate and universal and ready for use by any security analyst. 1. Introduction Information security technology does not reduce information risk very effectively [Blakley et al, 2001] because information security is primarily a human problem. Whatever forms an information asset takes, a risk assessment must be undertaken to understand which are best security measures suited for protecting information security framework: confidentiality, integrity and availability. Firstly, we observe that there are many standards, books and documents named Information Security Best Practices targeted at IT managers which prove the importance of information security. In these documents we notice almost similar limitations and inconsistencies in description of risk assessment methodologies that [Sherer and Alter, 2004] follow in information systems risk literature: inconsistent or too general definitions of risk assessment; lack of rigour applicability of risk assessment models depends on the analyst knowledge and business context, lack of an exhaustive and up-to-date database of risk vulnerabilities and exposure applicable in quantitative models many standards with little or no substantial differences The above mentioned arguments will be supported by a short review of the main standards or best practices. The Guidelines for the Security of Information Systems and Networks [OECD, 2002] shows that the process of risk assessment should be sufficiently broad-based to encompass key internal and external factors, such as technology, physical and human factors, policies and third-party services with security implications. Analyzing that definition we observe that without risk key factors the assessment depends in fact on the analyst judgment and qualification. Components for analyzing and interpreting risk presented in Table 1 are described relatively analytically in [Swanson and Guttman, 1996] but in more general terms in [Swanson, 1998]: Table 1. Components for analyzing and interpreting risk Components of risk Major factors in risk [Swanson and management Guttman, 1996] [Swanson, 1998] Asset Valuation The value of the system or application Consequence Threats Assessment Threat Identification Vulnerabilities Safeguard Analysis Effectiveness of current or proposed safeguards Likelihood Assessment Standards mentioned do not offer any explanation on techniques, methods or tools that could be used effectively later in a safeguard evaluation. Questionnaires containing specific control objectives and techniques against which an unclassified system or group of interconnected systems can be tested and measured are provided in [Swanson, 2001]. These are examples and not mandatory techniques. Risk management process should not be treated primarily as a technical function carried out by the IT experts who operate and manage the information system, but as an essential management function of the organization [Stonebruner et al., 2002]. The risk assessment process includes identification and evaluation of risks and risk impacts, and recommendations of risk-reducing measures. Risk mitigation refers to prioritizing, implementing, and maintaining the appropriate risk-reducing measures recommended from risk assessment process. The methodology presented contains nine steps: System Characterization Threat Identification Vulnerability Identification Control Analysis Likelihood Determination Impact Analysis Risk Determination Control Recommendations Results Documentation Standard security risk metrics are defined in [Swanson et al., 2003] for the first time as percentage

Managing Information in the Digital Economy: Issues & Solutions 228 of systems that had formal risk assessments performed and documented. The OCTAVE (Operationally Critical Threat, Asset, and Vulnerability Evaluation) approach focuses on technology in relation with security practices. In risk analysis an organizational set of impact evaluation criteria are defined as to establish a common basis for determining the impact value: high, medium, or low, due to threats at critical assets. This is a version of qualitative assessment. BS 7799 Part 1 Code of Practice and Part 2 Specification for Information Security Management Systems is a standard that has not been mandatory in Europe. Part 1 of this standard was adopted as ISO 17799. The former BS 7799 Part 2 was adopted as ISO 27001 in 2005. Both standards focus on risk management but there is no methodology included for risk assessment that we can use. In fact BS 7799 Part 2 states that an appropriate risk assessment shall be undertaken and the risk assessment shall identify the threats to assets, vulnerabilities and impacts on the organization and shall determine the degree of risk. In other words an organization should adopt a risk assessment methodology including the elements mentioned above and does not include methodology. The standard that outlines the principles of risk assessment with some guidance and advice on the interpretation of these principles is ISO TR 13335 Guidelines for the Management of Information Security Part 3. Based on specification of the standard, the risk management process is presented in Figure 1. Figure 1. General Security Risk Assessment (Source: ASIS INTERNATIONAL GUIDELINES COMMISSION) ISO has released the ISO Guide 73:2002 - Risk management Vocabulary Guidelines for use in standards. Based on this guide, the definition of the terms used in ISO standards are: Risk - the probability of an event that has negative consequences. Risk analysis - systematic use of information to identify sources and to estimate the risk. Risk assessment - is the overall process of risk identification, risk analysis and risk evaluation. Risk evaluation - process of comparing the estimated risk against given risk criteria to determine the significance of risk. Risk management - typically includes risk assessment, risk treatment, risk acceptance and risk communication (exchange or sharing of information about risk between the decision-maker and other stakeholders). Risk treatment - treatment process of selection and implementation of measures to modify risk. All these documents are very good instruments in supporting management decision-making. Obviously, security analysts need detailed tools because information security risk management is an iterative process. 2. Qualitative Approach Limitations Good data is a prerequisite to qualitative risk analysis, and the lack of good data may be the main reason qualitative analysis of information security risk is not usually performed [Blakley,2001]. Qualitative assessment use risk assessment matrix and questionnaires [Peltier, 2001]. In the case of risk matrix, risks are ranked as low, medium or high. In the questionnaires people use a risk scale for risk ranking. In this case, the qualitative assessment undergoes a transformation and becomes a qualitative-quantitative one. Estimating the likelihood of threats quantifiable as financial loss is difficult because it is based first of all on judgment and professional standing of the analyst. Consequently, the sensitivity and the value of the assets that could be impacted are relative to that judgment: the cost of loss of customer confidence or market share are very difficult to measure in the lack of any kind of standardization or information about the market [Bruce, 2003]. And if the analyst has only a technical background such an evaluation can not be done in realistic terms. We can talk here about moral hazard of the analyst. What competencies were used for such an assessment? Based on what competencies he made the assessment? If the questions used are not comprehensive, the answers will follow the same pattern. As we have much information/data within companies with a monetary value, and its estimation is quite difficult security risk assessment must be data centric process, not only technologically or technically

All text sho

In the lack of an unified language to describe the concepts and terms used in security assessment,

Managing Information in the Digital Economy: Issues & Solutions 229 oriented. Before beginning the assessment, the analyst must understand the business process and its functions; the legal framework and business context; the characteristics of the technology used in business; management philosophy and psychological aspects; business constraints; interdependences and interactions between information system components. Valuation of an asset (physical or logical) can differ depending on who is asking to give a value of that asset: an end-user, an accountant, a security analyst or a manager. For example, if we analyze information availability we have three points of view of what availability means at different levels: organizational, users and network presented in Table 2. Table 2. Three points of view on information availability Organizational Users access Users authentication Users authorization Users policy and operational procedures Logical and technical controls Users Systems performance and response time Computer network Data distribution across the network and vulnerabilities points Calculating the risks is a subjective estimation, in terms of: low, medium or high. In this case, we simply use a matrix risk-value and a risk impact ranking, but we can estimate exactly what value risks have. Most qualitative risk analysis methodologies make use of these elements and the assessment depends on the experience and judgment of the professional who made the analysis or identify quality elements that have an impact on information security. In fact, we talk about a what-if analysis. The analysts use qualitative variables but the result must be finally quantitative to serve the scope of a management decision. I propose a scenario to argue that hypothesis: an organization has a database server that can be accessed by the employees via an application and potentially attacked by a hacker. From the three layer architecture (database, web and application server) I discuss only the database server. In [Peltier, 2001] we find a good collection of qualitative risk assessment methods. Based on these, I take into account four elements: Asset value: the database server Threats: a vulnerability in database management system that give users database administrator rights Vulnerability: a database exploit that was released on a warez web sites Controls: in this case, security team can react when the exploit occur, and must patch the database management system with the last patch or disconnect the server from the network if the patch is not available. The assessment goal is to determine if the existing risk exposure is addressed by security controls that are in place. Given the above, when we analyze deeply this scenario there are some aspects that analysts must address: What is the generally accepted technique used in estimating correct value of information/transactions managed by the database server? Even the database is backed up, what is the monetary value on their information taking into account information confidentiality, integrity and availability? The historical cost of creating the database files can be influenced by the future security controls? Is there a specific technological risk? Is there an inherent risk of the application that uses the database server and the database management system? Some particular transaction on the database can create specific risks? Because these findings are of significance for the assessment, the analyst can not use only the qualitative approach described in most standards. To establish the value of the database server the analyst can use their accounting value: purchase price depreciation or the total cost of ownership. But for the rest of the elements he needs more information to substantiate a conclusion.

It is true that qualitative methods are easy to understand and apply but we can not make a cost benefits analysis because the probability of loss is not based upon mathematical certainty. For an analyst it is very difficult to collect historical data about the security events and every factor that can affect the probability of a security incident. The ISO 17799 /BS 7799 Part 1 is a policy-based standard that defines the risk assessment as the assessment of threats to, impacts on and vulnerabilities of information and information processing facilities and the likelihood of their occurrence. Information security risk can be analyzed taking into account four factors: Business assets values Threats against business assets Controls or counter measures to protect those assets Vulnerabilities that exploit the threats Because only accounting assets have a monetary value the cost of these physical assets should be at least the replacement cost. Also we can use the total cost of ownership of the asset, but in this case the analyst must decide what formula to use because there is not a general one accepted.

Managing Information in the Digital Economy: Issues & Solutions 230 Moreover, if this scenario happens the database administrator is reactive and makes the changes on the database server at the moment when he knows the vulnerability. The patches applied to the database server after the release of the exploit do not address the problems earlier described and an exact business impact analysis can not be made 3. Quantitative Approach Limitations Most security analysts agree that it is always better to have some quantitative information than none at all. Qualitative risk assessment is a scenario-based approach while quantitative analysis assigns monetary values to the components identified in the risk assessment phase. Securing the organizations information system means that the analyst identifies the assets, the threats that could have an effect on these assets and the vulnerabilities associated with the identified assets. But identification of the information system asset value is a qualitative process which must include quantitative elements: the financial value of the asset (ex: hardware cost). the cost to build the asset (ex: software cost) the cost to protect the asset (ex: support cost) the value of the asset to the competition. the cost to recover the asset (ex: replacement cost). The business value of the data contained on the database server from the scenario described in Section 2 is difficult to express in monetary terms. If we assume that the database server stores financial information, the value of the data may be based on two factors: the data contribution to the financial goals of the company and the value of the data to an external individual or organization. As a consequence, the indirect value of the database server is the most difficult assessment. In [Dillard, 2004] is presented a five step process to determine the value of an asset and some security metrics: Assign a monetary value to each asset class. Input the asset value for each risk. Produce the single loss expectancy value (SLE). Determine the Annual Rate of Occurrence (ARO). Determine the Annual Loss Expectancy (ALE). Single loss expectancy (SLE) represents the expected impact of a specific threat event and can be computed by multiplying the exposure factor of a given threat by the financial value of the asset (AV). The exposure factor (EF) is the percentage of asset loss caused by identified threat and can be calculated by multiplying the threat frequency level (TL) with the impact factor (IF). The threat frequency level (TL) is calculating by multiplying the threat probability (TP) by the risk factor (RF), where the risk factor is the criticality factor (CF) of the attack divided by the effort (E) required performing the exploit. EF = (((TP (C / E)) (VF AP)) / 100) Calculating the exposure factor following this formula does not take into account the time variable. This formula should be adjusted by estimating other three elements: average time period for threat identification, average time period for releasing technical procedures to reduce or accept threat and average time period necessary till the system becomes operational and the threat eliminated. We will name the sum of these three variables exposure time. Exposure factor will be bigger when exposure time is longer. Because estimating exposure time is based on security historical incidents, its value will be highly subjective. The annualized rate of occurrence (ARO) is the probability of a threat occurring during a one-year time frame. The annual loss expectancy (ALE) is the single loss expectancy (SLE) multiplied by the annualized rate of occurrence (ARO). ALE = SLE * ARO It is not easy to apply these formulas and to realize a cost-benefit analysis by taking the ALE and subtracting the initial cost of the countermeasure and the annual recurring cost of the countermeasure. The main problem when the analyst applies this formula is the numerical expression of the variables included. A rare threat is different from a threat that will never appear. Lacking of exhaustive threat probability database, the analyst puts in this formula a value based on a qualitative assessment. The impact factor and threat frequency level are variables that depend on statistical estimation of the threat. The Canadian Guide to Risk Assessment and Safeguard Selection for Information Technology Systems (MG3, 1996) state that an asset's value is based on: its replacement cost; its intrinsic value (for example, goodwill); and/or the consequences, impact or injury resulting from asset compromise. The output of the asset sensitivity analysis is an inventory of assets with their sensitivity (that is, their value) with respect to confidentiality, integrity, availability and replacement value. In many cases, information processed by the system will be the most significant asset. Moreover, the guide recommends the use of software tools to efficiently manage data. Intangible assets that are protected by laws (trademarks, patents, copyright) are easy to be assigned a value while know-how, goodwill, technical processes, customer list, the marketing budget, productivity value of the asset are more difficult to be identified in terms of earnings and profit they generate. For a good estimation, the security analyst must collaborate with the financial team because no profession has a monopoly on

Managing Information in the Digital Economy: Issues & Solutions 231 logical thinking and analytical reasoning. The analysis of intangible assets may be considered a multidisciplinary activity. No one set of professional qualifications or academic training grants an individual a monopoly license to practice intangible asset valuation [Reilly, 1999]. Because these kinds of assessments are more complex, specialized software packages were developed. This software uses a combination of qualitative and quantitative assessments because it is difficult to assign a value to an asset that is characterized by a quality and conduct automated risk analysis or risk assessment and vulnerability assessments of information systems. 4. Conclusion and future research Measuring security risk is a difficult task that is almost impossible for the entire information system. In the information security literature we can observe only few detailed quantitative methods (Jones and Ashenden, 2005) that describe the security metrics produced by some companies or institutions. Most books or standards are management instruments that describe the general framework of risk management. Concrete and complete security metrics are not presented in literature and a magic formula can not be applied. This situation arises from the fact that organizations do not recognize all security breaches/incidents and are aware about pure technical incidents. It is almost impossible for a security analyst with only technical background to quantify security risk for intangible assets. He can perform a quantitative or qualitative evaluation using dedicated software to improve the security of the information systems, but not a complete risk assessment for the whole information system. Qualitative assessment based on questionnaires use in fact statistical quantitative methods to obtain results. Statistical estimation represents the basis for quantitative models. We conclude that in both of the approaches the moral hazard of the analyst has influence on the results because human nature is subjective. He must use a sliding window approach according to business and information systems features, balancing from qualitative to quantitative assessment In the future I want to extend the research on two directions: first of all toward the analysis of the security risk assessment knowledge obtained after a recognized security certification. In the second toward another formulas used in quantitative assessment and their limitations. 8. References [1] Alter, S., Sherer S., (2004), A General, but readily adaptable model of information system risk, Communication of the Association For Information Systems, (14:7), 2005 [2] Ma, Q., Pearson, J., M., ISO 17799: Best Practices in Information Security Management, , Communication of the Association For Information Systems, (15:32), 2005 [3] Blakley, B., McDermontt, E., Geer, D., Information security is information risk management in the Proceedings of the 2001 workshop on New security paradigms, ACM Digital Library, 2001, p. 97 [4] Dillard, K., Pfost J., The Security Risk Management Guide, Microsoft Press, 2004 [5]Jones A., Ashenden D., Risk Management for Computer Security: Protecting Your Network & Information Assets, Butterworth-Heinemann, 2005 [6] Peltier T. R., Information Security Analysis, Auerbach, 2001, pp. 21-50 [7] Reilly F. R., Schweihs P. R., Valuing Intangible Assets, McGraw-Hill, 1999 [8] ISO Guide 73:2002, Risk management Vocabulary Guidelines for use in standards, Geneva, 2002 [9] ASIS INTERNATIONAL GUIDELINES COMMISSION, General Security Risk Assessment April 11, 2005 at Guideline, (2003), Retrieved http://www.asisonline.org/guidelines/guidelinesgsra. pdf [10] Canadian Guide to Risk Assessment and Safeguard Selection for Information Technology Systems, 1996, Retrieved February 12, 2005, at http://www.csecst.gc.ca/en/documents/publications/gov_pubs/itsg/m g3.pdf, [11] OCTAVE - Operationally Critical Threat, Asset, and Vulnerability Evaluation, Retrieved June 3, 2005 at http://www.cert.org/octave/approach_intro.pdf, [12] OECD Guidelines for the Security of Information Systems and Networks, p. 11-12,
Retrieved June 21, 2005 at

http://www.oecd.org/dataoecd/16/22/15582260.pdf, [13] Stoneburner G., Goguen A., Feringa A., NIST 800-30 - Risk Management Guide for Information Technology Systems, 2002, Retrieved October 2, 2005 at http://csrc.nist.gov/publications/nistpubs/80030/sp800-30.pdf , [14] Swanson M., NIST SP 800-18 - Guide for Developing Security Plans for Information Technology Systems, 1998, Retrieved October 1, 2005 at http://csrc.nist.gov/publications/nistpubs/80018/Planguide.PDF, [15] Swanson M., Bartol N., Sabato J., Hash J., Graffo L., NIST 800-55 - Security Metrics Guide for

Managing Information in the Digital Economy: Issues & Solutions 232 Information Technology Systems, 2003, Retrieved
November 13, 2005 at

http://csrc.nist.gov/publications/nistpubs/80030/sp800-30.pdf, [16] Swanson M., Guttman B., NIST 800-14 Generally Accepted Principles and Practices for Securing IT Systems, 1996, Retrieved August 8, 2005 at http://csrc.nist.gov/publications/nistpubs/80014/800-14.pdf,