Iso

2014

June
4
A lightweight, flexible evaluation

framework to measure the ISO 27002
information security controls
Karin Huijben
Master Computing Science
Radboud University, Nijmegen, The Netherlands
S o f t w a r e I m p r o v e m e n t G r o u p ( S I G )

Summary
The security of organizational processes is often weak [2] and could be improved
by using the ISO 27002 standard. In this master thesis project, an ISO 27002
compliant, lightweight evaluation framework was created that shows in a short
time period how secure organizational processes are in comparison with other
organizations. This evaluation could be used in diverse organization sizes and
types, but it was specially developed for organizations that produce software and
organizations that depend heavily on software. The reason to focus on these
organizations is the increasing importance of software in both the society and in
the economy in combination with the recent increase in cyber security incidents.

During the literature study, seven standards were compared with the ISO 27002
to determine which standard should become the basis of the evaluation
framework. There were only three with the possibility to be ISO 27002
compliant besides the ISO 27002 standard itself. The ISO 27002 standard was
the smallest of the standards as it has the least amount of pages, so the ISO
standard was chosen. In addition to the basic requirements for the framework, it
was checked if there were other comparable evaluation frameworks. None of
those evaluation frameworks met all requirements that was tried to meet.

The evaluation framework contains two parts:
• A questionnaire
• The rating system

The questionnaire was created in three phases:
• Trial questionnaire
• Full questionnaire
• Improved questionnaire

The trial questionnaire is a try-‐out version that covers only a part of the ISO
27002 standard. A validation was done inside SIG and the results were
promising for further investigation, but the time to fill in the questionnaire was
already long. In the second phase, the full questionnaire was designed that
covers all ISO 27002 control aspects. At the end of the phase, the validations
inside SIG with both the employees that have implemented the security controls
and the selected consultants were carried out. Both validations were positive,
but there were some remarks to improve the questionnaire (e.g. using an
introduction for the questionnaire). In the third phase, the full questionnaire was
improved by using the feedback of the second phase. The results were promising
that the questionnaire could be a SIG service in the future.

The rating system converts the answers of the questionnaire to the end results:
a star rating from one to five stars and suggested actions to improve the security.
Each question has five defined security levels. The star rating is based on a risk
profile where each higher security level has to meet a specific compliance
percentage of the total number of questions.

2

The evaluation framework can be improved with a lifecycle approach depending
on the results of previous answers. Additionally it has the ability to be updated
with new features and technologies when new standards are released.

3

Acknowledgements
I would like to express great appreciation to Haiyun Xu, Joost Visser and Erik Poll
for their valuable suggestions and constructive criticism during the research
process.

Furthermore I would like to thank the employees of SIG who helped me with the
validation session. The quality of the evaluation framework is improved a lot
with the help of your useful feedback.

I would also like to thank the two external organizations for helping me with the
validation of the evaluation framework.

4

Table of contents

Summary .............................................................................................................................. 2
Acknowledgements ........................................................................................................... 4
1 Introduction ................................................................................................................. 7
1.1 Problem statement ......................................................................................................... 7
1.2 Research definition ......................................................................................................... 8
1.3 Context ................................................................................................................................ 9
1.4 Relevance ........................................................................................................................ 10
1.5 Previous work ............................................................................................................... 10
1.6 Overview ......................................................................................................................... 11
2 Background ............................................................................................................... 12
2.1 Standards ........................................................................................................................ 12
2.1.1 ISO 27000 series .................................................................................................................... 13
2.1.2 NIST Special Publication 800 ........................................................................................... 17
2.1.3 The 2011 Standard of Good Practice for Information Security ........................ 21
2.1.4 10 steps to cyber security .................................................................................................. 22
2.1.5 COBIT 5 ...................................................................................................................................... 23
2.1.6 BSIMM ........................................................................................................................................ 24
2.1.7 PAS 555:2013 ......................................................................................................................... 26
2.1.8 SAS70 and its successors ................................................................................................... 27
2.2 Comparison of standards ........................................................................................... 29
2.3 Evaluation frameworks for ISO 27002 .................................................................. 32
2.3.1 Approach of Karabacak ...................................................................................................... 32
2.3.2 Approach of Wright .............................................................................................................. 33
2.3.3 Approach of Bandopadhyay ............................................................................................. 33
2.3.4 Approach of Praxiom ........................................................................................................... 33
2.4 Comparison of evaluation frameworks ................................................................ 33
3 Construction of the questionnaire ..................................................................... 36
3.1 Phase 1: the trial questionnaire .............................................................................. 37
3.1.1 Development ........................................................................................................................... 37
3.1.2 Validation inside SIG ............................................................................................................ 39
3.2 Phase 2: the full questionnaire ................................................................................ 41
3.2.1 Development ........................................................................................................................... 42
3.2.2 Validation inside SIG ............................................................................................................ 42
3.2.3 Validation with consultants .............................................................................................. 44
3.3 Phase 3: the improved questionnaire ................................................................... 45
3.3.1 Processing feedback of validations ............................................................................... 45
3.3.2 Validation with external organizations ....................................................................... 46
4 Construction of the rating system ...................................................................... 48
4.1 The SIG star rating ........................................................................................................ 48
4.1.1 The SIG approach .................................................................................................................. 48
4.1.2 The risk approach ................................................................................................................. 51
4.2 The actions for improving the security ................................................................. 52
5 Discussion .................................................................................................................. 53
5.1 Evaluation framework improvement .................................................................... 53
5.1.1 Continuous improvement .................................................................................................. 53
5.1.2 Event-‐based improvement ................................................................................................ 54
5.2 Potential risks in the evaluation framework ...................................................... 55
5

6 Conclusion .................................................................................................................. 56
References ........................................................................................................................ 58
Appendix ........................................................................................................................... 61
A: Differences of ISO 27002:2005 and ISO 27002:2013 ............................................. 61
B: Detailed transition table .................................................................................................. 65

6

1 Introduction
At the start of this chapter, a problem statement is given. This problem statement
describes the global security problem and the problem to solve during this
research process. Thereupon, the research is introduced including the research
questions. Furthermore, the context and the relevance of the master thesis are
discussed. In addition, the information about the earlier prototype is given,
which forms the starting point for this research. At the end, an overview of the
master thesis is presented.

1.1 Problem statement

Nowadays many organizations depend heavily on IT systems according to the
information security breaches survey 2008 that was carried out for the U.K.
Government [1]. This dependence creates an opportunity for large negative
effects on the business when a security attack occurs. In case there is no security
attack then it will be no problem, but it is shown that a large number of
organizations had many security breaches last year [2]. Some examples of
possible security breaches are the GGZ Eindhoven case1, where the employees
sent medical data to an IT company, and the famous Edward Snowden case.

Security flaws occur often in the system implementation, but it occurs also in
how the organization handles sensitive information. Code reviews and using
special security tools like Fortify2 could help to find some of the implementation
flaws. It should be noted that only the presence of security flaws could be
proven, but not the absence of them. Searching for security flaws in handling
information is much harder than the implementation of them. According the
survey that is done for the British Department for Business Innovation & Skills,
the security of organizational processes is sometimes weak or generally weak [2]
(more details see Table 1). This should be improved to a more secure situation.

Table 1 ‘10 steps to cyber security' survey results [2]
The ten steps Large organizations Small businesses

Information risk Some good, some weak Some good, some weak
management
User education and Some good, some weak Generally weak
awareness
Home and mobile working Some good, some weak Generally weak
Incident management Some good, some weak Generally weak
Managing user privileges Some good, some weak Some good, some weak
Removable media controls Some good, some weak Generally weak
Monitoring Some good, some weak Generally weak
Secure configuration Some good, some weak Some good, some weak
Malware protection Generally good Some good, some weak

1
http://www.nu.nl/binnenland/3154721/ggz-eindhoven-stuurt-medische-gegevens-ict-bedrijf.html
2
http://www8.hp.com/nl/nl/software-solutions/software.html?compURI=1338812#.Uz5sNa2SzBc
7

Network security Generally weak Generally weak

A possible way to improve the security around the organizational processes is to
use the ISO 27001:2013 and the associated ISO 27002:2013 standard. These two
standards describe an Information Security Management System (ISMS) where
an organization can manage the processes seriously and helps to improve
securing the actions and environment around the organizational processes.

1.2 Research definition

Many organizations encounter problems to implement the ISO 27002 standard
due to the large amount of information. This information does not tell exact what
an organization has to do and it is made for organizations in general. This means
that the organizations have to determine which processes, actions and controls
are applicable to them. After implementing the required ISMS and the associated
security controls, many organizations try to get an ISO 27001 certificate. This
certification process requires high effort from an organization. Organizations
could question themselves: how could the organizations know if they have
implemented ISO 27002 controls ‘well enough’ in a short time with low costs?

Currently, the Software Improvement Group (SIG) uses a simple checklist as an
assistant tool during the Security Risk Assessment (SecRA) for the organisational
process security. In a SecRA, a system is investigated to find possible
vulnerabilities and security risks [42]. Hereby, the investigation is carried out
from several perspectives such as the architecture and source code perspective.
To obtain a more comprehensive and focused check on organisational process
security, a new evaluation framework is formulated. This master thesis’
framework is based on the total 114 controls defined in the ISO 27002:2013
standard and it will be used for Information Security Process Assessments
(ISPA) of SIG. The master thesis’ framework targets mainly on software
development companies or companies rely heavily on software systems. The
new evaluation framework contains two parts, which are shown in Figure 1. In
this master thesis, the questionnaire and the rating system are created.

•  Input: Information of organization

Questionnaire •  Output: Answers of questionnaire
session
•  Input: Answers of questionnaire

•  Output: SIG star rating, actions, ...
Rating system

Figure 1 Evaluation framework
8

The following requirements are defined for the new framework:
• Lightweight: The evaluation framework has to be understandable for
anyone and it have to be easily checkable in a short time.
• Flexible: The evaluation framework can be adjusted to different
situations. The different situations are for example quick, detailed or
special-‐focus scanning in both small as large organizations.
• Measurable: The evaluation framework contains the ISO 27002 controls
with some checkpoints that can measure the ISO control in an objective
way.
• ISO 27002 compliant: The evaluation framework is based on ISO 27002,
so it could be used to check the large amount of implemented security
aspects in the company.
• Market conformity: The result of the evaluation framework shows if the
organization has a better or worse security control implementation
compared with other organizations.

In this research a special focus is getting a basic evaluation framework to
measure the ISO controls. The research question is formulated based on this
measuring focus.

How can the security of the organizational processes be measured in a flexible and
lightweight way?

For this main research question, several sub questions are brought forward:

• How can each ISO 27002:2013 control specifically be measured?
• Which ISO 27002:2013 controls have priority?
• How can the ISO 27002:2013 controls be grouped?
• Which security risks are not taken into consideration in the evaluation
framework?
• How can the number of checkpoints be reduced to an acceptable amount?

1.3 Context
The Software Improvement Group (SIG)3 provides objective advice about IT
systems for clients based on their static analysis tool. This static analysis tool
performs an analysis on several code metrics such as code complexity. SIG
divides possible services into 4 groups, which are shown in Figure 2. The current
service that uses the static analysis tool is based on the ISO 25010 standard [45].
The ISO 25010 standard describes system and software quality models. As can
been seen in the figure, this service is part of ‘modeling & measurement’ in the
development of a product. One aspect mentioned in the ISO 25010 standard is
security. Currently SIG provides Security Risk Assessment (SecRA) services, and
SIG aims at extending their services with what they call an Information Security
Process Assessment (ISPA). This ISPA consists of an analysis on the security in
organizational processes based on ISO 27002.

3
http://www.sig.eu/en/
9

2 Existing approaches to IT security
Rapid increases of attacks and reports of security incidents leave organizations worried about how
secure their systems are. Security control is hard to attain both in terms of processes and products.
Figure 1. shows an overview of the variety of existing approaches to security .

process
BSIMM
NEN 7510
OpenSAMM
ISO 27001
MS-SDL
process management SAS 70
models systems
development operations
Common modeling & testing & intrusion

Criteria measurement monitoring detection
CWE/SANS ethical
ISO 25010
Top-25 hacking
OWASP CVSS
penetration
ASVS
testing
product

Figure 2 Synthesis
Figure 1o-f Synthesis
cyber security
of cyber assessment and risk mand
security assessment itigation approaches
risk mitigation (made by SIG [42])
approaches
1.4 Relevance
The dimensions
This research used in tthe
aims o pcircle in this
rovide image are thetsoftware
a framework life cycle
o evaluate (from development
organizational to op-
processes
based on ISO 27002:2013. This framework will be part of an Information cycle.
erations) and a distinction between the software product and the processes governing its life
Security Process Assessment, which is an extra service to SIG’s clients. The
results of the framework will indicate to organizations how market conform they 2
are in handling their organizational processes secure.

1.5 Previous work

During a previous research project, a prototype of an evaluation framework was
created [6]. This small research project had the same goals as this research,
namely: flexible, lightweight, measurable, market conformance and ISO 27002
compliant. The previous version was based on ISO 27002:2005[13]. Those (old)
standards had 11 chapters and the previous research itself covered only chapter
10 about communications and operations security. Chapter 10 contains 32 ISO
27002 controls, while the full standard has 133 ISO controls. Each aspects of the
requirements were taken apart and it was checked what is required to achieve
the requirement.

The idea to make the approach lightweight was to create checkpoints that could
be easily marked as ‘implemented’ or ‘done’. In addition, the checkpoints had to
be written down unambiguous. To meet this requirement, the SMART-‐technique
(Specific, Measurable, Attainable, Relevant and Time-‐bound) was used.

There were several ideas to make the approach flexible. One was using
exceptions if the possible actions are different for the size of the organizations. In
addition, a baseline was created for the previous version of the evaluation
framework based on several papers[4][5][7]. Finally, there were 25 baselines
out of 133 ISO controls. Beside the baselines, the approach is made more flexible
by applying the 27 groups for the ISO controls based on the master thesis of
Altena[4].

10

The third goal was to be measurable. In case an ISO control has to be measurable,
it has to be unambiguous, complete and objective. The completeness is
‘guaranteed’, because the checkpoints are fully based on the ISO 27002
implementation guidelines. Unambiguousness and objectiveness were
implemented by the above mentioned SMART-‐technique. Only this technique
explodes the number of checkpoints in the prototype. Another option is to write
it more general, but this is less implicit about what has to be measured.

Beside the prototype itself, a rating system was created. On the question level
(ISO subchapter) of the prototype, the maturity model concept is applied. This
means that the rating is based on the completeness of all checkpoints of a
specific level. In addition to that, each low level (ISO subchapter) has its own
weight between 1.0 and 2.0 based on the number of baselines it holds. For the
full prototype an average is calculated based on the rating of the low levels (ISO
subchapters) and their weight.

The result of this previous research was a prototype. At the beginning there were
492 checkpoints after applying all the requirements of SIG on the 32 ISO
controls. After re-‐thinking about the amount of checkpoints, a possibility for
reduction was found. One option was combining some checkpoints and skip
some checkpoints that are overlapping with some other checkpoints in other ISO
chapters. Finally, the prototype was reduced to 212 checkpoints.

After finishing the prototype, a pilot test was performed at SIG. This pilot test
showed that the trial version still has some problems. Some checkpoints could be
answered in different ways, because the answer could be different on how
critical a system is. For example a critical system will be reviewed more often
than a non-‐critical system. Furthermore another issue was detected, namely
there are some exceptions that are not mentioned in ISO 27002. An example is
that an organization can reduce the risk of viruses by using virus scanner.
However, an iOS system might have a lower risk than Windows, therefore, an
organization might accept this risk.

1.6 Overview
In Chapter 2 the background of the research is given including standards ISO
27000 series, NIST special publication 800, the 2011 Standard of Good Practices
for Information Security, 10 steps to cyber security, COBIT 5 framework, BSIMM,
PAS 555, SAS70 and its successors. Chapter 2 also introduces several research
papers of similar research topics. In both cases (the standards and similar
research) some comparisons are made. Next Chapter 3 describes what is
involved in the construction of the questionnaire and the validation of the
questions. Then Chapter 4 describes the construction of the rating system, which
translates the answers of the questionnaire to a star rating and actions for
improvement. Chapter 5 discusses several aspects connected to the evaluation
framework, namely how to keep the evaluation framework up-‐to-‐date and the
potential risks of the evaluation framework. At the end of this master thesis
there is a conclusion with answers to the research questions and possible further
work.
11

2 Background
Each institute has its own names for a document, which describes norms,
requirements and so on. Some examples are standard, model and framework. In
this master thesis the term standard is used for all the documents with norms
and requirements.

Section 2.1 describes eight standards on information security. After this
description, a comparison of the eight standards is done in Section 2.2 in order to
decide which standard could be used as a basis for the new evaluation
framework. In the third section of this chapter, several evaluation frameworks
for the ISO 27001/ISO 27002 are described. The fourth section contains the
comparison between the evaluation frameworks. The difference between the
standards and evaluation frameworks is that the standards are the security
guidelines and the evaluation frameworks measure those security guidelines.
Especially the two comparison sections are important parts in this chapter
(Section 2.2 and 2.4).
2.1 Standards
There already exist many standards for securing the organizational processes
and measuring the effectiveness of them. One example is the ISO standard (ISO
27002:2013), which the new evaluation framework has to be compliant with.
Further examples are Building Security In Maturity Model (BSIMM) and Control
Objectives for Information and related Technology (COBIT).

In general the choice for standards is based on how acceptable they are in the
information security society, together with some other factors specific to the
standards that will be explained below. The following standards are discussed in
the upcoming sections:
• ISO 27000 series
The reason to choose the ISO 27000 standard series is that these
standards are generally accepted for information security management,
which makes them a good starting point of the new evaluation
framework.
• NIST Special Publication 800
This set of standards is chosen, because it is based on several
international standards and best practices including ISO 27002. This
means that NIST Special Publication contains a detailed and (almost)
complete list of the security controls, which are important.
• The 2011 Standard of Good Practice for Information Security
This best practices document refers to the ISO standard. That is why this
standard is chosen.
• 10 steps to cyber security
The reason to check this guide is that it is a well-‐defined and concise
guide, which is generally known. This guide does not only contain the
options for managing the risks, but also explains what the risks are. The
explanation of the possible risks for the key areas creates awareness
under the personnel that has to implement possible security controls.
12

• COBIT 5 framework
This chosen framework is globally accepted and it also specifies the
aspects for information security.
• BSIMM
The reason to check this model is that it is specially made for
organizations that produce software, which are a specific target for the
new evaluation framework.
• PAS 555
This standard is chosen, because it focuses on the outcomes of the
security control implementation. This focus results in a technology
independent standard.
• SAS70 and its successors (SSAE16 and ISAE3402)
The reason to check this standard is that it is a standard where an
organization could get a certificate. This certificate also covers
information security.

In the next sections, the above-‐mentioned standards are briefly introduced. A
comparison of those standards is done in Section 2.2. This comparison is to
verify which standard is the best option to use as the basis for the new
evaluation framework. The result of the comparison was that the ISO 27002
standard was the best option for the new evaluation framework. Other standards
were also very interesting, but some could not guarantee ISO 27002 compliance.
Other standards that could guarantee ISO compliance were so detailed, that
there was a large amount unnecessary security controls for the organizational
process security.

2.1.1 ISO 27000 series

The most important standard series for the master thesis research: the ISO
27000 series, are named formally ISO/IEC 27000[9]. These standards consist of
several standards for information security, which are developed by the
International Organization for Standardization (ISO)4 and the International
Electrotechnical Commission (IEC)5. The information security of these standards
is based on an Information Security Management System (ISMS) that needs to be
implemented in an organization. The type of organization is not limited to a
specific field, because the standard could be applied to all types of organizations
such as software development organizations and clothes factories. There are six
major standards of the ISO 27000 series that are widely used and they form
together the basis of the series.

1. ISO 27001 [10][12]
This standard describes the specification for an ISMS.
2. ISO 27002 [11][13]
This standard describes a reference for selecting controls for an ISMS
implementation.
3. ISO 27003 [14]

4
http://www.27000.org/
5
http://www.iec.ch/
13

This standard describes guidance for implementation of an ISMS.
4. ISO 27004 [15]
This standard describes guidance on the development and use of measures
and measurement for the effectiveness validation of the security controls.
5. ISO 27005 [16]
This standard describes information security risk management.
6. ISO 27006 [17]
This standard describes guidance for accreditation of organizations offering
ISMS certification.

During this research, there is a focus on the ISO 27002:2013 standards as can be
seen in the research questions. Other standards ISO 27001:2013 and ISO
27004:2009 are also discussed and the remaining standards are not applied for
this master thesis.

2.1.1.1 ISO 27001:2013
This ISO 27001 standard describes the specification for an ISMS. The most
important aspect mentioned is that there is a defined lifecycle in an ISMS. In a
previous version of ISO 27001 (the 2005 version) a specific lifecycle type was
mentioned, namely the Plan-‐Do-‐Check-‐Act (PDCA). A PDCA model structures
how the organizational processes could be improved.
Act Plan
Check Do

Figure 3 Plan-‐Do-‐Check-‐Act model
This PDCA model contains four phases:

• Plan
In this phase the organization has to establish the objectives and
processes, which are required to achieve to desired result.
• Do
In this phase the organization has to implement the plan of the previous
phase.
• Check
In this phase the organization has to validate if the implemented plans of
previous phases has the expected result.
• Act
In this phase the organization has to take some actions if the expected and
actual results are not the same.

14

To connect this PDCA model to the master thesis research is that the focus is on
the check-‐phase and partially on the act-‐phase. The new evaluation framework
verifies how far the security controls of ISO 27002 are implemented. The result
is a rating that shows how market conform an organization has implemented
their security controls. The act-‐phase is involved in giving the found results and
which actions have to be done in order to improve the current security
implementation.

2.1.1.2 ISO 27002:2013

The ISO 27002 standard describes security controls to give organizations the
best practices recommendations for an ISMS.

First, a global overview of the ISO 27002:2013 standard is provided. The
standard consists of three ISO layers:
• ISO layer 1: ISO chapter
There are 14 ISO chapters, which are mentioned in Table 2. Each ISO
chapter consists out of one or multiple ISO subchapters (ISO layer 2).
• ISO layer 2: ISO subchapter
There are 35 ISO subchapters. An example can be found in Table 3. Each
ISO subchapter consists out of one or multiple ISO controls (ISO layer 3).
• ISO layer 3: ISO control
There are 114 ISO controls. An ISO control is described as ISO control
name (including id) and a full description. An example can be found in
Table 3.

Table 2 Overview ISO 27002:2013 chapters [11]
ISO chapters
5. Information security policies
6. Organization of information security
7. Human resource security
8. Asset management
9. Access control
10. Cryptography
11. Physical and environmental security
12. Operations security
13. Communications security
14. System acquisition, development and maintenance
15. Supplier relationships
16. Information security incident management
17. Information security aspects of business continuity management
18. Compliance

Table 3 Example -‐ ISO control 5.1.1 of ISO 27002:2013 [11]
ISO chapter 5. Information security policies

ISO subchapter 5.1 Management direction for information security
ISO control 5.1.1 Policies for information security
15

ISO control A set of policies for information security shall be defined,
description approved by management, published and communicated
to employees and relevant external parties.

2.1.1.3 ISO/IEC 27004:2009

Besides the ISO 27001 and ISO 27002 standards, the ISO 27004:2009 standard is
possibly also important for the master thesis research. This ISO 27004 standard
describes guidance on the development and use of measures and measurement
for the assessment. These measurements could be used to verify the
effectiveness of an implemented ISMS and the ISO controls. This measurement
input and output are in all 4 steps of the PDCA lifecycle, which is mentioned in
ISO 27001.

More specially, the standard describes a method on how to create the
measurements for each control, which is implemented in the organization. In
addition, it describes how an organization could measure security and report the
results to the management. In total, the organization follows a four phase
procedure for this standard.

1. Development of measurements
The organization develops measurements for each implemented ISO
control or set of ISO controls.
2. Do the measurements in the organization
The organization measures the implementation of ISO controls in the
organizations with the developed measurement of phase 1.
3. Do the data analysis and report the result
The organization has gathered some data about the effectiveness in phase
2, which have to be analyzed. The analysis shows if the implemented
control is effective or not. The result of the data analysis will be reported
in a document.
4. Information security measurement program evaluation and improvement
The organization has the results of the measurements to evaluate. If the
results are positive then no action is needed. Otherwise the organization
verifies why the ISO control has not achieved the desired result and takes
actions to improve the results in the future.

The development of the master thesis’ framework in this research could be
regarded as a high-‐level version of first phase in the above-‐mentioned
procedure. The framework measures which ISO controls are implemented and in
which degree they are implemented. Phase 2 of ISO 27004 is meant to measure
the effectiveness of the implemented ISO controls.

2.1.1.4 Differences between de versions 2005/2013 of ISO 27001 and ISO 27002
Now it is clear how the ISO standards are structured and what is in the standard,
it is time to go deeper into the details. In October 2013, both ISO 27001 and ISO
27002 were updated. The new ISO versions made large changes in the standards,
but what are the actual differences? This section describes the main differences
between the 2005 version and the 2013 version.
16

The ISO 27001:2013 differs with ISO 27001:2005 on one major point.
• Most of the general information for the ISO 27000 series were defined in
each ISO standard itself like the terms and definitions. In the 2013
version, this common information is moved to the ISO 27000 standard
itself.

Then there is also a change in the security controls in Annex A of the ISO 27001.
These security controls are more detailed described in ISO 27002, so the changes
for ISO 27002 are described to show the differences in the ISO controls.

There are many differences that could be noticed if somebody compares the
different versions. Many of the small differences are caused by one large change.
The reason for this is that the new ISO 27002 version is technically and
structurally revised in comparison with the ‘old’ 2005 version. This resulted in
the following differences:

• The ISO 27002:2013 has 114 controls instead of 133 ISO controls of the
previous version. This new number of ISO controls is caused by the
following actions:
o 22 ISO controls are added to the new standard
o 38 ISO controls are deleted from the old standard
o Three times two ISO controls are combined to one control, so the
six old ISO controls are combined to three ISO controls.
• The ISO controls are structured in 14 chapters instead of the 11 chapters
in the 2005 version.
• Controls are sometimes renamed to a more obvious title.
• The ISO 27002:2013 has more third-‐party services controls than the 2005
version, because many organizations start using third-‐party services
instead of implementing everything by themselves.
• The 2013 implementation guide is changed in comparison with the old
version. In some cases it has more detailed information. In some other
cases, the information is more limited than in the older version.

The comparison between the old and new versions of ISO 27k is also done by
other organizations [18][19]. The found differences between the old and new
versions of ISO 27k are not exactly the same in the three comparisons. The
reason for these differences is the interpretation of the reader. One person could
interpret a large change in an ISO control as alteration, but another one could see
it as removing an old ISO control and creating a new ISO control.

2.1.2 NIST Special Publication 800

Next to the ISO standard, there are standards created by the National Institute of
Standards and Technology (NIST). NIST is a science institution and is part of the
United States federal government. The NIST is primarily involved in creating
standards and guidelines. These standards and guidelines are commonly used,
especially in the United States.

17

An example of the NIST standards series is the special publication 800 series,
which is specially designed for security related topics. Some of the security
related topics are about risk management processes in organizations.

Tier 1
Organization
Tier 2
Mission / Business processes
Tier 3
Information systems

Figure 4 Three-‐tiered risk management approaches
The first tier is an overall level, which means that all actions and decisions in the
first tier influences the possible actions and approaches in the other tiers. This
tier has a view from an organizational perspective and provides context for all
risk management activities.

After the first tier of the risk management approach, there is the second tier
which checks the risk management processes from a mission and/or business
perspective. This tier watches what mission and business processes are
required, prioritizes the mission and business processes and so on. The second
tier influences the third tier.

The lowest level is the third tier that checks from the information system
perspective. In this level there is a Risk Management Framework (RMF), which
has 6 steps:

• Step 1: Categorize Information Systems
More information about this can be found in FIPS 199[22] and SP 800-‐
60[23][24]
• Step 2: Select Security Controls
More information about this can be found in FIPS 200[25] and SP 800-‐
53[26]
• Step 3: Implement Security Controls
More information about this can be found in SP 800-‐70[27]
• Step 4: Assess Security Controls
More information about this can be found in SP 800-‐53A[28]
• Step 5: Authorize Information Systems
• Step 6: Monitor Security Controls

18

The third tier is especially important for the master thesis research, because this
tier contains the RMF that describes a lifecycle to create an effective information
security program. In the NIST Special Publication 800-‐53 revision 4[26] a global
overview of the security control catalogue of all NIST security controls is given.
The catalogue consists of two layers:

• NIST layer 1: Family
The catalogue contains 18 families, which can be found in Table 4.
• NIST layer 2: Security control
The catalogue contains 240 security controls. Each security control has
some basic information: the family, the name, control (‘basic control’) and
supplemental guidance. An example of a security control is given in Table
5. In addition to the basic information, the following information is given:
o Control Enhancements
Each security control can have several additional requirements. First
an organization has to implement the ‘basic control’ and then, if more
security is required, an organization can add some implementations of
control enhancements.
o Reference
The reference describes where more information could be found,
o Priority
A priority has four possible options:
§ P0: Undefined
§ P1: First to implement
§ P2: Next to implement
§ P3: Last to implement
o Baseline selection
Whether a specific security control is a baseline depends on how high
the risk of the information system is.

Table 4 Overview NIST families [26]
ID Family
AC Access Control
AT Awareness and Training
AU Audit and Accountability
CA Security Assessment and Authorization
CM Configuration Management
CP Contingency Planning
IA Identification and Authentication
IR Incident Response
MA Maintenance
MP Media Protection
PE Physical and Environmental Protection
PL Planning
PS Personnel Security
RA Risk Assessment
SA System and Services Acquisition
SC System and Communications Protection
19

SI System and Information Integrity
PM Program Management

Table 5 Example -‐ NIST security control AC-‐21 [26]
Family Access Control

Security AC-‐21 Information sharing
control name
Control The organization:
a. Facilitates information sharing by enabling authorized
users to determine whether access authorizations
assigned to the sharing partner match the access
restrictions on the information for [Assignment:
organization-defined information sharing circumstances
where user discretion is required]; and
b. Employs [Assignment: organization-defined automated
mechanisms or manual processes] to assist users in making
information sharing/collaboration decisions.
Supplemental This control applies to information that may be restricted in
Guidance some manner (e.g., privileged medical information, contract-‐
sensitive information, proprietary information, personally
identifiable information, classified information related to special
access programs or compartments) based on some formal or
administrative determination. Depending on the particular
information-‐sharing circumstances, sharing partners may be
defined at the individual, group, or organizational level.
Information may be defined by content, type, security category,
or special access program/compartment.
Related control: AC-‐3.
Control (1) Information sharing | Automated decision support
Enhancements The information system enforces information-‐sharing
decisions by authorized users based on access
authorizations of sharing partners and access restrictions
on information to be shared.
(2) Information sharing | Information search and retrieval
The information system implements information search
and retrieval services that enforce [Assignment:
organization-‐defined information sharing restrictions].
References None
Priority P2
Baseline LOW: not selected
Allocation MOD: AC-‐21
HIGH: AC-‐21

20

2.1.3 The 2011 Standard of Good Practice for Information Security
The 2011 Standard of Good Practice for Information Security is created by
Chaplin and Creasey and published by the Information Security Forum (ISF)[32].
The standard provides insights, best practice standards and tools, which address
each aspect of the model to aid organizations in enhancing their information
security environment.

At this moment the 2013 version has already been published, but this 2013
version was not available during this research. The older 2011 version is used in
this thesis.

The standard is part of a much larger aspect, namely an Information Risk
Management Business Cycle of ISF that consists of 4 phases.
• Define
In the define phase an organization can use The Standard of Good Practice
for Information Systems.
• Implement
In the implementation phase an organization can use the ISF Information
Risk Analysis Methodology (IRAM)6
• Evaluate
In the evaluation phase an organization can use the ISF benchmark7
• Enhance
In this phase an organization can use the results of the evaluation phase
to know the weaknesses in security and can use the standard of good
practices to select new security controls.

During the first and the last phase of the cycle, an organization could use the
standard of good practices. The standard consists of many possible security
controls in several groups. These controls are based on the ISO 27001 standard,
the ISO 27002 standard and COBIT 4. A short overview of the control framework
that contains three layers is given.
• ISF layer 1: Area
This standard contains 20 areas and the list of areas could be found in
Table 6.
• ISF layer 2: Topics
Each area has two or more topics. The topics are described as an ID,
name, principle and objective. An example can be found in Table 7.
• ISF layer 3: Control
Each topic has multiple controls. The control is described as an ID and
some description. An example can be found in Table 7.

Table 6 Overview ISF areas [32]
ID Area
CF1 Security Policy and Organization
CF2 Human Resource Security

6
https://www.securityforum.org/tools/isf-risk-manager/
7
https://www.securityforum.org/tools/isf-benchmark-service/
21

CF3 Asset Management
CF4 Business Applications
CF5 Customer Access
CF6 Access Management
CF7 System Management
CF8 Technical Security Infrastructure
CF9 Network Management
CF10 Threat and Vulnerability Management
CF11 Incident Management
CF12 Local Environments
CF13 Desktop Applications
CF14 Mobile Computing
CF15 Electronic Communications
CF16 External Supplier Management
CF17 System Development Management
CF18 Systems Development Lifecycle
CF19 Physical and Environmental Security
CF20 Business Continuity

Table 7 Example ISF control [32]
Area CF1 Security Policy and Organization

Topic CF1.1 Information Security Policy

Principle
A comprehensive, documented information security policy should
be produced and communicated to all individuals with access to
the organization’s information and systems.

Objective
To document the governing body’s direction on and commitment to
information security, and communicate it to all relevant
individuals.
Control CF 1.1.1
Control There should be a documented information security policy, ratified
description at board level that applies across the organization. There should be
an individual (or a group of individuals) responsible for
maintaining the policy.

2.1.4 10 steps to cyber security

Furthermore a 10 steps to cyber security guide is created by Government
Communications Headquarters (GCHQ), The Department for Business
Innovation & Skills (BIS) and the Centre for the Protection of National
Infrastructure (CPNI)[33]. According to the creators of the guide it is possible to
stop 80% of the cyber attacks with basic information risk management. For the
last 20%, an organization has to implement more advanced steps in ten key
areas (see Table 8). The 10 steps to cyber security document describe the ten
22

key areas in a summary, the possible risks in the area and how the risk can be
managed.

Table 8 Overview -‐ Key areas [33]
Key area
Home & Mobile Working
User Education & Awareness
Incident Management
Information Risk Management Regime
Managing User Privileges
Removable Media Controls
Monitoring
Secure Configuration
Malware Protection
Network Security

Examples of risks for the area ‘Home & Mobile Working’ that are being
overlooked are loss of credentials. In the area there are also six controls to
prevent the possible risks, including education of users and maintaining their
awareness.

2.1.5 COBIT 5
The Information Systems Audit and Control Association (ISACA) and the IT
Governance Institute (ITGI) created the Control Objectives for Information and
related Technology (COBIT). COBIT is a framework for governance and
management of enterprise IT. The framework is based on globally accepted
principles, practices, analytical tools and models to help increase the trust in
information systems. The current version is COBIT 5[35] and contains several
parts including COBIT 5 for information security[36] (see Figure 5).

Figure 5 Overview COBIT 5 framework [35]
23

For this research, the COBIT 5 for information security document is useful,
because it provides more detailed and practical guidance for information
security. The first part of the document describes the COBIT 5 framework and
the enablers for using COBIT 5 for information security. Furthermore a mapping
is given of COBIT 5 for information security and other information security
standards (including ISO 27002). The detailed guidance is given in seven parts:
• The principles, policies and frameworks enabler
• The processes enabler
• The organizational structures enabler
• The culture, ethics and behavior enabler
• The information enabler
• The services, infrastructure and applications enabler
• The people, skills and competencies enabler
2.1.5.1 Aligning COBIT 4.1, ITIL V3 and ISO/IEC 27002 for Business Benefit
Next to using only the COBIT framework, it is possible to use the combination of
the COBIT framework with other standards including ITIL8 and ISO/IEC 27002.
The IT Governance Institute describes the combination of these three standards
in the book ‘Aligning COBIT 4.1, ITIL V3 and ISO/IEC 27002 for Business
Benefit’[40]. This book includes why best practices are so important to use in an
organization. Additionally the IT Governance Institute gives an overview on what
to expect of each of the three standards (COBIT, ITIL and ISO 27002). In the book
they states that:

“COBIT and ISO/IEC 27002 helping to define what should be done and ITIL
providing the how for service management aspects”

Furthermore in the book, the best ways to implement the best practices are
mentioned: tailoring, prioritizing, planning, avoiding pitfalls and aligning the
best practices of the three standards. In the appendix of the book, the mapping
between COBIT, ITIL and ISO 27002 can be found.

2.1.6 BSIMM
Another standard that is generally known is Build Security In Maturity Model
(BSIMM)[7]. BSIMM focuses on organizational security, but is more on a
software development side than ISO 27001 (see Figure 2). In addition, BSIMM is
different in comparison with other models, namely BSIMM describes what
organizations actually do. This means that BSIMM is a descriptive model. Other
models define what an organization has to do to get the organizational processes
secure, so those models are prescriptive. The current version is BSIMM-‐V and it
was released in October 2013.

A global view of the BSIMM model is given to understand how the model is built
up in some activities. This model has four BSIMM layers:

8
http://www.itil-officialsite.com/
24

• BSIMM layer 1: BSIMM domain
The model describes four domains, which can be found in the top row in
Table 9. Each domain contains three BSIMM practices (BSIMM layer 2).
• BSIMM layer 2: BSIMM practices
The model describes twelve practices, which can be found in Table 9.
Each practices divided BSIMM activities (BSIMM layer 4) into three
BSIMM maturity levels (BSIMM layer 3).
• BSIMM layer 3: BSIMM maturity levels
The model describes three different maturity levels, namely 1, 2 and 3. An
example can be found in Table 10.
• BSIMM layer 4: BSIMM activities
The model describes 112 activities, all are described as an ID with a name
and a full description. An example can be found in Table 10.

Table 9 Software Security Framework (SFF) of BSIMM [7]
Governance Intelligence SSDL Touchpoints Deployment

Strategy and Attack Models Architecture Penetration
Metrics Analysis Testing
Compliance and Security Features Code review Software
Policy and Design Environment
Training Standards and Security Testing Configuration
Requirements Management and
Vulnerability
Management

Table 10 Example -‐ BSIMM activity SM1.1 of BSIMM-‐V [7]
BSIMM domain Governance

BSIMM practice Strategy and Metrics
BSIMM maturity level SM Level 1: Attain a common understanding of
direction and strategy
BSIMM maturity level Managers must ensure that everyone associated with
full description creating, deploying, operating, and maintaining
software understands the written organizational
software security objectives. Leaders must also ensure
that the organization as a whole understands the
strategy for achieving these objectives. A common
strategic understanding is essential for effective and
efficient program execution.
BSIMM activity SM1.1 Publish process (roles, responsibilities, plan),
evolve as necessary
BSIMM activity full The process for addressing software security is
description broadcast to all participants so that everyone knows
the plan. Goals, roles, responsibilities and activities are
explicitly defined. Most organizations pick and choose
from a published methodology such as the Microsoft
SDL or the Cigital Touchpoints and then tailor the
25

methodology to their needs. An SSDL process evolves
as the organization matures and as the security
landscape changes. In many cases, the methodology is
published only internally and is controlled by the SSG.
The SSDL does not need to be publically promoted
outside of the firm to count.

For each BSIMM activity it is measured how many times it occurs in the
organizations from a data set. The data set compromises 161 distinct
measurements collected from 67 different firms. The most common activity is
marked for each of the twelve practices. The results of a company for each
domain will be represented in a spider diagram, which an example is displayed
in Figure 6.
Strategy &
Metrics
Conqig. Mgmt. & 3 Compliance &
Vuln. Mgmt 2.5 Policy
2
Software Env. 1.5 Training
1
0.5 Company A
Pen. Testing 0 Attack Models
Company B
Sec. Features &

Sec. Testing
Design
Standards &
Code review
Req'ts
Arch. Analysis

Figure 6 Example results of BSIMM activities for company A and B
2.1.7 PAS 555:2013

Besides ISO and NIST, the British Standards Institution (BSI) developed the
standard PAS 555:2013[31]. Many standards and guidelines defined good
practices as to how cyber security could be achieved, but the focus of PAS 555 is
different. PAS 555 takes into account that there are rapid changes in technology,
so they see that there are many ways to achieve the goals. That is why PAS 555
describes a fundamental set of outcomes that the controls, systems and
processes aim to achieve. They describe fourteen main-‐outcomes and there are
in total fifteen sub-‐outcomes.

26

Table 11 Outcomes of security implementation [31]
Outcomes
Management structure
Commitment to cyber security culture
Security context
Business architecture strategy
Capability development strategy
Supplier and partner strategy
Technology strategy
Business resilience
Compliance with legislation and other standards
Risk assessment
• Asset management
• Threat assessment
• Vulnerability assessment
Protection and mitigation
• People security
• Physical security
• Technical security
• Resilience preparedness
Detection and response
• External awareness
• Internal monitoring
• Protective monitoring
• Cyber security incident management
Recovery
• Investigation
• Data integrity reassurance
• Business-‐as-‐usual restoration
• Legal process
Compliance analysis and continual improvement

The outcomes are connected to several controls and requirements that are
mentioned in other standards including the ISO 27002:2005 standard. In the
overview in Annex A of PASS 555:2013 is shown that almost all outcomes are
related to one or multiple controls of ISO 27002:2005.

2.1.8 SAS70 and its successors

The American Institute of Certified Public Accountants (AICPA) developed the
Statement on Auditing Standards No. 70 (SAS70)9. SAS70 is an international
norm for internal control of service organizations. This norm is used to get a
SAS70-‐statement, which describes if the controls of the service organization are
described correctly, the effectiveness of the design and which controls are used.

9
More information on SAS70: http://sas70.com/
27

The SAS70 distinguishes two types of audits:
• Type I:
o The audit is done at a point in time.
o Only covers design effectiveness of internal controls
• Type II:
o The audit is done in a period of time, where the time period is in
many cases at least six months.
o Covers the type I design and it also covers operational
effectiveness of internal controls.

In contrast to many other standards, SAS70 has no required control objectives
themselves. The control objectives that they used are from other frameworks as
for example the COBIT framework (more details about COBIT can be found in
Section 2.1.5).

SAS70 is replaced by two different norms, being:
• The International Standard on Assurance Engagements (ISAE) 3402.
ISAE 3402 is developed by the International Auditing and Assurance
Standards Board (IAASB), which is part of the International Federation of
Accountants (IFAC).
• Statement on Standards for Attestation Engagements (SSAE) 16.
SSAE 16 is developed by the Auditing Standards Board of the American
Institute of Certified Public Accountants (AICPA).

According to an article from Deloitte [20], these two standards are fairly similar.
The differences can be viewed in Table 12. Deloitte suggests that when choosing
between SSAE16 and ISAE 3402 that SSAE16 is for organizations located in U.S.
and the ISAE 3402 for the rest of the world. This is also a reason for not using
SSAE16, because SIG is an international organization with their headquarter in
the Netherlands and many clients are European organizations.

Table 12 Differences SSAE16 and ISAE 3402 according [20]
U.S. standard: SSAE16 International standard: ISAE

3402
Use of Report required to specifically Report required to state that it
report state that it is restricted to the is only intended for user
intended users entities and their auditors, but
may also include restrictive use
language
Intentional Service auditor considers Silent on this requirement
acts impact of intentional acts on
the report
Subsequent Service auditor to consider Does not require auditor to
events Type 2 subsequent events after consider events after the report
the report date date
Reporting Does not enable a service Enables a service auditor to
auditor to conclude that a conclude that a deviation
deviation identified when identified when performing
28

performing tests of controls tests of controls involving
involving sampling is not sampling is not representative
representative of the of the population from which
population from which the the sample was drawn
sample was drawn
2.2 Comparison of standards

In the previous sections, an introduction is given for eight different standards.
The next question is which one suits the best as a basis for the new evaluation
framework. For selecting the best standard, a comparison is done with eight
aspects:
• High-‐level
This aspect describes whether the standard contains requirements for
implementation of security controls in technical detail or in high-‐level.
High-‐level means fewer adjustments to new technologies, but describes
less clear how an organization could implement these aspects. Preferably
a combination is wanted: a high-‐level document with a few technical
aspects.
• Information security controls
This aspect describes if the standard contains security controls that are
specific related to organizational processes. For the basis of the
evaluation framework this aspect has to be met.
• ISO 27002 compliance
This aspect describes if the standard covers all the ISO 27002 security
controls so that it is compliant with the ISO standard. For the basis of the
new evaluation framework this aspect has to be met.
• Certifiable/auditable
This aspect describes if an organization could be formally audited and /or
certified compliant to the standard.
• Internationally used
This aspect describes whether the standard is used all over the world or
that it is specific to a country.
• Number of pages
This aspect describes the number of pages of the standard to give a rough
indication of the complexity and level of detail. In the most cases, the
number of pages is based on the full standard. Preferably the standard is
very small, but all required information has to be in the document.
• Price
This aspect describes the prices for buying the standard to use it in the
organization.

The comparison of the standards on the above-‐mentioned aspects is shown in
Table 13.

29

Table 13 Comparison standards

ISO 27002 compliance

Certifiable/auditable
Information security
International used
Number of pages*

High-‐level
controls
Price*
ISO 27002 standard X √ √ √ √ 90 ± €150
NIST Special Publication 800 X √ √ X √ 460 Free
ISF: The 2011 Standard of Good X √ √ X √ 292 ± €3570
Practice for Information Security**
The 10 steps to cyber security X √ X X √ 22 Free
COBIT 5 framework X √ √ X √ 220 ± €130
BSIMM X √ X X √ 67 Free
PAS 555 √ X X X √ 32 ± €100
SAS70 and its successors √ X X √ √/X N/A N/A
* The number of pages and prices are based on the full standard, except the NIST SP 800, where only
NIST SP 800-‐53 is used. For the COBIT 5 framework only the framework document itself and the
document COBIT 5 for information security is used (non-‐member price). Other exception is the
SAS70 standard where the documents are only required for the auditor, so an organization has only
to pay the audit. The prices of an audit are unknown, because it depends on several factors.
** The information is based on the 2011 version. At this moment you could only buy the 2013
version, so the prices is based on the 2013 version. The aspects and number of pages could not
checked in the 2013 version, because this version is not available to us.

As can be seen in Table 13, only PAS 555 and SAS70 are high-‐level documents
and all others are more technical and detailed. A high-‐level document is
desirable, because the number of changes in the future is limited. Besides that, it
is also possible to create measurements for those high-‐level documents.
However, a large disadvantage of these high-‐level documents is that the cause of
a bad result is unknown. To know the cause, a further analysis has to be
performed. When using a more detailed document, the cause is in most cases
already known. The above-‐mentioned disadvantage is so important that there is
chosen for the more detailed document. A high-‐level document is PAS 555, which
describes the goals of implemented security controls and it does not matter how
these are achieved. This means that an organization could use different
technologies, methods and attributes to achieve the goals. The other high-‐level
standard is the SAS70 standard, which has a different approach. Another high-‐
level standard is SAS70 and its successors, which describes more how to capture
the current situation of an organization in a document. This means that there are
no goals described and also no security controls to achieve secure organizational
processes. This means that SAS70 does not improve the security of the
organizational processes, but only gives predefined requirements how an
auditor could describe how organizations secure their processes. All other six
standards describe how to secure processes and systems in more technical
detail. These standards contain lists of security controls. The standards BSIMM
30

and the 10 steps to cyber security only describe the security control list, but the
other four describe even more details, such as a life cycle approach to improve
the information security management system.

Only the standards that have technical details (aspect 1) contain a list of security
controls (six out of eight). The fact that security controls are only described in
the standards with technical details is no coincidence, because the security
controls are mostly based on methodologies and technologies. As there is only
searched for standards that are connected or are similar to the ISO 27002
standard that contains information security controls, all standards with security
controls contain at least information security controls. In some cases the amount
of information security controls are extensive in comparison with the ISO
standard (NIST and ISF standards). Other standards have about the similar size
to the ISO standard (COBIT5 and BSIMM) and the 10 steps to cyber security have
less information security controls.

The fourth aspect for the standards is ISO 27002 compliance. It means that if an
organization implements the security controls of that specific standard and uses
the related method, than it would be possible to get an ISO certificate. This could
only be possible when the standard contains at least a security control list. This
list has to cover (almost) all aspects of the ISO standard. Three of the standards
(NIST, ISF and COBIT 5) refer to the ISO 27002 standard and some additional
standards. This means that these three standards will cover the aspects of the
ISO standard, but the authors added some extra elements from other standards.
The 10 steps of cyber security is a more compact standard than the ISO standard,
so it misses some (important) aspects of ISO including the lifecycle and some
topics. Another not ISO compliant standard is BSIMM that also misses the
lifecycle of the information security management system. In addition, BSIMM
miss some topics of the ISO standard. PAS 555 and SAS70 are both not ISO
compliant, because it has no lifecycle approach and has no security controls.

The fifth aspect is the possibility of the auditing and certification of the standard.
Only two standards have these possibilities. The first one is ISO that has their ISO
certificate. The ISO certificate states that the organization handles information
well, based on security controls that are applicable for that specific organization
type. The other one is SAS70, which has as SAS70 statement. This statement
describes the situation in the organizations based on what the auditor has seen
and heard. The difference between the ISO certificate and the SAS70 statement is
that the ISO certificate shows that the organizations follow the ISO standard,
which means that they have implemented the security controls that are
applicable and a lifecycle for the information security management system. The
SAS70 statement only describes what the organization has done for security. The
certificate does not guarantee a standard package of security controls like ISO or
the standard usage of a lifecycle to improve the information security
management like ISO. All other standards are not certifiable, but it could be
sometimes used for other certifications. For example an organization can
implement the BSIMM activities and could describe the implementation with the
SAS70 statement. The standards with ISO compliance could get the ISO
certification if they implemented the security controls in a right way.
31

The number of pages for a standard has a big range from 22 pages up to 460
pages. For the SAS70 standard there is no document about security controls and
so on, but only about the audit. The reason of not including the audit document
pages is that it could not be used as a guideline to check how and which security
controls are implemented and have to be implemented. For almost all others the
number of pages for the full standard is mentioned. The exceptions for this are
the NIST Special Publication 800 and the COBIT 5 framework, where only the
relevant documents pages are mentioned.

The decision to choose a specific standard is not based on the price, but this
aspect is given to provide extra information. The prices of the standards vary
from free of charge up to €3570. The BSIMM is published on a Creative Common
License, which means that everyone could use it if they reference BSIMM. For the
other free standards it is unknown if it is published under the Creative Common
License. The prices of the documents are based on the full standard except the
exceptions of the ‘number of pages part’ ISO, NIST and COBIT, where only the
price is for the relevant parts.

Our decision to use standards with information security controls and the
guarantee ISO compliance limited us to 4 possible standards: ISO, NIST, the
Standard of Good Practices and the COBIT 5 framework. The 3 standards
(besides ISO itself) are larger than ISO and more detailed, but it has not a large
amount of extra value to the new evaluation framework. The choice goes to the
ISO 27002 standard itself.

A special remark that has to be mentioned is that BSIMM is specially made for
software development organizations (our target clients). If there is no
requirement for ISO compliance, BSIMM is a good option to use.

2.3 Evaluation frameworks for ISO 27002

Over many years, several approaches have been proposed in the research
literature on how to create an evaluation framework for ISO 27002. Each of
those researches uses its own terms and is slightly different from the others.
Only the most important ones for this research are mentioned.

2.3.1 Approach of Karabacak
In 2006, Karabacak and Sogukpinar proposed a quantitative method for ISO
17799 gap analysis [41]. In this research, they created an evaluation framework
for ISO 1779910 that is based on ISRAM. ISRAM (Information Security Risk
Analysis Method) is a risk analysis tool that focuses on compliance processes. It
is a web-‐based survey with at maximum one question for each ISO control. The
results of the question can be translated to a compliance percentage. Because of
this each answer and question has to have a specific weight for the compliance
percentage.

10
The ISO 17799 is renamed to ISO 27002:2005, but contains the same content.
32

2.3.2 Approach of Wright
In 2006 Wright created a white paper about measuring the effectiveness of
security using ISO 27002 in 2006[34]. This includes that the measurement can
be reproduced and compared with the evidence found in for example previous
years. Wright states the objectives, benefits and challenges of measuring security
effectiveness. Furthermore he describes what should and what needs to be
measured during the security measurement and he gives some examples of
possible measurements.

2.3.3 Approach of Bandopadhyay

In 2011 Bandopadhyay et al. did research into a quantitative methodology for
information security control gap analysis [37]. This methodology is based on ISO
27002:2005 and also compliant to other information security standards, like
COBIT and NIST SP 800-‐53. First an organization has to be categorized in one of
the three levels: low, medium or high. This categorization is based on several
parameters including legal impact. Each level has its specific mandatory controls
and optional controls. After the categorization the analysis is done using a
questionnaire, where each ISO control has one or more questions. Each question
has a relative weight between 0 and 1, so that the level of implementation can be
measured in a percentage.
2.3.4 Approach of Praxiom

The ISO/IEC Praxiom Research Group Limited created 27001:2013 Information
Security Gap Analysis Tool[43]. The tool shows the gaps between the ISO
27001:2013 standard and the current security practices in the organization.
Based on those gaps, the tool can give an overview of what an organization have
to implement to be ISO compliant. The tool consists of 517 questions for the gap
analysis. Each of the questions has to be answered with yes or no.

2.4 Comparison of evaluation frameworks

In the previous section, several evaluation frameworks for ISO 27002 are
described. A comparison is done between the evaluation frameworks based on
some aspects including the requirements for the new evaluation framework. The
following six aspects are used:
• ISO 27002 compliance
This aspect describes if the measurement in the evaluation framework
contains a check for the ISO 27002 security controls so that it is compliant
with the ISO standard. This means that all aspects of the ISO standard
have to be covered.
• Flexible
This aspect describes if the measurement in the evaluation framework
has the possibility to be used in different situations (e.g. small/large
organizations) and different type of checks (e.g. partly and fully
checkable).
• Measurable
This aspect describes whether the measurement in the evaluation
framework can objectively be measured by an auditor.
• Lightweight
33

This aspect describes if the measurement in the evaluation framework is
easy to answer (e.g. unambiguous) and the evaluation framework is small
(in short time doable).
• Market conformance
This aspect describes whether the results of the measurement are
represented into a market conform end mark. In other words: this means
the result tells if the organization implemented the security controls
better or worse than other organizations.
• Method
This aspect describes the method that is used in the evaluation
framework.

The new evaluation framework aims to satisfy the first five aspects.
All the mentioned research papers had some differences with the master thesis
research as clarified in the table below.
Table 14 Comparison approaches

ISO 27002 compliance
Market conformance
Lightweight
Measurable
Flexible
Method
Approach of Karabacak [41] √ √ / X √ √ X Questionnaire
Approach of Wright [34] Measurement
√ √ √ X X
of facts *
Approach of Bandopadhyay [37] √ √ / X √ X X Questionnaire
Approach of Praxiom [43] √ X X X X Questionnaire
* This whitepaper describes evidence-‐based measurements to check the effectiveness of the current
implementation of ISO controls. This result of the measurement only indicates whether the
organization improved in comparison with the last measurement. The method measures some
specific facts inside the organization to show the ISO compliance.

The second aspect was flexibility. The most flexible approach is from the
approach of Wright where each ISO control has a specific measurement and does
not depend on other measurements. As an organization you can select only the
measurements that are applicable. The approach of Karabacak and the approach
of Bandopadhyay also have flexibility, but are both more restricted. These two
research papers [37][41] could measure a part of ISO (by headings), but it leaves
no free space for extra ISO compliance actions and exceptions. The approach of
Praxiom is not flexible, because an organization has to check the ISO standard
fully for getting the end result.

Furthermore the evaluations that the frameworks produce have to be
measurable. The approaches of Karabacak and Bandopadhyay have questions
and predefined answers, which all are objectively measurable. The approach of
34

Wright is different, because it is based on measurements on facts. An example of
fact-‐based measurement is: ‘in the last year, how much time is system x down?’.
However, the approach of Praxiom consists of several questions that are difficult
to answer, because the aspects (e.g. performance, suitability) that were asked are
not specific measurable. An example is ‘Do you improve the performance of
your ISMS?’. The possible answers are: yes and no. For an employee it is hard to
answer this question. The employee can wonder: what does my organization
have to improve to have a better performance? How can I measure that? The
answer is not objective, because an employee could have other opinions on it.

The fourth aspect is how lightweight the approach is. These research papers
have a large difference on the number of questions or measurements. Besides
that some have predefined answers. Predefined answers have the advantage to
make it easier to compare the answers from different organizations. A
disadvantage is that the correct answer for an organization might not be given in
the predefined answers. The approach of Karabacak is the only evaluation
framework that has maximal one question for each ISO control. This means that
in the ISO 27001:2005 version, the number of questions is in total 133. An
additional benefit of this evaluation framework is that it has several predefined
answers. All other frameworks contain more questions. Further, all others have
predefined answers except the approach of Wright.

Another requirement for the new evaluation framework is that the result of the
evaluation framework shows how market conformant an organization is. All
evaluation frameworks show the results of the framework in percentage of ISO
compliance instead of market conformity.

The last aspect for the comparison is which methodology was used in the
evaluation framework. In three of the four cases, a questionnaire is used. Only
one uses something else, namely fact based measuring like how much time the
system is down. In this case the evaluation framework does not base the results
on answers of the employees, but on facts that can be measured. Measuring the
implementation based on facts would be ideal to get an objective way of
measurement. However, it will much more time consuming to measure the
security of organizational processes with facts.

As mentioned at the start of this section, the new evaluation framework has to
meet the first five aspects of the comparison: ISO compliance, flexibility,
measurability, light weightiness and market conformity. None of the four
checked evaluation frameworks did meet all of the requirements.
35

3 Construction of the questionnaire
In 2013, a previous research project was initiated to investigate the possibilities
for this master thesis’ evaluation framework [6]. The result of the previous
research project was an earlier prototype that contained a number of
checkpoints. A checkpoint is a statement that could be answered with a
predefined answer (preferably only yes/no), for example, ‘A security awareness
training is given every six months’. One of the problems encountered during the
development process of the earlier prototype was that the question list was too
long. It contained 212 checkpoints for only 32 ISO controls. The size of
checkpoint list has to become much smaller during the current research process,
so a reduction is needed. Another possibility is to start all over again, because
creating new checkpoints is probably less time consuming than performing the
reduction. Along with being a smarter way of reduction, a benefit for starting all
over again is that a new version of ISO 27002 was just released. The new
evaluation framework has to be adjusted to this new version. It was decided to
start all over because of these two factors (reduction and the new ISO 27002
version).

The requirements for this new evaluation framework are the same as in the
earlier prototype, namely:
• Lightweight
• Flexible
• Measurable
• Market conformity
• ISO 27002 compliance.

In this chapter, the development process of the questionnaire is described. The
process was separated into three phases:
• Phase 1: the trial questionnaire
In this phase a small questionnaire is created as a trial for the full
questionnaire. This trial questionnaire shows possible issues for the full
questionnaire. This trial framework consists of 25 questions for 38 ISO
controls.
• Phase 2: the full questionnaire
The full questionnaire is designed in this phase. This questionnaire
consists of 50 questions for all ISO controls.
• Phase 3: the improved questionnaire
In this phase the feedback of the validation inside SIG and the validation
with consultants is processed. This questionnaire consists of 52 questions
for all ISO controls.
In all phases validation sessions are applied to confirm that the created
questionnaire works as intended. In these validation sessions, there was a
special focus on the following four aspects:
• Is the questionnaire useful?
• Is the questionnaire easy to answer?
• Could the questionnaire be answered in a short timeframe?
36

• Does the questionnaire measure the right things?
In addition, the requirement completeness is also taken into account as extra
information to improve the questionnaire on missing aspects.

3.1 Phase 1: the trial questionnaire

During the first phase, the trial questionnaire was created. The idea of this trial
questionnaire is to create a questionnaire of 25 questions. SIG has five
responsibility functions that are involved to the ISO 27002 implementation,
namely: Management (CSO), IT, Software Development (Lab), HR and Facility
management (Office). For each responsibility function there are five questions to
answer to get insight about how good the trial questionnaire performs. The five
groups are used for the validation inside SIG.
3.1.1 Development
During the development of the earlier prototype [6] it was already noticed that
the checkpoints were not the right choice because of the large volume of the
earlier prototype. In the earlier prototype it was tried to use questions with
predefined answers instead of those checkpoints to reduce the volume. The
implementation of questions satisfies the reduction expectations. The reason to
use a questionnaire is a result of the satisfied expectations and the technique is
the most used technique in other evaluation frameworks (see Section 2.3). The
other technique that was used in other evaluation frameworks was measuring of
facts. The advantage of this technique is that it produces objective result, but this
technique is not chosen for several reasons. One reason is that fact-‐based
measuring assumes that an ISMS has already been well implemented. This does
not have to be the case. Besides that the measurement is more dependent on the
way the ISMS has been implemented, so when this technique is used then the
evaluation framework has to be adjusted more to new technologies and ways to
implement the ISO controls.

Another point what is learned from the earlier prototype was that the evaluation
framework will increase very easily in size when too much details of the ISO
controls are used.

After learning from the earlier prototype, there were some design constraints
described for the trial evaluation framework:
• A maximum of one question for each ISO control
• Questions with pre-‐defined answers for making it easy to analyze the
results
• Questions are posed in such a way that every organization could answer
it, so the results can be compared.
• Answering the full evaluation framework may only take one day at
maximum

During the start of the research, the trial questionnaire with the 25 questions is
created. At first instance, it was tried to create one question for one control, but
it was noticed that it was possible to reduce much more. At the end there were
25 questions, which were based on 38 ISO controls. There are 114 ISO controls
37

in total, so approximately a third of all ISO controls were covered in the trail
questionnaire. One example of a question that was created can be found in Table
15. The question is asked in a relatively open manner, because otherwise
multiple questions have to be used to get the same amount of information. To
help the participants understand the question and to get them to find the correct
answer, pre-‐defined answers were used in the questionnaire.

Beside the four design constraints, the possibility to add some extra comments is
created, because in some cases people prefer to give a bit more details or wants
to describe an exception.

Table 15 Example -‐ question trial questionnaire
ISO control 9.2.1 User registration and de-‐registration

Control context A formal user registration and de-‐registration process shall be
implemented to enable assignment of access rights.
Question Are there any actions used to manage the user IDs?
Answer(s) ☐ No
☐ Yes:
☐ Usage of unique user IDs
☐ Immediately disabling or removing user IDs of users
who left the organization
☐ Periodically identifying and removing or disabling
redundant user IDs

Comments and/or extra actions:
……………..…………..…………..…………..…………..…………..……
……………..…………..…………..…………..…………..…………..……
……………..…………..…………..…………..…………..…………..……

It is considered to do a selection of questions in the evaluation based on a risk
assessment. In this way the trial questionnaire is more flexible and the questions
are more relevant to the client organization. However, if you use different
questions for evaluations then it is hard to compare the results of the
evaluations. This means that SIG could only say how well is implemented, but not
how market conform the implementation is. Because comparability and
objectivity is wanted, there is chosen not to pre-‐select questions based on a risk
assessment. The same approach is taken in all evaluation models of SIG.

There was a design constraint of one question for one ISO control, but it is
preferred to have fewer questions. That is why there is chosen to combine
several ISO controls in one question. It was possible to combine ISO controls if
the following requirements hold:
• The ISO controls share a related topic (e.g. both are questions about
assets)
• One person could have the knowledge about both ISO controls.
A simple example of a combination of two ISO controls can be found in Table 16.

38

Table 16 Two combined ISO controls
ISO controls 6.1.3 Contact with authorities

6.1.4 Contact with special interest groups
Question Is there a procedure which says when, how and which
organizations have to be contacted?
Answer(s) No
Yes, for:
Authorities (e.g. law enforcement, regulatory bodies,
supervisory authorities)
Special interest groups (e.g. specialist security forums)
Suppliers (e.g. source development organization)
Clients

Comments and/or extra actions:
……………..…………..…………..…………..…………..…………..……
……………..…………..…………..…………..…………..…………..……
……………..…………..…………..…………..…………..…………..……

3.1.2 Validation inside SIG
The first phase validation was done for the trial questionnaire. This trial
questionnaire consists of 25 questions, which are equally divided under the five
functions (Chief Security Officer, Lab, IT, Office and HR) for the ISO 27002
implementation inside SIG. During the validation, five one-‐hour sessions are
done (one for each department).

During a session the employee that is responsible for the implementation of the
security controls has to fill in the answers of the five questions that are related to
their department. Further they have to give feedback on those questions. For
feedback there were three statements described, where participants could agree
or disagree on:
• The question is useful
• The question can be easily answered (no ambiguity and so on)
• The question is complete
For each statement the employee can answer 1 (totally disagree) to 5 (totally
agree). There is also an option to give comments on each aspect. The assessment
of the three statements has to be given for all five questions so that it is know
which questions had to be improved.

The assessment of the three statements for the 25 questions on the aspects
usefulness, easiness and completeness indicated promising results for the full
questionnaire if it was created similar to the trial questions. All three aspects
obtained more than 50% of the answers agree or totally agree. Besides that in all
three aspects the answers disagree and totally disagree are lower than 25%. Of
course, there is room for improvement. The results of the assessments are
shown in Figure 7. The questions the participants were asked had no overlap.

39

There is feedback given to improve the questions. The most notable remarks
during the feedback will be discussed.

A problem encountered during the validation was that filling in the questions
and the discussion were mixed. This made it harder to measure the exact time of
filling in the questionnaire, although it was noticed that it took a lot of time. In
the second phase of the development, it was needed to make sure that the
questionnaire was reduced even further or something different to reduce the
time needed to answer the evaluation form.

It was also noticed based on the given answers and the discussion that the right
aspects of organizational process were measured.
Easy to
Useful answer Complete
Totally
0 0 agree
0
Agree
1 3
6 6 5
6 8
Neutral
7
5 Disagree
10
10 8
Totally
disagree

Figure 7 Validation inside SIG phase 1 -‐ results
3.1.2.1 Usefulness

Multiple times it was mentioned that some questions or possible answers are
only useful for a specific type of organizations (e.g. only banks). Possible
differences in companies or other organizations can be for example reliance on
software, security level, location and whether the building is shared with other
organizations. An example answer of a question was the AIVD screening for a
new employee. This AIVD screening is only necessary in a highly secured
environment. In ‘standard’ organizations there have to be some checks, but the
AIVD screening is not needed for most organizations. The above-‐mentioned
problem where questions and/or answers are not applicable for the
organization, the participant described the not applicable question as ‘totally
disagree’. However, the same question could be potentially useful for other types
of organizations. This means that the ‘totally disagree’ mark does not mean that
the question has to be removed from the questionnaire.

3.1.2.2 Easiness
In the feedback on the questions whether the questions were easy to answer, it
was shown that multiple times the participants preferred to have some
40

examples. For instance one question about contact with authorities and special
interest groups would become much easier to understand by using examples.
The question asks if there is a procedure which specifically say when, how and
which organizations should be contacted. The initial two options are yes and no,
but when you choose yes you can choose out of several options: the authorities,
special interest groups, third party services and contractors. Somebody who has
to fill in the questionnaire does not have to know what is in those four groups.
The usage of the examples makes it clearer how to answer the question.

Besides using more examples, in some questions there were still some words
that caused unambiguity. For example the word ‘system changes’ is used, which
led to a discussion what is and is not covered under system changes.
3.1.2.3 Completeness
The three questions where participants were disagreeing about the
completeness had two causes: missing examples (see Section 3.1.2.2) and
missing predefined answers to the questions. One of those three questions was
about the terms and conditions in contracts. At this moment the trial framework
distinguishes two groups: employees and contractors, but there are more. In the
organizations, there are for example also interns, self-‐employed workers and
employees of an employment service provider. These groups may have different
terms and conditions or regulation around the terms and conditions, so these are
special cases that some organizations did not think about. The second question
that missed predefined answers was the question about system security testing.
The question has some general answers about how the system security test is
done and how it is organized, but it is preferred to have more detail:
white/grey/black box testing, inside or outside the organization, code review or
runtime test. The third question was also missing some predefined answers just
like the second question.

3.2 Phase 2: the full questionnaire

During the second phase, the trial questionnaire is extended to the full
questionnaire. The goal of this questionnaire is to cover all 114 ISO controls and
use the feedback of phase 1 to improve the questionnaire. After finishing phase 2
several validations are scheduled:
• Validation inside SIG (similar to phase 1 validation only for full
questionnaire)
• Validation with consultants
During the validation, the same aspects as in phase 1 of the development are
validated. In these validation sessions, there is a special focus on these three
aspects:
• Could the questionnaire be answered in a short timeframe?
Further completeness is taken into account as extra information to improve the
questionnaire on missing aspects.

41

In this phase, two validations with two different groups are done, because both
groups have different functions. The validation inside SIG is done with people
who have the knowledge of the ISO 27002 implementation inside SIG, so they
can answer the questions. The validation with consultants is done, because those
consultants will help clients fill in the questionnaire in the future consultancy. It
is therefore important that the consults have experience with the questionnaire
as well.
3.2.1 Development
In the validation of the trial questionnaire, it was shown that answering 25
questions already takes a lot of time. Therefore it was difficult to fulfill the design
constraint of maximal one day to answer the full questionnaire. This resulted
into an extra design constraint, a maximum of 60 questions. Of course having
less questions is still preferable. This new constraint meant that the
questionnaire could not have 114 questions anymore. The rest of the design
constraints remained intact.

In the first phase there were 25 questions for 38 controls. When this data is
extrapolated then there would be 75 questions for 114 controls. This means that
the design process has to be adjusted to be able to combine more ISO controls for
meeting the project constraints. In some cases more ISO controls were able to be
combined than in phase 1, so this made it possible to satisfy the design
constraints. Two ISO controls can be combined if they fulfill the same two
requirements as used in phase 1:
• The ISO controls share a related topic (e.g. both are questions about
organization assets)
• The person required to answer the question should have knowledge
about both ISO controls.
A simple example of a combination of two ISO controls can be found in Table 16.
Eventually, a questionnaire with 50 questions was created that fully cover the
114 ISO controls.

3.2.2 Validation inside SIG

The questions of the full questionnaire were divided under the five departments
ISO 27002 implementation inside SIG (CSO, IT, Lab, HR, Office). During the
validation there were in total five sessions (one for each department), where
each session took 30 minutes. Only the IT department had a one-‐hour session,
because they have more questions. Before the session, the questions were given
to the employees. These questions had to be answered before the session,
because the validation session is only intended for receiving feedback.

For feedback some statements are made again, to which the participants needed
to give their opinion. This measurement was to determine how good or bad the
questionnaire was overall. There are two statements described:
• The questionnaire is useful
• The questionnaire can be easily answered (no ambiguity and so on)
42

For each statement the employee can give an answer from 1 (totally disagree) to
5 (totally agree). Furthermore there is an option to place some comments
concerning each aspect.

The assessment of the two statements indicates that the full questionnaire is
accepted as a good option for the service inside SIG. They agreed on that it was
useful and easy to answer. The results of the assessment are shown in Figure 8.
The figure shows for each statement (useful, easy to answer) the collective
results of the five sessions, which in each session a specific group if questions are
given that are linked to the task of the participant. Besides the measurement the
two opinions, there was also feedback supplied by the participants to improve
the questions.

Useful Easy to answer

0 0
1 1 1 Totally agree
Agree
0.5 Neutral
Disagree
4 2.5 Totally disagree

Figure 8 Validation inside SIG phase 2 – results
Everyone agreed that the (current) questionnaire is useful. This is a positive
indication for the future. If people think the questionnaire is useful then there is
a higher chance for it to be used properly.

Most of the participants also said that the questionnaire was easy to answer.
Only two of the five were less positive. One person did not think that all
questions were easy to answer and this resulted in an answer between agree and
neutral. This is showed in Figure 8 in the 0.5 agree and 0.5 neutral. The reason
for this result was that in some cases the participant had to ask some small
questions to get it clarified.

The IT department disagreed about the easiness. The reason to disagree on
easiness was that the questionnaire is generally formulated. This means that if
there is a question about the systems then the participant has to think of: which
systems do we have? What do we do for all those systems? And so on. This
means that you have to create an overview for yourself which systems you have
and which security controls are implemented. This extra step makes it harder to
give the answers. This is especially the case in the IT department questions, so
that is why other participants (lab, office, CSO, HR) did not encounter this
43

problem. In addition the time to fill in the full questionnaire was under in the
predefined requirement (1 day).
3.2.3 Validation with consultants
A validation session with the consultants of SIG is done. Consultants are
employees that give professional advice on a specific subject. The selection of the
participating consultants was based on their special security consultancy
capability. In total there were four consultants chosen. For each of those four
consultants, a 30-‐minute validation session is done. Before the session, the full
questionnaire is given to the consultants, so they could take a look and write
some notes or feedback.

For feedback the consultants fill in their opinion on two statements about the
quality of the questionnaire.
• The questionnaire is useful
• The questionnaire can be easily answered
For each statement the employee can give a rating ranging from 1 (totally
disagree) to 5 (totally agree). Furthermore there is an option to place some
comments.

The results of the small questionnaire indicate that questionnaire is accepted as
a good option for the service inside SIG. All consultants agreed on the fact that
the questionnaire was useful. However, they encountered some difficulties with
the aspect ‘easy to answer’. The main solution to improve the easiness is to write
a little introduction at the start of the questionnaire. The results of the
assessment are shown in Figure 9. The figure shows for each statement (useful,
easy to answer) the collective results of the four sessions.
Useful Easy to answer

0 0
Totally agree
1 1 1
Agree
Neutral
Disagree
3
2 Totally disagree

Figure 9 Validation with consultants – results
Summarizing the feedback from both security control owners and the
consultants, all participants agreed on that the questionnaire was useful. This
results confirm the agreement on usefulness of the validation inside SIG.

Another outcome was that the consultants gave a lower mark to easiness. There
were three major comments for the lower mark.
44

-‐ More structure in the questionnaire based on for example subjects, and
functions. This was already known, but not implemented yet.
-‐ An introduction before the questionnaire for the participants.
-‐ A document with information how a consultant could get the end result
(star rating and actions).
Possible explanation for the differences between the validation inside SIG and
the validation obtained from the consults is that the participants of the validation
inside SIG have seen the questionnaire (in the trial questionnaire validation)
earlier, so they are more familiar with questionnaire. Furthermore, the security
control owners should be able to understand the questions better since most of
the questions are about their job functions. Besides that, consultants has a
different type of perspective, because they must do the consultancy and do not
have to answer the questions. They view the questionnaire from a different
perspective.

The reason that one of the consultants chose for the option disagree was that the
questions are quite general. This is the same feedback as the IT department has
given. There was some discussion about the subject and the result of the
discussion was that the organization has to describe the actions for the ‘weakest’
removable media. The weakest removable media is the removable media that
has the lowest implemented security controls to protect the content. In case this
is mentioned at the start of the questionnaire then it would be okay according to
the consultant.

3.3 Phase 3: the improved questionnaire

During the third phase, the full questionnaire is improved based on the feedback
of the validation inside SIG and the validation with the consultants. After
completing the adjustments of the questionnaire, there were two validations
done with external organizations. This way the questionnaire could be validated
in practice with people that were not involved in the development process.
During the validation, the same aspects as in earlier phases of the development
are checked. The validation focus on these three aspects:
• Could the questionnaire be answered in a short time period?
In addition, the requirement completeness is taken into account as extra
information to improve the questionnaire on missing aspects.
3.3.1 Processing feedback of validations
During the validations sessions, some aspects to improve were provided by the
participants. Most of this feedback only required some minor adjustments to the
text.

Besides the adjustments to the text, there was some confusion about how to fill
in the questionnaire in general cases (see end of Section 3.3.2) and how the
consultant has to process the evaluation framework. An introduction for the
questionnaire is created to provide extra information for the client organization.
After finishing the questionnaire, a translation from the answers to a SIG rating
45

was designed. The introduction and the rating system solve the problems
encountered by the consultants.

3.3.2 Validation with external organizations

Validations with potential clients of SIG are done at the end of phase three. The
questions are divided differently than inside SIG. In the validation from SIG
internally, all questions are spread over five functions (CSO, IT, Lab, Office and
HR). However, for the external organizations all questions are spread over three
functions. The reason to do the division differently is that the HR and Office have
a limited amount of questions. We would like to limit the number of contact
person in order to facilitate the further sessions planning. In many cases, the CSO
has the knowledge to answer the HR and Office questions. This means that the
CSO could replace the two other functions, so that planning a conversation would
be easier. In case the CSO has not enough knowledge on the HR and Office
controls, he/she could distribute the questionnaire for obtaining the correct
answers. Further it is preferred to divide the amount of questions equally over
the employees. In the validation inside SIG this was not the case, but the new
division took this into account. Preferably the questions are divided over the
following three functions:
• Chief Security Officer (CSO) or a similar function
• Head of IT or a similar function
• A person from the software development department (if applicable)
It is difficult to give an exact reason about choice for above functions. The
division is based on the current division of the ISO controls inside SIG.

For the validation with external organizations, there were found two
organizations willing to participate into the validation. Both organizations are IT
related, but do not develop software themselves. One organization is considered
small (50-‐200) and the other one business is medium-‐sized (200-‐500). Both
organizations were positive about the questionnaire, but there were some
(small) remarks.

During the validation sessions, it was stated that these questionnaire is a
concrete (check-‐) list of actions for what has to be done by the organization to
keep their information protected. Both organizations agreed on that it was a
good start for getting ISO 27001 certified in the future. One organization said
that when they started to get an ISO certificate themselves, they used an external
organization to help them. However, this questionnaire would have helped them
more than the external organization did.

A critical point that was mentioned was that some questions could be
interpreted in different ways. This makes it harder to compare the given
answers, because possibly you compare two different things. This point has to be
solved in the future work. In the end, there have to be a balance between the
clearness of the questions, the generality of the questions and the time to answer
them.

46

Furthermore both organizations did think the content of the questionnaire was
quite complete. One organization did say it would be also nice to include a
question about who is ultimate responsible. For example this could be a
manager, but also other employees with lower functions. The reason for this
addition is that it is important that a high function (e.g. a manager) is the
ultimate responsible, because security requires high management commitment.
Another addition could be a general risk analysis.

Another observation was that the third part of the questionnaire about software
development has to be handed to an experienced employee such as the head of
development. An inexperienced employee has possibly not enough knowledge to
fill in the questionnaire properly.

The participants reminded me that some organizations prefer less paperwork.
These organizations could have implemented many security controls, but do not
write most of this down. In the questions, it is asked if there are policies and
documentation about implementation, but are these documents really
necessary? This could be investigated in a future work.

Furthermore, the time spent on filling in the questionnaire was in both
organizations under the defined limit of one day.

47

4 Construction of the rating system
In Section 3, the construction of the questionnaire is described, but a translation
from the answers of the questionnaire to a rating and actions is still to be
defined. In this chapter this aspect of the evaluation framework is described.

The translation from the chosen answers to an actual rating is based on how
market conform the chosen answers are. This means that the rating scheme
should be kept up to date. During the research process, the validations were
tested in cooperation with only a limited number of external organizations. This
made it difficult to give a market conformity rating at this stage in the project.
The actual rating has to be further calibrated by SIG employees once more data
points are available.
4.1 The SIG star rating

There are several options to calculate the SIG star rating. One option is to use the
techniques that are already used inside other SIG services (‘The SIG approach’)
[46]. However, there are other ways to do it. Another option that is considered
was the risk approach, which is more adjusted to the organization situation. In
the following two sections both approaches are described. In case of the
evaluation framework, there is chosen for the SIG approach over the risk
approach. The reason for this choice is that the added value of including risk in
the approach is too low in comparison with the cost of time and the complexity.
Besides that it would be difficult to compare organizations and the SIG approach
is more consistent to other SIG services.
4.1.1 The SIG approach
In the research of the earlier prototype [6] it was decided to do a simple rating
scheme to get to the SIG star rating. In this rating scheme, the answers for each
question are classified in a specific box. There are five boxes from 1 star until 5
stars, where the 1-‐star box is easier to satisfy and the 5-‐star box is the most
difficult one. An example of this is shown in Table 17. This approach uses
maturity levels of grading the question. This means that for two stars all answers
from the one star box (answers 1 and 2) and also the answers of the two stars
box (answer 5). The classification of the answers into the boxes are not given to
the participating organization on beforehand due to the possibility to choose
specific answer to get a specific result.

Table 17 Example – boxes
Stars Needed answers

✭ Answer 1, Answer 2
✭✭ Answer 5
✭✭✭ Answer 3, Answer 4
✭✭✭✭ Answer 6
✭✭✭✭✭ Answer 7

The SIG star rating is based on which boxes are met in the questions.

48

In the earlier prototype, the SIG star rating is made by taking the average of the
box star (1 to 5) for each question. For example you have two questions: one
question has all required answers to be in box 4 and the second question has the
answers for box 2. The SIG star rating is then (4+2)/2 = 3.

The same technique as in the earlier prototype could be used, but it was found
that there are some problems with the technique [44]. The problem was that
when you take the average with many inputs (in this case 52 questions), the
average would be coming to the middle. This means that mostly a 3 star rating is
given. This is not what is desired for a security check.

This could be solved with a transition table. In SIG they use already another
technique (transition technique), which is quite similar as the used technique of
the earlier prototype. The only real difference is that they use a transition table
instead of the average. This could be a good solution for the above-‐mentioned
problem. This transition technique puts answers of the questions into five
categories:
• Level 0 (Does not meet requirements for any level)
• Level 1
• Level 2
• Level 3
• Level 4
This level system is almost the same as shown in Table 17, where each answer is
in a specific box (or in this case a level) and for a specific level all answers of the
lower levels have to be met. Depending on the number of answers, a question is
in a specific category (see Table 18).

Table 18 Example -‐ level system
Level 1 Level 2 Level 3 Level 4

Question 1
Question 2 X X
Question 3 X
Question 4 X X X X
Question 5 X X X

After classifying the questions into five categories, a transition table is used. The
used transition table can be seen in Table 19. The percentages mean the amount
of questions has to be at least that level or higher. The example data set of Table
18 is used to illustrate the idea. There you see four crosses in the column ‘level
1’, but there are five questions. This means that level 1 has a compliance of
(4/5)*100 = 80%. If you also check this for the other levels, you will find the
following results: level 2 at 60%, level 3 at 40% and level 4 at 20%. In the table
below, it is possible to look up and see that all the requirements are met for a 3
star rating, but not for a 4 star rating. This means that the example has a 2 star
rating.

Table 19 Transition table
SIG star rating Level 1 Level 2 Level 3 Level 4
49

compliance compliance compliance compliance
✭ 30% 0% 0% 0%
✭✭ 60% 35% 15% 5%
✭✭✭ 85% 55% 30% 10%
✭✭✭✭ 90% 80% 50% 15%
✭✭✭✭✭ 100% 95% 60% 20%

Not only the above-‐mentioned transition table is tried, but there are three
scenarios used. It was checked which one fits the best:
• An easy to meet transition table: a table with low percentages to get a
high star rating.
• An optimistic transition table: a table with high percentages to get a high
star rating.
• A transition table that is in between the two transition tables mentioned
above.
Firstly, the rating system is used to calculate the results of the questionnaire
sessions of the three participating organizations (SIG and two external
organizations). The compliance percentages of those organizations can be found
in Table 20. It is known that two of the organizations have an ISO certificate and
all three have high security implementation. This meant that all organizations
has to be at least three stars or higher. The three transitions tables were used
and it was found that the transition in the middle has the most representative
percentages. This transition table can be found in Table 19 and the SIG star
rating results for the organizations are shown below.

Table 20 Results validations
Organizat-‐ Level 1 Level 2 Level 3 Level 4 SIG star rating

ion compliance compliance compliance compliance
A 90,0 62,0 42,0 16,0 ★★★ (2.7)
B 93,9 83,7 51,0 16,3 ★★★★ (3.6)
C 94,0 76,0 36,0 12,0 ★★★ (2.8)

Besides the small transition table, a detailed transition table is made where the
SIG rating can be determined more accurately (see Appendix B: Detailed
transition table). The results vary between 0.5 and 5.5. For each level there are
six data points that are shown in Table 21. Between the data points, there is a
linear function. In Figure 10, the piecewise linear function is given for Level 2.

Table 21 Six data points
Data Rate Level 1 Level 2 Level 3 Level 4

point compliance compliance compliance compliance
1 0.5 30% 0% 0% 0%
2 1.5 60% 35% 15% 5%
3 2.5 85% 55% 30% 10%
4 3.5 90% 80% 50% 15%
5 4.5 100% 95% 60% 20%
6 5.5 100% 100% 100% 100%
50

Level 2
100
90
80
70
60
50
40
30
20
10
0
0.5 1.5 2.5 3.5 4.5 5.5

Figure 10 Level 2 piecewise linear function

To know the exact grade, the following steps are done:
1. For each level, you check the position of the found compliance in the
table. This gives you a specific rate (e.g. 3,5)
2. The end result is the minimum of the found rates in step 1. For example
when you found the following ratings: 3,7; 3,5; 2,1; 2,2. Then the end
result is 2,1
4.1.2 The risk approach

An option is to base the ratings on the risks of the organizations is called the risk
approach. A bank for example has different risks than a telecommunication
organization.

The risk approach uses weights for questions based on the risk. The weight can
for example be set from one to ten. This risk is based on a risk assessment for the
organization. The higher the risk, the higher the weight. Each different weight
requires a specific percentage of implementation to be marked as ‘Good’/ ‘Ok’ /
‘Bad’. The higher the weight, the more difficult it is to be marked as ‘Good’.

The results of the questions (good, ok, bad) will actually be decisive for the final
rating. This could be done with various techniques, like a point system where
goods give more points. At the end there is an overall rating.

This approach could also be used as described in the SIG approach, but when
calculating the star rating you need to take the weights into account.
51

Another implementation of the risk approach is possible, like for instance:
• Adding more or less risk groups.
• Changing the weight factor of the implementation percentages based on
type of organization. For example a bank would have to be more secure
than a software development organization to get the same rating.
4.2 The actions for improving the security

Besides the SIG star rating, an organization will benefit from suggested actions to
improve the security. This extra information will help to see what has to be done
so that the vulnerabilities can be reduced to a minimum if the actions are
implemented well.

The answers of a questionnaire show the strong and weak points of the security
implementations in the organizational processes. Based on those weak points,
the necessary actions for the organization can be determined. In the overview of
the questions with the associated category (see Table 18), you know to which
questions the organizations score either good or bad. In case questions score
badly, SIG could formulate some actions to help improve the company’s security.
When there are many actions required then it is necessary to give an order for
implementation, because an organization cannot implement everything at the
same time. This order could be based on the costs, the likelihood that an incident
related to the action occurs, the impact of the incident and so on. An example for
such an action is that the organization has to use encryption on removable
media.

52

5 Discussion
5.1 Evaluation framework improvement
The evaluation framework has to be maintained to be effective and relevant
during the usage. New security attacks and technologies have to be taken into
account

During this research process an evaluation framework was made, but the type of
security attacks are always changing. The reason for the change of security
attacks is caused by continuously changing technologies to newer ones and new
attacks. This means that the security controls have to be adjusted to fully protect
the situation at that moment. This can result into an outdated evaluation
framework.

One possibility is making the evaluation framework so general that the changing
technology does not affect the evaluation framework. As mentioned in Section
2.4, it is not possible to have no changes. That is why two techniques are
proposed to improve this evaluation framework.
• Continuous improvement: A lifecycle
• Event-‐based improvement: after a new ISO 27002 standard is released
5.1.1 Continuous improvement
As mentioned in section 5.1, the evaluation framework has to be adjusted to the
changing work environment and knowledge. This updating process could be
realized in different ways.

An option is to do a continuous research on new ways to protect the
organizational processes, but this will costs a lot of time and money. The value of
an up-‐to-‐date framework is not so high as the price and time it will take to
update it like this.

Another way is to integrate the update into the evaluation framework itself. This
could be done with a lifecycle, which consist of three phases:
-‐ Do
SIG carries out several assessments with the evaluation framework in the
Do-‐phase. These assessments could be ordered by organizations as a
service.
-‐ Check
The outcomes of the assessments are checked and verified if some
answers are still used. Besides that it is counted how many times a new
answer is added by a client.
-‐ Act
The evaluation framework is adjusted based on the results in the Check-‐
phase. Possible adjustments could be:
• Predefined answers could be updated (added, changed or
removed)
• Questions could be updated
53

• The answers could be classified in a higher or lower maturity level
• Transition table could be changed

Table 22 Continuous improvement -‐ Lifecycle
Do
Act Check

In case an answer is not mentioned for x amount of times (e.g. 20 times) then it is
removed from the evaluation framework. When an answer is added when at
least y times (e.g. 5 times) an extra answer is mentioned by an organization. This
check can possibly be done automatically.

This lifecycle does not cost much effort, because the input of new technologies
and new processes is already given through clients in the Do-‐phase. The check-‐
phase has also be done for the SIG service, so only a small amount has to be spent
on the small check for the lifecycle. The act part is only required if there is some
special result from the check-‐phase.

5.1.2 Event-‐based improvement
Besides the continuous improvement, it would be necessary to check a new
standard sometimes as for instance a new ISO 27002 standard. The reason for
this is that it is unknown if there is sufficient information from the assessments
to know all new up-‐to-‐date facts. In this research an improvement of the
evaluation framework based on a release of a standard is an event-‐based
improvement. This event-‐based improvement consists out of checking the
difference between the old and new versions. After that the new ISO controls and
removed ISO controls are identified. Based on this information, SIG could
consider adjusting the current questions, but also adding or removing questions.
It would also be an option to do this improvement when there is a new release of
other standards such as BSIMM.

54

5.2 Potential risks in the evaluation framework
Every standard, method, evaluation framework and so on have advantages and
disadvantages. During the creation of those questions, you have to find a balance
between the requirements, such as being lightweight, and the amount of ISO
coverage. Some questions that showed up during the process of the new
evaluation framework were:
-‐ Does it have to cover all aspects?
-‐ Does it have to be lightweight?
-‐ Is there only a check involving a questionnaire or are the practical
implementations also checked?
At the end, when creating an evaluation framework some choices had to be
made. One choice was to mark the wish to be lightweight as important. This
choice has advantages that the evaluation framework is easy to use. However,
the choice can also bring risks. Only the highest risks are mentioned below. It is a
fundamental tradeoff between how light-‐ or heavyweight the approach is and
how detailed it is and how confident we can be about the end result.

First of all, it is not possible to cover all aspects of information security in an
evaluation framework. There are always new upcoming technologies and
security threats, which means that it is required to change the evaluation
framework constantly. This means that the evaluation framework is limited to
the mentioned information security controls in the ISO 27002 standard. This
amount of information security controls is manageable for the evaluation
framework.

The second risk in the evaluation framework is associated with the wish to be
lightweight. Lightweight means that only a limited amount of information is
checked, so it will be more a global check instead of a detailed check. This means
also that not all aspects will be checked and there is a chance that an unsecure
ISO control implementation will not be caught via the evaluation framework.

Another risk is that the wish to be lightweight made it impossible to check all
documents and practical implementation. This made us choose the use the
questionnaire and to have the possibility to do some selected checks. This choice
makes it impossible to know 100% sure that what they say is true and how good
the quality is of the document or implementation. The results are based on the
honesty of the employees of the client.

In addition, there is a possible risk that the knowledge of the participants are
outdated and/or not complete. The knowledge problem could lead to wrong
answers and that results in a wrong star rating. If the knowledge problem is
known, the participant could ask assistants from colleagues to help them to give
the correct answers.

55

6 Conclusion
There are many standards and evaluation frameworks that are related to this
research. However as discussed in Chapter 2, none of those meet all
requirements (e.g. lightweight, measurable, flexible) that SIG desires. This
research aims to meet all these requirements. However, it was a challenge to
implement the combination of all requirements (e.g. lightweight and complete).
Objective measurement was a difficult challenge even without the restriction of
the other requirements. The final solution was created by looking at what is
acceptable and what is not, coming to a compromise between the requirements.
Ideally, the evaluation framework should be objective and quantitative, but in
reality is always partially subjective and qualitative.

In an earlier prototype checkpoints were used that could be regarded as
statements where either yes or no could be the answer. This concept does not
work well in large questionnaires, because to obtain enough information a large
number of checkpoints is required. The option to have several predefined
answers to a question instead of only yes/no-‐checkpoints is in this case better.
These predefined questions allow for a much smaller questionnaire. Predefined
questions were used in this evaluation framework, but have to be used with
caution. Using only predefined answers does not allow the interviewee to
describe exceptions or give extra information. This possibly important
information would not reach the analyzers. For this reason comment lines were
added to the questionnaire.

First a trial evaluation framework is created consisting of 25 questions divided
over the five persons inside SIG that are responsible for the ISO 27002
implementation. The number of questions for each person was enough to verify
whether the framework was designed with a good method. The validation inside
SIG of this trial evaluation framework made it able to do a reduction to get a
smaller, more compact evaluation framework. During this reduction several
correlated ISO controls were combined. In the end, there was an evaluation
framework with 52 questions.

In the validation sessions with external organizations, the questions are divided
over three functions (CSO, head of IT, head of software development). All
organizations did not encountered problems with the division. In both external
organizations, there was one person that could answer both the CSO part and the
head of IT part. However, SIG does have separate employees for the two parts.
Further it was noticed that these functions are connected with specific ISO
chapters. This is very common, because in the ISO standard a specific type of
controls are grouped together. These groups are associated with some tasks and
processes, which are again associated with a specific function.

It was noticed that the questionnaire is more easily understandable to people
who are familiar with the ISO 27002 standards, but other people should be able
to understand the questions.

56

At the start of the research process, it was stated that this evaluation framework
is especially for software development organizations and organizations that
heavily rely on software. When looking back to this statement, it still holds that
the full evaluation framework is suited for this group. However, if the
questionnaire part about the software development is left out, it could also be
used for other type of organizations. This part about software development is
around 15% of the full questionnaire.

Section 2.3 described four other evaluation frameworks. These evaluation
frameworks have some similarities with the designed framework, for instance:
the new evaluation framework uses questions and three of the other frameworks
also use questions. They however also have big differences with the new
evaluation framework:

All frameworks except the new evaluation framework are based on an older ISO
27002:2005 version.

Furthermore the new evaluation framework provides a framework that is
lightweight. Three of the four frameworks ([34], [37] and [43]) are heavyweight.
Only the approach of Karabacak [41] is lightweight, but that approach requires
that multiple persons answer each question of the questionnaire. This means
that the approach of Karabacak is also more time-‐consuming than the used
approach in this master thesis.

Another difference is that this master thesis’ approach show how good or bad
implementation of security processes in an organization is in comparison with
other organizations (market conformance). All other approaches do not compare
the results of organizations with each other, but they only check the level of
implementation.

Future work can be done on this research in several topics.
• Refining the rating system based on more datasets than the used three
datasets (SIG and two external organizations). The star rating has to be
spread well (e.g. not all organizations have 4 stars). In case it is not spread
well than the rating system has to be adjusted to allow for better
comparison with other companies. This could be done by using other
percentages in the transition table or moving answers to another
maturity level.
• Creating a tool that automatically calculates the star rating based on the
answers of organizations. Possibly, the tool could give some suggestions
of actions to improve. In a more advanced tool, the refinement of the
rating system (mentioned in the previous point of future work) could be
done automatically. This tool is a ‘nice-‐to-‐have’, because it helps reducing
the analysis time.
• A research is required to investigate how trustworthy the evaluation
framework and the results are.

57

References
[1] PricewaterhouseCoopers (2008). Information security breaches survey
2008: executive summary. Retrieved from
http://www.berr.gov.uk/files/file45713.pdf
[2] PricewaterhouseCoopers (2013). Information security breaches survey
2013: executive summary. Retrieved from
http://www.pwc.co.uk/assets/pdf/cyber-‐security-‐2013-‐exec-‐
summary.pdf
[3] Reuijl, A.; Koers, M.; Paans, R.; van der Veer, R.; Roukens, R.; Kok, C.;
Breeman, J. (2014) Grip op SSD Het proces. Retrieved from
http://www.cip-‐overheid.nl/wp-‐content/uploads/2014/04/Grip-‐op-‐
SSD-‐Het-‐proces-‐v1-‐03.pdf
[4] Altena, J. (2012). ISO/IEC 27002 baseline selection. Master thesis,
Nijmegen: Radboud University.
[5] German Bundesamt für Sicherheit in der Informationstechnik. (2007). IT
security guidelines. IT Security Management and IT-‐Grundschultz.
[6] Huijben, K. (2013). From ISO 27001 towards a flexible and light-‐weight
security evaluation framework. Research B project, Nijmegen: Radboud
University.
[7] McGraw, G., Migues, S., & West, J. (2013). Build Security In Maturity Model
V.
[8] Xu, H. (2013). ISO 27k controls & security process model. Not published.
[9] International Organization of Standardization (2013). Information
technology – security techniques – information security management
systems – overview and vocabulary. ISO 27000:2013
systems – requirements. ISO 27001:2013
technology – security techniques – code of practice for information security
controls. ISO 27002:2013
systems – requirements. ISO 27001:2005
technology – security techniques – code of practice for information security
controls. ISO 27002:2005
technology — Security techniques — Information security management
system implementation guidance. ISO 27003:2010
technology — Security techniques — Information security management —
Measurement. ISO 27004:2009
technology — Security techniques — Information security risk
management. ISO 27005:2008
58

technology — Security techniques — Requirements for bodies providing
audit and certification of information security management systems. ISO
27006:2007
[18] Lineman, D. (2013). ISO 27002:2013 Change Summary Heatmap.
Retrieved from http://www.informationshield.com/security-‐
policy/2013/11/iso-‐270022013-‐change-‐summary-‐heatmap/
[19] PWC (2013). New releases of ISO 27001:2013 and ISO 27002:2013.
Retrieved from
http://www.pwc.com.cy/en/publications/assets/iso27001-‐27002-‐
2013.pdf
[20] Deloitte (2013). ISAE 3402 and SSAE 16 (replacing SAS 70):
Reinforcing confidence through demonstration of effective controls.
Retrieved from http://www.deloitte.com/assets/Dcom-‐
Luxembourg/Local%20Assets/Documents/Brochures/English/2011/lu_en_
isae3402-‐ssae16_09092011.pdf
[21] National Institute of Standards and Technology. (2011) Managing
Information Security Risk. Special publication 800-‐39.
[22] National Institute of Standards and Technology. (2004) Standards
for Security Categorization of Federal Information and Information
Systems. Federal information process standards publication 199.
[23] National Institute of Standards and Technology. (2008) Volume
I: Guide for Mapping Types of Information and Information Systems to
Security Categories. Special publication 800-‐60.
[24] National Institute of Standards and Technology. (2008) Volume II:
Appendices to Guide for Mapping Types of Information and Information
Systems to Security Categories. Special publication 800-‐60.
[25] National Institute of Standards and Technology. (2006) Minimum
Security Requirements for Federal Information and Information Systems.
Federal information process standards publication 200.
[26] National Institute of Standards and Technology. (2013) Security
and Privacy Controls for Federal Information Systems and Organizations.
Special publication 800-‐53.
[27] National Institute of Standards and Technology. (2011) National
Checklist Program for IT Products—Guidelines for Checklist Users and
Developers. Special publication 800-‐70.
[28] National Institute of Standards and Technology. (2010) Guide for
Assessing the Security Controls in Federal Information Systems and
Organizations. Special publication 800-‐53A.
[29] National Institute of Standards and Technology. (2010) Guide for
Applying the Risk Management Framework to Federal Information Systems.
Special publication 800-‐37.
[30] National Institute of Standards and Technology. (2011)
Information Security Continuous Monitoring (ISCM) for Federal
Information Systems and Organizations. Special publication 800-‐137.
[31] British Standards Institution. (2013) Cyber security risk –
Governance and management – Specification. PAS 555:2013.
[32] Chaplin, M.; Creasey, J. (2011) The 2011 Standard of Good Practices
for Information Security. Information Security Forum Limited
59

[33] Department for Business Innovation & Skills (2012) 10 steps to
cyber security guidance sheets. Retrieved from
http://www.bis.gov.uk/assets/biscore/business-‐sectors/docs/0-‐9/12-‐
1121-‐10-‐steps-‐to-‐cyber-‐security-‐advice-‐sheets
[34] Wright, S. (2006) Measuring the effectiveness of security using ISO
27001, Whitepaper. SANS institute. Retrieved from
http://www.iwar.org.uk/comsec/resources/iso-‐27001/measuring-‐
effectiveness.pdf
[35] Information Systems Audit and Control Association (2012) COBIT
5 framework.
[36] Information Systems Audit and Control Association (2012) COBIT
5 for information security
[37] Bandopadhyay, S.; Sengupta, A.; Mazumdar, C. (2011) A
quantitative methodology for information security control gap analysis.
Proceedings of the 2011 International Conference on Communication,
Computing & Security. Pages 537-‐540
[38] Järveläinen, J. (2013) IT incidents and business impacts: validating
a framework for continuity management in information systems.
International Journal of Information Management volume 33, issue 3.
Pages 583-‐590.
[39] Breier, J.; Hudec, L. (2012). Towards a security evaluation model
based on security metrics. International conference on computer systems
and technologies ‘ 12.
[40] IT Governance Institute (2008) Aligning COBIT 4.1, ITIL V3 and
ISO/IEC 27002 for business benefit. Retrieved from
http://www.isaca.org/Knowledge-‐
Center/Research/Documents/Aligning-‐COBIT-‐ITIL-‐V3-‐ISO27002-‐for-‐
Business-‐Benefit_res_Eng_1108.pdf
[41] Karabacak, B.; Sogukpinar, I. (2006) A quantitative method for ISO
17799 gap analysis. Computer & Security. Volume 25, Issue 6. Pages 413-‐
419
[42] Software Improvement Group (2013). How secure is your
software?. Whitepaper -‐
www.sig.eu/blobs/Whitepapers/SIG_whitepaper_secure_software.pdf
[43] Praxiom ISO IEC 27001 2005 Information Security Gap Analysis
Tool. Retrieved from http://www.praxiom.com/iso-‐27001-‐gap.htm
[44] Xu, H. (2014). Input research meeting at SIG
[45] International Organization of Standardization (2011). Systems and
software engineering – Systems and software Quality Requirements and
Evaluation (SQuaRE) – System and software quality models. ISO
25010:2011
[46] Heitlager, I.; Kuipers, T.; Visser, J. (2007) A practical model for
measuring maintainability. 6th International Conference on the Quality of
Information and Communications Technology (QUATIC 2007). Pages. 30–
39.

60

Appendix
A: Differences of ISO 27002:2005 and ISO 27002:2013
The following table gives an overview of the differences in the ISO 27002:2005
controls and the ISO 27002:2013 controls. The corresponding ISO controls can
be found in the same row. Each control has a specific color:
• Red: A removed ISO control
• Blue: A added ISO control
• Green: Almost the same ISO control, but only renamed
• Grey: Almost the same ISO control (not renamed)

Remark: almost all ISO controls had some small to large update in the new ISO
standard (2013)

ISO 27001:2005 ISO 27001:2013
ID Name ID Name
A.5.1.1 Information security policy document 5.1.1 Policies for information security
A.5.1.2 Review of the information security policy 5.1.2 Review of the policies for information security
A.6.1.1 Management commitment to information
security
A.6.1.2 Information security co-‐ ordination
A.6.1.3 ’ Allocation of information security
6.1.1
Information security roles and responsibilities
responsibilities
A.8.1.1 ‘ Roles and responsibilities
A.6.1.4 Authorization process for information
processing facilities
A.10.1.3 Segregation of duties
6.1.2
Segregation of duties
A.6.1.6 Contact with authorities 6.1.3 Contact with authorities
A.6.1.7 Contact with special interest groups 6.1.4 Contact with special interest groups
6.1.5 Information security in project management
A.11.7.1 Mobile computing and communications 6.2.1 Mobile device policy
A.11.7.2 Teleworking 6.2.2 Teleworking
A.8.1.2 Screening 7.1.1 Screening
A.8.1.3 Terms and conditions of employment 7.1.2 Terms and conditions of employment
A.8.2.1 Management responsibilities 7.2.1 Management responsibilities
A.8.2.2 Information security awareness, education 7.2.2 Information security awareness, education and
and training training
A.8.2.3 Disciplinary process 7.2.3 Disciplinary process
A.8.3.1 Termination responsibilities 7.3.1 Termination or change of employment
responsibilities
A.7.1.1 Inventory of assets 8.1.1 Inventory of assets
A.7.1.2 Ownership of assets 8.1.2 Ownership of assets
A.7.1.3 Acceptable use of assets 8.1.3 Acceptable use of assets
A.8.3.2 Return of assets 8.1.4 Return of assets
A.7.2.1 Classification guidelines 8.2.1 Classification of information
A.7.2.2 Information labeling and handling 8.2.2 Labeling of information
8.2.3 Handling of assets

A.10.7.1
Management of removable media 8.3.1 Management of removable media
A.10.7.3 Information handling procedures
A.10.7.2 Disposal of media 8.3.2 Disposal of media
A.10.8.3 Physical media in transit 8.3.3 Physical media transfer
61

ISO 27001:2005 ISO 27001:2013
ID Name ID Name
A.10.7.4 Security of system documentation
A.11.1.1 Access control policy 9.1.1 Access control policy
9.1.2 Access to networks and network services

A.11.2.1
User registration 9.2.1 User registration and de-‐registration
9.2.2 User access provisioning

A.11.2.2
Privilege management 9.2.3 Management of privileged access rights
9.2.4 Management of secret authentication
information of users
A.11.2.4 Review of user access rights 9.2.5 Review of user access rights
A.8.3.3 Removal of access rights 9.2.6 Removal or adjustment of access rights
A.11.3.1 Password use 9.3.1 Use of secret authentication information
A.11.6.1 Information access restriction 9.4.1 Information access restriction
A.11.5.1 Secure log-‐on procedures 9.4.2 Secure logon procedures
A.11.5.2 User identification and authentication
A.11.5.3 Password management system 9.4.3 Password management system
A.11.5.4 Use of system utilities 9.4.4 Use of privileged utility programs
A.12.4.3 Access control to program source code 9.4.5 Access control to program source code
A.11.5.5 Session time-‐out
A.11.5.6 Limitation of connection time
A.11.6.2 Sensitive system isolation
A.12.3.1 Policy on the use of cryptographic controls
10.1.1
Policy on the use of cryptographic controls
A.12.3.2 Key management 10.1.2 Key management
A.9.1.1 Physical security perimeter 11.1.1 Physical security perimeter
A.9.1.2 Physical entry controls 11.1.2 Physical entry controls
A.9.1.3 Securing offices, rooms and facilities 11.1.3 Securing offices, rooms and facilities
A.9.1.4 Protecting against external and environmental 11.1.4 Protecting against external and environmental
threats threats
A.9.1.5 Working in secure areas 11.1.5 Working in secure areas
A.9.1.6 Public access, delivery and loading areas 11.1.6 Delivery and loading areas
A.9.2.1 Equipment sitting and protection 11.2.1 Equipment siting and protection
A.9.2.2 Supporting utilities 11.2.2 Supporting utilities
A.9.2.3 Cabling security 11.2.3 Cabling security
A.9.2.4 Equipment maintenance 11.2.4 Equipment maintenance
A.9.2.7 Removal of property 11.2.5 Removal of assets
A.9.2.5 Security of equipment off-‐ premises 11.2.6 Security of equipment and assets off premises
A.9.2.6 Secure disposal or re-‐use of equipment 11.2.7 Secure disposal or re-‐use of equipment
A.11.3.2 Unattended user equipment 11.2.8 Unattended user equipment
A.11.3.3 Clear desk and clear screen policy 11.2.9 Clear desk and clear screen policy
A.10.1.1 Documented operating procedures 12.1.1 Documented operating procedures
A.10.1.2 Change management 12.1.2 Change management
A.10.3.1 Capacity management 12.1.3 Capacity management
A.10.1.4 Separation of development, test and 12.1.4 Separation of development, testing and
operational facilities operational environments
A.10.4.1 Controls against malicious code 12.2.1 Controls against malware
‘’
A.10.4.2 Controls against mobile code
‘’
A.10.5.1 Information back-‐up 12.3.1 Information backup
A.10.10.1 Audit logging 12.4.1 Event logging
‘’’
A.10.10.2 Monitoring system use
‘’’
A.10.10.3 Protection of log information
12.4.2
Protection of log information
62

ISO 27001:2005 ISO 27001:2013
ID Name ID Name
A.10.10.4 Administrator and operator logs 12.4.3 Administrator and operator logs
A.10.10.5 Fault logging
A.10.10.6 Clock synchronization
12.4.4
Clock synchronization
A.12.4.1 Control of operational software 12.5.1 Installation of software on operational systems
A.12.6.1 Control of technical vulnerabilities 12.6.1 Management of technical vulnerabilities
12.6.2 Restrictions on software installation

A.15.3.1
Information systems audit controls 12.7.1 Information systems audits controls
A.15.3.2 Protection of information systems audit tools

13.1.1 Network controls
13.1.2 Security of network services

A.11.4.5
Segregation in networks 13.1.3 Segregation in networks
A.11.4.1 Policy on use of network services
A.11.4.2 User authentication for external connections
A.11.4.3 Equipment identification in networks
A.11.4.4 Remote diagnostic and configuration port
protection
A.11.4.6 Network connection control
A.11.4.7 Network routing control
A.10.8.1 Information exchange policies and procedures
13.2.1
Information transfer policies and procedures
A.10.8.2 Exchange agreements 13.2.2 Agreements on information transfer
A.10.8.4 Electronic messaging 13.2.3 Electronic messaging
A.10.8.5 Business information systems
A.6.1.5 Confidentiality agreements
13.2.4
Confidentiality or non-‐disclosure agreements
A.12.1.1 Security requirements analysis and 14.1.1 Information security requirements analysis and
specification specification
14.1.2 Securing application services on public networks
14.1.3 Protecting application services transactions

A.12.2.1
Input data validation
A.12.2.2 Control of internal processing
A.12.2.3 Message integrity
A.12.2.4 Output data validation

14.2.1
Secure development policy

A.12.5.1
Change control procedures 14.2.2 System change control procedures
A.12.5.2 Technical review of applications after 14.2.3 Technical review of applications after operating
operating system changes platform changes
A.12.5.3 Restrictions on changes to software packages 14.2.4 Restrictions on changes to software packages
14.2.5 Secure system engineering principles
14.2.6 Secure development environment

A.12.5.4
Information leakage
A.12.5.5 Outsourced software development 14.2.7 Outsources development
14.2.8 System security testing
14.2.9 System acceptance testing

A.12.4.2
Protection of system test data 14.3.1 Protection of test data
15.1.1 Information security policy for supplier
relationships

A.6.2.1
Identification of risks related to external
parties
A.6.2.2 Addressing security when dealing with
customers
A.6.2.3 Addressing security in third party agreements 15.1.2 Addressing security within supplier agreements
15.1.3 Information and communication technology
supply chain

A.10.2.1
Service delivery

63

ISO 27001:2005 ISO 27001:2013
ID Name ID Name
A.10.2.2 Monitoring and review of third party services 15.2.1 Monitoring and review of supplier services
A.10.2.3 Managing changes to third party services 15.2.2 Managing changes to supplier services
A.13.2.1 Responsibilities and procedures 16.1.1 Responsibilities and procedures
A.13.1.1 Reporting information security events 16.1.2 Reporting information security events
A.13.1.2 Reporting security weaknesses 16.1.3 Reporting information security weaknesses
16.1.4 Assessment of and decision on information
security events
16.1.5 Response to information security incidents

A.13.2.2
Learning from information security incidents 16.1.6 Learning from information security incidents
A.13.2.3 Collection of evidence 16.1.7 Collection of evidence
17.1.1 Planning information security continuity
17.1.2 Implementing information security continuity

A.14.1.1
Including information security in the business
continuity management process
A.14.1.2 Business continuity and risk assessment
A.14.1.3 Developing and implementing continuity
plans including information security
A.14.1.4 Business continuity planning framework
A.14.1.5 Testing, maintaining and re-‐ assessing 17.1.3 Verify, review and evaluate information security
business continuity plans continuity
17.2.1 Availability of information processing facilities
A.15.1.1 Identification of applicable legislation 18.1.1 Identification of applicable legislation and
contractual requirements
A.15.1.2 Intellectual property rights (IPR) 18.1.2 Intellectual property rights
A.15.1.3 Protection of organizational records 18.1.3 Protection of records
A.15.1.4 Data protection and privacy of personal 18.1.4 Privacy and protection of personally identifiable
information information
A.15.1.5 Prevention of misuse of information
processing facilities
A.15.1.6 Regulation of cryptographic controls
18.1.5
Regulation of cryptographic controls
A.6.1.8 Independent review of information security 18.2.1 Independent review of information security
A.15.2.1 Compliance with security policies and 18.2.2 Compliance with security policies and standards
standards
A.15.2.2 Technical compliance checking 18.2.3 Technical compliance review

A.10.3.2
System acceptance
A.10.6.1 Network controls
A.10.6.2 Security of network services
A.10.9.1 Electronic commerce
A.10.9.2 On-‐line transactions
A.10.9.3 Publicly available information
A.11.2.3 User password management

‘ : ISO controls A.6.1.3 and A.8.1.1 are combined to one ‘new’ ISO control 6.1.1
‘’: ISO controls A.10.4.1 and A.10.4.2 are combined to one ‘new’ ISO control 12.2.1
‘’’: ISO controls A.10.10.1 and A.10.10.2 are combined to one ‘ new’ ISO control 12.4.1

64

B: Detailed transition table
Rating Level 1 Level 2 Level 3 Level 4
0.5 30 0 0 0
0.6 33 3.5 1.5 0.5
0.7 36 7 3 1
0.8 39 10.5 4.5 1.5
0.9 42 14 6 2
1 45 17.5 7.5 2.5
1.1 48 21 9 3
1.2 51 24.5 10.5 3.5
1.3 54 28 12 4
1.4 57 31.5 13.5 4.5
1.5 60 35 15 5
1.6 62.5 37 16.5 5.5
1.7 65 39 18 6
1.8 67.5 41 19.5 6.5
1.9 70 43 21 7
2 72.5 45 22.5 7.5
2.1 75 47 24 8
2.2 77.5 49 25.5 8.5
2.3 80 51 27 9
2.4 82.5 53 28.5 9.5
2.5 85 55 30 10
2.6 85.5 57.5 32 10.5
2.7 86 60 34 11
2.8 86.5 62.5 36 11.5
2.9 87 65 38 12
3 87.5 67.5 40 12.5
3.1 88 70 42 13
3.2 88.5 72.5 44 13.5
3.3 89 75 46 14
3.4 89.5 77.5 48 14.5
3.5 90 80 50 15
3.6 91 81.5 51 15.5
3.7 92 83 52 16
3.8 93 84.5 53 16.5
3.9 94 86 54 17
4 95 87.5 55 17.5
4.1 96 89 56 18
4.2 97 90.5 57 18.5
4.3 98 92 58 19
4.4 99 93.5 59 19.5
4.5 100 95 60 20
4.6 100 95.5 64 28
4.7 100 96 68 36
65

4.8 100 96.5 72 44
4.9 100 97 76 52
5 100 97.5 80 60
5.1 100 98 84 68
5.2 100 98.5 88 76
5.3 100 99 92 84
5.4 100 99.5 96 92
5.5 100 100 100 100

66

Iso

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Iso

Enviado por

Direitos autorais:

Formatos disponíveis

2014

A lightweight, flexible evaluation

1.1 Problem statement

The ten steps Large organizations Small businesses

1.2 Research definition

• Input: Information of organization

• Input: Answers of questionnaire

Common modeling & testing & intrusion

are in handling their organizational processes secure.

1.5 Previous work

2.1.1 ISO 27000 series

This PDCA model contains four phases:

2.1.1.2 ISO 27002:2013

ISO chapter 5. Information security policies

2.1.1.3 ISO/IEC 27004:2009

2.1.2 NIST Special Publication 800

Family Access Control

Area CF1 Security Policy and Organization

2.1.4 10 steps to cyber security

Governance Intelligence SSDL Touchpoints Deployment

BSIMM domain Governance

Sec. Features &

2.1.7 PAS 555:2013

2.1.8 SAS70 and its successors

U.S. standard: SSAE16 International standard: ISAE

2.2 Comparison of standards

ISO 27002 compliance

Number of pages*

2.3 Evaluation frameworks for ISO 27002

2.3.3 Approach of Bandopadhyay

2.3.4 Approach of Praxiom

2.4 Comparison of evaluation frameworks

3.1 Phase 1: the trial questionnaire

ISO control 9.2.1 User registration and de-­‐registration

ISO controls 6.1.3 Contact with authorities

3.2 Phase 2: the full questionnaire

3.2.2 Validation inside SIG

Useful Easy to answer

Useful Easy to answer

3.3 Phase 3: the improved questionnaire

3.3.2 Validation with external organizations

4.1 The SIG star rating

Stars Needed answers

Level 1 Level 2 Level 3 Level 4

SIG star rating Level 1 Level 2 Level 3 Level 4

Organizat-­‐ Level 1 Level 2 Level 3 Level 4 SIG star rating

Data Rate Level 1 Level 2 Level 3 Level 4

4.1.2 The risk approach

4.2 The actions for improving the security

Você também pode gostar

•  Input: Information of organization

•  Input: Answers of questionnaire

ISO control 9.2.1 User registration and de-‐registration

Organizat-‐ Level 1 Level 2 Level 3 Level 4 SIG star rating