IS AUDITING GUIDELINE

POST-IMPLEMENTATION REVIEW
DOCUMENT G29

The specialised nature of information systems (IS) auditing and the skills necessary to perform such audits require standards that apply
specifically to IS auditing. One of the goals of the Information Systems Audit and Control Association (ISACA) is to advance globally
applicable standards to meet its vision. The development and dissemination of the IS Auditing Standards are a cornerstone of the ISACA
professional contribution to the audit community. The framework for the IS Auditing Standards provides multiple levels of guidance:
„ Standards define mandatory requirements for IS auditing and reporting. They inform:
– IS auditors of the minimum level of acceptable performance required to meet the professional responsibilities set out in the ISACA
Code of Professional Ethics
– Management and other interested parties of the profession’s expectations concerning the work of practitioners
® ®
– Holders of the Certified Information Systems Auditor (CISA ) designation of requirements. Failure to comply with these standards
may result in an investigation into the CISA holder’s conduct by the ISACA Board of Directors or appropriate ISACA committee
and, ultimately, in disciplinary action.
„ Guidelines provide guidance in applying IS Auditing Standards. The IS auditor should consider them in determining how to achieve
implementation of the standards, use professional judgement in their application and be prepared to justify any departure. The objective
of the IS Auditing Guidelines is to provide further information on how to comply with the IS Auditing Standards.
„ Procedures provide examples of procedures an IS auditor might follow in an audit engagement. The procedure documents provide
information on how to meet the standards when performing IS auditing work, but do not set requirements. The objective of the IS
Auditing Procedures is to provide further information on how to comply with the IS Auditing Standards.

COBIT resources should be used as a source of best practice guidance. The COBIT Framework states, "It is management's responsibility to
safeguard all the assets of the enterprise. To discharge this responsibility as well as to achieve its expectations, management must establish
an adequate system of internal control." COBIT provides a detailed set of controls and control techniques for the information systems
management environment. Selection of the most relevant material in COBIT applicable to the scope of the particular audit is based on the
choice of specific COBIT IT processes and consideration of COBIT information criteria.

As defined in the COBIT Framework, each of the following is organised by IT management process. COBIT is intended for use by business
and IT management, as well as IS auditors; therefore, its usage enables the understanding of business objectives, communication of best
practices and recommendations to be made around a commonly understood and well-respected standard reference. COBIT includes:
„ Control objectives—High-level and detailed generic statements of minimum good control
„ Control practices—Practical rationales and “how to implement” guidance for the control objectives
„ Audit guidelines—Guidance for each control area on how to obtain an understanding, evaluate each control, assess compliance and
substantiate the risk of controls not being met
„ Management guidelines—Guidance on how to assess and improve IT process performance, using maturity models, metrics and critical
success factors. They provide a management-oriented framework for continuous and proactive control self-assessment specifically
focused on:
– Performance measurement—How well is the IT function supporting business requirements? Management guidelines can be used
to support self-assessment workshops, and they also can be used to support the implementation by management of continuous
monitoring and improvement procedures as part of an IT governance scheme.
– IT control profiling—What IT processes are important? What are the critical success factors for control?
– Awareness—What are the risks of not achieving the objectives?
– Benchmarking—What do others do? How can results be measured and compared? Management guidelines provide example
metrics enabling assessment of IT performance in business terms. The key goal indicators identify and measure outcomes of IT
processes, and the key performance indicators assess how well the processes are performing by measuring the enablers of the
process. Maturity models and maturity attributes provide for capability assessments and benchmarking, helping management to
measure control capability and to identify control gaps and strategies for improvement.

Glossary of terms can be found on the ISACA web site at www.isaca.org/glossary. The words audit and review are used interchangeably.

Disclaimer: ISACA has designed this guidance as the minimum level of acceptable performance required to meet the professional
responsibilities set out in the ISACA Code of Professional Ethics. ISACA makes no claim that use of this product will assure a successful
outcome. The publication should not be considered inclusive of any proper procedures and tests or exclusive of other procedures and tests
that are reasonably directed to obtaining the same results. In determining the propriety of any specific procedure or test, the controls
professional should apply his/her own professional judgement to the specific control circumstances presented by the particular systems or
information technology environment.

The ISACA Standards Board is committed to wide consultation in the preparation of the IS Auditing Standards, Guidelines and Procedures.
Prior to issuing any documents, the Standards Board issues exposure drafts internationally for general public comment. The Standards
Board also seeks out those with a special expertise or interest in the topic under consideration for consultation where necessary. The
Standards Board has an ongoing development programme and welcomes the input of ISACA members and other interested parties to
identify emerging issues requiring new standards. Any suggestions should be e-mailed (standards@isaca.org), faxed (+1.847. 253.1443) or
mailed (address at the end of document) to ISACA International Headquarters, for the attention of the director of research standards and
academic relations. This material was issued 15 October 2004.
 1. BACKGROUND

1.1 Linkage to Standards

1.1.1 Standard S6 Performance of Audit Work states, “During the course of the audit, the IS auditor should obtain sufficient,
reliable and relevant evidence to achieve the audit objectives. The audit findings and conclusions are to be supported by
appropriate analysis and interpretation of this evidence."
1.1.2 Standard S8 Follow-up Activities states, “After the reporting of findings and recommendations, the IS auditor should
request and evaluate relevant information to conclude whether appropriate action has been taken by management in a
timely manner.”

1.2 Linkage to COBIT

1.2.1 High-level control objective M4, Provide for Independent Audit, states, “Control over the IT process of providing for
independent audit that satisfies the business requirement to increase confidence levels and benefit from best practice
advice is enabled by independent audits carried out at regular intervals and takes into consideration:
„ Audit independence
„ Proactive audit involvement
„ Performance of audits by qualified personnel
„ Clearance of findings and recommendations
„ Follow-up activities
„ Impact assessments of audit recommendations (costs, benefits and risks)
1.2.2 Detailed control objective M4.6, Performance of Audit Work, states, “Audits should be appropriately supervised to provide
assurance that audit objectives are achieved and applicable professional auditing standards are met. Auditors should
ensure that they obtain sufficient, reliable, relevant and useful evidence to achieve the audit objectives effectively. The
audit findings and conclusions should be supported by appropriate analysis and interpretation of the evidence.”

1.3 Reference to COBIT

1.3.1 The COBIT references offer the specific objectives or processes of COBIT to consider when reviewing the area addressed
by this guidance. Selection of the most relevant material in COBIT applicable to the scope of the particular audit is based
on the choice of specific COBIT IT processes and consideration of COBIT’s information criteria.
1.3.2 In a post-implementation review, the first review after the implementation of an IT solution, the following processes are
more relevant:
„ PO2—Define the Information Architecture
„ PO4—Define the IT organisation and relationship
„ PO5—Manage the IT investment
„ PO8—Ensure Compliance with External Requirements
„ PO9—Assess risks
„ PO10—Manage projects
„ PO11—Manage quality
„ AI1—Identify automated solutions
„ AI2—Acquire and maintain application software
„ AI3—Acquire and maintain technology infrastructure
„ AI5—Install and accredit systems
„ AI6—Manage changes
„ DS7—Educate and Train Users
„ DS11—Manage Data
„ M1—Monitor the processes
„ M2—Assess Internal Control Adequacy
„ M3—Obtain Independent Assurance
„ M4—Provide for Independent Audit
1.3.3 The information criteria most relevant to the post-implementation review are:
„ Primary—Effectiveness and efficiency
„ Secondary—Availability, compliance, confidentiality, reliability and integrity
1.3.4 International Federation of Accountants (IFAC) Information Technology Committee (ITC) Guidelines include:
„ Implementation of Information Technology Solutions
„ Managing Information Technology Planning For Business Impact

1.4 Purpose of the Guideline

1.4.1 The purpose of this guideline is to describe the recommended practices in carrying out the post-implementation review of
information technology solutions, so that the relevant standards for information systems auditing are complied with during
the course of the review.
1.4.2 Organisations implement various IT solutions to meet their business requirements. Once the solutions are implemented,
post-implementation reviews are generally carried out by IS auditors to assess the effectiveness and efficiency of the IT
solutions and their implementation, initiate actions to improve the solution (where necessary) and serve as a learning tool
for the future.
1.4.3 Certain practices recommended in this guideline may also be appropriate for reviews of projects where implementations
are unsuccessful or aborted prior to implementation.
1.4.4 This guideline provides guidance in applying IS Auditing Standards S6 Performance of Audit Work and S8 Follow-up
Activities while conducting a post-implementation review. The IS auditor should consider it in determining how to achieve
implementation of these standards, use professional judgment in its application and be prepared to justify any departure.

Page 2 of 6 IS Auditing Guideline Post-implementation Review

1.5 Guideline Application
1.5.1 When applying this guideline, the IS auditor should consider its guidance in relation to other relevant ISACA guidelines

1.6 Definition and General Coverage

1.6.1 Post-implementation review, for the purpose of this guideline, means the first or subsequent review of an IT solution and/or
the process of its implementation, performed after its implementation, to assess any or all of the following:
„ Whether the intended objectives of the solution are realised
„ Whether actual costs and benefits are compared against budget
„ The effectiveness and appropriateness of the implementation process
„ Causes of time and/or cost overruns, and quality and/or performance issues, if any
„ Productivity and performance improvements resulting from the solution
„ Whether business process and internal controls are implemented
„ Whether user access controls are implemented in accordance with organisational policy
„ Whether users have been appropriately trained
„ Whether the system is maintainable and can be further developed effectively and efficiently
„ Whether available features and procedures, as relevant, have been implemented
„ Compliance with relevant statutory requirements and organisational policies
„ Compliance with COBIT Control Objectives and COBIT Management Guidelines, as relevant
„ Opportunities for further improvement in either the solution or implementation process
1.6.2 The objectives of a post-implementation review might include:
„ Ensure that the intended objectives of implementing the IT solution are met and aligned to meet the business
objectives of the organisation
„ Evaluate the adequacy of procedures and controls over input, processing and output to ensure that information
captured is complete and accurate, information processing complies with required business rules, and information
generated is accurate, reliable and timely
„ Evaluate the adequacy of procedures and controls over the maintenance and monitoring of the management trail
produced by the IT solution
„ Verify the accuracy of financial and management reports generated by the IT solution
„ Ensure the adequacy of application-level access control enforced by the IT solution
„ Verify the adequacy of availability features inherent in the IT solution to recover from any unexpected shutdowns and
maintain data integrity
„ Ensure that the IT solution can be supported and maintained efficiently and effectively in the absence of the specific
personnel responsible for its development and implementation
„ Identify potential risks and weakness in controls, as well as provide solutions to mitigate risks and strengthen
controls
1.6.3 The post-implementation review essentially seeks to determine whether the investment in the IT solution was worthwhile
(as determined and measurable by the organisation) and whether the delivered IT solution can be adequately managed
and controlled. These investment returns can be covered as a unique, separate review often called a benefits realisation
review (section 8.1). The scope of a post-implementation review should consider:
„ The nature of the IT solution
„ The intended usage of the IT solution (for what purpose, by whom, when and where)
„ The criticality of the IT solution in achieving business objectives
„ The scope of the review agreed with the auditee (organisation) management
„ Whether the IT solution was subject to audit review during the initiation, development and testing stage
„ Where there has been any non-audit involvement of IS auditors during the project implementation

2. AUDIT CHARTER

2.1 Mandate
2.1.1 Before commencing a post-implementation review, the IS auditor should have the requisite mandate to carry out the
review. Where the review is initiated by a third party, the IS auditor should obtain reasonable assurance that the third party
has the appropriate authority to commission the review.

3. INDEPENDENCE

3.1 Professional Objectivity

3.1.1 Before accepting the assignment, the IS auditor should provide reasonable assurance that his/her interest, if any, in the IT
solution that is the subject of the post-implementation review will not impair the objectivity of the review. Any possible
conflict of interest should be communicated explicitly to the organisation, and if possible, a written statement of the
organisation’s awareness of the conflict should be obtained before accepting the assignment.
3.1.2 Where the IS auditor had any non-audit roles in the implementation of the IT solution being reviewed, the IS auditor
should consider guideline G17 Effect of Nonaudit Roles on the IS Auditor’s Independence which provides guidance.

4. PROFESSIONAL ETHICS AND STANDARDS

4.1 Pre- and Post-implementation Reviews

IS Auditing Guideline Post-implementation Review Page 3 of 6

 4.1.1 As compared to a pre-implementation review, post-implementation review is ordinarily performed when the IT solution has
been in operation for a reasonable period (ordinarily a number of months or process cycles) and user procedures, as well
as application-level securities, have been implemented.
4.1.2 Pre-implementation reviews examine the conceptual design of controls and management trails, or how they operate in
test environments. Post-implementation reviews examine how controls and management trails are operating once the IT
solution is installed, configured and operating in the production environment. Where a pre-implementation review has
been performed satisfactorily, the IS auditor should use his/her discretion whether to limit the post-implementation review
to examine actual operation of the system in production.
4.1.3 It is preferable to perform both pre-implementation and post-implementation reviews, if resources are available,
4.1.4 When carrying out a post-implementation review the IS auditor should provide reasonable assurance that the project
owner responsible for implementing the IT solution and the project team is involved in the review process. Team members
consulted as part of the review should typically include:
„ People connected with the design, development and deployment of the IT solution
„ People with working knowledge of the area under review, and current and proposed business processes
„ People with relevant technical knowledge
„ People with knowledge of the organisation’s business strategy and the proposed contribution of the IT solution to the
achievement of the strategy
„ People involved in the measurement and monitoring of the benefits realisation process

5. COMPETENCE

5.1 Skills and Knowledge

5.1.1 The IS auditor also should provide reasonable assurance that he/she possess the relevant skills and knowledge to carry
out the post-implementation review of the IT solution. Where expert input is necessary, appropriate input should be
obtained

6. PLANNING

6.1 Scope and Objectives of the Review

6.1.1 The IS auditor, in consultation with the organisation as appropriate, should clearly define the scope and objectives of the
Post Implementation Review. The aspects to be covered by the review should be stated explicitly as part of the scope.
6.1.2 For the purpose of the review, the stakeholders in the implementation should be identified.
6.1.3 The findings and conclusions of any prior reviews of the IT solution or implementation process—pre-implementation or
concurrent reviews—should be considered in determining the scope and in audit planning.

6.2 Sign-off for the Terms of Reference

6.2.1 Depending on the organisational practices, the IS auditor should obtain the concurrence of the relevant parties in the
organisation for the terms of reference and the approach. If the review is being initiated by a third party, they should also
agree to the terms of reference.

6.3 Approach
6.3.1 The IS auditor should formulate the approach to provide reasonable assurance that the scope and objectives of the review
can be fulfilled in an objective and professional manner. The approach should be appropriately documented. The use of
expert input should be specified as part of the approach. Post-implementation reviews are not limited to the first review
after implementation of the IT solution. Multiple reviews may be performed to identify improvements in the implemented
solution.

7. PERFORMANCE OF AUDIT WORK

7.1 Execution of Post-Implementation Review

7.1.1 A post-implementation review should be scheduled at a reasonable time after the IT solution has been implemented.
Typical periods can range from four weeks to six months, depending upon the type of solution and its environment.
7.1.2 A post-implementation review is intended to be an assessment and review of the final working IT solution. Ideally, there
should have been at least one full implementation and reporting cycle completed to perform a proper review. The review
should not be performed while still dealing with initial issues and teething troubles, or while still training, and educating
users. However, where possible, the review should be performed while the opportunity remains to incorporate final
improvements to derive optimum benefit from the IT solution.
7.1.3 Review procedures should include the study of available documentation (such as business case, business requirements
including business controls, feasibility study, system, operational and user documentation, progress reports, minutes of
meetings, cost/benefit reports, testing and training plans), discussions with stakeholders, hands-on experimentation and
familiarisation with the IT solution, observation and inquiry of business and project personnel, and examination of
operational and control documentation.
7.1.4 Appropriate resources to carry out the post-implementation review should be identified and allocated, and the
performance of the review should be planned in conjunction with relevant auditee personnel.
7.1.5 Agreement should be reached regarding the format, content, audience and timing, where possible, of reporting the results
of the post-implementation review.

Page 4 of 6 IS Auditing Guideline Post-implementation Review

7.1.6 The stated objectives of the IT solution, costs and benefits should be studied in detail. The extent of achievement of the
objectives and actual costs and benefits should be evaluated together with the processes and systems used to capture,
monitor and report performance, costs and benefits. As part of this exercise, the productivity/performance improvements
delivered by the IT solution should also be studied. Suitable measurement criteria should be used in this context. The cost
and/or time overrun, if any, should be analysed by reference to their causes and their effects. Controllable and
uncontrollable causes should be identified separately.
7.1.7 The process followed for defining and implementing the IT solution should be evaluated with reference to its
appropriateness, as well as its effectiveness.
7.1.8 The adequacy and effectiveness of education and training provided to users and staff supporting the IT solution should be
reviewed.
7.1.9 The reports of any prior reviews performed either internally or by external reviewers on a pre-implementation basis or
concurrently with the implementation process should be studied, and the status of recommendations and actions taken
verified.
7.1.10 Since the post-implementation review is examining an IT solution, in general, the IT solution should satisfy appropriate
COBIT control objectives. The extent of compliance with relevant control objectives and the effect of noncompliance should
be analysed and reported. Further, critical success factors, key goal indicators, key performance indicators and maturity
model benchmarks from COBIT Management Guidelines should be adapted as appropriate for the IT solution and
implementation process being reviewed.
7.1.11 Appropriate management trails should be maintained for the data gathered, analysis made, inferences arrived at as well
as corrective actions recommended.
7.1.12 The extent of compliance with statutory and regulatory requirements and organisational policies and standards of the IT
solution and implementation process should be reviewed.
7.1.13 Where appropriate, automated testing tools and CAATs may be used to test relevant aspects of the IT solution.
7.1.14 The review should highlight risks and issues for necessary corrective action, together with opportunities for improvement
in controls or increased effectiveness of the implementation process.
7.1.15 Reported findings, conclusions and recommendations should be based on an objective analysis and interpretation of the
information and evidence obtained during the post-implementation review.

8. BENEFITS REALISATION REVIEWS

8.1 Benefits Realisation Review

8.1.1 All IT projects are actually business projects and should have a business rationale from the outset. Their success or
failure should be measured either in financial terms or as a contribution to achievement of the strategic business plan.
Benefits realisation reviews should focus not only on what has been achieved but what remains to be done. Organisations
that undertake benefits realisation reviews to fine tune best practices and learn lessons reap benefits when their next
project is undertaken.

8.2 Benefits Realisation Review Objectives

8.2.1 The objectives of a benefits realisation review are to evaluate the operational success of the new IT solution, and to
assess the actual costs, benefits and savings in comparison with budgeted amounts. The review may also examine the
effectiveness of the process used to deliver and implement the IT solution. A key consideration is whether or not the
original system objectives and schedules have been achieved. This requires a detailed understanding of as-is and to-be
processes, to assess the extent to which the objectives of the to-be processes have been achieved.
8.2.2 The benefits realisation component of a post-implementation review report should address:
„ Actual costs compared to budgeted costs
„ Actual benefits compared to budgeted benefits
„ Return on investment
„ Actual savings compared to budgeted savings
„ Actual project completion dates compared to planned completion dates
„ Original objectives compared to accomplished objectives
„ Assessment of the adequacy and quality of documentation and controls, including management trails
„ Actual IT solution performance compared to anticipated performance
„ Overall user satisfaction and understanding of the new IT solution system
„ Performance improvement suggestions for future IT solution implementation projects

9. OUTSOURCING

9.1 Outsourcing of IT
9.1.1 Where the organisation has partially or fully delegated some or all of its IT solution implementation to an external provider
of such services (the service provider), the IS auditor should assess the effect of such arrangements and review the
adequacy of, and conformance/compliance with, contracts, agreements and regulations with the service provider.
9.1.2 The IS auditor should obtain an understanding of the nature, timing and extent of the outsourced services. Also, the IS
auditor should establish what controls the service user has put in place to address the business requirements and controls
required by the organisation (refer to guideline G4 Outsourcing of IS Activities to Other Organisations).

10. REPORTING

IS Auditing Guideline Post-implementation Review Page 5 of 6

 10.1 Report Content
10.1.1 The report on the post-implementation review should address the following aspects depending on the objectives and
scope of the review:
„ The scope, objectives, methodology followed and assumptions made
„ Assessment of whether the intended objectives of implementing the IT solution has been met, and whether the IT
Solutions are aligned to meet the business objectives.
„ An overall assessment of the implementation process in terms of key strengths and weaknesses as well as the likely
effects of the weaknesses
„ Recommendations to overcome the significant weaknesses and to improve the implementation process
„ Potential risks and means to mitigate such risks
„ The extent of compliance with COBIT’s information criteria
„ Recommendations to improve future IT solutions and implementation processes
„ Training of the users on the IT solution implemented
„ Acceptance and adaptability of the IT solutions across the organisations
10.1.2 The observations and recommendations should be validated with the stakeholders and organisation (and service provider
if applicable), as appropriate, and responses obtained before finalising the report.

10.2 Weaknesses
10.2.1 Weaknesses identified during the post-implementation review, either due to lack of controls, poor implementation
processes or nonmitigation of associated risks to acceptable levels, should be brought to the attention of the business
process owner and to IS management responsible for the implementation of the IT solution. Where weaknesses identified
during the post-implementation review are considered to be significant or material, the appropriate level of management
should be advised immediately to allow early corrective action.
10.2.2 Since effective controls over IT solutions are dependent on general IT controls, any weaknesses in these areas should
also be reported. In the event that general IT controls are not examined, this fact should be included in the report.
10.2.3 The IS auditor should include appropriate recommendations in the report to strengthen controls to mitigate associated
risks.

11. FOLLOW-UP ACTIVITIES

11.1 Timeliness
11.1.1 The effects of any weaknesses identified by the post-implementation review are likely to be wide-ranging and high-risk.
Therefore, the IS auditor should carry out, where appropriate, sufficient, timely follow-up work to verify that management
action is taken to address weaknesses and manage risk effectively.

12. EFFECTIVE DATE

12.1 This guideline is effective for all information systems audits beginning 1 January 2005. A full glossary of terms can be
found on the ISACA web site at www.isaca.org/glossary.

APPENDIX

Reference
IS Auditing Guideline G23 System Development Life Cycle (SDLC) Review

Information Systems Audit and Control Association 2004-2005 Standards Board

Chair, Sergio Fleginsky, CISA PricewaterhouseCoopers, Uruguay
Svein Aldal Aldal Consulting, Norway
John Beveridge, CISA, CISM, CFE, CGFM, CQA Office of the Massachusetts State Auditor, USA
Claudio Cilli, Ph.D., CISA, CISM, CIA, CISSP Value Partners, Italy
Christina Ledesma, CISA, CISM Citibank NA Sucursal, Uruguay
Andrew MacLeod, CISA, CIA, FCPA, MACS, PCP Brisbane City Council, Australia
V. Meera, CISA, CISM, ACS, CISSP, CWA Microsoft Corporation, USA
Ravi Muthukrishnan, CISA, CISM, FCA, ISCA NextLinx India Private Ltd., India
Peter Niblett, CISA, CISM, CA, CIA, FCPA WHK Day Neilson, Australia
John G. Ott, CISA, CPA Aetna Inc., USA
Thomas Thompson, CISA Ernst & Young, UAE

© Copyright 2004
Information Systems Audit and Control Association
3701 Algonquin Road, Suite 1010
Rolling Meadows, IL 60008 USA
Telephone: +1.847.253.1545
Fax: +1.847.253.1443
E-mail: standards@isaca.org
Web site: www.isaca.org

Page 6 of 6 IS Auditing Guideline Post-implementation Review