Supreme Audit Institution Performance Measurement Framework (SAI PMF)

Presented by: Silje M. Sandstad, INTOSAI Development Initiative Horacio Saboia Vieira, TCU Brazil ICGFM Conference, Miami, 20-24 May 2013
SAI Performance Measurement Framework

Presentation Outline
1. Introduction 2. The development of SAI PMF 3. SAI PMF performance report: Audit Service Sierra Leone 4. How to do a SAI PMF? 5. Why do a SAI PMF? 6. What is next on the SAI PMF 7. Questions and answers
SAI Performance Measurement Framework 2

1. Introduction

SAI Performance Measurement Framework

 We are. Who are you?

- Respond to a few questions with the clicker for us to get to know you

SAI Performance Measurement Framework

Who are you?

Use the clicker: 1. Working for a donor agency 2. Working for a Supreme Audit Institution 3. Working for another agency

SAI Performance Measurement Framework

Have you heard about PEFA?

Use the clicker:
1. Yes 2. No

SAI Performance Measurement Framework

Have you heard about SAI PMF?

Use the clicker:
1. Yes 2. No

SAI Performance Measurement Framework

Analysis of who you are

Who are you? Donors: 40 SAIs: 20 Others: 40

SAI Performance Measurement Framework

Analysis of who you are

Who are you
PEFA SAI PMF Use SAI PMF 100 %

60 % 30 % 4% Donors
SAI Performance Measurement Framework

40 %

5%
SAIs

10 %

5%
Others

1%

2. The development of SAI PMF

SAI Performance Measurement Framework

10

Why was the SAI PMF developed?

Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010
INTOSAI Framework on the Value and Benefits of SAIs adopted 2010

Increased focus on use of country public financial management systems within the international donor community

SAI Performance Measurement Framework

11

How is the SAI PMF being developed?

Mandated by Johannesburg Accords, INTOSAI Congress, INCOSAI XX, 2010 Responsibility of Working Group on the Value and Benefits of SAIs SAI PMF Task Team
INTOSAI Regional Organizations: African Organization of English Speaking SAIs(AFROSAI-E) and African Organization of Francophone SAIs (CREFIAF) SAIs: Bahrain, Brazil, Chile, Germany, India, Mexico, Russia, United Kingdom Development Agencies: Inter American Development Bank Coordinator:INTOSAI-Donor Secretariat @ IDI

SAI PMF Reference Group

INTOSAI Regional Organizations: Pacific Association of SAIs (PASAI) SAIs: Canada, Denmark, Estonia, European Court of Auditors, Japan, Netherlands, Norway, Peru, Sweden Development Agencies: African Development Bank, OECD, European Commission, World Bank Civil Society Organizations: International Budget Partnership, Global Reporting Initiative (GRI), PEFA Secretariat

SAI Performance Measurement Framework

12

3. SAI Performance Report: Audit Service Sierra Leone

SAI Performance Measurement Framework

13

Country Context

Country Governance Environment

Sierra Leone: Country Context & Institutional Arrangements

Public Financial Management Environment

SAI Environment

SAI Performance Measurement Framework

14

Sierra Leone: Country Context

53% people live on less than $1.25 per day

Recurring conflict 1991-2001: 2 million people displaced

Foreign aid 25% of GNI

180th out of 187 on UN Human Development Index Conflict fuelled and financed by illegal diamond sales
SAI Performance Measurement Framework

Subsistence economy agriculture, fishing

15

Country Governance Environment

Corruption

Political Competition

Media Freedoms

Democracy

Country Governance

Rule of Law

Governance indicators below the median in Africa, but improving faster than its peers
SAI Performance Measurement Framework 16

Public Financial Management Environment

Lack of Performance Objectives Limited Implementation of Audit Recommendations Weak Financial Record Keeping & Reconciliations

Public Accounts Committee Hearings Resuming

Weak Internal Control & Internal Audit

Incomplete or Non-Existent Financial Statements

SAI Performance Measurement Framework 17

SAI Environment
Underqualified staff Vacancies Limited training Education levels Languages Financial accumen Proffesional qualifications Communication challenges Supervision challenges Between central locations To regions Internationally

Limited Financing

Limited recruitment pool

No physical office or IT network

Travel limitations

Capacity development support from DFID, INTOSAI and AFROSAI-E

Development partners budget support
SAI Performance Measurement Framework 18

The Assessment
Findings from the detailed assessment

The indicator scoring

SAI Performance Measurement Framework

19

Independence & Legal 3 4 Framework 4

Constitutional independence of SAI & AG Mandate for public sector Financial, compliance & performance audit Freedom to publish reports Financial independence:

Strategy for Organizational Development 3 Strategy guides organizational development & delivery of mandate Monitoring weaknesses limits continual improvement of audit methods & public reporting of SAI performance Audit Standards & Methodology 2 1 1 2 2 Risk-based financial & Documentation & quality control compliance audit manual Supervision & review Training based on manual Performance audit capacity Quality assurance system Planning & monitoring delivery Performance audit manual & cost of audits
Communication & Stakeholder 3 1 2 Management Reporting to PAC Working with PAC Radio debates Internal comms. Comms. with audited entities

Human Resources & Leadership 2 3 1 External training support Link staff plan to mandate Staff accountability & reward for performance Attract & retain staff Managerial vacancies Training resources SAI Performance Measurement Framework

Management & 2 2 Support Structures Code of ethics, job rotation, structures Monitoring staff Budget process & receipt utilization Infrastructure & IT of funds

SAI Reporting 3 2 3 2 2 Financial audit coverage Timely submission & publication of audit reports Selection of performance audit topics Performance audit coverage Follow-up of audits Reporting performance
20

Independence & Legal Framework

Strategy for Organizational Development

SAI Reporting 3 2 3 2 2 COVERAGE

Audit Standards & Methodology 2 1 1 2 2

SUBMISSION PUBLICATION FOLLOW-UP

AUDIT QUALITY

Human Resources & Management & Support Structures Leadership

Communication & Stakeholder Management

REPORTING PERFORMANCE

SAI Performance Measurement Framework

21

The Analysis
What supports or hampers SAI performance?
Internal factors External factors

SAI Performance Measurement Framework

22

Internal Factors Impacting on SAI Performance

Managerial vacancies

Quality control not implemented

Weak supervision & review

Quality assurance system

Poor documentation

Quality reports. Coverage. Timeliness.

Audit manuals

SAI Performance Measurement Framework

23

External Factors Impacting on SAI Peformance

Context PFM SAI Legal Framework SAI Resources

Strong donor support Participation in INTOSAI & AFROSAI-E programs Weak accountability undermines cooperation with SAI
Weak record keeping, internal control & internal audit Incomplete or missing financial statements Unclear performance objectives & no measurement

Weak financial independence Otherwise strong legal framework

Inadequate budget to hire & train sufficient, appropriate staff Inadequate premises, vehicles, IT networks to function efficiently 24

SAI Performance Measurement Framework

Value & Benefits of the SAI to Citizens

Fiscal Discipline Identification of: Major revenue discrepancies Failings in collection of fisheries revenues Audit of public debt management Efficient Resource Allocation Ensures funds spent for purposes intended Effective Service Delivery Legality of spending Identification of failings in: Anti-malarial drug delivery systems Access to clean water program Distribution of agricultural inputs

Promoting fiscal transparency and accountability Deterring, preventing & detecting corruption Communicating with & engaging citizens: demonstrating relevance to stakeholders
SAI Performance Measurement Framework 25

What Constrains the SAI from Delivering Greater Value & Benefits?
Fighting Corruption
Effective service delivery Aggregate fiscal discipline

SAI Performance Measurement Framework

26

Prospects for Further Improvement Strategic planning Donor support & practices INTOSAI & AFROSAI-E membership Country governance trends

SAI Performance Measurement Framework

27

4. How to do the SAI PMF assessment?

SAI Performance Measurement Framework

28

How will the SAI PMF be used?

Who decides to initiate a SAI PMF? Who decides when to do a SAI PMF? Head of SAI Head of SAI

Who decides how to do a SAI PMF?

Head of SAI

Who decides if, when and how to publish a SAI PMF?

Head of SAI

SAI Performance Measurement Framework

29

How will the SAI PMF Assessment be conducted?

1. SAI Self Assessment 2. Peer Review 3. External Assessment

SAI Involvement The approach will not necessarily affect how the assessment is carried out but the SAI may use the results for different purposes depending on the approach
30

SAI Performance Measurement Framework

How to assure the quality of the assessment?

Quality Assurance should cover:

Is the report factually correct? Has SAI PMF methodology been adhered to?

Is there sufficient evidence to justify the indicator scores?

Does the executive summary provide an in-depth analysis of the evidence and inter-relations between the various factors?
31

SAI Performance Measurement Framework

Quality Assurance
No single defined arrangement of quality assurance Most appropriate approach depends on purpose of the assessment
IDI offer assistance to the QA: Terms of Reference, Draft reports and final reports
SAI Performance Measurement Framework 32

5. Why do a SAI PMF assessment?

SAI Performance Measurement Framework

33

Why do a SAI PMF assessment?

Manage own performance:
Where are we? Monitor progress over time

Implementation of International Audit Standards Demonstrate accountability Provide evidence to donors on SAI Performance
SAI Performance Measurement Framework 34

6. What is next in the development of SAI PMF?

SAI Performance Measurement Framework

35

SAI PMF Roll-out

Training Piloting Guidance Support function @ IDI

SAI Performance Measurement Framework

36

6. Questions and Answers

SAI Performance Measurement Framework

37

For more information:

www.idi.no
E-mail: INTOSAI.Donor.Secretariat@idi.no

SAI Performance Measurement Framework

38

Thank you!

SAI Performance Measurement Framework

39