Escolar Documentos
Profissional Documentos
Cultura Documentos
Don Scott-Kemmis
November 2009
Contents
Summary
B.
C.
Case Studies:
B.1
United Kingdom
28
B.2
Canada
44
B.3
Singapore
60
B.4
Netherlands
68
79
1.
Introduction
While there has been over the last few years a sustained increase in the
emphasis on innovation in the public sector it remains the case that the basis of
systematic empirically-based analysis remains limited. The empirical base is
limited both with regard to the nature of innovation in the public sector its
drivers, characteristics, barriers etc and also to the types of innovation support
initiatives that are effective.
It is clearly the case that the recent focus on innovation in the public sector isnt
because the public sector has recently become innovative. In many countries the
public sector has been highly innovative, and it appears that it has become more
so over time. It is the case however that there are rising expectations on the
public service to deliver better services and policies in new ways, at lower costs,
and often in response to increasingly complex issues. It is also the case that,
with the extraordinary potential of IT, and the high level capabilities of the
human resources in the public services, there are opportunities to deliver on
those expectations.
The knowledge base for informing initiatives to increase innovation in the public
sector derives from the large body of knowledge about innovation in the private
sector (although the extent of applicability to the public sector is uncertain), a
large number of case studies (often based on quite different methodologies and
with considerable uncertainty about the extent to which experience in one type
of innovation in one context provides general lessons), some broad surveys of
innovations (usually derived from applicants who are winners in innovation
award competitions and often without a rigorous conceptual methodology), some
more systematic survey-based studies, fairly normative frameworks based on
direct experience, and/or involvement in research in particular domains.
Drawing on this diverse literature a framework has been developed to guide the
development and interpretation of a set of case studies of how a number of
comparator countries are responding to the challenge of raising the level of
innovation in the public sector. This framework is outlined below. Our primary
interest in these case studies is in how that goal is being pursued and what has
been achieved.
The following discussion in this introductory section is organised into the
following four sections:
What is Innovation in the Public Sector?
Managing Innovation: Processes, Competencies and Context
Addressing the Challenges of Managing Innovation in the Public Sector.
2.
The drivers for innovation in the public sector arise from several sources, that
are more or less common to the public sector across the OECD, and include:
pressure on government budgets; rising public expectations for more accessible
and flexible services and greater participation in service and policy development
and review; and complex social, environmental and economic challenges. The
more proximate drivers arise from: the priorities of politicians; the specific
problems that arise in areas of policy, administration, and services; and, the
identification of options for improvement.
The term innovation is a heterogeneous category. The Publin project 1 provides
the following examples of innovation in the public sector:
new or improved services (for example, health care at home)
process innovation (a change in the manufacturing of a service or
product)
administrative innovation (for example, the use of a new policy
instrument, which may be a result of policy change)
system innovation (a new system or a fundamental change of an existing
system, for instance by the establishment of new organizations or new
patterns of co-operation and interaction)
conceptual innovation (a change in the outlook of actors; such changes
are accompanied by the use of new concepts, for example integrated
water management or mobility leasing)
radical (or paradigmatic) changes of belief systems or rationalities
(meaning that the world view or the mental matrix of the employees of an
organization is shifting, eg joined-up-government)
A complementary set of categories is that of Bekkers et al. 2 :
Product or service innovation, focused on the creation of new public
services or products.
Technological innovations that emerge through the creation and use of
new technologies, such as the use of mobile devices and cell broadcasting
to warn citizens in the case of an emergency;
Process innovations, focused on the improvement of the quality and
efficiency of the internal and external business processes, like the direct
filing and automated assessment of taxes;
Koch, P., Cunningham, P., Schwabsky, N. and Hauknes, J. Innovation in the Public
Sector- Summary and policy recommendations Publin Report No. D24 Published by NIFU
STEP Studies in Innovation, Research and Education
http://www.step.no/publin/reports/d24-summary-final.pdf
2
V. Bekkers, H. van Duivenboden and M. Thaens, Public Innovation and Communication
technology: relevant backgrounds and concepts, in: Information and Communication
Technology and Public Innovation, V. Bekkers, H. van Duivenboden and M. Thaens, eds,
IOS Press, Amsterdam/Berlin/Oxford/Tokyo/Washington DC, 2006, pp. 321.
Private Sector
Pursuit of Profit, Stability or Growth of
Revenues, Market Share, Return on
Investment while minimising risk and
surviving.
Firms of many sizes, with options for
new entrants.
Performance
Metrics
Return on Investment
Management
Issues
Relations
with:
~ End-Users
Public Sector
Enactment of Public Policies.
10
~ Sources of
Knowledge
Time Horizon
Based on: Koch, P., Cunningham, P., Schwabsky, N. and Hauknes, J. Innovation in the
Public Sector- Summary and policy recommendations Publin Report No. D24 Published
by NIFU STEP Studies in Innovation, Research and Education
http://www.step.no/publin/reports/d24-summary-final.pdf
3.
11
Figure1:BringinganInnovationfromIdeatoApplication
Subprocesses:
3.
1.
Imagining
the Dual
(TechnoMarket)
Insight
2.
Mobilizing
Interest and
Endorsement
Incubating
to Define
Commercializability
BuildingtheValueofanInnovation
5.
9.
7.
Demonstrating
Sustaining
Promoting
Contextually
Full Implementation
Adoption
in
8.
Products
4.
Mobilizing
6.
and
Mobilizing
CompleMobilizing
Resources Processes
mentary
Market
for DemonAssets for
Constituents
stration
Delivery
Bridges:SatisfyingandMobilizingStakeholdersatEachStage
Source: V.J. Jolly (1997) Mind to Market. Harvard Business Press
Crawford,C.B.(2001)LeadershipandInnovation:ChampionsandTechiesasAgentsofInfluence.Association
ofLeadershipEducators.;Shane,S.,Venkataraman,SlandMacMillan,I.(1995)CulturalDifferencesin
InnovationChampioningStrategies.JournalofManagement.21(5):931952.;Howell,J.etal(2005)Champions
ofproductInnovations:defining,developingandvalidatingameasureofchampionbehaviour.Journalof
BusinessVenturing20:641661.
12
Radical
Uncertainty
&Complexity
Major
Incremental
Processes
Services
Policies
13
Generating new
knowledge in-house
through eg research,
development, engineering
Risk Management
Road-mapping
Business case analysis
Financial analysis
Impact assessment
Risk analysis
Decision making
Collaboration management
Alliance management
Licensing
IP management
Networking
Stage gate
Product development
Knowledge transfer
Negotiation
Valuation
Relevant science
and technology,
organisational
application,
regulatory
knowledge
Creativity
IP management
Research
Design
Choosing an innovation
focus appropriate to the
opportunity & capability
Executing projects
Evaluation
Portfolio management
Project Management
Team Management
Problem Solving
Marketing
Cross Functional teams
IP management
Budgeting
Implementing change in
the organisation
Learning through the
evaluation of experience
and the incorporation of
Change management
Organisational
change dynamics
Review
Monitoring
Communication
Persuasion
Leadership
Communication
Team building
Delegation
Empowerment
Motivation
Conflict resolution
Leadership
Self management
Stress management
Change management
Problem solving
14
Communication
Failure tolerance
Team building
Training
Delegation
TQM
Empowerment
Codification
Motivation
Experiment
Auditing
Idea management
Incentive systems
Continuous improvement
Human resource mngmt
Knowledge management
Based on: Tidd, J. Bessant, J. and Pavitt, K. (2005) Managing Innovation. Integrating,
Technological, Market and Organizational Change. 3rd Edition. Chichester. Wiley; Carlopio, J.
(2003) Changing Gears. The Strategic Implementation of Technology. Palgrave Macmillan;
IMProve for Innovation Management Professionals. The IMProve Approach The IMProve
Platform. Version 1.3 November, 2007,
http://ec.europa.eu/enterprise/newsroom/cf/document.cfm?action=display&doc_id=3048&userser
vice_id=1; Rae, D.M. (1997), Teaching entrepreneurship in Asia: impact of a pedagogical
innovation, Entrepreneurship, Innovation and Change, Vol. 6 No. 3, pp. 193-227.
Processes
& Culture
Capabilities
Organisational
Innovativeness
Source: Author
15
16
Developing
innovativeideas
fromstaff,
suppliers,
customers,other
organisations
Implement
innovations:
leadership,risk
management,
addressbarriers
Scaleupfor
wider
implementation
Learning:aboutgeneratingideas,incentivesthatwork,project
management,leadership,scalingup
Buildingconfidence,networks,linkagesoutsidetheorganisation,
reputation,neworganisationalarrangements
Source: Modified from UK, Comptroller and Auditor General (2009) Innovation Across Central
Government. National Audit Office, HC 12 Session 2008-9
Key
Requirement
Leadership
Resources
Networks
Culture
Competencies
Ideas
Learning
Organisational
strategies
Criteria
Champions who set goals and provide organisational support and protect
the ideas from premature judgement
Commitments of resources through each stage of the innovation process.
Short term budget & planning horizons can limit sustained commitment.
Informal networks linking individuals to sources of capability and to
communities of practice; formal networks linking organisations to others
related vertically or horizontally or outside the public sector.
Cultures that support the identification and exploration of ideas from any
source, experimentation and risk taking, that supports learning; good
internal communications; lack on internal politicking
Accessible competencies inside or outside the organisation.
Ideas, the starting point for innovation, may come from any source.
Analysis of the external environment
Individuals, teams and organisations learn from training activities, case
studies, experience, reviews
Strategies that recognise the role of renovating systems and capabilities for
innovation, and that develop performance evaluation approaches to assess
the effectiveness of the organisations innovation systems; future
orientation
Based on Albury (2006), Borins (2006), Roste (2004);Koch & Hauknes, Publin
(2005);Mulgan & Albury (2003); Mulgan (2007); UK, NAO (2006); LSE Public Policy
Group (2008).
17
18
Sources: Publin (2007) ; iDea (2005); Mulgan (2007); Mulgan & Albury (2003); Vigoda-Gadot et al (2005)
19
5.
Promoting Innovation in the Public Sector: Developing
Competencies and Processes and Improving the Context.
Initiatives to improve innovation performance have three types of outcome:
Specific innovation activities that may result in the implementation of a
successful innovation;
Experiences and learning processes by all involved that may build
capability at the individual level, at the group level, and at the
organisational level (eg via developing external links); and
Learning about how to promote and support innovation which may be
diffused more widely.
All or some of these outcomes could range from highly negative to highly positive,
depending on how the process is managed. For example, an unsuccessful
innovation project might nevertheless yield vital learning which improves overall
This list is largely based on Ren Kemp & Rifka Weehuizen Policy learning, what does it
mean and how can we study it? Publin report D15
20
Training Programs
Capability
Building
Learning
Diffusion
Tables 7 and 8 provide some additional checklists for agency and department
level innovation development initiatives.
21
Support Resources
External
Establishment of
External Innovation
Support Unit
Establishment of
External Training
Facility/Program
Central
Direct Initiatives
Central
Development of On-Line
Support Resources
Development of Case
Studies & Support
resources
Research on PS
Innovation Aust. &
International
Surveys of PS
Innovation &
development of metrics
Department/Agency
Organisation Level Innovation
Unit to Lead or Support
Projects
Embedding in Performance
Assessment
Central
Requirement for Organisation
Level Innovation Strategies
Development of Frameworks
& Metrics for Innovation
Performance Assessment
Department/Agency
22
Mechanism
How
Examples
Training Programs
On-line resources
support tools
Development of case
studies, resources
Central fund for
supporting trials
Central organisation
for capturing and
diffusing experience
Innovation oriented
procurement policies
and/or support
funding.
Innovation awards
and competitions
Innovation units at
the Central level
10
www.betteregulation.gov.uk/
23
Research on PS
innovation
Surveys of PS
innovation- with
metrics
Organisation level
innovation strategies
Organisation level
innovation
performance
assessment
Innovation units at
the Departmental
level
Although there are risks of missing the wood for the trees, many
public sector organisations have introduced forms of knowledge
management.
The Department for Business Innovation & Skills in the UK assists
departments in developing their innovation strategies.
NESTA in the UK is developing a framework for public sector
innovation metrics
UK-The Department for Education set up an innovation unit which
has supported imaginative communities of practice, and the
Department of Health has established an NHS Institute for
Innovation and Improvement. Within individual agencies, too,
smaller innovation funds have been widely used to give front line
managers a chance to try out new ideas.
Support for pilots and trials is increasingly common throughout the
UK public sector. In the UK NESTAs Lab is an initiative to develop
and test new policies, particularly social policy.
11
24
Innovation units
outside government
Benchmarking - intra
and inter-national
Leadership that
supports innovation
Position guidelines
and selection criteria
Individual and group
incentives
Procurement
programs that
incorporate a specific
objective of promoting
innovation by
suppliers
12
www.isb.gov.uk
Wessner, C.W., Converting Research into Innovation and Growth, SBIR, the University and the Park, National Research Council, April
10 2008. http://www.unece. org/ceci/ppt_presentations/2008/fid/Charles%20Wessner.pdf
13
25
What steps have been taken to determine the most important fields,
issues, and problems for innovation? These include: fields of relative
policy or delivery failure; areas where new technologies create
opportunities; cross-cutting fields.
Who has board and ministerial level responsibility for innovation?
What units, teams or groups are there to organise innovation? Whose
job is it to scan internationally for promising ideas; to scan
domestically; and to learn from neighbouring fields?
How are broad and specific budgets to support innovation determined,
and what methods are used to determine spending levels, metrics etc?
What processes are used to promote innovation; take stock of
successes and failures and determine which innovations should be
scaled up (eg spending reviews, strategy reviews)?
What mechanisms exist to develop promising ideas into workable
prototypes, either through mixed in-house teams or arms length
bodies?
What steps are taken to ensure recruitment and retention of creative,
entrepreneurial people?
What mix of pilots, pathfinders, ventures is used and why?
How are users, consumers and citizens engaged in innovation for
example through networks, holding funds, etc?
What methods are used to define and measure success?
Testing &
Measurement
Leadership
Culture shaping
Networks
Risk management
26
FROM
Distant
Vertical
Design logic
Power-based
rule-based
Efficiency
Independence
Policy-based steering
Political accountability
Discrete organization
Professional autonomy
Detailed central steering
Indirect participation
TO
Open
Horizontal
Action logic
Trust-based
Context-based
Responsiveness
Interdependence
Frontline steering
Societal responsibility
Embedded organization
Professional responsibility
Indirect, global steering
Direct participation
Source: H. van Duivenboden and M. Thaens (2008) / ICT-driven innovation and the
culture of public administration.Information Polity 13 (2008) 213232, p.228
Even if serious attempts are being undertaken to deal with cultural change
issues, it is increasingly difficult to make a success out of them because it is no
longer a matter of changing one particular organizational culture but of changing
a number of routines, values, rites, rules and styles of several parties at the
same time. After all, due to the linkage capacity of modern technology and the
penetration of ICT into the primary processes of public administration, many
ICT-innovations have an inter-organizational and/or relational character. This
observation stresses the necessity of collaboration between relevant
stakeholders and the emergence of intermediary organizations, like trusted third
parties or shared service centres, which facilitate collaboration. Hereby,
(personal and/or mutual) trust can be seen as an important condition for actors
27
14
28
The strong focus on innovation in the UK public sector is driven by the search for
productivity in the context of tightening budgets, the challenges arising from
complex social and environmental issues, and rising demands from the public for
more accessible and responsive and flexible services. Almost all organisations
saw innovation as a potential contributor to efficiency, policy development,
improved procurement, internal administrative processes, and communication
with users, staff training, delivery of services, and changing citizen behaviour.
The recent report reviewing public sector innovation performance 15 considers
that:
These factors will mean that government cannot simply do more of what
it has always done, but that it will need to develop radical and new
approaches and seize ideas within and outside organisations that can lead
to greater efficiency and effectiveness.p11.
According to the Comptroller and Auditor General (2009) the government
allocated over 3 billion a year for innovation through departmental innovation
budgets and will allocate a further 2.5 billion to support public sector innovation
from 2008-9.
Over the past two years the promotion of innovation in the public sector has
become a new and important responsibility for a number of organisations that
are themselves new. Hence, while there are bold plans and initiatives it remains
to be seen how effective these will be.
Organisations responsible for promoting innovation include both those with a
role across government and those at the department or agency level. The
Department of Department for Business Innovation & Skills (DBIS) has the
primary responsibility for innovation in the public sector. This role is set out in
the White Paper (of March, 2008), Innovation Nation. The Prime Ministers
Strategy Unit in the Cabinet Office also has a responsibility to promote
innovation in policy development 16 and delivery, as part of wider responsibilities
to promote long term and cross cutting strategic issues, and hence there is
some overlap of responsibility. These two organisations work closely together.
The Treasurys Operational Efficiency Program includes strategies for facilitating
front line innovation. Innovation is now also one of the components of the skills
included in the competency framework for the public sector.
15
UK, Comptroller and Auditor General (2009) Innovation Across Central Government.
National Audit Office, HC 12 Session 2008-9. p.15.
16
See UK Cabinet Office. Excellence and fairness: Achieving world class public services,
August 2008; and UK Cabinet Office. Transformational Government, Enabled by
Technology, November 2005.
29
30
At the Department for Environment Food and Rural Affairs Innovation Centre
policy or delivery teams can run workshops designed to enable the generation of
innovative solutions to problems.
The Department for Work and Pensions IT Innovation Centre and Solutions
Centre are designed to inspire creativity and innovation as well as being a site
where new ideas can be tested before implementation. The Centres are also
available for use by other government bodies.
The Ministry of Defences Centre for defence Enterprise invites proposals for
funding and support from companies with scientific or technological innovations
that have a potential application in defence.
The NHS National Innovation Centre (part of the NHS Institute) supports the
adoption of technological innovation from industry. It uses a web-based screening
tool to allow innovators to self- assess potential ideas, and assistance for the most
promising ones to be developed within the health service. The Government
Gateway team in the Department for Work and Pensions is working on an adapting
the screening tools so that they can be made available across government.
In local government, the Social Innovation Lab for Kent helps council staff
solve local problems. For instance academic experts have used ethnographic
techniques to help the council understand the experience of service users, leading
to changes such as services for fathers at childrens centres and better internet
access to information on care services.
Source: UK, Comptroller and Auditor General (2009) Innovation Across Central
Government. National Audit Office, HC 12 Session 2008-9
17
31
NESTA has created the Lab to meet this need for new ideas that work. By
bringing together experience and ingenuity from across the public, private and
third sectors, and drawing on the insights of citizens and consumers, the Lab
plays a vital role in making public services fit for the 21st century. The Lab
provides the freedom, flexible capital and expertise to undertake radical
experiments. It tests out new ways of finding and spreading the best ideas this might be by running a challenge prize, building a social ventures incubator,
or creating powerful new teams of users, front-line staff and decisionmakers. 21
2.
The UK has a long history of public sector innovation from minor improvements
in services to such initiatives as the formation of the BBC, Open University, the
National Health Service. Innovations include those that increase efficiency,
improvements in the quality of services, new services or ways of delivering
services.
Organisations draw on ideas for innovation from a wide range of sources but the
major source in practice tends to be internal, particularly senior management
as is the perspective that arises from similar surveys in the private sector.
19
32
33
Description
A user-generated website and independent, non-profit company,
founded by a social entrepreneur, which enables patients to share
opinions about their health care with the NHS and each other.
An experimental approach to support youth sport in deprived areas,
which are delivered through various local organisations.
A program bringing together a range of agencies to support parents
and young children, leading to several hundred local programs.
Stimulated local action to develop safe routes to school, leading to
local initiatives to improve safety.
A local initiative to provide social care for older and disabled people.
Police officers specifically for low level crime and anti-social
behaviour.
Fund to support enterprise in disadvantaged communities and social
groups.
Informal networks and intermediary organisations that link local
interest groups in the art and creative industry communities.
Mechanism to listen to patient views I the assessment and redesign
of services.
Developed by the Education departments Innovation Unit the
program facilitates links between schools and local authorities in
order to support innovate practices and to disseminate the findings
of these developments.
22
LSE Public Policy Group (2008) Innovation in Government Organisations, Public Sector
Agencies and Public Service NGOs. Innovation Index Working Paper NESTA.
23
Su Maddock (2007) Creating the Conditions for Public Innovation. National School of
Government; The Young Foundation and NESTA. p57
34
3.
The Role of Information Technology as a major driver of
Innovation
There is surprisingly little discussion of the role of IT in the recent reviews of
innovation in the UK public sector. The reviews focus on more generic issues.
However, the rising significance of the internet had attracted strong interest
from government. It was clear that the internet raised issues inter alia,
regarding the reuse of information generated in the public sector, and regarding
participation by the public sector in the new media. The internet is undoubtedly
one of the most significant innovations of the modern era. In the UK internet
usage grew from 9% of households in 1998 to almost 60% in 2006.
It became clear that:
new forms of large-scale self-help on public policy issues had been
emerging online and gathering public attention;
new social and economic value were being created from public information
using new technology;
government initiatives regarding the web had been ad hoc and had mixed
effect;
The role for government and its capability were unclear.
There was little systematic information about these developments and
their significance
The Policy Review Building on progress: Public services had raised awareness in
government regarding new forms of online activity:
The Government should support the development of new and innovative
services that provide tailored advice to specific groups (for example the
netmums.com website which provides a discussion and advice forum for
mothers). These are outside governments direct influence, but government has
a role to play in supporting them for example by ensuring that they are not
undermined by government programs or websites with similar objectives, and
have easy access to publicly available information. 24
Recognising these trends the government commissioned a report on the
appropriate response by government. The independent reviewed 25
recommended that government should respond to three particular challenges:
engaging in partnership with user-led online communities;
24
35
For example, the report suggests extending to other areas of the public sector
the BBCs model for innovation in its backstage service which encourages
people to innovate by re-using the BBCs data and services. The report
recommends that UK central government should create a backstage capability
to unlock the innovation potential of the information held in government data
bases.
Recognising that when mainstreaming any innovation, systemic culture and
behaviour change is required, the Taskforce makes the case for initiatives to
bring into the mainstream of UK government the innovative approaches it
recommends. The report therefore calls for action to help the public sector to
acquire the new skills and practices required to support this.
26
36
To promote innovation across the public sector DBIS has selected five focal
areas 29 :
Creating the conditions for innovation by aligning the major forces of the
public sector to be pro-innovative.
Leading innovation by promoting awareness at the highest levels of the
importance of innovation and of the principal tools to support it.
Supporting and disseminating exemplars.
Drawing on all sources of innovation by engaging users and front-line staff
and looking at innovation systems in the third sector, private sector,
Devolved Administrations and public sectors in other countries.
Realising the potential of innovation as an enabling force in driving related
policy initiatives.
5.
29
37
6.
In preparing the 2009 report on innovation the National Audit Office conducted a
survey of 27 government departments, held an online discussion with 120 front
line public servants and carried out 11 case studies. The overall report concluded
that the need for innovation is being emphasised more strongly by the highest
levels of the public sector. The report found that many more organisations were
establishing innovation units, conducting customer research and staff suggestion
schemes.
The NAO (2009) review concludes that government organisations are not
systematically pursuing approaches to procurement that promote innovation.
While there has been a series of initiatives to encourage government
organisations to seek and use feedback from users, and many such
organisations have introduced discussion groups, workshops, online
communities, and message boards, the 2009 NAO review found little evidence of
these having an impact on innovation 32 . The reviews findings in relation to
progress in implementing their earlier recommendations provide some insight
into performance:
Progress in implementing recommendations from the 2006 NAO report.
Summary of recommendation
Summary of progress
32
In 2008 the Cabinet Office launched a new standard, Customer Service Excellence, to
support customer focused service delivery.
38
7.
Lessons of Experience
The National Audit Office examined innovation performance in its 2006 report
Achieving Innovation in Central Government Organisations. It found that
approaches tended to be, focused at the top level of organisations, ad hoc, there
was little incentive for individual managers to take initiatives and that
information on the costs and outcomes of innovation was generally not collected.
It proposed more systematic approaches to these issues and to seeking ideas
from front line staff, encouraging learning from other organisations, and using
piloting for developing innovations. The development of new initiatives will
inevitably mean that some will fail and most will need substantial modification
before full implementation.
A set of focus group discussions was convened by the LES Public Policy Group in
2006 in order to provide greater insight for the 2006 NAO report on innovation
in central government 33 . These groups were composed of public servants and
consultants who work with government. The groups felt that the perspective on
innovation that came from the NAO survey of organisations tended to underplay
the role of front line staff in innovation, but strongly supported the view that
ideas needed support from senior managers if they were to progress, that
change was particularly complex due to the cautious hierarchical approaches,
lack of working across groups, that risk avoidance due to the fear of failure was
a barrier, as was often a lack of clarity over costs and benefits (due to having a
range of objectives).
The 2009 report of the NAO makes observations on the barriers and
opportunities for innovation, and identifies a range of lessons based on its case
studies, surveys and focus group discussions. The figure below, drawn from the
survey carried out for the 2009 NAO report identifies the factors considered to
support or hinder innovation in the public sector:
33
39
The report suggests that the case studies of recent innovations show 34 :
good performance and cost information can help identify where innovation
is needed and would be beneficial;
customer insight can be used to identify areas for innovation and possible
solutions;
technological innovations can be applied to service delivery to generate
efficiency and service improvements;
34
40
35
41
36
42
8.
Sources
43
44
Drivers
There are many drivers that are leading to an increased emphasis on innovation
in the Canadian public service.
The public sector faces high levels of client expectations for services, evolving
service delivery models that require new skill sets and higher levels of
knowledge for frontline workers, and financial constraints. Just when
organizations most need a strong, dedicated workforce to meet service and fiscal
demands, the workforce is shifting and decades of knowledge and experience
are starting to head out the door. The challenge is how to attract, retain and
engage employees to achieve high levels of organizational performance. 37
A major driver of innovation in federal government has been the pressure on
budgets due to revenue declines in the 1990s. The focus has remained on
improving efficiency and service quality while containing expenditure. According
to Glor the approach through the 1990s centred on cost saving through cutting
functions, reducing support to third parties, introducing user fees and to
privatise or form partnerships to maintain functions which could be financed by
users 38 :
The Department of Fisheries and Oceans did more with less: it
retired ships, double-crewed some ships and transferred ships among
regions, leading to a 10% increase in sea days, a 24% increase in
efficiency, and a $1 million reduction in annual operating costs (IPAC,
1993). Improved service through greater use of computers and the
Internet have been emphasized in Industry Canada, Natural
Resources Canada and Health Canada. A paper burden reduction
initiative has led to redesign of the Record of Employment, cutting
the number of forms from six to two, and reducing employers costs
by $100 million a year. The first paperless court was introduced in
Canada in the federal Competition Tribunal (internationally, a
paperless court had previously been introduced in the Netherlands):
Glor (2006) p118
Organisations, Strategies and Programs
The Treasury Board has been perhaps the key organisation in promoting
innovation and an important foundation for continuing initiatives to promote
innovation in the Canadian Public sector has been the Treasury Board publication
Results for Canadians in 2000 which signalled an attempt link whole-ofgovernment performance management and performance management systems
37
45
39
46
42
47
46
http://www.smarttape.ca/SmartTape
http://www.ccaf-fcvi.com/IRCSymposium/english/IRC-TakingChances.pdf
48
http://www.ccaf-fcvi.com/IRCSymposium/english/IRC-SymposiumProceedings.pdf
49
Glor 2006)A Gardener Innovators Guide to Innovating in Organisations.
www.innovation.cc/books/guide_innovate_organization.pdf
47
48
2.
The available evidence suggests that much of the innovation activity has been in
improving service delivery. Establishing the organisational arrangements,
partnerships and policies that enabled that involved a great deal of
organisational and policy innovation. The implementation of the service
improvement also involved extensive use of IT.
Two important innovations in Canada were actively developed and promoted by
the Service and Innovation Sector 51 of the Treasury Board Secretariat. One
major innovation in Canada in 2005, which has attracted international attention,
was the formation of a new department, Service Canada as a one stop point for
citizens to receive all government services, via telephone internet or in person.
In developing this innovation similar models around the world, including
Australias Centrelink, were reviewed. The Treasury Board Secretariat was also
the central player in the development of Government On-Line, as a tool for
service delivery. The implementation took six years, cost C$880m and involved
collaboration with 34 participating departments and agencies, who also funded
their own participation 52 . The Government-on-Line project has developed into a
comprehensive strategy to provide integrated services and to develop
partnerships across both government departments and agencies and the private
and voluntary sectors 53 .
Looking at submissions to the IPAC awards over the 1990-2003 period
Galimberti suggests that most involved new organisational forms or processes or
new service delivery (particularly involving greater empowerment toward the
end of the service chain and more partnerships with third parties, eg citizen
centred approaches). Most were in some way IT enabled. He comments that few
of the submissions involved new policy innovations, despite attempts by IPAC to
encourage such submissions, although in recent years there has been some
increase in policy-related innovations 54 .
50
Federal Public Service Employee Survey (1999) quoted in Teofilovic (2002), p15.
This unit was closed following a 2003 Expenditure Review.
52
Joyce (2007); and Tan, K. C. & Mechling, J. (2007) Service Canada A New Paradigm
in Government Service Delivery. John F. Kennedy School of Government. Harvard
University.
53
Teofilovic, N. (2008) The Reality of Innovation in Government. The Innovation Journal:
The Public Sector Journal Volume 7, No. 3, 2002www.innovation.cc/peerreviewed/reality.pdf p.12
54
Galimberti , J. (2003) Chronicling public sector renewal in Canada:the IPAC Award for
Innovative Management. VII Congreso Internacional del CLAD sobre la Reforma del
Estado y de la Administracin Pblica, Lisboa, Portugal, 8-11 Oct. 2002
http://unpan1.un.org/intradoc/groups/public/documents/CLAD/clad0043605.pdf
51
49
55
50
3.
The Role of Information Technology as a major driver of
Innovation
IT has clearly had a systemic role in much public sector innovation in Canada.
According to Galimberti technology has played a part in almost all of the IPAC
award submissions received. IT applications for electronic transactions and for
electronic kiosks have been common 57 .
The more extensive use of IT has been a major driver of innovation in the
Canadian government. Teofilovic suggests that the widespread use of IT in the
private sector transformed expectations of the government services 58 . The
government response began with a strategy blueprint paper in 1994.
Teofilovic suggests that, while there have been comprehensive programs to
introduce IT based service innovations, there has been no overarching federal
strategy to address grass roots organisational change, through, for example
greater empowerment, leadership and intra and inter-departmental
collaboration. Initiatives introduced by Human Resources Development Canada
and by the Treasury Board Secretariat in the late 1990s have focused on
developing capabilities in managers, and providing guidelines, for balancing
accountability and decision making scope.
Canada has been particularly active in using IT to deliver new and improved
services. Two reports by Statistics Canada indicate the public sector is an active
user of IT. A 2002 report based on an analysis of trends over 1998-2000 found
that, standardised by size, public sector organisations were more likely to have
implemented technological change than were private sector organisations. The
education and health service sectors were particularly active. This performance
was in part driven by the Y2K, but the report suggests that e-government was
also a major driver 59 . The 2004 study again found little difference between the
public and private sector in the rates of adoption of new technology. The study
also found that the public sector was more likely to develop training programs as
part of the IT implementation process. Over half of the public sector IT adoption
51
4.
60
Earl, L. (2004) Technological Change in the Public Sector, 2000-2002. Working Paper.
Statistics Canada.
61
Teofilovic, N. (2008) The Reality of Innovation in Government. The Innovation Journal:
The Public Sector Journal Volume 7, No. 3, 2002www.innovation.cc/peerreviewed/reality.pdf p.12
62
(http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp).
63
http://dor.myflorida.com/dor/report/performance_measures_fy200708_q3.pdf)
64
http://www.iccs-isac.org/en/cmt/benchmarking.htm
65
(http://www.utexas.edu/research/cswr/survey/site/)
66
http://www.tbs-sct.gc.ca/faq/wbmt-eng.asp
52
The federal structure of the Canadian system of government means that many
functions are performed by provincial government or jointly by the federal and
provincial governments. A foundation for much innovation in service delivery has
been the development of alternative service delivery (ASD), which was an
innovation responding to the realities of a large dispersed country with a federal
structure. This foundation has enabled a great deal more flexibility and
innovation, particularly involving horizontal and vertical collaboration across
government. 67
Brodtrick summarises the stages of reform in the Canadian public sector, which
forms a background to the current context, in terms of five phases, as shown
below.
Phases of Reform in the Canadian Public Sector
67
53
6.
68
Armstrong, J. and Ford, R. (2002) Public Sector Innovations and Public Interest
Issues. Discussion Paper The Innovation Journal. http://www.innovation.cc/discussionpapers/ps-innovation-public-interest.htm
69
Wilkins J. (2002) Learning from Canadian Innovations in Alternative Service Delivery.
CAPAM Biennial Conference. 2002.
70
http://www.ccaf-fcvi.com/IRCSymposium/english/IRC-SymposiumProceedings.pdf
54
7.
Lessons of Experience
This study has not been able to identify an integrated and comprehensive
strategy to promote innovation across the Canadian public sector.
Consequently, it is not clear how the lessons from programs that, directly or
indirectly, promote innovation can be drawn together.
However, the lessons from the sustained efforts to improve service performance
have been identified. A comprehensive review of the policies and strategies that
have enabled the sustained improvement in services details the sequential
development of the organisations that supported these developments. A key
organisation has been the Public Sector Service Delivery Council (PSSDC),
involving federal and provincial collaboration. Over the past decade the PSSDC
has had three leadership roles:
collaborative research
collaborative learning, and
collaborative service improvement. 73
The recent review of the development of service improvement in Canada
suggests that the keys to the success of the service improvement strategies
have been ..the implementation of:
71
55
74
56
8.
Sources
1.
2.
3.
4.
76
See also: Schmidt, F., 2004, Workplace Well-Being in the Public Sector - A Review of
the Literature and the Road Ahead www.hrma-agrh.gc.ca/hr-rh/wlbpseeoppfps/documents/WorkplaceWell-Being.pdf).
77
Joyce (2007)
57
6.
7.
8.
9.
10.
11.
12.
13.
14.
Additional References
Albury, D. (2005). Fostering Innovation in Public Services. Public Money &
Management, 25(1), 51-56.
58
Kernaghan, K., B.Marson and S.Borins, 2000, The New Public Organization,
Ottawa, Institute for Public Administration in Canada.
59
60
B.3 Singapore
1.
The current overarching policy framework for public sector development is PS21.
The PS21 Office is within the Prime Ministers Department 78 . This framework
places a strong emphasis on continual, engagement, empowerment and
individual responsibility for seeking opportunities for innovation and
improvement. This is clear form Table 1 which summarises the objectives and
focus of PS21, and from Table 2 which provides an innovation manifesto for the
public service.
Table 1:
PS21 is about the Singapore Public Service's commitment to Anticipate, Welcome and
Execute change, influencing developments in order to provide Singapore with the best
conditions for success.
PS21 has two objectives:
to foster an environment which embraces and activates perpetual change to
remain effective and relevant, whilst paying attention to employee engagement
and recognition; and
to nurture an attitude of service excellence
Focus of PS21
Empowering and Enabling Officers for Continuous Improvement. PS21 is a mass
movement and at its core lies our people. Every officer, regardless of level or
nature of work, has to feel empowered to take responsibility for his work and think
about how best to deal with challenges and unexpected situations that may arise.
We need to create the right conditions to empower our officers to seek and to
contribute to continuous improvement. At the same time, we recognise that we
need to build the skills and capacity of our people and to provide them with
learning opportunities to develop themselves.
Employee Engagement and Recognition. We promote employee engagement as we
recognise that officers will be committed to their jobs and enthusiastic about doing
it well if they feel engaged, valued and recognised. Efforts in recognising staff
excellence and promoting staff well-being will help to strengthen the morale of our
officers and foster pride within the Public Service.
Service Excellence. With rising public expectations, and increasingly complex
transactions, the Public Service seeks to continually improve its service and to
remain relevant. We encourage an attitude of service
Excellence amongst public officers and search for better ways to cater to the needs
of our customers. Public agencies take ownership and responsibility for promoting
service excellence within their organisations, and contributing to the overall
improvement of the service provided by the Singapore Public Service.
http://app.ps21.gov.sg/newps21/
78
http://app.ps21.gov.sg/newps21/
61
Table 2:
for all in the Knowledge Based Society. The Public Service for the 21 century, as an
integral component of society, has to continually re-invent itself to support the
innovative and enterprise movement so as to better anticipate, welcome and execute
change. In doing so, we will be guided by the following principles:
People want challenge in their work and recognition for what they do. People
want to contribute and know that their contributions matter. Hence, they are
motivated to contribute to a worthwhile higher purpose and cause, beyond selfinterest.
Everyone has talent and ability. Each individual has something to contribute and
diversity of views must be encouraged for non-linear thinking and analysis.
People want to improve themselves and can do so. People have an inherent thirst
to learn. They can improve given time, opportunity and training.
Individuals best realise and maximise their creative value through collaboration
with others. Collaboration may vary from a network of relations to an integrated
organisation. Innovation thrives best in a vibrant environment as opposed to
being in a vacuum.
Everyone thinking and doing will achieve more than a few thinking and doing.
This is especially critical for Singapore with our limited manpower, to succeed, we
will need to leverage on the diverse knowledge, skills and expertise of every
single individual.
The managers role is to facilitate and allow his staff to optimise their innovative
capacity. By instilling a sense of purpose and creating the broad framework and
safe environment in which the staff could operate - new ideas, experiments and
change become the norm rather than the exception. Supervisors must move from
managing resources to leading and inspiring people. Leadership skills must be
honed for the New Economy.
We challenge everyone to ask themselves: HAVE YOU INNOVATED TODAY?
Singapore Public Service
62
79
80
63
2000
2000
2002
2003
2005
The inaugural TEC Seminar was held with the objective of providing a
platform for innovators to showcase their TEC innovations and to look
beyond the TEC trial for new sources of funding, and attract potential
business partners and customers. The seminar helped link up TEC
innovators with various public agencies that play the role of adopters, and
venture capitalists as potential investors to some of these innovations.
2006
The inaugural TEC Public Service Innovation Award 2006 was successfully
held. A media conference was held at the NLB Pod on 6 November to
showcase the award winners of the TEC Public Service Innovation Award
2006. The award was given out by Mr Teo Chee Hean, Minister in charge of
the Civil Service at the Public Service Exhibition on 15 November.
Table 4:
Education
----Advanced Government Integrated Learning Environment (AGILE)
----Educational Innovation for the Knowledge-Based Economy:
----Development of a Problem-Based Learning (PBL) Education Model
----Diagnostics Tutorial & Assessment System (DTAS)
----Math Explorer - A 100% Web-based Assessment System for Math
----Intelligent Content Assessment Marking System (ICAM)
----Ide'Lite, Holistic Service Training Pedagogic Approach
Environment
----New Soil Improvement Technology
----Bioscrubber for Odour Treatment
----Biology Treatment of Industrial Food Waste
81
http://www.tec.gov.sg/TEC%20Home/home1.htm
64
65
http://www.tec.gov.sg/
5.
As noted above, the Singapore government has a strong and well earned
reputation for robust, pragmatic and innovative approaches to social, economic
and development policy. The continuity of the administration, with little
likelihood of a change of government, provides a foundation of stability which
enables long term planning, and a history of independent analysis and action.
The Singapore government has a particularly strong orientation toward policies
to drive evolution toward the development of a knowledge economy. This
perspective has a systemic influence on policy throughout the dense networks
that link the political, administrative and business groups in Singapore.
6.
Apart from the evidence provided by the TEC program no systematic evidence of
the improvements in innovation-related performance is available.
7.
Lessons of Experience
Perhaps the key lesson of this unique experience is the potential of a program
like the TEC to initiate bold projects that would be unlikely to happen if left to
individual departments. A second lesson is the potential for such a scheme to
elicit ideas and engagement from throughout the economy and in so doing
66
Sources
67
68
B.4 Netherlands
1.
82
http://www. Innovatieplatform.nl/en/projecten/overhead_en_innovatie/index.html
www.innovatieplatform.nl
84
www.minbzk.nl; www.minfin.nl
85
www.nsob.nl
86
www.elo.nl/elo/english/kelo/index.jsp
87
http://www.n21.nl
83
69
Public Sector Innovation and Quality bureau in the Ministry of the Interior and
Kingdom Relations developed a database of 850 examples of innovation in
several sectors of the public sector were assembled into a Public Sector
Innovation and Quality Database.
www.benchmarkenindepubliekesector.nl (in Dutch)
The innovations compiled in the Public Sector Innovation and Quality Database,
cover both new services or products and improvements in services or
administration processes, and the majority (according to one source, 70%) were
incremental. In public administration most innovations originate with
management, whereas in healthcare and the police most were initiated by
employees. Most incremental innovations were initiated by management
whereas most radical innovations were initiated by employees or external
organisations 88 .
3.
The Role of Information Technology as a major driver of
Innovation
The Netherlands has been a leader in e-government, with major initiatives in the
early 1990s, the Electronic Government Action Plan in 1998 and a
comprehensive strategy, The Dutch Digital Delta, in 1999, and new action
program, Another Government, in 2003. Major innovations have been
implemented across the public sector, including in voting, tax, and one-stop
digital desks for public access to government information and transactions. The
Ministry of Economic Affairs coordinates IT policy. The Dutch Organisation for
ICT and Government (ICTU), created in 201, coordinates inter-departmental
programs. 89 . Several quite detailed studies of the development and application
of ICT innovations in the Dutch public sector are available 90 .
4.
The Dutch government has been an active participant in the innovation systems
supporting water management and construction/housing. The links between
public policy, research, regulation, procurement and the private sector are well
recognised and there are many mechanisms for dialogue and collaboration
among the many players in these sectors 91 .
88
Eshuis P.H. & Muizer, A. P. Nd How Innovative is the Public Sector. This is a short
paper which provides very limited analysis of the database.
89
CAIMED (c.2004) Best Practices in The European Countries. The Netheralnds.
http://www.caimed.org.
90
For example: Korteland, E. & Bekkers, V. (2007) Diffusion of E-government
innovations in the Dutch Public Sector: the case of digital community policing.
Information Polity 12: 139-150. Bekkers has written several other reports.; van
Duivenboden, H. and Thaens, M. (2008) / ICT-driven innovation and the culture of
public administrationInformation Polity 13 (2008) 213232
91
See for example: Bossink, B. (2002) A Dutch Public-Private Strategy for Innovation in
Sustainable Construction. Construction Management and Economics 20: 633-642
70
The Dutch government became convinced that the goal of greater environmental
sustainability required major change in the functional systems of energy,
transport and agriculture, and as a result conceptualised the quest toward
sustainability as an issue of managing transitions in functional systems. 92 . This
new approach (it dates from c.1999) was the result of a process of policy
learning associated with: a more systems orientation to both innovation and the
socio-technical systems of energy etc; a foresight-based approach to long term
assessment, and; a high level of inter-departmental cooperation. In essence the
focus is on system innovation rather than innovation simply at the level of a
particular technology or specific policy. Although beginning largely in the
environment department the strongest proponent of transition management
became the Ministry of Economic Affairs.
This changed perspective had major implications for the policy and the role of
government. Policy became more concerned with transition paths, rather than
specific outcomes. Government became one of the coordinators of transition
processes in a long term cooperative approach which involved joint learning,
investment and coordination with other social groups (particularly business). The
new approach involved a shift in policy toward a more active approach to
learning, about a variety of options and in collaboration with others ie learning
at a system level about systemic change, involving a new structure of collective
governance emerges whereby government is at the same time facilitator [eg
through network building] and one of the players. In particular government
facilitates a process involving a continuous cycle of:
Problem structuring, organising transition arenas, developing
sustainability visions;
Creating arena of arenas, developing transition images, and agendas;
Mobilising transition networks and executing projects and experiments;
Evaluating, monitoring and learning. 93
It appears that this explicit approach to managing transitions leads to a changed
policy process:
The Dutch transition approach is innovation-oriented and bottom-up with longterm visions guiding societal experiments. Various paths are explored
simultaneously to avoid lock in adherence to certain paths. This makes sense
given the uncertainty about what option is best. In doing so Dutch authorities
rely on the wisdom of variation and selection processes rather than the
intelligence of planning. A mechanism of self-correction based on policy
learning and social learning is part of transition management. It offers a
framework for policy integration, helping different Ministries to collaborate.p26
This approach is also as an example of reflexive governance or evolutionary
governance 94 .
92
71
The Chapeau awards for quality in service provision, awarded every two
years starting in 1996
The Kordes awards for annual reports, launched in 1996
The Nachwacht awards, for programs in the health and welfare sectors
The award for innovation in Police and Security services, first launched in
1992
The INK awards for quality in the Dutch public administration
www.ink.nl
In 2002 the government introduced an award for Innovation in the Public Sector.
There has been a strong emphasis on the quality of services and on the
continual adaptation of services and service delivery to meet community needs.
Several mechanisms are used to drive continual improvement:
Surveys on citizen satisfaction
A survey on satisfaction is undertaken every two years. A total of 14,000
questionnaires are distributed to executives and officials in various public
administrations, including the Education, Defence, and Security and Police
Ministries. At the local level surveys on citizen satisfaction are widely used
in order to improve services provided to citizens and to direct
modernisation and change processes.
Benchmarking and the use of best practices for improving public
administration quality
Benchmarking activities are often included in broader projects managed
by single ministries.
Initiatives to introduce benchmarking and best practices
The INK quality model, promoted by the Dutch Institute for Quality, is
widely used in the Dutch public administrations. This model is very similar
to the EFQM (European Foundation for Quality Management) model and is
currently used by about 50% of public organisations in order to guarantee
quality services. w w w . i n k . n l
Groups outside of government shaping change
The Public Cause is a Dutch group of citizens aiming to increase the
quality of public services and policy. www.publiekezaak.nl. The
OmslagGroep (Foundation of Change) is an organisation of companies
72
5.
It is clear that in the Dutch social context there are high expectations of the
quality of government policy and services and also on the quality of engagement
with the community in both policy and service development. Government is a
participant, a facilitator, in many areas of Dutch economic and social activity.
Hence, the context is one of a small and concentrated community with a long
history of engaged and consultative government.
6.
73
7.
Lessons of Experience
The most useful lessons from the Dutch experience appear to be the approach to
policy and service innovation through collaborative relationships with business
and community groups. In the area of water management, health and
particularly sustainability approaches to policy innovation are largely based on
collaborative strategies.
8.
Sources
1. Bekkers, V. and V. Homburg, eds, The information ecology of egoverrnment, IOS Press, Amsterdam/Berlin/Oxford/ Tokyo/Washington
DC, 2005.
2. Bekkers, V. (2007) Modernization, public innovation and information and
communication technologies: The emperors new clothes? Information
Polity 12 (2007) 103107
3. Kemp, R. & Loorbach, D. (2005) Dutch Policies to Manage the Transition
to Sustainable Energy. In Beckenbach, F. et al Jahrbuch Okologische
Okonomik: Innovationen und Transformation. Band 4 Metropolis,
Marburg, 125-150.
4. Kickert, W. & Toonen, T ( 2007) Public Administration In The Netherlands:
Expansion, Diversification And Consolidation. Public Administration.
Volume 84 Issue 4: 969-987
5. van Duivenboden, H. V. Bekkers and M. Thaens, Creative destruction of
Public Administration Practices, in: Informationand Communication
Technology and Public Innovation, V. Bekkers, H. van Duivenboden and
M. Thaens, eds, IOS Press,Amsterdam/Berlin/Oxford/Tokyo/Washington
DC, 2006, pp. 230243.
EGPA Conference
Innovation in the Public Sector, September 3-6 2008
Victor Bekkers, Arthur Edwards, Rebecca Moody, Henri Beunders: New media, micro-mobilization and political
agenda setting: how young people have used Web 2.0 to
change the education agenda in the Netherlands
Alexandra Collm: Strategizing in the Public Sector: Roles within Top Management Teams in the IT Strategy
Process
95
Bekkers,V.(2007)Modernization,publicinnovationandinformationandcommunicationtechnologies:The
emperorsnewclothes.InformationPolity12:103107.
74
75
76
Anne Marie Berg: Organizing for innovation - autonomy versus control in public sector organizations
Yetano, Ana, Acerete, Basilio, Royo, Sonja: WHAT IS DRIVING THE INCREASING PRESENCE OF CITIZEN
PARTICIPATION INITIATIVES?
P.H. Eshuis, A. P. Mouser: How innovative is the public sector? Lessons to be Learned on the basis of a crosssectoral evaluation
Monica Dimitriu: Bringing citizens closer to public administration. Innovative ideas leading to an increased public
participation within the decision making process
Lars Fuglsang, John Storm Pedersen: How different is public and private innovation?
Gnan Luca, Alessandro hinna, Debora Tomasi: BOARDS 'BEHAVIOR RELEVANCE
ON PUBLIC SECTOR INNOVATION PROCESSES. Theoretical issues for a new research agenda
Kattel raiser, Liisa Vask: What is public sector innovation?
Fabio Monteduro, Alessandro hinna: STAKEHOLDER DIALOGUE AND INNOVATIONS IN GOVERNANCE
Wilma van der Scheer, Mirko Noordegraaf, Pauline Meurs: Institutional innovations in health care. How health
care executives PERCEIVE Pursue innovation and legitimation
Georgios Vardaxoglou, Mira Slavova, David Allen, Tom Wilson: Perceptions and adoption of technological
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102