Alexandra Polido

1
, Toms B. Ramos
2
CENSE, Center for Environmental and Sustainability Research, Departamento de Cincias e Engenharia
do Ambiente, Faculdade de Cincias e Tecnologia, Universidade Nova de Lisboa, Campus da Caparica.
2829-516 Caparica, Portugal.
Email:
1
apolido@fct.unl.pt;
2
tabr@fct.unl.pt
Evaluation of Strategic Environmental Assessment
Scoping Effectiveness
Introduction
Strategic Environmental Assessment (SEA) practice has increased in the last years (Retief, 2007) and a concern about
SEA effectiveness has emerged. This subject has been largely addressed and discussed in the institutional and
scientific literature (Dalal-Clayton & Sadler, 2005; Retief, 2007). Different authors sought to establish a concept for SEA
effectiveness (e.g. Baker & McLelland, 2003; Therivel, 2004; Retief, 2007; Fischer, 2010; Theophilou et al., 2010) but
generally, the proposal made in 1996 by Sadler, is the most adapted. Sadler (1996) proposed three types of
effectiveness procedural, substantive and transactive and three levels of assessment: system-wide reviews, decision
audits and component specific. Fischer & Gazzola (2006) suggested that effectiveness evaluation criteria is not equally
valid for all contexts and, furthermore (Dalal-Clayton & Sadler, 2005), there is a need to develop criteria according to the
national practices and legal framework. As pointed out by Retief (2007) there is no systematic approach to develop SEA
effectiveness evaluation and, therefore, it is challenging to design a research strategy and methodology.
In Portugal the practice of SEA is very recent, it was introduce in 2007 with the transposition of the EU Directive
2001/42/CE and has became a legal requirement for the environmental evaluation of certain plans or programs. By
September 2010, and according with the public database of the Portuguese Environmental Agency (APA) at the time,
there were thirty SEA processes completed. At present, there are over sixty-five processes.
Due to the recent outburst of SEA practice in Portugal and since the methodologies still are being developed and
discussed, there is a need to understand what is being done, how it is being developed and how it affects the overall
process of planning with the aim of improving the outcomes.
Methodology
The lack of systematic research on SEA effectiveness evaluation demands a flexible research approach (Retief, 2007)
and it is necessary to develop evaluation criteria according to the specific context as recommend by Fisher & Gazzola
(2006). In this research a set of criteria for the procedural effectiveness evaluation was developed. The research was
developed in three main stages: case study selection; design of the criteria for evaluation of the scoping effectiveness;
and, content analysis.
Case study selection
Twenty SEA SR were selected that met the following requisites:
Territorial coverage (ensuring a spread distribution in Portuguese territory);
Diversity of technical teams ensuring a non-predominance of a team;
Reports availability at APA.
The case studies are divided into 16 SR of Local Spatial Plans (LSP) and 4 SR of non-LSP.
Scoping effectiveness evaluation criteria
Twenty-one criteria were developed, divided into six dimensions (Table 1): (i) Plan or Program (PP) context, (ii) the
strategic reference framework, (iii) sustainability issues and SEA objectives, iv) baseline information, (v) communication
and public participation and (vi) integration of SEA with the different phases of the PP. These criteria were
operationalized by questions, that were developed taking into account the national regulations and guidelines, and
international guidelines and studies.
Content analysis
A content analysis was developed to reduce the subjectivity of the study. Content analysis is a systematic approach
used to retrieve relevant information from the documents to analyse (GAO, 1996). Due to its wide scope, different
scientific areas may use it (Fischer & Gazzola, 2006). The methodology used in this research was adapted from GAO
(1996), Roberts (1997) and Neuendorf (2002). The content analysis developed is qualitative and, therefore, the answers
expected to the evaluation criteria were Meet requirements, Partially meet requirements and Does not meet
requirements.
a) CEQ (1981); b) Sadler (1996); c)
DEAT/CSIR (2000); d) Fischer (2002); e)
Verheem & IAIA (2002); f) Baker &
McLelland (2003); g) Abaza et al. (2004); h)
Therivel (2004); i) Ahmed et al. (2005); j)
ODPM (2005); k) Schmidt et al. (2005); l)
GRDP (2006); m) OECD (2006); n) Scottish
Executive (2006); o) APA (2007); p) Retief
(2007); q) DGOTDU (2008); r) Theophilou et
al. (2010).
Results and Discussion
There are two Good Practice Guides in Portugal (APA 2007; DGOTDU, 2008), where the first has a wide-range scope
and the latter is focused on SEA of LSP. For the SR of LSP completed after 2008 it was expected that they use the good
practice guide from DGOTDU (2008), however, all twenty LSP-SEA SR used the good practice guide from APA (2007).
The Portuguese guidelines do not include a systematic approach for presenting information on the SR (as e.g. Scottish
Executive, 2006; ODPM, 2005), even though DGOTDU (2008) have guidelines for the presentation of some points of
the SR, which results in unnecessary and descriptive information rather than objective and focused.
The summary of the content analysis is provided in Figure 1. Generally, it shows that what is mandatory or is present in
the Portuguese guidelines are largely met, but what constitutes a best-practice, that creates proximity between the
process and the stakeholders, is not met, i.e., the communication and public participation criteria is largely dismissed,
showing that only what is mandatory is prepared.
The first and second sets of criteria (criteria from 1 to 6) are mandatory for the Environmental Report (ER) phase,
however the Portuguese guidelines put them in the SR phase. They were largely met. In the third set of criteria (from 7
to 9), only criterion 7 is fully met. This criterion is, according to APA (2007), what defines the scoping phase and if it is
not present constitutes a significant gap.
The fourth set of criteria (criterion 10) is not met. Some authors put this criteria in the scoping phase (e.g. CEQ, 1981;
ODPM, 2005; GRDP, 2006; DGOTDU, 2008), but APA (2007) does not. It may be discussed the necessity of this type of
information so early in the SEA process since it must be further developed in future phases of the process.
The fifth set is of most importance, since one of the aims of SEA is to promote public participation in order to develop
transparent processes. Furthermore, criteria such as criteria 15, 16, 17 and 19 reflect the possibility of assessing if,
during the scoping phase, public participation was carried out and how its results were introduced in the process. They
are not met, with the exception of criterion 12.
It was expected that the sixth set of criteria would be fully met, since it is of most importance that the planning process
and SEA process are completed simultaneously. This is not what was found. None of the SR identifies the integration
moments between the planning process and SEA.
Aim
The aim of the present research was to analyse how the scoping phase in SEA is developed and conducted in Portugal
through the effectiveness assessment of twenty selected Portuguese SEA Scoping Reports (SR). This evaluation was
supported by a content analysis.
Conclusions
The 21 criteria developed covered the whole scoping process, since the beginning of the planning process. It can be
discussed if some of the criteria should be present so early in the SEA process (e.g. criteria 10) or even if they make
sense to be used in all types of SEA (e.g. criteria 14). However, they were introduced assuming that they should be
analysed with some flexibility and refocused in future evaluations. This is of most importance since the SEA
methodologies are still being developed and hence, the criteria may be adapted and improved and may integrate
guidelines for the evaluation of SEA scoping effectiveness.
The evaluation carried out to the different types of SEA SR shows that SR of LSP and SR of non-LSP used the same
guidelines, despite that it could be expected that SR of LSP completed after 2008 used the DGOTDU (2008) guidelines.
This may happen because SEA practice is recent in Portugal and the users could be driven to adopt the first published
guides or the ones that practitioners assume as more reliable.
In order to make the scoping phase effective it is necessary to develop methods for public participation in all its steps
and a systematic coordination with the planning team. Also, to improve the outcomes, a set of strict guidelines could be
developed, weighting socio-cultural, decision-making and geographic differences, as discussed by Fischer & Gazzola
(2006).
CRITERIA ADAPTED FROM
Plan or Program (PP) context
1. Describes the PP content? j) l) n) o) q)
2. Describes the PP objectives? j) l) n) o) q)
3. Describes the strategic issues? c) o) q)
Strategic reference framework
4. Refers the relevant Policies, Plans or Programs (PPP)? d) e) j) l) m) n) o) q)
5. Refers the sustainability and strategic objectives of the
relevant PPP?
d) e) j) l) m) n) o) q)
6. Refers how these PPP are linked with the PP under
evaluation?
j) l) m) n) o) q)
Sustainability issues and SEA objectives
7. Identifies the key sustainability and environmental issues
relevant for the SEA?
d) e) g) h) j) l) n) o) q)
8. Identifies the indicators and sources of information for each
key sustainability and environmental issue?
d) e) h) o)
9. Defines the SEA objectives? d) e) g) h) j) l) n) o) q)
Baseline information
10. Identifies the baseline information and its probable evolution
without PP?
c) g) h) j) l) n) o) q)
Communication and public participation
11. Defines the communication strategy for the all SEA
process?
f) k) m) n) o) q)
12. Establishes the need for consultation to competent
authorities in the scoping phase?
b) c) d) e) f) g) h) j) l) o)
q)
13. Establishes the need for consultation to the general public
in the scoping phase?
b) c) d) e) f) g)
14. Establishes the need for consultation to institutions and
expert consultants in the scoping phase?
b) f) g)
15. During the scoping phase stakeholders were consulted? f) i)
16. Identifies how the outcomes were introduced in the scoping
report?
a) f) i)
17. Describes how consultation was conducted? a) f)
18. Promotes participation in the scoping phase? a) f) m) o) r)
19. Focus the contributions and concerns of stakeholders? a) f) p) r)
Integration of SEA with the different phases of the PP
20. Defines a methodology/timeline for the all SEA process? a) h) m) n) o) q)
21. Are the PP and SEA being developed simultaneously? h) j) l) n) o) p) q) r)

Table 1 - Scoping effectiveness evaluation criteria
Figure 1 Overall results for the SEA SR content analysis
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
N
u
m
b
e
r
o
f

S
E
A

S
R
Criteria
Meet requirements Partially meet requirements Does not meet requirements
References
Abaza, H., Bisset, R., & Sadler, B. (2004). Environmental Impact Assessment and Strategic Environmental Assessment: Towards an Integrated Approach. Geneva: UNEP.
Ahmed, K., Mercier, J. R., & Verheem, R. (2005). Strategic Environmental Assessment - Concept and Practice. Washington: World Bank.
Baker, D., & McLelland, J. (2003). Evaluating the effectiveness of British Columbia's environmental assessment process for first nations' participation in mining development. Environmental
Impact Assessment Review , 23, pp. 581-603.
CEQ - Council on Environmental Quality. (1981). Memorandum for General Counsels, NEPA Liaisons and Participants in Scoping. Washington D.C.: Executive Office of the President.
Dalal-Clayton, B., & Sadler, B. (2005). Strategic Environmental Assessment: a sourcebook and reference guide to international experience. London: Earthscan/International Institute for
Environment and Development.
DEAT/CSIR - Department of Environmental Affairs and Tourism/ The Council for Scientific and Industrial Research. (2000). Strategic Environmental Assessment in South Africa: Guideline
document. Pretoria: Department of Environmental Affairs and Tourism.
DGOTDU Direcco-Geral do Ordenamento do Territrio e Desenvolvimento Urbano. (2008). Guia da Avaliao Ambiental dos Planos Municipais de Ordenamento do Territrio. Lisboa:
DGOTDU/APA.
Fischer, T. (2002). Strategic Environmental Assessment Performance Criteria - The same requirements for every assessment? Journal of Environmental Assessment Policy and Management
, 4 (1), pp. 83-99.
Fischer, T. (2010). Reviewing the quality of strategic environmental assessment reports for English spatial plan core strategies. Environmental Impact Assessment Review , 30, pp. 62-69.
Fischer, T., & Gazzola, P. (2006). SEA effectiveness criteria - equally valid in all countries? The case of Italy. Environmental Impact Assessment Review (26), pp. 396-409.
GAO United States General Accouting Office. (1996). Content analysis: a methodology for structuring and analyzing written material. Washington: US GAO.
GRDP - Greening Regional Development Programmes. (2006). Handbook on SEA for Cohesion Policy 2007-2013. Greening Regional Development Programmes.
Neuendorf, K. (2002). The content analysis guidebook. Thausand Oaks, California: Sage Publications.
ODPM- Office of the Deputy Prime Minister. (2005). A Practical Guide to the Strategic Environmental Assessment Directive. Londres: Office of the Deputy Prime Minister.
OECD - Organisation for Economic Co-Operation and Development. (2006). Applying Strategic Environmental Assessment: Good Practice Guidance for Development Co-Operation. OECD.
APA (2007). Guia de Boas Prticas para Avaliao Ambiental Estratgica - Orientaes Metodolgicas. Amadora: Agncia Portuguesa do Ambiente.
Retief, F. (2007). A performance evaluation of strategic environmental assessment (SEA) process within the South Afican context. Environmentl Impact Assessment Review , 27 (1), pp. 84-
100.
Roberts, C. (1997). Text analysis for the social sciences: methods for drawing statistical inferences from text and transcripts. Mahwah, New Jersey: Lawrence Erlbaum Asociates, Inc.,
Publishers.
Sadler, B. (1996). International Study of the Effectiveness of Environmental Assessment - Environmental Assessment in a Changing World: Evaluating practice to improve performance.
Minister of Supply and Services, Canada.
Schmidt, M., Joo, E., & Albrecht, E. (Eds.) (2005). Implementing Strategic Environmental Assessment. Berlim: Springer.
Scottish Executive/Natural Scotland. (2006). Strategic Environmental Assessment Tool Kit (version 1). Edinburgh: Scottish Executive.
Theophilou, V., Bond, A., & Cashmore, M. (2010). Application of the SEA Directive to EU structural funds: Perspectives on effectiveness. Environmental Impact Assessment Review , 30, pp.
136-144.
Therivel, R. (2004). Strategic Environemtnal Assessment in Action. London: Earthscan.
Verheem, R., & IAIA. (2002). Avaliao Ambiental Estratgica: critrios de desempenho. Fargo: IAIA.
Alexandra Polido
apolido@fct.unl.pt
xanapolido@gmail.com
Toms B. Ramos
tabr@fct.unl.pt
Acknowledgments:
We would like to acknowledge the valuable collaboration of Clara Cintro from the Portuguese Environmental Agency