Managing to Outcomes at

Calgary Counselling Centre

Karen Mason
Student Number: 10124284
University of Calgary
School of Social Work

SOWK 695: Becoming an Evidence-Based Leader

Dr. Robbie Babins-Wagner
August 9, 2014

2
Within the social service sector there has been a move toward measuring client
outcomes. This move has increased accountability, organizational effectiveness, and ability
to demonstrate mission achievement. Through carefully selecting outcome measures,
organizations can work toward creating programs and services that benefit the vulnerable
populations that they serve. Calgary Counselling Center (CCC) has collected outcome
measure for approximately 10 years seeking to improve client services. This report will
review the rational for utilizing outcome measures, the model of change proposed for use
with depression clients, key findings in the 2011 annual research review,
recommendations, implementation plan, and the impacts of implementing these
recommendations.
Rational for Utilizing Outcome Measures
To work strategically we must understand what we seek to achieve. The mission at
the CCC is improving the well-being of individuals and families and strengthening
communities by delivering best practices in counselling, training, and research (Calgary
Counselling Centre, 2014). CCC evaluates its services through a performance management
approach. It is a matter of creating the conditions where basic operating parameters are
designed so that work drives the organizations strategic interests-so that work optimizes
the organizations ability to achieve its goals and meet its obligations (Hunter, 2013, p. 29).
Outcome measures are selected, collected, evaluated, and then utilized to shape service
delivery. Managing to outcomes is an overarching ethic or rigorously pursuing
meaningful, measureable good for those we serve (Morino, 2011, p. 59). This approach
mirrors the philosophical orientation of CCC.

3
It is essential that as a nonprofit agency CCC works in the most efficient and
effective means possible. If outcomes are not measured non profits can end up providing
the same services for years without ever really knowing if they could be doing something
different that would lead to greater benefits (Morino, 2011, p. 95). Even more disturbing
is the assertion by Hunter that while hundreds of thousands of social service organizations
work incredibly hard to help structurally disadvantage and socially marginalized
individuals, families and groups build better lives and life prospects for themselves and
their children, few do so effectively (2013, p. xiv). CCC has worked to develop the
organizational capacity to manage to outcomes and this approach must continue to be
utilized to ensure effective and efficient services.
Most importantly, measuring to outcomes provides a means of ensuring ethical
practice at CCC. Despite the best of intentions, nonprofits will make mistakes, and those
mistakes can cause harm to clients (Morino, 2011, p. 96). We must measure the outcomes
of our service to be able to identify if and when unintentional harm is occurring, and
correct these practices. To do any less would be unethical. As Morino asserts
poor performance by social sector organizations under cuts the initative and
hopes of their intended beneficiaries; it demoralizes those who are trying as hard as
they can to better themselves and improve their lives; and ultimately it helps sustain
social inequality (2011, p.12).
Providing quality, effective, evidence-based practices is not only an ethical imperative, but
it is a cornerstone of CCC vision.
The tool selected by CCC to collect client outcome measures is the Outcome
Questionnaire developed by Lambert. The OQ was designed to measure three areas of

4
patient functioning: symptomatic distress, interpersonal problems, and social role
adjustment (Lambert, Okiishi, Finch & Johnson, 1998, p. 64). It is designed to measure key
aspects of client wellness. This tool appears to capture the right measures as it is a direct
measure of CCCs ability to achieve its mandate and improve individual and family
wellbeing. As well, the OQ is reliable; test-retest coefficients were in the .70s and .80s and
internal consistency in the low .90s (Lambert et al., 1998 p. 64). Moreover it was found to
be relatively stable in nontreated individuals while being sensitive to change in patients
undergoing treatment (as cited in, Lambert et al, 1998, p. 64). It is capable of indicating
that there is a greater than chance likelihood that change resulted from treatment
provided. The tool is both valid and reliable with strong internal and external consistency.
This tool is administered to each client upon arrival for each appointment, and it serves as
the basis for the annual research review being examined.
Model of Change for Clients with Depression
The most common definition of depression is the one provided by the Diagnostic
and Statistical Manual of Mental Disorders IV-R (DSM-IV-R). It terms depression as a
Major Depressive Disorder and it is classified as a mood disorder. To qualify for a diagnosis
clients must have five or more symptoms of the nine criteria described and at least one of
the symptoms is either depressed mood or loss of interest or pleasure (American
Psychological Association, 2000, p. 168). The criteria includes
significant loss of weight or change in appetite; inability to get to sleep; or sleeping
too much; psychomotor agitation or retardation; loss of energy or fatigue; feelings of
worthlessness or excessive or inappropriate guilt; decreased of centration or
indecisiveness; or recurring thoughts of death (Scott & Solovey, 2007, p. 138).

5
It is this cluster of symptoms that is often used to describe the experiences of people who
are depressed. The definition provided by the DSM-IV-R is representative of the medical
model. In this model biological explanations and interventions are sought. The
monoamine hypothesis of depression posited that depression is caused by a deficiency of
the monoamines, noradrenaline, serotonin or both, in the brain and that antidepressant
drugs restore these to normal (Palazidou, 2012, p. 136). Recommended treatment
involves early and effective antidepressant drug treatment (Palazidou, 2012, p. 142). This
approach to the treatment of depression has dominated the field of psychiatry.
However, other conceptualizations of depression focus on intrapersonal thought
processes and relationships. Depression is often described as the triad of cognitions
described by Beck the self as worthless, the world as pointless and the future as hopeless
(Harris, 2001, p. 18). The two leading treatments for depression, cognitive-behavioral
therapy (CBT) and interpersonal psychotherapy (IPT), focus their treatment on each of the
two factors supposed to underlie depressions: cognitions and relationships (Scott &
Solovey, 2007, p. 136). Differences in brain chemistry are attributed to the quality and
quantity of relationships. Our social lives directly shape our brain chemistry and
powerfully affect he way we think and feel (Yapko, 2009, p.39).
Support for the concept of depression being related to environment can be found in
the writings of social theorists. Harris, 2007, indicates the one of the best known statistics
in psychiatric epidemiology is the two-fold greater rate of depression among women as
compared with men (p. 547). She notes gender differences and role identities, along with
attachment styles as being influential in the development of depression. In her 2001 work
she also notes the role of social class differences, social capital, and past adversity, and

6
social supports play in the prevalence of depression (Harris, 2001, p 23). It is apparent that
the social environment, and the persons relationship to those around them is a key aspect
in the development, diagnosis, and treatment of depression.
Much research has focused on determining model of treatment that is most
effective, and a relational element emerges. Martin, Garske, & Davis note an inability of
researchers to find a consistent difference in the effectiveness of psychotherapy across
orientations in the past 20years (2000, p. 438). The factor that has emerged as influential
is the therapeutic alliance. It is defined broadly as the collaborative and affective bond
between therapist and patient (p. 438). As Martin et al. indicates here is a consistent
finding that the quality of the alliance is related to subsequent therapeutic outcome (2000,
p. 438). It appears an essential factor in the successful treatment of clients is tied to a
relational context.
As such, at CCC improving relationships, and not biological interventions, is the
focus of treatment. Yapko asserts that skill and relationship building therapies can match
the success rates of antidepressant mediations and can even exceed their effectiveness in
key areas (2009, p. 964). People to people connectedness can outdo pharmaceuticals in
treating depression (Yapko, 2009, p. 19). These results are achieved without the harmful
side effects that are common in antidepressant use. As well, for individuals struggling with
low incomes can avoid the costs associated with medication. Furthermore, specific trials
that have combined medications and psychotherapy, do not typically find a statistically
significant effect (Dobson, Scherrer, & Haubert, 2006, p. 244). As there is no clear evidence
that biological interventions are advantageous for clients they are not a focus of treatment
at CCC. Instead the focus is on building relationships. Individual therapies focus on

7
building and maintaining a therapeutic alliance, regardless of the theoretical orientation of
clinicians.
Group sessions are also utilized to teach skills, shift individual cognitions, and
increase exposure to relationships, in a context where a therapeutic alliance exists. This
approach is supported by evidence that shows group psychotherapy is efficacious in its
ability to reduce depressive symptomatology in depressed individuals, and that treatment
gains are typically maintained for several months to one year (McDermut, Miller, & Brown,
2001, p. 113). This finding is in line with viewing depression through a relational lens.
How people develop their best selves is largely, though not entirely, achieved in the
context of positive relationships with other people (Yapko, 2009, p. 44). Group sessions
are posited to be evidence based and are an effective form of treatment that are a central
piece of service delivery at CCC.
Model of Change

Individual Counselling
Theraputic Alliance

Change Cognitions, Relationship skills

Group Sessions
Theraputic Alliance

Practice skills, Exposure to

relationships

Improved Client Wellbeing

As measured by the OQ Scores

8
Annual Research Summary Report - Depression
Client Demographics
It is interesting to note that there was an increase in the number of participants in
2011 that were single (60.5%) compared to the aggregate (50.8%) and most
participants had a grade 9-12 education (37.8%) or a university level education (37.8%)
and most participants were working full time (37.8%) while the majority of 2011s
participants were earning $5,001-$25,000 (Calgary Counselling Centre, 2012, p. 9). This
increase of single educated employed people may be reflective of a strong economy and
growing population-seeking employment. However, the majority of people are earning low
wages that cannot sustain an individual or family in this economy. If this trend continues it
may be worth further investigation as there may be additional referrals to outside agencies
required for this group, or other service adaptions required.
It also appears that we are serving a client group that is in significant need of
services. For the five year aggregate, a total of 4039 depression clients 87.6% had OQ
scores that were above the clinical cut off (Calgary Counselling Centre, 2012, p. 4). This is
a substantial sample size and it appears that the majority of clients report a level of distress
that indicates they are in need of clinical services. The same can be said for group
participants. For the aggregate group 94.4% of client had an initial OQ score over the
clinical cut off (Calgary Counselling Centre, 2012, p. 6). To be effective and efficient in our
service we want to be working with clients who are most in need of services. The data
support the assertion that we are working with the targeted population for people with
depression.

9
Client Change
It appears that the services provided are making change for many clients. For those
in the aggregate group, receiving individual counselling, the mean OQ score improved by
16.0 points from 88.1 to 72.1 (Calgary Counselling Centre, 2012, p. 5). With the OQ tool a
change of 14 points is sufficient to indicate that the change that occurred is greater than the
probability it occurred due to chance. It is termed the reliable change index and it is a
measure that indicates clients have improved, even though they may not achieve an overall
score that is below the clinical cut off. Moreover, for the 5 year aggregate 51% of clients
improved or recovered by their last session for 2011 53.2% of clients improved or
recovered by their last session (Calgary Counselling Centre, 2012, p. 5). This percentage is
comparable to industry benchmarks. It appears we are producing change for clients, but
there also remains room for improvement.
When client outcomes are sorted into the types of services clients receive,
comparing individual and group services, an interesting picture appears. For the 5 year
aggregate, from first session to group pre-test, 5 clients (10.9%) improved or recovered by
at least 14 points (Calgary Counselling Centre, 2012, p. 7). The individual counselling
appears to be less effective in producing client change than group processes with only 10%
of clients improving. As individual counselling is a labor-intensive process it may be worth
exploring if a reduction in the number of individual sessions can still produce overall
positive outcomes for clients.
The greatest gains appear to be achieved in a group setting. More clients reliably
improved from first counselling session to the end of group (post-test), than from first
counseling session to the first session of group (pretest), for the aggregate (Calgary

10
Counselling Centre, 2012, p. 7). It appears that attending group improves outcome
measures. This finding is supported by the qualitative data where clients indicated they
had an increased ability to face lifes challenges and through skill development, increased
self-awareness and a feeling of support (Calgary Counselling Centre, 2012, p. 10). While
some change occurs in individual counselling, services are most effective when individual
and group services are provided.
Individual and group services also appear to be beneficial even to those who are
significantly unwell. Of the aggregate group 28.2% of clients who begin group have a
clinically significant issue, 54.8% have scores high enough to indicate that they may have
some suicidal ideation, and 12.1% have severe distress, with suicidal ideation being likely
(Calgary Counselling Centre, 2012, p.15). A large number of very unwell clients are
accessing services, and they are making improvements through group settings. All
aggregate data showed statistically significant improvements, indicating that participating
in the group led to positive changes in mood, levels of self-esteem, stress levels, and overall
treatment progress (Calgary Counselling Centre, 2012, p. 17). The data supports the
assertion that a group context, with a focus on relationships and skills helped even clients
in severe distress.
As it appears that group is beneficial to clients we must consider what we know
about clients who complete group vs. those who do not complete. It does not appear that
level of client distress is a factor. There was no significant difference in the number of
completers and on-completers who scored above the clinical cut-off on their initial OQ
score (Calgary Counselling Centre, 2012, p. 6). Similarly, the number of individual session
prior to attending group does not appear influential. In the aggregate group the clients

11
that completed group attended an average of 12.9 individual counselling sessions, while
the clients that did not complete group attended an average of 13.4 individual counselling
sessions (Calgary Counselling Centre, 2012, p. 4). It appears that further research into the
factors that lead to people completing group is required. An initial step in this process
would be to meet with clinical staff to gain their insights into the factors they believe are
influential.
The most striking finding emerges when we consider the impact of returning to
individual counselling after completing group services. In the five-year aggregate group
The man OQ score actually increased by 3.82 points between the final session of group and
the final counselling session (Calgary Counselling Centre, 2012, p. 8). For the 2011 group
a similar trend appeared with the mean OQ score deteriorated 4.9 points between the final
session of group and the final session of counselling (Calgary Counselling Centre, 2012, p.
8). Both the aggregate group and the 2011 group had similar findings, which suggest;
added counselling after group for these clients actually resulted in deterioration (Calgary
Counselling Centre, 2012, p. 8). Currently, we recommend that all clients completing group
return to their individual Counsellor for follow up sessions. However, it appears that this
practice may actually be detrimental to our clients.
Moreover, there needs to be continued improvement of the services we provide to
our clients who have depression. Overall, depression clients in the 5 year aggregate
improved on their OQ scores from the first counselling session to last counselling session
by 20.3 points, however the OQ scores at the final counselling session ere still above the
clinical cut-off of 63 (Calgary Counselling Centre, 2012, p. 10). While change is occurring,
many clients with depression do not achieve a level of wellness that is below the clinical cut

12
off. This measure is indicative of a level of wellness that does not require clinical services.
Many clients do not fall below this mark, indicating that they have not achieved recovery.
Obviously more work is required to improve services and increase the number of clients
achieving this goal.
Recommendations
The findings support the assertion that working to improve skills and cognitions
and improve relationships has merit. Clients are making reliable change and improving in
their wellbeing, without a specific focus on providing medications. Individual counselling,
and group sessions which improve clients relationship skills and opportunity to build new
supportive relationships, appears to have a significant impact on client outcomes. It
appears that clients who are both moderately and severely unwell are benefiting from
services that focus on relationships. As such, it is recommended that this model of
intervention continue to be used by CCC.
In fact, clients made greater gains from group settings than in individual counselling.
As such, it appears necessary to determine the minimum number of individual sessions
that are required to support continued improvements in client outcomes. As it appears
more effective, efforts must be made to increase the use of group sessions for clients. As
such, it is recommended that staff be required to recommend group sessions to clients.
It is imperative that the policy requiring clients to return to individual counselling
after completing group work be eliminated. It appears that we are unintentionally creating
harm for those we seek to help as clients are degenerating upon their return for services. Of
course, clients will be given the opportunity to return for services if they request to do so,
as is in line with our client centered philosophy, however this will no longer be a

13
requirement. Once this change is implemented it would be prudent to continue to track if
those who choose to return without prompting continue to degenerate. If this is the case
additional changes to how we provide services may need to be made.
A number of recommendations for future research can be made. Firstly, it is
important to continue to track client demographics. If a trend continues to emerge that is
reflective of an increase of single, educated, but low earning population continues,
adaptations to services may required. Secondly, it may be necessary to delve deeper into
the research to determine the minimum number of individual sessions prior to group work
that are effective and to determine what factors contribute to group drop out rates. This
will result in huge benefits to our clients if we are able to identify the fastest and most
effective means of treatment. Lastly, as a large database has been created, it may be
possible to build relationships with outside researchers that track connections between
social determinants of health and depression. This work would be a substantial
contribution to understanding the links between social determinants and mental health in
Canada.
Summary of Recommendations:
1. Continue to utilize the model of change with a focus on relationships.
2. Identify the minimum number of counselling sessions required prior to group to
maintain client gains.
3. Recommend group sessions to all clients experiencing depression.
4. Eliminate the requirement that clients return to individual counselling following
group work.
5. Continue to utilize research to identify best practices.

14
Implementation
Prior to implementing any changes it is essential that the findings of this report be
reviewed with staff. Staff may have essential insights into findings and may be able to
create hypothesis that can be pursued. This process ensures quality services and
understanding the rational for changes increases the likelihood that staff will buy into the
change process. Fortunately, a culture of learning has been created at CCC. Learning
cultures value honest appraisal, open dissent, and constructive feedback. It promotes
intelligent risk taking in pursuit of both insight and impact. It considers the relevant
context of an assessment and makes difficult decisions based on evidence (Marino, 2011,
p. xx). Staff at CCC has been socialized into the use of outcome measures to drive policy and
service delivery decisions. As such, it is reasonable to believe that staff will be willing and
able to cope with the proposed changes.
Continued research is key to implementing the recommended changes. It is likely
that the most effective means of identifying a minimum number of individual sessions
required is through evaluating OQ data. If initial research finds validity in the assertion
that the number of sessions be reduced a pilot may be run comparing a small number of
group sessions versus treatment as usual. This research effort, along with those
recommended earlier is essential to identifying best practices for clients. It is probable that
adding these research requirements are achievable in current time frames and research
practices.
It is likely that the recommendation that staff refer all clients to group services is a
quickly achievable change. Evidence can be provided that group makes effective change for
clients, and it does not require significant changes in process for individual clinicians. To

15
assist with this process a copy of the referral form for group can be provided along with a
script clinicians can use to recommend group. A review of referrals by management can
identify those clinicians who are having challenges referring to group, and this can be
addressed through clinical supervision.
However, changing the policy that currently requires group participants to return to
individual counselling may require more concerted efforts. It will be necessary to identify
the rational for the current policy, which staff members advocated for its creation, and
possible implications of changing the policy. Education on the findings driving this change
and the content of proposed replacement processes will need to occur prior to
implementation. In the words of Jim Collins the good to great companies did not focus
principally on what to do to become great; the focused equally on what no to do and what
to stop doing (2001, p. 11). It is essential that we stop practices that appear to be
detrimental.
The financial implications of these changes must also be considered. Fortunately,
recommendations in one area may produce cost savings that can be utilized to fund
increased services in another area. The changes will result in a reduction of individual
counselling services. The number of individual counselling sessions will be reduced, and
there will also be a reduction as clients are not required to return for follow up
appointments. However, there is likely to be an increase of group services, as staff is
required to encourage clients to attend sessions. As individual services are more labor
intensive and less cost effective than group services it is likely these changes will offset
each other. However, it is important that the financial implications of these changes be

16
monitored. In addition, if more staff are required to provide group services it will be
necessary to ensure training requirements are met for those facilitating groups.
Implications of Managing to Outcomes
Managing to outcomes, that is implementing changes based on the client outcomes
that have been collected, is critical to providing quality care for clients. Improving client
wellbeing is the primary purpose of CCC and managing to outcomes demonstrates a
passion to be a effective as we possibly can for those we serve (Morino, 2011, p. 40). This
must remain central to all that the agency does. A nonprofit should measure outcomes for
a single reason: to improve the quality of services for clients (Morino, 2011, p 95). The
prospect of being able to use outcome measures to become more effective at helping those
in need is not only impressive; it is exciting.
There are also benefits for the staff that work in our agency. By providing staff
with information to help them refine and adjust their work, and organization can empower
staff to continually improve the quality of services they provide to clients (Morino, 2011,
p. 98). This is a thrilling prospect to those in social services. Being part of an organization
that values quality services and continued improvements can increase staff members
sense of efficacy. This has benefits in terms of reducing burn out, and maintaining the
passion required to provide services to vulnerable clients. Creating a culture where
outcomes are measure, once the correct people are on the bus, has the potential to increase
staff retention.
As an organization the potential to continually improve performance and better
meet mandate positions CCC to be an industry leader. Any journey from good to great
requires relentlessly adhering to these input variables, rigorously tracking your trajectory

17
on the output variables and then driving yourself to even higher level of performance and
impact (Collins, 2005, p. 14). Utilizing this approach will assist us to better achieve our
mandate. Mission effectiveness will become an increasingly urgent issue in the decade
ahead. To begin with funders have come under immense fiscal pressures funders and
donors will be forced to choose where and how to cut (Morino, 2011, p. xxii). CCC will be
attractive to funders, as it will have well demonstrated mission effectiveness.
Social workers advocate for those who are marginalized and vulnerable. It is our
responsibility to work towards social justice, and demonstrating effectiveness is essential
to this continued work. Those who rely on social services in order to overcome personal,
economic, and societal challenges need to social sector to embrace performance
management, to manage to outcomes with dedication, commitment, and passion (Hunter,
2013, p. 12). This provides accountability to clients, service providers, governments, and
funders. Public funders and eventually private funders as well will migrate away from
organizations with stirring stories alone, toward well-managed organizations that can also
demonstrate meaningful, lasting impact (Morino, 2011, p.2). If we are to make social
change we must be able to demonstrate the effectiveness of our work and improve our
ability to achieve our mandates.
Through developing outcome measures that truly measure the good we do clients,
and not simply focus solely on fiscal responsibility, we could demonstrate the impact we
have o individual lives, communities and society as a whole. For a business, financial
returns are a perfectly legitimate measure of performance. For a social sector organization,
however, performance must be assessed relative t mission, not financial returns (Collins,
2005, p. 10). With the skills to manage to outcomes in place, the social sector can

18
determine the measures by which our success is assessed as opposed to having indicators
forced upon us from funders seeking accountability and unintentionally pulling us off
mandate. Managing to outcomes could serve as the banner under which many of us with
diverse skills, talents, an offerings could come together to meet Druckers challenge and
convert a promising movement into a potential force (Morino, 2011, p. 40). It is only
through managing to outcomes in a systemic way that true social change can be made.

19
Reference
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental
disorders (4th ed., txt rev.) Washington, DC: Author.
Calgary Counselling Centre. (2012). Annual Research Summary Report, Depression, 2011.
Calgary Counselling Centre. (2014). Retrieved from www.calgarycounsellingcentre.com
Collins, J. (2001). Good to Great why some companies make the leap and others dont.
Random House.
Collins, J. (2005). Why Business Thinking is not the Answer Good to Great in the Social
Sectors. Retrieved from Amazon.com.
Dobson, K.S., Scherrer, M.C., & Haubert, L.c. (2006). Depression. In J. Fisher & W.T.
ODonohue (Eds.), Pratitioners guide to evidence-based psychotherapy. New York:
Springer.
Harris, T. (2001). Recent developments in understanding the psychosocial aspects of
depression. British Medical Bulletin, 57, 17-32.
Harris, T. (2007). Vulnerable to depression. British Journal of Psychotherapy, 23(4), 547562.
Hunter, D. E. K. (2013). Working Hard and Working Well. Hunter Consulting LLC. Rocky
River. OH.
Lambert, M. J., Okiishi, J. C., Finch, A. E., & Johnson, L. D. (1998). Outcome assessment: From
conceptualization to implementation. Professional Psychology: Research and
Practice, 29(1), 63-70.

20
Martin, D.J., Garske, J.P., & Davis, K.M. (2000). Relation of the therapeutic alliance with
outcome and other variables: a meta-analytical review. Journal of Consulting and
Clinical Psychology, 68(3), 438-450.
McDermut, W., Miller, I.W., & Brown, R. A. (2001). The efficacy of group psychotherapy for
depression: A meta-analysis and review of the empirical research. Clinical
Psychology: Science and Practice, 8(1), 98-116.
Morino, M. (2011). Leap of Reason. Managing to outcomes in an era of Scarcity. Washington,
DC. Venture Philanthropy Partners.
Palazidou, E. (2012). The neurobiology of depression. British Medical Bulletin, 101, 127-145
Scott, F. J., & Solovey, A.D. (2007). Second-order change in psychotherapy: the golden thread
that unifies effective treatments. Washington, DC. American Psychological
Association.
Yapko, M.D. (2009). Depression is contagious (I-book version). Retrieved from
SimonandSchuster.com