The Benefits andChallenges of TrainingChild ProtectionSo.docx

The Benefits and
Challenges of Training
Child Protection
Social Workers in
Father Engagement
It is widely recognised that, in a child protection context,
practitioners tend to focus on
working with mothers more than fathers. This may undermine
risk management and limit
the resources available for the care of children. This paper
discusses the process of
developing and running a training intervention for child
protection social workers, designed
to improve father engagement (with ‘fathers’ defined
inclusively). A short course was
provided, consisting of one day of awareness-raising about the
importance of work with
fathers and one day of motivational interviewing skills training.
The emphasis in the paper is
on insights from the qualitative elements of the mixed-method
process evaluation, namely,
observation and pre- and post-course interviews. In particular,
there is discussion of the
potential benefits and challenges of this kind of training, with
consideration given to the
general issue of father engagement and more specifically the
potential for using motivational
interviewing in child protection practice. Copyright © 2012
John Wiley & Sons, Ltd.
KEY PRACTITIONER MESSAGES:

• The pilot of the Cardiff University Fathers and Child
Protection course resulted in
improved engagement of fathers, according to social workers
self-report.
• Motivational interviewing has potential for developing
practitioners’ skills in working
with fathers when children are at risk.
• There are some inherent challenges in attempting to improve
the engagement of
fathers in a child protection context.
KEY WORDS: fathers; child protection; training; process;
motivational interviewing
Children’s services are often criticised for their relatively poor
engagementof men, which can lead to ineffective risk
management and reduced
resources for the care of children. Whilst acknowledging the
importance of
the legal concept of ‘parental responsibility’, we use the term
‘fathers’ in this
paper more inclusively to refer to any men who are involved in
parenting
practices, whether they are biological fathers, step fathers or
mothers’
boyfriends, male friends or relatives. We do so because it is the
failure to
engage men who have a significant role in the child’s life, and
not just those
men who legally have parental responsibility or are biological
fathers, that is
problematic for the safeguarding of children.
* Correspondence to: Jonathan Scourfield, Cardiff School of

Social Sciences, Cardiff University, Cardiff,
UK. E-mail: [email protected]
Nina Maxwell
Jonathan Scourfield*
Sally Holland
Cardiff School of Social Sciences,
Cardiff University, UK, CF10 3WT
Brid Featherstone
Open University, UK
Jacquie Lee
Cardiff Metropolitan University, UK
‘The process of
developing and
running a training
intervention for child
protection social
workers’
‘Children’s services
are often criticised for
their relatively poor
engagement of men’
Copyright © 2012 John Wiley & Sons, Ltd. Accepted: 9 March
2012
Child Abuse Review Vol. 21: 299–310 (2012)
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/car.2218
Although there seems to have been some recent progress in

improving the
involvement of fathers in early years and family support
services, there has been
little change in child protection work. In the course of child
protection work, it can
feel to social workers as though they are bombarded with men
who are posing a
risk to children, through physical abuse, sexual abuse and
emotional maltreatment
(Scott and Crooks, 2004). Fathers may be intimidating or
intoxicated and abusive
to workers, leading workers to be reluctant to confront or
engage with them, or to
purposefully avoid them for fear of their violent reactions
(O’Donnell et al.,
2005). In some circumstances, the risks posed by the
involvement of men in par-
enting may outweigh any potential benefits, for instance, in
cases of substantial
domestic violence. However, this should be judged on
assessment evidence rather
than taken-for-granted gendered assumptions about masculine
identities. Munro’s
(1999) review of child abuse reports highlights that
practitioners are strikingly
slow to revise judgments made early in a case, which may lead
to a premature
avoidance of engaging with fathers who present as violent but
who may also
afford protective factors and may indeed be vulnerable
themselves (Frosh, 1994).
The failure to engage these men can lead to ‘mother blaming’ in
terms of
‘failure to protect’ rather than engaging the man concerned
(Humphreys and

Absler, 2011). Whilst there is some literature on working with
men perceived
as difficult or hostile, there is, as Peled (2000) notes, limited
literature on
abusive men as fathers, although there are some notable
exceptions (e.g.
Harne, 2011). In this context, it is perhaps not surprising that
men can be
perceived as being dangerous non-nurturers (Ferguson and
Hogan, 2004). If,
however, men are labelled as violent without recognition of
their role as
fathers, this not only negates any chance of changing the
negative aspects of
these fathers’ behaviours to children but also may do little to
stop them from
leaving the home and moving on to new relationships with new
children.
Several barriers to father engagement have emerged from the
research literature.
Social work has a tradition of focusing upon the mother in
relation to child protec-
tion issues regardless of who is responsible for abusing the
child or who the child
lives with (Ashley et al., 2006; Daniel and Taylor, 2001;
Scourfield, 2003; Strega
et al., 2008). Evidence from serious case reviews in England
(Brandon et al.,
2009) suggests that social workers can and do neglect to
identify and locate
fathers, fail to systematically gather or record information about
fathers and have
a tendency to categorise men as either risk or resource for
children, rather than
recognise the possibility that they can be simultaneously both of

these things.
Scourfield’s (2003) ethnographic research has suggested that
occupational culture
(i.e. the attitudes, knowledge and beliefs of front-line staff that
shape routine prac-
tice) is powerful in this regard, with received ideas and familiar
responses to
mothers and fathers taking hold in the culture of the social work
team. In addition
to practitioner effects, there are alsovarious barriers to father
engagement set up by
mothers and by fathers themselves (see Maxwell et al., 2012).
Although there is evidence about the nature of the problem and
some isolated
practice initiatives, there is little systematic evidence about
what might help
improve father engagement in a child protection context. The
one published exam-
ple of an evaluated training intervention with practitioners is
described by English
et al. (2009). This half-day awareness-raising training course
resulted in some
increased engagement with fathers, as evidenced in case
records. The current
paper presents findings from an intervention research project
which, like English
et al.’s study, aimed to improve social workers’ engagement of
fathers. Unlike the
‘It can feel to social
workers as though
they are bombarded
with men who are
posing a risk to
children’

‘Limited literature on
abusive men as
fathers, although there
are some notable
exceptions’
‘A tendency to
categorise men as
either risk or resource
for children’
300 Maxwell et al.
Copyright © 2012 John Wiley & Sons, Ltd. Child Abuse Rev.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
English et al. project, our training course was longer – two days
in duration – and
the training content was different. Our course employed
awareness-raising (work-
ing with knowledge and values) about the importance of
engaging with fathers,
combined with specific skills development. Motivational
interviewing (MI) has
most recently been defined as ‘a collaborative, person-centred
form of guiding
to elicit and strengthen motivation to change’ (Miller and
Rollnick, 2009,
p. 137). The purpose of MI is to evoke the person’s own
motivation and commit-
ment to behavioural change through the use of ambivalence
between the person’s

own values and actual behaviours. Thus MI generates
motivation for behavioural
change as the individual seeks to create congruence between
their values and
behaviours. Whilst there is no direct evidence of the
effectiveness of MI for enga-
ging fathers in a child protection context, the training course we
developed also
included skills-based training in MI, as this approach has been
found to be effec-
tive in other allied fields such as substance misuse (Lundahl et
al., 2010) and has
considerable promise for the engagement of reluctant service
users.
Method
Two local authorities in South Wales were targeted as both
served relatively
large populations (in the Welsh context) with rates of child
protection registra-
tions above the Welsh average. Both authorities agreed to take
part in the pilot
of the training, with all costs being covered by the research
grant (see
Acknowledgements). It was agreed that, where possible, whole
teams of social
workers would be trained. The rationale for this was the
importance of occupa-
tional culture in maintaining gendered practice, as noted earlier.
The training intervention development involved the input of
consultants with
expertise in the field of father engagement or working
effectively with men. These
consultants were from the Family Rights Group, the Probation

Service and Children
in Wales, the national umbrella children’s organisation. The
development phase also
included a review of research evidence (Maxwell et al., 2012)
and semi-structured
interviews in the two main local authorities with four social
work managers, six
social work practitioners and eight service users (5 fathers and
3 mothers).
The pilot training course then consisted of two full days, one
week apart. It
was envisaged that social workers would find it easier to attend
the course on
one day in each week as opposed to having two consecutive
days away from
the office. Participants were divided into two groups, each
consisting of
roughly equal numbers of staff from both authorities. This mix
ensured that
sufficient staff from each team remained in the office. The 50
social workers
who attended the two-day training course included a few
individuals from three
other local authorities, to ensure sufficient numbers in the third
cohort for a
viable group, but attendance was concentrated on the two core
authorities.
A mixed-method process evaluation was conducted. Observation
field notes
were taken by four observers (3 members of the research team
and 1 indepen-
dent observer), with each training session assessed under three
main categories
(trainer’s presentation style, success of activities and delegates’

responses).
Approximately three weeks following the training, 21
participants were invited
to participate in a short (15 minute) telephone interview. A total
of ten inter-
views were completed. The interview consisted of six main
questions relating
to the course (e.g. strengths, weaknesses and overall
impression) and a further
five questions relating to practice (e.g. Have you used anything
you learned
‘Awareness-raising
about the importance
of engaging with
fathers, combined
with specific skills
development’
‘Where possible,
whole teams of
social workers would
be trained’
‘This mix ensured that
sufficient staff from
each team remained in
the office’
301Training Child Protection Social Workers in Father
Engagement
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218

from the course in your daily practice?). In addition, the three
trainers were inter-
viewed about their experiences of delivering the training and
thoughts about how
the training was received. For this pilot study, no data were
collected from service
users – either parents or children – after the training had taken
place.
Pre-Intervention Qualitative Research
The overall picture was that the interviews conducted in the
development phase
reflected the themes that emerged from our literature review
(Maxwell et al.,
2012); for example, highlighting the role of mothers as
gatekeepers, practical
difficulties in arranging meetings with fathers who work, and
fathers’ avoid-
ance, absenteeism and reluctance to engage. However, of more
interest to the
current paper are the main themes that emerged in response to
the question
for social workers: Have you got any thoughts on your own
training needs in
relation to working with fathers? and the question for parents:
Are there any
things you’d like to see social workers do differently?
Of the four managers, two recommended guidance on how to
manage the
process of challenging difficult people. This reflects that both
mothers and
fathers may present aggression and/or hostility when their

parenting skills
are called into question. Similarly, a third manager emphasised
the need for
good communication skills that are based upon strategies of
enabling ways
of talking to people (not just fathers), especially those who do
not want to
engage or may be evasive when questioned. In regard to
working with fathers,
the fourth manager highlighted the need to raise awareness
about the barriers to
working with fathers so that social workers can understand and
adopt a more
patient, persistent approach. As for social workers, two echoed
the need for train-
ing on how to manage challenging behaviour, how to engage
parents and how to
work with violent parents. In relation to specific training for
father engagement,
two social workers wanted more information on legal aspects,
especially around
parental responsibility. One practitioner suggested it would be
beneficial to hear
from fathers themselves to find out what their perspectives are
in working with
social workers. We were able to respond to all these points in
the training design.
When asked what they would like to see social workers do
differently, all
eight of the service users (including 5 fathers) complained that
they were dic-
tated to with little attempt made at understanding their
particular situation. The
majority of fathers felt that they were talked at, with one stating
that he did not

understand what he was being told to do and another suggesting
it would be
helpful to be kept informed of any progress he had made. Whilst
one father felt
that his social worker was ‘on a crusade’ against men, both he
and the rest of
the parents interviewed were all able to recall periods where
they had worked
with what they perceived to be a good social worker.
Overwhelmingly, good
social workers were perceived to be those who listened,
understood and worked
with the family. These findings also informed the training
design.
Intervention Design
As well as the expert consultants, the course was designed in
collaboration with
the three trainers. Two of these trainers had a background in
social work
‘Have you got any
thoughts on your own
training needs in
relation to working
with fathers?’
‘We were able to
respond to all these
points in the
training design’
‘Good social workers
were perceived to be
those who listened,

understood and
worked with the family’
302 Maxwell et al.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
training, as well as specialist knowledge from research and
(crucially) practice
experience of working with fathers in a child protection context.
The third
trainer’s expertise was in relation to MI. The aim of the course
was to improve
social workers’ engagement of men in the child protection
process and there
were four main objectives:
1. To recognise the benefits of working with men for improving
the safety of children.
2. To enhance social workers’ knowledge in relation to work
with fathers.
3. To enhance inter-personal skills for engaging with reluctant
clients.
4. For social workers to feel more confident and effective in
working with fathers.
As learning readiness and knowledge gain have been found to
predict
training transfer (Antle et al., 2008), day one of the course
acknowledged the
difficulty and complexity of the roles involved in child
protection, encouraged

social workers to consider their own values and beliefs and
highlighted (with
reference to research evidence) the rationale for father
engagement as a means
of encouraging social workers to consider new ways of working.
This approach
was especially pertinent as we would not expect all practitioners
to be equally
committed to increasing father involvement in the child
protection process
(McBride et al., 2001). In the light of the pre-intervention
qualitative research
summarised earlier, we decided that an important aspect of any
training
intervention would be skills training for work with reluctant
clients. This was
therefore the main focus of day two, via an introduction to MI.
Both days com-
bined a range of teaching methods including information-giving,
discussion,
group activities and role-play.
Intervention Outcomes
Data on the outcomes of the training for social workers are
presented fully in a
separate paper (Scourfield et al., 2012) so are only summarised
very briefly
here. Quantitative measures were completed by course
participants at the start
of the first training day and again two months after completion
of the course.
Self-efficacy in relation to work with fathers improved over
time. There was
strong evidence (p < 0.001) of positive change in trainees’
responses about

their confidence levels in relation to each one of 17 different
statements about
work with fathers. Increase in confidence ranged between one
and two points
on a ten-point scale (see Holden et al., 2002). The magnitude of
change was
greatest for trainees’ confidence in discussing problematic and
abusive beha-
viour. Changes in team culture were modest. Although a metric
of all responses
to questions about teams added together showed significant
change, for indivi-
dual questions there was only significant change in relation to
two questions: In
my team, staff are comfortable working with fathers (p = 0.05)
and I myself
would feel able to offer advice and consultation to others on
work with fathers
(p < 0.001).
Self-efficacy does seem to have followed through to practice.
Trainees were
asked about categories of fathers on their caseload and how
many men had
been worked with. For the category of men whose behaviour
puts children at
risk of harm, there was no change over time. For the category of
men living
with children who are not putting them at risk of harm, there
was an increase
in the rate of engagement following the training (p = 0.03).
Finally, for fathers
‘Learning readiness
and knowledge
gain have been

found to predict
training transfer’
‘Self-efficacy in
relation to work with
fathers improved
over time’
‘Self-efficacy does
seem to have followed
through to practice’
Engagement
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
whose whereabouts are known but who are not living with their
children, there
was almost a doubling of the rate of engagement from 26.5 per
cent to 52
per cent (p = 0.01).
Qualitative Insights into the Training Process
Drawing on data from observation notes taken during the course
and telephone
interviews conducted after the training, the following sections
present findings
in relation to the benefits and challenges of attempting to
improve father
engagement and to use MI in child protection practice.

The Benefits and Challenges of Attempting to Improve Father
Engagement in
Child Protection
One of the main themes emerging from the data was that the
training high-
lighted the need for perseverance, effort and time in order to
engage fathers.
One observer commented thus on the views of trainees: ‘Much
of the emphasis
seemed to be on what hard work, how difficult and time-
consuming working
with men was’ (Observer field notes)
Interviewees noted that they had come away from the course
recognising the
importance of considering their beliefs and understanding the
father’s perspec-
tive, both of which could be related to three of the four course
objectives (2–4).
One example of such a comment was this:
‘Personally I like any course that kind of flags up a kind of
minority position in a way, or a
minority view in a way that asks people to stop and reflect and
kind of put themselves in the
shoes of the person who occupies them in a minority position.’
This comment is of interest in a number of ways. Firstly, it
constructs fathers
as a minority (and possibly a victimised minority?). Secondly,
the use of the
word ‘minority’ might be seen to reinforce the idea that practice
with fathers
is a discrete process rather than part of engaging with families

in a more
holistic way. The issues around engaging fathers can be seen to
feed into a wider
set of concerns about contemporary practices and even into
debates about the nat-
ure of social work. One observer noted that to trainees, work
with men who are
abusive did not seem to be seen as part of the social work role
and therefore when
there are issues in relation to abuse, the social work focus
remains on the mother.
It is of interest that one participant noted that their team did try
to work with
the whole family and that was part of her ethos, but the
following comment
from another interviewee suggests that for some people at least
this was in fact
a major learning point from the course: ‘The message came loud
and clear I
think that the focus of the training was for us not to forget there
is more than
one parent in any situation.’
The importance of not assuming that because the father is not
resident, he is
not interested or does not have an important role to play was
something that
many respondents focused on. Also there was an interesting
point made about
how assumptions made at an earlier stage in a case may need to
be revisited.
One respondent noted that it was important to try to unpack
whether what
was presented as aggressive behaviour was actually frustration
and to seek to

go below the surface of the presenting behaviours. Interestingly,
there was little
‘The training
highlighted the need
for perseverance,
effort and time in order
to engage fathers’
‘The use of the word
‘minority’ might be
seen to reinforce the
idea that practice with
fathers is a discrete
process’
304 Maxwell et al.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
reference in respondents’ comments to the importance of re-
thinking assumptions
about fathers’ physical appearance as noted by Ferguson and
Hogan (2004) and
also on the Family Rights Group (2009) DVD that was shown on
the course.
A number of interviewees noted that their practice had already
changed
since the training course, primarily in terms of making more
effort to find
out where the father was and get his details and his views. One

noted that a
child had gone to live with the father and another felt that the
father had got
more involved and was being very supportive as a result of her
contacting
him. Another important issue identified in terms of benefits was
that the course
had sparked conversations at a team level. This development, if
sustained, is
likely to be important, given the power of occupational culture.
For one parti-
cipant, an important change had been her perseverance with
fathers who are
violent. Her experience had been that they did not want to speak
to her but
through perseverance she was getting them to talk to her. She
also referred
explicitly to the assumption that a man in that situation might
have about a
woman social worker being on the woman’s side.
There was a concern that if non-resident fathers are engaged
with, workers
would get caught up in battles within relationships, an issue
which has been
discussed more generally in the literature. It raises the question
of whether
social workers are trained to engage with the complexities of
the relationships
they encounter, especially in diverse family constellations
(Featherstone,
2004). Rather than recognising complexity, the observers noted
that there
was a tendency for course participants to fall into thinking that
‘fathers’
referred to birth fathers, despite attempts by the trainers to

emphasise inclusive
use of this term. This is perhaps indicative of the challenges of
terminology in
this field. In designing the training course, we followed recent
UK policy
discourse in using the term ‘fathers’, but the course participants
did not necessa-
rily associate this term with a wider group of men in some kind
of parenting role,
including, for example, a mother’s fairly new boyfriend. Use of
language will
shift over time and according to context, but it may still be
difficult to maintain
a focus on a wider group of men if we use the term ‘fathers’ in
training.
When asked about any gaps in the training, most of those
identified seemed
to relate both to the risks attached to engaging fathers and the
risks posed by
fathers/men. Whilst overall the training was rated very
positively, social work-
ers perceived the course to be focused upon engaging those who
were difficult
to engage rather than those who were actually aggressive.
Social workers stated
that they would have preferred more on working with aggressive
men. The
feeling that risk could have been dealt with more prominently
connects to a
more general issue raised by one of the observers, namely, that
it is very
difficult to maintain a consistent focus on both risk and
resource over a two-
day course. There is a tendency for the discussion to veer
towards either a sole

focus on men as risk or a predominant focus on men as resource
for children.
To an extent – and this is not meant to absolve the trainers’
responsibilities –
trainees will pick up what they want to pick up on an issue that
has such
considerable personal resonance and is so affected by life
experience. Ideally,
as Ferguson (2011) concludes, education and training should
‘provide opportunities for workers to critically reflect on their
assumptions and attitudes
towards men, women and gender roles, and their own
experiences of being fathered, so that learn-
ing can occur about how these influence their understanding of
masculinity and practice’ (p. 163).
‘Little reference in
respondents’
comments to the
importance of re-
thinking assumptions
about fathers’ physical
appearance’
‘A tendency for course
participants to fall into
thinking that ‘fathers’
referred to birth
fathers’
‘Very difficult to
maintain a consistent
focus on both risk
and resource over a
two-day course’

Engagement
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
Whilst our course did start with some discussion about
participants’ own
values in relation to fathering, the material was not generally
geared towards
a more personal biographical dimension. To tackle underpinning
beliefs in
more depth, this would probably be necessary.
The Benefits and Challenges of Using MI in Child Protection
Practice
In general, course participants responded very positively to day
two of the
course which was primarily focused on MI. This element was
frequently
mentioned in the end-of-course evaluative comments (when
participants were
asked, in turn, to verbally state what they would take away from
the course)
and in the anonymous end-of-course written evaluation form.
Whilst MI skills
were clearly fresh in trainees’ minds at the end of the course,
interestingly, they
were mentioned far less frequently in the post-course qualitative
interviews
which took place a few weeks later than the general message of

the need to
engage fathers more. Nonetheless, a small number mentioned
specific commu-
nication skills that they had learned and tried to use in the
month since
completing the course. Here are two examples:
‘I think that with some clients it has worked particularly well. I
can think of one in parti-
cular who wants to talk to me but, sort of, can’t. It’s about sort
of helping him find the words
almost and I think that the Motivational Interviewing facilitates
his side of it you know to
bring out in conversation.’
‘I’ve tried a little bit of Motivational Interviewing [laughs]. . .
it at least got me thinking
about well look perhaps I could try approach these situations
differently and perhaps I could
be putting more time aside in order to try and get underneath
the problem by allowing more
space for people to kind of provide their own look on things
where you are using more open-
ended questions and by trying to get people to try and engage in
that way.’
It is possible that once the practitioners returned to their busy,
everyday
practice the overall theme of the training was easier to
remember, put into
practice and report than the micro-skills of MI. The statutory
social worker role
is complex and involves much more than engaging, listening
and enabling
change. This is summarised by one of the trainers, herself an
experienced child

protection social worker:
‘Motivating people to change is a really important bit of what
social workers do but I think
that looking back, we would have done better to acknowledge
that they also have to investi-
gate, they have to gather information, and they have to convey
information.’
This trainer went on to state that MI is an important part of the
practitioner’s
repertoire of communication styles. During one of the courses,
two participants
debated the applicability of MI to their work:
‘This is much easier to use in substance misuse services and
when people want to be there
than in children’s services. We always have the balance of risk.
We often have to demand.’
‘But [another intervenes] we could use the decisional balance
with them and they lay out
the map and we can use that to navigate through with them.
Karen [trainer] says this is a
really good way of describing MI.’ (Observer field notes)
Two of the participants who took part in qualitative post-course
interviews
mentioned that they did not think MI had a role in their current
caseload or
even team. One stated (emphasis added),
‘Course participants
responded very
positively to day two
of the course which

was primarily focused
on MI’
‘Easier to remember,
put into practice and
report than the micro-
skills of MI’
‘During one of the
courses, two
participants debated
the applicability of MI
to their work’
306 Maxwell et al.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
‘If I was in a long-term team then that [MI] would have been
very beneficial but you know
in Duty and Assessment if you are doing an initial assessment
it’s in-out, you get the informa-
tion you don’t tend to be able to . . . use it to its full potential
really because you just haven’t
got the time. You know an initial assessment will be completed
in an hour’s visit you know and
the Motivational Interviewing is something that I feel would be
used further down the line.’
This interviewee suggests that time restrictions on her team’s
roles mean that
they have to concentrate on gaining information, ‘it’s in-out. . .

get the informa-
tion’ rather than engagement activities that are perceived to be
lengthy. The
course appeared to encourage many to reflect on their usual
style, which many
mentioned tended to be questioning, interrogatory and above
all, speedy. Many
participants noted that they tend to rush in and out of
assessment interviews,
because of the time pressures of the statutory frameworks
(Broadhurst et al.,
2010). However, some participants observed that with just a
slight change of
pace and style much more could be learned about a service
user’s life and
perspective. In day two of the training course, two of the
trainers role-played
an initial assessment meeting with a father using MI
approaches. This
provoked the following feedback from participants (emphasis
added):
‘You seemed to slow down the process.’
‘There were periods when you didn’t talk at all and he had the
chance to say more.’
‘It seemed relaxed. He talked a lot.’
‘He was quite hyper when he came in but you managed to calm
him down.’
It is notable that all of these comments were concerned with
pace and style.
Participants here observed that the slower pace allowed them to
learn more
about this father. Two went on to contrast this to their usual
style.

‘What’s interesting about this is the contrast between this
interview and one we all do
which is all about getting info and filling in forms.’
One whispered to the other –
‘I do that when I go out. I start interviewing straight away.’
(Observer field notes)
It can be seen that the MI training was a style that contrasted
fairly starkly
with some of these participants’ usual ways of working. This
may be a reflec-
tion of the procedural and informationally driven nature of
contemporary child
protection practice (Parton, 2008) overshadowing a more
relational approach to
working with parents (Ruch, 2005). MI’s emphasis on the
person rather than
the behavioural ‘problem’ refocuses practice on building
collaborative relation-
ships and rapport with parents. Indeed, one participant, in
responding to the
trainer’s comment that people are more likely to change if they
decide to do
so themselves, rather than being told to by someone else, stated
(with humour)
‘So what’s the point of social workers?’ (observer field notes).
There was a risk
that this training course could be seen as preaching by
academics who understand
little about the everyday realities of practice and who do not
value the effort that
practitioners are already putting in to engage with fathers and to
use an engaging
style of communication. Fortunately, the trainers were
experienced current child
protection practitioners, which seem to have been particularly

valued by partici-
pants. This was mentioned in almost every post-course
interview:
‘They were so knowledgeable, and you could see that they had
worked on our level, they
knew the pitfalls.’ (Post-course interview 3)
‘The course appeared
to encourage many to
reflect on their
usual style’
‘It is notable that all of
these comments were
concerned with pace
and style’
‘There was a risk
that this training
course could be seen
as preaching by
academics’
Engagement
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
‘I always find that very valuable having somebody who’s got
that kind of amount of knowl-
edge and practice of the field to bring to the course.’ (Post-

course interview 9)
There were three important risks in devoting one day of the
course to using
MI with fathers. One was that this approach would be seen as
irrelevant
because, as has been seen, it contrasts with the predominant
culture of commu-
nication in child protection which can be dominated by
information-giving
(telling parents what the concerns are), evidence gathering and
confrontation
(Forrester et al., 2008a). It has been seen in this section that
this was recognised
by many course participants, but that most showed an
enthusiastic willingness
to reflect on their usual communication style with fathers and
other service
users and to try new ways of working. The second risk was that
the course
may be seen as something imposed by academics who know
little about statu-
tory social work and who do not value the considerable
experience and existing
skills of the participants. The trainers’ practitioner credibility
and ability to use
recent examples from their own practice helped considerably in
avoiding this
scenario. Thirdly, there was the risk that only one day of skills
training might
not be enough to enable the application of new communication
styles to practice.
There is evidence that some practitioners have indeed tried to
use MI in their
practice. Further evaluation, using an experimental design,
would be required

to know for sure whether the training ‘worked’ for practitioners
and families.
Conclusion
In response to the main aim of the intervention, namely, that
social workers’
engagement of fathers could be improved via a short course, the
qualitative
findings reported provide some tentative support. Social
workers who attended
the course appeared to reframe their thinking about fathers and
demonstrated
greater awareness of the need to persevere and make greater
efforts to engage
fathers. MI was seen as a useful tool for social workers,
although there was a
clear distinction made as to when MI was useful and when
interviewing had
to be directed towards gathering important information and
‘dictating’ what
behaviours had to be changed. This follows Forrester et al.
(2008a, 2008b)
who found a tendency for social workers to adopt
confrontational styles of
communication, but also found that MI training achieved a
moderate level of
success in improving social work practice, with workers
displaying lower levels
of confrontation and higher levels of listening to parents.
In respect of the difficulties social workers face in balancing
risk and
resource, the course had a modest aim in highlighting the need
to gather basic
information about fathers for effective risk management. To this

end, the
course was perhaps effective. However, the difficulties in
maintaining a focus
on both risk and resource have been noted. Post-intervention
interviews sug-
gested some social workers were fearful about engaging fathers
who may pose
a risk, and especially aggressive men, and felt that the course
did not ade-
quately address these issues, although this needs to be balanced
against the
findings from the quantitative research that the greatest increase
in self-efficacy
was in relation to discussing problematic and abusive
behaviour. The useful-
ness of MI to gain a wealth of information by slowing down the
pace and style
of interviewing clients was noted. However, there could perhaps
have been
‘Three important risks
in devoting one day of
the course to using MI
with fathers’
‘Only one day of skills
training might not be
enough to enable the
application of new
communication styles
to practice’
‘A modest aim in
highlighting the need
to gather basic
information about

fathers for effective
risk management’
308 Maxwell et al.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
more attention paid to the links between possessing the skills to
engage reluctant
fathers and encouraging them to talk as a means of effective
risk management.
This paper has described a two-day training course delivered to
whole teams
of social workers and aimed at encouraging child protection
workers to
question their values and attitudes toward fathers, further their
understanding
about fathers and learn skills that have proven successful at
engaging reluctant
clients, albeit in different fields. There was some evidence that
social workers
were spending more time and effort in engaging fathers with a
modicum of
success. Initial quantitative findings from pre-post measures
from social work-
ers are presented in a separate paper (Scourfield et al., 2012),
but we note that
more rigorous evaluation using an experimental design is
needed in order to
evidence any practical applications and changes made to
practice as a direct

result of the course. Perhaps the most important evidence gaps
are the lack
of data on actual as opposed to perceived behaviour change, the
reactions of
fathers themselves and whether training on engaging fathers can
be associated
with more beneficial outcomes for women and children. In this
pilot study, we
did not collect any follow-up data from service users – either
parents or
children – and it would be important to do this in any further
research.
Training can serve as a catalyst to changing occupational
culture. However,
it is important to note on concluding this paper that although
training may be
necessary, it may not be sufficient to achieve cultural change
without a
whole system approach that embeds and sustains the cultural
shift within the
organisation, for example, through the use of reflective clinical
supervision
and review of systems and procedures.
Acknowledgements
The project was funded by the National Institute for Social Care
and Health
Research (Wales). We are very grateful for the cooperation of
two Welsh local
authorities in the piloting of the training course. Polly Baynes,
Daryl Dugdale,
Karen Marsh, Ian Bickerton, Sean Haresnape and Tony Ivens all
made valuable
contributions to the training design.

References
Antle BF, Barbee AP, van Zyl MA. 2008. A comprehensive
model for child welfare training
evaluation. Children and Youth Services Review 30(9): 1063–
1080.
Ashley C, Featherstone B, Roskill C, Ryan M, White S. 2006.
Fathers Matter: Research Findings
on Fathers and their Involvement with Social Care Services.
Family Rights Group: London.
Brandon M, Bailey S, Belderson P, Gardner R, Sidebotham P,
Dodsworth J, Warren J, Black J.
2009. Understanding Serious Case Reviews and Their Impact: A
Biennial Analysis of Serious
Case Reviews 2005–7, Research report DCSF-RR129. London,
DCFS.
Broadhurst K, Wastell D, White S, Hall C, Peckover S,
Thompson K, Pithouse A, Davey D.
2010. Performing ‘Initial Assessment’: Identifying the Latent
Conditions for Error at the
Front-Door of Local Authority Children’s Services. British
Journal of Social Work 40(2):
352–370.
Daniel B, Taylor J. 2001. Engaging with fathers: practical
issues for health and social care.
Jessica Kingsley Publishers: London.
English DJ, Brummel S, Martens P. 2009. Fatherhood in the
Child Welfare System: Evalua-
tion of a Pilot Project to Improve Father Involvement. Journal
of Public Child Welfare 3(3):

213–234.
‘Some evidence that
social workers were
spending more time
and effort in
engaging fathers’
Engagement
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
Family Rights Group. 2009. Fathers Matter DVD: The views
and experiences of fathers on their
involvement with local authority children’s services. Family
Rights Group: London.
Featherstone B. 2004 Family Life and Family Support: A
Feminist Analysis: Palgrave Macmillan:
Basingstoke.
Ferguson H. 2011. Child Protection Practice. Palgrave
Macmillan: Basingstoke.
Ferguson H, Hogan F. 2004. Strengthening Families Through
Fathers: Developing Policy and
Practice in Relation to Vulnerable Fathers and their Families.
The Centre for Social and Family
Research, Waterford Institute of Technology.
Forrester D, McCambridge J, Waissbein C, Rollnick S. 2008a.

How do child and family social
workers talk to parents about child welfare concerns? Child
Abuse Review 17: 23–35. DOI:
10.1002/car.981
Forrester D, McCambridge J, Waissbein C, Emlyn-Jones R,
Rollnick S. 2008b. Child risk and
parental resistance: Can Motivational Interviewing improve the
practice of child and family
social workers in working with parental alcohol misuse? British
Journal of Social Work
38: 1302–1319.
Frosh S. 1994. Sexual Difference, Masculinity and
Psychoanalysis. Routledge: London.
Harne L. 2011. Violent Fathering and the Risks to Children: The
need for change. Policy Press:
Bristol.
Holden G, Meenaghan T, Anastas J, Metrey G. 2002. Outcomes
of social work education: The
case for social work self-efficacy. Journal of Social Work
Education 38: 115–133.
Humphreys C, Absler D. 2011. History repeating: Child
protection responses to domestic
violence. Child and Family Social Work 16(4): 464–473.
Lundahl BW, Kunz C, Brownell C, Tollefson D, Burke BL.
2010. A Meta-Analysis of Motiva-
tional Interviewing: Twenty-Five Years of Empirical Studies.
Research on Social Work
Practice. 20(2): 137–160.
Maxwell N, Scourfield J, Featherstone B, Holland S, Tolman R.

2012. Engaging with fathers in
child welfare: A narrative review of recent research evidence.
Child and Family Social Work
17(2): 160–169.
McBride BA, Rane TR, Bae J. 2001. Intervening with teachers
to encourage father/male invol-
vement in early childhood programs. Early Childhood Research
Quarterly 16(1): 77–93.
Miller WR, Rollnick S. 2009. Ten things that motivational
interviewing is not. Behavioural and
Cognitive Psychotherapy 37(2): 129–140.
Munro E. 1999.Common errors of reasoning in child protection
work. Child Abuse & Neglect
23(8): 745–758.
O’Donnell JM, Johnson WE, D’Aunno LE, Thornton HL. 2005.
Fathers in child welfare:
caseworkers’ perspectives. Child Welfare LXXXIV(3): 387–
414.
Parton N. 2008. Changes in the form of knowledge in social
work: From the ‘social’ to the
‘informational’? British Journal of Social Work 38(2): 253–269.
Peled E. 2000. Parenting by men who abuse women: Issues and
dilemmas. British Journal of
Social Work 30(1): 25–36.
Ruch G. 2005. Relationship-based practice and reflective
practice: Holistic approaches to
contemporary child care social work. Child and Family Social
Work 10(2): 111–123.

Scott KS Crooks CV. 2004. Effecting change in maltreating
fathers: Critical principles for
intervention planning. Clinical Psychology: Science and
Practice 10: 95–111.
Scourfield J. 2003. Gender and Child Protection. Palgrave
Macmillan: London.
Scourfield J, Tolman R, Maxwell N, Holland S, Bullock A,
Sloan L. 2012. Results of a training
course for social workers on father engagement in child
protection. Children and Youth
Services Review, published online in advance of print. DOI:
10.1016/j.childyouth.2012.03.022
Strega S, Fleet C, Brown L, Dominelli L, Callahan M,
Walmsley C. 2008. Connecting father
absence and mother blame in child welfare policies and
practice. Children and Youth Services
Review 30(7): 705–716.
310 Maxwell et al.
Vol. 21: 299–310 (2012)
DOI: 10.1002/car.2218
Copyright of Child Abuse Review is the property of John Wiley
& Sons, Inc. and its content may not be copied
or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission.
However, users may print, download, or email articles for

individual use.
Review the Resources and reflect on the various strategies
presented throughout the course that may be helpful in
disseminating effective and widely cited EBP.
This may include: unit-level or organizational-level
presentations, poster presentations, and podium presentations at
organizational, local, regional, state, and national levels, as
well as publication in peer-reviewed journals.
Reflect on which type of dissemination strategy you might use
to communicate EBP.
Post at least two dissemination strategies you would be most
inclined to use and explain why. Explain which dissemination
strategies you would be least inclined to use and explain why.
Identify at least two barriers you might encounter when using
the dissemination strategies you are most inclined to use. Be
specific and provide examples. Explain how you might
overcome the barriers you identified.
References:
· Melnyk, B. M., & Fineout-Overholt, E. (2018). Evidence-
based practice in nursing & healthcare: A guide to best practice
(4th ed.). Philadelphia, PA: Wolters Kluwer.
Chapter 10, “The Role of Outcomes on Evidence-based Quality
Improvement and enhancing and Evaluating Practice Changes”
(pp. 293–312)
Chapter 12, “Leadership Strategies for Creating and Sustaining
Evidence-based Practice Organizations” (pp. 328–343)
Chapter 14, “Models to Guide Implementation and

Sustainability of Evidence-based Practice” (pp. 378–427)
· Gallagher-Ford, L., Fineout-Overholt, E., Melnyk, B. M., &
Stillwell, S. B. (2011). Evidence-based practice, step by step:
Implementing an evidence-based practice change. American
Journal of Nursing, 111(3), 54–60.
doi:10.1097/10.1097/01.NAJ.0000395243.14347.7e. Retrieved
from
https://journals.lww.com/ajnonline/Fulltext/2011/03000/Evidenc
e_Based_Practice,_Step_by_Step_.31.aspx
· Newhouse, R. P., Dearholt, S., Poe, S., Pugh, L. C., & White,
K. M. (2007). Organizational change strategies for evidence-
based practice. Journal of Nursing Administration, 37(12), 552–
557. doi:0.1097/01.NNA.0000302384.91366.8f. Retrieved from
http://downloads.lww.com/wolterskluwer_vitalstream_com/jour
nal_library/nna_00020443_2007_37_12_552.pdf
· Melnyk, B. M. (2012). Achieving a high-reliability
organization through implementation of the ARCC model for
systemwide sustainability of evidence-based practice. Nursing
Administration Quarterly, 36(2), 127–135.
doi:10.1097/NAQ.0b013e318249fb6a
· Melnyk, B. M., Fineout-Overholt, E., Gallagher-Ford, L., &
Sustaining evidence-based practice through organizational
policies and an innovative model. American Journal of Nursing,
111(9), 57–60. doi:10.1097/01.NAJ.0000405063.97774.0e.
Retrieved from
https://www.nursingcenter.com/nursingcenter_redesign/media/E
BP/AJNseries/Sustaining.pd
· Melnyk, B. M., Fineout-Overholt, E., Giggleman, M., & Choy,
K. (2017). A test of the ARCC© model improves
implementation of evidence-based practice, healthcare culture,
and patient outcomes. Worldviews on Evidence-Based Nursing,
14(1), 5–9. doi:10.1111/wvn.12188. Retrieved from
https://sigmapubs.onlinelibrary.wiley.com/doi/abs/10.1111/wvn.
12188
· Melnyk, B. M., Fineout-Overholt, E., Gallagher-Ford, L., &

Sustaining evidence-based practice through organizational
policies and an innovative model. American Journal of Nursing,
111(9), 57–60. doi:10.1097/01.NAJ.0000405063.97774.0e.
Retrieved from
https://journals.lww.com/ajnonline/Fulltext/2011/09000/Evidenc
e_Based_Practice,_Step_by_Step__Sustaining.27.aspx
55
Journal of Social Work Practice in the Addictions, 9:55–70,
2009
Copyright © Taylor & Francis Group, LLC
ISSN: 1533-256X print/1533-2578 online
DOI: 10.1080/15332560802533612
WSWP1533-256X1533-2578Journal of Social Work Practice in
the Addictions, Vol. 9, No. 1, January 2009: pp. 1–25Journal of
Social Work Practice in the Addictions
Collocation: Integrating Child Welfare
and Substance Abuse Services
CollocationE. Lee et al.
EUNJU LEE, PHD
Senior Research Scientist, Center for Human Services Research,
School of Social
Welfare, University at Albany, New York, USA
NINA ESAKI, PHD
Research Scientist, Center for Human Services Research,

School of Social
ROSE GREENE, MA
Associate Director, Center for Human Services Research,
School of Social
This article presents findings from a process evaluation of a
pilot
program to address parental substance abuse in the child welfare
system. By placing substance abuse counselors in a local child
welfare office, the collocation program was designed to
facilitate
early identification, timely referral to treatment, and improved
treatment engagement of substance-abusing parents. Frontline
child welfare workers in 6 of the 7 pilot sites endorsed the
program
as they found that the collocated substance abuse counselors
pro-
vided additional resources and facilitated case processing.
Findings
suggest that clearly defined procedures and sufficient staffing
of
qualified substance abuse counselors could lead to better
programs.
KEYWORDS child welfare, parental substance abuse, service
integration
Received May 16, 2007; accepted February 8, 2008.
This research was funded by the Children’s Bureau (#90CW-
1111), Administration of

Children and Family, U.S. Department of Health and Human
Services.
Address correspondence to Eunju Lee, Center for Human
Services Research, School of Social
Welfare, University at Albany, 135 Western Ave., Albany, NY
12222, USA. E-mail: [email protected]
albany.edu
56 E. Lee et al.
Parental substance abuse is a well-known risk factor affecting
families in the
child welfare system. However, both the child welfare and
substance abuse
service systems have faced challenges in identifying, engaging,
and provid-
ing effective treatment to substance-abusing parents
investigated for child
maltreatment.
Challenges include the different goals, legal mandates, and
practices
between the child welfare and substance abuse fields. As a
result of the
Adoption and Safe Families Act of 1997, the timelines for
placement deci-
sions and family reunification were shortened, placing
unrealistic demands
on substance-abusing parents to make significant life changes.
Additionally,
the child welfare system seeks to protect children and,
whenever possible,
to keep families together. Substance abuse treatment providers

view addic-
tion as a chronic, relapsing condition and traditionally place
primary focus
on the individual client.
In response to these challenges, policymakers and
administrators have
invested in service integration models. Promising results from
the Illinois
Title IV demonstration program will further generate interest in
service inte-
gration (Marsh, Ryan, Choi, & Testa, 2006; Ryan, Marsh, Testa,
& Louderman,
2006). Despite this recent advance, empirical research on
service integration
models is still limited and few studies have rigorously examined
the imple-
mentation issues of these models.
A collocation program piloted in a northeastern state is a
service integra-
tion model designed to address parental substance abuse in the
child welfare
system. The program consists of placing credentialed
alcoholism and sub-
stance abuse counselors (CASACs) in local child welfare offices
to work with
frontline child welfare workers to increase the level of
substance abuse iden-
tification, treatment referral, and treatment engagement. This
article presents
the results of a process evaluation of the collocation model
using data from
interviews, focus groups, and administrative records. The
program model,
implementation process, implementation challenges, perceived

effects, and
suggestions for future service integration models are examined.
LITERATURE REVIEW
Prevalence and Risk of Substance Abuse in the
Child Welfare System
Although substance abuse is considered a serious risk factor for
child
maltreatment, current prevalence rates of parental substance
abuse in child
welfare cases vary widely due to differences in definitions and
methodology
(Besinger, Garland, Litrownik, & Landsverk, 1999; Semidei,
Radel, & Nolan,
2001; Young, Boles, & Otero, 2007). For example, the Child
Welfare League
of America (1998) estimated that at least 50% of confirmed
cases of child
maltreatment involve parents with substance abuse problems.
Semidei et al.
Collocation 57
(2001) found substance abuse contributed to child maltreatment
for one
third to two thirds of the families involved with child welfare
agencies.
Parental alcohol or drug use has been also strongly associated
with the sub-
stantiation of abuse or neglect allegations (Sun, Shillington,
Hohman, & Jones,
2001; Wolock, Sherman, Feldman, & Metzger, 2001). Estimates

of parental
substance abuse for children entering foster care have been even
more stag-
gering: About 80% of children placed out of home due to
maltreatment have
parents with substance abuse issues (Besinger et al., 1999; U.S.
Department of
Health and Human Services [USDHHS], 1999). The prognosis
for families with
substance abuse problems in the child welfare system is dismal.
Child mal-
treatment cases involving parental substance abuse often result
in recurring
maltreatment allegations, longer stays in foster care, and
reduced likelihood
of family reunification (Ryan et al., 2006; Smith & Testa, 2002;
USDHHS, 1999;
U.S. Government Accounting Office, 1998; Wolock & Magura,
1996).
Barriers to Service and Treatment
Unfortunately, less than half of all parents with substance abuse
issues in the
child welfare system enter and complete necessary alcohol and
drug services
(Young, Gardner, & Dennis, 1998). Gregoire and Schultz (2001)
found that few
parents complete assessment or treatment. Engaging and
retaining these clients
in treatment has been a critical problem (Choi & Ryan, 2006;
USDHHS, 1999).
There have been clinical and systemic barriers for engagement
and retention of
parents in treatment (McAlpine, Marshall, & Doran, 2001).
These issues revolve

around the nature of the child welfare job, the types of
substance abuse treat-
ment services readily available in communities, federal and
state policies, and
the differing perspectives of the child welfare and substance
abuse fields.
First, child welfare staff lacks the training and experience to
accurately
assess the extent of substance abuse problems of parents
investigated for
child maltreatment (Semidei et al., 2001; Tracy, 1994; Young et
al., 1998).
Parents in the child welfare system are likely to deny their
alcohol and other
drug problems as well as their need for help, in part, because
they fear
removal of their children (Dore, Doris, & Wright, 1995; Jessup,
Humphreys,
Brindis, & Lee, 2003). Child welfare workers whose primary
focus is the
safety of children are also not experienced in helping parents
with
substance problems (Marsh & Cao, 2005; Tracy & Farkas, 1994)
and view
substance-abusing parents as difficult to treat (Semidei et al.,
2001).
Effective treatment designed for parents, especially women with
young
children, is not easily available in many communities. Many
providers are
not prepared or equipped to address the complex physical,
mental, social,
and economic issues facing these women and their children
(USDHHS,

1999). In addition, these parents, particularly mothers, often
lack critical
concrete supports (e.g., child care, transportation) necessary to
begin and
complete treatment (Azzi-Lessing & Olsen, 1996; Carlson,
2006).
58 E. Lee et al.
Despite a lengthy recovery process and the need for concrete
services
to enter and complete treatment, federal and state policies place
demanding
timelines on such families. Under the Adoption and Safe
Families Act
(ASFA) of 1997, parents must resolve their problem within a
12-month
period or risk permanent loss of their children (Green, Rockhill,
& Furrer,
2006; Smith, 2001). These policies not only place demands on
substance-
abusing parents to make significant life changes in relatively
brief periods of
time, but also place undue burdens on child welfare services to
accelerate
accurate assessment, referral, and case management services
(McAlpine
et al., 2001).
Finally, the child welfare and substance abuse treatment
systems have
different perspectives (Feig, 1998; Young & Gardner, 1998).
Substance
abuse treatment staff members who are knowledgeable about

addiction
focus almost exclusively on the drug abuser. In contrast, child
welfare
workers who are more knowledgeable about the consequences of
addiction
on the other family members might have a punitive attitude
toward sub-
stance abusers and focus on the maltreated child. In addition,
given the
often different background and training experiences of workers
in these
two fields, child welfare workers and substance abuse treatment
providers
typically know very little about the other area (Carlson, 2006).
Need for Collaboration Between the Two Systems
To address the challenges associated with substance abuse in
child welfare,
strategies for integrating substance abuse treatment and child
welfare
services have gained increased popularity (Horwath &
Morrison, 2007; Ryan
et al., 2006). Historically, the implicit model in child welfare
depended on
the child welfare worker acting in isolation to motivate the
substance-abusing
client to seek treatment. However, more recently, policymakers,
practitio-
ners, and scholars have come to believe that collaboration
between sub-
stance abuse and child welfare systems can be more effective in
engaging
the parents in treatment (Colby and Murrell, 1998; Cornerstone
Consulting
Group, 2002; McAlpine et al., 2001; Peterson, Gable, &

Saldana, 1996; Ryan
et al., 2006; Semidei et al., 2001; Young & Gardner, 2002).
Some research suggests collaboration between substance abuse
treatment
and other social service systems improves treatment outcomes,
especially for
women (Dore & Doris, 1998; Kraft & Dickinson, 1997; Marsh,
D’Aunno, &
Smith, 2000; Randolph & Sherman, 1993; Walsh & Young,
1998; Young &
Gardner, 1998). Dore and Doris (1998) found that nearly half of
the women in
their study were able to complete treatment through a placement
prevention
initiative staffed by both child welfare workers and substance
abuse specialists.
For women with children, improved access to treatment,
specifically the provi-
sion of transportation, outreach, and child-care services, showed
a negative
relationship with continued substance abuse (Marsh et al.,
2000).
Collocation 59
A number of states have initiated collaborative efforts between
the
child welfare and substance abuse systems to build effective
new partner-
ships. Although some show promising results (Cornerstone
Consulting
Group, 2002; Maluccio & Ainsworth, 2003; Young & Gardner,
2002), there

has been limited empirical evidence to demonstrate the impact
of these
collaborative efforts on child welfare outcomes (Barth,
Gibbons, & Guo,
2006; Marsh et al., 2006). One exception has been a recent
study (Ryan et
al., 2006) that demonstrated positive results after provision of
intensive
case management to link substance abuse services and child
welfare ser-
vices in Illinois.
Collocation: A Service Integration Model
Collocation refers to strategies that place multiple services in
the same
physical space (Ginsburg, 2008). It has been suggested as a
strategy for
integrating different service systems for clients with multiple
service needs
(Agranoff, 1991; Austin, 1997). Clients with multiple needs
face difficulties
in navigating fractured systems with different sets of rules and
expecta-
tions. As a result, they are less likely to receive needed services
and more
likely to experience poor outcomes (Marsh et al., 2006). A
recent study
indicated that child welfare outcomes are substantially enhanced
when
families receive appropriate substance abuse services (Green,
Rockhill, &
Furrer, 2007).
A collocation model, which places substance abuse counselors
at local

child welfare agencies, serves as a simple, concrete, and
straightforward
mechanism for facilitating collaboration between the two
systems. The
model has the potential to increase early identification of
substance-abusing
parents in the child welfare system. It could also address some
of the barri-
ers to treatment, thereby engaging and retaining substance-
abusing parents
in treatment that might, in turn, lead to improved child welfare
outcomes.
Substance abuse specialists are trained to utilize empirically
based tech-
niques, such as the transtheoretical model of change (Prochaska
&
DiClemente, 1984; Prochaska & Norcross, 1999) and
motivational interview-
ing (Miller & Rollnick, 2002), a process of engagement that is
designed to
overcome child welfare clients’ denial of abuse and to motivate
them to
enter treatment. These specialists, working in concert with child
welfare
workers, can address the logistical and psychosocial barriers to
treatment,
can build a trusting relationship during the “window of
opportunity” when
parents feel highly vulnerable, and can successfully obtain the
parents’
acceptance of care plan goals within federal and state time
constraints.
Unfortunately, literature specific to the topic of collocation is
limited.
Several descriptive studies regarding collocation have been

conducted in
such venues as human services in schools (Briar-Lawson,
Lawson, Collier, &
Joseph, 1997; Tapper, Kleinman, & Nakashian, 1997), mental
health service
60 E. Lee et al.
providers in buildings of primary care physicians for the
treatment of
depressed patients (Valenstein et al., 1999), and substance
abuse providers
in departments of social services for the assessment of
Temporary Assis-
tance to Needy Families (TANF) recipients (Center on
Addiction and Sub-
stance Abuse, 1999). Similarly, research regarding the
collocation of
substance abuse specialists in child protective services (CPS) is
sparse, and
although encouraging regarding intermediate outcomes
(McAlpine et al.,
2001), remains inconclusive regarding longer term child welfare
outcomes
(Marsh et al., 2006). McAlpine and colleagues (2001) examined
a program
that included collocating substance abuse specialists in child
welfare offices.
They found substantial increased use of the substance abuse
specialist by
the child welfare office in less than 1 year—from an initial rate
of 10 staff
members making requests for 169 investigations to 32 staff
members

making requests for 282 investigations. A recent evaluation of
the Illinois
Title IV-E demonstration program showed promise of service
integration for
substance-abusing parents whose children were removed from
their care
(Ryan et al., 2006).
Despite encouraging outcomes, additional research is needed
regarding
service integration models for child welfare clients. Particularly
useful
would be studies examining implementation issues. The
Maryland Title IV-E
demonstration was terminated due to several factors, but some
were related
to program implementation (USDHHS, 2005), indicating
difficulties of
service integration regardless of its promise.
METHODOLOGY
To address the issue of substance abuse in families involved in
the child
welfare system, the child welfare and substance abuse state
agencies in a
northeastern state issued a request for proposals (RFP).
Collocation was one
of the suggested models funded under this RFP, using TANF
prevention
funds. For this model, CASACs were to be collocated in child
welfare offices
to identify and assist parents with substance abuse problems.
Treatment
agencies were eligible to apply for the funding in partnership
with child

welfare offices in their region. In 2001, nine programs began to
serve child
welfare clients and the pilot programs ended in most sites by
2004.
Study Design
From 2004 to 2005, the authors conducted a process study as
part of an
evaluation of the pilot collocation program. The study included
seven sites;
four programs in primarily rural locations and three programs in
primarily
metropolitan areas. Two of the original sites were eliminated
from the
study. One site was defunded in the first year due to the
inability of the
Collocation 61
substance abuse treatment agency to establish a working
relationship with
the local child welfare office. The second site adopted a blended
interven-
tion model of the collocation and family drug court programs,
which was
unfavorable to an evaluation of the collocation model.
The study’s goal was to examine the implementation processes
and to
assess whether program sites varied in implementation success.
Specifically,
the authors were interested in examining the following
questions: 1) Were

the target populations served? 2) Did collocation increase
collaboration and
understanding between the child welfare and substance abuse
agencies?
3) Was the program implemented as intended? and 4) What were
the barri-
ers to successful implementation?
Data and Analysis
Data were collected from focus groups and individual
interviews at each of
the seven collocation sites, as well as from interviews with
stakeholders at
the state agencies. Information gathered from stakeholders
included the
planning and startup of the program, the operations, processes
for case
identification and referrals, the relationship between the child
welfare and
substance abuse fields, and administrative procedures and
protocols. In
each collocation site, a focus group consisting of 10 to 15 child
welfare
workers and a separate focus group for 6 to 12 child welfare
supervisors
were conducted. Interviews were also held with at least one key
child
welfare administrator, often the individual with responsibility
for overseeing
the program at each program site. Separate interviews were
conducted with
each CASAC and his or her supervisor from the treatment
agency. To elimi-
nate bias, two investigators were present at each of the focus
groups and

interviews, and sessions were tape-recorded. In total, 14 focus
groups and
18 interviews were conducted. Additionally, progress reports
and other
administrative records, such as the original contracts, were
reviewed.
After each site visit, the tapes from the interviews and focus
groups
were transcribed and categorized. To ensure accuracy and to
eliminate bias,
the transcribed notes were compared with the notes taken by the
two
authors. Data were then analyzed using the constant comparison
method
(Glaser, 1978) by writing down emerging themes and by
comparing similar-
ities and differences within and across sites (Miles &
Huberman, 1994;
Patton, 2002).
RESULTS
Despite initial start-up difficulties, all but one of the seven sites
succeeded
in implementing the collocation model. At the one site where
implementa-
tion did not occur, staff at the child welfare office and at the
treatment
62 E. Lee et al.
agency disagreed on program goals and operating procedures
and could

not establish a strong working relationship.
In general, child welfare workers who admitted to being
initially skep-
tical about yet another new initiative ended up embracing the
program.
Similarly, substance abuse counselors who typically provide
services within
their clinics grew to realize the benefits of home visits as a way
to identify
and assess substance abuse issues and to elicit greater
awareness of client
needs. Both agreed that the collocation program improved their
under-
standing of each other’s system and perceived that the program
improved
early identification, timely referral to treatment, and treatment
outcomes of
substance-abusing parents in the child welfare system.
Challenges
ACCEPTANCE BY CHILD WELFARE STAFF
Although frontline child welfare workers were advised of the
new initiative,
specific mechanisms were not established about how to work
with the
collocated substance abuse worker. In addition, many of the
child welfare
workers were skeptical about the introduction of yet another
new program
in their offices. As a result, the burden of implementation fell
heavily on the
CASACs and their supervisors.

The lack of established procedures made implementation
difficult,
especially in the first year. All of the collocated counselors
encountered a
number of startup difficulties, particularly in obtaining
acceptance from the
child welfare workers and in achieving an adequate number of
case refer-
rals. Although the concept of collocation implies an egalitarian
partnership,
it was the CASACs who had to make an extra effort to ingratiate
themselves
to the child welfare staff and to make personal appeals for case
referrals.
Two CASACs were replaced early on because they were unable
to develop
close working relationships with child welfare workers.
MODEL VARIATIONS
Although the program framework was identified in the RFP, the
design of
the program mechanisms was determined by the localities. At
six out of the
seven sites, the collocated counselors consistently provided two
core
services: assessment of substance abuse and referral to
treatment services.
However, the programs varied on how the counselors provided
these ser-
vices and whether they provided additional services beyond
these two core
activities.
Two basic variations of the program emerged: one in the
metropolitan

sites and one in the rural sites. In the metropolitan programs,
the client
interviews, assessments, and referrals were conducted in the
child welfare
Collocation 63
office. In the rural programs, the counselors conducted home
visits and their
services were not physically limited to the child welfare offices.
Additionally,
in the rural sites, the CASACs continued to work with the client
over a longer
period of time than in the metropolitan programs by providing
case manage-
ment services, such as transportation, for the duration of their
treatment.
Similarly, there were two different processes for how the case
was
referred to the collocated counselors. Identification of substance
abuse cases
occurred either through a call to the child abuse hotline or after
the initial
investigation. In some sites, the hotline call that identified
parental substance
abuse was forwarded directly to the substance abuse counselor,
although
this represented a minority of referrals to the program. Most
often, cases
were referred to the collocated counselor after the investigation
was initiated
by the child welfare worker. Child welfare workers were
generally willing to

involve the CASACs in such cases to obtain additional
assistance and coun-
sel. However, they were inconsistent regarding the types of
cases that were
referred and when the referrals were made. No consistent rules
were estab-
lished, resulting in individual child welfare workers using their
own discretion.
TARGET POPULATIONS AND CAPACITY
Overall, the collocation programs served the intended
populations, TANF
parents affected by substance abuse. In most cases, the CASACs
served
mothers who were being investigated for child maltreatment.
However, on
occasion, the counselors would provide services to other family
members.
In some of the smaller rural counties, the collocated counselors
worked
with a significant number of adolescents with substance abuse
issues
involved in persons in need of supervision (PINS) cases, who
were neither
the perpetrators nor victims of the CPS reports.
As for capacity, even in the smallest county, a single CASAC
could not
serve all eligible clients, especially when the CASAC was
conducting both
home visits and providing case management services. Due to the
level of
funding, the sites were limited to hiring one or two CASACs.
Although child
welfare workers generally respected the collocated counselors

for their
ability to engage the clients as well as for their knowledge of
appropriate
treatment services, they expressed frustration about the limited
service
capacity that could be offered by one or two CASACs. Child
welfare work-
ers in one focus group expressed a desire for 10 substance abuse
counse-
lors to be assigned to their local program.
CONFIDENTIALITY
At a number of sites, there was confusion and apprehension
among the child
welfare workers about sharing information. Child welfare
workers felt that
they had to obtain consent forms from their clients to share
information with
64 E. Lee et al.
the CASACs. This process slowed down the CASACs’ effort to
quickly engage
clients and provide them with appropriate assessments and
treatment referrals
during the short investigation period. Eventually, some sites
developed mem-
oranda of understanding (MOUs) between the two agencies that
addressed
this issue. In compliance with the Health Insurance Portability
and Account-
ability Act (HIPAA) laws, CASACs obtained a signed consent
form from clients

to share client information with child welfare workers.
Addressing the issues
of information sharing and confidentiality prior to
implementation is impor-
tant to reduce confusion and difficulties for workers on both
sides.
Benefits
IMPROVED COORDINATION OF SERVICES
At the programmatic level, there was an improved relationship
between the
child welfare and substance abuse fields as demonstrated by the
enhanced
coordination of service delivery. This could be partly attributed
to an
increased awareness on both sides of the goals, objectives, and
challenges
of each other’s field. Similarly, the physical proximity of the
CASAC made a
difference for child welfare workers and their clients. Child
welfare workers
were able to contact the CASAC immediately and have the
client meet with
the substance abuse specialist in a timely fashion, which was
extremely
important due to policies imposing time limitations in case
determination.
The child welfare workers believed the program led to less
recurrence
of child maltreatment and consequently fewer subsequent CPS
reports.
However, this impression has yet to be verified by a
comprehensive review

of the administrative data.
INCREASED SUBSTANCE ABUSE IDENTIFICATION AND
BETTER REFERRAL
The child welfare workers agreed that the substance abuse
counselors were
better equipped to persuade child welfare clients to admit to
substance
abuse problems. Two possible explanations can be offered.
First, unlike the
child welfare workers, the counselors were trained specifically
in tech-
niques for engaging clients with substance abuse problems.
Second, the cli-
ents were not as threatened by the counselors as they were by
the child
welfare workers, who could ultimately remove their children
from the
home. Therefore, they were more willing to be honest about
their substance
abuse issues and were more motivated to resolve their problems
with assis-
tance from an experienced substance abuse counselor.
Some counselors helped clients access treatment services and
worked
with them to remain in treatment. In the rural sites, the
counselors followed
the clients beyond the referral stage by providing additional
case management
services, such as arranging transportation and removing other
barriers that

Collocation 65
might impede clients from obtaining treatment. In all of the
sites, the coun-
selors had discretionary funds to assist clients in this capacity.
DISCUSSION
Findings from this study offer insight into the challenges and
potential
benefits of implementing a program to collocate substance
abuse counse-
lors in child welfare offices. The collocation programs faced
issues similar to
those that plague many new initiatives. Suggestions for
successful implemen-
tation of a collocation program include careful planning,
engaging child
welfare workers, standardizing procedures, and providing strong
leadership.
Planning
To facilitate communication and processing of cases between
child welfare
workers and counselors, child welfare offices and collaborating
treatment
agencies would benefit from detailing policies on
confidentiality in MOUs.
Similarly, providing adequate physical facilities for collocated
counselors
should be planned to enhance their integration into the child
welfare offices.
In the planning phase, administrators might want to consider the
specific

qualities that would maximize the acceptance of the collocated
counselor by
the child welfare office. Early on, it needs to be recognized that
the collocated
substance abuse counselors are entering a potentially
unwelcoming culture.
Although good clinical skills are important, the collocated
substance abuse
counselor also needs a flexible personality, as demonstrated by
a willingness
to work with child welfare workers, an aptitude for learning
new rules, and
an open-mindedness toward the culture of child welfare offices.
Engaging Child Welfare Workers
Programs that engage both child welfare workers and substance
abuse coun-
selors in advance of program implementation are likely to
experience greater
success. Informing workers of the program and soliciting their
feedback
beforehand will lead to easier program implementation when
formally intro-
duced. Providing the workers with information regarding the
program, espe-
cially the benefits to both them and their clients, is essential.
Child welfare
workers are often wary of new initiatives that tend to add more
work to their
already heavy caseloads. The successful implementation of the
collocation
program was partly due to the fact that the CASACs provided
additional
resources to child welfare workers, thus lessening some of their
burden.

Similarly, substance abuse counselors need to understand that
their role is
to be complementary to that of the child welfare workers. They
need to be
trained on the policies and practices of the child welfare system
from the
66 E. Lee et al.
beginning, especially the laws, requirements, and timelines
pertinent to the child
welfare system. To be accepted and effective, they need to
overcome precon-
ceived notions about the child welfare system and adapt to the
agency’s culture.
Standardizing Procedures
Collocation programs would benefit from clearly stated
procedures outlining
the program model, program eligibility, and the process for
identification,
referral, and follow-up of clients. The lack of such procedures
is not condu-
cive to collaboration, as workers from the two systems could be
left with
differing expectations.
Standardization may include the identification and referral of
all child
welfare cases with parental substance abuse issues directly to
the collocated
substance abuse counselors as soon as possible. Specifically,

cases with
substance abuse issues identified in the initial hotline call may
be auto-
matically referred to the counselors. Similarly, all other cases
that are inves-
tigated by child welfare workers should be screened, if possible,
using a
brief standardized tool. The earlier the intervention, the better
the potential
outcomes for the families. The CPS investigation provides a
window of
opportunity to engage child welfare clients when they are
feeling vulnerable
and perhaps more receptive to treatment services.
In addition, it might be advantageous to implement an
automated
information system to track cases that are referred to the
CASAC. By so
doing, both the child welfare workers and the CASAC can
identify trends,
such as tracking the duration between case intake and referral to
the
CASAC, and to make informed program adjustments.
Leadership
Although the collocation program depends primarily on
collaboration
among frontline workers from two service agencies, leadership
at each
agency plays a critical role in successful implementation. The
collocation of
frontline staff is not just a new initiative, but a sign of a
burgeoning relation-
ship between workers in two systems that have long held

different views
and have operated on different sets of mandates.
For better outcomes, substance abuse treatment agencies must
continue
to provide support to the collocated counselors and maintain
collaborative
relationships with the child welfare agencies at higher levels of
manage-
ment. The senior management teams in both organizations
should be in
regular communication and should address any programmatic
issues in a
collaborative, expeditious fashion to keep the program running
smoothly.
The implementation of the pilot program was successful partly
due to the
leaders from both agencies being willing to listen to and work
with each
other, including replacing ineffective project members when
necessary.
Collocation 67
Limitations of the Study
There are several limitations to this study. The primary data for
this study
were gathered through focus groups and interviews. Focus
groups include
the tendency for certain types of socially acceptable opinions to
emerge and
for certain types of participants to dominate the research
process (Smithson,

2000). Steps were taken at the beginning of each focus group to
emphasize
the confidential nature of the information that was being
collected as well
as to encourage participants to “speak up, even if you disagree
with every-
one else in the group.” Although steps were taken to reduce
these biases,
these elements could not be completely avoided.
In addition, although there were a minimum of two researchers
who
participated in each of the focus group and interview sessions,
there is the
possibility that key conclusions might have been biased by the
perspectives
of the researchers. Although information gathered from the
program partici-
pants suggests some positive outcomes of the program, it will
only be
through a quantitative outcome study that actual impact can be
determined.
CONCLUSION
The findings of this process study are encouraging in regard to
the possi-
ble impact of the collocation model on coordination of services
between
the child welfare and substance abuse systems. Given the
prevalence of
substance abuse in the child welfare population, it is important
that new
and innovative interventions are developed and tested to
improve child
welfare outcomes for vulnerable families who are in need of

services.
Although this study was limited to a small-scale pilot program,
the initial
findings provide a strong foundation on which a quantitative
outcome
study can be conducted to determine what impact, if any, the
program
might actually have. It is through the pilot testing of new
programs and
process studies such as this one that program developers can
learn about
various factors that facilitate or hinder successful
implementation of any
program. The successful implementation of a program is the
first step
toward assessing its efficacy.
REFERENCES
Agranoff, R. (1991). Human services integration: Past and
present challenges in
public administration. Public Administration Review, 51, 533–
542.
Austin, M. J. (1997). Service integration: Introduction.
Administration in Social
Work, 21(3–4), 1–7.
Azzi-Lessing, L., & Olsen, L. (1996). Substance abuse-affected
families in the child
welfare system: New challenges, new alliances. Social Work,
41(1), 15–23.
68 E. Lee et al.

Barth, R. P., Gibbons, C., & Guo, S. (2006). Substance abuse
treatment and the
recurrence of maltreatment among caregivers with children
living at home: A
propensity score analysis. Journal of Substance Abuse
Treatment, 30, 93–104.
Besinger, B. A., Garland, A. F., Litrownik, A. J., & Landsverk,
J. (1999). Caregiver
substance abuse among maltreated children placed in out-of-
home care. Child
Welfare, 78, 221–239.
Briar-Lawson, K., Lawson, H. A., Collier, C., & Joseph, A.
(1997). School-linked
comprehensive services: Promising beginnings, lessons learned,
and future
challenges. Social Work in Education, 19, 136–148.
Carlson, B. E. (2006). Best practices in the treatment of
substance-abusing women
in the child welfare system. Journal of Social Work Practice in
the Addictions,
6, 97–115.
Center on Addiction and Substance Abuse. (1999). Building
bridges: States respond
to substance abuse and welfare reform. New York: Author.
Child Welfare League of America. (1998). Alcohol and other
drug survey of state
child welfare agencies. Washington, DC: Author.
Choi, S., & Ryan, J. P. (2006). Completing substance abuse
treatment in child

welfare: The role of co-occurring problems and primary drug of
choice. Child
Maltreatment, 11, 313–325.
Colby, S. M., & Murrell, W. (1998). Child welfare and
substance abuse services:
From barriers to collaboration. In R. L. Hampton, V. Senatore,
& T. P. Gullotta
(Eds.), Substance abuse, family violence, and child welfare:
Bridging perspec-
tives (pp. 188–219). Thousand Oaks, CA: Sage.
Cornerstone Consulting Group. (2002). A review of the four
child welfare IV-E waivers
related to substance abuse services in Delaware, Maryland, New
Hampshire, and
Illinois. Houston, TX: Author.
Dore, M. M., & Doris, J. M. (1998). Preventing child placement
in substance-abusing
families: Research-informed practice. Child Welfare, 77, 407–
426.
Dore, M. M., Doris, J. M., & Wright, P. (1995). Identifying
substance abuse in mal-
treating families: A child welfare challenge. Child Abuse &
Neglect, 19, 531–543.
Feig, L. (1998). Understanding the problem: The gap between
substance abuse pro-
grams and child welfare services. In R. L. Hampton, V.
Senatore, & T. P. Gullotta
(Eds.), Substance abuse, family violence, and child welfare:
Bridging perspec-
tives (pp. 62–95). Thousand Oaks, CA: Sage.

Ginsburg, S. (2008). Colocating health services: A way to
improve coordination of
children’s health care? New York: The Commonwealth Fund.
Glaser, B. G. (1978). Theoretical sensitivity: Advances in the
methodology of
grounded theory. Mill Valley, CA: Sociology Press.
Green, B. L., Rockhill, A., & Furrer, C. (2006). Understanding
patterns of substance
abuse treatment for women involved with child welfare: The
influence of the
Adoption and Safe Families Act (ASFA). American Journal of
Drug and Alcohol
Abuse, 32, 149–176.
Green, B. L., Rockhill, A., & Furrer, C. (2007). Does substance
abuse treatment
make a difference for child welfare case outcomes? A statewide
longitudinal
analysis. Children & Youth Services Review, 29, 460–473.
Gregoire, K. A., & Schultz, D. J. (2001). Substance-abusing
child welfare parents:
Treatment and child placement outcomes. Child Welfare, 80,
433–452.
Collocation 69
Horwath, J., & Morrison, T. (2007). Collaboration, integration
and change in
children’s services: Critical issues and key ingredients. Child
Abuse & Neglect,
31(1), 55–69.

Jessup, M. A., Humphreys, J. C., Brindis, C. D., & Lee, K. A.
(2003). Extrinsic barriers
to substance abuse treatment among pregnant drug dependent
women. Journal
of Drug Issues, 33, 285–304.
Kraft, M. K., & Dickinson, J. E. (1997). Partnerships for
improved service delivery:
The Newark target cities project. Health & Social Work, 22,
143–148.
Maluccio, A. N., & Ainsworth, F. (2003). Drug use among
parents: A challenge for
family reunification practice. Children and Youth Services
Review, 25, 511–533.
Marsh, J. C., & Cao, D. (2005). Parents in substance abuse
treatment: Implications
for child welfare practice. Children and Youth Services Review,
27, 1259–1278.
Marsh, J. C., D’Aunno, T. A., & Smith, B. D. (2000). Increasing
access and providing
social services to improve drug treatment for women with
children. Addiction,
95, 1237–1247.
Marsh, J. C., Ryan, J. P., Choi, S., & Testa, M. F. (2006).
Integrated services for
families with multiple problems: Obstacles to family
reunification. Children
and Youth Services Review, 28, 1074–1087.
McAlpine, C., Marshall, C. C., & Doran, N. H. (2001).
Combining child welfare and

substance abuse services: A blended model of intervention.
Child Welfare, 80,
129–149.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data
analysis: An expanded
sourcebook (2nd ed.). Thousand Oaks, CA: Sage.
Miller, W. R., & Rollnick, S. (2002). Motivational interviewing:
Preparing people to
change (2nd ed.). New York: Guilford.
Patton, M. Q. (2002). Qualitative research and evaluation
methods (3rd ed.).
Thousand Oaks, CA: Sage.
Peterson, L., Gable, S., & Saldana, L. (1996). Treatment of
maternal addiction to
prevent child abuse and neglect. Addictive Behaviors, 21, 789–
801.
Prochaska, J. O., & DiClemente, C. C. (1984). The
transtheoretical approach:
Crossing traditional boundaries of change. Homewood, IL:
Dorsey.
Prochaska, J. O., & Norcross, J. C. (1999). Systems of
psychotherapy: A transtheoretical
approach (4th ed.). New York: Brooks/Cole.
Randolph, L. A., & Sherman, B. R. (1993). Project CONNECT:
An interagency
partnership to confront new challenges facing at-risk women
and children in
New York City. Journal of Community Health, 18, 73–81.

Ryan, J. P., Marsh, J. C., Testa, M. F., & Louderman, R. (2006).
Integrating substance
abuse treatment and child welfare services: Findings from the
Illinois alcohol
and other drug abuse waiver demonstration. Social Work
Research, 30, 95–107.
Semidei, J., Radel, L. F., & Nolan, C. (2001). Substance abuse
and child welfare:
Clear linkages and promising responses. Child Welfare, 80,
109–128.
Smith, B. D. (2001). Child welfare and substance abuse: Toward
partnerships with
parents and communities. In A. L. Sallee, H. A. Lawson, & K.
Briar-Lawson
(Eds.), Innovative practices with vulnerable children and
families (pp. 121–143).
Dubuque, IA: Eddie Bowers.
Smith, B. D., & Testa, M. F. (2002). The risk of subsequent
maltreatment allegations in
families with substance exposed infants. Child Abuse &
Neglect, 26(1), 97–114.
70 E. Lee et al.
Smithson, J. (2000). Using and analysing focus groups:
Limitations and possibilities.
International Journal of Social Research Methodology, 3, 103–
119.
Sun, A., Shillington, A. M., Hohman, M., & Jones, L. (2001).
Caregiver AOD use,

case substantiation, and AOD treatment: Studies based on two
southwestern
counties. Child Welfare, 80, 151–177.
Tapper, D., Kleinman, P., & Nakashian, M. (1997). An
interagency collaboration
strategy for linking schools with social and criminal justice
services. Social
Work in Education, 19, 176–188.
Tracy, E. M. (1994). Maternal substance abuse: Protecting the
child, preserving the
family. Social Work, 39, 534–540.
Tracy, E. M., & Farkas, K. J. (1994). Preparing practitioners for
child welfare practice
with substance-abusing families. Child Welfare, 73, 57–68.
U.S. Department of Health and Human Services. (1999).
Blending perspectives and
building common ground: A report to Congress on substance
abuse and child
protection. Washington, DC: U.S. Government Printing Office.
U.S. Department of Health and Human Services. (2005,
September). Child welfare dem-
onstration projects: Evaluation methodologies. Retrieved
January 14, 2007, from
http://www.acf.hhs.gov/programs/cb/programs_fund/cwwaiver/s
ubstanceabuse/
evaluation.htm#t5
U.S. Government Accounting Office. (1998). Foster care:
Agencies face challenges
securing stable homes for children of substance abusers
(GAO/HEHS-98–182).

Washington, DC: Author.
Valenstein, M., Klinkman, M., Becker, S., Blow, F. C., Barry,
K. L., Sattar, A., et al.
(1999). Concurrent treatment of patients with depression in the
community:
Provider practices, attitudes, and barriers to collaboration.
Journal of Family
Practice, 48, 180–187.
Walsh, M. A., & Young, J. J. (1998). Innovative strategies for
strengthening systemic
links between child welfare services and services for alcohol
and other drug
problems: The DHS IMPACT program. The Source, 8(2), 12–13.
Wolock, I., & Magura, S. (1996). Parental substance abuse as a
predictor of child
maltreatment re-reports. Child Abuse and Neglect, 20, 1183–
1193.
Wolock, I., Sherman, P., Feldman, L. H., & Metzger, B. (2001).
Child abuse and
neglect referral patterns: A longitudinal study. Children and
Youth Services
Review, 23(1), 21–47.
Young, N. K., Boles, S. M., & Otero, C. (2007). Parental
substance use disorders and
child maltreatment: Overlap, gaps, and opportunities. Child
Maltreatment, 12,
137–149.
Young, N. K., & Gardner, S. L. (1998). Children at the
crossroads. Public Welfare,
56(1), 3–11.

Young, N. K., & Gardner, S. L. (2002). Navigating the
pathways: Lessons and
promising practices in linking alcohol and drug services with
child welfare.
Rockville, MD: Substance Abuse and Mental Health Services
Administration.
Young, N. K., Gardner, S. L., & Dennis, K. (1998). Responding
to alcohol and other
drug problems in child welfare: Weaving together policy and
practice. Washington,
DC: CWLA Press.
Workbook
for
Designing
a Process
Evaluation
Produced for the
Georgia Department of Human

Resources
Division of Public Health
By
Melanie J. Bliss, M.A.
James G. Emshoff, Ph.D.
Department of Psychology
Georgia State University
July 2002
Evaluation Expert Session
July 16, 2002 Page 1
What is process evaluation?
Process evaluation uses empirical data to assess the delivery of
programs. In contrast to outcome evaluation, which assess the
impact of the program, process evaluation verifies what the
program is and whether it is being implemented as designed.
Thus,
process evaluation asks "what," and outcome evaluation asks,
"so
what?"

When conducting a process evaluation, keep in mind these three
questions:
1. What is the program intended to be?
2. What is delivered, in reality?
3. Where are the gaps between program design and delivery?
This workbook will serve as a guide for designing your own
process
evaluation for a program of your choosing. There are many
steps involved
in the implementation of a process evaluation, and this
workbook will
attempt to direct you through some of the main stages. It will be
helpful to
think of a delivery service program that you can use as your
example as
you complete these activities.
Why is process evaluation important?
1. To determine the extent to which the program is being
implemented according to plan
2. To assess and document the degree of fidelity and variability
in
program implementation, expected or unexpected, planned or
unplanned
3. To compare multiple sites with respect to fidelity
4. To provide validity for the relationship between the
intervention

and the outcomes
5. To provide information on what components of the
intervention
are responsible for outcomes
6. To understand the relationship between program context
(i.e.,
setting characteristics) and program processes (i.e., levels of
implementation).
7. To provide managers feedback on the quality of
implementation
8. To refine delivery components
9. To provide program accountability to sponsors, the public,
clients,
and funders
10. To improve the quality of the program, as the act of
evaluating is
an intervention.

Stages of Process Evaluation Page Number
1. Form Collaborative Relationships 3
2. Determine Program Components 4
3. Develop Logic Model*
4. Determine Evaluation Questions 6
5. Determine Methodology 11
6. Consider a Management Information System 25
7. Implement Data Collection and Analysis 28
8. Write Report**
Also included in this workbook:
a. Logic Model Template 30
b. Pitfalls to avoid 30
c. References 31
Evaluation can be an exciting,
challenging, and fun experience
Enjoy!
* Previously covered in Evaluation Planning Workshops.
** Will not be covered in this expert session. Please refer to
the Evaluation Framework

and Evaluation Module of FHB Best Practice Manual for more
details.
Forming collaborative relationships
A strong, collaborative relationship with program delivery staff
and management will
likely result in the following:
Feedback regarding evaluation design and implementation
Ease in conducting the evaluation due to increased cooperation
Participation in interviews, panel discussion, meetings, etc.
Increased utilization of findings
Seek to establish a mutually respectful relationship
characterized by trust, commitment,
and flexibility.
Key points in establishing a collaborative
relationship:
Start early. Introduce yourself and the evaluation team to as

many delivery staff and
management personnel as early as possible.
Emphasize that THEY are the experts, and you will be utilizing
their knowledge and
information to inform your evaluation development and
implementation.
Be respectful of their time both in-person and on the
telephone. Set up meeting places
that are geographically accessible to all parties involved in the
evaluation process.
Remain aware that, even if they have requested the evaluation,
it may often appear as
an intrusion upon their daily activities. Attempt to be as
unobtrusive as possible and
request their feedback regarding appropriate times for on-site
data collection.
Involve key policy makers, managers, and staff in a series of
meetings throughout the
evaluation process. The evaluation should be driven by the
questions that are of
greatest interest to the stakeholders. Set agendas for meetings
and provide an
overview of the goals of the meeting before beginning. Obtain
their feedback and
provide them with updates regarding the evaluation process.

You may wish to
obtained structured feedback. Sample feedback forms are
throughout the workbook.
Provide feedback regarding evaluation findings to the key
policy makers, managers,
and staff when and as appropriate. Use visual aids and
handouts. Tabulate and
summarize information. Make it as interesting as possible.
Consider establishing a resource or expert "panel" or advisory
board that is an official
group of people willing to be contacted when you need feedback
or have questions.
Determining Program Components
Program components are identified by answering the questions
who, what, when, where,
and how as they pertain to your program.

Who: the program clients/recipients and staff
What: activities, behaviors, materials
When: frequency and length of the contact or intervention
Where: the community context and physical setting
How: strategies for operating the program or intervention
BRIEF EXAMPLE:
Who: elementary school students
What: fire safety intervention
When: 2 times per year
Where: in students’ classroom
How: group administered intervention, small group practice
1. Instruct students what to do in case of fire (stop, drop and
roll).
2. Educate students on calling 911 and have them practice on
play telephones.
3. Educate students on how to pull a fire alarm, how to test a
home fire alarm and how to
change batteries in a home fire alarm. Have students practice
each of these activities.
4. Provide students with written information and have them take
it home to share with their
parents. Request parental signature to indicate compliance and
target a 75% return rate.
Points to keep in mind when determining program
components

Specify activities as behaviors that can be observed
If you have a logic model, use the "activities" column as a
starting point
Ensure that each component is separate and distinguishable
from others
Include all activities and materials intended for use in the
intervention
Identify the aspects of the intervention that may need to be
adapted, and those that should
always be delivered as designed.
Consult with program staff, mission statements, and program
materials as needed.
Your Program Components

After you have identified your program components, create a
logic model that graphically
portrays the link between program components and outcomes
expected from these
components.
Now, write out a succinct list of the components of your
program.
WHO:
WHAT:
WHEN:
WHERE:

HOW:
What is a Logic Model
A logical series of statements that link the problems your
program is attempting to
address (conditions), how it will address them (activities), and
what are the expected
results (immediate and intermediate outcomes, long-term goals).
Benefits of the logic model include:
helps develop clarity about a project or program,
helps to develop consensus among people,
helps to identify gaps or redundancies in a plan,
helps to identify core hypothesis,
helps to succinctly communicate what your project or program
is about.

When do you use a logic model
Use...
- During any work to clarify what is being done, why, and with
what intended results
- During project or program planning to make sure that the
project or program is logical and
complete
- During evaluation planning to focus the evaluation
- During project or program implementation as a template for
comparing to the actual program
and as a filter to determine whether proposed changes fit or
not.
This information was extracted from the Logic Models: A
Multi-Purpose Tool materials developed by Wellsys
Corporation for the Evaluation Planning Workshop Training.
Please see the Evaluation Planning Workshop
materials for more information. Appendix A has a sample
template of the tabular format.

Determining Evaluation Questions
As you design your process evaluation, consider what questions
you would like to answer. It is only after
your questions are specified that you can begin to develop your
methodology. Considering the importance
and purpose of each question is critical.
BROADLY....
What questions do you hope to answer? You may wish to turn
the program components that you have just identified
into questions assessing:
Was the component completed as indicated?
What were the strengths in implementation?

What were the barriers or challenges in implementation?
What were the apparent strengths and weaknesses of each step
of the intervention?
Did the recipient understand the intervention?
Were resources available to sustain project activities?
What were staff perceptions?
What were community perceptions?
What was the nature of the interaction between staff and
clients?
These are examples. Check off what is applicable to you, and
use the space below to write additional broad,
overarching questions that you wish to answer.
SPECIFICALLY ...
Now, make a list of all the specific questions you wish to
answer, and organize your questions categorically. Your
list of questions will likely be much longer than your list of
program components. This step of developing your
evaluation will inform your methodologies and instrument
choice.
Remember that you must collect information on what the

program is intended to be and what it is in reality, so you
may need to ask some questions in 2 formats.
For example:
How many people are intended to complete this intervention
per week?"
How many actually go through the intervention during an
average week?"
Consider what specific questions you have. The questions below
are only examples! Some may not be appropriate
for your evaluation, and you will most likely need to add
additional questions. Check off the questions that are
applicable to you, and add your own questions in the space
provided.
WHO (regarding client):
Who is the target audience, client, or recipient?
How many people have participated?
How many people have dropped out?
How many people have declined participation?
What are the demographic characteristics of clients?
Race
Ethnicity
National Origin
Age
Gender
Sexual Orientation
Religion
Marital Status
Employment
Income Sources
Education

Socio-Economic Status
What factors do the clients have in common?
What risk factors do clients have?
Who is eligible for participation?
How are people referred to the program? How are the
screened?
How satisfied are the clients?
YOUR QUESTIONS:
WHO (Regarding staff):
Who delivers the services?
How are they hired?
How supportive are staff and management of each other?
What qualifications do staff have?
How are staff trained?
How congruent are staff and recipients with one another?
What are staff demographics? (see client demographic list for
specifics.)
YOUR QUESTIONS:

WHAT:
What happens during the intervention?
What is being delivered?
What are the methods of delivery for each service (e.g., one-
on-one, group session, didactic instruction,
etc.)
What are the standard operating procedures?
What technologies are in use?
What types of communication techniques are implemented?
What type of organization delivers the program?
How many years has the organization existed? How many
years has the program been operating?
What type of reputation does the agency have in the
community? What about the program?
What are the methods of service delivery?
How is the intervention structured?
How is confidentiality maintained?
YOUR QUESTIONS:

WHEN:
When is the intervention conducted?
How frequently is the intervention conducted?
At what intervals?
At what time of day, week, month, year?
What is the length and/or duration of each service?
July 16, 2002 Page 10
YOUR QUESTIONS:
WHERE:
Where does the intervention occur?

What type of facility is used?
What is the age and condition of the facility?
In what part of town is the facility? Is it accessible to the
target audience? Does public transportation access
the facility? Is parking available?
Is child care provided on site?
YOUR QUESTIONS:
WHY:
Why are these activities or strategies implemented and why
not others?
Why has the intervention varied in ability to maintain interest?
Why are clients not participating?
Why is the intervention conducted at a certain time or at a
certain frequency?
YOUR QUESTIONS:

July 16, 2002 Page 11
Validating Your Evaluation Questions
Even though all of your questions may be interesting, it is
important to narrow your list to questions that
will be particularly helpful to the evaluation and that can be
answered given your specific resources, staff,
and time.
Go through each of your questions and consider it with respect
to the questions below, which may be helpful in
streamlining your final list of questions.
Revise your worksheet/list of questions until you can answer
"yes" to all of these questions. If you cannot answer
"yes" to your question, consider omitting the question from your
evaluation.
Validation
Yes
No

Will I use the data that will stem from these questions?
Do I know why each question is important and /or valuable?
Is someone interested in each of these questions?
Have I ensured that no questions are omitted that may be
important to
someone else?
Is the wording of each question sufficiently clear and
unambiguous?
Do I have a hypothesis about what the “correct” answer will be
for each
question?

Is each question specific without inappropriately limiting the
scope of the
evaluation or probing for a specific response?
Do they constitute a sufficient set of questions to achieve the
purpose(s) of
the evaluation?
Is it feasible to answer the question, given what I know about
the
resources for evaluation?
Is each question worth the expense of answering it?
Derived from "A Design Manual" Checklist, page 51.

July 16, 2002 Page 12
Determining Methodology
Process evaluation is characterized by collection of data
primarily through two formats:
1) Quantitative, archival, recorded data that may be managed
by an computerized
tracking or management system, and
2) Qualitative data that may be obtained through a variety of
formats, such as
surveys or focus groups.
When considering what methods to use, it is critical to have a
thorough
understanding and knowledge of the questions you want
answered. Your
questions will inform your choice of methods. After this section
on types of
methodologies, you will complete an exercise in which you
consider what method
of data collection is most appropriate for each question.
Do you have a thorough understanding of your
questions?
Furthermore, it is essential to consider what data the

organization you are
evaluating already has. Data may exist in the form of an
existing computerized
management information system, records, or a tracking system
of some other
sort. Using this data may provide the best reflection of what is
"going on," and it
will also save you time, money, and energy because you will not
have to devise
your own data collection method! However, keep in mind that
you may have to
adapt this data to meet your own needs - you may need to add or
replace fields,
records, or variables.
What data does your organization already have?
Will you need to adapt it?
If the organization does not already have existing data, consider
devising a
method for the organizational staff to collect their own data.
This process will
ultimately be helpful for them so that they can continue to self-
evaluate, track
their activities, and assess progress and change. It will be
helpful for the
evaluation process because, again, it will save you time, money,
and energy that
you can better devote towards other aspects of the evaluation.
Management
information systems will be described more fully in a later
section of this
workbook.

Do you have the capacity and resources to devise
such a system? (You may need to refer to a later
section of this workbook before answering.)
July 16, 2002 Page 13
Who should collect the data?
Given all of this, what thoughts do you have on who should
collect data for your
evaluation? Program staff, evaluation staff, or some
combination?
Program Staff: May collect data from activities such as
attendance, demographics,
participation, characteristics of participants, dispositions, etc;
may
conduct intake interviews, note changes regarding service
delivery,
and monitor program implementation.
Advantages: Cost-efficient, accessible, resourceful, available,

time-efficient,
and increased understanding of the program.
Disadvantages: May exhibit bias and/or social desirability, may
use data for critical
judgment, may compromise the validity of the program; may put
staff in uncomfortable or inappropriate position; also, if staff
collect
data, may have an increased burden and responsibility placed
upon
them outside of their usual or typical job responsibilities. If you
utilize staff for data collection, provide frequent reminders as
well
as messages of gratitude.
Evaluation staff: May collect qualitative information regarding
implementation,
general characteristics of program participants, and other
information that may otherwise be subject to bias or distortion.
Advantages: Data collected in manner consistent with overall
goals and timeline
of evaluation; prevents bias and inappropriate use of
information;
promotes overall fidelity and validity of data.
Disadvantages: May be costly and take extensive time; may
require additional
training on part of evaluator; presence of evaluator in

organization
may be intrusive, inconvenient, or burdensome.
July 16, 2002 Page 14
When should data be collected?
Conducting the evaluation according to your timeline can be
challenging. Consider how
much time you have for data collection, and make decisions
regarding what to collect
and how much based on your timeline.
In many cases, outcome evaluation is not considered appropriate
until the program has
stabilized. However, when conducting a process evaluation, it
can be important to start
the evaluation at the beginning so that a story may be told
regarding how the program
was developed, information may be provided on refinements,
and program growth and
progress may be noted.
If you have the luxury of collecting data from the start of the
intervention to the end of

the intervention, space out data collection as appropriate. If you
are evaluating an
ongoing intervention that is fairly quick (e.g., an 8-week
educational group), you may
choose to evaluate one or more "cycles."
How much time do you have to conduct your evaluation?
How much time do you have for data collection (as opposed to
designing the evaluation,
training, organizing and analyzing results, and writing the
report?)
Is the program you are evaluating time specific?
How long does the program or intervention last?
At what stages do you think you will most likely collect data?
Soon after a program has begun
Descriptive information on program characteristics that will not
change; information
requiring baseline information
During the intervention
Ongoing process information such as recruitment, program
implementation
After the intervention
Demographics, attendance ratings, satisfaction ratings

July 16, 2002 Page 15
Before you consider methods
A list of various methods follows this section. Before choosing
what methods are
most appropriate for your evaluation, review the following
questions. (Some may
already be answered in another section of this workbook.)
What questions do I want answered? (see previous section)
Does the organization already have existing data, and if so,
what kind?
Does the organization have staff to collect data?
What data can the organization staff collect?
Must I maintain anonymity (participant is not identified at all)
or confidentiality
(participant is identified but responses remain private)? This
consideration
pertains to existing archival data as well as original data

collection.
How much time do I have to conduct the evaluation?
How much money do I have in my budget?
How many evaluation staff do I have to manage the data
collection activities?
Can I (and/or members of my evaluation staff) travel on site?
What time of day is best for collecting data? For example, if
you plan to conduct
focus groups or interviews, remember that your population may
work during the
day and need evening times.
July 16, 2002 Page 16
Types of methods

A number of different methods exist that can be used to collect
process
information. Consider each of the following, and check those
that you think would
be helpful in addressing the specific questions in your
evaluation. When "see
sample" is indicated, refer to the pages that follow this table.
√ Method Description
Activity,
participation, or
client tracking log
Brief record completed on site at frequent intervals by
participant or deliverer.
May use form developed by evaluator if none previously exists.
Examples: sign
in log, daily records of food consumption, medication
management.
Case Studies
Collection of in-depth information regarding small number of
intervention
recipients; use multiple methods of data collection.
Ethnographic
analysis
Obtain in-depth information regarding the experience of the
recipient by
partaking in the intervention, attending meetings, and talking
with delivery staff

and recipients.
Expert judgment
Convene a panel of experts or conduct individual interviews to
obtain their
understanding of and reaction to program delivery.
Focus groups
Small group discussion among program delivery staff or
recipients. Focus on
their thoughts and opinions regarding their experiences with the
intervention.
Meeting minutes
(see sample)
Qualitative information regarding agendas, tasks assigned, and
coordination and
implementation of the intervention as recorded on a consistent
basis.
Observation
(see sample)
Observe actual delivery in vivo or on video, record findings
using check sheet
or make qualitative observations.
Open-ended
interviews –
telephone or in
person

Evaluator asks open questions (i.e., who, what, when, where,
why, how) to
delivery staff or recipients. Use interview protocol without
preset response
options.
Questionnaire
Written survey with structured questions. May administer in
individual, group,
or mail format. May be anonymous or confidential.
Record review
Obtain indicators from intervention records such patient files,
time sheets,
telephone logs, registration forms, student charts, sales records,
or records
specific to the service delivery.
Structured
interviews –
telephone or in
person
Interviewer asks direct questions using interview protocol with
preset response
options.

July 16, 2002
Page 17
Sample activity log
This is a common process evaluation methodology because it
systematically records exactly what is happening during
implementation. You may wish to devise a log such as the one
below and alter it to meet your specific needs. Consider
computerizing such a log for efficiency. Your program may
already have existing logs that you can utilize and adapt for
your
evaluation purposes.
Site:
Recorder:
Code
Service
Date
Location
# People

July 16, 2002

Meeting Minutes
Taking notes at meetings may provide extensive and invaluable
process information that
can later be organized and structured into a comprehensive
report. Minutes may be taken
by program staff or by the evaluator if necessary. You may find
it helpful to use a
structured form, such as the one below that is derived from
Evaluating Collaboratives,
University of Wisconsin-Cooperative Extension, 1998.
Meeting Place: __________________ Start time: ____________
Date: _____________________________ End time:
____________
Attendance (names):
Agenda topic:
_________________________________________________
Discussion:
_____________________________________________________
Decision Related Tasks Who responsible Deadline

1.
2.
3.
Agenda topic:
_________________________________________________
Discussion:
_____________________________________________________
Decision Related Tasks Who responsible Deadline
1.
2.
3.
Sample observation log
July 16, 2002

Observation may occur in various methods, but one of the most
common is
hand-recording specific details during a small time period. The
following is several rows
from an observation log utilized during an evaluation examining
school classrooms.
CLASSROOM OBSERVATIONS (School Environment Scale)
Classroom 1: Grade level _________________ (Goal: 30
minutes of observation)
Time began observation: _________Time ended
observation:_________
Subjects were taught during observation period:
___________________
PHYSICAL ENVIRONMENT
Question
Answer
1. Number of students
2. Number of adults in room:
a. Teachers
b. Para-pros
c. Parents

Total:
a.
b.
c.
3. Desks/Tables
a. Number of Desks
b. Number of Tables for students’ use
c. Any other furniture/include number
(Arrangement of desks/tables/other furniture)
a.
b.
c.
4. Number of computers, type
5. How are computers being used?
6. What is the general classroom setup? (are there walls,
windows, mirrors,
carpet, rugs, cabinets, curtains, etc.)

7. Other technology (overhead projector, power point, VCR,
etc.)
8. Are books and other materials accessible for students?
9. Is there adequate space for whole-class instruction?
12. What type of lighting is used?
13. Are there animals or fish in the room?
14. Is there background music playing?
15. Rate the classroom condition
Poor Average Excellent

16. Are rules/discipline procedures posted? If so, where?
17. Is the classroom Noisy or Quiet?
Very Quiet Very Noisy
Choosing or designing measurement instruments
Consider using a resource panel, advisory panel, or focus group
to offer feedback
July 16, 2002
Page 20
regarding your instrument. This group may be composed of any
of the people listed
below. You may also wish to consult with one or more of these
individuals throughout
the development of your overall methodology.
Who should be involved in the design of your instrument(s)
and/or provide feedback?

Program service delivery staff / volunteers
Project director
Recipients of the program
Board of directors
Community leader
Collaborating organizations
Experts on the program or service being evaluated
Evaluation experts
_________________________
_________________________
_________________________
Conduct a pilot study and administer the instrument to a group
of recipients, and then
obtain feedback regarding their experience. This is a critical
component of the
development of your instruments, as it will help ensure clarity
of questions, and reduce
the degree of discomfort or burden that questions or processes
(e.g., intakes or
computerized data entry) elicit.
How can you ensure that you pilot your methods? When will
you do it, and whom will you use
as participants in the study?
Ensure that written materials are at an appropriate reading
level for the population.
Ensure that verbal information is at an appropriate terminology
level for the population.
A third or sixth-grade reading level is often utilized.

Remember that you are probably collecting data that is
program-specific. This may
increase the difficulty in finding instruments previously
constructed to use for
questionnaires, etc. However, instruments used for conducting
process evaluations of
other programs may provide you with ideas for how to structure
your own instruments.
July 16, 2002
Page 21
Linking program components and methods (an example)
Now that you have identified your program components, broad
questions, specific
questions, and possible measures, it is time to link them
together. Let's start with your
program components. Here is an example of 3 program
components of an intervention.
Program Components and Essential Elements:

There are six program components to M2M. There
are essential elements in each component that must
be present for the program to achieve its intended
results and outcomes, and for the program to be
identified as a program of the American Cancer
Society.
Possible Process Measures
1) Man to Man Self-Help and/or Support Groups
The essential elements within this component are:
• Offer information and support to all men
with prostate cancer at all points along the
cancer care continuum
• Directly, or through collaboration and
referral, offer community access to
prostate cancer self-help and/or support
groups
• Provide recruitment and on-going training
and monitoring for M2M leaders and
volunteers
• Monitor, track and report program
activities
• Descriptions of attempts to schedule and advertise
group meetings

• Documented efforts to establish the program
• Documented local needs assessments
• # of meetings held per independent group
• Documented meetings held
• # of people who attended different topics and speakers
• Perceptions of need of survey participants for
additional groups and current satisfaction levels
• # of new and # of continuing group members
• Documented sign-up sheets for group meetings
• Documented attempts to contact program dropouts
• # of referrals to other PC groups documented
• # of times corresponding with other PC groups
• # of training sessions for new leaders
• # of continuing education sessions for experienced
leaders
• # and types of other on-going support activities for
volunteer leaders
• # of volunteers trained as group facilitators
• Perceptions of trained volunteers for readiness to
function as group facilitators
July 16, 2002

2) One-to-One Contacts
• Offer one-to-one contact to provide
information and support to all men with
prostate cancer, including those in the
diagnostic process
• Provide recruitment and on-going training
and monitoring for M2M leaders and
volunteers
activities
• # of contact pairings
• Frequency and duration of contact pairings
• Types of information shared during contact pairings
• # of volunteers trained
• Perception of readiness by trained volunteers
• Documented attempts for recruiting volunteers
• Documented on-going training activities for volunteers

• Documented support activities
3) Community Education and Awareness
• Conduct public awareness activities to
inform the public about prostate cancer
and M2M
activities
• # of screenings provided by various health care
providers/agencies over assessment period
• Documented ACS staff and volunteer efforts to
publicize the availability and importance of PC and
screenings, including health fairs, public service
announcements, billboard advertising, etc.
• # of addresses to which newsletters are mailed
• Documented efforts to increase newsletter mailing list

Linking YOUR program components, questions, and methods
Consider each of your program components and questions that
you have devised in an earlier section of this workbook, and the
methods that you checked off on the "types of methods" table.
Now ask yourself, how will I use the information I have
obtained from this question? And, what method is most
appropriate for obtaining this information?
Program Component
Specific questions that go with this
component
How will I use this
information?
Best method?

Program Component
Specific questions that go with this
component
How will I use this
information?
Best method?

July 16, 2002
Page 25
Data Collection Plan
Now let's put your data collection activities on one sheet - what
you're collecting, how you're doing it, when, your sample, and
who will collect it. Identifying your methods that you have just
picked, instruments, and data collection techniques in a
structured manner will facilitate this process.
Method
Type of data (questions, briefly
indicated)

Instrument used
When
implemented
Sample
Who collects
E.g.: Patient
interviews in health
dept clinics
Qualitative - what services they are
using, length of visit, why came in,
how long wait, some quantitative
satisfaction ratings
Interview created
by evaluation team
and piloted with
patients
Oct-Dec; days
and hrs
randomly
selected

10 interviews
in each
clinic
Trained
interviewers

July 16, 2002
Consider a Management Information System
Process data is frequently collected through a management
information system (MIS) that
is designed to record characteristics of participants,
participation of participants, and
characteristics of activities and services provided. An MIS is a
computerized record
system that enables service providers and evaluators to
accumulate and display data
quickly and efficiently in various ways.
Will your evaluation be enhanced by periodic data presentations
in tables or other

structured formats? For example, should the evaluation utilize a
monthly print-out of
services utilized or to monitor and process recipient tracking
(such as date, time, and
length of service)?
YES
NO
Does the agency create monthly (or other periodic) print outs
reflecting
services rendered or clients served?
YES
NO
Will the evaluation be conducted in a more efficient manner if
program
delivery staff enter data on a consistent basis?
YES
NO
Does the agency already have hard copies of files or records

that would be
better utilized if computerized?
YES
NO
Does the agency already have an MIS or a similar computerized
database?
YES
NO
If the answers to any of these questions are YES,
consider using an MIS for your evaluation.
If an MIS does not already exist, you may desire to design a
database in which you can
enter information from records obtained by the agency. This
process decreases missing
data and is generally efficient.
If you do create a database that can be used on an ongoing
basis by the agency, you may

consider offering it to them for future use.
Page 27
July 16, 2002
Information to be included in your MIS
Examples include:
Client demographics
Client contacts
Client services
Referrals offered
Client outcomes
Program activities
Staff notes
Jot down the important data you would like to be included in
your MIS.

Managing your MIS
What software do you wish to utilize to manage your data?
What type of data do you have?
How much information will you need to enter?
How will you ultimately analyze the data? You may wish to
create a database directly in
the program you will eventually use, such as SPSS?
Will you be utilizing lap tops?
Page 28
July 16, 2002
If so, will you be taking them onsite and directly entering your
data into them?

How will you download or transfer the information, if
applicable?
What will the impact be on your audience if you have a laptop?
Tips on using an MIS
If service delivery personnel will be collecting and/or entering
information into the MIS
for the evaluator's use, it is generally a good idea to provide
frequent reminders of the
importance of entering the appropriate information in a timely,
consistent, and regular
manner.
For example, if an MIS is dependent upon patient data
collected by public health officers
daily activities, the officers should be entering data on at least a
daily basis. Otherwise,
important data is lost and the database will only reflect what
was salient enough to be
remembered and entered at the end of the week.
Don't forget that this may be burdensome and/or inconvenient
for the program staff.
Provide them with frequent thank you's.

Remember that your database is only as good as you make it.
It must be organized and
arranged so that it is most helpful in answering your questions.
If you are collecting from existing records, at what level is he
data currently available?
For example, is it state, county, or city information? How is it
defined? Consider whether
adaptations need to be made or additions need to be included for
your evaluation.
Back up your data frequently and in at least one additional
format (e.g., zip, disk, server).
Consider file security. Will you be saving data on a network
server? You may need to
consider password protection.
Page 29
July 16, 2002

Allocate time for data entry and checking.
Allow additional time to contemplate the meaning of the data
before writing the report.
Page 30
July 16, 2002
Implement Data Collection and Analysis
Data collection cannot be fully reviewed in this workbook, but
this page offers a few tips
regarding the process.
General reminders:
THANK everyone who helps you, directs you, or participates
in anyway.
Obtain clear directions and give yourself plenty of time,

especially if you are traveling
long distance (e.g., several hours away).
Bring all of your own materials - do not expect the program to
provide you with writing
utensils, paper, a clipboard, etc.
Address each person that you meet with respect and attempt to
make your meeting as
conducive with their schedule as possible.
Most process evaluation will be in the form of routine record
keeping (e.g., MIS). However, you
may wish to interview clients and staff. If so:
Ensure that you have sufficient time to train evaluation staff,
data collectors, and/or
organization staff who will be collecting data. After they have
been trained in the data
collection materials and procedure, require that they practice
the technique, whether it is
an interview or entering a sample record in an MIS.
If planning to use a tape recorder during interviews or focus
groups, request permission
from participants before beginning. You may need to turn the
tape recorder off on
occasion if it will facilitate increased comfort by participants.

If planning to use laptop computers, attempt to make
consistent eye contact and spend
time establishing rapport before beginning. Some participants
may be uncomfortable with
technology and you may need to provide education regarding
the process of data
collection and how the information will be utilized.
If planning to hand write responses, warn the participant that
you may move slowly and
Page 31
July 16, 2002
may need to ask them to repeat themselves. However, prepare
for this process by
developing shorthand specific to the evaluation. A sample
shorthand page follows.

July 16, 2002
Annual Evaluation Reports
The ultimate aim of all the Branch’s evaluation efforts is to
increase the intelligent use of
information in Branch decision-making in order to improve
health outcomes. Because we
understand that many evaluation efforts fail because the data are
never collected and that even
more fail because the data are collected but never used in
decision-making, we have struggled to
find a way to institutionalize the use of evaluation results in
Branch decision-making.
These reports will serve multiple purposes:
The need to complete the report will increase the likelihood
that evaluation is done and
data are collected.
The need to review reports from lower levels in order to
complete one’s own report
hopefully will cause managers at all levels to consciously
consider, at least once a year,
the effectiveness of their activities and how evaluation results
suggest that effectiveness
can be improved.

The summaries of evaluation findings in the reports should
simplify preparation of other
reports to funders including the General Assembly.
Each evaluation report forms the basis of the evaluation report
at the next level. The contents
and length of the report should be determined by what is mot
helpful to the manager who is
receiving the report. Rather than simply reporting every
possible piece of data, these reports
should present summary data, summarize important conclusions,
and suggest recommendations
based on the evaluation findings. A program-level annual
evaluation report should be ten pages
or less. Many my be less than five pages. Population team and
Branch-level annual evaluation
reports may be longer than ten pages, depending on how many
findings are being reported.
However, reports that go beyond ten pages should also contain a
shorter Executive Summary, to
insure that those with the power to make decisions actually read
the findings.
Especially, the initial reports may reflect formative work and
consist primarily of updates on the
progress of evaluation planning and implementation. This is
fine and to be expected.
However, within a year or two the reports should begin to
include process data, and later actual
outcome findings.
This information was extracted from the FHB Evaluation
Framework developed by Monica Herk and Rebekah Hudgins.

July 16, 2002
Suggested shorthand - a sample
The list below was derived for a process evaluation regarding
charter schools. Note the use of general shorthand as
well as shorthand derived specifically for the evaluation.
CS
Charter School
mst
Most
Sch School b/c Because
Tch Teacher, teach st Something
P Principal b Be
VP Vice Principal c See

Admin Administration, administrators r Are
DOE Dept of Education w/ When
BOE Board of Education @ At
Comm Community ~ About
Stud Students, pupils = Is, equals, equivalent
Kids Students, children, teenagers ≠ Does not equal, is not the
same
K Kindergarten Sone Someone
Cl Class # Number
CR Classroom $ Money, finances, financial, funding,
expenses, etc.
W White + Add, added, in addition
B Black < Less than
AA African American > Greater/more than
SES Socio-economic status ??? What does this mean? Get more
info on, I'm confused…
Lib Library, librarian DWA Don't worry about (e.g. if you wrote
something unnecessary)
Caf Cafeteria Ψ Psychology, psychologist
Ch Charter ∴ Therefore
Conv Conversion (school) ∆ Change, is changing
S-up Start up school mm Movement
App Application, applied ↑ Increases, up, promotes
ITBS Iowa Test of Basic Skills ↓ Decreases, down, inhibits
LA Language arts X Times (e.g. many x we laugh)
SS Social Studies ÷ Divided (we ÷ up the classrooms)
QCC Quality Core Curriculum C With
Pol Policy, politics Home, house
Curr Curriculum ♥ Love, adore (e.g. the kids ♥ this)
LP Lesson plans Church, religious activity
Disc Discipline O No, doesn't, not
Girls, women, female 1/2 Half (e.g. we took 1/2)
Boys, men, male 2 To

July 16, 2002
F
Father, dad
c/out
without
P
Parent
2B
To be

M
Mom, mother
e.g.
For example
i.e.
That is
…
If the person trails off, you missed
information
Appendix A
Logic Model Worksheet
Population Team/Program Name
__________________________ Date
_______________________

If the following
CONDITIONS
AND
ASSUMPTIONS
exist...
And if the following
ACTIVITIES are
implemented to
address these
conditions and
assumptions
Then these
SHORT-TERM
OUTCOMES may
be achieved...
And these
LONG-TERM
OUTCOMES
may be
acheived...
And these LONG-
TERM GOALS can
be reached....

July 16, 2002
Appendix B
Pitfalls To Avoid
Avoid heightening expectations of delivery staff, program
recipients, policy makers, or
community members. Ensure that feedback will be provided as
appropriate, but may or may
not be utilized.
Avoid any implication that you are evaluating the impact or
outcome. Stress that you are
evaluating "what is happening," not how well any one person is
performing or what the
outcomes of the intervention are.
Make sure that the right information gets to the right people -
it is most likely to be utilized
in a constructive and effective manner if you ensure that your
final report does not end up on

someone's desk who has little motivation or interest in utilizing
your findings.
Ensure that data collection and entry is managed on a
consistent basis - avoid developing an
evaluation design and than having the contract lapse because
staff did not enter the data.
Page 36
July 16, 2002
Appendix C
References
References used for completion of this workbook and/or that
you may find helpful for
additional information.
Centers for Disease Control and Prevention. 1995. Evaluating
Community Efforts to Prevent
Cardiovascular Diseases. Atlanta, GA.

Centers for Disease Control and Prevention. 2001. Introduction
to Program Evaluation for
Comprehensive Tobacco Control Programs. Atlanta, GA.
Freeman, H. E., Rossi, P. H., Sandefur, G. D. 1993. Workbook
for evaluation: A systematic
approach. Sage Publications: Newbury Park, CA.
Georgia Policy Council for Children and Families; The Family
Connection; Metis Associates,
Inc. 1997. Pathways for assessing change: Strategies for
community partners.
Grembowski, D. 2001. The practice of health program
evaluation. Sage Publications: Thousand
Oaks.
Hawkins, J. D., Nederhood, B. 1987. Handbook for Evaluating
Drug and Alcohol Prevention
Programs. U.S. Department of Health and Human Services;
Public Health Service; Alcohol,
Drug Abuse, and Mental Health Administration: Washington, D.
C.
Muraskin, L. D. 1993. Understanding evaluation: The way to
better prevention programs.
Westat, Inc.
National Community AIDS Partnership 1993. Evaluating
HIV/AIDS Prevention Programs in
Community-based Organizations. Washington, D.C.
NIMH Overview of Needs Assessment. Chapter 3: Selecting the
needs assessment approach.

Patton, M. Q. 1982. Practical Evaluation. Sage Publications,
Inc.: Beverly Hills, CA.
Page 37
July 16, 2002
Posavac, E. J., Carey, R. G. 1980. Program Evaluation: Methods
and Case Studies.
Prentice-Hall, Inc.: Englewood Cliffs, N.J.
Rossi, P. H., Freeman, H. E., Lipsey, M. W. 1999. Evaluation:
A Systematic Approach. (6th
edition). Sage Publications, Inc.: Thousand Oaks, CA.
Scheirer, M. A. 1994. Designing and using process evaluation.
In: J. S. Wholey, H. P. Hatry, &
K. E. Newcomer (eds) Handbook of practical program
evaluation. Jossey-Bass Publishers: San
Francisco.
Taylor-Powell, E., Rossing, B., Geran, J. 1998. Evaluating
Collaboratives: Reaching the
potential. Program Development and Evaluation: Madison, WI.

U.S. Department of Health and Human Services; Administration
for Children and Families;
Office of Community Services. 1994. Evaluation Guidebook:
Demonstration partnership
program projects.
W.K. Kellogg Foundation. 1998. W. K. Kellogg Foundation
Evaluation Handbook.
Websites:
www.cdc.gov/eval/resources
www.eval.org (has online text books)
www.wmich.edu/evalctr (has online checklists)
www.preventiondss.org
When conducting literature reviews or searching for additional
information, consider using
alternative names for "process evaluation," including:
formative evaluation
program fidelity
implementation assessment
implementation evaluation
program monitoring

The Benefits andChallenges of TrainingChild ProtectionSo.docx

More Related Content

Similar to The Benefits andChallenges of TrainingChild ProtectionSo.docx

More from arnoldmeredith47041

Recently uploaded

The Benefits andChallenges of TrainingChild ProtectionSo.docx