|
|
| |
|
 |
|
 |
A growing sense of urgency regarding the need to reform present patterns of delivering human services to children and families underscores the need for systems to have mechanisms that ensure accountability. One strong element of this reform is that human services need to become accountable for achieving measurable outcomes rather than continuing to focus on technical compliance with rules or on simple demonstration of service need. Doug Nelson, Executive Director of the Annie E. Casey Foundation notes, "It has become a well-worn observation that success in human services is too often measured by persons served or services provided and too rarely by results achieved. Difficult though it may be, the reform required is clear. Helping agencies, service programs and schools need to be held genuinely accountable for progress on specific, publicly articulated and accurately tracked outcomes for the children and families they serve." (Nelson, 1993).
Historically, human services often have received funding based on the intrinsic good of the services provided (Burchard and Schaefer, 1992). Moreover, government has tended to judge the effectiveness of public programs based on how much they cost, irrespective of whether they achieve stated goals (Gore and Brown, 1993). Recent trends in health care reform combined with efforts to demonstrate the effectiveness of systems of care for youth with serious emotional disturbance (Stroul and Friedman, 1986) have underscored the need for service systems to have mechanisms that ensure accountability. Shifting toward results-based accountability raises hopes that systems will respond more flexibly to those they serve, that public faith in the ability of human service institutions to accomplish their intended purposes will be restored, and that communities will be better able to plan their support of children and families (Schorr, Farrow, Hornbeck, and Watson, 1994).
Despite increasing demands for accountability from a variety of sources, public agencies continue to expend enormous energy in collecting information that has little or no bearing about what it is they are accomplishing. Burchard and Schaefer (1992) concur with this observation and find that human service administrators are in no way certain that the services they provide make a difference. Randall Feltman, Director of Ventura County Mental Health, provided testimony to the U.S. Congressional Select Committee on Children, Youth, and Families (1991) stating that mental health services must be able to "monitor outcomes for a child over time and across interagency environments." Without this ability, Feltman believes that services will flounder and taxpayers will find little reason to spend their money on public services.
Interest in outcome accountability is leading to discussion about the strategies that must be created in order to move systems from compliance oriented data collection to outcome-based measurement and from rule-driven decision making to decision making based on practical data (e.g. Burchard and Schaefer, 1992; Cohen and Ooms, 1993; Usher, 1993 a, b; Weiss and Jacobs, 1988). This paper provides a conceptual and practical framework for building and using outcome-oriented information systems for public and private child-serving agencies. This framework, The Ecology of Outcomes, is intended to help agencies serving children and families create strategies to collect and utilize information about who is being served, what services are being offered, and what functional outcomes are being achieved. The assumption underlying the Ecology of Outcomes is that an outcome-oriented information system is essential to the successful functioning of any child-serving system.
Emphasis on outcome accountability in mental health is not a new concept. The Community Mental Health Centers (CMHC) Amendments in 1975 (P.L. 94-63) contained extensive program evaluation requirements. CMHCs were required to evaluate the efficacy of their own services through a self-evaluation process that included citizen review and participation (Flaherty and Windle, 1981). This measurement process was intended to improve functioning of CMHCs in that systematically collected information on client outcome would lead to better decision making and improved program operations (Elmore, 1978; Flaherty and Windle, 1981; U.S. General Accounting Office, 1976; Wortmann, 1975). Unfortunately, the promise of outcome-based accountability in CMHCs failed to receive sufficient attention by the field of mental health.
Tensions between program evaluation methodologies and the needs of practicing managers have contributed to the inadequacy of information system development in child-serving systems (Bickman, Heflinger, Pion and Behar, 1992; Stevenson and Longabaugh, 1980; Usher, 1993, b; Weiss and Jacobs, 1988). Some problems associated with evaluation approaches include: adversarial relationships between program staff and evaluators; detached measurement of outcomes from a position of scientific objectivity; measurement of long-term outcomes with less regard for assessing how a particular intervention leads to outcomes, and lack of ongoing feedback from evaluator to program staff so findings can be readily incorporated into agency plans (Usher, 1993, a; Cohen and Ooms, 1993). Weiss and Greene (1992) believe that many approaches to evaluation have played into a never-ending search for the "perfect" program model, rather than offered an understanding of how to make promising programs work better. These criticisms are not meant to be exhaustive. However, they underscore several of the fundamental difficulties that have stalled the development of effective and useful information systems. Heather Weiss, Director of the Harvard Family Research Project, enumerated additional aspects of program evaluation methods that prevent accountable systems from developing (Cohen and Ooms, 1993). Most notably, many methods do not aid programs in internalizing a process of assessment, feedback, and mid-course correction so that evaluation can be ongoing and facilitate self-correction. In addition, key stakeholders typically do not provide input on outcomes used in evaluations. Evaluators often select outcomes that bear a limited relationship to the goals of the program under evaluation. Consequently past evaluation efforts have not been explicitly tied to accountability, and information generated by many methods of evaluation have not been useful for management decision making and improvements in service delivery.
Renewed interest in accountability and the role evaluation can play is leading to discussion about the information strategies that must be created in order to move systems from compliance-oriented data collection to outcome-based measurement and from rule-driven decision making to decision making based on practical data (Burchard and Schaefer, 1992; Cohen and Ooms, 1993; Usher, 1993 a, b; Weiss and Jacobs, 1988). A central theme of this approach is that pertinent outcome indicators, tracked with regularity over time, can provide timely and useful feedback to an organization, so that improvements in service delivery can occur as needed (Casas, 1992). The focus in this approach is more related to how information can be used so that an organization improves over time. Foremost, evaluations must have managerial relevance (Bartlett and Cohen, 1993; Greene and Newman, 1993; Stevenson and Longabaugh, 1980). This is in contrast to evaluations that provide reports within a time frame that has no correspondence to the time frames within which managers operate. Further, the data that is gathered by outside evaluations is often more relevant to a particular theoretical framework than to the practical issues faced by both consumers and service providers.
Usher (1993b) provides a comprehensive description of how an organization's self-improvement strategy should be developed. According to his model, self assessment begins with a collaborative process which involves all key stakeholders for establishing the service system's desired outcomes. Next, the program identifies the intended recipient of services and the form of services provided. As noted above, this information helps the agency or program to determine whether services are provided as they were intended. This descriptive information allows an agency to interpret the meaning of their outcome data. For example, outcome data may indicate that an agency's program is not leading to expected change in a selected outcome. However, information about what services were delivered may indicate that the most plausible reason for this result was due to problems in implementing the service program as intended. This type of internal evaluation strategy will enable a service system to continually improve its service delivery.
The development of recent quality improvement strategies also coincides with new approaches to program evaluation. Typically quality assurance processes hold programs accountable to minimum standards (Berwick, 1989). Berwick believes that these standards usually become "ceilings" rather than "floors" and prevent achievement of excellence. They also usually reflect technical compliance with rules related to practice standards (Evans, Faulkner, Hodo, Mahrer, and Bevilacqua, 1992). New approaches are shifting the focus from compliance with predetermined standards to an approach that promotes continuous internal evaluation and corrective action (Berwick, 1989; Fountain, 1992). Total quality management is one of these approaches to improve service quality (Fountain, 1992). Here, an organization is oriented toward a focus on the structure, process and outcome of its services. Additionally, regular and structured feedback is disseminated throughout the organization.
Information on the effectiveness of services, as well as the ability of service providers to respond to information about their effectiveness, is relevant given the recent attention to health care reform and managed care approaches. Greene and Newman (1993) point out that mental health services in the public sector are rapidly undergoing revision as more states adopt managed care approaches. These approaches require fast and easy access to more reliable and valid data than other approaches to the delivery of mental health services (Greene and Newman, 1993). However, much of the information collected in managed care is focused on the costs of services and the management of resources instead of attention to the client outcomes that are of importance to consumers and to the public at large.
Newman and Sorensen (1985) suggest that managed care entities can combine fiscal information with outcome data in order to maximize the quality of care within available resources. Managed care approaches that are effective achieve a synthesis of outcome and cost (Eckert, 1994; Lonborg and Fenster, 1992; O'Leary, 1992). In simple terms this synthesis requires knowing who is being served, what services and costs are associated with the service population and the associated outcomes of these services. Adding measurable benefits or outcomes to the managed care formula is crucial if a reform towards accountability is to be included in the current trends in health care reform. Bartlett and Cohen (1993) state that the recent and rapid increase in managed mental health and substance abuse care is actually part of a more general movement towards outcome monitoring throughout all of health care. However, as Bartlett and Cohen (1993) indicate, no other area of health care reform has as much at stake nor needs as much evidence of positive outcome as do the areas of mental health and substance abuse.
Return to The Ecology of Outcomes Home Page
 |
 |
 |
CFS Home | CFS Centers & Projects | CFS Publications | CFS News | CFS Faculty & Staff | CFS Divisions Copyright © 2005, Dept. of Child & Family Studies, Louis de la Parte Florida Mental Health Institute -- see terms of use. |
|
To contact us
about this website, write us at cfswebmaster@fmhi.usf.edu To correspond with employees of the department, write to them care of: The Department of Child and Family Studies Louis de la Parte Florida Mental Health Institute University of South Florida 13301 Bruce B. Downs Blvd. Tampa, FL 33612-3807 |