Program Development & Evaluation

Program Development & Evaluation

Home

Our Services

About Us

Recent Clients

Referral

Contact Us

Our Program Development Processes Program development is only as strong as its weakest link. Development of successful programs involves two central ideas. The first concept is that programs are developed, beginning with certain theoretical notions of partipants' crucial needs and relevant mechanisms of change. The second concept is that the movement from theory to the completed program involves a series of steps or stages. What these stages are will vary widely across organizations and projects. Due to such variability, a generalizable model of program development is in demand, one that integrates several theories and steps from different schools of thought. Sussman's (2001) Six-Step Chain model is an effective integrated model that isolates and strengthens the weakest link in the process of steps that integrate several well-known and empirically supported methods of program development. We have adopted this model based on its utility and comprehensive integration of several effective program development methods. The six steps are as follows: *Assessment of Organization's Needs* This includes staff at all levels, and consumers or customers of the organization. Concurrent to assessment, program mediation is developed to address the emergent needs. *Systematic Pooling and Warehousing* This includes searching for promising, empirically supported organizational activities or interventions to test. *Systemization of Perceived Efficacy Studies* In this process we screen among promising activity and intervention ideas gathered in the pooling and warehousing stage. Our efforts result in effective contrasting of found activities and interventions that narrow down the pool to acceptable, relatively time-and-cost effective program components. *Systemization of Immediate-Impact Studies* This process determines the workability of individual program components. We evaluate how well or ineffective components are together which allows us to narrow the margin of error in program implementation. *Systemization of Program Construction & Pilot Testing* We consider program content and process sequencing, along with the pragmatics of testing a complete program. *Refinement* In this process we refine a set of immediate post-test/post-treatment activity measures that predict longer-term outcomes from short-term measures. Throughout the completion of each of these steps, ancillary methods are applied as needs for fine tuning are identified. One of these ancillary methods include *Concept Mapping. Concept Mapping* This technique of collecting and evaluating programming data is utilized for representing knowledge in graphs. The knowledge graphs that are produced in the mapping process are networks of concepts related to the data collected. These networks consist of nodes (points/vertices) and links (arcs/edges). Nodes represent concepts and links represent the relationships between those concepts. Concepts and links may be categorized, they can be simply associative, specified, or divided in categories such as causal or temporal relationships. Concept Mapping has many uses. For program development purposes concept mapping is often utilized for: Generate ideas (brain storming, etc.); Designing complex information structures (long texts, hypermedia, large web sites, etc.) Communicating complex ideas Facilitating learning by explicitly integrating new and old knowledge Assessing understanding or diagnosing misunderstanding Concepts and links may be categorized, they can be simply associative, specified, or divided in categories such as causal or temporal relationships. Concept Mapping has many uses. For program development purposes concept mapping is often utilized for: Generate ideas (brain storming, etc.); Designing complex information structures (long texts, hypermedia, large web sites, etc.) Communicating complex ideas Facilitating learning by explicitly integrating new and old knowledge Assessing understanding or diagnosing misunderstanding		Our Program Evaluation Processes We adhere to the program evaluation standards put forth by the Joint Committee on Standards for Educational Evaluation which are accredited by the American National Standards Institute. Our evaluation process is comprehensive and encompasses three broad steps. These steps often do follow each other succinctly, however sometimes overlapping. *Developing an Evaluation Plan* This process involves us reviewing our evaluation toolbox to decide upon the method or methods to utilize. This step sets the stage for step 2. Depending on the program and organization, the methods used will consist of either one or a combination of the following: Proactive Evaluation - This involves needs assessment and research synthesis. In this method of evaluation a review of exemplary practice is performed and an establishment of benchmarks occurs. Clarificative Evaluation - This method of evaluation identifies the "real goals" or intentions of a given program as distinct from the stated ones. This allows for management of the salient characteristics of a program and accurate formulation of processes and outcomes. This method also allows us to identify unrealistic goals that are unachievable through the program and for which managers could or should not be held accountable. The end result of this method is a clarification and better understanding of the underlying program logic. Interactive Evaluation - This method of evaluation allows direct organizational control over the program, how it is delivered, and how it is evaluated. As the evaluators we provide input and support for the organization's agenda. The evaluation efforts are internal and are influenced strongly by key individuals within the organization who are close to the action. Our role is to employ interactive strategies to encourage utilization of findings. This method relies heavily on the existing context within the organization and our craft knowledge to drive the working knowledge of the key individuals within the organization. This method focuses on organizational change and improvement on a continuous basis. It is an empowerment method for organizations and their people. Monitoring Evaluation - This method of evaluation is formative or ongoing. It involves checking the delivery of specific program plans to ensure they are on-track and that specified levels of outcomes are being achieved. It often involves developing management information services that can provide responsive, valid, and useful information for assessing program delivery and outcomes. This method provides evidence that stakeholders, administrators, or managers can report on when having to present progress to funding agencies or community groups. It also develops mechanisms by which programs can be fine-tuned on the basis of the findings provided. Impact Evaluation - This method of evaluation determines the range and extent of outcomes of a program. It investigates whether or not a program has been implemented as planned and how implementation has affected outcomes. This method also provides evidence to funders, senior managers, and any-other stakeholders about the extent to which resources allocated to a program have been spent wisely. By utilizing this method we help organizations make decisions about replication or extension of a program. *Implementing an Evaluation Design* The emphasis in this stage is on data management or the collection and analysis of evidence. We take an interest in the programs we evaluate and take a stance of hopeful curiosity. This allows us to adopt a "critical friend" position as a means of influencing meaningful and effective use of the findings. *Disseminating Findings to Stakeholders* Our staff has the craft-knowledge and communication skills to disseminate our findings to the stakeholders in the organization. We adopt a pragmatic perspective, and often select from a limited range of evidence from which to present our findings, make conclusions, and meet deadlines. It is our commitment to not transform our evaluation project into a larger project than the one we are evaluating. When we have fulfilled this commitment with solid effectiveness, we know we are providing a reputable and respectable evaluation service.