Program Evaluation

What is program evaluation?

Program evaluation is the systematic measurement of program processes and/or outcomes with the aim of increasing program success. At data2insight, we work closely with stakeholders to design and implement an evaluation that answers critical questions regarding program effectiveness, efficiency, and impact. For example, during program implementation, we can provide formative evaluation findings that program staff can use to inform program improvement decisions. We can also provide summative evaluation towards the end of a program or upon its completion, often required by funding agencies and used to make decisions about program continuation, expansion, and funding. Finally, developmental evaluation combines aspects from both formative and summative evaluation in order to facilitate continuous program improvement.

How do evaluation and research differ?

Evaluators use many of the same qualitative and quantitative methodologies used by researchers. While data2insight program evaluation is as rigorous and systematic as academic research, the primary purpose of evaluation is to answer the questions of program staff, funders and other stakeholders rather than answer our own research questions. We also aim to provide timely and constructive information for decision-making about your programs. Accordingly, data2insight is client-focused. We tailor the design of evaluations to meet the needs of your program with your time and budget constraints.

How much does program evaluation cost?

The cost of an evaluation is entirely contingent upon the scope and nature of the evaluation activities and questions to be answered. The National Science Foundation’s “rule of thumb” about evaluation budgets is 10% of the total grant amount. Data2insight is committed to providing cost effective evaluation plans that are both responsive to your program’s evaluation needs and time and budget constraints. We work with a wide-range of programs and projects which vary in scope and budget.

Can you help me with the evaluation section of my grant proposal?

Many federal agencies and foundations require that proposals include information about how the effectiveness of the proposed program will be evaluated. This section usually contains a brief description of possible metrics for program outcomes and a plan for both formative and summative evaluation of the program. Data2insight can provide text for the evaluation section of your proposal at no charge with the understanding that we will work together when your grant is funded. Depending on time and available resources, we can also provide feedback on the grant as a whole and guidance on the development of your goals and outcomes. The best way to begin is to as early in the proposal process as possible.

When should we get an evaluator involved in our grant and/or program?

We like to get involved with a project or program as early as possible as we find that it can save time and money for everyone involved. Data2insight can help inform program design with our knowledge of evaluation methods, improve your grant proposal by including a robust evaluation section, and help your program get off to a great start by introducing evaluation practices from the outset. We recognize that project time lines and planning processes may not always be ideal. Data2insight is responsive to your time constraints and will work with you to meet proposal deadlines. We do request that before including data2insight as an evaluator in your grant proposal that you contact us.

Past project:

Veronica Smith acted as co-lead of the external evaluation for this National Science Foundation (NSF) funded program designed to increase the recruitment and retention of women into course at 8 different community colleges.

The purpose of this program evaluation was to gather feedback from project stakeholders to inform project decision-making and report progress to NSF. Our evaluation included collection of data to review progress made toward meeting outcomes, an assessment of technical assistance and support quality, and an overall program performance review. The evaluation also included tracking and reporting of targeted course enrollment and completion rates.

Our team gathered information from project participants, including the project Key and Co-Leaders, Leadership Team members, college instructors and female students from targeted courses, and the program’s Principal Investigator. In year four, the evaluation team administered surveys, conducted key informant interviews, observed and presented at an annual Project Partner Meeting, reviewed documents, and collected and analyzed targeted course enrollment and completion data. The team also completed an outcome analysis that looked at the relationship between program implementation by each of the eight community colleges and changes in female student enrollment and completion. The outcome analysis also looked at the relationship between a measure of college leadership participation and changes in female student enrollment and completion at each college.

Evaluation methods employed to answer questions regarding progress-to-date, technical assistance and support quality and program performance:

Self-Report Surveys

Online and paper surveys were used to gather information about the program from Key and Co-Leaders, Leadership Team members, instructors and female students. Survey questions solicited feedback in closed and open-ended questions about project implementation and dissemination and the quality of project extension services. All surveys were refined per NSF Reverse Site Visit recommendations and included questions about attitudes toward STEM. Surveys were created in collaboration with program staff and then cognitively tested for readability and understanding. Technical assistance was available to respondents during survey administration. Survey data were analyzed using quantitative statistical applications or qualitative methods, depending on the question format.

Interviews

Information phone interviews were conducted with all Key and/or Co-Leaders. In addition, a small sample of instructors was interviewed, as well as the Principal Investigator. A semi-structured interview protocol was used to gather information about challenges and successes faced during college plan implementation, retention strategies and resources, project quality and support, and recommendations for improving the program. Interviews were audio-recorded to capture direct quotations and analyzed using qualitative methods.

Observation of Project Partner Meeting

Two evaluation team members were present at the Project Partner Meeting. Observation data and notes taken at the meeting were used in the development and refinement of measures used in the evaluation and this report.

Document Review

Project documents such as monthly phone call minutes, college project plans, training surveys, etc. have been reviewed and findings were used to inform evaluation results.

Enrollment and Completion Reporting

With the intention of helping CWT leaders and instructors better understand enrollment and completion patterns and inform decisions, the evaluation team redesigned the enrollment and completion reports. New and enhanced features include:

• Introductory and advanced course summaries in addition to an all course overview
• Summary chart featuring average number of females enrolling and completing targeted courses
• Reporting of average student number per course by term for robust comparison across courses and colleges
• Percentages and averages reported out to one decimal place for higher level of accuracy
• Color coding for quick orientation.

These reports were updated quarterly and were with the PI and college Key and Co-Leaders to inform decision-making regarding instructor training and program improvements. The report data was also used to measure changes in enrollment and completion rates for introductory and advanced courses by college.

Implementation & Outcome Analysis

College leadership teams participating in this program were initially provided training that informed them of the range best practices (AKA strategies) for recruiting and retaining female students. College teams committed to the following strategies: maintaining a web site, creating and disseminating posters and other recruitment collateral, attending the annual project partner meetings and providing data for evaluation. Beyond these commitments, program staff recommended that the leadership teams include at least three recruitment strategies and three retention strategies in their strategic plans. The teams then determined which strategies to include in their yearly plans. During monthly calls with program staff, teams typically reviewed progress made against their strategic plan and program staff provided resources, solutions and recommendations on implementation. Leadership teams and program staff reviewed what action items had been completed and made adjustments to the plan as deemed necessary. The evaluation team categorized the recruiting and retention strategies listed in each college’s project implementation plan and then determined the extent of the implementation of categorized strategies for each college. A rating scale from 0-4 was used to measure completion of action items as follows:
Recruiting & Retention Plan Implementation Rating Scale
0=nothing accomplished to date
1=progress made, but not completed
2=completed action item more than 2 months after targeted date
3=completed action item within 2 months after targeted date
4=completed action item on time

A percentage of plan completion was determined by dividing the overall actual plan rating by a possible overall rating (possible overall rating = number of action items in the plan multiplied by four). The percentage of plan completion served as an indicator of implementation fidelity. The relationship between implementation fidelity and the change in enrollment and completion rates was examined at the college level with a bivariate correlation analysis that used the percentage of plan completion as the independent variable. An examination of the data using histograms and scatterplots was conducted and revealed no serious threats to underlying distributional assumptions of the residuals.

Participation & Outcome Analysis

The evaluation team and program staff identified participation indicators. These indicators included attendance on monthly phone conference calls in year four, annual site visit attendance, project partner meeting attendance, and the number of CWT presentations made since program inception. An Overall Participation Indicator (OPI) was calculated by totaling the number of attendees at each call, site visit and/or project partner meeting. If there was a Key- or Co-Leader in attendance the total attendance was multiplied by two to capture a factor of influence by team leads. The number of presentations made by each school was then added to that total. The relationship between the OPI and the change in enrollment and completion rates was then explored at the college level with a bivariate correlation analysis that used OPI as the independent variable. An examination of the data using histograms and scatterplots was conducted and revealed no serious threats to underlying distributional assumptions of the residuals.

• Services

  • Data Visualization
  • Evaluation
  • Research
  • Strategic Planning