|
The following summary is based on information provided by 13 survey respondents. The summary should not be seen as describing the evaluation-related thoughts and experiences of community-based organization staff in general. We cannot generalize on the basis of thirteen responses. Keeping that caution carefully in mind, you may find that the information is interesting and thought-provoking in some ways.
Who led the evaluation?
The majority of respondents—10—reported on an evaluation project that was led by someone on their staff. Two respondents reported on an experience with an external evaluator, and one reported on an evaluation carried out through some other arrangement.
Why were you involved with evaluation?
Respondents were asked about the factors that had led to their having an evaluation project. Below are the answers and the number of times each answer was selected:
3 answers of “We needed to have a literature review for a grant proposal.”
7 answers of “We needed to include an evaluation plan in a grant proposal.”
4 answers of “We received a grant that had an evaluation requirement.”
3 answers of “Our board of directors advocated for evaluation.”
7 answers of “Evaluation activities were something that my staff and I wanted.”
2 answers of “We wanted guidance in logic model construction or a critique of a logic model
we designed.”
How did the evaluator do?
Using a scale of 1 (poor) to 4 (excellent), respondents were asked to rate the evaluator in the following areas:
- Affordability
- General competency
- Familiarity with our program area
- Understanding of our organization and its functioning
- Ability to relate to our program participants with insight, respect, and sensitivity
- Ability to work smoothly with the staff
- Working with us in a spirit of collaboration rather than being a dictator
- Welcoming our input while designing the evaluation
- Designing an evaluation that would satisfy the funder and also produce useful findings for our organization
- Choosing an outcome measurement approach that took into account the staff’s perceptions of the positive effects of program participation
- Using academic concepts or evaluation tools in ways that are workable in real-life settings
- Willingness to take on a small project when a limited amount of help was what we needed
- Doing the work that was promised
- Doing a reasonable amount of work in return for the money paid
- Communicating in terms that I and other members of the staff could understand
- Communicating with staff, when appropriate, throughout the evaluation
- Conveying findings to me and my staff reasonably soon after the evaluation was completed
- Conveying findings in an understandable and useful way
Most respondents seemed to be reporting on good experiences; the majority of ratings were 4s. As one respondent commented, “All evaluation was designed and implemented in-house by our staff—so of course we were all 4s!” There were some ratings of 2 and 3, however, and one respondent reported a uniformly bad experience—a rating of 1 was given in every single area. Although you might guess that in-house evaluators would be rated more favorably than outside evaluators, there was no pattern in which one category of evaluators received top marks and the other did not. In fact, the uniformly bad experience (ratings of 1 in every area) was with an in-house evaluator.
What might you do next time?
Respondents were asked what strategies they thought they might use the next time they had research- or evaluation-related needs. Below are the answers and the number of times each answer was selected:
6 answers of “contract with an outside, external evaluator/researcher”
4 answers of “use a staff person who is responsible for evaluation, either full time
or part-time”
3 answers of “invest in capacity-building for all of us on the staff”
2 answers of “invest in training for someone on the staff”
2 answers of “hire a new employee with evaluation skills”
2 answers of “use internet resources”
2 answers of “borrow evaluation ideas and/or techniques from another organization”
1 answer of “use on-paper guides”
1 answer of “use guidelines, measures, or other materials provided by a national or
regional office of the program for which my agency is a site”
1 answer of “use a volunteer researcher(s)”
Evaluation education interests or needs?
Respondents were asked what evaluation topics they would prefer if free or low-cost educational sessions were available for them or their staff. Each of the following topics was chosen by more than one respondent:
- Logic models
- Understanding and defining outcomes
- Understanding basic evaluation concepts
- How to cope with the research-related or evaluation parts of an RFP (Request for Proposals)
- How to stretch your dollars and make the best use of an evaluator’s time
- Finding a balance between do-it-yourself and paying for outside help
- Start-up information and coping tips for the leadership of new organizations
Overall…
In response to the question, “Do you have any complaints or frustrations related to having to think about and arrange for evaluations?”, one survey respondent said, “Hard to get entire buy in from staff; expensive if hired, very slow if donated.”
What Is Your Organization Like?
The survey respondents worked in organizations that ranged in size from 2 to 25,000 FTE staff members. The organizations were in the categories of advocacy, education, health care, and social service. One respondent was in a public agency; all others were in private nonprofit settings. All of the organizations were in large metropolitan areas. One was in the western United States; the rest were in the Washington, DC area.
Respondents were asked about the sources of funding for their organization, with the importance of each source indicated. More than one respondent reported funding from each of these categories:
- Federal government
- State government
- Local government
- Foundations
- Individual donors
- In-kind donations
and one respondent reported corporate funding as a source. There was such a variety of funding “mixes” that it is not possible to describe a typical mix.
Who Are You?
The survey respondents were development directors, executive directors, or program managers—busy people, in other words! We are sincerely, immensely grateful to them for taking the time to respond to the survey. The respondents provided information on which we can build as we pursue the question of what effects are being felt in the nonprofit world from present-day research- and evaluation-related demands.
|