Step Three: Decide on Evaluation Methods and Tools
Best Methods
Frequently people wonder what evaluation approaches are the best ways to measure success, but it is not an either/or question. While some approaches work better than others in different circumstances, you don’t have to choose between number gathering (quantitative) and story gathering (qualitative). They are both equally compelling. Arts in the Parks evaluation embraces what’s called mixed method evaluation: no stories without numbers and no numbers without stories.
Quantitative methods: focus on what can be counted, such as the number of attendees, volunteers or venues. Demographic information such as age, household income or postal code can be broken into ranges and counted.
Qualitative methods: focus on the quality of an encounter and provide ways for respondents to express their experience through a subjective lens.
Evaluation Tools
Arts in the Parks uses a variety of evaluation tools depending on the stakeholder group:
- Artists and Arts Groups: surveys and focus groups
- Audience members: surveys, field interviews, creative responses
- Community Organizers: surveys, focus groups
- Volunteers: surveys
- Summer Staff: interviews and creative reflection
Crafting Questions
Whether you are asking questions in a focus group or an interview or using an online survey, creating the right questions is an important step in an effective evaluation plan. Focus group members and interviewees need to know that you welcome both negative and positive feedback. Surveys shouldn’t be too long.
Helpful survey tips:
- Be clear and concise
- Tailor your questions and wording to suit the group you are surveying
- Avoid jargon, technical terms and acronyms
- Avoid leading questions (a type of question that sways respondents to answer in a specific manner)
- Ask one thing at a time – don’t put too much into one question
- Provide opportunities for respondents to share other information and observations in addition to the questions posed
In creating a survey, offer options such as “I Don’t Know”, “Prefer Not to Answer” or “Other”. The “Other” option, followed by an open field or writing space, gives respondents an opportunity to contribute their own ideas. Also leave a space at the end of interviews or surveys for anything else the respondent would like to share. You can create a small or larger box depending how much feedback you’d like to receive.
Counting Attendees
Getting an audience count is easy if you sell tickets. If the event is free you will need to do a head count. It’s best to have a few people doing the counting so numbers can be compared and averaged. Our approach is to collect information at the beginning of the activities when most of the audience is present. A head count gives an approximate number but still provides information about the scale of the event (small, medium, or large) and the number of people attending in different areas of the city.
Social Media Tools
You can get information about the impact of your project by looking to social media. If you created a website for your event, how many views did you get? If you record the event and post it on YouTube, how many comments do you collect? You can look at Facebook shares and Twitter retweets in the same way.
Designing and Distributing Surveys
People go to events to enjoy the activities so it’s important not to take up too much of their time. AITP uses long and short form audience surveys. Short surveys provide key feedback; they take only a few minutes, which encourages more people to participate. Long surveys take a bit more time and dig a little deeper to help us better understand the experience of the attendees. See Appendix for Arts in the Parks survey questions used in 2018.
Planning Interviews and Focus Groups
Interviewing people or gathering focus groups is an excellent way to understand the impact of your program. Recording (audio or visual) captures what was said but if you can’t record the discussion, create a note template with a section for each question. This will make it easier to pick out themes. Interviews can be structured so that they don’t deviate off into other topics, or you can approach interviews more casually, with a sense of what you want to ask but open to interviewees taking things in another direction.
Focus groups bring people together to share their experiences and often act as a catalyst for future networking. Focus groups with Arts in the Parks artists created an opportunity for them to share experiences working in the parks but also to network with each other. Though they worked in different disciplines and in different parks, the focus group provided feedback about challenges they faced and helped us make changes to the program.
Exploring Different Angles
Creative reflection can be used to gain a deeper level of engagement than surveys, interviews or focus groups. It’s most appropriate to use with people who have invested more time in an experience than a few hours; for example, those who have spent a whole day, a week or several months with a community arts project.
Arts in the Parks used creative reflection to look at the experiences of summer staff who worked from May to August in 2018. Their on-the-ground experience in parks gave them the opportunity to feel and observe park settings. In an evaluative session dedicated to creative reflection, they were asked to consider their experience by thinking about what they expected when they started, what happened along the way, and how they were feeling at the end of the program. They then created a collage (using paper, marking pens, magazines, scissors and glue) and shared it with other staff members while talking about their experience.