Assessment and Evaluation of Arts Education
M. Christine Dwyer
RMC Research Corporation, Portsmouth
Evaluation of Artists in Residence and Arts Education Projects
(from a workshop presented at the 1999 Arts in Education Conference)
The Purposes of Evaluation are:
to identify what participants have learned; and to reflect on what to preserve, what to replicate, what to change. In this way, evaluation is different from documentation, research, and assessment.
Steps in Conducting an Evaluation
1. Identify guiding evaluation questions. You can get to those questions by thinking about: What do you really want to know? What do your colleagues want to know?
What does your principal want to know? What would other artists want to know? What does your funder want to know? What do the parents of the students want to know?
2. Determine approach for collecting information based on the time available from participants, the nature of the project being evaluated, you reasons for conducting the evaluation, and how much information you can reasonably summarize.
3. Design the instruments or procedures you will need.
4. Schedule the evaluation into your overall planning. Notify participants about your intenti0ons as you have appropriate opportunities and schedule their participation as part of the overall process.
5. Implement the approach and collect the data.
6. Extract key themes and summarize the learnings and the value of the project. Share those summaries with colleagues, principal, funders, parents, artists, students.
Ways to Approach Collecting Information
Below are seven ideas for approaches to consider that emphasize reflection. As you will quickly see, many of the ideas can be used in combination to provide a richer evaluation experience.
1. Summarize ratings of project achievements/outcomes from different perspectives
Summarize lists of possible specific project outcome statements in different categories (e.g., categories might include learning about making and appreciating art, learning about oneself and one's interaction with others through art). Ask different groups of participants to identify which statements are true of the project. Also ask them to identify the most important outcome achieved in each category. Summarize by tallying the results across participants within groups. In other words, list and compare the achievements that were most highly rated by parents, students, teachers, participating artists, etc.
2. Hold structured discussions and focus groups
Organize small groups of people who have participated in the project in different role, teachers, artists, parents, students. Address a few large themes or questions, i.e., What was the most important skill that teachers and students learned from their participation in the project?, and gather feedback from the full group in response to those questions. The goal is to have rich, deep discussions around a few topics and allow everyone's perspective to be heard. For more variety, incorporate other approaches in the discussion such as signaling priorities by voting or writing "last word" comments. Allow time at the end for the whole group to work on summarizing the big points that participants made.
3. Create a logic or theory map of the project
Work with a core team that was closely engaged with the project to "map" the overall project, focusing on how activities and processes produced short term and longer term outcomes. It is helpful to do such a map at the beginning of a project to imagine the unfolding of the project. Once the project has concluded, the team would modify the map to reflect how things really evolved. A map could be as simple as filling in columns labeled project elements, project activities, short term outcomes, long term outcomes. Examples of short term outcomes for the stages in a group-developed dance might be total group engagement in practicing movement sequences and development by each child of 1 to 2 ideas that could be built into a dance. Longer term outcomes might be about development of understanding that dance requires a variety of specific skills and that choreographers develop their ideas for dances from motions, forms, feelings, and stories.
4. Review project portfolios/collections of work/demonstrations of work
Collect student work from different stages of the process along with the student, teacher, and artist commentary on the work in the form of notes or tape recorded comments. Use the portfolio of work from a few students or the full group participating as the basis for a review and discussion of what students have learned from the project. Focus on: (a) the way work is evidence of individual student development; (b) characterizing the common outcomes for students evident in a variety of work; (c) what specific instructional strategies helped to produce the optimal growth in students. Portfolio/demonstration review can be a terrific opportunity for engaging people who have not been very close to the project, e.g., a School Board member or a teacher from another grade level, along with some project participants.
5. Engage an expert reviewer to comment on a portfolio of work/set of demonstrations
In some projects it might be ideal to involve an expert or two, e.g., another artist from the same discipline who has not been involved in the project, in review of a portfolio of work or set of demonstrations as described above. The expert reviewer would review the work and the commentaries, and then develop some oral or written summaries from his/her perspective about what outcomes the project produced, what the strengths of the project were, and what were the most important learnings. It would be critical for a core team of project participants to interact with the expert reviewer and probe the summary comments to learn more about the independent perspectives. A powerful reflection would be for project participants to carry out a similar exercise on their own and compare their findings with the expert's comments.
6. Shadow a student through the project and develop an individual student profile
Concentrate on describing the interactions of one student with the project from beginning to end, gathering work and perspectives along the way from a variety of sources. develop a written description of the student's interaction with the project as the basis for a detailed reflection by a core group on: (a) what the student obtained from participation; (b) how that compares with the experiences of other students; (c) what experiences and strategies were especially powerful in producing student outcomes; (d) how could the project have been even more powerful for the student. Capture the summary themes from the reflection.
7. Ask a participating classroom teacher to keep a written, email-ed, or tape-recorded diary
This approach is similar to the student profile but taken from the perspective of one teacher's journey through the project. As with the student, the value is in group and individual reflection about the meaning of the experiences. Focus on: (a) what is the teacher learning about her student; (b) what is the teacher learning about herself/himself as a teacher; (c) what are the most powerful experiences; (d) what are the low points, stresses, or points of doubt about project value.
Contact: M. Christine Dwyer
RMC Research Corporation
1000 Market Street
Portsmouth, NH 03801
800/258-0802 603/422-8888 fax: 603/436-9166
Click here for assessment and evaluation resources
