The Comprehensive Assessment Of Team Member Effectiveness: A New Peer Evaluation Instrument

Abstract

Abstract NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract The Comprehensive Assessment of Team Member Effectiveness: A New Peer Evaluation Instrument Abstract A multi-university research team has designed a peer evaluation instrument that is simple to use. The system gathers data from students through a web interface, ensuring the confidentiality of the peer ratings. The system analyzes the data to calculate suggested grade adjustments for equitably distributing a team’s grade among the team’s members. The system also provides extensive feedback to faculty as to certain dynamics of student teams that can be discerned from the peer evaluation data. Reliability and validity studies are under way. This paper includes a description of the behaviorally anchored rating scales, the electronic interface, and the feedback it provides. The design of a new instrument The instrument was developed from a broad base of teamwork research. The identification of potential items from the teamwork literature, the creation of a Likert-scale instrument, and the use of exploratory factor analysis and confirmatory factor on a large survey sample to reduce the instrument and identify the factor structure are in press elsewhere.1 Our earlier published work described the importance of assessing teamwork in the engineering classroom and the challenges it represents and laid out the ambitious assessment plan that would help develop an instrument that is easy to use and yet meaningful for both faculty and students, 2 described and demonstrated the benefit of a behaviorally anchored rating system,3 detailed the process of creating a new behaviorally anchored rating scale to simplify administration, data analysis, and reporting, and make feedback more understandable.4 This paper shows how the behaviorally anchored rating scales are incorporated into an electronic interface, how the database is designed, and the feedback it provides. Designing a complicated database to keep the administration and reporting simple To make the peer evaluation simple for students and faculty, the database schema became quite complicated, summarized in a 20-page functional requirements document. The system includes views for administrators, faculty, and students, and has modules for password protection and consent and reporting. The instrument is called the Comprehensive Assessment of Team Member Effectiveness, referred to as CATME for short. Administrator interface: The administrator grants access to the system after confirming that the request has come from a faculty member. The interface shows active and pending faculty accounts, keeps track of each faculty’s last login date and time. This interface also provides access to the raw data for surveys released for research purposes. Faculty interface: Participating faculty enter information about classes using the system, populate a class with students, populate teams of students within a class, and set up surveys for team activities. The faculty interface also controls the instructions given to the students, the

Topics & Concepts