Assessing verbal interaction of adult learners in computer-supported collaborative problem solving

Abstract

The objective of this study is to explore new ways of assessing collaborative problem solving (CPS) processes based on different modalities of audio data and their combination. The data collection took place in an educational lab setting during an experiment with adult teams from professional contexts who collaboratively solved multiple problems as part of a CPS training. From audio data, both verbal (ie, speech) and non-verbal (ie, pitch) aspects were extracted. Four analysis methods were used, including (a) content analysis; (b) linguistic inquiry and word count; (c) verbal entrainment analysis; and (d) acoustic-prosodic entrainment based on pitch data. Insights are given into the CPS processes of the participating groups using these measures and relevant relationships between some of these measures are further investigated. Based on content analysis, it was found that most of the interactions during the CPS process are task oriented, whereas team-oriented interactions are less present. Second, three measures of proportion of contribution in CPS were investigated and clear differences in participation patterns between and within teams were found. We suggest that a combination of utterance count and words per sentence could provide valuable insights for quantity and equality of participation. Third, the study explored pronoun use and found that the most frequently used personal pronouns were first-person singular. Next, the results indicated a relationship between pronoun use and the relative frequency of interactions. Fourth, a rather weak relationship between lexical entrainment measures and the acoustic-prosodic measures were found, suggesting that these measures are indicative of separate communicative aspects in CPS. This study contributes to a better understanding of which type of audio-based data is most informative to teachers and students as a feedback or assessment tool. This study complements previous research as it focuses on spoken human-to-human communication collected in an authentic context.Practitioner notesWhat is already known about this topicSupport and guidance systems for learning coaches, teachers and learners are needed to foster the educational quality of collaborative problem solving (CPS) activities.CPS is a complex process and measuring the quality of CPS processes remains challenging.Multimodal learning analytics, focusing on verbal and non-verbal data sources and using content analysis, linguistic inquiry and word count and verbal and acoustic entrainment measures could be valuable to measure the quality of CPS.What this paper addsThe majority of interactions during CPS processes are task oriented or cognitive of nature, whereas team-oriented interactions are less present.Utterance count and words per sentence should be used in combination, as they are indicative of different aspects.Pronoun use in learners' discourse is related to the types of CPS interactions.Lexical entrainment measures and acoustic-prosodic are indicative of distinctive communicative aspects in CPS.Implications for practice and/or policyQuality indicators of CPS processes should include both verbal and non-verbal measures of students' interactions.Educational researchers and the (Edtech) industry should further leverage their forces to foster the development of (semi-)automated systems for measuring the quality of CPS processes.It should be further investigated how quality indicators of CPS processes can be most meaningful to trainers, teachers and learners, for example, through the use of dashboards.