Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States
Amy Zhang, Michele Igo, Marc Facciotti, and David Karger
Determining affective states such as confusion from students' participation in online discussion forums can be useful for instructors of a large classroom. However, manual annotation of this data by instructors or paid crowd workers is both time-consuming and expensive. Additionally, it can be difficult, especially for paid crowd workers, to distinguish confusion from other affects that appear similar, such as curiosity. Instead, we harness affordances prevalent in social media to allow students to self-annotate their forum posts with hashtags and emojis. This allows instructors and students to locate salient messages as well as provides an easier way to acquire a labeled dataset of nuanced emotions. From a dataset of over 25,000 discussion posts from two courses self-annotated by students, we demonstrate how we can identify linguistic differences between posts expressing confusion versus curiosity, achieving 82% accuracy at distinguishing between the two affective states.