Comment Ranking Diversification in MOOC Forum Discussions
Curtis Northcutt, Kimberly Leon, Karson Ota, and Naichun Chen
Viewing consumption of online course forums with hundreds of comments relies largely on ranking, since most users only view top-ranked comments. When comments are ranked by an ordered score (e.g. number of replies or up-votes) without ad- justing for semantic similarity among near-ranked comments, top-ranked comments are more likely to emphasize the ma- jority opinion, as well as incur redundancy. In this paper, we propose a top K comment diversification re-ranking model using Maximal Marginal Relevance (MMR) and evaluate its impact in three interest areas, (1) semantic diversity, (2) in- clusion of the semantics of lower-ranked comments, and (3) redundancy, within the context of a HarvardX course discus- sion forum.
We conducted a small-scale re-ranking evaluation experiment requiring subjects to choose among two unidentified ordered lists of comments, (1) the top 5 comments of our diversified ranking and (2) the top 5 comments of a baseline ranking ordered only by score, their true identities unknown. For three subjects, across 100 trials, we found a significant (1) in- crease in diversity, (2) increase in inclusion, and (3) decrease in redundancy using a 75% score, 25% similarity trade-off. Within each interest area, inter-rater reliability showed mod- erate consistency, with typical Cohen-Kappa scores near 0.2. Our findings suggest that our model improves (1) diversifica- tion, (2) inclusion, and (3) redundancy, among top K ranked comments in online course forum discussions.