There is no simple path that will take us immediately from the contemporary amateurism of the college to the professional design of learning environments and learning experiences. The most important step is to find a place on campus for a
team of individuals who are professionals in the design of learning environments — learning engineers, if you will. Herbert Simon

The emerging discipline of Learning Engineering is focused on putting into place tools and processes that use the science of learning as a basis for improving educational outcomes. An important part of Learning Engineering focuses on improving the effectiveness of educational software. In many software domains, A/B testing has become a prominent technique to achieve the software’s goals. Many large companies (Amazon, Google, Facebook, etc.) run thousands of AB tests and present at the Annual Conference on Digital Experimentation (CODE), but that venue is too broad to address AB testing issues specific to EdTech platforms. We see a need to address issues with running large-scale A/B tests within the educational context, where the use of A/B testing lags other industries. This workshop will explore ways in which A/B testing in educational contexts differs from other  domains and proposals to overcome current challenges so that this approach can become a more useful tool in the learning engineer’s toolbox. Issues to be addressed are expected to include:

  • managing unit of assignment issues
  • measurement, including both short and long-term outcomes
  • practical considerations related to experimenting in school settings, MOOCs, & other contexts
  • ethical and privacy issues
  • relating experimental results to learning-science principles
  • understanding use cases (core, supplemental, in-school, out-of-school, etc.)
  • accounting for aptitude-treatment interactions
  • A/B testing within adaptive software
  • adaptive experimentation
  • attrition and dropout
  • stopping criteria
  • User experience issues
  • Educator involvement and public perceptions of experimentation
  • Balancing practical improvements with generalizable science

We welcome participation from researchers and practitioners who have either practical or theoretical experience related to running A/B tests and/or randomized trials. This may include researchers with backgrounds in learning science, computer science, economics and/or statistics.

Organizers

  • Steve Ritter, Carnegie Learning
  • Neil Heffernan, Worcester Polytechnic Institute
  • Joseph Jay Williams, University of Toronto
  • Burr Settles, Duolingo
  • Phillip Grimaldi, Rice University
  • Derek Lomas, Delft University of Technology

Registration

To register for this workshop, please select this workshop when registering for Learning @ Scale 2020.