TITLE: 
Getting Too Personal(ized): The Importance of Feature Choice in Online Adaptive Algorithms

ABSTRACT:
Digital educational technologies offer the potential to customize students' experiences and learn what works for which students, enhancing the technology as more students interact with it. We consider whether and when attempting to discover how to personalize has a cost. We explore these issues in the context of using multi-armed bandit (MAB) algorithms to learn a policy for what version of an educational technology to present to each student, varying the relation between student characteristics and outcomes and also whether the algorithm is aware of these characteristics. Through simulations, we demonstrate that the inclusion of student characteristics for personalization can be beneficial when those characteristics are needed to learn the optimal action. In other scenarios, this inclusion decreases performance of the bandit algorithm, and we characterize the size of that negative impact in settings similar to those that occur in practice. Moreover, including unneeded student characteristics can systematically disadvantage students with less common values for these characteristics. Because curriculum designers and researchers do not a priori know the true relationship between student characteristics and intervention effectiveness, this raises questions of fairness, which we also explore by considering a more exploitative bandit algorithm. Despite the potential risks, our simulations do suggest that real-time personalization can be helpful in certain real-world scenarios, and we illustrate this through case studies using existing experimental results in ASSISTments (Selent et al. 2016). Overall, our simulations show that adaptive personalization in educational technologies can be a double-edged sword: real-time adaptation improves student experiences in some contexts, but the slower adaptation and potentially discriminatory results mean that a more personalized model is not always beneficial.

AUTHORS:
Zhaobin Li, Luna Yee, Nathaniel Sauerberg, Irene Sakson, Joseph Jay Williams, Anna Rafferty

NOTE:
Presented in the workshop as part of the ENCORE track. A version of this paper was previously published at Educational Data Mining 2020, and PDF is available at: http://tiny.cc/GettingTooPersonalized