Exploring the Need for Transparency in Educational Recommender Systems

Jordan Barria-Pineda

DC paper’s DOI: https://doi.org/10.1145/3340631.3398676

Educational Recommender Systems (EdRecSys) are different in nature from conventional Recommender Systems (RecSys) -mostly related to e-commerce- as the main goal of EdRecSys is supporting students learning’ instead of maximizing users’ satisfaction from consuming the recommended items. Thus, research on transparency for traditional RecSys is hard to transfer from e-commerce contexts to educational scenarios, as the level of knowledge of the end-user (i.e. the student) is crucial for generating and evaluating the impact of the recommendations on students’ learning.

Research Questions

• RQ1: What are the most common learning goals that students set to guide their learning in online learning platforms that could be supported by providing learning content recommendations?.

• RQ2: What type of (a) explanations and (b) presentation formats (e.g. textual, visual)
explanatory interfaces would be most appealing and meaningful to students for obtaining information about why specific learning materials were recommended to them?

– RQ2.1: What information about the recommendation process (e.g., input data, algorithm) should be disclosed?

– RQ2.2: How should the display format (e.g., visual, textual or a combination) and the interactivity of the explanations?

– RQ2.3: What are the trade-offs between the amount of information provided and the format/interactivity of the explanations? (in terms of complexity and usefulness for learners).

• RQ3: What are the effects of providing learning content recommendations’ explanations on students’ behavior within an online educational system and their associated perceptions and learning?

Proposed Approach

I plan to use different HCI research methods (e.g. semi-structured interviews, rapid prototyping) to elicit the real-world students’ needs for support that could be covered by a learning content recommender system. By knowing this, I will explore what are the most appropriate explanatory interfaces that students will value more to get information about how the recommendation process works and how it helps them to support their learning process.

Progress-to-Date

We have already performed some classroom experiments as initial exploratory attempts for addressing the aforementioned research questions. We developed two variants of an online programming learning system (called Mastery Grids, see Fig. 1 and Fig. 2) that include different recommendation approaches for suggesting programming learning activities along with different explanatory interfaces:

• A classroom experiment was run where we implemented a Mastery Grids version in an introductory Java course with a rule-based recommendation engine designed for supporting students’ knowledge expansion [1] (see Fig. 1). The group that received explanations exhibited more persuasive recommendations in terms of adopting the suggested material as part of their learning process.

Figure 1. Explanatory interface for content recommendations in *Mastery Grids* (visual OLM + textual description.

• In a second related attempt, we developed a Mastery Grids version with remedial recommendations (focused on fixing students’ misconceptions) for an introductory database course. Here, students had access to an explanatory interface which visualized struggling concepts. Four experimental treatments were implemented here regarding the type of recommendation explanation format they had access to (see Fig. 2) [2]. Learners exhibited lower conversion rates for recommended problems than for non-recommended ones when any type of explanations (i.e., visual or textual) were provided. On the subjective side, remedial recommendations were perceived to be of better quality when they were justified by partial recommendation explanation (either only textual or visual) than when received none or complete explanations.

Figure 2. Four versions of the *Mastery Grids* system studied in a study that involved different formats of recommendations’ explainability: (1) Visual + textual explanation (2) No explanation (3) Only textual explanation (4) Only visual explanation

References

[1] Jordan Barria-Pineda and Peter Brusilovsky. 2019. Making Educational Recommendations Transparent through a Fine-Grained Open Learner Model. In IUI’19 Workshops.

[2] Jordan Barria-Pineda, Kamil Akhuseyinoglu, Peter Brusilovsky, Kerttu Pollari-Malmi, Teemu Sirkiä, Lauri Malmi. 2020. Personalized Remedial Recommendations for SQL Programming Practice System. In APCSE’20 Workshop co-located with UMAP’20.