![Personal characteristics
Need for cognition
• Measurement of the tendency for an individual to engage in, and enjoy, effortful cognitive activities
• Measured by test of Cacioppo et al. [1984]
Visualisation literacy
• Measurement of the ability to interpret and make meaning from information presented in the form of
images and graphs
• Measured by test of Boy et al. [2014]
Locus of control (LOC)
• Measurement of the extent to which people believe they have power over events in their lives
• Measured by test of Rotter et al. [1966]
Visual working memory
• Measurement of the ability to recall visual patterns [Tintarev and Mastoff, 2016]
• Measured by Corsi block-tapping test
Musical experience
• Measurement of the ability to engage with music in a flexible, effective and nuanced way
[Müllensiefen et al., 2014]
• Measured using the Goldsmiths Musical Sophistication Index (Gold-MSI)
Tech savviness
• Measured by confidence in trying out new technology
7](https://image.slidesharecdn.com/intrs-recsys-220824142351-af785a0e/75/Explaining-recommendations-design-implications-and-lessons-learned-7-2048.jpg)
![Evaluation method
Between-subjects – 240 participants recruited with AMT
Independent variable: settings of user control
2x2x2 factorial design
Dependent variables:
Acceptance (ratings)
Cognitive load (NASA-TLX), Musical Sophistication, Visual Memory
Framework Knijnenburg et al. [2012]
15](https://image.slidesharecdn.com/intrs-recsys-220824142351-af785a0e/75/Explaining-recommendations-design-implications-and-lessons-learned-15-2048.jpg)
![Labor Market Explorer Design Goals
33
[DG1] Exploration/Control
Job seekers should be able to control
recommendations and filter out the information
flow coming from the recommender engine by
prioritizing specific items of interest.
[DG2] Explanations
Recommendations and matching scores should be
explained, and details should be provided on-
demand.
[DG3] Actionable Insights
The interface should provide actionable insights to
help job-seekers find new or more job
recommendations from different perspectives.](https://image.slidesharecdn.com/intrs-recsys-220824142351-af785a0e/75/Explaining-recommendations-design-implications-and-lessons-learned-33-2048.jpg)
Uploaded byKatrien Verbert
Explaining recommendations: design implications and lessons learned
The document discusses designing explainable recommender systems, outlining the Augment/HCI research group's work on explaining recommendations to increase user trust, enable interaction with recommendation processes, and providing various application domains for explainable recommender systems including learning analytics, jobs, nutrition, and healthcare. It also discusses challenges and lessons learned in explaining recommendations to users.
More Related Content
Similar to Explaining recommendations: design implications and lessons learned
Explaining recommendations: design implications and lessons learned
- 1. Explaining recommendations: design implicationsand lessons learned Women @ RecSys – 24 Aug 2022 Katrien Verbert Augment/HCI - KU Leuven https://augment.cs.kuleuven.be @katrien_v
- 2. Human-Computer Interaction group ExplainableAI - recommender systems – visualization – intelligent user interfaces Learning analytics & human resources Media consumption Precision agriculture Healthcare Augment Katrien Verbert ARIA Adalberto Simeone Computer Graphics Phil Dutré LIIR Sien Moens E-media Vero Vanden Abeele Luc Geurts Kathrin Gerling 2
- 3. Augment/HCI team Robin DeCroon Postdoc researcher Katrien Verbert Professor Francisco Gutiérrez Postdoc researcher Tom Broos PhD researcher Nyi Nyi Htun Postdoc researcher Houda Lamqaddam Postdoc researcher Oscar Alvarado Postdoc researcher https://augment.cs.kuleuven.be/ Diego Rojo Carcia PhD researcher Maxwell Szymanski PhD researcher Arno Vanneste PhD researcher Jeroen Ooge PhD researcher Aditya Bhattacharya PhD researcher Ivania Donoso Guzmán PhD researcher 3
- 4. Interactive recommender systems Coreobjectives: • Explaining recommendations to increase user trust and acceptance • Enable users to interact with the recommendation process
- 5. TalkExplorer 5 Verbert, K., Parra,D., Brusilovsky, P. and Duval, E. (2013). Visualizing recommendations to support exploration, transparency and controllability. In Proc. of the 17th Int. Conf. on Intelligent User Interfaces (IUI’13), 12 p. ACM.
- 6. Explanations 6 Millecamp, M., Htun,N. N., Conati, C., & Verbert, K. (2019, March). To explain or not to explain: the effects of personal characteristics when explaining music recommendations. In Proceedings of the 24th International Conference on Intelligent User Interfaces (pp. 397-407).
- 7. Personal characteristics Need forcognition • Measurement of the tendency for an individual to engage in, and enjoy, effortful cognitive activities • Measured by test of Cacioppo et al. [1984] Visualisation literacy • Measurement of the ability to interpret and make meaning from information presented in the form of images and graphs • Measured by test of Boy et al. [2014] Locus of control (LOC) • Measurement of the extent to which people believe they have power over events in their lives • Measured by test of Rotter et al. [1966] Visual working memory • Measurement of the ability to recall visual patterns [Tintarev and Mastoff, 2016] • Measured by Corsi block-tapping test Musical experience • Measurement of the ability to engage with music in a flexible, effective and nuanced way [Müllensiefen et al., 2014] • Measured using the Goldsmiths Musical Sophistication Index (Gold-MSI) Tech savviness • Measured by confidence in trying out new technology 7
- 8. User study Within-subjectsdesign: 105 participants recruited with Amazon Mechanical Turk Baseline version (without explanations) compared with explanation interface Pre-study questionnaire for all personal characteristics Task: Based on a chosen scenario for creating a play-list, explore songs and rate all songs in the final playlist Post-study questionnaire: Recommender effectiveness Trust Good understanding Use intentions Novelty Satisfaction Confidence 8
- 9. Results 9 The interaction effectbetween NFC (divided into 4 quartiles Q1-Q4) and interfaces in terms of confidence
- 10. Design implications Explanationsshould be personalised for different groups of end- users. Users should be able to choose whether or not they want to see explanations. Explanation components should be flexible enough to present varying levels of details depending on a user’s preference. 10
- 11. User control Users tendto be more satisfied when they have control over how recommender systems produce suggestions for them (Konstan and Riedl, 2012) Control recommendations Douban FM Control user profile Spotify Control algorithm parameters TasteWeights 11 Konstan, J.A. and Riedl, J., 2012. Recommender systems: from algorithms to user experience. User modeling and user-adapted interaction, 22(1-2), pp.101-123.
- 12. Controllability Cognitive load Additionalcontrols may increase cognitive load 12 Ivana Andjelkovic, Denis Parra, andJohn O’Donovan. 2016. Moodplay: Interactive mood-based music discovery and recommendation. In Proc. of UMAP’16. ACM, 275–279.
- 13. Different levels ofuser control 13 Level Recommender components Controls low Recommendations (REC) Rating, removing, and sorting medium User profile (PRO) Select which user profile data will be considered by the recommender high Algorithm parameters (PAR) Modify the weight of different parameters Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 13-21). ACM.
- 14. User profile (PRO)Algorithm parameters (PAR) Recommendations (REC) 8 control settings No control REC PAR PRO REC*PRO REC*PAR PRO*PAR REC*PRO*PAR 14
- 15. Evaluation method Between-subjects– 240 participants recruited with AMT Independent variable: settings of user control 2x2x2 factorial design Dependent variables: Acceptance (ratings) Cognitive load (NASA-TLX), Musical Sophistication, Visual Memory Framework Knijnenburg et al. [2012] 15
- 16. Results Main effects:from REC to PRO to PAR → higher cognitive load Two-way interaction: does not necessarily result in higher cognitive load. Adding an additional control component to PAR increases the acceptance. PRO*PAR has less cognitive load than PRO and PAR High musical sophistication leads to higher quality, and thereby results in higher acceptance 16
- 17. More high riskapplication domains 17 Application domains
- 18.
- 19. Learning analytics Src: SteveSchoettler 19
- 20.
- 21. Explaining exercise recommendations Howto automatically adapt the exercise recommending on Wiski to the level of students? How do (placebo) explanations affect initial trust in Wiski for recommending exercises? Goals and research questions Automatic adaptation Explanations & trust Young target audience Middle and high school students Ooge, J., Kato, S., & Verbert, K. (2022, March). Explaining Recommendations in E-Learning: Effects on Adolescents' Trust. In 27th International Conference on Intelligent User Interfaces (pp. 93-105) 21
- 22.
- 23.
- 24. Trust Multidimensional trust • Trustingbeliefs (Competence, benevolence, integrity) • Intention to return • Perceived Transparency One-dimensional trust Direct trust measurement Ask about trust factors with 7-point Likert-type questions Indirect trust measurement Log whether students accept recommendations or not 24
- 25. Results: Real explanations… …did increase multidimensional initial trust … did not increase one-dimensional initial trust … led to accepting more recommended exercises compared to both placebo and no explanations 25
- 26. Results: Placebo explanations… …did not increase initial trust compared to no explanations … may undermine perceived integrity … are a useful baseline: • how critical are students towards explanations? • how much transparency do students need? 26
- 27. Results: No explanations Canbe acceptable in low-stakes situations (e.g., drilling exercises): indications of difficulty level might suffice Personal level indication: Easy, Medium and Hard tags 27
- 28.
- 29. Explaining job recommendations 29 Gutiérrez,F., Charleer, Sven, De Croon, Robin, Nyi Nyi Htun, Goetschalckx, Gerd, & Verbert, Katrien. (2019) “Explaining and exploring job recommendations: a user-driven approach for interacting with knowledge-based job recommender systems”. In Proceedings of the 13th ACM Conference on Recommender Systems. ACM, 2019
- 30.
- 31. 3 1 Ranking of parametersas voted by participants
- 32.
- 33. Labor Market ExplorerDesign Goals 33 [DG1] Exploration/Control Job seekers should be able to control recommendations and filter out the information flow coming from the recommender engine by prioritizing specific items of interest. [DG2] Explanations Recommendations and matching scores should be explained, and details should be provided on- demand. [DG3] Actionable Insights The interface should provide actionable insights to help job-seekers find new or more job recommendations from different perspectives.
- 34.
- 35. Final Evaluation 35 66 jobseekers (age 33.9 ± 9.5, 18F) 8 Training Programs, 4 Groups, 1 Hour. 1 2 3 4 5 6 7 8 ResQue Questionnaire + two open questions. Users explored the tool freely. All interactions were logged.
- 36.
- 37.
- 38. Results Explanations contributeto support user empowerment. A diverse set of actionable insights were also mentioned by participants. Participants in the technical group engaged more with all the different features of the dashboard. Non-native speakers, sales and construction groups engaged more with the map. The table overview was perceived as very useful by all user groups, but the interaction may need further simplification for some users. 38
- 39.
- 40. https://www.jmir.org/2021/6/e18035 Nutrition Nutrition advice (7) Diets(7) Recipes (7) Menus (2) Fruit (1) Restaurants (1) Doctors (4) Hospital (5) Thread / fora (3) Self-Diagnosis (3) Healthcare information (5) Similar users (2) Advise for children (2) General health information Routes (2) Physical activity (10) Leisure activity (2) Wellbeing motivation (2) Behaviour (7) Wearable devices (1) Tailored messages (2) Routes (2) Physical activity (10) Leisure activity (2) Behaviour (7) Lifestyle Specific health conditions Health Recommende r Systems 40
- 41. Recommender systems forfood 41
- 42.
- 43. Design and Evaluation 43 GutiérrezF., Cardoso B., Verbert K. (2017). PHARA: a personal health augmented reality assistant to support decision-making at grocery stores. In: Proceedings of the International Workshop on Health Recommender Systems co-located with ACM RecSys 2017 (Paper No. 4) (10-13).
- 44. Design 44 Gutiérrez Hernández F.,Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented reality applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii International Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan 2019.
- 45.
- 46. User study WithinSubjects n = 28 (1F, 27M) Ages from 22 to 38 (M = 25.81, SD = 4.57) Post-Questionnaires TAM (Technology Acceptance) NASA-TLX (Task Load Index) 46 Gutiérrez Hernández F., Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented reality applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii International Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan 2019.
- 47. Results PHARA allowsusers to make informed decisions, and resulted in selecting healthier food products. Stack layout performs better with HMD devices with a limited field of view, like the HoloLens, at the cost of some affordances. The grid and pie layouts performed better in handheld devices, allowing to explore with more confidence, enjoyability and less effort. 47 Gutiérrez Hernández F., Htun NN., Charleer S., De Croon R., Verbert K. (2018). Designing augmented reality applications for personal health decision-making. In: Proceedings of the 2019 52nd Hawaii International Conference on System Sciences Presented at the HICSS, Hawaii, 07 Jan 2019-11 Jan 2019.
- 48.
- 49. RECOMMENDE R ALGORITHMS MACHINE LEARNING INTERACTIVE DASHBOARDS SMART ALERTS RICHCARE PLANS OPEN IoT ARCHITECTURE 49
- 50.
- 51. Evaluation methodology 12nurses used the app for three months Data collection Interaction logs Resque questions Semi-structured interviews 51
- 52. Results Iterative designprocess identified several important features, such as the pending list, overview and the feedback shortcut to encourage feedback. Explanations seem to contribute well to better support the healthcare professionals. Results indicate a better understanding of the call notifications by being able to see the reasons of the calls. More trust in the recommendations and increased perceptions of transparency and control Interaction patterns indicate that users engaged well with the interface, although some users did not use all features to interact with the system. Need for further simplification and personalization. 52 Gutiérrez, F., Htun, N. N., Vanden Abeele, V., De Croon, R., & Verbert, K. (2022, March). Explaining Call Recommendations in Nursing Homes: a User-Centered Design Approach for Interacting with Knowledge-Based Health Decision Support Systems. In 27th International Conference on Intelligent User Interfaces (pp. 162-172).
- 53.
- 54. Explaining health recommendations 6different explanation designs Explain WHY users are given a certain recommendation for their (chronic) pain based on their inputs 54 Maxwell Szymanski,Vero Vanden Abeele and Katrien Verbert Explaininghealthrecommendationstolayusers: Thedosand don’ts – Apex-IUI2022
- 55. 6 different designs 55 textinline reply tags
- 56. 5 Word cloud Featureimportance Feature importance+ % 6 different designs
- 57.
- 58. Results Feature Importance Bars(+%) favoured by most users 58
- 59. Keywords / themesas to why users (dis)like the explanation type 59
- 60. Research challenges Insightvs. information overload Visual representations often difficult for non-expert users Limitations of online user studies 60
- 61. Take away Involvementof end-users has been key to come up with interfaces tailored to the needs of non-expert users Domain expertise of users and need for cognition important personal characteristics Need for personalisation and simplification Conversational approaches seem promising 61
- 62. Career challenges 62 PhD. researcherOct. 2003 – Feb. 2008 Post-doc Feb. 2008 – Dec. 2012 Assistant Professor Jan. 2013 – Dec 2013 62 Assistant Professor Jan. 2014 – Dec 2014 Assistant Professor Oct. 2014 – Sept 2018 Associate Professor Oct. 2018 – Sept 2022 Professor Oct. 2022 - … • Worked at TU/e and VUB • Research visits at CMU, Pitt, RWTH, BYU, FIT
- 63. Career challenges 63 PhD. researcherOct. 2003 – Feb. 2008 Post-doc Feb. 2008 – Dec. 2012 Assistant Professor Jan. 2013 – Dec 2013 63 Assistant Professor Jan. 2014 – Dec 2014 Assistant Professor Oct. 2014 – Sept 2018 Associate Professor Oct. 2018 – Sept 2022 Professor Oct. 2022 - … Building up strong CV in research, teaching, leadership, service
- 64. Career challenges 64 PhD. researcherOct. 2003 – Feb. 2008 Post-doc Feb. 2008 – Dec. 2012 Assistant Professor Jan. 2013 – Dec 2013 64 Assistant Professor Jan. 2014 – Dec 2014 Assistant Professor Oct. 2014 – Sept 2018 Associate Professor Oct. 2018 – Sept 2022 Professor Oct. 2022 - … • Diverse funding: national - international • Basic + applied research • Collaboration with industry
- 65. Peter Brusliovsky NavaTintarevCristinaConati Denis Parra Collaborations Bart Knijnenburg Jurgen Ziegler 65
- 66.
Editor's Notes
- #5 Bring tools that have been researched for decades within reach of lay-humanists – traditional scholar not well versed with computational methods
- #9 The procedure contains the following steps: \begin{enumerate} \item \textit{Tutorial of study} - Participants were invited to read the description of the user study and to choose a scenario for generating a play-list. Then, they were asked to watch a task tutorial. Only the features of the particular setting were shown in this video. The ``Start'' button of the study was only activated after finishing the tutorial. Users logged in with their Spotify accounts to our experimental system, so that our recommenders could leverage the Spotify API and user listening history to generate ``real'' recommendations. \item \textit{Pre-study questionnaire} - This questionnaire collects user demographics and measures user's personal characteristics such as musical sophistication and visual memory capacity. %and their trust in recommender systems. The visual memory capacity is measured by ``Corsi block-tapping test''. In the test, a number of tiles are highlighted one at a time, and participants are asked to select the tiles in the correct order afterward. The number of highlighted tiles increases until the user makes too many errors. In Experiments 1 and 3, we used a test with a more sophisticated implementation of the Corsi test~\footnote{\url{https://www.humanbenchmark.com/tests/memory}, accessed June 2018}, which allows us to better distinguish participants by the level of visual memory capacity. In Experiment 2, to control the workload of participants in the within-subjects design, we chose a simple version of the Corsi test~\footnote{\url{http://www.psytoolkit.org/experiment-library/corsi.html}, accessed June 2018} for measuring visual short-term memory. \end{itemize} \item \textit{Manipulating Recommender and rating songs} - To ensure that participants spent enough time to explore recommendations, the questionnaire link was only activated after 10 minutes. After tweaking the recommender, participants were asked to rate the top-20 recommended songs that resulted from their interactions. \item \textit{Post-study questionnaire} - Participants were asked to evaluate the perceived quality, perceived accuracy, perceived diversity, satisfaction, effectiveness, and choice difficulty of the recommender system. After answering all the questions, participants were given opportunities to provide free-text comments of their opinions and suggestions about our recommender. \end{enumerate}
- #10 Figure 2 shows that participants with low NFC are reporting a higher confident in their playlist with the explanations in- terface than in the baseline. Participants with a high NFC reported the opposite. Hence, the participants with low NFC have more confidence in the explanation interface than in the baseline, in contrast to user with low NFC. An explanation might be that low NFC participants benefited from the expla- nations because they did not spontaneously engage in much extra reasoning to justify the recommendations they received, and when they received the rational from the explanation this increased their confidence in their songs selection. Figure 2 also indicates that as the NFC increased, the con- fidence of participants in the playlist created in the baseline also increased. This result indicated that participants with a high NFC were more willing to understand their own musical preference in relation to the attributes of the recommended songs. This may have resulted in a higher confidence in their playlist. We did not see the same increase in trust as NFC increases in the explanation interface. As Figure 2 shows, the NFC scores in the third quartile were almost the same for both interfaces. At the highest NFC level, participants had a higher confidence in the baseline than in the explanation interface. The reduced confidence within the explanation interface could be an indication that users with a high NFC have less need for explanations.
- #16 We employed a between-subjects study to investigate the effects of interactions among different user control on acceptance, perceived diversity, and cognitive load. We consider each of three user control components as a variable. By following the 2x2x2 factorial design we created eight experimental settings (Table~\ref{tab:table1}), which allows us to analyze three main effects, three two-way interactions, and one three-way interaction. We also investigate which specific \textit{personal characteristics} (musical sophistication, visual memory capacity) influence acceptance and perceived diversity. Each experimental setting is evaluated by a group of participants (N=30). Of note, to minimize the effects of UI layout, all settings have the same UI and disable the unsupported UI controls, e.g., graying out sliders. As shown in section ~\ref{evaluation questions}, we employed Knijnenburg et al.'s framework~\citep{knijnenburg2012explaining} to measure the six subjective factors, perceived quality, perceived diversity, perceived accuracy, effectiveness, satisfaction, and choice difficulty~\citep{knijnenburg2012explaining}. In addition, we measured cognitive load by using a classic cognitive load testing questionnaire, the NASA-TLX~\footnote{https://humansystems.arc.nasa.gov/groups/tlx}. It assesses the cognitive load on six aspects: mental demand, physical demand, temporal demand, performance, effort, and frustration. The procedure follows the design outlined in the general methodology (c.f., Section \ref{sec:general-procedure}). The \textit{experimental task} is to compose a play-list for the chosen scenario by interacting with the recommender system. Participants were presented with play-list style recommendations (Figure~\ref{fig:vis1}c). Conditions were altered on a between-subjects basis. Each participant was presented with only one setting of user control. For each setting, initial recommendations are generated based on the selected top three artists, top two tracks, and top one genre. According to the controls provided in a particular setting, participants were able to manipulate the recommendation process.
- #17 Main effects: REC has lowest cgload and highest acceptance Two-way: All the settings that combine two control components do not lead to significantly higher cognitive load than using only one control component. combing multiple control components potentially increases acceptance without increasing cognitive load significantly. visual memory is not a significant factor that affects the cognitive load of controlling recommender systems. In other words, controlling the more advanced recommendation components in this study does not seem to demand a high visual memory. In addition, we did not find an effect of visual memory on acceptance (or perceived accuracy and quality). One possible explanation is that users with higher musical so- phistication are able to leverage different control components to explore songs, and this influences their perception of recommenda- tion quality, thereby accepting more songs. Our results show that the settings of user control significantly influence cognitive load and recommendation acceptance. We discuss the results by the main effects and interaction effects in a 2x2x2 factorial design. Moreover, we discuss how visual memory and musical sophistication affect cognitive load, perceived diversity, and recommendation acceptance. \subsubsection{Main effects} We discuss the main effects of three control components. Increased control level; from control of recommendations (REC) to user profile (PRO) to algorithm parameters (PAR); leads to higher cognitive load (see Figure \ref{fig:margin}c). The increased cognitive load, in turn, leads to lower interaction times. Compared to the control of algorithm parameters (PAR) or user profile (PRO), the control of recommendations (REC) introduces the least cognitive load and supports users in finding songs they like. We observe that most existing music recommender systems only allow users to manipulate the recommendation results, e.g., users provide feedback to a recommender through acceptance. However, the control of recommendations is a limited operation that does not allow users to understand or control the deep mechanism of recommendations. \subsubsection{Two-way interaction effects} Adding multiple controls allows us to improve on existing systems w.r.t. control, and do not necessarily result in higher cognitive load. Adding an additional control component to algorithm parameters increases the acceptance of recommended songs significantly. Interestingly, all the settings that combine two control components do \textit{not} lead to significantly higher cognitive load than using only one control component. We even find that users' cognitive load is significantly \textit{lower} for (PRO*PAR) than (PRO, PAR), which shows a benefit of combining user profile and algorithm parameters in user control. Moreover, combing multiple control components potentially increases acceptance without increasing cognitive load significantly. Arguably, it is beneficial to combine multiple control components in terms of acceptance and cognitive load. \subsubsection{Three-way interaction effects} The interaction of PRO*PAR*REC tends to increase acceptance (see Figure \ref{fig:margin}a), and it does not lead to higher cognitive load (see Figure \ref{fig:margin}c). Moreover, it also tends to increase interaction times and accuracy. Therefore, we may consider having three control components in a system. Consequently, we answer the research question. \textbf{RQ1}: \textit{The UI setting (user control, visualization, or both) has a significant effect on recommendation acceptance?} It seems that combining PAR with a second control component or combing three control components increases acceptance significantly. %KV: this paragraph refers to different RQs: either rephrase or omit? -SOLVED \subsubsection{Effects of personal characteristics} Having observed the trends across all users, we survey the difference in cognitive load and item acceptance due to personal characteristics. We study two kinds of characteristics: visual working memory and musical sophistication. \paragraph{Visual working memory} The SEM model suggests that visual memory is not a significant factor that affects the cognitive load of controlling recommender systems. The cognitive load for the type of controls used may not be strongly affected by individual differences in visual working memory. In other words, controlling the more advanced recommendation components in this study does not seem to demand a high visual memory. In addition, we did not find an effect of visual memory on acceptance (or perceived accuracy and quality). Finally, the question items for diversity did not converge in our model, so we are not able to make a conclusion about the influence of visual working memory on diversity. \paragraph{Musical sophistication} Our results imply that high musical sophistication allows users to perceive higher recommendation quality, and may thereby be more likely to accept recommended items. However, higher musical sophistication also increases choice difficulty, which may negatively influence acceptance. One possible explanation is that users with higher musical sophistication are able to leverage different control components to explore songs, and this influences their perception of recommendation quality, thereby accepting more songs. Finally, the question items for diversity did not converge in our model, so we are not able to make a conclusion about the influence of musical sophistication on diversity.
- #30 A second dashboard was designed for both job seekers and job mediators. The dashboard explains recommendations by showing compentence and compentence gaps.
- #31 We used a user-centered design methodology consisting of: focus groups, co-design sessions, usability evaluations and a final evaluation with 66 job seekers.
- #32 The results are presented here: job title and distance to a job were ranked frequently on the first positiion by both job seekers and job mediators. Many other parameters were ranked frequently on the next positions, including type of contract, competences, studies and work experience.
- #33 We then elaborated designs that put focus on the important parameters: location and distance are for instace visualised in all tthese design, as well as competences that are asked in jobs. We captured additional input of job seekers and mediators in co-design sessions and evaluated first prototypes.
- #34 Results of these sessions were captured and articulated in the following design goals: Exploration / control: the need to control recommendations and highlight items of interest was expressed Job seekers also wanted to see explanations of recommendations that go beyong the typical matching score There is also a need to support actionable insights: for instance by visualising which competences are currently in high demand in the job market
- #35 We used a user-centered design methodology consisting of: focus groups, co-design sessions, usability evaluations and a final evaluation with 66 job seekers.
- #36 We conducted a final evaluation with 66 job seekers, in different training programs at VDAB – including highly technical users, sales, construction and non-native speakers. We asked the users to freely explore the dashboard and then used the ResQue questionnaire to capture user experience feedback together with two open questions. We also logged all interactions.
- #37 Technical users were a bit more negative with respect to overall satisfaction (Q5). Perceived accuracy (Q1) was a bit lower for non-native speakers: As language is not trivial for this group, also working with an interface in a different language as well as finding suitable jobs is evidently much more difficult. Perceived usefulness (Q2) was a bit lower for the construction group. The wide range of very specific competencies in their domain was one of the reasons they highlighted for this lower score.
- #38 Participants with technical background engaged the most with the job vacancies table, where they performed most of the interactions in a significant different way compared to the other groups. Participants in the construction group and non-native speakers engaged less with the job vacancies table. We can observe that these groups engaged more with the map component.
- #39 Responses to the questions indicate that job seekers value the use of interactive visualization techniques to find relevant job vacancies.. In general, the approach is perceived as effective to explore job recommendations. The interaction patterns also indicate that participants engaged well with the interface. Also explanations seem to contribute well to better support of user empowerment. Many of the responses hint to better understanding of the job recommendations by being able to see which competencies are required. A diverse set of actionable insights were also mentioned by participants. Participants indicate that the overview of competencies enables them to explore the job market and understand whether they have the needed competencies. They also gain insight into regions where most jobs are offered, supporting potential location-based job mobility. Other actionable insights include the observation that their user profile is not up to date. Such insights are key as well, as they trigger job seekers to update their profile and, in turn, receive better recommendations.
- #43 We found that the stack visualisation performs better with HMD devices with a limited field of view, like the HoloLens, at the cost of some usability affordances (RQ4). For handheld devices, the grid and the pie tended to score higher in terms of confidence in decision making, compared to the list and stack layouts (RQ1, RQ4).
- #44 The prediction model shows the impact of the food product on the weight of the participant. Opacity is used to represent the uncertainty of this prediction. (POINT to third card)
- #45 The prediction model shows the impact of the food product on the weight of the participant. Opacity is used to represent the uncertainty of this prediction. (POINT to third card)
- #46 We compared four different layouts to represent this information: a stack layout, a list layout, a grid layout and a pie layout. We compared their use in two implementations: one using the Microsoft HoloLens, and a second one using an Smartphone
- #47 We evaluated these layouts in a user study with 28 participants in a lab setting. and measured both subjective and objective data collected from the use of our application,
- #48 We found that the stack visualisation performs better with HMD devices with a limited field of view, like the HoloLens, at the cost of some usability affordances (RQ4). For handheld devices, the grid and the pie tended to score higher in terms of confidence in decision making, compared to the list and stack layouts (RQ1, RQ4).
- #60 Most users prefer more information (holistic overview of inputs)
- #61 challenges you experienced and learned both from your research and your life choices/career path.
- #63 I think I faced the typical challenges in the development of an academic career: first one was the need for international mobility – which was not trivial with a family. But I than got some great opportunities: I moved to Paul De Bra’s group in the Netherlands and I also had some amazing opportunities to conduct research visists at CMU, PITT, and was able to work with very influencial researcers in the domain such as Peter Brusilovsky from whom I’ve learned a lot. These visits where typically around three months but resulted in long-lasting and very productive collaborations and I would highly recommend them to any young researcher: they were key to the success of my career.
- #64 Wasn’t an easy path. What I learned is that it is important to be excellent in all the different dimensions: research teaching leadership but also service – both internationally both also internally: so I’ve also taken up larger roles within our organisation such as being a member of the research council of our university: which was a lot of work, but I also learned a lot from it