The design of adaptive acquisition of users feedback an empirical study (rcis'14)

www.bournemouth.ac.uk
The Design of Adaptive Acquisition of Users
Feedback: an Empirical Study
Malik Al Maliki, Cornelius Ncube and Raian Ali
Faculty of Science and Technology
Bournemouth University, UK
RCIS, Marrakesh, Morocco, 28-30 May, 2014malmaliki@bournemouth.ac.uk

www.bournemouth.ac.uk 2
Introduction
malmaliki@bournemouth.ac.uk REFSQ, Essen, Germany, 7-10 April, 2014RCIS, Marrakesh, Morocco, 28-30 May, 2014

Self-Adaptive Software
“Self-adaptive software evaluates its own behavior and changes behavior
when the evaluation indicates that it is not accomplishing what the software
is intended to do, or when better functionality or performance is possible.”
(Laddaga 1997)
• Why do we need it?
• Adapt to internal and external changes.
• Maximize operation in different conditions.
• Reduce the cost and time of manual adaptation.
• Keep the satisfaction of requirements.
malmaliki@bournemouth.ac.uk RCIS, Marrakesh, Morocco, 28-30 May, 2014

Towards Social Adaptation
• Self-adaptivity is highly dependent on
feedback that is provided by the system itself.
• Feedback is on current state and its environment.
• System feedback is the driver for triggering
and taking autonomous adaptation actions.
• There is no emphasis on the role of users in
the adaptation process.

Social Adaptation
“the system autonomous ability to analyse users’ feedback and choose upon an
alternative behaviour which is collectively shown to be the best for meeting
requirements in a context”
(Ali et al. 2012)
• Users’ judgments on the quality and validity of the different behaviours of a
system drive adaptation
• Socially-driven adaptation
• Giving users’ a voice in tailoring adaptation actions
• Maximize users’ satisfactions.
Social Adaptation Loop (Ali et al. 2012)

Problem

Feedback Quality and Users’
Behavior
• The quality of collected feedback is highly
affected by users’ behaviour.
• Then, how do users behave to feedback
acquisition? What do they like? What type of
feedback? which method? How is their behaviour socially
affected? etc.

Aim of the Study

Aim of the study
• The aim of this study was to :
Empirically understand users’ different
perspectives and behavioural aspects to feedback
acquisition for socially-adaptive software.

Study Design

Study design
• Mixed Method approach (sequential-exploratory).
• First phase (qualitative):
•Interviews.
•7 participants
•Served as a foundation for the second phase.
• Second phase (quantitative):
•Questionnaires.
•100 participants (BU and overseas participants).
•Good response rate (100 out of 180).
•The survey script contained 31 questions discussing and investigating the results
of the first phase.
•Improved the quality and generalizability of the first phase results.

• First phase participants:
• Second phase participants:
Participants’ Characteristics
Participants Age Gender Education Level Home Country
P1
P2
P3
P4
P5
P6
P7
19
29
24
19
23
28
26
Male
Male
Female
Female
Male
Male
Female
Undergraduate
Postgraduate
Postgraduate
Undergraduate
Undergraduate
Undergraduate
Postgraduate
UK
Nigeria
Nigeria
KSA
UK
USA
KSA
Age Range Gender
18-25 26-34 35-54 55-64 Total Male Female Total
level of
education
High school 3 0 0 0 3 3 0 3
Bachelor’s degree 9 3 6 0 18 13 5 18
Master’s degree 6 36 10 3 55 30 25 55
Professional degree 0 0 1 1 2 1 1 2
Doctorate degree 3 11 5 0 19 10 9 19
Others 0 3 0 0 3 2 1 3
Total 21 53 22 4 100 59 41 100

Qualitative Findings
(Phase 1)

Overview of interviews analysis

Qualitative Findings
Theme1: Explicit Feedback Advantages
[1.1] Evident channel for delivering users’ voice and raising
developers' awareness
[1.2] Better for ethical reasons
Theme3: Feedback Acquisition Methods
[3.1] Email is preferable:
o More personalized
o More preferable for qualitative feedback
o More time space and less interruption
o Reasonable number of feedback requests
[3.2] Passive feedback forms are preferable
[3.3] Quantitative feedback request is preferable
[3.4] Combination of qualitative and quantitative (not only
quantitative)
Theme2: Motivation for Accepting/Ignoring Feedback
Requests
[2.1] Visibility of feedback effect on the system
[2.2] Usability and simplicity:
o Language used
[2.3] Disagreement of existing feedback
[2.4] Reasonable number of feedback requests
[2.5] The exciting nature of feedback subject
[2.6] Positive experience
[2.7] Negative experience and needs for improvement
[2.8] Less interruption and distraction
[2.9] Device used
[2.10] Raising public awareness
[2.11] Being forced by the software:
o Low quality feedback
Theme4: Pause of Feedback Requests
[4.1] Same feedback is given
[4.2] Lack of interest
[4.3] Passive feedback is preferable
Theme5: Timing for Feedback Requests
[5.1] Enough time before requesting feedback
[5.2] On recent service or product
[5.3] Reminder is needed
[5.4] Avoid work time or hours
Theme6: Feedback Visibility
[6.1] Ability to see what others said
[6.2] The trend of current feedback

Quantitative
Findings
(phase2)

Overview of the Questionnaire
Introduction to the survey
A snapshot of the questionnaire

Quantitative Findings
• Typical users of popular software applications
• The majority of the participants represent a typical set of software
users.
• Their feedback reflects their experience with popularly used software
applications.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Desktop
Applications
(i.e. MS Office)
E-commerce
(i.e. Ebay)
Search
Engines
Social
Networking
Web
Applications
Mobile Apps
Applications used by users

• Do users like to be asked for feedback by software
applications?
• Not really!, this reflects
• the need for novel mechanisms to increase users’ engagement as
evaluators of software applications.
0%
10%
20%
30%
40%
50%
Yes, very
much
Yes, to a
certain
extent
Not much Not at all
Users’ likeness of feedback request

• Feedback Acquisition Methods and Feedback Types
• Feedback Types
• Explicit feedback
• Qualitative feedback
• Quantitative feedback
• A combination of qualitative and quantitative
• Implicit feedback ( 19%)
0%
10%
20%
30%
40%
50%
60%
Qualitative
feedback
Quantitative
Feedbak
Qualitative and
Quantitative
Explicit Feedback Types

• Feedback Acquisition Methods
• Passive feedback acquisition (51%)
• Offline feedback acquisition (33%)
• Online feedback acquisition (54%)
• Using Hints or tips (31%)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Passive
Method
Offline
Method
Online
Method
Hint/tip
Method
Acquisition Methods: Users' Preferences

• Motivations for Accepting/Ignoring Feedback
Requests
• Users’ Experience
• Simplicity of feedback requests (64%)
• Timing for feedback requests (75%)
• Awareness of the usage and impact of the feedback on the
system (54% )
• Privacy (31%)
• Familiarity (42%)
0%
20%
40%
60%
80%
Users’ Motivations: Users’ Experience
factors

• Interface Design
• Language used (52%)
• Graphical design (31%)
• Simplicity and complexity (74%)
• Fitness of the design and content (81%)
• Information provided (24%)

• Social Factors
• Visibility and similarity of others feedback (47%)
• Volume of already given feedback (52%)
• Social recognition (57%)
• Feedback acquisition as a social activity (63%)
• Volume and Frequency of Feedback Requests
Effect of High Feedback Requests Volume on Users
Percentage
of Users
It is fine with me, I like to give feedback often 3%
It is fine with me as long as I am not forced to give answers 13%
I tend to respond to some of them 14%
I tend to give less focused or less truthful feedback 10%
It leads me to give a negative feedback as the requests make me feel annoyed 7%
I tend to ignore all of them and I tend to consider it as a spam 53%
I tend to stop using the software sending me these requests 21%

Users Clusters

Users Clusters
• Initial clusters of users’ behaviour to feedback acquisition:
Clusters N Likeness
to be
asked
Method Explicit/
Implicit
Reminder Visibility-
Willingness
increases
Social
Activity-
interest
Social
recognition
-willingness
increases-
impact
Feedback
Volume
Feedback
Similarity
Cluster 1 (feedback
antagonists)
38 No Online No No No No No No
Cluster 2
(passive and stingy
people)
27 No Passive No No No No No No
Cluster 3
(privacy fanatic and
generous people)
21 Yes Offline Very
Explicit
Yes Yes_ If able
to see others
feedback first
No Yes Few-
increase
50%
Cluster 4
(privacy tolerant and
socially ostentatious
people)
14 No Hint+
Online
Implicit is
also OK
No Yes_ If able
to see others
feedback first
Yes Yes Large-
increase
Similar-
increase

Conceptual Framework for
Adaptive Feedback
Acquisition

Conceptual Framework for
Adaptive Feedback Acquisition
Conceptual Framework for an adaptive acquisition of users’ feedback.

Conclusion and future work
• An empirical mixed method study to investigate users’ behaviour to
feedback acquisition in software applications was reported.
• A conceptual framework for adaptive feedback acquisition was
proposed.
• Users’ were studied first qualitatively and then quantitatively to enhance
our results and allow for more generalization.
• Users’ behaviour with regard to feedback acquisition highly varies and is
influenced by a number of behavioural factors.
• Systematic approaches and novel mechanisms to conduct an adaptive
feedback acquisition are needed.
• Availability of such systematic approaches can greatly improve the
quality of users’ feedback, users’ satisfaction and the quality of socially-
adaptive software.

Acknowledgement
• We would like to thank:
• Participants who took part in our study for their valuable input.
• Dr. Hamid Bouchachia and Dr. Emilio Balaguer for insights on conducting clusters
analysis.
• The anonymous reviewers for their valuable feedback.
• The research is supported by:
• Bournemouth University through the Software System Research Centre (SSRC) fund.
• Saudi Ministry of Higher Education through the 4th scholarship program.
• FP7 Marie Curie Grant (the SOCIAD project)
• Bournemouth University Fusion Investment Fund (BBB, VolaComp and BUUU projects)

Feedback?

The design of adaptive acquisition of users feedback an empirical study (rcis'14)

More Related Content

Viewers also liked

Similar to The design of adaptive acquisition of users feedback an empirical study (rcis'14)

More from Engineering and Social Informatics (ESOTICS)

Recently uploaded

The design of adaptive acquisition of users feedback an empirical study (rcis'14)