Perspective Chapter: Social Awareness in HRI

1. Introduction

More and more, robots interact with humans in daily life. Possibly, even an ordinary person interacts with a robotic vacuum cleaner, such as Roomba. In many stores, robots are being used as information panels, helping customers to find products or to answer questions about them, such as Pepper. In some cities, as in Los Angeles and San Francisco, robots are delivering goods, such as those from Serve Robotics. Or transporting people, like Waymo and other companies. Robotic assistants like ElliQ are being installed in nursing homes and retirement houses, helping to keep elderly people mentally active. Robotic toys such as AIBO and MISA are toys that keep children entertained.

In factories, robots are not kept behind fences anymore. A revised ISO10218 [1] standard is about to be published in 2025, bringing a necessary review since IMRs (industrial mobile robots) and collaborative robots are more and more frequent on the shop floor and in warehouses, working in an environment close to humans.

Some pivotal works discuss sociable robots in general, such as Fong et al. [2], Breazeal [3, 4], Mahdi et al. [5], and Leite et al. [6]. Other works discuss applications, such as education [7, 8, 9] or health care [10, 11]. These works were fundamental to help organize the list of general requirements discussed here.

The key point of this chapter is to reframe how we think about robots, and to add social awareness as an underlying requirement in all specific aspects. In fact, not only interactions with humans but also humans of all ages and needs (autism, impaired, blind, etc.), which imposes distinct requirements on the social behavior of robots, but also other sentient beings such as dogs and cats, which are present in daily life in our houses and streets. We discuss the general implications of social awareness in morphology, dialog, effective communication, individuality, personality, navigation, privacy, and ethics as fundamental aspects in robotics affected by social awareness. Also, we discuss briefly the impacts of social awareness in functionality, since it varies strongly from one robot application to another: impacts on social awareness for a vacuum cleaner robot are quite distinct from those for an assistant robot.

2. Social robots

The work of Fong et al. [2], though not recent, remains a cornerstone in the field of social robotics. Drawing upon the foundational definition by Dautenhahn and Billard, cited by Fong et al. [2], social robots are described as “embodied agents that are part of a heterogeneous group: a society of robots or humans. They are able to recognize each other and engage in social interactions, they possess histories (perceive and interpret the world in terms of their own experience), and they explicitly communicate with and learn from each other.” Kirby et al. [12] says, “social robots are designed to interact with people in human-centric terms and to operate in human environments alongside people. Many social robots are humanoid or animal-like in form, although this does not have to be the case. A unifying characteristic is that social robots engage people in an interpersonal manner, communicating and coordinating their behavior with humans through verbal, nonverbal, or affective modalities”. These illustrative definitions, among others, agree on the following characteristics:

• Physical embodiment, i.e., a social robot has a physical body;

• Social skills, i.e., a social robot interacts with humans and other sentient beings such as animals, following the social rules relevant to its role;

• Autonomy, i.e., a social robot makes decisions by itself.

Fong et al. [2] also reference Breazeal when categorizing social robots into four primary classes:

• Function-oriented robots: These robots are primarily designed to perform specific tasks or functions with some level of social interaction. Their main objective is practical utility, such as providing companionship or assisting in daily tasks.

  • Example: Robotic vacuums, like Roomba, fall into this category. But until now, they exhibited little social understanding.

• Companion robots: These robots are designed to provide social interaction and emotional support, mimicking social behavior and emotions. They are often employed to support humans emotionally, offering comfort and interaction.

  • Example: Paro, a therapeutic robot for elderly care, and Pepper, a humanoid robot for customer service, are notable examples.

• Interactive robots: Designed for engagement, these robots respond to human gestures, speech, or actions, often used in educational or entertainment contexts.

  • Example: Jibo, a social robot for family interaction, and Moxie, designed for educational purposes, exemplify this category.

• Socially-aware robots: These robots engage proactively with humans to satisfy internal social aims, such as drives or emotions. They require sophisticated models of social cognition.

  • Example: Kismet, developed at MIT, demonstrates internal states guiding its reactions.

The important aspect of a classification of social robots, being this or any other as those in Mavridis [13] and Bunt et al. [14] is that, clearly, social robots differ significantly from conventional industrial robots, teleoperated robots, and AGVs/AMRs and also from existing robots such as Roomba or Pepper, therefore bringing the need to consider social awareness as requirement in HRI design of robots interacting with humans.

3. Morphology

Morphology influences social interaction by shaping expectations: all humans judge based on appearance. The first look: that is the primary social awareness impact. For instance, a dog-like robot will elicit different human reactions compared to an anthropomorphic robot. However, a human-like appearance may not always be desirable due to the “uncanny valley” effect conceptualized by Mori [15] shown in Figure 1.

Figure 1 illustrates how familiarity varies according to various types of artifacts. It displays familiarity with both moving and still entities. An industrial robot shows little familiarity when compared to a humanoid robot. A prosthetic hand is frequently weird, exemplifying the drop of familiarity. Mori’s original picture, as in Figure 1, was later somewhat refuted, particularly because of humanoid robots, which do not always display familiarity. Works such as Berns and Ashok [16] and Yam et al. [17] tried to investigate which anthropomorphism aspects result in familiarity, adding or removing them as “humanizing” or “dehumanizing” robot appearances. The results are not conclusive but clearly illustrate the phenomenon.

Fong et al. identify several morphological types:

• Anthropomorphic: Resembling a human.

• Zoomorphic: Resembling an animal.

• Caricatured: Simplified or stereotypical embodiments.

• Functional: Prioritizing function over form.

Other classifications were proposed, such as Mahdi et al. [5], but the above is useful to classify most products discussed in this chapter: “functional” such as vacuum cleaners; caricatured, as most assistants such as ElliQ; anthropomorphic as Pepper, zoomorphic as Spot, Boston Dynamics dog. Morphology is the first drive of human expectation, a central aspect of social interaction.

4. Dialogue

Social interactions extend beyond simple commands and rely on conversational context. Here “dialog” is used to include all forms of communication, both verbal and nonverbal: voice, screens, touch screens, buttons, etc. Following Mavridis [13], some desired goals include, besides the “simple command”:

• Multiple speech acts as in ISO24617-2 [14].

• Mixed-initiative dialog, as the robot should be able to initiate the dialog.

• Situated language and the symbol grounding problem.

• Affective interaction, as discussed deeper in Section 5.

• Motor correlates and nonverbal communication, also discussed in Section 5.

• Purposeful speech and planning, i.e., how much cheap chat is meaningful in HRI?

While human-robot communication takes many forms, Fong et al. [2] identify three primary types of communication media:

• Low-level (pre-linguistic): Basic, nonverbal exchanges.

• Nonverbal: Gestures or other visual cues.

• Natural language: Conversational interactions enabled by advancements in large language models.

Among these, natural language has become increasingly feasible due to recent technological developments such as ChatGPT.

Dialog is absolutely central to social awareness.

5. Affective communication

Affective communication plays a critical role in human behavior. It includes:

• Verbal communication, as prosody conveying emotions.

• Nonverbal communication, particularly body gestures, is not only related to anthropomorphic robots but also to all other types of morphology: consider, for instance, your vacuum cleaner blinking an LED if it finds a harmful situation. Facial gestures are particularly relevant when conveying emotions. But body language in general is an important emotional display, using arms, hands, shoulders, and general posture. Touch should also be included in this category.

Emotional models in robotics are typically categorized into [18]:

• Discrete approaches: Using specific labels (e.g., happiness, sadness) to classify emotions.

• Dimensional approaches: Employing continuous values to represent emotional dimensions (e.g., arousal, valence).

• Componential theories, such as Scherer et al. [18], attempt to integrate discrete and dimensional approaches.

Affective communication in social robots is a subject with a large bibliography, such as Kirby et al. [12], Paterson [19], and Abdollahi et al. [20] to cite a few.

6. Navigation

Robot navigation in the presence of humans presents unique challenges in the field of navigation, as it necessitates the search for a socially acceptable path. The survey by Kruse et al. [21], although somewhat dated, remains a foundational reference on this topic.

A fundamental challenge in socially acceptable motion techniques is the accurate detection of individuals within the environment. Essential tasks for achieving this include pedestrian detection [22, 23], people tracking [24], and the recognition of human actions and activities [25, 26], among others. These steps are prerequisites for enabling socially compliant navigation.

To provide a general understanding of the problem, Kruse et al. [21] discuss a scenario illustrated in Figure 2, where the robot is tasked with guiding Person A to Person B without disturbing other individuals in the environment.

Traditional trajectory planning techniques, such as obstacle avoidance, are employed to identify a feasible path. However, an added layer of complexity involves minimizing disturbances to Persons C, D, and E along the way. The robot must maintain sufficient space for Person A to follow it (or walk alongside it, depending on the specific task) while adhering to socially acceptable distances (approximately 1 meter, as suggested by Kruse) and maintaining an appropriate speed.

Kruse et al. identify key aspects of socially acceptable navigation:

1. Comfort: The absence of annoyance or stress for humans interacting with robots.

2. Naturalness: The extent to which the robot’s low-level behavior patterns resemble those of humans.

3. Sociability: Adherence to explicit high-level cultural conventions.

In this context, “comfort” is considered a more nuanced concept than mere safety, as it encompasses the need for appropriate distancing. For human-to-human interactions, Hall cited by Kruse et al. [21] proposed the values summarized in Table 1.

However, further research is required to determine whether these values are equally applicable to HRI.

In addition to these social considerations, robots must also achieve conventional navigation goals, including task completion, energy efficiency, time efficiency, and ensuring safety.

7. Individuality

Fong et al. [2] introduce an important discussion regarding the distinction between individual and collective robots. Individual robots operate based on their unique experiences, whereas collective robots share knowledge within a network.

Research about robotic swarms, in general, relates to the resulting functionality, i.e., about the internal swarm behavior leading to some results, such as discussed in Duan et al. [27] and Bredeche and Fontbonne [28]. But collective robots can leverage learning, passing their experience to others, therefore helping to fulfill its function. As a simple example, a vacuum cleaner sharing his knowledge about a house may help in its replacement or help others in the same building to understand the house topology. In sections X and Y, we discuss some consequences in privacy and ethics of knowledge sharing.

An individual robot may learn from his experience, acquiring specific behavior based on these unique experiences. Still, in the simple example of a vacuum cleaner, it can understand where it’s frequently dirtier and adapt its behavior to this. This concept ties into the notion of individuality. A social robot can exhibit individuality and even be recognized as an “electronic person,” a term proposed by the European Parliament’s Committee on Legal Affairs in a draft report on civil law rules for robotics [29]. This term envisions a legal status for sophisticated autonomous robots, granting them “specific rights and obligations, including that of making good any damage they may cause,” and applying electronic personality in cases where robots make autonomous decisions or interact independently with third parties. Robot rights are derived from the legal discussion about animal rights and inspired the so-called “machine question” [30].

8. Personality

Personality significantly influences social interaction [31, 32]. Should a robot exhibit a distinct personality?

Consider the Star Wars franchise robots since they are exemplary models of social awareness, particularly R2-D2 and C-3PO.

R2-D2 assumes the role of a “mechanical technician” (referred to as an “astromech droid” in the Star Wars universe), adept at repairing machinery, interacting with systems, and responding to human commands. Additionally, it acts proactively, often anticipating human needs. Although R2-D2 operates under the instructions of a master, it occasionally circumvents legal constraints in service of its master’s objectives, raising intriguing ethical considerations. R2-D2 does not speak in human language but understands it, communicating instead through “beeps” that humans interpret as a unique linguistic system. Its actions reflect courage, as it undertakes critical and dangerous tasks without hesitation. However, these behaviors are purely mechanical responses, devoid of human emotion. R2-D2 exemplifies loyalty, frequently risking its existence for its master and adhering to Asimov’s Laws of Robotics [33]. Its understated heroism is marked by humility, as it seeks neither recognition nor praise, embodying the ideal functionality of a machine.

C-3PO, self-described as a “protocol robot,” boasts the ability to communicate in over six million languages and comprehends a vast array of cultures, including their customs, traditions, etiquette, and ceremonial practices. Its primary function is to facilitate interaction among humans and other beings by providing translation and ensuring cultural appropriateness. In contrast to R2-D2, C-3PO features an anthropomorphic design, enhancing its relatability to humans. Notably, it exhibits a highly anxious demeanor, often fixating on minor details or potential dangers with repeated exclamations of “We’re doomed!” It frequently highlights risks and expresses discomfort in unpredictable or chaotic situations. Despite its critical contributions, C-3PO tends to underestimate its capabilities, viewing its responsibilities as burdensome. The robot is verbose, often sharing excessive or tangential information, earning it the epithet “mindless philosopher” from Princess Leia. Its approach to problem-solving emphasizes logic and practicality, though it often defaults to pessimistic assumptions. Moreover, its literal thinking limits its ability to grasp sarcasm, adding to its endearing yet occasionally exasperating personality.

The richly developed personalities of R2-D2 and C-3PO often lead viewers to momentarily overlook their mechanical nature. These characters illustrate the potential for robots to engage humans on a social level while raising thought-provoking questions about the ethical and functional dimensions of advanced robotics.

9. Privacy

Privacy has become a critical concern in contemporary society. Personal data is continuously collected through various means such as GPS, cameras, smart devices, on-demand television, and more. Companies, and occasionally governments, utilize this data to uncover habits and tailor marketing strategies. To address these issues, many countries are enacting laws to protect data privacy. Robots, however, introduce additional complexity to this challenge as they access novel forms of data. For instance, home appliances like vacuum cleaners and ovens can collect information that traditional devices cannot capture.

To achieve optimal performance, robots must gather what can be termed “intimacy data,” a category of personal information that extends beyond conventional data privacy considerations. This introduces ethical challenges, particularly with the emergence of robot swarming, where robots or humans share data among themselves. Imagine a scenario where your vacuum cleaner informs your neighbor about breadcrumbs under your bed or shares intimate photos without consent. Such possibilities highlight the privacy concerns tied to robotic data sharing.

These concerns raise important questions: Do privacy risks deter humans from adopting social robots? Is there a “privacy paradox” in which the benefits of social robots are weighed against fears of privacy loss?

Lutz and Tamo-Larrieux [34] conducted a study involving approximately 500 U.S. citizens aged 18–74. Using the model depicted in Figure 2, they explored the relationship between robot use intention and factors such as trust, privacy concerns, perceived benefits, scientific interest, and social influence. Additionally, the study examined how social influence impacts these factors. The experimental factors and their relationship are represented in Figure 3.

Their findings regarding physical privacy revealed:

• Trusting beliefs and privacy concerns had no significant effect on robot use intention, leading to the rejection of hypotheses H1 and H2.

• Perceived benefits positively influenced the intention to use social robots, supporting H3.

• Physical privacy concerns, trusting beliefs, and perceived benefits were significantly correlated and aligned with expectations, supporting H4.

• While physical privacy concerns were unaffected by social influence, social influence positively impacted trusting beliefs and perceived benefits, partially supporting H5.

• Social influence had a significant positive effect on robot use intention, supporting H6.

• Scientific interest did not significantly affect robot use intention, rejecting H7, but it was positively influenced by social influence, supporting H8.

The study also examined the effects of these factors on institutional informational privacy (concerning data usage by companies and governments) and social informational privacy (related to hacking and data breaches). Respondents expressed minimal concern about physical privacy but were significantly more worried about institutional privacy—specifically, data protection by manufacturers. There was moderate concern about malicious uses of social robots by other users, such as stalking or hacking. Overall, while respondents displayed moderate privacy concerns about social robots, other studies show notable apprehension about whether smart speakers like Alexa, Siri, or Google Home adequately safeguard privacy [35].

10. Ethics

Ethics form the foundation of all social interactions, serving as the guiding principles by which individuals and entities navigate complex relationships and dilemmas. This importance extends to the realm of robotics, where ethical considerations are critical in ensuring that technology aligns with human values. Discussions of robot ethics often invoke the “trolley problem” or “crash problem” in the context of self-driving cars [36]. For example, if a robot must choose between hitting a minivan with five passengers or a roadster with one person, it confronts a profound moral uncertainty. Here, the robot is compelled to make a life-altering decision, embodying the ethical dilemmas intrinsic to its programming.

However, moral uncertainty is not confined to vehicles. Consider agricultural robots operating in environments populated by animals. These systems must address ethical considerations involving sentient beings. For instance, should a robotic harvester prioritize the safety of a turtle crossing its path? Such scenarios underscore the need for ethical frameworks that guide robots in balancing operational efficiency with the preservation of sentient life.

Social robots raise unique ethical questions due to their direct interactions with humans. Empathy is a key attribute for these robots. For instance, personal assistant robots may face ethical dilemmas about prioritizing emotional well-being over truthfulness. Is it morally acceptable for a robot to lie to an elderly user by saying, “Your son called,” when he did not? Such decisions involve weighing the benefits of emotional comfort against the intrinsic value of honesty.

Nursing robots, anticipated as essential in aging societies, present another layer of ethical complexity. Delegating decisions about people’s care and well-being to algorithms raises significant concerns. Can an algorithm adequately consider the nuances of human dignity, autonomy, and emotional needs? Furthermore, elderly users often anthropomorphize their robotic companions, developing deep emotional attachments. This phenomenon, observed since the advent of ELIZA and continuing today with advanced conversational agents like ChatGPT, highlights ethical concerns about fostering dependency or escapism through prolonged interactions with robots.

The issue of deception by robots is another pressing ethical question. Should robots deceive humans through behavior or speech? Isaac and Bridewell [37] suggest that robots might need to employ “white lies” to better meet human expectations and maintain trust. However, such deception risks eroding the moral fabric of human-robot relationships and potentially manipulating users in ways that undermine their autonomy.

The ethical landscape becomes even more intricate when considering sex robots and military robots (“warbots”). These applications challenge societal norms and values in profound ways. For example, the on-demand series Westworld depicts an amusement park where robots enable the fulfillment of any human desire without consequence. This fictional scenario prompts critical reflection on the ethical implications of using robots to satisfy desires that might be harmful or morally questionable if directed toward humans. Such narratives force us to consider the boundaries of acceptable robot behavior and the societal impacts of normalizing certain actions through robotic intermediaries.

The integration of social robots into human life necessitates a robust ethical framework. This framework must address the moral uncertainty inherent in robotic decision-making, the balance between empathy and truthfulness, the risks of anthropomorphism, and the implications of deception. As robots become increasingly autonomous and entwined with human society, the ethical questions they raise will only grow more complex, requiring open and transparent discussions and probably new laws, under careful consideration to ensure that technology serves humanity in an equitable and just manner.

11. Social awareness applied to functionality

Social awareness affects functionality in specific ways for each robot application. To grasp this impact, imagine a vacuum cleaner equipped with social awareness. This capability could entail the following:

• It would refrain from cleaning if the baby sleeps in the room or if the user is watching a movie, demonstrating an understanding of human activities and preferences.

• It would recognize obstacles such as food dropped by a baby or pet waste, avoiding actions that could exacerbate messes instead of resolving them. (Personal anecdote: my robotic vacuum cleaner once spread dog urine across the bedroom, misidentifying it as a typical liquid.)

• It would communicate effectively with its user, potentially employing voice interaction for greater accessibility.

• It would provide relevant information about its operation, such as recommending more frequent activation based on observed needs.

• It would respect privacy by not sharing sensitive data with other devices, such as disclosing the types of debris found in the home to a neighbor’s vacuum cleaner (assuming inter-device communication capabilities).

Such a vacuum cleaner would require contextual understanding and appropriate behavioral responses, executing or refraining from tasks based on situational demands. It would need to demonstrate empathy, effectively manage unexpected situations—possibly seeking human input—and communicate in a manner that aligns with human interaction norms (e.g., voice or visual feedback rather than buttons or complex interfaces). Moreover, it would need to adhere to ethical principles and maintain user privacy. While affectivity might not be essential, an understanding of emotional states would be crucial for effective human-robot interaction.

12. Conclusion

In this chapter, we postulate that robots with strong interaction with humans should be thought of as social robots.

A brief introduction to social awareness was presented, discussing its core components: morphology, dialog, effective communication, navigation, individuality, personality, privacy, and ethics. Even in well-developed fields such as navigation, taking social awareness into consideration brings new challenges.

But robots, as considered in this text, are not “general machines”: they are built to perform a specific task, such as a vacuum cleaner. So, it’s necessary to analyze each application to understand its social role and determine the characteristics to apply. A simple and partial example of a hypothetical vacuum cleaner with social awareness was present.

Social awareness represents perhaps the ultimate level of human-robot interaction, going far beyond interfacing with voice, video processing, and other basic functions in robotics. It does not require AGI (artificial general intelligence); it can be built with today’s technology. But it’s expected AGI includes complete social awareness.

We marvel at the new parkour of biped robots, but the real challenge to build a new generation of robots working close to humans is to build social awareness.

Acknowledgments

The authors thank Minerva Institute and CPS-PUC-SP for their support on this research.