Emp[æsthetics] AI – Inês Rodolfo

STUDIES AND PRACTICES OF INTERACTION

The project encourages students to conceptualize an AI persona, considering how it can be integrated naturally and ubiquitously into our daily lives, helping to improve interactions with the environment. Students are tasked with designing the interaction interfaces of empathetic AI systems, utilizing storytelling techniques to shape conversations and engagement with users by studying different aapproaches of aesthetics. Emphasis is placed on human-centered design principles, where AI should be contextually aware and responsive to the immediate needs of its users, whether through voice, gesture, facial recognition, or physiological data.

Role & Place: Invited Assistant Professor at Faculty of Fine Artes, University of Lisboa (3hours/week).

IMPORTANT NOTE: The outcomes presented in this work were developed as part of a curricular unit within the Bachelor’s program in Communication Design. This unit functions as an independent minor course, bringing together students from various disciplines within the Faculty. The project briefing was inspired on a research project called Ethereal which explores empathy in human-AI interaction.This article presents selected case studies from the students with different backgrounds (e.g.,art multimedia, communication design and interior design, among others). All students provided consent by email for presenting their work in this article. All authorship rights to the materials of the project are reserved and belong exclusively to the students. The image displayed on the cover of this article was selected from the project "Cassie AI". The materials are included in this portfolio solely for academic purposes, such as disseminating knowledge, fostering creative dialogue, and showcasing the collaborative achievements between the students and their professor, in compliance with fair use principles and educational rights.

Empathetic AI

Conversational agents

Human-AI Interaction

Multimodal Interaction

CONTEXT

The progress of digital technology has led to the creation of conversational agents such as chatbots and personal assistants, which are increasingly intelligent. Empathetic AI is an emerging field aiming to replicate human empathy through computational models. However, studies indicate that designing the personality of AI or its persona has been reinforceing stereotypes in society instead of promoting new behaviors that support human diversity.

AI agents have become social actors in daily life, raising questions about trust, relationships, and anthropomorphism. Several studies identify that agents are progressively more human-like, not only in terms of appearance but also in how they mimic emotions and personality traits. In contrast, previous studies suggest addressing artificial empathy in human-computer interaction, based on human-centered design, developing conversational agents that are more focused on human needs, adopting a distinct identity as assistive technology, separate from human appearance. According to this perspective, it is suggested that AI can be more focused on people and their context, using storytelling methods for interacting with agents, where empathy is created and expressed through machine-translated perceptions, rather than understanding the AI agent as a human, involving feelings that the machine can only read through emotional contagion.

Additionally, other research lines propose that the next generation of AI agents be more ubiquitous and invisible, listening to the environment and proactively providing intelligent services and recommendations to assist humans.
Studies involving design have approached voice assistant interfaces by visually representing emotions, where facial expressions are mimicked through icons, body movements through text boxes, and undulating elements reflecting the voice, presenting positive results in user engagement during interaction design.

CHALLENGE

Students are invited to conceptualize the interaction design of an AI that demonstrates empathy towards humans. The design of the AI persona is requested, in the context of assistive technology, along with the presentation of its communication model and interface design, applying a storytelling process to shape its interaction/conversation with the human actor.
The proposal may be a product, service, system, or artistic installation combining physical and digital elements to offer a multimodal and orchestrated user experience, answering the following question:

- How can AI agents be naturally integrated ubiquitously into or social context and daily life, demonstrating empathy and enhacing human interaction with their environment?

methodology

Students may address the question by thinking on several possibilities on how AI can be envisioned as an entity integrated into the physical environment, manifested through visual and/or auditory stimuli in conventional graphical interfaces such as mobile devices and wearables or, alternatively, through architectural surfaces that can adopt the interface modality (e.g., a house mirror or other equipment).

The design should depend on the context and the physical environment where the actor/user is located, reacting through their voice, text (chatbots), gesture, facial recognition, or reading of their physiological data. The AI is expected to act as an invisible social agent manifested by the human actor, according to their requested needs or needs detected by the AI, offering anticipatory responses.

Its presence may be perceived through multimodal interaction, providing (bio)feedback strategies through motion design, lighting, sound effects, and voice modulation (if conversation occurs), promoting natural integration with the environment.

The work proposal includes the choice and justification of one (or more) of the aesthetics approaches derived from the "aesthetic turn" in the history of human-computer interaction design, by Udsen and Jørgensen (2005), which were discussed during classes.

Students were asked to develop a UX/UI case study, structured within the design thinking framework, with a focus on the interactive design part, applying the studies from several authors that were discussed on the expositive classes.

01. Observe / Research

1.1 Create a documentation structure for the study and practice: development of a case study that describes the work's development and applied design process (see examples in the UX/UI design case study community). The structure should explain the steps of the design thinking methodology and describe the methods requested in this briefing, from planning to results.

1.2 Definition of the problem and central theme: a guide for developing the prototype or experience and identifying its aesthetic characteristics in interaction design, by identifying authors and related works. A 500-word text. This text should be included in the case study.

1.3 Contextual analysis: collection of ethnographic data and observation records through text articles, photography, videos, screenshots of apps or works by other authors, sound samples, and examples of human interaction with AI agents, AI personality types, and behavior collection. Create a research wall in Figma to present and relate the collected data as a precedents analysis to inform the study.

02. Imagine / Ideate

2.1 Sketching on paper about the concept and visualizing the idea.

2.2 Audience description: define user groups for the product or experience actors and their personas (explain if they are users or actors).

2.3 Definition of the AI Persona as a response to human personas. Define its language and interaction mode (identify the AI agent as an experience actor).

2.4 Conceptual diagram of the AI's multimodal system.

2.5 Journey Map – user or actor experience journey. It should include empathy as a metric and a storyboard. The storyboard can be generated using an AI tool (e.g., DALL·E). If so, include the prompts associated with each scenario. It should include an inventory of touchpoints between the physical and digital world.

03. Make / Prototype / Install

3.1 Paper prototyping: sketching AI interfaces, layout of interfaces or dialogue boxes. Create wireframes that will inform the next method.

3.2. Wireflow of the actor or user's interaction with the AI system. This method should present system navigation through wireframe articulation according to functionalities in an interaction scenario.

3.3 Final mockups: select the main wireframes and specify their interaction in detail (through captions on the interface layout and side descriptions).

3.4 Design System (visual design, behavior classification, icon or conversational prompt library, typography, colors, avatars, and AI expression formats). The Design System is similar to a style guide, though with more refined complexity – it should present technical specifications.

3.5 Interactive prototype/experience: animate the final mockups. Include a button in Figma with the embedded link to the functional prototype. If the work is an installation of an interactive experience, it should be demonstrated in a space and date at the faculty, to be determined with the professor (or choose a context where it can happen – e.g., a gallery). The work can also be a simulation of an AI interaction and be done through a film.

SPECIFICATIONS

Prototype Deliverables

The prototype or experience should apply human-centered design principles and reflect or innovate on a behavior identified during the research phase. The student may use different digital tools and approaches to carry out the interaction design of the proposal.

embodied interaction & physical environment • visual and sound stimuli • Arquitectural interactive surfaces • gestural interfaces • biofeedback• AI personas

Program Contents

Study of (bio)feedback strategies through motion design, lighting, sound effects, and voice modulation (if conversation occurs), promoting natural integration with the environment. The student's work should include the choice and justification of one (or more) of the aesthetics approaches presented in the article of Udsen and Jørgensen (2005):

Cultural approach • Functional approach • Approach focused on the qualities of experience • Ubiquitous or techno-futurist approach.

The student’s work could focus on the functionalist study of human factors alone (e.g., usability, emotions, or affective computing) or on non-informational artifacts (such as games, artworks, or common applications integrated into culture). The outcome depends on the choice of the type of aesthetics they wish to study and practice in the domain of interaction design. The program aligns with the goals and teacthing methodologies of the curricular unit of Studies and Practices of Interaction, directed by the Communication Design Department of Fine Arts, University of Lisbon.

outcomes

This section presents four selected case studies from the class. The selection criteria applied the following metrics: 1) overall quality of the work, 2) representativeness of the learning outcomes of the class in response to the briefing, 3) responsiveness to suggested improvements and argumentation capacity, or 4) originality.

The case studies utilized different prototyping techniques according to the backgrounds of the students (e.g., multimdeia, motion design, graphic design, 3D modeling, interactive high-fidelity functional prototyping). Interaction design methods, such as wireflows, were mandatory.

Students chose different aesthetics, but the majority followed a functional approach (e.g., Ally) which was applied alone or combined with others. The second most chosen approach was the techno-futurist or ubiquitous approach (e.g., dREM and Synthia). Some works that combined more than one approach, showed more exploratory tendency. For example, dREM also included a significant focus on the experience-based approach, while Synthia incorporated elements of the functional approach. The cultural approach can be observed in Cassie’s work, which adopts a satirical tone towards speculative design.

To assess the case studies, signing into the Figma platform (https://www.figma.com/signup) is necessary to visualize the material, except for Cassie.

case studies

ALLY

by Rita Cabral, communication design course

dREM

by Rodrigo Nunes, communication design course

SYNTHIA

by Catarina Bessa and Margarida Prates, multimedia art course

CASSIE

by Duarte Rei, communication design course

Briefing References

1. Liu-Thompkins, Y.; Okazaki, S.; Li, H. Artificial empathy in marketing interactions: Bridging the human-AI gap in atective and social customer experience. Journal of the Academy of Marketing Science 2022,50, 1198-1218.

2. Alazraki, L.; Ghachem, A.; Polydorou, N.; Khosmood, F.; Edalat, A. An empathetic AI coach for selfattachment therapy. In Proceedings of the 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI), 2021; pp. 78-87.

3. Montemayor, C.; Halpern, J.; Fairweather, A. In principle obstacles for empathic AI: why we can’t replace human empathy in healthcare. AI & society 2022, 37, 1353-1359.

4. Montiel-V.zquez, E.C.; Ram.rez Uresti, J.A.; Loyola-Gonz.lez, O. An Explainable Artificial Intelligence Approach for Detecting Empathy in Textual Communication. Applied Sciences 2022, 12, 9407.

5. Nallur, V.; Finlay, G. Empathetic AI for ethics-in-the-small. AI & SOCIETY 2022, 1-2.

6. Zhu, Q.; Luo, J. Toward Artificial Empathy for Human-Centered Design: A Framework. arXiv preprint arXiv:2303.10583 2023.

7. Lawton, G. Google Empathy Lab uses 'design feeling' in search of a more human UI.

8. Pradhan, A.; Lazar, A. Hey Google, do you have a personality? Designing personality and personas for conversational agents. In Proceedings of the Proceedings of the 3rd Conference on Conversational User Interfaces, 2021; pp. 1-4.

9. Seymour, W.; Van Kleek, M. Exploring interactions between trust, anthropomorphism, and relationship development in voice assistants. Proceedings of the ACM on Human-Computer Interaction 2021, 5, 1-16.

10. Alabed, A.; Javornik, A.; Gregory-Smith, D. AI anthropomorphism and its etect on users'selfcongruence and self–AI integration: A theoretical framework and research agenda. Technological Forecasting and Social Change 2022, 182, 121786.

11. Haas, G.; Rietzler, M.; Jones, M.; Rukzio, E. Keep it Short: A Comparison of Voice Assistants’ Response Behavior. In Proceedings of the Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022; pp. 1-12.

12. Spitale, M.; Okamoto, S.; Gupta, M.; Xi, H.; Matarić, M.J. Socially Assistive Robots as Storytellers that Elicit Empathy. J. Hum.-Robot Interact. 2022, 11, Article 46, doi:10.1145/3538409.

13. Mathur, L.; Spitale, M.; Xi, H.; Li, J.; Matarić, M.J. Modeling user empathy elicited by a robot storyteller. In Proceedings of the 2021 9th International Conference on Atective Computing and Intelligent Interaction (ACII), 2021; pp. 1-8.

14. Zhao, H. Emotion in interactive storytelling. In Proceedings of the FDG, 2013; pp. 183-189.

15. Tabassum, M.; Kosiński, T.; Frik, A.; Malkin, N.; Wijesekera, P.; Egelman, S.; Lipford, H.R. Investigating users' preferences and expectations for always-listening voice assistants. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2019, 3, 1-23.

16. Udsen, L. E., & J.rgensen, A. H. (2005). The aesthetic turn: unravelling recent aesthetic approaches to human-computer interaction. Digital creativity, 16(04), 205-216.

Inês Rodolfo, PhD

Human-Centered Design