We are investigating concerns that conversational AI may reinforce negative gender stereotypes in a three-year project by the Engineering and Physical Sciences Research Council (EPSRC).
Conversational AI systems are rapidly developing from purely transactional systems to social companions, which can respond to a wide variety of user requests.
Examples of conversational AI systems include:
- Amazon’s Alexa
- Apple’s Siri
- Google’s Assistant.
Recently, a report by United Nations Educational, Scientific and Cultural Organization (UNESCO) pointed out that the current clearly female-gendered systems and their submissive behaviour reinforce gender stereotypes.
A three-year EPSRC funded project explores this claim tying together three, so-far separate fields of research:
- conversational AI
- social stereotyping
- digital education.
This intersection enables us to assess and anticipate social impacts of digital personas, as well as to address some of the underlying issues of biased technology design.
Designing an artificial woman
The idea of designing an artificial woman is not new: Our previous blog post tells the 2,000-year-old story of Galatea and Pygmalion. The Greek myth describes how the sculptor Pygmalion fell in love with one of his statues, which was granted life by Aphrodite.
We can expect modern instantiations of female gendered artificial personas soon to be found in every household: according to a recent survey, 38% of UK adults owned a smart speaker at the start of 2021 and the trend is increasing. Despite their ubiquitous use, we currently know little about how to measure and anticipate the harm these systems may cause, nor how to mitigate and prevent them.
The team at Heriot-Watt University, led by Professor Verena Rieser and hosted at the National Robotarium, is a two-time finalist and winner of third place at the prestigious Amazon Alexa Prize challenge. The Amazon Alexa Prize is a challenge where university teams get the chance to test their bots with US Amazon customers.
As part of the challenge, Professor Rieser and her team found that 5% of interactions contained abuse, with sexual harassment being the most prominent form of abuse. This may be on the low side for such systems; other research (PDF, 138KB) reports up to 30% of user interactions as abusive.
The aforementioned UNESCO report highlights that the assistants’ submissiveness in the face of gender-based abuse is harmful since it is normalising verbal assault in human-human interactions. We explore this claim as part of this project.
Three main themes
We also investigate three main themes within abuse mitigation:
- abuse detection
- strategies for counter speech
- preventative measures in terms of the bot’s persona design.
For example, for abuse detection, our results show that the distribution of abuse is vastly different compared to abuse and toxicity on social media. There is more sexually tinted aggression towards the virtual persona of conversational AI systems.
This means that current machine learning models perform poorly on this task since they only have been trained on social media data. In response to this shortcoming, the project has released new data and machine learning models, which are tailored to detect abuse directed at conversational systems.
Once we can detect system-directed abuse, system designers face the difficult task of how to respond. Here, the project has run online studies on what different people regard as an ‘appropriate’ response.
Our results show that good responses are context-dependent and user specific. For example, older users dislike answers that contain a joke, whereas younger users dislike avoidance strategies.
We also find in our research (PDF, 358KB) that there is an interaction with the agent’s gender, where responses by agents who have female voices are in general rated lower. One possible explanation is that counterspeech by women is regarded as less socially acceptable.
The outcomes of these studies have been used to inform the development of customer-facing systems, such as those deployed by the BBC and Alana AI.
Collaboration with Meta AI Research
The project has also started a collaboration with Meta AI Research (formerly Facebook AI Research) to investigate how current conversational AI systems developed in research respond in safety critical situations. Current state-of-the-art systems use deep machine learning to predict a likely next response.
Our results show that this response is often not safe, meaning it may inflict physical or psychological harm on the user. The paper calls for more research on how responses by deep learning based systems can be made safer, for example, by increasing control over ‘black box’ machine learning models.
The project also focuses on long-term measures on how systems can be designed to prevent abuse from occurring in the first place.
In a recent study, the project confirms that most users perceive current commercial conversational assistants to be female-gendered and highly anthropomorphised, such as possessing human traits. This is despite proclaimed efforts by companies to counteract this trend.
Our goal is to change user perceptions and provide concrete guidelines on how to design ‘better’ AI agents, which will receive less abuse from the user. This requires a fundamental understanding of how artificial voices are perceived by humans and how they react to them.
Assessing on two dimensions
Professor Benedict Jones’ team at the University of Strathclyde have been investigating these issues. Their analyses of data from seven hundred participants show that people primarily assess conversational assistants’ voices on two dimensions.
The first dimension reflects users’ impressions of how trustworthy the voices sound and the second dimension reflects users’ impressions of how aggressive the voices sound. The aggressiveness dimension is very strongly predicted by the pitch of the voices, with voices with lower pitch being those that users judge to sound particularly aggressive.
Their work also shows that lowering the pitch of the conversational agents’ voices even slightly makes the voices sound much more aggressive.
Together, these findings suggest that considering the role of pitch on aggressiveness-related perceptions might be one route through which conversational agents’ voices can be designed to reduce the frequency with which they are abused.
The team is now testing this idea more directly by studying how altering the pitch of conversational agents’ voices influences users’ interactions with them.
Educating children and young people
In the second half of the project, we will develop inclusive learning materials to educate children and young people on the use of conversational assistants and the technology behind them.
Our goal is to demystify AI by informing the learners about how machine learning and speech technology works in an age-appropriate way. An important aspect of this work is to gather views from children and young people about their experiences with conversational agents at home and what they consider to be appropriate interactions.
We will also explore what they would like the role of conversational agents to be in their lives now, and in the future.
Top image: Credit: Six_Characters, E+ via Getty Images
Professor Verena Rieser is a Professor of Computer Science at Heriot-Watt University, the Head of Responsible Innovation at Alana AI and Director of Ethics for the UK the National Robotarium.
Find out more about Professor Verena Rieser.