Introducing Girl Effect’s GenAI Whitepaper: Girl Effect’s Journey to Smarter, Safer Health Chatbots

You can download the whitepaper here.

Girl Effect is excited to unveil our latest whitepaper, documenting the progress we have made integrating Generative AI in our products over the past year, as well as looking ahead to the next step changes in impact that this technology can offer.

How the Journey Began

Twelve years ago, our Chief Creative and Technology Officer, Karina Rios Michel, uncovered a powerful insight while working on Girl Effect’s Ni Nyampinga program in Rwanda: girls had no one they could turn to for answers to their most personal questions about growing up. 

In response, we created Baza Shangazi (Ask Aunty), an advice column published in a nationally distributed magazine, and later expanded via SMS and radio shows. It was designed to answer one powerful question:

“What would it look like if girls had a safe space to ask the things they were too afraid to say out loud?”

Central to Girl Effect’s approach has always been an understanding of the necessity to reach our target audience through the channels they are familiar with. Over the years, we have seen the widespread availability of mobile phones in the countries where we operate, with instant messaging services such as WhatsApp becoming the ‘killer app’, the place where people spend most of their time communicating with friends and family. We realised that these channels represented the perfect medium of communication for our programmatic objectives. Instant Messaging services are private, interactive and intuitive in their use. A persona could be created for the chatbot, which could be seen by the user as one of their contacts, increasing the feeling of a bespoke and personal experience.

Girl Effect launched Big Sis in 2018. In its first incarnation, Big Sis provided key messaging around SRH topics and offered an FAQ service where users could ask questions, with answers to the most popular submissions published on a weekly basis. This feature was incredibly popular, proving the demand for a digital version of our agony aunt service. However, at the time, the technology was not advanced enough to give our users what they really wanted- an immediate answer to their question.

Harnessing the Power of Generative AI

With any digital solution there is a trade-off between the personal connection created through in-person communication and the ability to reach users at scale. Big Sis had created the feeling of providing a 1:1 experience but did not have the ability to respond directly to the user. With the advancement in artificial intelligence and machine learning, Girl Effect has now addressed this issue. We were first able to create our own Natural Language Processing (NLP)  model using our trove of user questions- this allowed us to classify user submissions and provide content suggestions. This was a big step to a more personal experience (we saw a 200% increase in meaningful content engagement). Yet we still couldn’t answer the user’s question directly.

When OpenAI released ChatGPT, we saw a clear pathway to achieving our vision through the generation of conversation with our users. By providing instructions through prompts about how to talk to our users, by referring to our existing content sets, and through ensuring the safety of the output provided by the Large Language Model (LLM) as a result of guardrails and an evaluation framework, we would be able to realize a step-change in the quality of engagement and depth of impact we could offer.

Since the release of our last Vision Paper 18 months ago, we have taken our research and the development of our approach into a functional product. Rather than starting from scratch, we were able to find the optimal locations in our chatbots to insert generative AI, building on an already successful product. 

Powerful Evidence

Currently, 50,000 users have submitted almost 150,000 questions to our GenAI-powered FAQ service. As part of our phased rollout, we ensured that we gathered evidence as to the quality of what we had designed, firstly through in person user testing and then through experimentation in the live chatbot. 

We conducted rigorous AB testing of our new feature compared to  its predecessor and saw clear results as to the effect of Generative AI:

• Users were 11% more likely to recommend the chatbot to a friend
• Users were 17.1% more likely to access key messaging and asked 200% more questions
• Users were 12.66% more likely to access service information

Not only did we see significant increases in engagement, we also saw that the system was both safe and cost effective. 

• Our evaluation framework showed that answers were 99% safe and 100% accurate
• Providing a response to users cost $0.007

Girl Effect’s Next Steps

The impact of Generative AI was irrefutable. Based on these findings, we are now pursuing novel uses of the technology, as we aim to expand the scope of user engagement augmented by Generative AI, including:

• Improvements in the conversational experience
• Personalization of the user experience based on past engagement
• Rollout in new languages for Kenya and India
• Longer form, guided user journeys around key thematic areas, powered by a multi-agent system

Now Read the Whitepaper!

The whitepaper places Generative AI within Girl Effect’s theory of change, details the steps taken to roll out GenAI features within our existing chatbots and the results of the experiments which were conducted to gauge the impact, and details the technical challenges related to building the architecture required to safely realise our ambitions, before looking to what the next set of innovations will look like.

At Girl Effect, and as a signatory of the Principles for Digital Development we are committed to advancing the knowledge of the community of practice in the field of ICT4Development. We hope that, by sharing our journey, we will be able to offer valuable insights and advice as to how to go about integrating Generative AI into products and programs with similar objectives.

You can download the whitepaper here.

We look forward to hearing your thoughts!