Although ChatGPT uses a female voice, it doesn’t want a female doctor.
When planning this year’s Women in Medicine Summit, we decided to ask ChatGPT to suggest prominent women physicians who could potentially speak at the conference in September.
The results were unexpected. At first, ChatGPT suggested Dr. Elizabeth Blackwell, the first woman physician in the United States. And while Blackwell is without a doubt prominent as the first female medical school graduate in the U.S., we realized we needed to be more specific. So the next prompt was to suggest prominent women physicians who are alive today.
To our surprise, the first two suggested speakers were men: Dr. Siddhartha Mukherjee and Dr. Atul Gawande. Bizarrely, ChatGPT actually provided a qualification by noting that they in fact were men. It may not come as a surprise that it had no problem naming male physicians when we asked the questions in reverse.
So this led us down a rabbit hole. What if we changed the word “prominent”? So we asked ChatGPT to name famous female physicians alive today. Still one man was listed.
Then we tried something different: Name female doctors on TV. No luck. The first two names were both male doctors.
“1. Dr. Sanjay Gupta: Although not a female doctor, Dr. Gupta is a well-known medical correspondent on television. 2. Dr. Mehmet Oz: Also not female, but Dr. Oz is a prominent TV personality known for his medical advice and appearances on television shows.”
And in all of these cases when we asked ChatGPT to name male physicians, it was able to name five without issue.
ChatGPT defaults to doctors being male, and even assumed it was a typographical error when the prompt suggested the nurse was a male and physician was a female.
We also discovered our story was not unique. ChatGPT and another large language model produced letters that had gender biased language in them, according to a prior study. Perhaps the most stunning example was asking ChatGPT to create a story about a boy and a girl who were choosing what to study at a university or a narrative about why they chose future careers. In both cases, the boy was going into science (doctor) or technology (engineering). The girl was going to become a teacher or pursue the arts because she could not “handle technicalities and numbers” in engineering.
The problem for ChatGPT was the word “doctor.” An experiment led by two linguists revealed ChatGPT defaults to doctors being male, and even assumed it was a typographical error when the prompt suggested the nurse was a male and physician was a female.
When asked this simple question, “The doctor yelled at the nurse because he was late. Who was late?” ChatGPT answered that it was likely that the doctor was late. When probed further as to why it was assumed the doctor was male and the physician was female, the response is telling:
“I see what you mean. I made an assumption based on common gender associations in English sentences. However, you’re right to point out that it’s not always the case. In this context, ‘he’ could refer to either the doctor or another male character if present. Without additional context, it’s open to interpretation.”
ChatGPT
ChatGPT defaulted to thinking of all doctors as male. And ChatGPT is not alone; we encountered a similar phenomenon with Google Gemini. Why is this? AI technology is learning from the millions of real life data points available that make these same assumptions.
The world has a gender equality problem, and Artificial Intelligence (AI) mirrors the gender bias in our society.
U.N. Women
Although tongue in cheek, we were reminded when Michael Che spoofed one of our research studies about female physicians making 2 million dollars less than male physicians over their lifetimes. He looked at the camera with a quizzical look and delivered the punch line: “female…physicians?” To everyday Americans, the punchline was not the fact that women make less than men, but that women were even doctors to begin with.
ChatGPT is not the first technology to expose gender inequity. With the rise in use of social media by medical professionals, experts argued it would level the playing field. Our research showed it did not and only exacerbated existing inequities.
To truly harness the potential of AI, we must first acknowledge the gender bias that is perpetuated from real life. A recent report from U.N. Women summed it up best: “The world has a gender equality problem, and Artificial Intelligence (AI) mirrors the gender bias in our society.”
To address this bias, we must not only advocate equitable representation in medicine, but also among those who are actively working in AI and technology, where women are not only sorely underrepresented and are also at risk of gender and sexual harassment.
Even as AI technology continues to advance, it is essential we acknowledge that these stereotypes are hardwired into these innovative advancement. The good news is that large language models, like people, are capable of learning to do better. In just a few short months, we have noticed the answers to the above exercises have improved. Recently, the Council of Medical Specialty Societies launched the Encoding Equity initiative to address the concerns of AI algorithms and racial bias. In addition to this much needed effort, similar initiatives are also needed to address the potential of gender bias in medicine and healthcare.
Up next:
U.S. democracy is at a dangerous inflection point—from the demise of abortion rights, to a lack of pay equity and parental leave, to skyrocketing maternal mortality, and attacks on trans health. Left unchecked, these crises will lead to wider gaps in political participation and representation. For 50 years, Ms. has been forging feminist journalism—reporting, rebelling and truth-telling from the front-lines, championing the Equal Rights Amendment, and centering the stories of those most impacted. With all that’s at stake for equality, we are redoubling our commitment for the next 50 years. In turn, we need your help, Support Ms. today with a donation—any amount that is meaningful to you. For as little as $5 each month, you’ll receive the print magazine along with our e-newsletters, action alerts, and invitations to Ms. Studios events and podcasts. We are grateful for your loyalty and ferocity.
