Garbage in, Garbage out: The MAHA Report

Today’s poem touches on many complex issues relating to both individual and societal health care and well-being today. I find it simultaneously upsetting and insightful. And as such, a good intro to the topic of the day – the disturbing, distorted, Make America Healthy Again (MAHA) report. The poem reminds us of the dangers of Dr. Google, of the allure of numbing the pain, of the importance of listening to truths from the body and from science. It comes to us in translation, with consequences repeatedly discussed on this blog and in this case emphasizing our reliance on the knowledgeable expertise of the translator as similar to our reliance on scientists to understand our health crisis.

The Following Scan Will Last Four Minutes

By Lieke Marsman
Translated By Sophie Collins

Before you sink away
into the morphinesweet unreality of the everyday
we would like to say something
about those spasms and fasciculations of yours
as well as that bump on your back

For years you have no doubt been googling every freckle
Just recently
you were at the doctor’s with a patch of dry skin on your leg
Diagnosis: too much shower gel
But on hearing the word chondrosarcoma
you went home and immediately unplugged your router
Do you know where your priorities lie?

Do you know what life has to offer
or did those endless therapy sessions
and that eight-week mindfulness course
simply teach you how to tolerate suffering
that every signal in your body
can be temporarily expelled
to the rhythm of some breathing exercise?
          Let the pain be
          To be free is to be free of need
Wrong
To be free is to need some fresh air
and to be able to get up and go outside

Don’t say we didn’t warn you
                                                                           Translated from the Dutch

Robert F Kennedy, Jr. and his HHS MAHA gang took Dr. Google to new heights when it recently released a document purportedly detailing childhood health reasons for the decline in American’s life expectancy. It seems clear to any professor out there that (at least portions of) the report were written by a Large Language Model (LLM) – and not a very sophisticated one at that. That’s because reviewers found one of the few clear tell-tale signs of LLM generated “research” writing – hallucinated sources – as well as another, the presence of ‘oaicite’ in the reference URLs, indicating use of OpenAI (ChatGPT) software.

What is a hallucinated reference? Basically, consider that an LLM generates predictive text (similar to your phone trying to complete sentences for you as you type). If you ask it to ‘reference’ material, it will generate predictive citations. These are based on probabilities. So it will take your basic topic, and pick out authors’ names that come up with some frequency in relation. It’ll do the same with journals, issues and page numbers. And it’ll generate an innocuous title like: “Child health: challenges and opportunities”. At first glance, it’ll seem quite real. Until you go to the journal, or ask the author for the article, and find there is no such thing. They are no more based in scientific fact than if hundreds of monkeys were typing away in a locked room.

Knowledge that LLMs behave this way has been around at least since just after the widescale release of ChatGPT at the end of 2022. Professors like me have been using this knowledge to identify AI written work and catch cheaters, and some AI companies have been working to figure out how to make their LLMs more reliable, e.g. drawing only from published journal articles, and citing them completely, like Scite.ai. This latter point means there are generative AI applications available (for a fee) that would have increased the credibility and reduced the detectability of the dependence on AI to write the report that were passed over in favor of a lesser quality LLM.

Why do this? We can’t be certain, because we are being told the original document had “formatting issues” (insert well-seasoned professor eyeroll here). It could be a combined set of incompetence and laziness, after so many scientists have been fired and partially replaced with loyalist flunkies. These would include not being able to write a scientific report in the first place, not knowing what the LLM was doing in their names, and not checking the work it generated. That’s the kind interpretation of the problem. Since the report is full of pseudo-science that seems to back RFK Jr’s bizarre ideas about health and well-being (which were well known at the time of his controversial confirmation), the more malicious explanation also holds water. That is, that an LLM relying on actual, peer reviewed science would not respond with sufficient support for already-decided, but not defensible, positions, while one drawing from a conspiracy-laden internet could.

I’m not a health scientist or health economist so I can’t fact check much of this report. But I can tell you that by page 8 dumb statements start to show in ways that highlight the administration’s penchant for exaggeration and misinterpretation of data. At the top of the page, for example, they claim “Over the past century, US GDP has grown over 30,000%” and cite the FRED data (Federal Reserve of St Louis, which tracks a lot of long term macroeconomic data). It is true that nominal US GDP (GDP that has not had inflation taken out) is currently about 30 trillion dollars (29.2 trillion USD in 2024). The FRED data only goes back to 1947, so the citation cannot be where they get 100 years of data, nor the 125 years they imply by mentioning a 1900 benchmark later in the paragraph.

It is possible to go back further with reconstructed data sets, but the numbers become small and relatively challenging to compare, in part because what we can do and measure as economic activity has changed (though not, as the report does intend to show, enough to provide real insight into well-being rather than just economic productivity). So sure, you can get a figure of 30,000% growth in nominal GDP. But it is hardly informative and factually distorted, and breeds deserved distrust in all the rest of the presentation of data, such as the claims of magnitude for rising autism and ADHD diagnoses (page 12), which suffer from some of the same challenges in documenting change over time as GDP – we can recognize, diagnose, and treat these differently today than in the past. The compounded errors here are almost as much as the seeming errors in calculating compounded real growth. Not excluding inflation is an introductory macroeconomics course level mistake. The long run annualized growth rate of US real GDP since the government started documenting this metric in the 1930s is about 2% and the overall percent change in real GDP is more like 2200%, which even ChatGPT will give you if you ask for the percentage growth in GDP over the past century (I asked).

This is all rather beside the larger point, of course. Which is that the report is full of red herrings regarding the steps we can take to improve quality of life for youth, and those surviving past it, right now.

The leading cause of death in American children and teens is from gun-related deaths. Guns are not mentioned once in the report. Recent actual scientific evidence has shown that much of the decrease in US life expectancy can be linked to “deaths of despair” and opioid and other substance abuse. Opioid use and abuse in children is however understudied but abuse levels may have been 3.6% of the youth (12-17) population as long ago as 2017, and intergenerational costs from household opioid abuse are significant for health and child welfare. Opioids are not mentioned in any context in the report, despite a long list of potential medication misuses on pg. 60 and following that includes rare treatments for gender affirming care (not labeled as such, of course).

Cartoon credit and story: https://www.reddit.com/r/PeterExplainsTheJoke/comments/18sxhtk/petahhh_plsss/

Cover image generated by WordPress AI using prompt generated by WordPress AI: Create a featured image that visually represents the tension between the allure of modern technology and the challenges of health care. Showcase curiosity but also confusion. Incorporate elements of both digital distortion and natural health, like a heart rate monitor and a plant, blending technology with nature. Use soft, diffused natural light to evoke a sense of introspection, while keeping the focus sharp and details highly defined. Request a high-resolution output with intricate textures and contrasts to accentuate the complexity of the themes discussed in the blog post.

Dutch (original) version of the poem:

DE VOLGENDE SCAN DUURT VIER MINUTEN
Voordat je wegzakt
in de morfinezoete onwerkelijkheid van alledag
willen we alvast even iets zeggen
over die spasmen en fasciculaties van jou
alsook over de bult op je rug

Je hebt natuurlijk jarenlang ieder
huidvlekje gegoogeld, laatst nog
zat je bij de huisarts met een uitgedroogd
stukje scheenbeen
Diagnose: te veel douchegel gebruikt
Maar bij het horen van het woord chondrosarcoom
trok je de internetstekker eruit
Weet jij eigenlijk wel waar je prioriteiten liggen?

Weet jij eigenlijk wel wat het leven je te bieden heeft
of heb je tijdens ellenlange therapiesessies
en die achtweekse cursus mindfulness
dusdanig geleerd om ieder lijden
verdraagzaam te ondergaan dat ieder signaal van je lichaam
bij voorbaat gedoemd was weg te drijven
op het ritme van een of andere ademhalingsoefening
laat de pijn er zijn
vrij zijn is vrij zijn van verlangen?

Fout
Vrij zijn is verlangen naar buitenlucht
en erheen kunnen lopen

Zeg niet dat we je niet gewaarschuwd hebben