A couple instances where errors were caused by automation, and I suspect that they were probably shoddy AI errors from generative AI chatbot technology.
Notes, references, transcript: https://chloehumbert.substack.com/p/probable-ai-error-anecdotes
References:
Data Brokers and the Sale of Americans’ Mental Health Data. The Exchange of Our Most Sensitive Data and What It Means for Personal Privacy. By: Joanne Kim, February 2023, Duke Sanford Cyber Policy Program Data broker 4 advertised highly sensitive mental health data to the author, including names and postal addresses of individuals with depression, bipolar disorder, anxiety issues, panic disorder, cancer, PTSD, OCD, and personality disorder, as well as individuals who have had strokes and data on those people’s races and ethnicities. Two data brokers, data broker 6 and data broker 9, mentioned nondisclosure agreements (NDAs) in their communications, and data broker 9 indicated that signing an NDA was a prerequisite for obtaining access to information on the data it sells.
Does CDC HICPAC want to make a mockery of infection control in healthcare? Chloe Humbert Aug 21, 2024 At a recent NNU webinar on the use of AI in healthcare, someone told the story about an automated shift change report that just makes a sheet with no human to human handoff between actual healthcare workers. In this case the automated sheet failed to show that the person coming into the hospital had “no immune system” and had the nurse not made the extra step of checking the patient’s chart, they would’ve put the immune compromised person in with the patient who had covid and flu.
Starbucks Abandons Borked AI Inventory Tool That Couldn’t Count: Report The company behind the tool said its on a “mission to count everything of value in the world.” By Matt Novak Published May 22, 2026, 12:40 pm ET Gizmodo The video advertises the product as having 99% accuracy. But according to Reuters, the tool was “frequently” miscounting and mislabeling items.
Transcript:
(00:00:00):
I’m Chloe Humbert and I have two little stories from the last year where I can see
(00:00:06):
like if you know how this chatbot LLM large language model AI stuff works you can
(00:00:16):
see how this type of automation error could happen and and how
(00:00:22):
Once you know how it happens, you’ll see it in more places.
(00:00:26):
So I have a few anecdotes that demonstrate this.
(00:00:31):
It seems very likely to me that it’s the shoddy AI LLM chatbot thing being implemented.
(00:00:38):
And so one of them is that information was erroneously added into...
(00:00:49):
A visit summary after a doctor’s appointment.
(00:00:52):
So in other words, I had a doctor’s appointment.
(00:00:56):
And afterwards, I checked the paperwork in my medical portal.
(00:01:03):
And it added something like about me that wasn’t true.
(00:01:11):
and like it seemed to be erroneous it wasn’t even something that was asked at the
(00:01:16):
visit or talked about or anything it was just it was just added in and I you know
(00:01:22):
contacted the provider and said you need to fix that because we never discussed
(00:01:28):
that that’s not true I never said that and where did that come from even
(00:01:35):
so they were like just they just fixed it and then a few months later all of a
(00:01:40):
sudden like you were asked this question about this particular issue like whenever
(00:01:47):
I had an appointment it was happening and it wasn’t just happening to me that was
(00:01:51):
the weird thing so I wondered if maybe there was a system-wide thing that you know
(00:01:56):
pushed this into everybody’s uh
(00:02:00):
like visit notes just arbitrarily.
(00:02:03):
It was very strange.
(00:02:04):
So the next thing that happened was I had a medical appointment.
(00:02:10):
This was another specialist appointment.
(00:02:12):
It was going to be a remote virtual televisit appointment.
(00:02:19):
So I signed in for my appointment and the nurse practitioner I was seeing at the
(00:02:25):
specialist office,
(00:02:28):
Tried to connect to the call twice, and it seemed like, and then they just dropped.
(00:02:35):
And then I got a message saying, sorry, your appointment is being canceled.
(00:02:39):
You can reschedule.
(00:02:41):
The provider couldn’t connect for whatever reason.
(00:02:45):
So, okay, I get on the messaging.
(00:02:48):
I’m like, okay, I’ll make another appointment.
(00:02:51):
This is a mess because essentially to make another appointment at this particular
(00:02:59):
specialist or a lot of them in my plan in network of my health insurance,
(00:03:07):
it was like a nine-month or more wait to get an appointment.
(00:03:11):
So... I was like wow.
(00:03:13):
Also it was getting towards the end of the year I think, and I knew my insurance would be changing on January first so then I was kind of hesitant to make an appointment not knowing if they’d be in my insurance the next year.
(00:03:29):
So I didn’t push it.
(00:03:31):
Like I didn’t follow up.
(00:03:32):
I put that on hold trying to wait and see if my provider list for my new insurance
(00:03:38):
the following year would include them.
(00:03:41):
So nothing.
(00:03:42):
And then some weeks later, maybe a month or two later, what happened
(00:03:48):
I was alerted to the fact that my insurance paid for a visit.
(00:03:52):
That never happened because I got a bill for a copay.
(00:03:58):
So I get the specialist co-pay bill,
(00:04:00):
and I’m thinking,
(00:04:01):
well,
(00:04:01):
wait a minute,
(00:04:03):
this visit didn’t happen.
(00:04:05):
It just didn’t happen, right?
(00:04:07):
So I go, I try to contact the medical, so I try to contact them.
(00:04:12):
I’ll contact the doctor’s office,
(00:04:14):
the specialist’s office,
(00:04:17):
and they don’t seem to know what’s going on.
(00:04:19):
And so I contacted my insurance, and my insurance needed me to file an appeal.
(00:04:27):
So I filed an appeal on this visit.
(00:04:29):
That never happened.
(00:04:30):
I explained everything.
(00:04:33):
That the call was dropped.
(00:04:34):
The
(00:04:35):
Provider couldn’t connect,
(00:04:36):
so we canceled the appointment because,
(00:04:38):
you know,
(00:04:39):
I was up in the air about,
(00:04:40):
you know,
(00:04:41):
what my insurance would be the next year.
(00:04:43):
You know, it just didn’t happen.
(00:04:45):
And it was one of the insurance people I talked to at my health insurance.
(00:04:52):
They said that apparently everything went through like it was a visit,
(00:04:56):
like it had all the paperwork and stuff,
(00:04:58):
like as if a visit happened.
(00:05:00):
But
(00:05:04):
when one of the people went in they said yeah it looks like all of the information
(00:05:09):
is there for a visit that happened yet
(00:05:13):
This person could also see notes in the portal that,
(00:05:19):
you know,
(00:05:19):
indicated,
(00:05:20):
yes,
(00:05:20):
I was talking about rescheduling the appointment.
(00:05:23):
So they could tell that,
(00:05:24):
like,
(00:05:24):
you know,
(00:05:25):
they could see the messages in the portal where I’m trying to,
(00:05:29):
you know,
(00:05:30):
sort out whether we’re going to reschedule the appointment and that the
(00:05:33):
appointment,
(00:05:34):
because the appointment didn’t happen.
(00:05:36):
And it was, you know, obviously there was contradictory information there.
(00:05:41):
Well, in the end, it was reversed.
(00:05:44):
They reversed it all.
(00:05:46):
And in the end,
(00:05:47):
my insurance,
(00:05:48):
somebody at my insurance,
(00:05:50):
when I followed up on the appeal,
(00:05:53):
they told me that the specialist’s office blamed the EPIC system,
(00:05:57):
which is...
(00:05:58):
introducing LLM chatbots to do visit summaries and stuff.
(00:06:04):
So in other words, the Epic system generated a visit summary for a visit that never happened.
(00:06:09):
So imagine how many actual visits are mischaracterized.
(00:06:13):
because if it could generate a summary out of thin air when the visit never
(00:06:19):
happened,
(00:06:19):
what is it doing when there is material to work with?
(00:06:23):
Who knows?
(00:06:24):
But that was how it was able to actually get to the point where it was actually a
(00:06:29):
visit that was confirmed with all of the necessary information to get the insurance
(00:06:34):
to pay for it.
(00:06:36):
So it just, quote-unquote, hallucinated a whole visit.
(00:06:43):
So that’s problematic because you could see how in a medical situation that could go way wrong.
(00:06:49):
And of course,
(00:06:50):
there are dozens of...
(00:06:53):
My situation is very minor because these are things that were completely,
(00:07:00):
fairly irrelevant.
(00:07:01):
Fairly irrelevant.
(00:07:02):
I mean,
(00:07:03):
not to say that billing isn’t irrelevant and getting a bill that you weren’t
(00:07:06):
supposed to.
(00:07:07):
That’s very serious, but...
(00:07:09):
I’m talking about life-threatening.
(00:07:11):
There are life-threatening examples I have heard of over the past few years.
(00:07:16):
And people have been warning from National Nurses United.
(00:07:20):
I’ve put a lot of this on.
(00:07:22):
I’ve talked about this before.
(00:07:23):
I’ve written about this.
(00:07:25):
So there are a lot more serious ramifications.
(00:07:28):
But these problems are everywhere.
(00:07:30):
And my point is that
(00:07:32):
It’s very serious when it’s medical.
(00:07:34):
And then it’s also causing problems when it’s a financial situation.
(00:07:39):
Because not everybody would have been able to pick up on that very quickly.
(00:07:45):
Maybe somebody without a co-pay would just go through and...
(00:07:50):
Yeah,
(00:07:50):
you could be denied another appointment because they could say,
(00:07:54):
well,
(00:07:54):
you had one already,
(00:07:56):
and then you’re left scrambling,
(00:07:57):
like,
(00:07:58):
well,
(00:07:59):
I didn’t have my yearly follow-up or whatever.
(00:08:02):
And this is my next story about...
(00:08:07):
It Affecting you financially is that last year,
(00:08:10):
I was planning on buying an apple device, actually I was planning on buying two Apple devices and ended up not buying either.
(00:08:22):
from Apple because I got turned down for an Apple card uh and I was like so
(00:08:28):
irritated because I have excellent credit like my credit my credit is in the 800s I
(00:08:34):
have a credit score in the 800s I have an immaculate like there’s nothing negative
(00:08:39):
on my credit reports at all and I’m not bragging about this but it was hard um
(00:08:46):
because uh
(00:08:48):
I was in a real problem in the Great Recession because I had a lack of credit and
(00:08:55):
then I had financial issues because of the Great Recession.
(00:08:59):
So anyway, I went through all of this.
(00:09:01):
So it took me years to get medical bills that were paid off or weren’t even
(00:09:08):
appropriate or weren’t even real.
(00:09:11):
I had stuff on my credit report like that.
(00:09:13):
It was medical, all medical.
(00:09:15):
Um, so I’m very irritated
(00:09:18):
If there is something now on my credit report that isn’t right,
(00:09:23):
like even so much as that and like the profile,
(00:09:26):
whatever,
(00:09:27):
past employer name or whatever.
(00:09:30):
I’m very sensitive to that because I’m always scrutinizing my credit report because
(00:09:35):
I’ve had to go through such hoops to fix errors like that were financial and were
(00:09:40):
devastating.
(00:09:41):
And now, you know, I don’t want to see anything wrong in those credit reports.
(00:09:46):
And
(00:09:47):
um so I was turned down for an apple card and the people at Goldman Sachs you know
(00:09:53):
customer service or whatever basically um it had to be escalated to like a
(00:09:58):
supervisor because I was like why am I being turned down and why am I not being given the reason for my turn down.
(00:10:02):
Because I know the law is that when you’re turned down for credit they have to tell you they have to give you a copy of your credit report or something, they have to give you a reasoning that you’re turning you down for credit
(00:10:23):
And I knew they couldn’t be turning me down for my credit, you know like I said it’s immaculate there’s no negatives my credit score is in the
(00:10:27):
800s there’s nothing wrong with my credit so I tell this to the person at Goldman
(00:10:36):
Sachs customer service and then they have to escalate it to a supervisor and I
(00:10:42):
talked to a supervisor who essentially told me it’s an identity issue an identity
(00:10:48):
issue and
(00:10:51):
so it didn’t have anything to do with my financial credit report it was a identity
(00:10:57):
confirmation issue and I’m like you can’t the person said you know we can’t confirm
(00:11:02):
your identity because of the credit report and so I went back and I was looking at
(00:11:10):
it and I think
(00:11:12):
like the person suggested that it was possibly a phone number or an address or
(00:11:17):
something like that that was they were cagey about it they wouldn’t seem they
(00:11:22):
wouldn’t
(00:11:23):
Admit to exactly what it was,
(00:11:26):
but the way that the person had run down the list of possibilities,
(00:11:31):
I suspect that it was either my name or my phone number,
(00:11:34):
which now,
(00:11:35):
this is really problematic,
(00:11:36):
because now your identity,
(00:11:38):
like,
(00:11:39):
out there when they collect information on you,
(00:11:43):
basically...
(00:11:44):
Your phone number has become almost like a social security number.
(00:11:48):
And your social security number was never supposed to be an identification number
(00:11:53):
in the first place.
(00:11:54):
But let’s put that aside.
(00:11:55):
But now your phone number has become that.
(00:11:58):
And it’s a nightmare to change your phone number.
(00:12:01):
And I’m not the only one who’s...
(00:12:04):
experience this but there’s they actually after you know trying to change my phone
(00:12:10):
number with things I found an article I think it was in Bloomberg um but uh yeah
(00:12:14):
that like it’s a nightmare and basically it’s because everything is attached to
(00:12:18):
your phone number now your identity and that’s how like data brokers track you is
(00:12:23):
one of the things is that of course data brokers are tracking you they’re buying
(00:12:27):
and selling your medical um
(00:12:30):
History and stuff. And what’s weird is that it’s illegal to hack a hospital. It’s illegal to hack a healthcare system and steal medical data. It’s illegal for a healthcare provider to sell your medical data. However.
(00:12:45):
Once your medical data is out there,
(00:12:47):
it is completely legal to buy and sell it for advertising,
(00:12:51):
for marketing,
(00:12:52):
for targeting,
(00:12:53):
for the government to buy it and use your mental health data against you.
(00:12:58):
And they have
(00:13:00):
mental health data right down to your physical address,
(00:13:03):
your phone number,
(00:13:04):
your name,
(00:13:05):
your diagnosis,
(00:13:07):
notes that go to your health insurance to justify paying for services,
(00:13:13):
etc.
(00:13:15):
It’s really not good.
(00:13:17):
But anyway, the point being that
(00:13:21):
I went through my credit report and I had just looked at it like probably not long
(00:13:26):
before I applied for the Apple card because I have freezes on all my credit my all
(00:13:35):
three credit bureaus I have
(00:13:37):
My credit is has like a voluntary,
(00:13:40):
like a freeze on it,
(00:13:41):
and I have to unfreeze it in order to apply for credit now.
(00:13:46):
So, and then I refreeze it.
(00:13:49):
And that’s to,
(00:13:50):
you know,
(00:13:50):
make sure that,
(00:13:51):
like,
(00:13:51):
in case of identity theft or whatever,
(00:13:54):
I just,
(00:13:55):
I don’t have to
(00:13:57):
you know it’s just gonna get turned down because I have a freeze on it so if some
(00:14:02):
bad actor tries to steal my identity and apply for credit or something so I
(00:14:08):
probably did look at it and I didn’t see this so I go over the whole thing and okay
(00:14:13):
so there were a couple of you know things changes my phone number also
(00:14:21):
there was a name in there now I have changed my name because I’m not and um my name
(00:14:28):
is Chloe Kaczenski Humbert and Humbert is my spouse’s last name and I have kept my
(00:14:36):
quote-unquote maiden name as my middle name and I had this changed by the court so
(00:14:42):
uh anyway it had a variation of my name and
(00:14:48):
Actually,
(00:14:48):
two variations of my name on my credit report saying,
(00:14:52):
you know,
(00:14:52):
they say,
(00:14:53):
what other names has this person gone by?
(00:14:55):
And if you change your name, it’ll have your, you know, previous names in there.
(00:15:01):
And...
(00:15:03):
It had a couple of variations, like mix and mashups of my name.
(00:15:08):
It even had one of them was with Humbert as my first name, I think, or something like that.
(00:15:14):
It was strange.
(00:15:15):
It was doing word jumbles, and it was...
(00:15:18):
configurations of my name that never were the case.
(00:15:22):
However,
(00:15:23):
what was interesting about one of those names is that the same jumbled name I have
(00:15:29):
seen in emails to one particular email address that goes through a domain name that
(00:15:36):
I have had.
(00:15:38):
And it’s an email address that gets this name attached to it.
(00:15:46):
And I get these
(00:15:47):
you know spam scammy kind of emails the kind of emails that like say you know we’re
(00:15:54):
watching you and we know what porn you’re watching or whatever bullshit.
(00:15:59):
you know and it clearly thinks that I’m a man because just you know going by the
(00:16:04):
type of spam scam extortion email type of thing it is and and I find these in my
(00:16:12):
spam folder usually they go straight to the junk because they’re you know they are
(00:16:16):
scams but this is where I’ve seen the name and this name wound up on a credit
(00:16:20):
report of mine so I’m thinking that they’re getting this information from data
(00:16:24):
brokers and
(00:16:26):
not only that but I think that data brokers are now using AI chatbot LLM tools to
(00:16:33):
process their data and then mashing it up and making like huge errors so that
(00:16:39):
basically the errors that were already because the data brokers start out with
(00:16:43):
errors and then now they’re high test errors because they’re putting them through
(00:16:47):
shoddy AI bullshit and that ends up on my official data
(00:16:54):
credit report is this obvious name that didn’t come from any bank.
(00:16:59):
It certainly didn’t come from any creditor.
(00:17:01):
It certainly wasn’t a name that was used by any employer I’ve ever had or any whatever.
(00:17:08):
They specifically got this name from a data broker because it’s not a name I’ve
(00:17:13):
ever used or ever had or,
(00:17:15):
you know,
(00:17:15):
whatever.
(00:17:16):
It was a mishmash that only scammers have used.
(00:17:20):
so and that yet that ended up on my official credit report not as my primary name
(00:17:26):
at least but as a past name the other error on my credit report was my current
(00:17:33):
address was correct
(00:17:35):
However, your credit report will have several of your old addresses.
(00:17:41):
Usually it’ll go back until the beginning of time to the first time you applied for
(00:17:47):
credit at all or had any kind of bill,
(00:17:51):
I guess,
(00:17:52):
or whatever.
(00:17:53):
And so there was a list of addresses in it.
(00:17:57):
And what was interesting about one of the addresses was that it was never my address.
(00:18:04):
but not only was it never my address it’s an address that doesn’t exist on any map
(00:18:10):
like it just doesn’t exist at all like it’s not a real address so not only was it
(00:18:17):
never my address it’s not even a real address that exists on any map anywhere so uh
(00:18:23):
What’s what was interesting about it,
(00:18:25):
though,
(00:18:26):
is that it looked like it could have been my address.
(00:18:29):
So I may have actually even seen it on a path like I may have seen it like on my
(00:18:35):
credit report and just kind of glossed over.
(00:18:39):
and just scrolled on by thinking, yeah, that looks right.
(00:18:43):
Because at first, it does look like an address I might have had.
(00:18:46):
And the reason for that is,
(00:18:49):
I think it was like a combination of two or three addresses I had in the past.
(00:18:55):
So that’s why it wasn’t a real address.
(00:18:57):
It was actually a...
(00:18:59):
It put together a couple of things from addresses that really were mine in the past.
(00:19:07):
And you could totally see this happening with a chatbot,
(00:19:11):
because a chatbot comes up with stuff that sounds right,
(00:19:14):
it looks right,
(00:19:16):
even if it isn’t.
(00:19:17):
But it’s like,
(00:19:18):
oh,
(00:19:19):
this is an address she’s likely to have had,
(00:19:21):
because she’s had these other addresses,
(00:19:23):
right?
(00:19:23):
Right?
(00:19:24):
So I think that it was an error by shoddy chatbot LLM technology being used in the
(00:19:33):
credit report or for somebody reporting to the credit report or whatever.
(00:19:38):
And because it just seemed like a classic kind of type of AI error that would happen.
(00:19:45):
An address that doesn’t exist, but looks like it could have been my past address.
(00:19:50):
So all of these are problems because,
(00:19:52):
and especially if the government is buying these things and then using them against
(00:19:59):
people and
(00:20:00):
And your credit,
(00:20:02):
you know,
(00:20:02):
if you’re going to apply for credit,
(00:20:04):
this is going to trip things up.
(00:20:06):
And I have to wonder if it’s doing this about your identity stuff,
(00:20:10):
like your name and your address and stuff like that.
(00:20:14):
I mean, how long is it before it starts messing up the actual financial data?
(00:20:19):
And I’m sure it has,
(00:20:20):
like it hasn’t happened to me,
(00:20:21):
but like I’m just wondering how many people’s
(00:20:25):
Or maybe it has happened to me and I just haven’t seen it because it hasn’t done
(00:20:28):
anything negative yet.
(00:20:30):
So maybe there are already errors in there.
(00:20:35):
What are you supposed to do?
(00:20:37):
Nobody should have to be going through this and meticulously finding all of these
(00:20:43):
errors if they’re using this shoddy technology that is known to be error-prone.
(00:20:50):
Why are they doing it?
(00:20:51):
Why is it even legal to use it if it’s so error-prone?
(00:20:57):
Just doesn’t make sense.
(00:20:59):
But the idea here, the warning that I think, you know, is that check your credit reports.
(00:21:05):
Like,
(00:21:06):
request your credit reports and check them for errors,
(00:21:09):
which you should be doing anyway,
(00:21:11):
you know,
(00:21:11):
even without AI errors.
(00:21:12):
There was always been error problems with those and also your medical reports and
(00:21:18):
see,
(00:21:19):
you know,
(00:21:19):
because I’ve heard some other stories about,
(00:21:22):
you know,
(00:21:22):
people saying,
(00:21:23):
oh,
(00:21:23):
I never said that at my doctor’s appointment.
(00:21:26):
Where did that come from?
(00:21:27):
You know, and you could end up being treated for something that isn’t even real. That’s not a good idea. And a lot of people, you know maybe you think well that wouldn’t happen to me because I know what whatever, I would notice that, but not everybody would. So. It’s not just about whether it’s going to hurt me. It’s going to hurt somebody.
(00:21:31):
Yeah, so check your...
(00:21:49):
credit report and check all your medical documents after your visits,
(00:21:55):
especially the visit summary,
(00:21:57):
because I have heard from medical professionals that this Epic isn’t the only one.
(00:22:03):
They’re using these AI chatbots to summarize visits.
(00:22:07):
They’re using it.
(00:22:08):
So we got to be on this, and I think it should be stopped.
