ChatRWD™ Beta from Atropos Health Exceeds Big Tech LLMs on Physician Trust and Ability to Answer Questions Completely with High-Quality Evidence in New Study
02 Julio 2024 - 8:07AM
Business Wire
The Research and Results are Published in
Preprint on arXiv, Continuing the Company’s Commitment to Provide
Transparency on its Methodology
Atropos Health, the pioneer in translating real-world clinical
data into high-quality personalized real-world evidence for care,
today announced the publication of “Answering Real-World Clinical
Questions Using Large Language Model Based Systems” which clearly
details ChatRWD’s superiority over other large language models
(LLMs) when it comes to trust, quality, and accuracy (limited
hallucinations). The paper comes on the heels of the successful
beta launch in Q42023.
ChatRWD™ reduces the time to produce high-quality
publication-grade real-world evidence from months to minutes
through a chat-based AI co-pilot. The study tested what evidence
can be produced using LLMs in two different ways, and then
evaluated each answer (or non-answer) on five quality measures.
Key findings:
- ChatRWD beta outperformed other LLMs with more completed
answers (94%)
- Independent physician reviewers found ChatRWD beta produced the
best answer on 60% of the questions and reported 44% of the answers
to be trustworthy enough to inform practice. The closest
alternative was OpenEvidence (which leverages existing literature)
with 46% and 30% respectively.
- ChatRWD beta provided answers with evidence 58% of the time.
OpenEvidence was the runner up at 24%. Other LLMs were <=
10%.
- Hallucinations, especially citing studies or reports that do
not exist, occurred for literature-based LLMs. Depending on the
LLM, up to 50% of the provided citations were unable to be found.
Top sources of error for ChatRWD beta were question definition and
that occurred less than 20% of the time when an answer was
generated.
"As generative AI technologies move from hype to utilization in
healthcare, it is critical that their relevance, reliability, and
actionability are rigorously measured and independently verified,"
said Atropos Health Chief Medical Officer and co-founder Dr.
Saurabh Gombar. "High-quality clinical evidence must remain the
cornerstone of value in healthcare. We put our technology to the
test, and the results were clear: ChatRWD has no alternative.
Independent physician evaluators trusted ChatRWD's results, and
while other LLMs were complimentary when answering well studied
questions, only ChatRWD consistently was able to create evidence
for novel questions."
ChatRWD is the first application incorporating chat-to-database
capability to help healthcare leaders advance and accelerate
evidence generation rapidly, saving organizations valuable time,
money, and staffing resources. ChatRWD users are guided through a
simple workflow to validate study design choices to reduce the risk
of hallucination. This unique vertical user experience is available
to healthcare institutions, data platforms, research institutions,
life sciences companies and more who install Atropos Health’s
Generative AI platform GENEVA OS™ (Generative Evidence Acceleration
Operating System). Questions are answered with data from the
Atropos Evidence™ Network, the largest federated data network.with
over 300 million patient records and every answer comes with a Real
World Fitness Score™ (RWFS) - evaluating the “fit-for-purpose” of
the data set used to answer the question. The average RWFS for
datasets used in the evaluation was 80 (out of 100), indicating
that the data was highly appropriate to answer the questions at
hand.
“Built on the core technology GENEVA OS, ChatRWD is not tied to
any single LLM ,” said Neil Sanghavi, President of Atropos Health.
“GENEVA OS is deployed locally at a customer's or data holder's
site and it does not expose any patient-level clinical or claims
data to LLMs; the LLMs in ChatRWD are used to create the query and
design the study, but GENEVA OS is used to execute the query -
without any connection to the LLMs.”
The publication outlines the approach and the benefits to
accuracy in the generation of evidence. Based on the outcome of
this study, the full launch of ChatRWD will follow with the
potential to democratize high-quality, accurate, transparent, and
personalized real-world evidence for everyone. To become a ChatRWD
beta user, sign-up at https://www.atroposhealth.com/chatrwd
About Atropos Health
Atropos Health is the developer of GENEVA OS™, the operating
system for rapid healthcare evidence across a robust network of
real-world data. Healthcare and life science organizations work
with Atropos Health to close evidence gaps from bench to bedside,
improving individual patient outcomes with data-driven care,
expediting research that advances the field of medicine, and more.
We aim to transform healthcare with timely, relevant real-world
evidence.
To learn more about Atropos Health, visit www.atroposhealth.com
or connect through LinkedIn or follow on X (Twitter)
@AtroposHealth.
View source
version on businesswire.com: https://www.businesswire.com/news/home/20240702832027/en/
Media Contact: Bruno Solari
atropos@solcomms.co