First Peer-Reviewed Study to Evaluate AI-Generated Impressions Demonstrates Radiologist Preference for Domain-Specific Versus General Models

Rad AI, the leader in AI-powered radiology workflow solutions, announced the publication of new peer-reviewed research in Nature Portfolio’s npj Digital Medicine, demonstrating that domain-specific AI models better meet radiologist expectations than general-purpose large language models (LLMs) when generating radiology impressions, the most critical component of the report guiding patient care. Conducted in collaboration with researchers at Moffitt Cancer Center, the study represents the first comprehensive, multi-stakeholder evaluations of AI-generated impressions and highlights a broader shift in how healthcare organizations assess AI: not just on technical performance but on how well it supports real-world clinical workflows, communication and decision-making.

The analysis included 200 oncologic CT reports, comparing radiologist-authored impressions with outputs from a radiology-specific AI model fine-tuned on institutional data from radiologists and oncologists, and a general-purpose LLM. The domain-specific AI model performed in close alignment with human radiologists across key quality measures, including completeness, correctness and conciseness. Generic LLMs, however, were consistently deprioritized by radiologists, with differences driven by conciseness and overall clinical usability ranging from approximately 28% to nearly 50%.

The findings showed the domain-specific AI model generated high-quality radiology impressions significantly faster than general-purpose LLMs while maintaining the concise, high-signal structure radiologists rely on. The results also reinforced that purpose-built AI can meet clinician expectations in real-world settings when tailored to each radiologist’s workflow. Patient harm ratings remained low across all impression types, with scores ranging from approximately 1.0 to 1.2 on a 3-point scale, where 1 represents no-to-minimal risk of harm.

“Impressions are the most critical part of the radiology report. This study demonstrated that, in addition to accuracy, customization matters to radiologists in order for them to feel confident that AI clearly and effectively communicates findings from impressions — and does so in a way that aligns with their clinical workflows,” said Andrew Del Gaizo, Chief Medical Information Officer at Rad AI and co-author of the study. “These findings highlight the importance of AI that’s purpose-built for radiology and designed to support how clinicians actually make decisions in practice.”

In radiology, impressions serve as the critical summary that informs downstream care decisions, where clarity and prioritization in reporting are essential. This study provides a novel look at how both radiologists and oncologists evaluate the quality of those impressions in practice. By taking on the time-intensive work of drafting impressions, domain-specific AI allows radiologists to focus on the clinical judgment and decision-making that are central to patient care. These results reinforce that generating accurate text alone isn’t enough, and the clinical usefulness of AI models depends on how effectively the information is communicated in practice. Radiologists consistently favored concise, high-signal summaries, while oncologists were more receptive to more detailed, explanatory outputs, underscoring the need for AI systems that can support different users across specialties.

“We saw meaningful variability in how radiologists and oncologists evaluated the same outputs, which has important implications for how health systems think about deploying AI,” said Trevor Rose, MD, MPH, diagnostic radiologist at Moffitt Cancer Center. “The study shows that impression quality can be inherently subjective even with clinicians in the same specialty. Rather than optimizing for a single ‘best’ output, organizations should be prioritizing tools that can adapt to different users, workflows and clinical preferences.”

As healthcare organizations continue evaluating AI solutions, the distinction between general-purpose models and clinically tuned systems is becoming increasingly important. Tools that align with real workflows and clinician expectations are more likely to deliver meaningful impact in practice, enabling radiologists to keep pace with growing demand while maintaining high standards of patient care.

Write to us [⁠wasim.a@demandmediaagency.com] to learn more about our exclusive editorial packages and programmes.