CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents
Researchers have introduced CodeClinic, a new benchmark designed to evaluate the ability of Large Language Model (LLM) agents to automate clinical reasoning tasks. Built on the MIMIC-IV dataset, this framework addresses the limitations of existing systems that rely on manually curated tools or inefficient zero-shot code generation. CodeClinic assesses whether LLMs can synthesize and compose reusable clinical skills through two complementary tasks: longitudinal ICU surveillance, which simulates patient monitoring with structured decisions, and compositional information seeking, involving complex multi-step reasoning across thousands of instances. The study also proposes an offline autoformalization pipeline that converts natural-language clinical guidelines into verified Python skill libraries using iterative LLM refinement. Results indicate that this approach significantly improves consistency and reduces per-query token usage by up to 40% compared to traditional methods. This development aims to reduce the expert effort required for maintaining clinical tool libraries while enhancing the reliability of automated patient state tracking and sepsis detection in intensive care units.
Wire timeline
CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents
Researchers have introduced CodeClinic, a new benchmark designed to evaluate the ability of Large Language Model (LLM) agents to automate clinical reasoning tasks. Built on the MIMIC-IV dataset, this framework addresses the limitations of existing systems that rely on manually curated tools or inefficient zero-shot code generation. CodeClinic assesses whether LLMs can synthesize and compose reusable clinical skills through two complementary tasks: longitudinal ICU surveillance, which simulates patient monitoring with structured decisions, and compositional information seeking, involving complex multi-step reasoning across thousands of instances. The study also proposes an offline autoformalization pipeline that converts natural-language clinical guidelines into verified Python skill libraries using iterative LLM refinement. Results indicate that this approach significantly improves consistency and reduces per-query token usage by up to 40% compared to traditional methods. This development aims to reduce the expert effort required for maintaining clinical tool libraries while enhancing the reliability of automated patient state tracking and sepsis detection in intensive care units.
cs.AI updates on arXiv.org