PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code
Researchers from arXiv have published a study addressing a critical flaw in using Large Language Models (LLMs) for scientific simulations: code that executes successfully may still solve the wrong physics. This mismatch, termed the 'comprehension-generation gap,' occurs when generated input files converge but encode governing equations differing from user intent. The team introduces the Intent Fidelity Score (IFS), a structural metric evaluating governing terms, boundary conditions, and coefficients within the MOOSE framework. They developed a PDE-grounded refinement loop that uses deterministic violation reports to iteratively correct code. Evaluated on MooseBench, a new 220-case multiphysics benchmark, their method significantly improved IFS, particularly in complex cases where direct generation failed. The study reveals that execution-only repairs leave nearly 40% of cases solving incorrect physics, highlighting executability and intent fidelity as distinct failure modes. Proof-of-concept experiments suggest this verification approach extends to other domain-specific languages like FEniCS and FreeFEM, emphasizing the need for mathematical structure verification over mere execution success in scientific computing.
Wire timeline
PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code
Researchers from arXiv have published a study addressing a critical flaw in using Large Language Models (LLMs) for scientific simulations: code that executes successfully may still solve the wrong physics. This mismatch, termed the 'comprehension-generation gap,' occurs when generated input files converge but encode governing equations differing from user intent. The team introduces the Intent Fidelity Score (IFS), a structural metric evaluating governing terms, boundary conditions, and coefficients within the MOOSE framework. They developed a PDE-grounded refinement loop that uses deterministic violation reports to iteratively correct code. Evaluated on MooseBench, a new 220-case multiphysics benchmark, their method significantly improved IFS, particularly in complex cases where direct generation failed. The study reveals that execution-only repairs leave nearly 40% of cases solving incorrect physics, highlighting executability and intent fidelity as distinct failure modes. Proof-of-concept experiments suggest this verification approach extends to other domain-specific languages like FEniCS and FreeFEM, emphasizing the need for mathematical structure verification over mere execution success in scientific computing.
cs.AI updates on arXiv.org