Hallucination in Multimodal Models is the first comprehensive research monograph dedicated to the growing challenge of hallucinations in large-scale multimodal AI systems, particularly vision-language models (VLMs) and multimodal large language models (MLLMs). The book systematically defines, categorizes, evaluates, and mitigates hallucinations — cases where models generate content that is factually inconsistent, visually unsupported, or commonsensically implausible. These hallucinations have become increasingly problematic in real-world applications of AI, including robotics, autonomous systems, and AI-generated media, where the consequences of inaccurate outputs can be severe.

The purpose of this book is threefold:

(1) to formalize the types and causes of hallucination in multimodal models;

(2) to present state-of-the-art evaluation frameworks, such as FaithScore and FIHA, for quantifying hallucination at a fine-grained level; and

(3) to introduce a unified mitigation framework.

The book presents recent research results, including FaithScore, FIHA, FIFA, FGAIF, and Dentist, together with empirical studies across leading large vision-language models. Of particular interest are its fine-grained approaches to atomic fact verification, semantic dependency modeling, unified text-video hallucination evaluation, reward-based alignment, and training-free hallucination mitigation. The book also examines how these methods can improve the faithfulness and reliability of multimodal model outputs, offering practical tools for both researchers and engineers.

This book complements and extends the existing literature on multimodal model evaluation (e.g., MME, SEED-Bench, LAMM) by moving beyond surface-level metrics to offer deeper, interpretable, and automated hallucination analysis. Unlike survey papers or benchmarks that only diagnose the problem, our monograph provides a cohesive solution path from diagnosis to mitigation, built upon novel technical contributions and real-world implementations.

As this is the first edition, it introduces original theoretical frameworks, algorithms, benchmarks, and design paradigms. It is intended to serve as a reference for graduate students, academic researchers, and industry practitioners working in natural language processing, computer vision, embodied AI, and trustworthy AI systems.

Liqiang Jing is a PhD candidate in the Computer Science Department at the University of Texas at Dallas. He was a research intern at Alibaba Damo Acadamy and Tencent AI Lab (Seattle). His research interests include multimodal learning and natural language processing. He has published over 15 papers in leading NLP and AI conferences, such as ICLR, ACL, EMNLP, ACM SIGIR, ACM MM, AAAI, etc. In addition, he has served as a reviewer for many top conferences and journals, such as NeurIPS, ICLR, ICML, and EMNLP.

Yue Zhang is s a PhD candidate in the Computer Science Department at the University of Texas at Dallas. Her research interests include multimodal learning and natural language processing.

Xinya Du is a tenure-track Assistant Professor at the University of Texas at Dallas, Computer Science Department. He earned a Ph.D. degree from Cornell University and was a Postdoctoral Research Associate at UIUC. His research is on NLP, Knowledge and Reasoning, and LLMs. The goal is to build intelligent machines with both faithful knowledge \& reasoning capabilities. His work has been published in leading NLP and ML conferences (ACL, EMNLP, ICLR). His work was included in the list of Most Influential ACL Papers by Paper Digest and has been covered by major media like New Scientist. He was named a Spotlight Rising Star in Data Science and was selected for the New Faculty Highlights program by AAAI. He is the recipient of the 2024 Amazon Research Award and the 2024 NSF CAREER Award