Coupling LLMs with KGs: Benchmarking LLM Models and Structured Reasoning Frameworks with a Classic Botanical Field Guide

Abstract

Large language models remain susceptible to hallucination and stochastic outputs when reasoning over structured domains. This thesis investigates whether coupling vision-language models with a typed knowledge graph can produce verifiable botanical identification, and whether ensemble disagreement across model runs can serve as a scalable signal for detecting prediction uncertainty without human annotation. The experimental platform is Newcomb’s Wildflower Guide, a hierarchical botan- ical key, whose feature-value pairs are encoded as a knowledge graph and paired with a corpus of expression-unlabeled field photographs from iNaturalist. A two-stage mapping pipeline, first an existence check followed by a blind multiple-choice clas- sification, is applied across six vision-language models. A human-curated reference set establishes perceptual alignment between models and the Newcomb vocabulary before evaluation on unlabeled images. The central finding is that ensemble disagreement is a reliable, annotation-free sig- nal for image-level uncertainty. Critically, same-model repetition for the most stable model produces sparser but more precise disagreement than architecturally diverse ensembles: when a stable model disagrees with itself on a specific observation, the disagreement is more concentrated on true errors, whereas cross-model disagreement conflates image-level ambiguity with model architectural bias. A further finding is that unanimous disagreement between models and the Newcomb key can surface candidate annotation errors at scale without any additional human effort. Together, these results establish a practical path for extending the pipeline to large corpora of expression-unlabeled images and for generalizing the methodology to domains with similarly structured knowledge.