Evaluating the Adequacy of Competency Question LLM-Generated Ontologies
KGC 2026
•
27m
https://drive.google.com/file/d/1dXvbNAZw9mdxcL74IRM705kESz4f-xpL/view?usp=share_link
We present an extensible framework specifically designed for the Competency Question-to-Ontology generation, which evaluates a growing set of models from major providers on their ability to generate formal ontologies from natural language requirements and associated SPARQL queries. Our framework systematically explores the high-dimensional space of LLM generation by testing combinations of three prompting strategies, zero-shot, few-shot, and CoT, across three levels of prompt expressivity, while a dedicated sensitivity analysis module evaluates the impact of parameters such as temperature, top_p, and max_tokens. Each generated ontology undergoes a multi-layered validation pipeline that includes syntax parsing, logical consistency checks and comprehensive structural matching against a set of reference ontologies. Beyond structural fidelity, the framework measures functional coverage by executing SPARQL queries to verify requirement fulfillment and assesses quality through automated checks of URI formatting, naming conventions, and documentation standards. By automating the entire lifecycle from prompt iteration to cost-tracking and metric aggregation, this framework provides the basis for a reproducible, data-driven foundation for integrating LLMs into production-grade ontology engineering workflows.
Up Next in KGC 2026
-
The Intelligence Multiplier
Hedge funds and trading firms face an insurmountable challenge: synthesizing insights from market data, SEC filings, earnings transcripts, expert interviews, and research reports at scale. This presentation demonstrates a production knowledge graph system that automatically bridges structured fin...
-
Grounding LLMs in Domain Knowledge
"https://drive.google.com/file/d/1oF6pLrXoVYKh_GRpyC-E4hmFRnzdOF9a/view?usp=share_link
Qualification processes in industrial settings require accurate, equipment-specific inspection criteria from technical documentation to ensure deliverable quality and minimize post-launch downtime and claims. ...
-
JIT-Resolution
Traditional Enterprise Knowledge Graphs rely on upfront Entity Resolution (ERRP) that strips away nuance. These heavy build-time commitments create brittle pipelines and discard the contextual signals necessary for handling unstructured data like SEC filings or risk disclosures.
Methodology JIT-...
