Neuralsymbolic Visual Understanding and Reasoning using Deep Learning and KGs
Deep Learning for and with Knowledge Graphs Track
•
25m
Visual AI has made incredible progress in basic vision tasks using deep learning techniques that can detect concepts in visual scenes accurately and quickly. However, the existing techniques rely on labelled datasets that lack common sense knowledge about visual concepts and have biased distribution of visual semantic relationships. As a result, these techniques have limited visual relationship prediction performance, limiting the expressiveness and accuracy of semantic representation and downstream reasoning. We employed deep neural networks to predict visual concepts, including objects and visual relationships, and linked them to generate symbolic image representation. To alleviate the challenges above, we leveraged rich and diverse common sense knowledge in heterogenous knowledge graphs to systematically refine and enrich the generated image representation. As a result, we observed significant improvement in recall rates of visual relationship prediction (7% increase in Recall@100), expressiveness of the representation, and the performance of downstream visual reasoning tasks, including image captioning (15% increase in SPICE score) and image reconstruction. The encouraging results depict the effectiveness of the proposed approach and the impact on downstream visual reasoning.
Up Next in Deep Learning for and with Knowledge Graphs Track
-
Knowledge Graph Completion using Embe...
Knowledge Graphs (KGs) are often generated automatically or manually which lead to KGs being in complete. Recent years have witnessed many studies on link prediction using KG embeddings which is one of the mainstream tasks in KG completion. Most of the existing methods learn the latent representa...
-
Efficient Knowledge Graph Constructio...
We aim to bring interested researchers uKnowledge graph construction which aims to extract knowledge from the text corpus, has appealed to researchers. Previous decades have witnessed the remarkable progress of knowledge graph construction on the basis of neural models; however, those models ofte...
-
Learning Concept Embeddings with a Tr...
We present a novel approach for learning embeddings of concepts from knowledge bases expressed in the ALC description logic. They reflect the semantics in such a way that it is possible to compute an embedding of a complex concept from the embeddings of its parts by using appropriate neural const...
