Lightweight visual question answering using scene graphs

Conference item

Visual question answering (VQA) is a challenging problem in machine perception, which requires a deep joint understanding of both visual and textual data. Recent research has advanced the automatic generation of high-quality scene graphs from images, while powerful yet elegant models like graph neural networks (GNNs) have shown great power in reasoning over graph-structured data. In this work, we propose to bridge the gap between scene graph generation and VQA by leveraging GNNs. In particular, we design a new model called Conditional Enhanced Graph ATtention network (CE-GAT) to encode pairs of visual and semantic scene graphs with both node and edge features, which is seamlessly integrated with a textual question encoder to generate answers through question-graph conditioning. Moreover, to alleviate the training difficulties of CE-GAT towards VQA, we enforce more useful inductive biases in the scene graphs through novel question-guided graph enriching and pruning. Finally, we evaluate the framework on one of the largest available VQA datasets (namely, GQA) with ground-truth scene graphs, achieving the accuracy of 77.87%, compared with the state of the art (namely, the neural state machine (NSM)), which gives 63.17%. Notably, by leveraging existing scene graphs, our framework is much lighter compared with end-to-end VQA methods (e.g., about 95.3% less parameters than a typical NSM).

Authors: Nuthalapati, SV; Chandradevan, R; Giunchiglia, E; Li, B; Kayser, M

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for Computing Machinery
• Host title: CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management
• Pages: 3353-3357
• Publication date: 2021-10-26
• Acceptance date: 2021-08-10
• Event title: 30th ACM International Conference on Information and Knowledge Management (CIKM 2021)
• Event location: Virtual Event
• Event website: https://www.cikm2021.org/
• Event start date: 2021-11-01
• Event end date: 2021-11-05
• DOI: 10.1145/3459637.3482218
• ISBN: 9781450384469
• Language: English
• Keywords: FFR