Learning to Generate Grounded Visual Captions without Localization Supervision

Ma, Chih-Yao; Kalantidis, Yannis; AlRegib, Ghassan; Vajda, Peter; Rohrbach, Marcus; Kira, Zsolt; Ma, Chih-Yao; Wu, Zuxuan; AlRegib, Ghassan; Xiong, Caiming; Kira, Zsolt; Ma, Chih-Yao; Lu, Jiasen; Wu, Zuxuan; AlRegib, Ghassan; Kira, Zsolt; Socher, Richard; Xiong, Caiming; Ma, Chih-Yao; Hang, Hsueh-Ming

Kevin Chih-Yao Ma

Grounded Visual Captions without Localization Supervision

Mon, 03 Jun 2019 00:00:00 +0000

Learning to Generate Grounded Visual Captions without Localization Supervision

Chih-Yao Ma, Yannis Kalantidis, Ghassan AlRegib, Peter Vajda, Marcus Rohrbach, Zsolt Kira
European Conference on Computer Vision (ECCV), 2020

[arXiv] [GitHub] [ML@GT]

10-min Video

Abstract

When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, that is whether the model uses the correct image regions to output particular words, or if the model is hallucinating based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference, for both image and video captioning tasks.

Proposed Concept

Qualitative Results

We conduct qualitative analysis for comparing the baseline (Unsup.) and the proposed method in the figure below. Each highlighted word has a corresponding image region annotated on the original image. The image regions are selected based on the region with the maximum attention weight. We can see that our proposed method significantly outperformed the baseline (Unsup.) in terms of both the quality of the generated sentence and grounding accuracy.

In addition, we also show a number of correct and incorrect examples of our proposed method in the figure below.

Captioning and Grounding Performance on Flickr30k-Entities

We first evaluate the proposed cyclical training regimen on the Flickr30k dataset for image captioning task. To understand how our proposed method performs on captioning as well as visual grounding, we compare with the proposed strong baseline with or without grounding supervision. We train the attention mechanism (Attn.) of the baseline method as well as adding the region classification task (Cls.) using the ground-truth grounding annotation. Using the resultant supervised baseline model as the upper bound, our proposed method with cyclical training achieves relative 20% to 15% grounding accuracy improvements for F1_all and F1_loc respectively and achieves around 12% improvements for F1_all_per_sent and F1_loc_per_sent, while maintaining the captioning evaluations performances.

Table 1: Performance comparison on the Flickr30k Entities test set. *: our results are averaged across five runs.

We also evaluate the proposed method on the ActivityNet-Entities dataset for video captioning task. Similarly, our proposed method significantly improve grounding accuracy while maintaining the captioning evaluations performances.

Table 2: Performance comparison on the Activity-Entities val set. *: our results are averaged across five runs.

Code and Paper

GitHub

arXiv

Citation

If you find this work useful, please cite our paper:


@inproceedings{ma2020learning,
    title={Learning to Generate Grounded Visual Captions without Localization Supervision},
    author={Ma, Chih-Yao and Kalantidis, Yannis and AlRegib, Ghassan and Vajda, Peter and Rohrbach, Marcus and Kira, Zsolt},
    booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2020},
    url={https://arxiv.org/abs/1906.00283},
}

The Regretful Navigation Agent for Vision-and-Language Navigation

Mon, 25 Feb 2019 00:00:00 +0000

Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, Zsolt Kira
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Oral)

[arXiv] [GitHub] [Poster]

Demo

Abstract

As deep learning continues to make progress for challenging perception tasks, there is increased interest in combining vision, language, and decision-making. Specifically, the Vision and Language Navigation (VLN) task involves navigating to a goal purely from language instructions and visual information without explicit knowledge of the goal. Recent successful approaches have made in-roads in achieving good success rates for this task but rely on beam search, which thoroughly explores a large number of trajectories and is unrealistic for applications such as robotics. In this paper, inspired by the intuition of viewing the problem as search on a navigation graph, we propose to use a progress monitor developed in prior work as a learnable heuristic for search. We then propose two modules incorporated into an end-to-end architecture: 1) A learned mechanism to perform backtracking, which decides whether to continue moving forward or roll back to a previous state (Regret Module) and 2) A mechanism to help the agent decide which direction to go next by showing directions that are visited and their associated progress estimate (Progress Marker). Combined, the proposed approach significantly outperforms current state-of-the-art methods using greedy action selection, with 5% absolute improvement on the test server in success rates, and more importantly 8% on success rates normalized by the path length.

Proposed Concept

Qualitative Results

The below figure shows the qualitative outputs of our model during successful navigation in unseen environments. In the example at the left side, the agent made a mistake at the first step, and the estimated progress at the second step slightly decreases. The agent then decides to rollback, after which the progress monitor significantly increases. Finally, the agent stopped correctly as instructed. At the middle, we show an example where the agent correctly goes up the stairs but incorrectly does it again rather than turning and finding the TV as instructed. Note that the progress monitor increases but only by a small amount; this demonstrates the need for learned mechanisms that can reason about the textual and visual grounding and context, as well as the resulting level of change in progress. In this case the agent then correctly decides to rollback and successfully walked into the TV room. At the right side, the agent misses the stairs, resulting in a very small progress increase. The agent decides to rollback as a result. Upon reaching the goal, the agent’s progress estimate is 99%. Please refer to the Appendix for the full trajectories and unsuccessful examples.

Navigation Performance on Room-to-Room

We evaluate our proposed Regretful Agent on the Room-to-Room dataset for Vision-and-Language Navigation task. our method achieves significant performance improvement over the existing approaches. We achieved 37% SPL and 48% SR on the validation unseen set and outperformed all existing work. Our best performing model achieves 41% SPL and 50% SR on validation unseen set when trained with the synthetic data. We demonstrate absolute 8% SPL improvement and 5% SR improvement on the test server over the current state-of-the-art method.

Code and Paper

GitHub

arXiv

Citation

If you find this work useful, please cite our paper:


@inproceedings{ma2019theregretful,
    title={The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation},
    author={Ma, Chih-Yao and Wu, Zuxuan and AlRegib, Ghassan and Xiong, Caiming and Kira, Zsolt},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2019},
    url={https://arxiv.org/abs/1903.01602},
}

Self-Monitoring Agent for Vision-and-Language Navigation

Thu, 27 Sep 2018 00:00:00 +0000

Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, Caiming Xiong

International Conference on Learning Representations (ICLR), 2019
(Top 7% of reviews)
[arXiv] [OpenReview] [GitHub] [Poster] [ML@GT]

Abstract

The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our self- monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set).

Proposed Concept

Qualitative Results

We qualitatively show how the agent navigates through unseen environments by following instructions as shown in the figure below. In each figure, the agent follows the grounded instruction (at the top of the figure) and decides to move towards a certain direction (green arrow). For the full figures and more examples of successful and failed agents in both unseen and seen environments, please see the supplementary material in our paper.

Navigation Performance on Room-to-Room

Our method achieves significant performance improvement compared to the state of the arts without data augmentation. We achieve 70% SR on the seen environment and 57% on the unseen environment while the existing best performing method achieved 63% and 50% SR respectively. When trained with synthetic data, our approach achieves slightly better performance on the seen environments and significantly better performance on both the validation unseen environments and the test unseen environments when submitted to the test server. We achieve 3% and 8% improvement on SR on both validation and test unseen environments. Both results with or without data augmentation indicate that our proposed approach is more generalizable to unseen environments.

Code and Paper

GitHub

arXiv

Citation

If you find this work useful, please cite our paper:


@inproceedings{ma2019selfmonitoring,
    title={Self-Monitoring Navigation Agent via Auxiliary Progress Estimation},
    author={Ma, Chih-Yao and Lu, Jiasen and Wu, Zuxuan and AlRegib, Ghassan and Kira, Zsolt and Socher, Richard and Xiong, Caiming},
    booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},
    year={2019},
    url={https://arxiv.org/abs/1901.03035},
}

Object Interactions for Fine-grained Video Understanding

Thu, 16 Nov 2017 00:00:00 +0000

Attend and Interact: Higher-Order ObjectInteractions for Video Understanding

Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan AlRegib, Hans Peter Graf
Computer Vision and Pattern Recognition (CVPR), 2018
[arXiv] [ML@GT Blog] [Poster]

Abstract

Human actions often involve complex interactions across several inter-related objects in the scene. However, existing approaches to fine-grained video understanding or visual relationship detection often rely on single object representation or pairwise object relationships.

In this paper, we propose to efficiently learn higher-order interactions between arbitrary subgroups of objects for fine-grained video understanding. We demonstrate the impact of modeling object interactions towards significantly improving accuracy for both action recognition and video captioning.

To the best of our knowledge, this is the first work modeling object interactions on open domain large-scale video datasets.

Fine-grained Human Activity Recognition

Relationship Grounded Video Captioning

Citation

If you find this work useful, please cite our paper:


@inproceedings{ma2018attend,
  title={Attend and Interact: Higher-Order Object Interactions for Video Understanding},
  author={Ma, Chih-Yao and Kadav, Asim and Melvin, Iain and Kira, Zsolt and AlRegib, Ghassan and Graf, Hans Peter},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018}
}

Activity Recognition with RNN and Temporal-ConvNet

Thu, 30 Mar 2017 00:00:00 +0000

TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

Chih-Yao Ma*, Min-Hung Chen*, Zsolt Kira, and Ghassan AlRegib
Signal Processing: Image Communication, 2018
[arXiv] [GitHub]
(* equal contribution)

Abstract

In this work, we demonstrate a strong baseline two-stream ConvNet using ResNet-101. We use this baseline to thoroughly examine the use of both RNNs and Temporal-ConvNets for extracting spatiotemporal information. Building upon our experimental results, we then propose and investigate two different networks to further integrate spatiotemporal information: 1) temporal segment RNN and 2) Inception-style Temporal-ConvNet.

Our analysis identifies specific limitations for each method that could form the basis of future work. Our experimental results on UCF101 and HMDB51 datasets achieve state-of-the-art performances, 94.1% and 69.0%, respectively, without requiring extensive temporal augmentation.

How we tackle Activity Recognition problem?

Demo

The GIFs demonstrate the top-3 predictions results of our TS-LSTM and Temporal-Inception methods. The text on the top is the ground truth, three texts are the predictions for each of the method, and the bar right next to the predictions are how confident the model makes predictions.

Code and Paper

GitHub

arXiv

Citation

If you find this work useful, please cite our paper:


@article{ma2018ts,
  title={TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition},
  author={Ma, Chih-Yao and Chen, Min-Hung and Kira, Zsolt and AlRegib, Ghassan},
  journal={Signal Processing: Image Communication},
  year={2018},
  publisher={Elsevier}
}

NCTU-3DFixation Dataset

Fri, 01 May 2015 00:00:00 +0000

NCTU-3DFixation Dataset

Chih-Yao Ma, Hsueh-Ming Hang
Journal of Vision, 2015

[Paper]

Abstract

The eye fixation data is essential for studying the way humans understand the image contents. Up to now, the public-domain 3D fixation data are scarce. Therefore, we conduct our eye fixation experiments on 3D images. The resultant dataset should be useful to the 3D visual attention research community. Our eye fixation dataset comprises 475 3D images and 16 subjects. We use a Tobii TX300 eye-tracker to track the eye movement of each subject. In addition, this database contains 475 computed depth maps. The database and the MATLAB source codes for evaluation and visualization are released on this webpage.

Proposed Concept

Download

All image contents provided in the NCTU-3DFixation Database are solely intended for private and research purposes only and copyrighted by their respective owners unless otherwise stated.

I have compressed all the files together to create one zip file which is currently PASSWARD protected.

NCTU-3DFixation.zip (4.7GB)

Password: c-o-m-m-l-a-b- (without "-")

If the above link doesn’t work or somehow gives you low download speed, please refer to this Google Drive link for downloading our dataset.

This compressed file includes:

475 3D images
Generated Depth maps
Gaze data from 16 subjects
MATLAB Codes for visualization and evaluation

If you have any questions regards to our database, please don’t hesitate to contact me.

Citation

If you find this work useful, please cite our paper:


@article{ma2015learning,
    title={Learning-based saliency model with depth information},
    author={Ma, Chih-Yao and Hang, Hsueh-Ming},
    journal={Journal of vision},
    volume={15},
    number={6},
    pages={19--19},
    year={2015},
    publisher={The Association for Research in Vision and Ophthalmology}
}