News

Congratulations to Junting Zhang for Passing His Qualifying Exam on 01/17/19. Her thesis proposal is titled with “IMAGE KNOWLEDGE TRANSFER WITH DEEP LEARNING TECHNIQUES”. Her Qualifying Exam committee includes: Jay Kuo (Chair), Sandy Sawchuk, Keith Jenkins, Panos Georgiou and Ulrich Neumann (Outside Member).

Abstract of thesis proposal:

In recent years, we have witnessed tremendous success in training deep neural networks to learn a surprisingly accurate mapping from input signals to outputs, whether they are images, languages, genetic sequences, etc. from large amounts of labeled data. One fatal characteristics of the current dominant learning paradigm is that it learns in isolation: given a carefully constructed training dataset, it runs a machine learning algorithm on the dataset to produce a model that is then used in its specific intended application. It has no intention to exploit the dependencies and relations among different tasks and domains, nor the effective techniques to retain, accumulate, and transfer knowledge gained from past learning experiences to solve new problems in the new scenarios.

The learning environments are typically static and strictly constrained. For supervised learning, labeling of training data is often done manually, which is prohibitively expensive in terms of labor resource and time, especially when the required label is fine-grained or it requires knowledge from a domain expert. Considering the real world is too complex with infinite possible tasks, it is almost impossible to label sufficient number of examples for every possible task or application. Furthermore, the world also changes constantly, and appearance of instances or the label of the same instance may vary from time to time, the labeling thus needs to be done continually, which is a daunting task for humans.

On the contrary, humans learn in a different way, where transfer [...]

Congratulations to Yuhang Song for passing his Qualifying Exam on January 10, 2019! Yuhang’s thesis proposal is titled with “High-Quality Image Inpainting with Deep Generative Models”. His qualifying exam committee consisted of Jay Kuo (Chair), Antonio Ortega, Alexander Sawchuk, Panayiotis Georgiou, and Ulrich Neumann (Outside Member).

We invited Yuhang to talk about his thesis proposal:

Image inpainting is the task to reconstruct the missing region in an image with plausible contents based on its surrounding context, which is a common topic of low-level computer vision. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, However, existing methods are limited to fill in small holes on low-resolution images, and very often generate unsatisfying results containing easily detectable flaws. In this thesis proposal, we specifically study two image inpainting related problems: 1) finetuning the image generation textures; 2) making use of the semantic segmentation information for higher quality image inpainting.

In order to overcome the difficulty to directly learn the distribution of high-dimensional image data, we divide the task into inference and translation as two separate steps and model each step with a deep neural network. We also use simple heuristics to guide the propagation of local textures from the boundary to the hole. We show that, by using such techniques, inpainting reduces to the problem of learning two image-feature translation functions in much smaller space and hence easier to train. We evaluate our method on several public datasets and show that we generate results of better visual quality than previous state-of-the-art methods.

The second research idea is motivated by the fact that existing methods based on generative models don’t exploit the segmentation information to constrain the object shapes, which usually lead to blurry [...]

Word embedding has obtained its popularity among various NLP tasks including sentiment analysis [1], information retrieval [2] and machine translation [3]. The goal for word embedding is transferring word to vector representation which embeds both syntactic and semantic information. In the meantime, relationship between words can be distinguished though measurements of corresponding word vectors.

Although embedded vector representations of words offer impressive performance on many natural language processing (NLP) applications, the information of ordered input sequences is lost to some extent if only context-based samples are used in the training. For further performance improvement, two new post-processing techniques, called post-processing via variance normalization (PVN) and post-processing via dynamic embedding (PDE), are proposed in this work. The PVN method normalizes the variance of principal components of word vectors, while the PDE method learns orthogonal latent variables from ordered input sequences [4]. Our post-processing technique could improve the performance on both intrinsic evaluations tasks including word similarity, word analogy and outlier detection, and extrinsic evaluation tasks including sentiment analysis and machine translation.

In the meantime, we are also interested in word embedding evaluation tasks. It can be divided into two categories: intrinsic evaluation and extrinsic evaluation. We are trying to understand more on the properties of word embedding as well as their evaluation methods. It is still an on-going project.

Further developments include contextualized word embedding [5] and pre-trained language models [6] are quite popular last year. Lots of exciting work can be done along this direction and performance is much better than previous models. Also, bilingual or multi-lingual word embedding could also be an interesting research area.

–By Bin Wang, working with Fenxiao (Jessica) Chen, Angela Wang and Yunchen (Joe) Wang

Reference:

[1] Shin, B.; Lee, T.; and Choi, J. D. [...]

May your Christmas sparkle with moments of love, laughter and goodwill. And may the year ahead be full of contentment and joy. Wish all our fellows a Merry Christmas!

The task of Visual Dialogue involves a conversation between an agent system and a human end-user regarding visual information presented. The conversation consists of multiple rounds of questions and answers. The key challenge for the agent system is to answer the questions of human users with meaningful information while keeping the conversation flow contiguous and natural. The current visual dialogue systems can be divided into two tracks, generative models and discriminative models. The discriminative models cannot directly generate a response but choose a response out of a pool of candidate responses. Although the discriminative models have achieved surprising results, they are usually not applicable in real scenarios where no candidate response pool is available. On the other hand, the generative models can directly generate a response based on the input information. However, most generative models based on maximum likelihood estimation (MLE) approach suffer from the tendency of generating generic responses.

We present a novel approach that incorporates a multi-modal recurrently guided attention mechanism with a simple yet effective training scheme to generate high quality responses in the Visual Dialogue interaction model. Our attention mechanism combines attentions globally from multiple modalities (e.g., image, text questions and dialogue history), and refines them locally and simultaneously for each modality. Generators using typical MLE-based methods only learn from good answers, and consequently tend to generate safe or generic responses. The new training scheme with weighted likelihood estimation (WLE) penalizes generic responses for unpaired questions in the training and enables the generator to learn from poor answers as well.

On benchmark dataset, our proposed Visual Dialogue system demonstrates state-of-the-art performance with improvement of 5.81% and 5.28 on recall@10 and mean rank, respectively.

–By Heming Zhang