weiwang

MCL Research on CNN Ensembles

CNN technology provides state-of-the-art solutions to many image processing and computer vision problems. Given a CNN architecture, all of its parameters are determined by the stochastic gradient descent (SGD) algorithm through backpropagation (BP). The BP training demands a high computational cost. Furthermore, most CNN publications are application-oriented. There is a limited amount of progress after the classical result in [1]. Examples include explainable CNNs [2,3,4] and feedforward designs without backpropagation [5,6].

The determination of CNN model parameters in the one-pass feedforward (FF) manner was recently proposed by Kuo et al. in [6]. It derives network parameters of a target layer based on statistics of output data from its previous layer. No BP is used at all. This feedforward design provides valuable insights into the CNN operational mechanism. Besides, under the same network architecture, its training complexity is significantly lower than that of the BP design CNN. FF-designed and BP-designed CNNs are denoted by FF-CNNs and BP-CNNs, respectively.

We focus on solving the image classification problem based on the feedforward-designed convolutional neural networks (FF-CNNs) [6]. An ensemble method that fuses the output decision vectors of multiple FF-CNNs to solve the image classification problem is proposed. To enhance the performance of the ensemble system, it is critical to increasing the diversity of FF-CNN models. To achieve this objective, we introduce diversities by adopting three strategies: 1) different parameter settings in convolutional layers, 2) flexible feature subsets fed into the Fully-connected (FC) layers, and 3) multiple image embeddings of the same input source. Furthermore, we partition input samples into easy and hard ones based on their decision confidence scores. As a result, we can develop a new ensemble system tailored to hard samples to further boost classification accuracy.  Although [...]

By |February 26th, 2019|News|Comments Off on MCL Research on CNN Ensembles|

Welcome New MCL Member Haoxuan You

Welcome New MCL Member Haoxuan You! Here is an short interview with Haoxuan:

1. Could you briefly introduce yourself and your research interests?

My name is Haoxuan You. I received my bachelor’s degree at Xidian University in China last July and spent my gap year at Tsinghua University as a research assistant. Now I am having my six-month visiting at USC. Generally speaking, my research interest lies in computer vision and machine learning. I hope to model the 3D environment by effectively 3D data processing.

2. What is your impression about MCL and USC?

The first impression of USC to me is the sunshine and beautiful campus. Then I found MCL a warm and big family with very kind people. The seminar lunch, the group meeting all give me a good opportunity to get involved in and help me to broaden my horizon. I feel really lucky to be with these lovely friends.

3. What is your future expectation and plan in MCL?

I hope to have a better understanding of how to be a good researcher and independent thinker from Prof. Kuo and other MCL members. And I wish to do some more valuable work such as the interpretable network in 3D data processing and 3D object detection.

By |February 16th, 2019|News|Comments Off on Welcome New MCL Member Haoxuan You|

Welcome New MCL Member Prof. Yongfei Zhang

Welcome new MCL member Prof. Yongfei Zhang! Prof. Zhang now is a visiting scholar at Media Communications Lab (MCL) at University of Southern California (USC) under the supervision of Prof. C.-C. Jay Kuo. He is an Associate Professor with the Beijing Key Laboratory of Digital Media, School of Computer Science and Engineering, and the State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. He was a visiting scholar at University of Missouri, Columbia and Sept. 2007 to Sept. 2009.

Here is a short interview with Prof. Zhang:

1. Could you briefly introduce yourself and your research interests?

I am Yongfei Zhang, an Associate Professor in Beihang University. I received my B.S. degree in Electrical Engineering and Ph.D. degree in Pattern Recognition and Intelligent Systems, both from Beihang University, Beijing, China, in 2005 and 2011 respectively. My current research interests include Real-time High Efficiency Video Coding/Decoding, Computer Vision (Image Retrieval, Vehicle/Person Re-ID), Construction & Referencing of Domain-Specific Knowledge Graph.

2. What is your impression about MCL and USC?

MCL is a large and warm group with great director and students. Group routine activities, like individual meetings and weekly group seminars, are clearly scheduled and well organized with the contributions of each MCL member, which make the research and work very efficient. Besides, Prof. Kuo’s passion for the academic research and group culture development impress me a lot.

3. What is your future expectation and plan in MCL?

Great thanks to Prof. Kuo for hosting me in the big family of MCL. I wish I could enhance myself, with the help and guidance of Prof. Kuo, on teaching, research, and group management. Also, I’d like to make friends with all MCL members. Academically, I wish I could proceed/cooperate on [...]

By |February 9th, 2019|News|Comments Off on Welcome New MCL Member Prof. Yongfei Zhang|
  • Permalink Gallery

    Congratulations to Yueru Chen for passing her Qualifying Exam

Congratulations to Yueru Chen for passing her Qualifying Exam

Congratulations to Yueru Chen for passing her Qualifying Exam on Jan. 23, 2019! The title of her Ph.D. thesis proposal is “OBJECT CLASSIFICATION BASED ON NEURAL-NETWORK-INSPIRED IMAGE TRANSFORMS”. Her Qualifying Exam Committee includes: Jay Kuo (Chair), Antonio Ortega, Shri Narayanan, Keith Chugg and Ulrich Neumann (Outside Member).

Abstract of thesis proposal:

Convolutional neural networks (CNNs) have recently demonstrated impressive performance in image classification and change the way building feature extractors from carefully handcrafted design to automatically deep learned from a large labeled dataset. However, a great majority of current CNN literature are application-oriented, and there is no clear understanding and theoretical foundation to explain the outstanding performance and indicate the way to improve. In this thesis proposal, we focus on solving the image classification problem based on the neural-network-inspired Saak (Subspace approximation with augmented kernels) transform and Saab (Subspace approximation with adjusted bias) transform.

Based on the lossy Saak transform, we firstly proposed an efficient, scalable and robust approach to the handwritten digits recognition problem. We conduct a comparative study on the performance of the LeNet-5 and the Saak-transform-based solutions in terms of scalability and robustness as well as the efficiency of lossless and lossy Saak transform under a comparable accuracy level. We also develop an ensemble method that fuses the output decision vectors of Saab-transform-based decision system (i.e. the FF-CNN model) to solve the image classification problem. To enhance the performance of the ensemble system, it is critical to increasing the diversity of FF-CNN models. To achieve this objective, we introduce diversities by adopting three strategies: 1) different parameter settings in convolutional layers, 2) flexible feature subsets fed into the Fully-connected (FC) layers, and 3) multiple image embeddings of the same input source. Furthermore, we partition [...]

By |February 5th, 2019|News|Comments Off on Congratulations to Yueru Chen for passing her Qualifying Exam|
  • Permalink Gallery

    Congratulations to Heming Zhang for Passing Her Qualifying Exam

Congratulations to Heming Zhang for Passing Her Qualifying Exam

Congratulations to Heming Zhang for Passing Her Qualifying Exam on Jan. 24, 2019! Her thesis proposal is titled with “LOCAL-AWARE DEEP LEARNING: METHODOLOGY AND APPLICATIONS”. Her Qualifying Exam Committee includes: Jay Kuo (Chair), Antonio Ortega, Keith Jenkins, Sandy Sawchuk and Stefanos Nikolaidis (Outside Member).

Abstract of thesis proposal:

Deep learning techniques utilize networks with multiple layers cascaded to map the inputs to desired outputs. To map the entire inputs to desired outputs, useful information should be extracted through the layers. During the mapping, feature extraction and prediction are jointly performed. We do not have direct control for feature extraction. Consequently, some useful information, especially local information, is also discarded in the process.

In this thesis proposal, we specifically study local-aware deep learning techniques
with: 1) multi-modal attention mechanism; 2) local cues reasoning; 3) local region
characteristics analysis. Specifically, we design a multi-modal attention mechanism for generative visual dialogue system, which simultaneously attends to multi-modal inputs and utilizes extracted local information to generate dialogue responses. We propose a proposal network for fast face detection system for mobile devices, which detects salient facial parts and
uses them as local cues for detection of entire faces. We extract representative fashion features by analyzing local regions, which contain local fashion details of humans’ interests.

By |February 4th, 2019|News|Comments Off on Congratulations to Heming Zhang for Passing Her Qualifying Exam|
  • Permalink Gallery

    Congratulations to Fenxiao (Jessica) Chen for Passing Her Qualifying Exam

Congratulations to Fenxiao (Jessica) Chen for Passing Her Qualifying Exam

Congratulations to Fenxiao (Jessica) Chen for Passing His Qualifying Exam on 01/22/19! Her thesis proposal is titled with “GRAPH EMBEDDING WITH DEEP LEARNING TECHNIQUES”. Her Qualifying Exam Committee include: Jay Kuo (Chair), Sandy Sawchuk, Panos Georgiou, Viktor Prasanna and Xiong Ren (Outside Member).

Abstract of thesis proposal:

Graph representation learning is an important task nowadays due to the fact that most real-world data naturally comes in the form of graphs in many applications. Graph data often come in high-dimensional irregular form which makes them more difficult to analyze than the traditional low-dimensional data. Graph embedding has been widely used to convert graph data into a lower dimensional space while preserving the intrinsic properties of the original data.

In this thesis proposal, we specifically study two graph embedding problems: 1) Developing effective and graph embedding techniques can provide researcher with deeper understanding of the collected data more efficiently; 2) Use the embedded information to conduct applications such as node classification and link prediction.

To find an efficient way to learn and encode graph into a low dimensional embedding. We first present a novel Deepwalk-assisted Graph PCA (DGPCA) method is proposed for processing language network data represented by graphs. This method can generate a precise text representation for nodes (or vertices) in language networks. Unlike other existing work, our learned low dimensional vector representations add flexibility in exploring vertices neighborhood information while reducing noise contained in the original data. To demonstrate the effectiveness, we use DGPCA to classify vertices that contain text information in three language networks. Experimentally, DGPCA is shown to perform well on the language datasets in comparison to several state-of-the-art benchmarking methods.

To solve the node prediction problem, we present A novel graph-to-tree conversion mechanism called the deep [...]

By |February 4th, 2019|News|Comments Off on Congratulations to Fenxiao (Jessica) Chen for Passing Her Qualifying Exam|
  • Permalink Gallery

    Congratulations to Junting Zhang for Passing Her Qualifying Exam

Congratulations to Junting Zhang for Passing Her Qualifying Exam

Congratulations to Junting Zhang for Passing His Qualifying Exam on 01/17/19. Her thesis proposal is titled with “IMAGE KNOWLEDGE TRANSFER WITH DEEP LEARNING TECHNIQUES”. Her Qualifying Exam committee includes: Jay Kuo (Chair), Sandy Sawchuk, Keith Jenkins, Panos Georgiou and Ulrich Neumann (Outside Member).

Abstract of thesis proposal:

In recent years, we have witnessed tremendous success in training deep neural networks to learn a surprisingly accurate mapping from input signals to outputs, whether they are images, languages, genetic sequences, etc. from large amounts of labeled data. One fatal characteristics of the current dominant learning paradigm is that it learns in isolation: given a carefully constructed training dataset, it runs a machine learning algorithm on the dataset to produce a model that is then used in its specific intended application. It has no intention to exploit the dependencies and relations among different tasks and domains, nor the effective techniques to retain, accumulate, and transfer knowledge gained from past learning experiences to solve new problems in the new scenarios.

The learning environments are typically static and strictly constrained. For supervised learning, labeling of training data is often done manually, which is prohibitively expensive in terms of labor resource and time, especially when the required label is fine-grained or it requires knowledge from a domain expert. Considering the real world is too complex with infinite possible tasks, it is almost impossible to label sufficient number of examples for every possible task or application. Furthermore, the world also changes constantly, and appearance of instances or the label of the same instance may vary from time to time, the labeling thus needs to be done continually, which is a daunting task for humans.

On the contrary, humans learn in a different way, where transfer [...]

By |February 4th, 2019|News|Comments Off on Congratulations to Junting Zhang for Passing Her Qualifying Exam|

Congratulations to Ye Wang for Passing His Qualifying Exam

Congratulations to Ye Wang for Passing His Qualifying Exam on 01/16/2019! His thesis proposal is titled with “VIDEO OBJECT SEGMENTATION AND TRACKING WITH DEEP LEARNING TECHNIQUES”. His Qualifying Exam committee includes: Jay Kuo (Chair), Sandy Sawchuk, Antonio Ortega, Shri Narayanan, and Joseph Lim.

Abstract of thesis proposal:

Unsupervised video object segmentation is a crucial application in video analysis without knowing any prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a given video clip. In this thesis proposal, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatial-temporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and then selected by motion information as hard negatives if they exist and positives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced in case that there are no hard negatives in the sequence and experiments demonstrate both kinds of negatives (distractors) are complementary. Finally, we conduct DOA using the positive, negative, and hard negative masks to update the foreground/background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, DAVIS 2016 and FBMS-59 datasets.

In addition, this thesis proposal reports a visible and thermal drone monitoring system that integrates deep-learning-based detection and tracking modules. The biggest challenge in adopting deep learning methods for drone detection is the paucity of training drone images especially thermal drone images. To address this issue, we develop two data augmentation techniques. One is a model-based drone augmentation technique that automatically generates visible drone images with a bounding box [...]

By |January 28th, 2019|News|Comments Off on Congratulations to Ye Wang for Passing His Qualifying Exam|
  • Permalink Gallery

    Congratulations to Yuhang Song for Passing His Qualifying Exam

Congratulations to Yuhang Song for Passing His Qualifying Exam

Congratulations to Yuhang Song for passing his Qualifying Exam on January 10, 2019! Yuhang’s thesis proposal is titled with “High-Quality Image Inpainting with Deep Generative Models”. His qualifying exam committee consisted of Jay Kuo (Chair), Antonio Ortega, Alexander Sawchuk, Panayiotis Georgiou, and Ulrich Neumann (Outside Member).

We invited Yuhang to talk about his thesis proposal:

Image inpainting is the task to reconstruct the missing region in an image with plausible contents based on its surrounding context, which is a common topic of low-level computer vision. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, However, existing methods are limited to fill in small holes on low-resolution images, and very often generate unsatisfying results containing easily detectable flaws. In this thesis proposal, we specifically study two image inpainting related problems: 1) finetuning the image generation textures; 2) making use of the semantic segmentation information for higher quality image inpainting.

In order to overcome the difficulty to directly learn the distribution of high-dimensional image data, we divide the task into inference and translation as two separate steps and model each step with a deep neural network. We also use simple heuristics to guide the propagation of local textures from the boundary to the hole. We show that, by using such techniques, inpainting reduces to the problem of learning two image-feature translation functions in much smaller space and hence easier to train. We evaluate our method on several public datasets and show that we generate results of better visual quality than previous state-of-the-art methods.

The second research idea is motivated by the fact that existing methods based on generative models don’t exploit the segmentation information to constrain the object shapes, which usually lead to blurry [...]

By |January 21st, 2019|News|Comments Off on Congratulations to Yuhang Song for Passing His Qualifying Exam|

MCL Research on Graph Embedding

Research on graph representation learning has gained increasing attention among researchers because many speech/text data such social networks, linguistic (word co-occurrence) networks, biological networks and many other multi-media domain specific data can be well represented by graphs. Graph representation allows relational knowledge about interacting entities to be stored and accessed efficiently. Analyzing these graph data can provide significant insights into community detection, behavior analysis and many other useful applications for node classification, link prediction and clustering. To analyze the graph data, the first step is to find an accurate and efficient graph representation. The steps of graph embedding are shown in Figure 1. The input is a graph represented by an adjacency matrix. Graph representation learning aims to embed the matrix into a latent dimension that captures the intrinsic characteristics of the original graph. For each node u in the network, we embed it to a d dimensional space that represent the feature of that node, as shown in Figure 2.

Obtaining an accurate representation for the graph is challenging because of several factors. Finding the optimal dimension of the representation is not an easy task. Representation with higher number of dimensions might preserve more information of the original graph at the cost of more space and time. The choose of dimension can also be domain-specific and depends on the type of input graph. Choosing which property of the graph to embed is also challenging given the plethora of properties graphs have.

In our research, we first focus on node prediction task in deep learning models. Specifically, we explore node classification using tree-structured recursive neural networks. Then we switch our goal to improve the accuracy and efficiency of the deep-walk based matrix factorization method.

 

— By Fenxiao(Jessica) [...]

By |January 14th, 2019|News|Comments Off on MCL Research on Graph Embedding|