News

Semantic segmentation can help people identify and locate objects, which provide important road information for upper-level navigational tasks. Due to the rapid development of deep Convolutional Neural Networks (CNNs)[1], the performance of image segmentation models has been greatly improved and CNNs is widely used for this task. However, maintaining the performance under different conditions is a non-trivial task. In the dark, rain, or fog environment, the quality of RGB images will be greatly reduced while other sensors may still get fair results. Thus, our model combines the information of RGB image and depth map. When driving, we often encounter obstacles, like the trash can, barrier, rubble, stones, and cargos. Recognizing and avoiding them is very crucial for safety. To address this problem, we apply multi-dataset learning. In this way, our model can learn more classes from other data sets, including obstacles.

In our experiment, we fully evaluate the RFNet[2] with different datasets and methods to combine them. Regarding the framework of our model, the inputs from different datasets will pass through the resizing module. Then, the depth map and RGB image are sent to the network. Ground truth will go to Relabeling module. Multi-dataset learning strategy is applied to Resizing, Relabeling and Modified Softmax layer. Finally, by comparing the results of relabeled ground truth and prediction, we can obtain the intersection of union(IoU) and value of cross-entropy loss.

Our result shows that our models have excellent performances in the urban environment and blended environment. However, in the field environment, the depth map helps the model very slightly. We also proposed a new thrifty relabeling, which can improve the performance of the model without increasing the complexity of the network. Moreover, more datasets can help the model [...]

Let us hear what she has to say about her defense and an abstract of her thesis.

Neural networks have been shown to be effective in many applications. To better explain the behaviors of the neural networks, we examine the properties of neural networks experimentally and analytically in this research.

In the first part, we conduct experiments on convolutional neural networks (CNNs) and observe the behaviors of the networks. We also propose some insight or conjectures for CNNs in this part. First, we demonstrate how the accuracy changes with the size of the convolutional layers. Second, we  develop a design to determine the size of the convolutional layers based on SSL. Third, as a case study, we analyze the SqueezeNet, which is able to achieve the same accuracy as AlexNet with 50x fewer parameters, by studying the evolution of cross-entropy values across layers and doing visualization. Fourth, we also propose some insight on  co-training based deep semi-supervised learning.

In the second part, we propose new angles to understand and interpret neural network. To understand the behaviors of multilayer perceptrons (MLPs) as classifiers,  we interpret MLPs as a generalization of a two-class LDA system so that it can handle an input composed by multiple Gaussian modalities belonging to multiple classes. An MLP design with two hidden layers that also specifies the filter weights is proposed. To understand the behaviors of multilayer perceptrons (MLPs) as a regressor we construct MLPs as a piecewise low-order polynomial approximator using a signal processing approach. The constructed MLP contains one input, one intermediate and one output layers. Its construction includes the specification of neuron numbers and all filter weights.  Through the construction, a one-to-one correspondence between the approximation of an MLP and that [...]

Let us hear what he has to say about his defense and an abstract of his thesis.

Deep learning has brought impressive improvements in many fields, thanks to end-to-end data-driven optimization. However, people have little control over the system during training and limited understanding about the structure of knowledge being learned. In this thesis, we study theory and applications of adversarial and structured knowledge learning: 1) learning adversarial knowledge with human interaction or by incorporating human-in-the-loop; 2) learning structured knowledge by modelling contexts and users’ preferences via distance metric learning.

In the first part, we teach a robotic arm to learn robust manipulation grasps that can withstand perturbations, through end-to-end optimization with a human adversary. Specifically, we formulate the problem as a two-player game with incomplete information, played by a human and a robot, where the human’s goal is to minimize the reward the robot can get. We then extend this idea to improve the sample efficiency of deep reinforcement learning by incorporating human in the training loop. We presented a portable, interactive and parallel platform for human-agent curriculum learning experience.

In the second part, we present two works that address different aspects of structured representation learning. First, we proposed a self-training framework to improve distance metric learning. The challenge is the noise in pseudo labels, which prevents exploiting additional unlabeled data. Therefore, we introduced a new feature basis learning component for the teacher-student network, which better measures pairwise similarity and selects high confidence pairs. Second, we address image-attribute query, which allows a user to customize image-to-image retrieval by designating desired attributes in target images. We achieve this by adopting a composition module, which enforces object-attribute factorization and an attribute-set synthesis module to deal with sample insufficiency.

Looking back, Prof. Kuo is [...]

Object tracking is a fundamental computer vision problem that finds a wide range of applications, such as video surveillance, smart traffic system, autonomous driving cars and so on. Nowadays, most state-of-the-art object trackers adopt deep neural networks for high tracking performance at the expense of huge computational resources and heavy memory use. Here, we seek a more lightweight solution that requires fewer resources for training and inference and has a much smaller model size, thus making real-time tracking possible on small devices such as mobile phone and autonomous drones.

The proposed object tracker is built upon the PixelHop framework so that it is called OTHop (Object Tracking PixelHop). The term “hop” denotes the neighborhood of a pixel. OTHop conducts spectral analysis using Saab transform on neighborhoods of various sizes centered on a pixel through a sequence of cascaded dimension reduction units, which naturally forms a multi-resolution feature extraction scheme, thus helping capture unusual patterns that we should pay more attention to during tracking. Then we adopt the XGBoost classifier as the binary predictor to differentiate foreground pixels and background pixels. The classifier is pre-trained on some offline dataset and then updated online using either the initial frame or preceding frames with Saab coefficients as the input. Base on the classification results we derive the object bounding box.

To sum up, OTHop has the following main steps:

Extract joint spatial-spectral features based on the PixelHop framework;
Predict the probability of a spatial region, which can be of various sizes, of being a foreground object or a background region with a trained XGBoost binary classifier;
Fuse results obtained at different hops in Steps 2 to obtain the ultimate object bounding boxes.

The tracker is tested on the [...]

As the number of Deepfake video contents grows rapidly, automatic Deepfake detection has received a lot of attention in the community of digital forensics. Deepfake videos can be potentially harmful to society, from non-consensual explicit content creation to forged media by foreign adversaries used in disinformation campaigns.

A light-weight high-performance Deepfake detection method, called DefakeHop, is proposed in this work. State-of-the-art Deepfake detection methods are built upon deep neural networks. DefakeHop extracts features automatically using the successive subspace learning (SSL) principle from various parts of face images. DefakeHop consists of three main modules: 1) PixelHop++, 2) feature distillation and 3) ensemble classification. To derive the rich feature representation of faces, DefakeHop extracts features using PixelHop++ units from various parts of face images. The theory of PixelHop++ have been developed by Kuo et al. using SSL. PixelHop++ has been recently used for feature learning from low-resolution face images but, to the best of our knowledge, this is the first time that it is used for feature learning from patches extracted from high-resolution color face images. Since features extracted by PixelHop++ are still not concise enough for classification, we also propose an effective feature distillation module to further reduce the feature dimension and derive a more concise description of the face. Our feature distillation module uses spatial dimension reduction to remove spatial correlation in a face and a soft classifier to include semantic meaning for each channel. Using this module the feature dimension is significantly reduced and only the most important information is kept. With a small model size of 42,845 parameters, DefakeHop achieves state-of-the-art performance with the area under the ROC curve (AUC) of 100%, 94.95%, and 90.56% on UADFV, Celeb-DF v1 and Celeb-DF v2 datasets, [...]