News

MCL Director, Professor C.-C. Jay Kuo, gave an opening keynote at the IEEE International Conference on Visual Communications and Image Processing (VCIP) on December 2, 2020. The meeting would originally be held from December 1-4, 2020, in Macau. However, due to the COVID-19 pandemic, it became a virtual one. The keynote is titled with “Interpretable and Effective Learning for 3D Point Cloud Registration, Classification and Segmentation.” Here is the abstract:

“3D point cloud analysis and processing find numerous applications in computer-aided design, 3D printing, autonomous driving, etc. Most state-of-the-art point cloud processing methods are based on convolutional neural networks (CNNs). Although they outperform traditional methods in terms of accuracy, they demand heavy supervision and higher training complexity. Besides, they lack mathematical transparency. In this talk, I will present three interpretable and effective machine learning methods for 3D point cloud registration, classification and segmentation, respectively. First, an unsupervised registration method that extracts salient points for matching is presented. Second, an unambiguous way to order points sequentially in a point cloud set is developed. Then, their spatial coordinates can be treated as geometric attributes of 1D data array. This idea facilitates the classification task. Third, for the segmentation task, we show how to leverage prior knowledge on point clouds to derive an intuitive and effective segmentation method. Extensive experiments are conducted to demonstrate the performance of the three new methods. I will also provide performance benchmarking between these interpretable methods and deep learning methods.”

The keynote was well attended with many questions during the 10-minute Q&A session. Professor Kuo’s keynote was sponsored by Tencent and called the Tencent Keynote Speech.

Knowledge graphs (KG) model human readable knowledge using entity and relation triples. One major branch of KG research is representation learning, in which we try to learn low dimensional embeddings for entity and relations. Simple arithmetic operations between embeddings of entities and relations can represent complex real world knowledge or even discover new ones. KGs are rapidly evolving with the enormous amount of new information generated everyday. Since it is infeasible to retrain KG embeddings whenever we encounter a new entity or relation, modeling unseen entities and relations remains a challenging task.

There are two main directions of research to handle unseen entities. One direction is to infer the embedding of new entities from its neighboring entities and relations that are observed during training. Researchers have either relied on Graph Neural Networks or designed specialized aggregation functions to collect the unseen nodes’ neighborhood information. The other path is to leverage feature information in entity nodes metadata. Specifically, entity name and descriptions are often available in textual format upon querying the KG. Recent advances in transformer language models have made it possible to extract high quality feature representation for contextual information after a minimal amount of fine tuning of the model. When transformer language models such as BERT are applied to extract entity representations, the model is capable of generating embedding for any entity with textual name or descriptions. As a result, the unseen entity problem is therefore resolved.

RotatE has been one of the most effective yet simple KG embedding models invented recently. In RotatE, entities and relations are models as complex vectors. Each element of the relation vector serves as an element-wise phase shifter that transforms source entity to target entity. We propose a specialized [...]

The visual tracking problem has a long history and has diverse applications in video surveillance, smart traffic system, autonomous driving cars and so on. Deep learning methods have gradually dominated the online single object tracking field because of the superior tracking accuracy. However, they usually require training on tremendous labeled videos which are expensive and time-consuming to acquire.

We proposed an explainable self-supervised salient-point-based approach to track general objects in real time, by utilizing attention and features from both the spatial domain and  the temporal domain. There are two major parts in our tracking system: tracking adjacent frames by matching salient points which represent spatial attention, and utilizing temporal information storing in salient points across different frames to identify loss of object or appearance change. In both parts, the salient point plays an important role in capturing spatial-temporal information. Here the feature of a salient point comes from the concatenation of two hop layer features in two-stage channel-wise Saab. The first hop contains PCA information of local patches at high resolution, while the second hop works at a lower resolution with larger receptive field, thus naturally forming a multi-resolution feature extractor which help capture unusual patterns that we should pay more attention to during tracking.

We have got some preliminary results on the current framework. We evaluate our method on the long-term tracking benchmark TB-50 [1] where the used metrics include success plots and precision plots in one pass evaluation (OPE) mode. This dataset includes 50 video sequences and 29491 frames in total. Mean success rate indicates the average overlapping ratio between the prediction and the ground truth, while mean precision rate shows how close their centers are. The higher the two values are, the better [...]

Image super-resolution (SR) is a classic image reconstruction problem in computer vision (CV), which aims at recovering a high-resolution image from a low-resolution image. As a type of supervised generative problem, image SR attracts wide attention due to its strong connection with other CV topics, such as object recognition, object alignment, texture synthesis and so on. Besides, it has extensive applications in real world, for example, medical diagnosis, remote sensing, biometric information identification, etc.

For the state-of-the-art approaches for SR, typically there are two mainstreams: 1) example-based learning methods, and 2) Deep Learning (CNN-based) methods. Example-based methods either exploit external low-high resolution exemplar pairs [1], or learn internal similarity of the same image with different resolution scales [2]. However, features used in example-based methods are usually traditional gradient-related or just handcraft, which may affect model performance. While CNN-based SR methods (e.g. SRCNN [3]) does not really distinguish between feature extraction and decision making. Lots of basic CNN models/blocks are applied to SR problem, e.g. GAN, residual learning, attention network, and provide superior SR results. Nevertheless, the non-explainable process and exhaustive training cost are serious drawbacks of CNN-based methods.

By taking advantage of reasonable feature extraction [4], we utilize spatial-spectral compatible cw-Saab features to express exemplar pairs. In addition, we formulate a Successive-Subspace-Learning-based (SSL-based) method to gradually partition data into subspaces by feature statistics, and apply regression in each subspace for better local approximation. By visualization the samples in representative subspaces, we find obvious sample similarity in pixel domain. This demonstrates the efficiency of our method in splitting samples into subspaces with semantic meaning. In the future, we aim at providing such a SSL-based explainable method with high efficiency for SR problem.

—  By Wei Wang

Reference:

[1] Timofte, Radu, [...]

Point cloud registration refers to the process of aligning two point clouds. The two point clouds to be aligned are commonly called source and target. The goal is to find a spatial transformation (3D rotation and translation) that needs to be applied to the source to optimally align it with the target.  Registration has become popular with the proliferation of 3D scanning devices like LiDAR and their applications in autonomous driving, robotics, graphics, mapping, etc.

Point clouds need to be registered in order to merge data from different sensors to obtain a globally consistent view, mapping a new observation to known data, etc. Registration is challenging due to several reasons. The source and the target point clouds may have different sampling densities and different number of points. Point clouds may contain outliers and/or be corrupted by noise. Sometimes, only partial views are available.

The problem of registration (or alignment) has been studied for a long while. Prior to point cloud processing, the focus has been on aligning lines, parametric curves and surfaces. The classical Iterative Closest Point (ICP) algorithm alternates between finding corresponding points and estimating the optimal rotation and translation. ICP just uses the spatial coordinates of points to establish point correspondences. More recently there has been a trend to use deep learning, feature based methods for registration. Two such popular methods include PointNetLK and Deep Closest Point (DCP). PointNetLK and DCP treat registration as a supervised learning problem and train end-to-end networks using deep learning. The supervision is in terms of class labels and ground truth rotation matrix and translation vector. We propose a method called ‘Salient Points Analysis (SPA)’ [1] for registration.  In contrast with the recent deep learning methods, our SPA method [...]