News

Congratulations to Zhanxuan Mei for passing his defense. Zhanxuan’s thesis is titled “Explainable and Lightweight Techniques for Blind Visual Quality Assessment and Saliency Detection.” His Dissertation Committee includes Jay Kuo (Chair), Antonio Ortega, and Ulrich Neumann (Outside Member). The Committee members praised the quality of Zhanxuan’s work very much. The MCL News team invited Zhanxuan for a short talk on his thesis and PhD experience. Here is the summary. We thank Zhanxuan for his kind sharing and wish him all the best on his next journey. A high-level abstract of Zhanxuan’s thesis is given below:

Thesis:

Explainable and Lightweight Techniques for Blind Visual Quality Assessment and Saliency Detection

The thesis contains four main research topics:

We begin by presenting our proposed GreenBIQA method, a novel approach to BIQA characterized by its compact model size, low computational complexity, and high performance. Building on the foundation of GreenBIQA, we extend its application to BVQA through the development of GreenBVQA. To further enhance the performance of GreenBIQA, we introduce a lightweight and efficient image saliency detection method, termed GreenSaliency. Ultimately, we integrate GreenSaliency with GreenBIQA, culminating in the development of the Green Saliency-guided BIQA method (GSBIQA).

PhD Experience Sharing:

The PhD journey at MCL has been an unforgettable experience. Over the course of this long and challenging path, I navigated through the unprecedented COVID era, multiple rounds of rigorous exams, and a diverse range of responsibilities, including teaching assistantships and intensive research projects. Each challenge presented an opportunity to grow, expand my perspectives, and enhance my skill set. I have honed my communication skills, developed the ability to tackle real-world problems through collaborative projects, and cultivated teamwork skills by engaging in discussions and joint efforts with exceptionally talented peers. These valuable experiences and abilities [...]

Objective image quality assessment (IQA) plays a crucial role in various multimedia applications and is generally categorized into three distinct types: Full-Reference IQA (FR-IQA), Reduced-Reference IQA (RR-IQA), and No-Reference IQA (NR-IQA). FR-IQA involves a direct comparison between a distorted image and its original reference image to evaluate quality. RR-IQA, in contrast, relies on partial information from reference images to assess the quality of the target images. NR-IQA, also known as blind image quality assessment (BIQA), is indispensable in situations where reference images are unavailable, such as at the receiver’s end or for user-generated content on social media platforms. The increasing prevalence of such platforms has driven a significant rise in the demand for BIQA. BIQA is critical for estimating the perceptual quality of images without reference, making it particularly relevant in the context of user-generated content and mobile applications, where reference images are typically not accessible.

The challenge of BIQA lies in its need to handle a wide variety of content and the presence of multiple types of distortions. Although many BIQA methods leverage deep neural networks (DNNs) and incorporate saliency detectors to improve performance, their large model sizes pose significant limitations for deployment on resource-constrained devices.

To overcome these challenges, we propose a novel non-deep-learning BIQA method, termed Green Saliency-guided Blind Image Quality Assessment (GSBIQA). GSBIQA is distinguished by its compact model size, low computational requirements, and strong performance. The method integrates a lightweight saliency detection module that aids in data cropping and decision ensemble processes, generating features that effectively mimic the human attention mechanism. The GSBIQA framework is composed of five key processes: 1) green saliency detection, 2) saliency-guided data cropping, 3) GreenBIQA feature extraction, 4) local patch prediction, and 5) saliency-guided decision ensemble. [...]

Objective image quality assessment (IQA) is pivotal in various multimedia applications. It can be categorized into three distinct types: Full-Reference IQA (FR-IQA), Reduced-Reference IQA (RR-IQA), and No-Reference IQA (NR-IQA). FR-IQA directly compares a distorted image against a reference or original image to assess quality. RR-IQA, on the other hand, uses partial information from the reference images to evaluate the quality of the target images. NR-IQA, also known as blind image quality assessment (BIQA), becomes essential in scenarios where reference images are unavailable, such as at the receiver’s end or for user-generated content on social media. The demand for BIQA has surged with the increasing popularity of such platforms. BIQA is an essential task that estimates the perceptual quality of images without reference. This field is increasingly relevant due to the rise in user-generated content and mobile applications where reference images are typically unavailable.

The challenge in BIQA lies in the diversity of content and the presence of mixed distortion types. While many BIQA methods employ deep neural networks (DNNs) and incorporate saliency detectors to enhance performance, their large model sizes limit deployment on resource-constrained devices.

To address this challenge, we introduce a novel and non-deep-learning BIQA method with a lightweight saliency detection module, called Green Saliency-guided Blind Image Quality Assessment (GSBIQA). It is characterized by its minimal model size, reduced computational demands, and robust performance. The lightweight saliency detector in GSBIQA facilitates data cropping and decision ensemble and generates useful features in BIQA that emulate the attention mechanism. The GSBIQA method is structured around five key processes: 1) green saliency detection, 2) saliency-guided data cropping, 3) Green BIQA feature extraction, 4) local patch prediction, and 5) saliency-guided decision ensemble. Experimental results show that the performance of [...]

Single image super-resolution (SISR) is an intensively studied topic in image processing. It aims at recovering a high-resolution (HR) image from its low-resolution (LR) counterpart. SISR finds wide real-world applications such as remote sensing, medical imaging, and biometric identification. Besides, it attracts attention due to its connection with other tasks (e.g., image registration, compression, and synthesis). To deal with such ill-posed problem, we recently proposed two methods, LSR[1] and LSR++[2], by providing reasonable performance and effectively reduced complexity.

LSR consists of three cascaded modules:

Unsupervised Representation Learning by creating a pool of rich and diversified representations in the neighborhood of a target pixel.

Supervised Feature Learning by Relative Feature Test (RFT [3]) to select a subset from the representation pool that is most relevant to the underlying super-resolution task automatically, and

Supervised Decision Learning by predicting the residual of the target pixel based on the selected features through regression via classical machine learning, and effectively fusioning the predictions for more stable results.

LSR++ is promoted based on LSR, with emphasis on sample alignment, a more promising sample preparation process which is suitable for all patch-based computer vision problems. As illustrated in Fig 1, based on gradient histograms of patches along the eight reference directions (Fig.1.a), patch alignment utilizes patch rotations and flipping to meet the standard templates of gradient histograms, where D_max is the direction with the largest cumulative gradient magnitude, and D_max_orth_b and D_max_orth_s refer to the orthogonal directions to D_max with big and small cumulative gradient magnitude, respectively. By modifying the set of (D_max, D_max_orth_b, and D_max_orth_s) of a patch, patch alignment can regularize the edge pattern with the patch by directions perpendicular the edge (D_max) and directions along the edge (D_max_orth_b, D_max_orth_s). The process of patch [...]

Understanding and retrieving information in 3D scenes poses a significant challenge in artificial intelligence (AI) and machine learning (ML), particularly in grasping complex spatial relationships and detailed properties of objects in 3D spaces. Multiple tasks are suggested to assess 3D understanding, such as 3D object retrieval, 3D captioning, 3D question answering, 3D vision grounding, etc.

Existing methods can be roughly divided into two categories. The first category utilizes large 2D foundational models for feature extraction and maps 2D pixel-wise features to 3D point-wise features for 3D tasks. For example, the 3D-CLR model [1] extracts 2D features from multiview images with the CLIP-LSeg model [2] and maps the 2D features to 3D points in a reconstructed neural radiance field compact representation. The reasoning process is performed via a set of neural reasoning operators. The 3D-LLM model [3] utilizes 2D vision-language models (VLM) as the backbone. It extracts 2D features with the ConceptFusion model [4] and maps them to 3D points. Then, the 3D information is injected into a large language model to generate text outputs.

Another group of methods directly handles 3D point clouds with a 3D encoder and tries to align the extracted 3D features with the features from other modalities. This group of methods may require the training of a 3D encoder and may need many computational resources. For example, the Uni3D [5] leverages a unified vanilla transformer structurally equivalent to a 2D Vision Transformer (ViT) as the backbone to extract 3D features. Downstream tasks can be achieved after feature alignment among different modalities. It is also possible to leverage pre-trained 3D encoders. Point-SAM [6] utilizes the point cloud encoder from the Uni3D to transform the input point cloud into embeddings. It starts by sampling [...]