Abstract

The rise of AI-driven voice technologies has transformed human-machine interaction, with voice assistants like Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana being prominent examples. A critical component of these systems is the keyword spotting (wake word detection) module, which identifies specific keywords in a continuous audio stream. This module plays a vital role in voice assistants by preventing the need for computationally expensive automatic speech recognition (ASR) to run continuously. User-defined keyword spotting (UDKWS), or custom keyword detection, has gained significant attention due to the increasing demand for personalized voice assistants. Unlike closed-vocabulary keyword spotting, UDKWS must identify keywords not seen during training, often functioning in a zero-shot manner. Traditional UDKWS methods rely on ASR to transcribe audio and perform string matching, but these approaches are resource-intensive. To address this, recent neural network-based embedding techniques using text for keyword enrollment have shown promising results. However, these systems struggle to differentiate between phonetically similar keywords (e.g., ”madame” vs. ”modem”) due to alignment mismatches between audio and text representations. In this thesis, we address these challenges at both the model and feature levels. At the model level, we propose a novel approach that leverages knowledge from a pre-trained text-to-speech (TTS) system to enhance the alignment between audio and text projections. The effectiveness of TTS-based transfer learning is examined by analyzing intermediate representations. Our experiments demonstrate that the proposed method significantly outperforms the cross-modality correspondence detector (CMCD) on the challenging LibriPhrase Hard dataset, achieving an 8.22% improvement in area under the curve (AUC) and a 12.56% reduction in equal error rate (EER). In addition to model-level improvements, we explore feature-level enhancements for UDKWS. Existing methods predominantly rely on mel-scale features such as MFCC and mel spectrogram, which do not fully capture temporal dynamics like pronunciation changes over time. To address this, we propose using shifted delta coefficients (SDC), which are known for capturing long-term temporal information. Our experimental results demonstrate that SDC outperforms MFCC, with an 8.32% improvement in AUC and an 8.69% improvement in EER on the Libriphrase Hard dataset, establishing SDC as a robust feature for UDKWS, especially in differentiating between phonetically similar keywords

Abstract

Robustness towards adversarial attacks is a vital property for classifiers in several applications such as autonomous driving, medical diagnosis, etc. Also, in such scenarios, where the cost of misclassifi- cation is very high, knowing when to abstain from prediction becomes crucial. A natural question is which surrogates can be used to ensure learning in scenarios where the input points are adversarially perturbed and the classifier can abstain from prediction? This paper aims to characterize and design surrogates calibrated in “Adversarial Robust Reject Option” setting. First, we propose an adversarial robust reject option loss l γd and analyze it for the hypothesis set of linear classifiers (H lin ). Next, we provide a complete characterization result for any surrogate to be (l γd , H lin )- calibrated. To demonstrate the difficulty in designing surrogates to l γd , we show negative calibration results for convex surrogates and quasi-concave conditional risk cases (these gave positive calibration in adversarial setting without reject option). We also empirically argue that Shifted Double Ramp Loss (DRL) and Shifted Double Sigmoid Loss (DSL) satisfy the calibration conditions. Finally, we demonstrate the robustness of shifted DRL and shifted DSL against adversarial perturbations on a synthetically generated dataset.

Abstract

Sensing, detection, and screening of bio/chemical species are essential in the fields of smart living, health, foods, agriculture, environmental monitoring, and so on. Intensive research has been going on for identifying, predicting, and diversifying promising sensor substrates that can produce better sensor performance in terms of sensitivity, selectivity, response and recovery, speed, etc. In the quest for minimum footprint with maximum output, metal clusters have emerged as promising candidates. Often the luminescence properties of the metal clusters have been utilized for sensing and detection. The present study explores the potential of gold clusters as semiconduction- and vibrational (Raman and IR) spectroscopy-based sensor substrates taking cysteine (Cys) amino acid as a model ligand-cum-target biomolecule. The sensor activity of a substrate is a convolution of many effects of both the sensor substrate and the target analyte. A detailed atomic/molecular level understanding of the specific systems is required to understand sensing properties. The study involves a density functional theory (DFT)-based systematic computational study to elucidate the structure, electronic properties, and interactions of varied nuclearity gold clusters, Aun (n = 1 – 8) as well as their complexes, Aun-Cys. We have optimized the structures and calculated bonding, charge distribution, binding energies, HOMO-LUMO energies, electron localization function (ELF), density of states (DOS), vibrational spectra of the bare Aun (n = 1-8) and Aun-Cys systems in gas and water solvent media. The study attempted to explore the correlations that exist among the structure, bonding, and electronic properties of clusters and their semiconduction- and Raman/IR spectroscopies-based sensing properties. The nature of interactions and sensing properties of the clusters vary depending on the nature of the media and the cluster nuclearity. The study revealed that gold clusters can be promising candidates for semiconduction- and vibrational spectroscopy-based sensing substrates. Furthermore, vibrational spectroscopy-based single gold atom sensors can be a possibility. The present study extends our understanding beyond the traditional cluster luminescence-based sensing to single-atom vibrational spectroscopic sensing to semiconduction or resistance-based sensing.

Abstract

Cancer remains a leading cause of morbidity and mortality worldwide. Despite advances in genomics, identifying clinically relevant subtypes of cancer remains challenging due to its complex and heterogeneous nature. This thesis explores the application of network-based stratification (NBS) approaches to stratify cancer types into clinically relevant subtypes, which can aid in precision medicine and targeted therapy efforts. First, we investigate the effectiveness of a standard NBS pipeline using somatic mutation data to stratify Renal Cell Carcinoma (RCC). We explore the impact of network composition on RCC stratification performance and extend the NBS approach to copy number variation (CNV) data for RCC subtyping. Next, we introduce DeepGraphMut (DGM), a novel graph-based deeplearning pipeline that integrates somatic mutation data with protein-protein interaction (PPI) networks. By employing a graph autoencoder with a graph attention layer and a node-level attention decoder, DGM generates patient-specific clinically relevant encodings for unsupervised and supervised tasks. We demonstrate the effectiveness of DGM across 16 cancer types comprising of 7352 samples from The Cancer Genome Atlas (TCGA). Unsupervised clustering reveals distinct subtypes with significant survival differences in 11 cancer types. In supervised analysis using a Cox regression model, DGM demonstrates excellent performance in predicting survival outcomes, achieving a high concordance index (c-index) value in the range of 0.7 across most cancers, underscoring its robust predictive performance using only somatic mutation data. Furthermore, DGM outperforms traditional methods and its lightweight variant in both unsupervised and supervised analyses. In summary, this thesis presents a promising approach for cancer subtype identification and prognosis, especially in resource-limited settings where multi-omics data may not be readily available. By leveraging the strengths of graph learning and network biology, DGM offers a valuable tool for advancing personalized medicine.

Abstract

Application of machine learning methods to two important problems, namely, detection of COVID- 19 using chest radiographs (X-rays and CT scans), and molecular subtyping of breast cancer using multi-omics data is carried out. The recent pandemic made clear the need for fast and reliable tech- niques in distinguishing pneumonia caused by the novel virus SARS-CoV-2 from pneumonia caused by other viral/bacterial/fungal infections. In this work, a basic CNN model was built from scratch and spatial attention-based mechanism (Attn-CNN) incorporated to detect the manifestations of COVID-19 in CXR and CT scan images with improved generalizability and explainability has been developed. The proposed spatial attention-based solution overcomes the need for lung segmentation and region-based annotations for training the CNN models while keeping the model complexity minimized, thus making it deployable in clinical settings. To verify the generalizability of the models, testing has also been carried out on external datasets and explainability has been provided using Grad-CAM visualization of the pixels, selected by the model for classification. Performance evaluation of the proposed approach against five state-of-the-art deep learning models showed 95% accuracy for CXRs and 96% for CT images and outperformed all other models and comparatively generalized well on external datasets. Advancements in the high-throughput techniques have generated large volumes of data, enabling genome-wide profiling of various omics data, such as protein-coding and non-coding (e.g., miRNA, lncRNA, etc.) genes, DNA methylation, and analysis of genetic variations (SNVs, CNVs, etc.). How- ever, identification of diagnostic and prognostic biomarkers is challenging due to heterogeneity at mul- tiple levels and the huge number of features associated with each. This heterogeneity is seen to affect the generalizability and explainability of ML models. To address the high dimensionality and explain- ability issues, a knowledge-based feature selection framework along with a filtering approach using pre- dominant correlations is proposed for multi-omics-based biomarker identification. Breast cancer being hormone-dependent cancer, we considered the molecular subtype classification based on the three hor- mone receptors, viz., estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2): Luminal (ER+, PR±, HER2±), HER2-enriched (ER–, PR–, HER2+), and Triple Negative (ER–, PR–, HER2–). DNA methylation data from protein-coding genes and long non- coding RNAs (lncRNAs) were integrated with gene expression data of the associated genes and copy variant genes for feature selection and classification. Using 172 features obtained from the proposed framework, stratified 5-fold cross-validation was carried out using five ML models. The best perfor- mance is obtained for Random Forest model with an accuracy value of 98.19% and AUC values ≥ 0.98 for all the three classes showing the effectiveness of the proposed approach.

Abstract

Traditional drug discovery and development processes are time-consuming and costly, with only a fraction of compounds progressing to clinical testing and an even smaller percentage making it to mar- ket. To address these challenges, computer-aided drug discovery (CADD) methodologies have emerged as efficient tools for designing and evaluating potential drug candidates. By utilizing computational al- gorithms to model drug-receptor interactions, CADD significantly reduces costs and timeframes associ- ated with lead identification and optimization while maintaining high quality. Integrating deep learning algorithms further enhances drug discovery pipelines by enabling the analysis of molecular structures, genetic data, and biochemical interactions to predict drug efficacy and toxicity, and optimize dosage regimens. Deep learning helps to design new small molecule drugs and peptide drugs. Nowadays, target-based drug design is giving promising results. We propose a novel computational pipeline that leverages single-cell transcriptomic data to identify crucial proteins as drug targets for malaria, a dis- ease with increasing resistance to conventional treatments. Through mutual-information-based feature reduction and protein-protein interaction network analysis, key proteins vital for the survival of Plas- modium falciparum are identified, and potential drug molecules are computationally predicted using deep learning techniques. We can use this pipeline to select targets for any disease for developing drugs. Additionally, we explored peptides as promising therapeutic agents due to their targeted interactions with biological targets and reduced side effects compared to small-molecule drugs. We introduce HY- DRA, a hybrid diffusion model for designing therapeutic peptides tailored to specific target receptors, exemplified by the design of peptides targeting Plasmodium falciparum Erythrocyte Membrane Protein 1 (PfEMP1) genes. HYDRA generates highly stable and diverse peptides based on a protein target. Gene expression is a multifaceted process crucial to understanding molecular biology and pharmacol- ogy. We worked on elucidating the intricate relationship between gene length and kinetic parameters, such as transcription rate (S i ), association rate of transcription factor to bind with DNA (K on ), dissoci- ation rate of transcription faction detached from DNA (K of f ), Burst size (SK of f ), which significantly influence the mean expression levels of genes.Using a two-state stochastic gene expression model im- plemented in Python, we analyzed single-cell transcriptomics data to predict kinetic parameters for each gene. We classified genes into short and long categories, revealing distinct patterns in the relationship between gene length and these parameters. Our results indicate that burst size plays a critical role in mean expression, highlighting its importance for identifying gene targets that require lower drug doses for therapeutic effects. This integrated computational framework holds promise for accelerating drug discovery efforts and combating drug-resistant diseases such as malaria.

Abstract

The convergence of computer vision and advertising analysis has seen progress, but existing adver- tisement datasets remain limited. Many are small subsets of larger datasets, and while larger datasets may offer multiple annotations, they often lack consistent organization across all images, making it challenging to structure ads hierarchically. This lack of clear categorization and overlap in labeling hinders in-depth analysis. To address this, we introduce MAdVerse 1 , a comprehensive, multilingual dataset of over 50,000 advertisements sourced from websites, social media, and e-newspapers. MAd- Verse organizes ads into a hierarchy with 11 primary categories, 51 sub-categories, and 524 specific brands, facilitating fine-grained analysis across a diverse range of brands. We establish baseline perfor- mance metrics for key ad-related tasks, including hierarchical classification, source classification, and hierarchy induction in other ad datasets and, in a multilingual context, thereby providing a structured foundation for advertisement analysis. In our second work, we investigate foundational aspects of out-of-distribution (OOD) detection. Existing OOD benchmarks typically focus on broad, class-level shifts but lack controlled environments for assessing how individual attribute changes such as color or shape affect OOD detection. To bridge this gap, we created two synthetic datasets, SHAPES and CHARS 2 , each designed to allow controlled experimentation with isolated shifts in attributes. Through variations in color, size, rotation, and other factors, these datasets facilitate a targeted examination of OOD detection performance under specific conditions, providing insights into how OOD detection is affected under different attribute shifts. Later, we apply OOD detection methods to advertisements, where models face real-world distribution shifts characteristic of diverse advertising styles. Our contributions, MAdVerse for structured ad analysis and SHAPES and CHARS for controlled OOD studies emphasize the importance of robust, adaptable models for both foundational research and practical applications in advertisement analysis.

Abstract

Geography markup language (GML) is an XML specification for expressing geographical features. Defined by Open Geospatial Consortium (OGC), it is widely used for storage and transmission of maps over the Internet. XML schemas provide the convenience to define custom features profiles in GML for specific needs as seen in widely popular cityGML, simple features profile, coverage, etc. Simple features profile is a simpler subset of GML profile with support for point, line and polygon geometries. SFP has been constructed to make sure it covers most commonly used GML geometries. Web Feature Service (WFS) serves query results in SFP by default. But it falls short of being an ideal choice due to its high verbosity and size-heavy nature, which provides immense scope for compression. Generic text-based compression models are simple and fast but there are two issues that needs to be addressed. They cannot leverage the structure that exists in data to achieve better compression and there is an unnecessary decompression step before the user can actually use the data. To address these issues, we came up with GMZ, a lossless compression model aimed at achieving high compression ratios. The decision to design GMZ exclusively for GML’s Simple Features Profile (SFP) seems fair because of the high use of SFP in WFS and that it facilitates high optimization of the compression model. Our experiments indicate GMZ achieves reasonably good compression ratios and can be useful in WebGIS based applications. In a typical server-client model such as Web Feature Service, the server is the primary creator and provider of GML, and therefore, requires compression and query capabilities. On the other hand, the client is the primary consumer of GML, and therefore, requires decompression and visualization capa- bilities. In the first part of our work, we demonstrated compression using a python script that can be plugged in a server architecture, and decompression and visualization in a web browser using a Firefox add-on. The focus of this work is to develop the already existing tools to provide query capability to server. Our model provides the ability to decompress individual features in isolation, which is an es- sential requirement for realizing query in compressed state. We construct an R-Tree index for spatial data and a custom index for non-spatial data and store these in a separate index file to prevent altering the compression model. This facilitates independent use of compressed GMZ file where index can be constructed when required. The focus of this work is the bounding-box or range query commonly used in WebGIS with provision for other spatial and non-spatial queries. The decrement in compression ra- tios due to the new index file is in the range of 1-3 percent which is trivial considering the benefits of querying in compressed state. With around 75% average compression of the original data, query support in compressed state and decompression support in the browser, GMZ can be a good alternative to GML for WFS-like services.

Abstract

The analysis of digital histopathology slides or Whole Slide Images (WSIs) is critical for several diagnoses. Recent advancements in computational techniques, particularly in the field of digital pathol- ogy, have shown promise in automating the classification process. Whole Slide Imaging (WSI), com- bined with deep learning and modern computer vision techniques, has emerged as a powerful tool in this domain. This thesis addresses two major medical challenges using deep learning and computer vi- sion techniques: the classification of Lupus Nephritis (LN) and low-grade gliomas into their respective subtypes. Systemic lupus erythematosus (SLE) is an autoimmune disease wherein the patient’s immune sys- tem attacks healthy tissues, leading to Lupus Nephritis (LN), a severe condition causing renal failure. Traditional methods for diagnosing LN require meticulous pathological assessment of renal biopsies, which is time-consuming. In the first architecture(chapter 3), We propose a novel pipeline that au- tomates this process by: 1) detecting various glomerular patterns in WSIs using Periodic Acid-Schiff (PAS) stained images, and 2) classifying each image based on these extracted glomerular features. This approach leverages deep learning to improve the accuracy and efficiency of LN classification. Low-grade glioma, a type of brain tumor originating from glial cells, also presents significant diag- nostic challenges due to the large size and complexity of WSIs. In the second architecture(chapter 4), our work involves the classification of low-grade gliomas into Astrocytoma and Oligodendroglioma. Given the computational infeasibility of training deep learning models on gigapixel images, we adopt a weakly supervised method to extract discriminative patches from WSIs, which represent the tumor regions. A Convolutional Neural Network (CNN) is then trained on these discriminative patches, and the results are aggregated to determine the WSI label. Evaluated on a dataset of 581,616 patches from 286 WSIs obtained from The Cancer Genome Atlas (TCGA) portal, our method achieved a slide-wise accuracy of 79.31%, which increased to 89.65% when trained only on discriminative patches. The methodologies presented in this thesis not only demonstrate significant improvements in classi- fication accuracy but also offer scalable and efficient solutions for enhancing the diagnostic processes in pathology, ultimately contributing to better patient outcomes and more efficient healthcare delivery.

Abstract

The rapid growth of deep neural network models in the face domain has led to their adoption in safety-critical applications. However, a crucial limitation hindering their widespread deployment is the lack of comprehensive understanding of how these models work and the inability to explain their de- cisions. Explainability is essential for ensuring the correctness, reliability, and fairness of AI systems, and there is a growing recognition of its importance across AI applications. Despite the significance of explainability, most current methods are designed for general object recognition tasks and cannot be directly applied to the face domain. Faces are highly structured objects, and face tasks often in- volve fine-grained details, making them unique and distinct from general object recognition. This thesis aims to bridge the gap in explainability literature for the face domain by providing novel methods for interpreting and analyzing deep face representations. In this thesis, we embark on a comprehensive journey of interpreting and analyzing deep face repre- sentations to uncover the underlying mechanisms behind DNN-based face-processing models. We first visualize face representations and introduce methods to identify functional concepts in face representa- tions using ’cross-task aware filters’ (CRAFT). Our approach includes an efficient task-aware pruning method using CRAFTs. We also present state-of-the-art Canonical Saliency Maps (CMS) to pinpoint critical input features. We thoroughly analyze deep face representations to understand the learned fea- tures and their functional relevance in different face tasks. To further enhance our understanding of human attention in the context of driving behavior, we investigate driver gaze patterns and develop DashGaze, a large-scale naturalistic driver gaze dataset . Using this dataset, we propose an innovative calibration-free driver gaze estimation algorithm that provides valuable information for studying and predicting driver behavior. The comprehensive overview, experimental studies, and analyses presented in this thesis contribute to the wider adoption of explainability methods in face-processing tasks, enabling safer and more trust- worthy deployment of deep-face algorithms in real-world applications. By shedding light on the inner workings of these models and their biases, this work paves the way for the responsible and ethical development of AI technologies in the face domain.