Abstract

Videos form an integral part of human lives and act as one of the most natural forms of perception spanning both the spatial and the temporal dimensions: the spatial dimension emphasizes the content, whereas the temporal dimension emphasizes the change. Naturally, studying this modality is an important area of computer vision. Notably, one must efficiently capture this high-dimensional modality to perform different downstream tasks robustly. In this thesis, we study representation learning for videos to perform two key aspects of video-based tasks: classification and generation. In a classification task, a video is compressed to a latent space that captures the key discriminative properties of a video relevant to the task. On the other hand, generation involves starting with a latent space (often a known space, such as standard normal) and learning a valid mapping between the latent and the video manifold. This thesis explores complementary representation techniques to develop robust representation spaces useful for diverse downstream tasks. In this vein, this thesis starts by tackling video classification, where we concentrate on a specific task of “lipreading” (transliterating videos to text) or in technical terms - classifying videos of mouth movements. Through this work, we propose a compressed generative space that self-augments the dataset improving the discriminative capabilities of the classifier. Motivated by the findings of this work, we move on to finding an improved generative space in which we touch upon several key elements of video generation, including unconditional video generation, video inversion, and video superresolution. In the classification task, we aim to study lipreading (or visually recognizing speech from the mouth movements of a speaker), a challenging and mentally taxing task for humans to perform. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from ‘Amyotrophic Lateral Sclerosis’ (ALS) often lose muscle control, consequently, their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting large-scale datasets of a patient needed to train modern data-hungry deep learning models is, however, extremely challenging. We propose a personalized network designed to lipread for an ALS patient using only one-shot examples. We depend on synthetically generated lip movements to augment the one-shot scenario. A Variational Encoder-based domain adaptation technique is used to bridge the real-synthetic domain gap. Our approach significantly improves and achieves high top-5 accuracy with 83.2% accuracy compared to 62.6% achieved by comparable methods for the patient. Apart from evaluating our approach on the ALS patient, we also extend it to people with hearing impairment, relying extensively on lip movements to communicate. In the next part of the thesis, we focus on representation spaces for video-based generative tasks. Generating videos is a complex task that is accomplished by generating a set of temporally coherent images frame-by-frame. This approach confines the expressivity of videos to image-based operations on individual frames, necessitating network designs that can achieve temporally coherent trajectories in the underlying image space. We propose INR-V, a video representation network that learns a continuous space for video-based generative tasks. INR-V parameterizes videos using implicit neural representations (INRs), a multi-layered perceptron that predicts an RGB value for each input pixel location of the video. The INR is predicted using a meta-network which is a hypernetwork trained on neural representations of multiple video instances. Later, the meta-network can be sampled to generate diverse novel videos enabling many downstream video-based generative tasks. Interestingly, we find that conditional regularization and progressive weight initialization play a crucial role in obtaining INR-V. The representation space learned by INR-V is more expressive than an image space showcasing many interesting properties not possible with the existing works. For instance, INR-V can smoothly interpolate intermediate videos between known video instances (such as intermediate identities, expressions, and poses in face videos). It can also in-paint missing portions in videos to recover temporally coherent full videos. We evaluate the space learned by INR-V on diverse generative tasks such as video interpolation, novel video generation, video inversion, and video inpainting against the existing baselines. INR-V significantly outperforms the baselines on several of these demonstrated tasks, clearly showcasing the potential of the proposed representation space. In summary, this thesis makes a significant cont

Abstract

Fingerprints are a widely recognized and commonly used method of identification. Contact-based fingerprints, which involve pressing the finger against a surface to obtain images, are a popular method of capturing fingerprints. However, this process has several drawbacks, including skin deformation, unhygienic conditions, and high sensitivity to the moisture content of the finger. These factors can negatively impact the accuracy of the fingerprint. Moreover, fingerprints are three-dimensional anatomical structures, and two-dimensional fingerprints do not capture the depth information of the finger ridges. While 3D fingerprint capture is less sensitive to skin moisture levels and avoids skin deformation, it is limited in adoption due to the high cost and system complexity associated with it. The complexity and cost are mainly attributed to the use of multiple cameras, projectors, and sometimes synchronously moving mechanical parts. Photometric stereo offers a promising solution to build low-cost, simple sensors for high-quality 3D capture using only a single camera and a few LEDs. However, the method assumes that the surface being imaged is lambertian, which is not the case for human fingers. Existing 3D fingerprint scanners based on photometric stereo also assume that the finger is lambertian, resulting in poor reconstruction results. In this context, we introduce the Split and Knit algorithm (SnK), a 3D reconstruction pipeline based on Photometric Stereo for finger surfaces. The algorithm splits the reconstruction of the ridge-valley pattern and finger shape and combines them to obtain the 3D fingerprint reconstruction for the full finger with a single camera for the first time. To reconstruct the ridge-valley pattern, SnK introduces an efficient way of estimating the direct illumination component by using a trained U-Net without extra hardware, which reduces the non-Lambertian nature of the finger image and enables a higher-quality reconstruction of the entire finger surface. To obtain the finger shape using a single camera, the algorithm introduced two novel approaches, a) using IR illumination and b) using a mirror and parametric modeling for the finger shape. Finally, we combine the overall finger shape and the ridge-valley point cloud to obtain a 3D finger phalange. The high-quality 3D reconstruction results in better matching accuracy of the captured fingerprints. Splitting the ridge-valley pattern from the finger provides an implicit way to convert 3D fingerprint into 2D fingerprint, making the SnK algorithm compatible with the 2D fingerprint recognition systems. To apply the SnK algorithm to fingerprints, we designed a 3D printed photometric stereo-based setup that captures contactless finger images and obtains their 3D reconstructions

Abstract

Breast cancer continues to be a major worldwide health burden, and therapy and prognosis are greatly influenced by hormone receptor status. Immunohistochemistry (IHC) is currently the gold standard for evaluating the status of the estrogen receptor (ER), the progesterone receptor (PR), and the human epidermal growth factor receptor 2 (HER2). However, this approach has drawbacks, such as the potential for labelling errors and inconsistency with intrinsic subtypes. To increase the precision of predicting hormone receptor status, this thesis introduces a unique predictive modelling approach employing DNA methylation and gene expression data. We created machine learning models that use data from The Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO) to include DNA methylation profiles and gene expression data to forecast ER, PR, and HER2 status. Using MSigDB and STRINGDB, we discovered genes that had varying levels of methylation in relation to each receptor status and looked into the functional significance of these findings. Additionally, we used machine learning models based on noisy label training to address the issue of noisy labels that may be present as a result of potential mislabeling by IHC-based approaches. The results of the study revealed that gene expression information and DNA methylation profiles are reliable indicators of hormone receptor status. Our models performed well as compared to traditional IHC techniques, indicating potential clinical value. Additionally, our method might predict brand-new biomarkers and offer deeper perceptions into the epigenetic pathways underlying breast cancer. However, there are still certain drawbacks, such as class imbalance problems and the high-dimensionality of DNA methylation data, which may be resolved with the development of machine learning techniques and larger, more representative datasets. This thesis emphasises the potential of DNA methylation-based prediction models to increase the precision of determining hormone receptor status, provide useful information for individualised therapy approaches, and enhance patient prognosis

Abstract

The field of autonomous driving has seen many advances in recent times. In this thesis, we try to look at some important challenges in autonomous driving from the perspective of a single (monocular) camera: Real-Time Scene (Road/Lanes) Layout and multi-object simultaneous localisation and mapping in the outdoor scene. To tackle the challenges faced by monocular setups/systems (such as scale ambiguity in monocular reconstruction, dynamic object localisation, and uncertainty in feature representation), we transpose the above problems into orthographic (bird’s-eye view) space. Thus, we perform road/lane layout reconstruction, localisation and mapping of dynamic participants and the ego camera vehicle in the scene with an orthographic view as the configuration space. By assuming only the height of the egocamera above the ground, we leverage single-view metrology cues to accurately localise the ego-vehicle and all other traffic participants in bird’s-eye view. We demonstrate that our system outperforms prior work that uses strictly greater information, highlighting each design decision’s relevance via an ablation analysis

Abstract

Barak valley is an area in the north-eastern part of India where traditionally the practice of shifting cultivation has been more prevalent than sedentary agriculture. In the recent past, population growth coupled with the geographic isolation of the area has further aggravated the decline of forest land cover in the area. The region that encompasses the Barak valley, shows varied land covers and has diverse population densities. While some land use and land cover changes are gradual and slow, some can be rapid and fast. Also, many of these changes are dependent on the biophysical and socio-economic drivers of change, including policy inputs. So, there is a need to understand these interactions and how these changes will be affected by the different factors in the near future. This thesis focuses on modelling land use changes in the Kathakal basin of Barak valley, primarily changes in the shifting cultivation patterns, over a period of 1988 to 2005 using an agent-based model after understanding changes in the recent past. Each district is modelled as an agent of change in the agent-based model, 1. to capture the interactions between the various drivers like population changes, infrastructure development and land use practices and 2. to decide on the land resource allocation across the district at the land parcel level including allotment of shifting cultivation regions. The model accounts for both socio-economic as well as geographic factors, like access to infrastructure while making the land use decision on a year-on-year basis. The model considers the need for shifting cultivation land area for both a staple crop – paddy (rice) and a non-staple crop based on the demand against the supply estimated from irrigated and rain-fed cropping regions. While the model has been fine-tuned based on the data till 1997, the simulated model outcome of 2005 is validated against an existing remote sensing derived land use map of the region. The model shows an accuracy of 98% for the test period till 1997 and achieves an overall accuracy of about 94% for 2005 predicted land use pattern. While the aggregated results across the region show good concurrence, the randomness inherent in the choice of the shifting cultivation land limits the ability to predict the precise locations. Further, the thesis will discuss the challenges of different land use conditions prevailing in the two districts and how the model is able to adapt to those in providing a good estimate of the land use changes in a region as diverse as Barak Valley. Based on the model and its applicability to the Barak valley region, the research further explores the impact of various policies on the choice of land use over a period of 30 years, i.e., the land use patterns between 2005 and 2035. Four different scenarios, including a business-as-usual (BAU) scenario were modelled, to highlight how either ecological or economic considerations affects the choice of land use in the two diverse existing land use conditions of the districts of Hailakandi and Mamit. The results show that depending on whether the district is largely rural or urban, has sparse or dense population, and has access to irrigation infrastructure, the land use responses are different. The BAU scenario shows that without intervention of any kind, it is difficult to stop the spread of shifting cultivation due to the land pressures from population growth and the need for increased economic returns from the land. The model shows that infrastructure development and incentivising settled forms of agriculture can have a positive impact on reducing shifting cultivation, though the rate of this change is dependent on both socio-economic and bio-physical conditions of the region.

Abstract

In the current digital era, about 80% of the digital data which is being generated is unstructured and unlabeled natural language text. Research efforts are going on for improving text mining techniques to automatically organize, analyze, and extract useful information from the voluminous text data. In the development cycle of information retrieval and text mining applications, text representation is the most fundamental and critical step as its effectiveness directly impacts the performance of the application. Three important properties of the text representation models, which make them suitable for practical applications, are representational power, interpretability of features, and unsupervised learnability. Words and word sequences (such as sentences and documents) are the natural units of the text which are subjected to vector representation to handle the given text data. With the advent of deep learning, a family of neural language models emerged to represent words as low dimensional, distributed, and dense vectors. These word representations are popularized as word embeddings. Word embeddings are learned from huge corpora, so they encode a lot of information and have high expressive power. In the case of word sequences, traditional methods provide interpretability and support unsupervised learning, but they suffer from the issues of low representational power and scalability. The deep learning based word embedding methods are successful in producing vectors with high representational power. However, most of the deep learning based methods are notorious for human uninterpretabilty and work only in supervised learning environments. In this thesis, we propose improved word sequence representation models by exploiting the frequency distributional and spatial distributional properties of the word embeddings. Firstly, we propose an alternative word sequence representation framework in the context of longer texts such as documents. The existing vector averaging based models represent the document as a position vector in the same word embedding space. As a result, they are unable to capture the multiple aspects as well as the broad context in the document. Also, due to their low representational power, the existing approaches perform poorly at document classification. Furthermore, the document vectors obtained using such methods have uninterpretable features. In this work, we propose an improved document representation framework that captures multiple aspects of the document with interpretable features. In this framework, instead of representing a document in word embedding space, it is represented in a distinct feature space where each dimension is associated with a potential feature word that has relatively high discriminatory power. A given document is modeled as the distances between the feature words and the document. We have proposed two criteria for the selection of potential feature words and a distance function to measure the distance between the feature word and the document. Experimental results on multiple datasets show that the proposed model consistently performs better at document classification over the baseline methods. Secondly, we propose a weighted averaging based word sequence embedding method in the context of shorter texts such as sentences. The proposed weighting scheme captures the contextual diversity of words based on their geometry in the embedding space. It is observed that the simple unweighted vector averaging ignores the discriminative power of words and treats all the words as equal in the given sentence. Assigning the weights to words based on their discriminative power helps in building better word sequence representations. Recent literature introduced word weighting schemes based on the word frequency distribution into the simple averaging model. The frequency-based weighted averaging models augmented with the denoising steps are shown to outperform many complex deep learning models. However, these frequency-based weighting schemes derive the word weights solely based on their raw counts and ignore the diversity of contexts in which these words occur. The proposed weighting algorithm is simple, unsupervised, and non-parametric. Experimental results on semantic textual similarity tasks show that the proposed weighting method outperforms all the baseline models with significant margins and performs competitively to the current frequency-based state-of-the-art weighting approaches. Furthermore, as the frequency distribution-based approaches and the proposed word embeddings geometry-based weighting approach capture two different properties of the words, we define hybrid weighting schemes to combine both varieties. We also empirically demonstrate that the hybrid weighting methods perform consistently better than the corresponding individual weighting schemes. Overall, we have proposed a new word sequence representation framework and wei

Abstract

In the healthcare domain, medical and patient interactions are crucial for diagnosis. Initially, AI models developed for healthcare centered only on monolingual data. However, such models do not cater to the multilingual regions, where most conversations are Code-Mixed. Our thesis concentrates on developing the Code-Mixed Medical Task-Oriented Dialog System. This task-oriented dialog system aims to help the user consult a medical specialist based on their symptoms. We train the dialog system on the code-mixed task-oriented dialog dataset ‘Su-Vaid’. The dataset contains 3005 Telugu-English Code-Mixed dialogues between patients and doctors with 29k utterances covering ten specializations with an average code-mixing index (CMI) of 33.3%. The major components of our dialog system are Natural Language Understanding (NLU), Dialog Manager (DM), and Natural Language Generation (NLG) modules. We manually annotated the conversational dataset with intent and slot labels. We also present baselines to establish benchmarks on the dataset using existing state-of-the-art Natural Language Understanding (NLU) models. We improved the existing baselines using contextual ground truth intent labels and processing the slots as chunks. We have used Rule-based Dialog Manager and template-based NLG module in our Dialog System. While interacting with the system, the user may fail to mention a few symptoms. In such a case, the dialog system must remind the user by suggesting relevant symptoms. We have incorporated a suggestion system into our dialog system. Apart from interacting with the user to recommend the medical specialist, the dialog system needs to be more human-like (empathetic). Also, human-human conversations in the healthcare domain are mostly empathetic. To achieve human-like behavior, we have included empathy via language accommodation in our dialog system. Also, interactions between patients and medical practitioners can sometimes be challenging. Health care workers are at greater risk of workplace abuse than most other workers, with nurses and family physicians rated as most at risk of abusive encounters with patients. Types of abusive encounters range in severity, from verbal threats to more extreme encounters, such as stalking and physical assault. Hate speech and physical abuse must be strongly condemned for a better function of health care. Such conversations must be identified by the system so that the system can take necessary action on such users. So we have included hate speech detection in our dialog system. Finally, we have evaluated all the major components of our dialog system using state-of-the-art baselines and classification metrics and we chose the top performing models to integrated them into the final version of the system.

Abstract

Autism Spectrum Disorder (ASD) is a lifelong heterogeneous developmental disorder that is characterized by abnormal development of the brain. Deficits in social skills, incapability to articulate language, abnormal sensory-motor movements, and stereotyped behaviors are mainly observed in children with ASD. The underlying biological markers for the diagnosis and treatment of ASD are not known yet. The current behavior-based diagnosis of ASD is arduous and requires expertise. It has been demonstrated that the non-invasive method such as resting-state functional magnetic resonance imaging (rs-fMRI) provides a means to examine functional connectivity patterns in the brain, which can help diagnose various neurodegenerative and psychiatric disorders, including ASD. Most previous studies associated ASD with atypical functional connectivity (FC) between different pairs of regions. However, brain connectivity is dynamic and varies extensively among brain states. In this thesis, we explored both static and dynamic functional connectivity based features to understand the fundamental group differences between ASD patients and typically developing (TD) subjects. Firstly, we proposed a Multilayer Perceptron (MLP) based classification model with autoencoder pretraining for classifying ASD from TD based on static functional connectivity (sFNC) extracted from rs-fMRI scans of the ABIDE-1 (a publicly available dataset from Autism Brain Imaging Data Exchange consortium). Our model achieves new state-of-the-art performance on the ABIDE-1 dataset with a 10- fold cross-validation accuracy of 74.82%. Further, we use the Integrated Gradients (IG) and DeepLIFT techniques to identify the correlations between brain regions that contribute most to the classification task. Our analysis identifies the following regions associated with ASD: Left Lingual Gyrus, Right Insula, Right Cuneus, Right Middle Frontal Gyrus, and Left Superior Temporal Gyrus. Interestingly, these regions in the brain are primarily responsible for social cognition, language, attention, decisionmaking, and visual processing, which are known to be altered in ASD. Secondly, we investigated the dynamic functional connectivity (dFNC) between 53 independent components among 188 ASD and 195 TD subjects sampled from the ABIDE-I consortium. We estimated dFNC using sliding window-based approaches and identified four distinct dynamic states through hard-clustering analysis. Hyper-connectivity within the cognitive control domain, between cognitive control and default mode network have been identified among ASD subjects. Hyper-connectivity within the default mode network has been found among TD individuals. Further, we estimated the dynamic temporal properties such as fractional time spent, mean dwell time per state and observed significant differences between ASD and TD groups. ASD subjects are found to have significantly longer dwell time in one of the states (4) when compared to TD individuals. We also found a significantly increased occurrence of the same state (4) in ASD subjects, whereas other states (1 and 3) are more frequent in TD subjects. Overall, both static and dynamic-based methods have been explored to find out the ASD biomarkers. While there is broad consensus in the brain network profiles between sFNC and dFNC, the temporal profile of brain state dynamics is additionally available with dFNC analysis and may potentially contribute to disease biomarkers. In addition, we have done extensive replication studies of both sFNC and dFNC based models for ASD classification or characterization in the literature. We observe great discrepancy between what is reported and what could be replicated or reproduced. Based on these analyses, we propose a set of recommendations for future studies that encompass factors such as dataset selection criteria, preprocessing pipepline, proper reporting of selected samples, atlas selection, and hyperparameter choices and reporting for the proposed models.

Abstract

In recent years, due to the advent of technology and the Internet, the amount of opinionated data targeting products has increased exponentially. With this increase, there has emerged a need to understand the opinions and their associated sentiment to enhance the feedback loop for organizations manufacturing the products and users looking for opinions about the usability of the same to base their future purchasing decisions on. The major bottleneck in achieving the above is the constraint on resources available for certain languages. We explore ways to work through these constraints and attempt to achieve good results for the tasks of Sentiment Analysis and it’s variants. In this thesis, We first attempt to extract language invariant features for downstream tasks like sentiment analysis to be able to retain decent performance in a low resources setting. We find that the ability to do so is task-dependent and hypothesize patterns in tasks where the approach can and cannot work effectively. We then take a closer look at the reasons for the poor performance of models in Aspect Term Extraction and Aspect Term Polarity Classification for Hindi, which is a variant of Sentiment Analysis based on Opinion Mining. After a detailed analysis of the same, we conclude that there is a gap in the state of the existing gold standard dataset for Hindi. We then go ahead and describe our methodology for developing a high-quality dataset parallel to the Gold English dataset for these tasks and establish that the new dataset adequately represents the task. To further improve the state of Aspect Term Extraction (ATE) and Aspect Term Polarity Classification (ATPC), we develop a novel architecture that achieves new state-of-the-art results for Hindi and near state-of-the-art results for English. We also show the fullness of our method in solving the task in a multilingual setting and achieving near-state-of-art results and hence establishing that for tasks where we cannot extract language invariant features, we can develop models that can learn features crucial for the task in a manner which can be leveraged to give high performance reliably.

Abstract

The rapid advancement of autonomous driving systems is transforming the future of transportation and urban mobility. However, these systems face significant challenges when deployed in complex and unstructured traffic environments, such as those found in many cities in south-east Asian countries like India. This thesis aims to address the challenges related to data collection, management, and generation for autonomous driving systems operating in such scenarios. The primary contributions of this work include the development of a data collection toolkit, the creation of a comprehensive driving dataset for unstructured environments (IDD-3D), and the proposal of a synthetic data generation framework (TRoVE) based on real-world data. The data collection toolkit presented in this thesis enables sensor fusion for driving scenarios by treating sensor APIs as separate entities from the data collection interface. This framework allows for the use of any sensor configuration and demonstrates the smooth creation of driving datasets using our framework. This toolkit is adaptable to different environments and can be easily scaled, making it a crucial step towards creating a large-scale dataset for Indian road scenarios. The IDD-3D dataset provides a valuable resource for studying unstructured driving scenarios with complex road situations. We present a thorough statistical and experimental analysis of the dataset, which includes high-quality annotations for 3D object bounding boxes and instance IDs for tracking. We highlight the diverse object types, categories, and complex trajectories found in Indian road scenes, enabling the development of robust autonomous driving systems that can generalize across different geographical locations. In addition, we provide benchmarks for 3D object detection and tracking using state-of-the-art approaches. The TRoVE synthetic data toolkit offers a framework for the automatic generation of synthetic data for visual perception, leveraging existing real-world data. By combining synthetic data with real data, we show the potential for improved performance in various computer vision tasks. The data generation process can be extended to different locations and scenarios, avoiding the limitations of bounded data volumes and variety found in manually designed virtual environments. This thesis contributes to the development of systems in complex environments by addressing the challenges of data acquisition, management, and generation. By bridging the gap between the current state-of-the-art and the needs of unstructured traffic scenarios, this work paves the way for more robust and versatile intelligent transportation systems that can operate safely and efficiently in a wide range of situations