Abstract

Given a premise, the aim of Natural Language Inference is to identify a hypothesis as Entailed, Refuted, or Neutral. To do such classification, a model must acquire the ability to reason over the premise. While entailment tasks have been extensively studied with unstructured text as the premise, there is an increasing demand for learning to reason over semi-structured and organised data formats such as tables, knowledge graphs, databases, and combinations thereof. Structured data forms differ from unstructured text in the way they capture information and relationships – not just via language, but also through position and structure. Particularly, tables capture the connections between cells, which represent isolated distinct entities. Tabular data is organised so that items of the same kind are grouped together in rows, columns, or both. Consequently, it is straightforward to infer rankings, trends, unique items, and aggregate values from tabular data. These sorts of reasoning are specific to structured data formats, which makes inference on tables a difficult task requiring separate effort from inference on plain text. Creating challenging tabular inference data for supervision is necessary for mastering complex reasoning. Prior research in this sector has predominantly employed two data generating methodologies. The first technique is human annotation. This results in data that is inventive, fluent, and linguistically diverse. However, human annotation is costly and time-consuming, making it difficult to scale. The second form of data production is through synthetic means, where the data is generated using a defined set of rules or context-free grammar. This system is easily scalable in terms of both time and cost, but it lacks originality. Its results are predictable and adhere to predetermined patterns and fixed vocabulary. This research presents a framework for semi-automatically "recasting" existing tabular data in order to mitigate the drawbacks of both of the aforementioned data generation techniques. Existing data is perturbed, modified, and augmented through recasting to conform to the specifications of a given target task, which is Tabular Inference in this case. This framework is used to construct tabular NLI instances from five datasets that were originally designed for tasks such as table-to-text generation, tabular question answering, and tabular semantic parsing. To demonstrate the utility and quality of these datasets, this thesis explains how recasted data may be utilised as evaluation benchmarks and augmentation data to improve performance on tabular NLI tasks such as TabFact. In addition, this work evaluates the efficacy of models trained on recasted data in the zero-shot setting and examines performance trends across different types of recasted datasets. This thesis concludes with a discussion of the limitations and potential future paths of this field of study

Abstract

The reconstruction and analysis of 3D objects by computational systems has been an intensive and long-lasting research problem in the graphics and computer vision scientific communities. Traditional acquisition systems are largely restricted to studio environment setup which requires multiple synchronized and calibrated cameras. With the advent of active depth sensors like time-of-flight sensors, structured lighting sensors made 3D acquisition feasible. This advancement of technology has paved way to many research problems like 3D object localization, recognition, classification, reconstruction which demand innovating sophisticated/elegant solutions to match their ever growing applications. 3D human body reconstruction, in particular, has wider applications like virtual mirror, gait analysis, etc. Lately, with the advent of deep learning, 3D reconstruction from monocular images garnered significant interest among the research community as it can be applied to in-the-wild settings. Initially we started exploration of classification of 3D rigid objects due to availabilty of ShapeNet datasets. In this thesis, we propose an efficient characterization of 3D rigid objects which take local geometry features into consideration while constructing global features in the deep learning setup. We introduce learnable B-Spline surfaces in order to sense complex geometrical structures (large curvature variations). The locations of these surfaces are initialized over the voxel space and are learned during training phase leading to efficient classification performance. Later on, we primarily focus on rather challenging problem of non-rigid 3D human body reconstruction from monocular images. In this context, this thesis presents three principle approaches to address 3D reconstruction problem. Firstly, we propose a disentangled solution where we recover shape and texture of the 3D shape predicted using two different networks. We recover the volumetric shape of non-rigid human body shapes given a single view RGB image followed by orthographic texture view synthesis using the respective depth projection of the reconstructed (volumetric) shape and input RGB image. Secondly, we propose PeeledHuman - a novel shape representation of the human body that is robust to self-occlusions. PeeledHuman encodes the human body as a set of Peeled Depth and RGB maps in 2D, obtained by performing ray-tracing on the 3D body model and extending each ray beyond its first intersection. We learn these Peeled maps in an end-to-end generative adversarial fashion using our novel framework - PeelGAN. The PeelGAN enables us to predict shape and color of the 3D human in an end-to-end fashion at significantly low inference rates. Finally, we further improve PeelGAN by introducing a shape prior while reconstructing from monocular images. We propose a sparse and efficient fusion strategy to combine parametric body prior with a non-parametric PeeledHuman representation. The parametric body prior enforces geometrical consistency on the body shape and pose, while the non-parametric representation models loose clothing and handles self-occlusions as well. We also leverage the sparseness of the non-parametric representation for faster training of our network while using losses on 2D maps. We evaluate our proposed methods extensively on a number of datasets. In this thesis, we also introduce 3DHumans dataset, which is a 3D life-like dataset of human body scans with rich geometrical and textural details. We cover a wide variety of clothing styles ranging from loose robed clothing like saree to relatively tight-fitting shirt and trousers. The dataset consists of around 150 male and 50 unique female subjects. Total male scans are about 180 and female scans are around 70. In terms of regional diversity, for the first time, we capture body shape, appearance and clothing styles for the South-Asian population. This dataset will be released for research purposes.

Abstract

The Himalayan region, spanning 2,500 kilometers in northern India, is highly prone to seismic activity. Situated in seismic zones IV and V, this region experiences frequent and devastating earthquakes, responsible for 70% of the world's fatal landslides. Factors such as steep slopes, heavy rainfall, uneven topography, geological conditions, climate, and unplanned urbanization exacerbate the susceptibility of the Himalayan landscape to landslides during earthquakes. The ongoing collision between the Indian and Eurasian plates generates faults and stress, making the region a high-risk area for future earthquakes. Khattri's 1999 research suggests a 56% probability of a magnitude 8.5 or greater earthquake occurring in the Himalayan seismic gap within the next century. Therefore, earthquake- induced landslides are a significant concern, necessitating enhanced preparedness to mitigate social and economic setbacks. To assess slope stability under seismic conditions, researchers employ deterministic, probabilistic, and statistical techniques. While predicting earthquakes with absolute certainty is impossible, seismic hazard studies estimate expected ground motion levels. These studies play a vital role in identifying ground shaking intensities that can trigger slope failures, quantifying hazards associated with specific locations. Integrated seismic hazard assessments, considering slope properties, enable the evaluation of likely ground motion scenarios. Therefore, conducting comprehensive and up-to-date seismic hazard studies is crucial to assess future landslide hazards in the earthquake-prone Himalayan region. Several seismic hazard analyses have been conducted in the Himalayan region, resulting in the development of peak ground acceleration for specific locations. However, challenges exist in using these ground intensities to accurately predict seismic vulnerability. Outdated or macro-level hazard maps, infrequent updates to earthquake databases, lack of expanded prediction equations, generalized GIS databases, data uncertainties, and stochastic-based earthquake catalogs contribute to these challenges. Depth ranges and maximum magnitude evaluations are crucial in seismic hazard assessments. Previous studies used standard or average depth ranges, but this study incorporated appropriate focal depths for point and linear sources. It also utilized a probabilistic approach called Regional Rupture Character (RRC) to estimate maximum magnitudes, setting it apart from studies using different ground motion prediction equations (GMPEs). Furthermore, the study introduced a fully probabilistic technique, called fully probabilistic seismic hazard assessment (FPSHA), to assess ground motion triggering landslides, a novel approach for the region. This study conducted seismic hazard analyses for the Darjeeling Sikkim Himalayan region using three frameworks: deterministic seismic hazard analysis (DSHA), probabilistic seismic hazard analysis (PSHA), and fully probabilistic seismic hazard analysis (FPSHA). DSHA emphasized seismic sources as the primary threat but did not consider uncertainties in the earthquake database and GMPE, potentially impacting results. While deterministic hazard maps evaluate intolerable failure consequences, they lack probabilistic information. PSHA, considering uncertainties in the earthquake database, provides estimates of ground motion exceedance over a specific time period. PSHA ground motions were significantly lower than those from DSHA, possibly due to inclusion of uncertainties. However, PSHA ground motions varied compared to FPSHA, which integrated PSHA with a dynamic slope stability model considering slope properties. FPSHA assessed the most probable ground motion scenarios for landslide triggering over the next 50 years for all slope models. Significant differences in ground motion levels were observed between FPSHA and both DSHA and PSHA, attributed to uncertainties in slope models, GMPE, and seismic source models. It is apparent from this research that seismic landslide hazards can be overestimated or underestimated when relying solely on DSHA and PSHA approaches. While PSHA provides probabilistic ground motions based on historical earthquakes, it is recommended for general seismic infrastructure design. FPSHA, on the other hand, estimates ground motions based on earthquake statistics and soil properties, offering suitable design ground motions for landslide triggering conditions. By considering all possible earthquake scenarios leading to slope instability, FPSHA accounts for specific conditions that can trigger landslides. The updated hazard maps and design charts developed in this study have various applications. They can be utilized in seismic infrastructure design, hazard zoning mapping, landslide monitoring, seismic slope stability analysis, land use planning, code requirements, and implementation of mitigation measures. These outc

Abstract

Autonomous systems perform various tasks across different industries ranging from finance to healthcare to space applications. However, these systems are often deployed in the open world, where it is hard to obtain complete specifications of the objectives and constraints. Operating based on an incomplete model can produce undesired effects, i.e., Negative Side Effects (NSEs). Negative side effects affect the system’s safety and reliability and can be of two types: Markovian and non-Markovian. In this thesis, we try to mitigate negative side effects in environments modeled as Markov decision processes (MDPs). Unlike previous works in this area that associate negative side effects with stateaction pairs, our framework associates them with entire trajectories, which is more general and captures non-Markovian dependence on states and actions. Non-Markovian negative side effects are produced when the agent executes a certain sequence of actions in the deployed environment. Prior works mitigate Markovian negative side effects and can not be easily extended to non-Markovian negative side effects. We build a framework, Controller-Assisted Safe Planning (CASP), for mitigating the non-Markovian negative side effects. Our primary contributions are: 1. We design a model based on Finite State Controllers (FSCs) that can predict the severity of negative side effects for a given trajectory. 2. We learn the model parameters using observed data containing state-action trajectories and the severity of the associated negative side effects. The model is learned such that it generalizes well to unseen data. Information about negative side effects is gathered through Oracle feedback and compactly represented as a finite state controller. 3. We develop a constrained MDP model that uses information from both the underlying MDP and the learned model for planning while avoiding negative side effects. Our empirical evaluation demonstrates the effectiveness of our approach in learning and mitigating Markovian and non-Markovian negative side effects.

Abstract

Document summarization aims to create a precise and coherent summary of a text document. There exist a plethora of deep learning summarization models that are developed mainly for English, which often requires (i) a large training corpus and (ii) efficient pre-trained language models and tools. However, English summarization models for low-resource languages like Indian languages are often limited by rich morphological variation and syntactic and semantic differences. Also, the restricted form of supervision limits the generality and usability of low-resource languages due to the lack of annotated corpora. The Graph Autoencoder (GAE) model has recently shown superior performance on several NLP tasks, even with limited resources. In this work, we propose GAE-ISUMM, an unsupervised Indic Summarization model that extracts summaries. In particular, our proposed model uses GAE and leverages: (i) learning document representations and (ii) jointly learning sentence representations and summary of the document. For evaluation purposes, we introduce TELSUM, a manually annotated summarization dataset comprised of 501 document-summary pairs. Extensive experiments on existing low-resource datasets (XL-Sum) and TELSUM provide the following insights: (i) our proposed model displays state-of-the-art results on XL-Sum and report benchmark results on TELSUM, (ii) Surprisingly, the inclusion of positional and cluster information in the proposed model further improved the performance of summaries. We open-source our dataset and code 1 . On the other hand, with the advancement of various deep-learning methodologies and transformerbased models, summarization has advanced to a new level. However, consistent and standard datasets must be produced to benefit from these deep learning algorithms fully. The creation of dedicated resources is rarely seen for low-resource Indian languages, unlike English which hinders the progress of summarization. To this end, we create summarization resources for Indian languages by introducing ISummCorp (Indic Summarization Corpora) and IndicSumm (Indic Language Summarization Models). IsummCorp is a highly abstractive summarization dataset sourced from the Times Of India (TOI). It is manually annotated by experts across eight Indian languages. Human and intrinsic evaluations demonstrate the high quality, abstraction, and compactness of ISummCorp. IndicSumm is a set of diverse monolingual and multilingual models based on ISummCorp. We refined IndicSumm, by finetuning the sophisticated, multilingual pre-trained mT5 model. With ISummCorp, we show that a model can perform better in a monolingual environment when trained with enough monolingual data than in a multilingual finetuning scenario. To investigate the potential of monolingual models, we finetuned mT5 using ISummCorp in both monolingual and multilingual situations and achieved better performance in a monolingual setting. Furthermore, we compare IndicSumm to other multilingual summarization models (XL-Sum and IndicBART) and achieve the state-of-the-art results.

Abstract

The goal of this thesis is to design a flow synthesis based visual servoing framework enabling longrange obstacle avoidance for Micro Air Vehicles (MAV) flying amongst tall skyscrapers. Recent deep learning based frameworks use optical flow to do high-precision visual servoing. This work explores the question: can a surrogate flow be designed for these high-precision visual-servoing methods which leads to obstacle avoidance? The concept of saliency is explored for identifying high-rise structures in/close to the line of attack amongst other competing skyscrapers and buildings as a collision obstacle. A synthesised flow is used to displace the salient object segmentation mask. This flow is computed in a way that ensures that the visual servoing controller maneuvers the MAV safely around the obstacle. In this approach, a multi-step Cross-Entropy Method (CEM) based servo control is employed to achieve flow convergence, resulting in obstacle avoidance. This novel pipeline is deployed on an MAV to successfully and persistently maneuver amongst high-rises and reach the goal in simulated and photo-realistic realworld scenes. Extensive experimentation is conducted and results are presented to compare the proposed approach with optical flow and short-range depth-based obstacle avoidance methods hence conclusively demonstrating the proposed framework’s merit. Additional visualisations can be found here.

Abstract

This paper proposes a buffer-assisted amplify-and-forward (AF) unmanned aerial vehicle (UAV) relay for communication between two ground nodes without a direct link. These communications are essential in disaster rescue areas where fairness plays a significant role. To achieve fairness, we aim to maximize the minimum information rate. The traditional way of prefixed scheduling the time slots to transfer and receive at the UAV do not guarantee that the communication system uses high signal-to-noise ratio (SNR) communication links instead of low SNR links. Therefore, we choose to employ a buffer at the UAV to store the information and transfer it to the destination nodes in the high SNR links. The pairing of time slots is necessary here since we are using a buffer, and we do not know in which time slot data is transmitted to the destination node after it is received at the UAV. Consequently, we formulate a fairness maximization problem by jointly optimizing the trajectory and power control. Unfortunately, this formulation results in a non-convex problem. We propose a solution based on the principles of the minorize-maximize (MM) algorithm and linear programming relaxation techniques to solve the fairness problem and pairing of time slots. Numerical results demonstrate that the trajectory, power control and paired slots favor the UAV and ground nodes to communicate in the high SNR channel links, thus maintaining fairness.

Abstract

Text-to-speech systems convert any given text to speech. They play a vital role in making Humancomputer interaction (HCI) possible. As humans, we don’t just rely on text (language) to communicate; we use many other mechanisms like voice, gestures, expressions, etc., to communicate efficiently. In natural language processing, vocabulary and grammar tend to take center stage, but those elements of speech only tell half the story. Affective prosody of speech provides larger context and gives meaning to words, and keeps listeners engaged. Current HCI systems largely communicate in text, and they lack a lot of prosodic information, which is crucial in a conversation. To make the HCI systems communicate in speech, text to speech systems should be able to synthesize speech that is expressive and controllable. But the existing text to speech systems learn the average variation in the dataset it’s trained on, which synthesizes samples in a neutral way without many prosodic variations. To this end, we develop a textto-speech system that can synthesize the given emotion where the emotion is represented as a tuple of Arousal, Valance and Dominance (AVD) values. Text to speech systems have a lot of complexities. Training such a system requires the data to be very clear, noiseless, and collecting such data is difficult. If the data is noisy, it will reflect unnecessary artifacts in the synthesized samples. Training emotion based text to speech models is considerably more difficult and not strait forward. The fact that obtaining emotion annotated data for the desired speaker is costly and very subjective makes it a cumbersome task. Current emotion based systems can synthesize emotions with some limitations. (1) Emotion controllability comes at the cost of loss in quality, (2) Have discreet emotions which lack the finer control, and (3) cannot be generalized to new speakers without the annotated emotion data. We propose a system that overcomes the above-mentioned problems by leveraging the largely available corpus of noisy speech annotated with emotions. Even though the data is noisy, our technique trains an emotion based text to speech system that can synthesize desired emotion without any loss of quality in the output. We present a method to control the emotional prosody of Text to Speech (TTS) systems by using phoneme-level intermediate variances/features (pitch, energy, and duration) as levers. We learn how the variances change with respect to emotion. We bring the finer control in the synthesized speech by using AVD values, which can represent emotions in a 3D space. Our proposed method also doesn’t require emotion annotated data for the target speaker. Once trained on the emotion annotated data, it can be applied to any system which has the prediction of the variances as an intermediate step. vi vii With thorough experimental studies, we show that the proposed method improves over the prior art in accurately emulating the desired emotions while retaining the naturalness of speech. We extend the traditional evaluation of using individual sentences for a complete evaluation of HCI systems. We present a novel experimental setup by replacing an actor with a TTS system in offline and live conversations. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly preferred over the state-of-the-art TTS system and adds the much-coveted ”human touch” in machine dialogue.

Abstract

We embark on a hitherto unreported problem of an autonomous robot (self-driving car) navigating in dynamic scenes in a manner that reduces its localization error and eventual cumulative drift or Absolute Trajectory Error. Modern autonomous vehicles (AVs) often rely on vision, LiDAR, and even radar-based simultaneous localization and mapping (SLAM) frameworks for precise localization and navigation. However, modern SLAM frameworks often lead to unacceptably high levels of drift (i.e., localization error) when AVs observe few visually distinct features or encounter occlusions due to dynamic obstacles. This work argues that minimizing drift must be a key desiderata in AV motion planning, which requires an AV to take active control decisions to move towards feature-rich regions while also minimizing conventional control cost. To do so, we have two distinct formulations of this problem. In the first approach, we learn actions that lead to drift-minimized navigation through a suitable set of reward and penalty functions. We use Proximal Policy Optimization, a class of Deep Reinforcement Learning methods, to learn the actions that result in drift-minimized trajectories. We show, by extensive comparisons on a variety of synthetic, yet photo-realistic scenes made available through the CARLA Simulator, the superior performance of the proposed framework vis-a-vis methods that do not adopt ` such policies. In the second approach, we first introduce a novel data-driven perception module that observes LiDAR point clouds and estimates which features/regions an AV must navigate towards for drift minimization. Then, we introduce an interpretable model predictive controller (MPC) that moves an AV toward such feature-rich regions while avoiding visual occlusions and gracefully trading off drift and control cost. Our experiments on challenging, dynamic scenarios in the state-of-the-art CARLA simulator indicate our method reduces drift up to 76.76% compared to benchmark approaches. We further proposed a novel approach that combines differentiablity with classical odometry algorithms (ICP) to identify drift minimizing features in a point cloud. We also developed a modular and scalable platform for testing and deploying self-driving systems on a real car. This works sets up the groundwork for scalable active navigation in real world

Abstract

The task of entity extraction has been thoroughly explored in the NLP community over the last two decades. There is a myriad of downstream tasks for Natural Language Understanding that utilizes entity extraction - ranging from Information Retrieval, Question Answering, Fact Extraction and Verification, Knowledge Graph Completion, etc. However, the datasets and domains that have been used for evaluation and benchmarking such entity extraction models have mostly been straightforward, structurally simple, semantically non-complex, and well-present across the breadth of training data. Most entity extraction tasks comprise of a significant overlap in entities across train and test sets. This does not mimic the real-world scenario, where rare emergent entities show a larger presence, and the entities themselves are often complex and semantically ambiguous. Our work investigates entity extraction in 2 settings - in the domain of scientific research papers, and in the area of linguistically low-resource structurally complex settings. In the first scenario, we establish an end-to-end pipeline that extracts certain task-specific entities from research documents, that help in a meaningful mapping of the research landscape. We show that the domain of scientific documents is non-trivial for entity extraction tasks because scientific entities follow a long-tail distribution. We incorporate multiple strategies like Distant Supervision, Graph Ranking, and Sequence Labelling to solve this task. Furthermore, we introduce multiple human-annotated gold standard datasets for both modular and complete evaluation of this entire pipeline. In the second setting, we investigate the task of low-resource semantically ambiguous complex entities. We experiment with multiple transformer-based architectures and pre-training strategies to solve this task. Our work significantly outperforms the baseline and outperforms multiple ensemble and gazetteer-based systems. We hope that our work will help undertake further research in this critical area of Natural Language Processing.