Abstract

In the field of artificial intelligence, the generation of human-like motion from natural language descriptions has garnered increasing attention across various research domains. Computer vision focuses on understanding and replicating visual cues for motion, while computer graphics aims to create and edit visually realistic animations. Similarly, multimedia research explores the intersection of data modalities, such as text, motion, and image, to enhance user experiences. Robotics and human-computer interaction are pivotal areas where language-driven motion systems improve the autonomy and responsiveness of machines, facilitating more efficient and meaningful human-robot interactions. Despite its significance, existing approaches still encounter significant difficulties, particularly when generating motions from unseen or novel text descriptions. These models often lack the ability to fully capture intricate, low-level motion nuances that go beyond basic action labels. This limitation arises from the reliance on brief and simplistic textual descriptions, which fail to convey the complex and fine-grained characteristics of human motion, resulting in less diverse and realistic outputs. As a result, the generated motions frequently lack the subtlety and depth required for more dynamic and context-specific applications. This thesis introduces two key contributions to overcome these limitations and advance text-conditioned human motion generation. First, we present Action-GPT, a novel framework aimed at significantly enhancing text-based action generation models by incorporating Large Language Models (LLMs). Traditional motion capture datasets tend to provide action descriptions that are brief and minimalistic, often failing to convey the full range of complexities involved in human movement. Such sparse descriptions limit the ability of models to generate diverse and nuanced motion sequences. Action-GPT leverages LLMs to create richer, more detailed descriptions of actions, capturing finer aspects of movement. By doing so, it improves the alignment between text and motion spaces, enabling models to generate more precise and contextually accurate motion sequences. This framework is designed to work with both stochastic models (e.g., VAE-based) and deterministic models offering flexibility across different types of motion generation architectures. Experimental results demonstrate that Action-GPT not only enhances the quality of synthesized motions—both in terms of realism and diversity—but also excels in zero-shot generation, effectively handling previously unseen text descriptions. Second, we introduce MoRAG, a sophisticated retrieval-augmented generation strategy tailored to enhance the performance of motion diffusion models. MoRAG adopts a multi-part fusion retrieval mechanism that allows for improved generalization of motion retrieval across a wide range of language inputs, addressing the limitations of current retrieval methods that struggle with unseen or atypical descriptions. By incorporating low-level, part-specific motion details into the retrieval process, MoRAG constructs more accurate and varied motion sequences. The retrieval process is refined by prompting LLMs to handle issues like spelling errors, rephrasing, and ambiguous language, ensuring that the retrieved motions are contextually relevant and diverse. These augmented motion samples are then used as additional knowledge within the motion generation pipeline, enhancing the system’s ability to generate complex and realistic motions from diverse textual inputs. This retrieval-augmented approach increases both the robustness and generalization capacity of the motion generation models, making them more adaptable to complex and unseen scenarios. Together, these contributions represent a substantial advancement in text-conditioned human motion generation. By refining both the action description process and the motion retrieval strategy, this work enhances the ability of models to generate diverse, realistic motions from natural language inputs, particularly in zero-shot settings and when handling detailed, complex descriptions.

Abstract

The performance of wireless communication systems is fundamentally constrained by the propagation environment, which is uncertain, volatile, and constantly changing due to factors such as multipath fading, doppler shift, path loss, etc. Legacy solutions such as multiple input multiple output, relays, etc., have circumvented these issues to a certain extent in order to establish reliable communication links. Nonetheless, as communication systems are heading towards a future with data rate requirements in the order of gigabits/second and latencies in the order of microseconds, efficient system design is the need of the hour. Especially, solutions that merely circumvent the ill-effects of wireless propagation environments are no longer sufficient. Towards this, a new technology named Reconfigurable Intelligent Surface (RIS) is emerging as a potential solution that has the capability to alter the propagation environment to a large extent, and hence is envisioned to be a part of next-generation cellular standards. There are many research directions for the implementation of RIS-aided communication systems, such as RIS hardware design, channel modeling, channel estimation, optimal beamforming, capacity and outage analysis, optimal RIS deployment, etc. These directions must be explored in-depth in order to realize the benefits of such RIS-aided communication systems fully. This thesis aims to explore RIS-aided communication systems design and their performance analysis in terms of capacity and outage, with a special emphasis on proposing low-complexity beamforming solutions and their performance analysis. The main contribution of the thesis focuses on developing statistically optimal, low-complexity beamforming and RIS-phase shift matrix solution for a RIS-aided downlink MISO system under different propagation environments: Rician-Rician, Rician-Rayleigh, and Rayleigh-Rayleigh. The study considers both correlated and i.i.d. fading scenarios and provides closed-form expressions for optimal transmit beamformers and RIS phase shifts. These solutions offer significant advantages over instantaneous CSI-based beamforming schemes by eliminating the need for reliable feedback channels. The work also derives the ergodic capacity and outage probability performances in closed-form, demonstrating how the number of RIS elements and the presence of LoS components influence system performance under various fading conditions. Next, the thesis investigates beamforming and outage probability analysis for RIS-aided MISO downlink systems under Rician fading along both the direct and the RIS-assisted indirect links. Toward this, the focus has been on maximizing the capacity for two transmitter architectures: fully digital and fully analog. This capacity maximization problem with optimally configured RIS is shown to be L1 norm-maximization with respect to the transmit beamformer. To obtain the optimal fully digital beamformer, a complex L1-PCA-based algorithm has been proposed that has low computational complexity. Another low-complexity optimal beamforming algorithm has been proposed to obtain the fully analog beamforming solution. Additionally, analytical upper bounds on the SNR achievable by the proposed algorithms are derived and used to determine lower bounds on outage probabilities. The derived bounds are numerically shown to be exact for a unit-rank channel matrix. The final part of the thesis explores the design of a RIS-NOMA-aided multi-user downlink system with a low-complexity transceiver architecture. It proposes a fully analog architecture for the base station (BS), using a single radio frequency (RF) chain, which limits it to single-user transmission. To enable multi-user communication, NOMA is integrated with the system. Moreover, the use of RIS enhances the receive SNR for each user and also makes the propagation environment conducive for NOMA, enhancing successive interference cancellation capability. Towards this, the sum rate and energy efficiency of the proposed system are investigated while ensuring minimum rates for all users. Furthermore, the non-convex optimization sum rate and energy efficiency problems are addressed using a quadratic transform-based approach. The proposed RIS-NOMA-aided fully analog system outperforms fully digital architecture in sum rate at lower SNR and energy efficiency across a wide SNR range, demonstrating that RIS can significantly reduce transceiver complexity while improving performance.

Abstract

Imagine a situation where an interior designer is working on a new project and plans to add a few objects in the home to enhance its look. He made all the purchases and now the objects are not fitting into the designed environment. Should he still use them? Instead he can check the suitability beforehand by placing them in the environment virtually and then buy if he is satisfied. Also, consider a scenario in a game where a player is surrounded by enemies approaching from all directions. The player must strategically place obstacles to hold off the enemies, but has no assistance in this task. Can the player survive under these conditions? Can he perform these actions just by voice control or text instruction? Motivated by these ideas, we devised a method to modify the scene based on text prompt in this thesis. Text driven diffusion models have shown remarkable capabilities in editing images. These edits include substituting the objects in scene, inserting new instances, deleting unwanted things and changing the textures. Execution of these edits is quick in images and does not need much guidance. However, when editing 3D scenes, existing works mostly rely on training a Neural Radiance Field (NeRF) for 3D editing. Recent NeRF editing methods leverage edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside the text prompt to identify the edit location. In absence of this guidance, edits are decentralized from target location and are diverse in each view. When the changes are different in multiple views, 3D reconstruction becomes inaccurate, changing the overall aesthetics of the scene. These methods are operational on small 3D scenes and are more generalized to a particular scene. They require training for each specific edit and cannot be exploited in real-time edits as each training cycle requires ample amount of time. These limitations make NeRF approach a gridlock for real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model’s space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits, which are certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and the location where the edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on the grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others. We hope this work motivates future research on text driven 3D scene edits using mesh representation. As this is an initial step towards training free approach for scene edits, adding few complex components can enhance the user experience that drives the research here after.

Abstract

As cyber threats grow in complexity and sophistication, vulnerabilities are increasingly exploited due to expanding attack surfaces. Knowing all possible vulnerabilities in advance is impractical. Traditional defense techniques often fall short as attackers can conduct thorough reconnaissance on static systems and launch targeted attacks at their convenience. Moving Target Defense (MTD) offers a promising alternative by dynamically altering system configurations, making it harder for attackers to gain a stable understanding of the attack surfaces. However, the implementation of MTD can be costly, with too frequent configuration changes increasing maintenance costs and impacting the quality of service, potentially negating its benefits. Thus, there is a pressing need for cost-efficient MTD strategies that continuously balance robust defense mechanisms with minimal switching overhead. Many existing techniques rely on assumptions about prior knowledge of the attacker’s intentions and payoffs, which limits their applicability in the real-world where such information is often not readily available. This thesis proposes a novel MTD framework designed to not only thwart attacks but also to be cost-aware and continuously adapt to the evolving attack landscape. We leverage the mathematical framework of Factored Markov Decision Process (FMDP) to develop effective defense policies without assuming prior knowledge of the attacker side payoffs. Instead, we incorporate real-time attacker responses into the defender’s MDP using a dynamic Bayesian network. By employing an FMDP, we create a compact yet realistic representation of the system which often consists of multiple changeable aspects that influence the attacker’s response and the rewards. We demonstrate theoretically that considering adaptive attackers in real-world scenarios and devising effective strategies presents inherent challenges, including a negative result on regret bound. However, with certain assumptions, we show that the average regret of our method can be bounded. We validate the efficacy of our method empirically in two domains: a web application and a network system. Our experiments cover various scenarios involving evolving and unknown attackers. The results highlight the potential of our approach in significantly improving defense outcomes and enhancing the scope of modelling complex, real-world scenarios. We include a related co-authored work for comparison, which addresses this problem using a multi-armed bandit (MAB) framework. This approach employs a "followthe-perturbed-leader" style algorithm to learn effective MTD strategies.

Abstract

Software product development can often result in the generation of Software Development Waste (SDW) at any stage of the software development life cycle. SDW is defined as any resource-consuming activity that does not add value to the client or the organization developing the software. SDW impacts a software project’s overall efficiency and productivity as the project scale and size increase. For example, developers may need to rework implementations that cater to ambiguous feature stories; sometimes artifacts may not go into production, resulting in unused artifacts, etc. After the COVID-19 pandemic, software development processes were profoundly impacted. Many organizations are working predominantly with remote or hybrid work models. Traditional practices, reliant on in-person interactions and co-located teams, were disrupted as organizations adapted to new, virtual environments. This transition led to the adoption of various communication and collaboration tools, fundamentally altering how software development was executed and perceived. The work setup required teams to navigate productivity, communication, and workflow management challenges, reshaping established development practices. Our research examined the effects of the pandemic-induced work-from-home situation on the production and handling of SDW. Starting with a multi-year study, we surveyed 615 participants and interviewed 31 from the software industry across eight countries specializing in various domains. We observed a rise in SDWs and identified a new type of SDW, and other wastes were reclassified into existing waste types. Additionally, we found that teams need more direct measures of SDW and rely instead on proxy measures such as productivity and delivery times. The lack of definitive measures to monitor and manage SDW is a concern. The rising adoption of open-source software (OSS) has prompted an examination of the causes of SDW within the use-case of OSS and the appropriate metrics for its measurement. The pandemic reshaped practices, fostering more collaborative and flexible approaches and accelerating OSS adoption. This shift provides a foundation to investigate the influence of these developments on SDW. To address this gap, we propose four measures, namely Stale Forks (SF), Project Diversification Index (PDI), PR Rejection Rate (PRR), Backlog Inversion Index(BII), and the Feature Fulfillment Rate (FFR) visualization to potentially identify unused artifacts, building the wrong feature/product, mismanagement of backlog types of SDW. We applied these measures to ten open-source projects. We observe that OpenCV has 0.85%, a small percentage of active forks, and 95.79%, a high percentage of backup forks, indicating many unused artifacts. Meanwhile, the low stale fork value at 1.43% indicates fewer unused artifacts. Rustlings has a high number of potentially stale (26.11%) and stale forks (10.02%), indicating a high percentage of unused artifacts. The PDI ranges from 0.0143 for Rust in one repository to 2.18 for bootstrap, indicating the rate at which the project’s plan differs from user expectations and how much it aligns with those expectations, respectively. The repository Rustlings has the lowest unused artifact count with a PRR rate of 0.22, while angular and react-native are on the opposite side of the spectrum with values of 6.59 and 19.35, respectively. The project bootstrap has a ’0’ on its BII, with kubernetes showing the highest value at 3.37. Finally, the FFR visualization shows variations in ‘backlog management’ practices among the different projects.

Abstract

Smart home technologies, which integrate automation, energy management, and security, are increasingly influencing modern urban living. This thesis investigates the adoption of smart home systems in Mumbai, a city with notable socio-economic diversity and rapid urbanization. The study examines consumer perceptions, financial constraints, privacy concerns, and the technical feasibility of smart home technologies in an emerging market, with particular focus on price sensitivity. While global smart home adoption is driven by technological infrastructure and growing consumer awareness, markets like India face unique challenges. These include limited affordability, privacy concerns, and a lack of consumer knowledge, which slow the adoption of such technologies. This thesis addresses these gaps, exploring how these factors shape smart home adoption in Mumbai. A mixed-methods approach was employed, involving surveys from 425 residents in Mumbai’s F/South Ward. Conjoint analysis was used to evaluate consumer preferences regarding three key smart home features: security, energy efficiency, and convenience. The research assesses how these features are prioritized by consumers, and how pricing impacts decision-making. The findings reveal that security features—such as smart locks, cameras, and motion sensors— are the most important drivers of adoption, with 41.88% of respondents ranking security as their top priority. This highlights the significant demand for personal safety solutions in Mumbai’s densely populated environment. On the other hand, while energy efficiency is recognized for its potential long-term benefits, it is generally seen as less urgent compared to immediate security concerns. Cost emerges as a key barrier to adoption, particularly for middle- and lower-income households. Respondents expressed reluctance to invest in complete systems priced above ₹70,000, even when acknowledging potential savings in the long run. However, higher-income consumers demonstrated a greater willingness to adopt comprehensive smart home solutions. These findings underscore the need for flexible pricing strategies that offer both entry-level options for price-sensitive consumers and premium packages for those seeking more advanced features.Additionally, privacy and data security concerns pose significant challenges, with 72.94% of respondents expressing fears over data misuse and system vulnerabilities. These concerns, coupled with doubts about the reliability of smart home systems, contribute to consumer hesitation. Addressing these issues through robust data protection, transparent communication, and trustworthy system reliability is essential to building consumer trust. The research recommends that smart home providers focus on offering tiered pricing models to cater to different income levels, along with subscription-based or financing options to reduce upfront costs. Furthermore, professional installation services and strong customer support are crucial in ensuring system reliability and addressing technical concerns. By implementing these strategies and addressing privacy concerns, providers can enhance consumer confidence and promote broader adoption. In conclusion, this thesis provides a comprehensive analysis of the factors influencing smart home technology adoption in Mumbai. By highlighting the importance of security, affordability, and privacy, it offers practical recommendations for industry stakeholders and policymakers. Tailored solutions that address these challenges will be essential for increasing adoption rates in Mumbai and other similar urban markets.

Abstract

The rise of AI-driven voice technologies has transformed human-machine interaction, with voice assistants like Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana being prominent examples. A critical component of these systems is the keyword spotting (wake word detection) module, which identifies specific keywords in a continuous audio stream. This module plays a vital role in voice assistants by preventing the need for computationally expensive automatic speech recognition (ASR) to run continuously. User-defined keyword spotting (UDKWS), or custom keyword detection, has gained significant attention due to the increasing demand for personalized voice assistants. Unlike closed-vocabulary keyword spotting, UDKWS must identify keywords not seen during training, often functioning in a zero-shot manner. Traditional UDKWS methods rely on ASR to transcribe audio and perform string matching, but these approaches are resource-intensive. To address this, recent neural network-based embedding techniques using text for keyword enrollment have shown promising results. However, these systems struggle to differentiate between phonetically similar keywords (e.g., ”madame” vs. ”modem”) due to alignment mismatches between audio and text representations. In this thesis, we address these challenges at both the model and feature levels. At the model level, we propose a novel approach that leverages knowledge from a pre-trained text-to-speech (TTS) system to enhance the alignment between audio and text projections. The effectiveness of TTS-based transfer learning is examined by analyzing intermediate representations. Our experiments demonstrate that the proposed method significantly outperforms the cross-modality correspondence detector (CMCD) on the challenging LibriPhrase Hard dataset, achieving an 8.22% improvement in area under the curve (AUC) and a 12.56% reduction in equal error rate (EER). In addition to model-level improvements, we explore feature-level enhancements for UDKWS. Existing methods predominantly rely on mel-scale features such as MFCC and mel spectrogram, which do not fully capture temporal dynamics like pronunciation changes over time. To address this, we propose using shifted delta coefficients (SDC), which are known for capturing long-term temporal information. Our experimental results demonstrate that SDC outperforms MFCC, with an 8.32% improvement in AUC and an 8.69% improvement in EER on the Libriphrase Hard dataset, establishing SDC as a robust feature for UDKWS, especially in differentiating between phonetically similar keywords

Abstract

Robustness towards adversarial attacks is a vital property for classifiers in several applications such as autonomous driving, medical diagnosis, etc. Also, in such scenarios, where the cost of misclassifi- cation is very high, knowing when to abstain from prediction becomes crucial. A natural question is which surrogates can be used to ensure learning in scenarios where the input points are adversarially perturbed and the classifier can abstain from prediction? This paper aims to characterize and design surrogates calibrated in “Adversarial Robust Reject Option” setting. First, we propose an adversarial robust reject option loss l γd and analyze it for the hypothesis set of linear classifiers (H lin ). Next, we provide a complete characterization result for any surrogate to be (l γd , H lin )- calibrated. To demonstrate the difficulty in designing surrogates to l γd , we show negative calibration results for convex surrogates and quasi-concave conditional risk cases (these gave positive calibration in adversarial setting without reject option). We also empirically argue that Shifted Double Ramp Loss (DRL) and Shifted Double Sigmoid Loss (DSL) satisfy the calibration conditions. Finally, we demonstrate the robustness of shifted DRL and shifted DSL against adversarial perturbations on a synthetically generated dataset.

Abstract

Sensing, detection, and screening of bio/chemical species are essential in the fields of smart living, health, foods, agriculture, environmental monitoring, and so on. Intensive research has been going on for identifying, predicting, and diversifying promising sensor substrates that can produce better sensor performance in terms of sensitivity, selectivity, response and recovery, speed, etc. In the quest for minimum footprint with maximum output, metal clusters have emerged as promising candidates. Often the luminescence properties of the metal clusters have been utilized for sensing and detection. The present study explores the potential of gold clusters as semiconduction- and vibrational (Raman and IR) spectroscopy-based sensor substrates taking cysteine (Cys) amino acid as a model ligand-cum-target biomolecule. The sensor activity of a substrate is a convolution of many effects of both the sensor substrate and the target analyte. A detailed atomic/molecular level understanding of the specific systems is required to understand sensing properties. The study involves a density functional theory (DFT)-based systematic computational study to elucidate the structure, electronic properties, and interactions of varied nuclearity gold clusters, Aun (n = 1 – 8) as well as their complexes, Aun-Cys. We have optimized the structures and calculated bonding, charge distribution, binding energies, HOMO-LUMO energies, electron localization function (ELF), density of states (DOS), vibrational spectra of the bare Aun (n = 1-8) and Aun-Cys systems in gas and water solvent media. The study attempted to explore the correlations that exist among the structure, bonding, and electronic properties of clusters and their semiconduction- and Raman/IR spectroscopies-based sensing properties. The nature of interactions and sensing properties of the clusters vary depending on the nature of the media and the cluster nuclearity. The study revealed that gold clusters can be promising candidates for semiconduction- and vibrational spectroscopy-based sensing substrates. Furthermore, vibrational spectroscopy-based single gold atom sensors can be a possibility. The present study extends our understanding beyond the traditional cluster luminescence-based sensing to single-atom vibrational spectroscopic sensing to semiconduction or resistance-based sensing.

Abstract

Cancer remains a leading cause of morbidity and mortality worldwide. Despite advances in genomics, identifying clinically relevant subtypes of cancer remains challenging due to its complex and heterogeneous nature. This thesis explores the application of network-based stratification (NBS) approaches to stratify cancer types into clinically relevant subtypes, which can aid in precision medicine and targeted therapy efforts. First, we investigate the effectiveness of a standard NBS pipeline using somatic mutation data to stratify Renal Cell Carcinoma (RCC). We explore the impact of network composition on RCC stratification performance and extend the NBS approach to copy number variation (CNV) data for RCC subtyping. Next, we introduce DeepGraphMut (DGM), a novel graph-based deeplearning pipeline that integrates somatic mutation data with protein-protein interaction (PPI) networks. By employing a graph autoencoder with a graph attention layer and a node-level attention decoder, DGM generates patient-specific clinically relevant encodings for unsupervised and supervised tasks. We demonstrate the effectiveness of DGM across 16 cancer types comprising of 7352 samples from The Cancer Genome Atlas (TCGA). Unsupervised clustering reveals distinct subtypes with significant survival differences in 11 cancer types. In supervised analysis using a Cox regression model, DGM demonstrates excellent performance in predicting survival outcomes, achieving a high concordance index (c-index) value in the range of 0.7 across most cancers, underscoring its robust predictive performance using only somatic mutation data. Furthermore, DGM outperforms traditional methods and its lightweight variant in both unsupervised and supervised analyses. In summary, this thesis presents a promising approach for cancer subtype identification and prognosis, especially in resource-limited settings where multi-omics data may not be readily available. By leveraging the strengths of graph learning and network biology, DGM offers a valuable tool for advancing personalized medicine.