Abstract

Modeling realistic human-object interaction in three-dimensional environments is a fundamental
challenge in computer vision, with broad implications for robotics, virtual reality, and embodied artificial intelligence. Despite significant progress in human motion generation and 3D scene understanding,
existing approaches often treat motion, interaction, and environment as separate problems, limiting their
ability to generate physically plausible and context-aware behavior.
This thesis addresses this challenge through a unified perspective on scene-aware human-object interaction, grounded in both real-world systems and data-driven modeling. We begin by developing a
system for real-time human pose understanding and interaction analysis based on joint-level reasoning,
formalized in our published patent application. This system demonstrates how structured representations of human motion can enable robust, interpretable, and deployable interaction-aware applications.
Building on this foundation, we introduce a large-scale dataset and benchmark for full-body human
motion with object interaction in realistic 3D scenes. This dataset enables the study of scene-aware
grasping and exposes key limitations of existing methods in handling interaction under environmental
constraints. We then propose a learning-based framework for generating physically plausible, sceneaware human-object interactions that explicitly models the interplay between body motion, object manipulation, and scene geometry.
To enable scalable deployment of such systems, we further develop a fast and robust geometry processing framework based on feature-aware mesh simplification, which preserves interaction-relevant
geometric properties while significantly reducing computational complexity.
We also explore the extension of interaction-aware modeling to conditional settings, including multimodal human motion generation. In particular, we highlight the role of object interaction in ensuring
consistency between generated motion and environmental constraints, demonstrating that naive integration of motion and interaction leads to physically implausible results.
Through extensive experiments and evaluations, we demonstrate that the proposed approaches significantly improve the realism, physical plausibility, and scalability of human-object interaction in 3D
environments. By bridging real-world systems, data-driven modeling, and geometry processing, this
thesis contributes towards the development of integrated frameworks for human-centric 3D understanding and interaction.

Abstract

Classical motion planning for robotic manipulation includes a set of general algorithms that aim to
minimize a scene-specific cost of executing a given plan. This approach offers remarkable adaptability,
as they can be directly used off-the-shelf for any new scene without needing specific training datasets.
However, without a prior understanding of what diverse valid trajectories are and without specially
designed cost functions for a given scene, the overall solutions tend to have low success rates within a
certain time limit. Conversely, deep-learning-based planners achieve high accuracy within their training
domains but are notoriously brittle when faced with out-of-distribution environments. To bridge this
gap, we first introduce Ensemble-of-costs-guided Diffusion for Motion Planning (EDMP) [2]. EDMP
leverages a diffusion model to learn a prior over kinematically valid trajectories while incorporating
scene-specific "collision-cost" guidance directly at the time of inference. By employing an ensemble of
multiple cost functions to capture diverse scene nuances, EDMP significantly improves success rates and
generalizes to novel tasks, such as manipulating arbitrary held objects, without retraining.
Despite its robustness, EDMP, and similar waypoint-based diffusion planners, faces a fundamental
bottleneck: the requirement for a substantial number of denoising steps, which leads to slow inference
speeds. To address this, we propose Guided Polynomial Diffusion for Motion Planning (GPD)[1],
where we introduce diffusion in the parametric space of trajectories, where the parameters are
represented as Bernstein coefficients. We show that this representation greatly improves the effectiveness
of the cost-function guidance and the inference speed. We also introduce a novel stitching algorithm
that leverages the diversity in diffusion-generated trajectories to produce collision-free trajectories
with just a single cost function-guided model. We demonstrate that our approaches outperform SOTA
diffusion-based motion planners for manipulators and provide an ablation study on key components.

Abstract

Document question answering has witnessed substantial progress with the development of large visionlanguage models. Despite these advances, most existing systems remain limited in their interpretability,
as they typically generate answers without explicitly and precisely identifying the visual evidence on
which those answers are based. This deficiency is particularly pronounced in real-world documents,
where supporting evidence may appear in curved text, be fragmented across multiple spatially separated
regions, be embedded within tables or charts, or exist at different semantic levels such as phrases, lines,
and blocks. Under such conditions, conventional grounding approaches based on coarse bounding boxes
are often insufficient, as rectangular regions fail to capture the irregular and fine-grained structure of
document evidence.
This thesis addresses these limitations through M3Grounder, a novel mask-based framework for
grounded document question answering. In contrast to prior approaches that represent evidence using
rectangular coordinates, the proposed framework formulates grounding as a segmentation problem
and predicts pixel-level evidence masks that are tightly aligned with the answer generation process.
The model autoregressively generates answer text interleaved with special grounding tokens, enabling
explicit correspondence between individual answer spans and their associated supporting regions. This
formulation naturally supports multi-span grounding, in which a single answer may depend on multiple
disjoint regions, and multi-granular grounding, in which evidence may be localized at the phrase, line,
or block level depending on the semantic scope required by the question. This design enables answer
generation interleaved with grounding tokens and hierarchical evidence prediction.
To realize this formulation, the thesis introduces a set of architectural and training mechanisms
designed to improve the precision and faithfulness of document grounding. In particular, M3Grounder
integrates document understanding with segmentation-based evidence prediction, employs a bleedsuppression objective to reduce spillover into irrelevant neighboring regions, and incorporates a hierarchical enclosure mechanism that encourages spatial consistency across phrase-, line-, and block-level masks.
By enforcing containment relationships across these granularities, the framework more faithfully reflects
the hierarchical structure of document layouts and supports more interpretable evidence attribution. Collectively, these components enable finer and more reliable localization than prior box-based grounding
methods, especially in documents containing irregular text geometry or densely structured layouts.
Beyond model design, this thesis also introduces GroundingDocQA, a large-scale data generation
pipeline for grounded document question answering, and GroundingDocQA-Bench, a benchmark for
the joint evaluation of answer correctness and grounding fidelity. The dataset construction pipeline is designed to support a broad range of document types, including text-rich pages, curved-text documents,
and graphics-heavy documents such as charts and infographics. This enables supervision for a wide
variety of grounded reasoning scenarios that extend beyond conventional extractive settings. The
benchmark complements standard answer-centric evaluation by explicitly assessing whether predicted
evidence is spatially accurate, semantically appropriate, and faithful to the generated answer. In this way,
the thesis promotes a more comprehensive evaluation paradigm for document question answering, one
that treats evidence quality as a first-class criterion alongside answer accuracy.
Extensive empirical results demonstrate that mask-based grounding constitutes a more expressive
and effective alternative to coarse region attribution for document question answering. The proposed
framework yields improvements in evidence faithfulness, better captures irregular and spatially distributed
support, and provides stronger grounding performance across challenging document layouts. More
broadly, this thesis argues that the next generation of document question answering systems should not be
evaluated solely by whether they answer correctly, but also by whether they can provide precise, verifiable,
and semantically appropriate evidence for their predictions. In this respect, the thesis advances document
question answering toward more interpretable, trustworthy, and practically deployable system

Abstract

Recent advances in text-to-image generation have enabled generation of highly realistic images. Text
alone, however, can only provide coarse control over the generation process. As a result, generative
models fail to serve applications which require precise control over various scene elements. One important scenario is multi-object scene control, where the aim is to generate a scene with precise control
over 3D object arrangements.
Most of the existing works for 3D control in generation fail to account for multi-object scenes,
primarily focusing on single object scenes only. Even methods that do allow some form of multi-object
control fail to generate inter-object occlusions, resulting in unrealistic images.
This thesis contributes two research works along this direction. Compass Control is the first work
which enables multi-object 3D orientation control in text-to-image generative models. This is achieved
by introducing special orientation tokens directly in the text prompt, one token per object in the scene.
The orientation token is output by a learnable MLP, which takes desired orientation for that object as
input. Further, a binding mechanism is introduced to bind each compass token to its corresponding
object, thus enabling disentangled multi-object control.
Compass Control, along with other works on 3D scene control, have a consistent limitation - they
fail to generate inter-object occlusions. We find that this limitation stems from lack of occlusion awareness in the input scene representations, which fail to model 'hidden objects' in the scene. Building on
these insights, we propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly
models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects
are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired
camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about
occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual tokens
derived from our rendered 3D representation. Furthermore, we apply masked self-attention to accurately
bind each object bounding box to its corresponding textual description, enabling accurate generation of
multiple objects without object attribute mixing.
The code and datasets for both these works have been made publicly available to encourage future
research in this direction.

Abstract

The global rise in pediatric myopia has intensified the need for objective biomarkers to understand
the functional retinal changes associated with refractive error. The pupillary light reflex (PLR) serves as
a non-invasive window into the health of rods, cones, and intrinsically photosensitive retinal ganglion
cells (ipRGCs). However, traditional pupillometry is often constrained by stationary hardware, narrow
spectral stimuli, and the challenges of testing pediatric populations. This thesis explores the use of a
Virtual Reality Head-Mounted Display (VR-HMD) with integrated infrared eye tracking as a portable,
high-precision platform for broad-spectrum chromatic pupillometry.
A cohort of 117 children (ages 6-14) was recruited and categorized into emmetropic, low myopic,
and high myopic groups. Participants were exposed to an eight-color paradigm (Red, Green, Blue,
Yellow, White, Purple, Indigo, and Black) generated on an OLED display, followed by a custom computational pipeline (PuReST algorithm) to extract quantitative metrics. The VR-HMD platform was
first validated by reproducing canonical PLR waveforms, demonstrating that it effectively disengages
the vergence-accommodation conflict through full-field stimulation.
Statistical analysis revealed that myopic children exhibit a significantly greater Maximum Constriction Amplitude (MCA) in response to high-illuminance stimuli (Yellow and White light) compared to
emmetropes, suggesting an altered dopaminergic modulation or increased photosensitivity to luminance.
Furthermore, myopes demonstrated a significantly faster Maximum Constriction Velocity (MCV) in response to long-wavelength Red light, supporting the hypothesis of an upregulated L-cone pathway adaptation to myopic defocus. A striking "anomalous" response was observed for Indigo light, where participants displayed three distinct phenotypes: constriction, dilation, or no change. A targeted follow-up
study (N=43 adults) deconstructed this effect, revealing that the anomaly was driven by a negative luminance contrast relative to the background, creating a perceptual conflict between chromatic constriction
and luminance-driven dilation. Finally, exploratory case studies of participants with strabismus, high
hyperopia, and amblyopia further demonstrated the system's sensitivity to diverse neuro-visual pathologies.
This research establishes the VR-HMD as a valid scientific instrument for quantitative pupillometry.
The findings provide new physiological evidence for functional neuroretinal differences in the myopic
eye and position VR-based chromatic pupillometry as a scalable solution for tele-ophthalmology and
the identification of predictive biomarkers for myopia progression.
Keywords: Chromatic Pupillometry, Virtual Reality, Myopia, Pediatric Vision, Retinal Photoreceptors, Pupillary Light Reflex.

Abstract

The rapid evolution of machine learning from task-specific systems to general-purpose decisionmakers has amplified concerns around transparency, reliability, and trust. Despite strong performance,
modern models remain opaque, often exhibiting unpredictable behavior and generating outputs that are
fluent yet unfaithful to the input or task constraints. These challenges necessitate principled oversight
mechanisms that can both explain model behavior and detect failures in a scalable and reliable manner.
This thesis addresses these challenges through the lens of behavioral faithfulness, focusing on two
complementary problems: (1) generating faithful explanations of model behavior, and (2) detecting unfaithful model outputs. We introduce SLED (Sample Learning to Explain Divergence), a framework
for producing global natural language explanations that characterize where two machine learning models converge and diverge in their predictions. SLED synthesizes representative inputs via gradient-based
optimization to identify regions of agreement and disagreement, and leverages large language models
to generate interpretable explanations grounded in these behavioral patterns. Unlike prior approaches
based on local feature attributions or training data access, SLED provides model-level insights with
minimal data requirements.
Experiments across 13 datasets and 11 model configurations show that SLED consistently outperforms baselines such as MaNtLE, LIME, and Anchors, achieving improvements of 18-24% in faithfulness and 10-22% in simulatability. Ablation studies demonstrate that the framework is sample-efficient,
robust to the choice of explainer models, and benefits from regularization that ensures realistic synthetic
samples. Human evaluations further validate its utility, with users achieving a simulatability score of
63.5% and reporting improved understanding of model behavior.
In parallel, this thesis formalizes hallucinations as failures of faithfulness, enabling systematic analysis across tasks. We propose a computationally efficient, NLI-based approach for detecting hallucinations in natural language generation tasks such as definition modeling, machine translation, and paraphrase generation. This framework models hallucination detection as a function of both model inputs
and training distributions, allowing it to generalize across domains.
Overall, this work establishes faithfulness as a unifying principle for trustworthy AI. By grounding
explanations in observable behavior and reframing hallucinations as measurable failures of that behavior, it bridges the gap between model decision processes and human understanding, contributing toward
more transparent and reliable machine learning systems.

Abstract

Remote laboratories have emerged as a critical component of modern engineering education by enabling students to perform hands-on experiments on real hardware from any location. However, building
reliable and secure IoT-driven remote labs at scale remains challenging due to heterogeneous hardware,
distributed communication channels, cloud dependencies, and continuous operational requirements.
This thesis addresses two fundamental aspects essential for maintaining dependable remote experimentation at IIIT Hyderabad's RLabs platform: security analysis of IoT-based remote labs and automated
experiment readiness testing using computer vision.
This thesis addresses these gaps through a systematic security assessment of the RLabs ecosystem by
analysing vulnerabilities across device firmware, network protocols, interoperability layers, and webfacing components. The study identifies common IoT weaknesses--such as insecure communication
paths, weak authentication, and unsafe hardware actuation pathways--and documents their relevance in
the context of educational remote labs. The resulting threat model highlights risks related to hardware
misuse, unauthorised control, and data exposure, offering practical mitigation strategies for strengthening system resilience.
The second part introduces an automated testing framework designed to eliminate manual validation of experiment nodes. The system emulates real user interaction through scripted workflows and
verifies hardware behaviour using a computer-vision pipeline tailored for each experiment. Integrated
into GitHub Actions, the framework supports continuous testing, early fault detection, and automated
reporting. Evaluations conducted on the Vanishing Rod and Focal Length experiments demonstrate that
the testing pipeline can reliably identify hardware faults, misalignment, incorrect actuation responses,
and degraded video feeds within minutes.
Together, the security analysis and automated testing system significantly enhance the reliability,
maintainability, and safety of remote laboratories. The thesis establishes a foundation for scalable deployments and paves the way for future extensions involving machine learning-based anomaly detection, intelligent fault prediction, and autonomous experiment calibration.

Abstract

The current project deals with the digitalization of the ICMR-NCTB toolkit through the use of digital
platforms while preserving the diagnostic validity of traditional pen-and-paper tools. It reports on the
design and development of a digital adaptation of verbal components within the ICMR-NCTB cognitive
assessment toolkit. The study aimed at translating the original test structure, its administration procedures, and scoring methods into an interactive, device-based format while preserving the functionality
of the traditional pen-paper tools. A prototype was developed that administered selected verbal tasks,
recorded participant responses by audio capture, and stored results in a structured digital format (JSON)
for subsequent and continuous evaluation. Preliminary validation was conducted with a cohort of young,
academically oriented students who were administered the selected verbal tasks in both digital and traditional modes. Results indicated that the digital version can replicate core assessment functions, although
more extended validation efforts are required to establish generalizability across diverse age groups and
clinical populations. Challenges identified include automated audio processing, user experience design,
and data-handling infrastructure, representing key avenues for further refinement. The contribution provides the basic framework for a fully digitized, scalable, clinically reliable tool for cognitive assessment,
matching current technological standards.

Abstract

Index Terms: Falsetto register, Glottal activity, High arousal speech, Intrinsic variations, Lombard speech, Modal register, Single frequency filtering, Speech synthesis, zero time windowing, zero frequency filtering.
High-arousal speech is typically produced in emotionally charged situations, during longdistance communication, or in the presence of background noise. At the extreme, this type of
speech is called "shouted speech." Emotions such as anger and happiness also commonly result
in high-arousal vocal expressions. This thesis analyzes, detects, and synthesizes high-arousal
(loud) speech.
The first part of the thesis investigates acoustic and glottal features critical for distinguishing high-arousal speech from neutral speech. Prior studies on high-arousal speech have utilized
segmental features such as spectral tilt and energy ratios across low and high-frequency bands.
Others have investigated subsegmental features derived from glottal flow waveforms, which are
often difficult to obtain. In contrast, this thesis adopts a suprasegmental approach. The glottal
activity is parameterized using the open quotient Oq, estimated from electroglottograph (EGG)
signals. In addition, an acoustic feature--the ratio of high to low-frequency spectral energy
(β) is derived using the zero-time windowing (ZTW) method, which provides high temporal resolution for spectral analysis. High-arousal speech is further analyzed across different
loudness levels using intrinsic variations among six sound units. Key acoustic parameters include the fundamental frequency (F0), the abruptness of excitation (Im) and short-term energy
(Se). These are computed around the glottal closure instants, and are derived using the zerofrequency filtering (ZFF) method. Building on this analysis, a novel approach is proposed for
high-arousal speech detection that does not require a reference to neutral speech, unlike most
machine learning-based methods. Instead, intrinsic suprasegmental variations in speech are
leveraged, offering a reference-independent and computationally efficient detection method.
Further, this thesis focuses on Lombard speech generation using different approaches. For
synthesis, both analysis-by-synthesis and data-driven methods are explored. Some studies use
spectral and source features to generate high-arousal speech, but their relative impact on synthesis is unknown. Hence, we study the fundamental frequency along with the duration modification of Lombard's speech. At the source level, previous studies have shown an increase in
fundamental frequency (F0), but the importance of F0 contour shape and F0 contour level in
Lombard speech is unclear. These perceptual studies carried out in this thesis highlight the role
of F0 contour shape and F0 contour level along with the durations. Six different experiments
were conducted by modifying durations and both F0 contour shape and level. These experiments are performed on samples from the Hurricane Challenge-2013 database. Perceptual
results show that the contribution of the F0 contour level is more than the F0 contour shape and
duration.
Finally, Lombard speech synthesis is achieved via transfer learning. Transfer learning eliminates the need for huge amounts of data to synthesize Lombard speech. This thesis uses
different neural parameter generation and vocoder models and compares the ability to adapt
to minimal Lombard speech that is available. In addition, a study based on metric learning
is conducted. Metric learning was initially studied on natural speech to enhance speech intelligibility in a noisy environment. The approach is adapted to the TTS system to improve
speech in noise. Metric learning is used to learn an intelligibility metric that generates a speech
with unique characteristics, such as increased loudness and higher pitch, compared to normal
speech. By aligning the learned metric with the acoustic features of the target speech domain,
the model improves its ability to synthesize realistic and intelligible Lombard speech from
textual input.

Abstract

Reliable and generalizable motion planning is essential for deploying robotic manipulators in realworld environments, where uncertainty, variability, and previously unseen conditions are common.
Classical optimization-based motion planners can be applied to any new scene without requiring taskspecific training, offering strong adaptability; however, they rely heavily on manually designed cost
functions tailored to each scene, which limits their scalability and performance in complex settings . In
contrast, deep-learning-based motion planning methods achieve high success rates across diverse trained
scenarios, but they often struggle to generalize to out-of-distribution scenes that differ from training data.
This thesis presents two complementary contributions that enhance generalization in robotic motion
planning and policy initialization for unseen in-distribution tasks, with extensions to multi-task learning.
First, we introduce EDMP (Ensemble-of-costs-guided Diffusion for Motion Planning) [7], detailed
in Chapter 3 which bridges classical planning with learned generative models. EDMP is trained on
diverse, kinematically valid trajectories and, at inference time, incorporates scene-specific cost signals,
such as collision and reachability costs, to guide the diffusion process. By leveraging an ensemble of
costs rather than a single handcrafted objective, EDMP produces feasible, diverse trajectories while
preserving the adaptability of classical planners. Our method exceeds the performance of classical
motion planners and matches the performance of state-of-the-art learned planning methods while offering
strong generalization to novel scenes. This work has been accepted for publication at IEEE International
Conference on Robotics and Automation (ICRA) 2024.
Second, we propose GenHyper, detailed in Chapter 4, a framework focused on zero-shot transfer
within an in-distribution task family. Using adversarially trained hypernetworks, GenHyper generates
strong policy initializations that perform well on unseen in-distribution tasks (demonstrated on MuJoCo
continuous-control benchmarks) with little or no environment interaction. Importantly, we also extend
this zero-shot framework to multi-task robotic manipulation settings using residual learning in latent
policy parameter space. Across benchmarks, GenHyper outperforms existing in-context, meta-RL, and
hypernetwork baselines on zero-shot transfer and enables faster adaptation with minimal interactions.
This work was accepted at the AAAI 2025 GenPlan and PRL Workshops.