Abstract

We introduce Sketchtopia, a large-scale dataset and AI framework designed to explore goal-driven, multimodal communication through asynchronous interactions in a Pictionary-inspired setup. Sketchtopia captures natural human interactions, including freehand sketches, open-ended guesses, and iconic feedback gestures, showcasing the complex dynamics of cooperative communication under constraints. It features over 20K gameplay sessions from 916 players, capturing 263K sketches, 10K erases, 56K guesses and 19.4K iconic feedbacks. We introduce multimodal foundational agents with capabilities for generative sketching, guess generation and asynchronous communication. Our dataset also includes 800 human-agent sessions for benchmarking the agents. We introduce novel metrics to characterize collaborative success, responsiveness to feedback and inter-agent asynchronous communication. Sketchtopia pushes the boundaries of multimodal AI, establishing a new benchmark for studying asynchronous, goal-oriented interactions between humans and AI agents.

Abstract

Modern generative models often struggle to synthesize structured objects from detailed part specifications. They frequently produce anatomically implausible outputs or hallucinated components. We introduce PLATO, a novel two-stage framework that bridges this gap by enabling precise, part-controlled object generation. The first stage is PLayGen, our novel part layout generator which takes a list of parts and object category as input and synthesizes high-fidelity layouts of part bounding boxes. To enhance PLayGen's ability to learn inter-part relationships, we introduce novel structure-based loss functions. In the second stage, PLayGen's synthesized layout is used to condition a custom-tuned ControlNet-style adapter, enforcing spatial and connectivity constraints. This results in anatomically consistent, high-fidelity object generations containing precisely the user-specified parts. We further propose new part-level evaluation metrics to rigorously quantify adherence to part specifications. Extensive experiments show that PLATO significantly outperforms state-of-the-art generative models and produces structurally coherent objects in a controllable manner — marking a step forward in modular, part-driven asset generation.

Abstract

India's gig-based food delivery platforms, such as Swiggy and Zomato, provide crucial income to marginalised communities but also entrench workers in cycles of invisible labour. Through 14 semi-structured interviews, we analyse waiting time and repetitive UI itneractions as key burdens that contribute to 'digital discomfort' for gig based food delivery agents. We find that workers employ creative strategies to navigate algorithmic management, yet remain constrained by platform-side 'gamification' and system opacity. We propose worker-centered GUI automation as a potential intervention to reduce friction while preserving agency. In conclusion, this position paper argues for rethinking HCI approaches in the Global South to prioritise worker autonomy over efficiency-driven design optimisations.

Abstract

Typically, in computer assisted language learning systems, pronunciation assessment of learners’ reading is a critical component when they’re practicing with long paragraphs. Thus, automatic methods need to be developed for assessing spoken paragraphs i.e. monologues. However, there is a limitation on the availability of monologues data, corresponding ratings and the works to assess them. To address the data and label scarcity, this work proposes a method to obtain vector (ProMoVec) representations to assess the pronunciation of monologue considering the i-vector modelling based on synthetic monologues data. The synthetic monologues are proposed to obtain by concatenating the utterances which are semantically similar and corresponding ratings predicted based on entropy criterion which uses utterance ratings. Experiments with SpeechOcean762 reveal that the PromoVecs used in downstream assessment task with clustering shows a relative improvement of 11.6% compared to the baseline method.

Abstract

Automatic pronunciation assessment is a critical component in computer assisted language learning. Typically, modeling pronunciation assessment tasks need labels, which are difficult to obtain as it requires expert annotators. Thus, it is essential to build an accurate model with less annotated data. In this work, an approach is proposed that considers a few speech samples using the i-vector framework. Each sample, first, is lengthened by T factor by concatenating the augmented samples of the same speech. The augmentation is obtained using time-scale modification (TSM), pitch-scale modification (PSM) and both. Next, phoneme-level goodness-of-pronunciation scores of concatenated speech are converted to a vector (GoP2Vec) with the i-vector framework. Experiments on two datasets revealed that the proposed GoP2Vec outperforms the state-of-the-art (SOTA) unsupervised methods and is on par with the SOTA supervised methods when it is used to train a simple neural model with a few samples.

Abstract

FinFET, now firmly established in leading VLSI industries for their superior performance, exhibit heightened aging susceptibility that poses significant reliability challenges. The aggressive scaling of technology nodes has further compromised circuit reliability in recent years, highlighting the need for effective aging mitigation techniques. Recent advancements in the miniaturization of nanoscale technology have demonstrated the potential of optimizing performance parameters in standard cells using machine learning models and optimization algorithms through device sizing modifications. Building on this progress, we propose a methodology for optimizing performance parameters in 16nm high-performance (HP) FinFET for the first time. The approach leverages a multi-objective optimization algorithm framework to mitigate aging impacts across PVT variations while addressing NBTI and HCI effects by optimally adjusting FinFET design parameters, including channel length (lg), width (tfin), and height (hfin). With SPICE simulations, time-series datasets were generated to train machine learning models that achieved an R² score exceeding 0.99 and a mean absolute percentage error below 1% across standard cells. Our approach yields a significant simulation speedup and a reduction in simulation workload compared to traditional SPICE simulations. Using the proposed optimization algorithm framework, we improved the power-delay product (PDP) by up to 36.97% under non-aging conditions and 34.94% with aging considered with respect to the nominal dimension at the fresh year, demonstrating significant performance gains for FinFET-based standard cells. The experimental results on 12 distinct complex cells validate the aging mitigation across years.

Abstract

Deep learning models are susceptible to backdoor attacks involving malicious attackers perturbing a small subset of training data with a trigger to causes misclassifications. Various triggers have been used including semantic triggers that are easily realizable without requiring attacker to manipulate the image. The emergence of generative AI has eased generation of varied poisoned samples. Robustness across types of triggers is crucial to effective defense. We propose Prototype Guided Backdoor Defense (PGBD), a robust post-hoc defense that scales across different trigger types, including previously unsolved semantic triggers. PGBD exploits displacements in the geometric spaces of activations to penalize movements towards the trigger. This is done using a novel sanitization loss of a post-hoc finetuning step. The geometric approach scales easily to all types of attacks. PGBD achieves better performance across all settings. We also present the first defense against a new semantic attack on celebrity face images.

Abstract

Centrality measures offer a mechanism to quantify various notions of the importance of vertices in a network. These measures find application in social network analysis, the study of disease spread, and the like. Since the networks of interest are usually large, parallelism has become a powerful tool for computing these measures. However, for some of these applications, it is crucial to study how these centrality values change when there are changes to the underlying network. As a result, parallel algorithms that examine how to update these values due to changes to the graph are gaining research interest. In this paper, we study the update of the percolation centrality measure in a dynamic graph. We present parallel algorithms to handle changes to the percolation value of a batch of vertices and the addition and deletion of a batch of edges to the graph. We leverage the graphs' structural properties to improve our algorithms' performance. To our knowledge, we are the first to propose such algorithms for percolation centrality. We implement and benchmark our dynamic algorithms on a server with two AMD EPYC CPUs and an Nvidia A100 GPU. Our experiments on a collection of real-world graphs indicate that our algorithms achieve speedups of 8.79x and 2.82x on CPU and GPU, respectively, for edge updates, and 309.44x and 18.71x on CPU and GPU, respectively, for vertex percolation updates over state-of-the-art static algorithms for a batch of 10000 edges and vertices, respectively.

Abstract

Imagine sitting in a presentation, trying to follow the speaker while simultaneously scanning the slides for relevant information. While the entire slide is visible, identifying the relevant regions can be challenging. As you focus on one part of the slide, the speaker moves on to a new sentence, leaving you scrambling to catch up visually. This constant back-and-forth creates a disconnect between what’s being said and the most important visual elements, making it hard to absorb key details—especially in fast-paced or content-heavy presentations, as in a conference talk. This will require an understanding of slides, especially text, graphics, and layout. We introduce a method that automatically identifies and highlights the most relevant slide regions based on the speaker’s narrative. By analyzing spoken content and matching it with textual or graphical elements in the slides, our approach ensures better synchronization between what you hear and what you need to attend to. We explore different ways of solving this problem and assess their success and failure cases. Analyzing multimedia documents is emerging as a key requirement for seamless understanding of content-rich videos, such as educational videos and conference talks, by reducing cognitive strain and improving comprehension. Code and dataset available at: https://github.com/meghamariamkm2002/Slide_Highlight

Abstract

Handwritten Text Recognition (HTR) presents a significant challenge in computer vision due to various factors such as individual writing styles, noise, blur, and other imperfections in the text. This challenge is further exacerbated when dealing with languages using Indian scripts, which are characterized by complex character structures, extensive character inventories, and specific cultural nuances. In this study, we address these challenges by focusing on enhancing handwritten text recognition for ten Indic languages: Hindi, Bengali, Telugu, Tamil, Gujarati, Gurumukhi, Oriya, Kannada, Malayalam, and Urdu. We aim to improve recognition accuracy by leveraging the Permuted Autoregressive Sequence Model (PARSeq), an extension of the transformer-based model. Our results demonstrate the superiority of the PARSeq model over existing approaches, particularly in achieving state-of-the-art performance across most languages. Additionally, we investigate the efficacy of transfer learning from printed text to handwritten text, revealing its potential to enhance recognition performance. The trained models and code are publicly available at https://github.com/LalithaEvani/Indic-HTR-CVIP-2024.