Abstract

A fundamental aspect of compositional reasoning in a video is associating people and their actions across time. Recent years have seen great progress in general-purpose vision/video models and a move towards long-video understanding. While exciting, we take a step back and ask: are today’s models good at compositional reasoning on short videos? To this end, we introduce VELOCITI, a benchmark to study Video-LLMs by disentangling and assessing the comprehension of agents, actions, and their associations across multiple events. We adopt the Video-Language Entailment setup and propose StrictVLE that requires correct classification (rather than ranking) of the positive and negative caption. We evaluate several models and observe that even the best, LLaVA-OneVision (42.5%) and GPT-4o (44.3%), are far from human accuracy at 89.6%. Results show that action understanding lags behind agents, and negative captions created using entities appearing in the video perform worse than those obtained from pure text manipulation. We also present challenges with ClassicVLE and multiple-choice (MC) evaluation, strengthening our preference for StrictVLE. Finally, we validate that our benchmark requires visual inputs of multiple frames making it ideal to study video-language compositional reasoning.

Abstract

In decision making settings such as medical diagnosis, investment choices, or sentencing in a court of law, an individual’s background and experience influence their decisions. In the legal domain, multiple factors like personal bias/beliefs, recent events, and the contemporary state of mind could affect decision-making, leading to inconsistencies in judgments between and within jurisdictions. It is widely reported in the literature that anomalies and disparities exist in judicial decisions, such as bail grants and sentence impositions, stemming from implicit bias or other contextual factors. Notably, in domains like sales and marketing, data cube-based systems are being used to extract interesting trends and anomalies in subspaces from large multidimensional databases. In this paper, we extend the data cube framework to explore possible anomalies and disparities in court sentences by conducting experiments on a sample Indian judgment dataset of criminal cases. The results show that the data cube-based framework could identify anomalies in the legal domain. We hope this work will encourage researchers to investigate a compre- hensive data cube-based framework to reduce disparities and improve justice delivery worldwide.

Abstract

The widespread adoption of machine learning (ML) has brought forth diverse models with varying architectures, data requirements, introducing new challenges in integrating these systems into real-world applications. Traditional solutions often struggle to manage the complexities of connecting heterogeneous models, especially when dealing with varied technical specifications. These limitations are amplified in large-scale, collaborative projects where stakeholders contribute models with different technical specifications. To address these challenges, we developed LoCoML, a low-code framework designed to simplify the integration of diverse ML models within the context of the Bhashini Project - a large-scale initiative aimed at integrating AI-driven language technologies such as automatic speech recognition, machine translation, text-to-speech, and optical character recognition to support seamless communication across more than 20 languages. Initial evaluations show that LoCoML adds only a small amount of computational load, making it efficient and effective for large-scale ML integration. Our practical insights show that a low-code approach can be a practical solution for connecting multiple ML models in a collaborative environment.

Abstract

As Machine Learning (ML) makes its way into aviation, ML-enabled systems—including low-criticality systems—require a reliable certification process to ensure safety and performance. Traditional standards, like DO-178C, which are used for critical software in aviation, don’t fully cover the unique aspects of ML. This paper proposes a semi-automated certification approach, specifically for low-criticality ML systems, focusing on data and model validation, resilience assessment, and usability assurance while integrating manual and automated processes. Key aspects include structured classification to guide certification rigor on system attributes, an Assurance Profile that consolidates evaluation outcomes into a confidence measure the ML component, and methodologies for integrating human oversight into certification activities. Through a case study with a YOLOv8-based object detection system designed to classify military and civilian vehicles in real-time for reconnaissance and surveillance aircraft, we show how this approach supports the certification of ML systems in low-criticality airborne applications.

Abstract

Large Language Models (LLMs) often inherit biases from the web data they are trained on, which contains stereotypes and prejudices. Current methods for evaluating and mitigating these biases rely on bias-benchmark datasets. These benchmarks measure bias by observing an LLM’s behavior on biased statements. However, these statements lack contextual considerations of the situations they try to present. To address this, we introduce a contextual reliability framework, which evaluates model robustness to biased statements by considering the various contexts in which they may appear. We develop the Context-Oriented Bias Indicator and Assessment Score (COBIAS) to measure a biased statement’s reliability in detecting bias, based on the variance in model behavior across different contexts. To evaluate the metric, we augmented 2,291 stereotyped statements from two existing benchmark datasets by adding contextual information. We show that COBIAS aligns with human judgment on the contextual reliability of biased statements (Spearman’s 𝜌 = 0.65, 𝑝 = 3.4 ∗ 10−60) and can be used to create reliable benchmarks, which would assist bias mitigation works.

Abstract

Chemical reaction prediction, encompassing forward synthesis and retrosynthesis, stands as a fundamental challenge in organic synthesis. A widely adopted computational approach frames synthesis prediction as a sequence-to-sequence translation task, using the commonly used SMILES representation for molecules. The current evaluation of machine learning methods for retrosynthesis assumes perfect training data, overlooking imperfections in reaction equations in popular datasets, such as missing reactants, products, other physical and practical constraints such as temperature and cost, primarily due to a focus on the target molecule. This limitation leads to an incomplete representation of viable synthetic routes, especially when multiple sets of reactants can yield a given desired product. In response to these shortcomings, this study examines the prevailing evaluation methods and introduces comprehensive metrics designed to address imperfections in the dataset. Our novel metrics not only assess absolute accuracy by comparing predicted outputs with ground truth but also introduce a nuanced evaluation approach. We provide scores for partial correctness and compute adjusted accuracy through graph matching, acknowledging the inherent complexities of retrosynthetic pathways. Additionally, we explore the impact of small molecular augmentations while preserving chemical properties and employ similarity matching to enhance the assessment of prediction quality. We introduce SynFormer, a sequence-to-sequence model tailored for SMILES representation. It incorporates architectural enhancements to the original transformer, effectively tackling the challenges of chemical reaction prediction. SynFormer achieves a Top-1 accuracy of 53.2% on the USPTO-50k dataset, matching the performance of widely accepted models like Chemformer, but with greater efficiency by eliminating the need for pre-training.

Abstract

Design codes and standards provide a range of values for structural behavior factor for precast reinforced concrete (RC) structural systems. However, most of the codes give common value for all kinds of precast RC structural systems, due to the lack of information from experimental studies. Thus, the present paper highlights experimental investigation on behavior of different precast RC structural systems through quasi-static reversed cyclic tests on precast RC systems subjected to displacement-controlled loading. Different precast RC structural systems tested are precast RC wall-column system, precast RC double leaf system and precast RC beam-column joints with different configurations. Damage assessment has been carried out through observations and data obtained from the experiments, which has been co-related with the grade of damage, damage index and deformation capacity. Consequently, structural behavior factor for different precast RC structural systems has been determined. The structural behavior factor thus obtained would be beneficial for practising engineers and structural designers for analysis and design of different precast RC structural systems.

Abstract

Summarizing movie screenplays presents a unique set of challenges compared to standard document summarization. Screenplays are not only lengthy, but also feature a complex interplay of characters, dialogues, and scenes, with numerous direct and subtle relationships and contextual nuances that are difficult for machine learning models to accurately capture and comprehend. Recent attempts at screenplay summarization focus on fine-tuning transformer-based pre-trained models, but these models often fall short in capturing long-term dependencies and latent relationships, and frequently encounter the "lost in the middle" issue. To address these challenges, we introduce DiscoGraMS, a novel resource that represents movie scripts as a movie character-aware discourse graph (CaD Graph). This approach is well-suited for various downstream tasks, such as summarization, question-answering, and salience detection. The model aims to preserve all salient information, offering a more comprehensive and faithful representation of the screenplay's content. We further explore a baseline method that combines the CaD Graph with the corresponding movie script through a late fusion of graph and text modalities, and we present very initial promising results. We have made our code and dataset publicly available.

Abstract

Pronunciation correction is crucial for Text-to- Speech (TTS) systems in production. Traditional methods, which rely on phoneme sequence manipulation, are often cumber- some and error-prone. To address this, we propose Prompt-to- Correct, an editing-based methodology for pronunciation cor- rection in TTS systems using voice prompts. Our approach enables accurate, granular corrections at test-time without the need for additional training or fine-tuning. Unlike existing speech editing methods, we eliminate the need for external alignment to determine edit boundaries. By simply providing a correctly-pronounced reading of a word in any voice or accent, our system successfully corrects mispronunciations while maintaining continuity. Experimental results demonstrate that our method outperforms traditional baselines and state-of-the- art speech editing techniques. Speech samples are available at: https://prompt-to-correct.github.io/P2C

Abstract

Recently, the exponential growth in capability and pervasiveness of Large Language Models (LLMs) has led to significant work done in the field of code generation. However, this generation has been limited to code snippets. Going one step further, our desideratum is to automatically generate architectural components. This would not only speed up development time, but would also enable us to eventually completely skip the development phase, moving directly from design decisions to deployment. To this end, we conduct an exploratory study on the capability of LLMs to generate architectural components for Functions as a Service (FaaS), commonly known as serverless functions. The small size of their architectural components make this architectural style amenable for generation using current LLMs compared to other styles like monoliths and microservices. We perform the study by systematically selecting open source serverless repositories, masking a serverless function and utilizing state of the art LLMs provided with varying levels of context information about the overall system to generate the masked function. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains to evaluate code quality and the degree of similarity between human and LLM generated code respectively. Along with our findings, we also present a discussion on the path forward for using GenAI in architectural component generation.