Abstract

With the primary focus on evaluating the ef- fectiveness of large language models for au- tomatic reference-less translation assessment, this work presents our experiments on mimick- ing human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evalua- tion task where we performed zero-shot learn- ing, in-context example-driven learning, and fine-tuning of large language models to pro- vide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA- 2-13B) achieves a comparable or higher overall correlation with human judgments for the con- sidered Indian language pairs (Refer figure 1).

Abstract

The objective of the MTIL track in FIRE 2023 was to encourage the development of Indian Language to Indian Language (IL-IL) Neural Machine Translation models. The languages covered in the track included Hindi, Gujarati, Kannada, Odia, Punjabi, Urdu, Telugu, Kashmiri, and Sindhi. The track consists of two tasks: (i) a General Translation Task and (ii) a Domain specific Translation Task with Governance and Healthcare being the chosen domains. For the listed languages, we proposed 12 diverse language directions for the general domain translation task and 8 each for healthcare and governance domains. Participants were encouraged to submit models for one or more language pairs. We witnessed the creation of 34 distinct models spanning various language pairs and domains. Model assessments were conducted using five evaluation metrics: BLEU, CHRF, CHRF++, TER, and COMET. The submitted model outputs were ranked based on the CHRF score.

Abstract

The demand for daily form digitization requires extensive manual effort. This paper presents an efficient pipeline for digitizing Indic handwritten forms, minimizing human intervention. We validate the pipeline using textit{Hindi} and textit{Bengali} forms, creating a dedicated test set, textit{IIIT-Indic-Form-Test}. Our approach enables form capture and orientation alignment via smartphone, extracting printed and handwritten fields with OCR enhanced by template annotations. A predefined word list supports post-processing for fields like textit{name}, textit{state}, and textit{country}. We compare our pipeline with tools like Google Parser and Microsoft Azure and conduct ablation studies on form style, rotation, and handwriting variations. A GUI-based application for digitization is also developed. The code and model are publicly available at https://github.com/shaoncvit/Indic_Handwritten_Form_dataset.

Abstract

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of “naturalistic” event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Abstract

Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality evaluators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.

Abstract

The integration of Vision-Language Models (VLMs) into various applications has highlighted the importance of evaluating these models for inherent biases, especially along gender and racial lines. Traditional bias assessment methods in VLMs typically rely on accuracy metrics, assessing disparities in performance across different demographic groups. These methods, however, often overlook the impact of the model’s disabilities, like lack spatial reasoning, which may skew the bias assessment. In this work, we propose an approach that systematically examines how current bias evaluation metrics account for the model’s limitations. We introduce two methods that circumvent these disabilities by integrating spatial guidance from textual and visual modalities. Our experiments aim to refine bias quantification by effectively mitigating the impact of spatial reasoning limitations, offering a more accurate assessment of biases in VLMs.

Abstract

The implementation of Internet of Things (IoT) as Cyber-Physical Systems (CPS) is essential for the advancement of smart healthcare within a highly digital environment. However, the development of medical CPS (MCPS) poses various challenges, particularly in terms of dependability and security. The complex nature of medical IoT components encompasses both computational and physical aspects including secure reliability. To mitigate these issues, this article introduces a cyber engineering, an innovative approach, that combines principles from complex system design in order to establish a systematic framework for combining the cyber and physical layers of IoT and MCPS together. This integration aims to provide an unified system designing approach that can produce more secure and robust systems. By embracing the cyber engineering ideology, the healthcare industry can overcome reliability challenges and pave the way for the development of secure and dependable smart healthcare solutions, while ensuring security by design perspective. The approach also provides ideal future possibilities to the upcoming researchers.

Abstract

This paper contributes to the expanding research on toxicity in online games, exploring the relationship between game design and player behaviours in India. Using an ethnographic approach and in-depth interviews with gamers, we capture firsthand accounts of toxic gaming experiences, delving into game design, game mechanics, and player interactions intersecting to shape player behaviours. While previous research on toxicity in gaming environments has largely focused on the role of game design, aspects of gender and, to a lesser extent, race, primarily within the context of the Global North, our study shifts the focus to India and, by extension, the Global South. We seek to uncover broader dimensions of gaming toxicity that go beyond design and gender. Our analysis examines the underlying factors contributing to the prevalence and persistence of toxic behaviours in Indian gaming environments by highlighting the influence of cultural and contextual elements in producing toxic behaviours. Expanding the scope of research to include these overlooked aspects, our study aims to provide a more comprehensive understanding of the complexities of toxicity in online gaming spaces within the Indian context.

Abstract

Indoor air pollution monitoring has been a region of interest in recent times. Multiple Internet of Things (IoT) enabled devices are available for this purpose. With the growing number of sensors in our daily environment, huge amounts of data are being collected and pushed to the servers through the Internet. This study aims to reveal that seemingly trivial indoor air pollution data containing particulate matter, carbon dioxide, and temperature can reveal complex insights about an individual’s lifestyle. Data was collected over a period of four months in a real-world environment. The study demonstrates the inference of cooking activities by using machine learning and deep learning techniques. The study further demonstrates that different food items and culinary practices have different air pollution signatures, which can be identified and distinguished with great accuracy (>90%). In the practice of inferential analysis, it is not necessary to rely on data characterised by high frequency or granularity. Less detailed data like hourly averages, can be used to make meaningful conclusions that might intrude on an individual’s privacy. With the rapid advancement in machine learning and deep learning, a proactive approach to privacy is needed to ensure that the collected data and its usage do not intentionally or unintentionally breach individual privacy.

Abstract

Analysis of foot pressure, also known as plantar pressure analysis, plays a pivotal role in biomedical assessments related to posture and gait analysis. Extensive research has been conducted on leveraging this technique for clinical purposes, leading to the development of flexible pressure sensors. In this letter, we present the use of in-shoe flexible pressure sensor array for determining the nature of the walking surface. The sensor system is fabricated using eight low-cost and robust, in-shoe pressure sensors that leverage the piezoresistivity of velostat. The sensor array was characterized for four different surface types. Random Forest (RF) algorithm was used to classify the surfaces with 86% accuracy. Based on this analysis, we propose a novel method for analyzing various surfaces based on their attributes such as firmness, rigidity, and penetrability. Such a device can be used for ascertaining surface characteristics after construction, or playing surfaces in a stadium.