keepitsimple at SemEval-2025 Task 3: LLM-Uncertainty based Approach for Multilingual Hallucination Span Detection
Saketh Reddy Vemula,Parameswari Krishnamurthy
International Workshop on Semantic Evaluation, SemEval, 2025
Abs | | bib Tex
@inproceedings{bib_keep_2025, AUTHOR = {Saketh Reddy Vemula, Parameswari Krishnamurthy}, TITLE = {keepitsimple at SemEval-2025 Task 3: LLM-Uncertainty based Approach for Multilingual Hallucination Span Detection}, BOOKTITLE = {International Workshop on Semantic Evaluation}. YEAR = {2025}}
Identification of hallucination spans in black-box language model generated text is essential for applications in the real world. A recent attempt at this direction is SemEval-2025 Task 3, Mu-SHROOM-a Multilingual Shared Task on Hallucinations and Related Observable Over-generation Errors. In this work, we present our solution to this problem, which capitalizes on the variability of stochastically-sampled responses in order to identify hallucinated spans. Our hypothesis is that if a language model is certain of a fact, its sampled responses will be uniform, while hallucinated facts will yield different and conflicting results. We measure this divergence through entropy-based analysis, allowing for accurate identification of hallucinated segments. Our method is not dependent on additional training and hence is cost-effective and adaptable. In addition, we conduct extensive hyperparameter tuning and perform error analysis, giving us crucial insights into model behavior.
Shared Task at Biolaysumm2025 : Extract then summarize approach Augmented with UMLS based Definition Retrieval for Lay Summary generation. Authors:
Aaradhya Gupta,Parameswari Krishnamurthy
Biomedical Language Processing, BioNLP, 2025
@inproceedings{bib_Shar_2025, AUTHOR = {Aaradhya Gupta, Parameswari Krishnamurthy}, TITLE = {Shared Task at Biolaysumm2025 : Extract then summarize approach Augmented with UMLS based Definition Retrieval for Lay Summary generation. Authors:}, BOOKTITLE = {Biomedical Language Processing}. YEAR = {2025}}
We present LayForge, a two-track lay sum-
mary generation system developed for the Bio-
LaySumm2025 shared task(Xiao et al., 2025)
. Task 1.1 addresses the lay summarization
using only the internal content of the article,
while Task 1.2 augments this process with do-
main knowledge such as biomedical definitions
and concept explanations. BioLaySumm em-
ploys a modular architecture that leverages
large language models (LLMs), a BioBERT-
based named entity recognizer (NER), and the
UMLS(Bodenreider, 2004) knowledge base to
create readable, informative, and faithful lay
summaries. Our system shows strong perfor-
mance on both tasks when evaluated on the
PLOS and elife subset(Goldsack et al., 2022),
particularly in readability and factuality metrics.
The architecture illustrates how modularity and
domain adaptation can be effectively combined
for accessible biomedical communication.
TableWise at SemEval-2025 Task 8: LLM Agents for TabQA
Harsh Bansal,Aman Raj,Akshit Sharma,Parameswari Krishnamurthy
International Workshop on Semantic Evaluation, SemEval, 2025
@inproceedings{bib_Tabl_2025, AUTHOR = {Harsh Bansal, Aman Raj, Akshit Sharma, Parameswari Krishnamurthy}, TITLE = {TableWise at SemEval-2025 Task 8: LLM Agents for TabQA}, BOOKTITLE = {International Workshop on Semantic Evaluation}. YEAR = {2025}}
In this study, we present our approach to SemEval Task 8: Question Answering over Tabular Data (Os’es Grijalba et al., 2025), where we develop a Large Language Models(LLM) based agent capable of answering questions over tabular data. Our agent leverages a set of custom defined tools incorporating structured table parsing and reasoning mechanisms to enhance semantic understanding of tabular data. Extending our methodology, we apply chain-of- thought prompting to further refine it’s understanding of the task. Our findings suggest that LLM-based agents, when properly adapted, can significantly improve their table-based question answering capabilities.
Progressive Perturbation with KTO for Enhanced Machine Translation of Indian Languages
Yash Bhaskar,Ketaki Mangesh Shetye,Vandan Mujadia,Dipti Mishra Sharma,Parameswari Krishnamurthy
MT Summit, MT-S, 2025
Abs | | bib Tex
@inproceedings{bib_Prog_2025, AUTHOR = {Yash Bhaskar, Ketaki Mangesh Shetye, Vandan Mujadia, Dipti Mishra Sharma, Parameswari Krishnamurthy}, TITLE = {Progressive Perturbation with KTO for Enhanced Machine Translation of Indian Languages}, BOOKTITLE = {MT Summit}. YEAR = {2025}}
This study addresses the critical challenge of
data scarcity in machine translation for Indian
languages, particularly given their morpholog-
ical complexity and limited parallel data. We
investigate an effective strategy to maximize
the utility of existing data by generating nega-
tive samples from positive training instances us-
ing a progressive perturbation approach. This
is used to align the model with preferential
data using Kahneman-Tversky Optimization
(KTO). Comparing it against traditional Su-
pervised Fine-Tuning (SFT), we demonstrate
how generating negative samples and leverag-
ing KTO enhances data efficiency. By creat-
ing rejected samples through progressively per-
turbed translations from the available dataset,
we fine-tune the Llama 3.1 Instruct 8B model
using QLoRA across 16 language directions, in-
cluding English, Hindi, Bangla, Tamil, Telugu,
and Santali. Our results show that KTO-based
preference alignment with progressive pertur-
bation consistently outperforms SFT, achieving
significant gains in translation quality with an
average BLEU increase of 1.84 to 2.47 and
CHRF increase of 2.85 to 4.01 compared to
SFT for selected languages, while using the
same positive training samples and under simi-
lar computational constraints. This highlights
the potential of our negative sample genera-
tion strategy within KTO, especially in low-
resource scenarios.
ExagTTS: An Approach Towards Controllable Word Stress Incorporated TTS for Exaggerated Synthesized Speech Aiding Second Language Learners
Anindita Mondal,Surtani Monica,Anil Kumar Vuppala,Parameswari Krishnamurthy,Chiranjeevi Yarra
Annual Conference of the International Speech Communication Association, INTERSPEECH, 2025
@inproceedings{bib_Exag_2025, AUTHOR = {Anindita Mondal, Surtani Monica, Anil Kumar Vuppala, Parameswari Krishnamurthy, Chiranjeevi Yarra}, TITLE = {ExagTTS: An Approach Towards Controllable Word Stress Incorporated TTS for Exaggerated Synthesized Speech Aiding Second Language Learners}, BOOKTITLE = {Annual Conference of the International Speech Communication Association}. YEAR = {2025}}
Computer-Assisted Language Learning systems provide speech exaggerating at mispronounced word locations as a feedback to the L2 learners. Traditionally, expert speakers recordings (which limit the scalability) are considered for this task though there are advancements in text-to-speech (TTS) that can generate native like natural sounding speech. To address these, this work proposes two novel controllable strategies for scalable speech exaggeration. One strategy is direct speech exaggeration that incorporates the proposed label conditioned tokenization in GlowTTS. Another strategy is cascading the state-of-the-art TTS to a WORLD vocoder with proposed energy and duration modifications. A subset of Tatoeba corpus, that we annotated with prominent words, is used for experimentation. Automatic and manual assessment reveals that the exaggerated speech quality from both direct and cascaded strategy with duration modification is closer to the prominent words in the native speaker's speech.
MTNLP-IIITH: Machine Translation for Low-Resource Indic Languages
Abhinav P M,Ketaki Mangesh Shetye,Parameswari Krishnamurthy
Conference on Machine Translation, WMT, 2024
@inproceedings{bib_MTNL_2024, AUTHOR = {Abhinav P M, Ketaki Mangesh Shetye, Parameswari Krishnamurthy}, TITLE = {MTNLP-IIITH: Machine Translation for Low-Resource Indic Languages}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2024}}
Machine Translation for low-resource languages presents significant challenges, primarily due to limited data availability. We have a baseline model and a primary model. For the baseline model, we first fine-tune the mBART model (mbart-large-50-many-to-many-mmt) for the language pairs English-Khasi, Khasi-English, English-Manipuri, and Manipuri-English. We then augment the dataset by back-translating from Indic languages to English. To enhance data quality, we fine-tune the LaBSE model specifically for Khasi and Manipuri, generating sentence embeddings and applying a cosine similarity threshold of 0.84 to filter out low-quality back-translations. The filtered data is combined with the original training data and used to further fine-tune the mBART model, creating our primary model. The results show that the primary model slightly outperforms the baseline model, with the best performance achieved by the English-to-Khasi (en-kh) primary model, which recorded a BLEU score of 0.0492, a chrF score of 0.3316, and a METEOR score of 0.2589 (on a scale of 0 to 1), with similar results for other language pairs.
Yes-MT’s Submission to the Low-Resource Indic Language Translation Shared Task in WMT 2024
Parameswari Krishnamurthy,Yash Bhaskar
Conference on Machine Translation, WMT, 2024
@inproceedings{bib_Yes-_2024, AUTHOR = {Parameswari Krishnamurthy, Yash Bhaskar}, TITLE = {Yes-MT’s Submission to the Low-Resource Indic Language Translation Shared Task in WMT 2024}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2024}}
This paper presents the systems submitted by the Yes-MT team for the Low-Resource Indic Language Translation Shared Task at WMT 2024, focusing on translating between English and the Assamese, Mizo, Khasi, and Manipuri languages. The experiments explored various approaches, including fine-tuning pre-trained models like mT5 and IndicBart in both Multilingual and Monolingual settings, LoRA finetune IndicTrans2, zero-shot and few-shot prompting with large language models (LLMs) like Llama 3 and Mixtral 8x7b, LoRA Supervised Fine Tuning Llama 3, and training Transformers from scratch. The results were evaluated on the WMT23 Low-Resource Indic Language Translation Shared Task’s test data using SacreBLEU and CHRF highlighting the challenges of low-resource translation and show the potential of LLMs for these tasks, particularly with fine-tuning.
Exploring News Summarization and Enrichment in a Highly Resource-Scarce Indian Language: A Case Study of Mizo
Abhinaba Bala,Urlana Ashok,Rahul Mishra,Parameswari Krishnamurthy
Workshop on Indian Language Data, ILD-W, 2024
@inproceedings{bib_Expl_2024, AUTHOR = {Abhinaba Bala, Urlana Ashok, Rahul Mishra, Parameswari Krishnamurthy}, TITLE = {Exploring News Summarization and Enrichment in a Highly Resource-Scarce Indian Language: A Case Study of Mizo}, BOOKTITLE = {Workshop on Indian Language Data}. YEAR = {2024}}
Obtaining sufficient information in one’s mother tongue is crucial for satisfying the information needs of the users. While high-resource languages have abundant online resources, the situation is less than ideal for very low-resource languages. Moreover, the insufficient reporting of vital national and international events continues to be a worry, especially in languages with scarce resources, like Mizo. In this paper, we conduct a study to investigate the effectiveness of a simple methodology designed to generate a holistic summary for Mizo news articles, which leverages English-language news to supplement and enhance the information related to the corresponding news events. Furthermore, we make available 500 Mizo news articles and corresponding enriched holistic summaries. Human evaluation confirms that our approach significantly enhances the information coverage of Mizo news articles.
Fine-tuning Pre-trained Named Entity Recognition Models For Indian Languages
Sankalp Sanjay Bahad,Pruthwik Mishra,Karunesh Arora,Rakesh Chandra Balabantaray,Dipti Mishra Sharma,Parameswari Krishnamurthy
NAACL Student Research Workshop, NAACL-SRW, 2024
@inproceedings{bib_Fine_2024, AUTHOR = {Sankalp Sanjay Bahad, Pruthwik Mishra, Karunesh Arora, Rakesh Chandra Balabantaray, Dipti Mishra Sharma, Parameswari Krishnamurthy}, TITLE = {Fine-tuning Pre-trained Named Entity Recognition Models For Indian Languages}, BOOKTITLE = {NAACL Student Research Workshop}. YEAR = {2024}}
Named Entity Recognition (NER) is a use- ful component in Natural Language Process- ing (NLP) applications. It is used in various tasks such as Machine Translation, Summa- rization, Information Retrieval, and Question- Answering systems. The research on NER is centered around English and some other ma- jor languages, whereas limited attention has been given to Indian languages. We analyze the challenges and propose techniques that can be tailored for Multilingual Named Entity Recog- nition for Indian Languages. We present a hu- man annotated named entity corpora of ∼40K sentences for 4 Indian languages from two of the major Indian language families. Addition- ally,we present a multilingual model fine-tuned on our dataset, which achieves an F1 score of ∼0.80 on our dataset on average. We achieve comparable performance on completely unseen benchmark datasets for Indian languages which affirms the usability of our model.
Transformer-based Context Aware Morphological Analyzer for Telugu
Chelpuri Abhijith,Dasari Priyanka,Nagaraju Vuppala,Mounika Marreddy,Radhika Mamidi,Parameswari Krishnamurthy
Workshop on Speech and Language Technologies for Dravidian Languages, DravidianLangTech, 2023
@inproceedings{bib_Tran_2023, AUTHOR = {Chelpuri Abhijith, Dasari Priyanka, Nagaraju Vuppala, Mounika Marreddy, Radhika Mamidi, Parameswari Krishnamurthy}, TITLE = {Transformer-based Context Aware Morphological Analyzer for Telugu}, BOOKTITLE = {Workshop on Speech and Language Technologies for Dravidian Languages}. YEAR = {2023}}
This paper addresses the challenges faced by Indian languages in leveraging deep learning for natural language processing (NLP) due to limited resources, annotated datasets, and Transformer-based architectures. We specifically focus on Telugu and aim to construct a Telugu morph analyzer dataset comprising 10,000 sentences. Furthermore, we assess the performance of established multilingual Transformer models (m-Bert, XLMR, IndicBERT) and mono-lingual Transformer model BERT-Te (trained from scratch on an extensive Telugu corpus comprising 80,15,588 sentences). Our findings demonstrate the efficacy of Transformer-based representations pre-trained on Telugu data improved the performance of the Telugu morph analyzer, surpassing existing multi-lingual approaches. This highlights the necessity of developing dedicated corpora, annotated datasets, and machine learning models in a mono-lingual setting. Using our dataset, we present benchmark results for the Telugu morph analyzer achieved through simple fine-tuning on BERT-Te. The morph analyzer dataset 1 and codes are open-sourced and available here.
Assessing Translation capabilities of Large Language Models involving English and Indian Languages
Vandan Mujadia,Urlana Ashok,Yash Bhaskar,Penumalla Aditya Pavani,Kukkapalli Shravya,Parameswari Krishnamurthy,Dipti Mishra Sharma
Technical Report, arXiv, 2023
@inproceedings{bib_Asse_2023, AUTHOR = {Vandan Mujadia, Urlana Ashok, Yash Bhaskar, Penumalla Aditya Pavani, Kukkapalli Shravya, Parameswari Krishnamurthy, Dipti Mishra Sharma}, TITLE = {Assessing Translation capabilities of Large Language Models involving English and Indian Languages}, BOOKTITLE = {Technical Report}. YEAR = {2023}}
Generative Large Language Models (LLMs) have achieved remarkable advancements in var- ious NLP tasks. In this work, our aim is to ex- plore the multilingual capabilities of large lan- guage models by using machine translation as a task involving English and 22 Indian languages. We first investigate the translation capabilities of raw large language models, followed by ex- ploring the in-context learning capabilities of the same raw models. We fine-tune these large language models using parameter efficient fine- tuning methods such as LoRA and additionally with full fine-tuning. Through our study, we have identified the best performing large lan- guage model for the translation task involving LLMs, which is based on LLaMA.
AbhiPaw@ DravidianLangTech: Abusive Comment Detection in Tamil and Telugu using Logistic Regression
Abhinaba Bala,Parameswari Krishnamurthy
Workshop on Speech and Language Technologies for Dravidian Languages, DravidianLangTech, 2023
@inproceedings{bib_Abhi_2023, AUTHOR = {Abhinaba Bala, Parameswari Krishnamurthy}, TITLE = {AbhiPaw@ DravidianLangTech: Abusive Comment Detection in Tamil and Telugu using Logistic Regression}, BOOKTITLE = {Workshop on Speech and Language Technologies for Dravidian Languages}. YEAR = {2023}}
Abusive comments in online platforms have become a significant concern, necessitating the development of effective detection systems. However, limited work has been done in low resource languages, including Dravidian languages. This paper addresses this gap by focusing on abusive comment detection in a dataset containing Tamil, Tamil-English and TeluguEnglish code-mixed comments. Our methodology involves logistic regression and explores suitablef embeddings to enhance the performance of the detection model. Through rigorous experimentation, we identify the most effective combination of logistic regression and embeddings. The results demonstrate the performance of our proposed model, which contributes to the development of robust abusive comment detection systems in low resource language settings. Keywords: Abusive comment detection, Dravidian languages, logistic regression, embeddings, low resource languages, code-mixed dataset.
AbhiPaw@ DravidianLangTech: Fake News Detection in Dravidian Languages using Multilingual BERT
Abhinaba Bala,Parameswari Krishnamurthy
Workshop on Speech and Language Technologies for Dravidian Languages, DravidianLangTech, 2023
@inproceedings{bib_Abhi_2023, AUTHOR = {Abhinaba Bala, Parameswari Krishnamurthy}, TITLE = {AbhiPaw@ DravidianLangTech: Fake News Detection in Dravidian Languages using Multilingual BERT}, BOOKTITLE = {Workshop on Speech and Language Technologies for Dravidian Languages}. YEAR = {2023}}
This study addresses the challenge of detecting fake news in Dravidian languages by leveraging Google’s MuRIL (Multilingual Representations for Indian Languages) model. Drawing upon previous research, we investigate the intricacies involved in identifying fake news and explore the potential of transformer-based models for linguistic analysis and contextual understanding. Through supervised learning, we fine-tune the ”muril-base-cased” variant of MuRIL using a carefully curated dataset of labeled comments and posts in Dravidian languages, enabling the model to discern between original and fake news. During the inference phase, the fine-tuned MuRIL model analyzes new textual content, extracting contextual and semantic features to predict the content’s classification. We evaluate the model’s performance using standard metrics, highlighting the effectiveness of MuRIL in detecting fake news in Dravidian languages and contributing to the establishment of a safer digital ecosystem. Keywords: fake news detection, Dravidian languages, MuRIL, transformer-based models, linguistic analysis, contextual understanding.x
AbhiPaw@ DravidianLangTech: Multimodal Abusive Language Detection and Sentiment Analysis using Transformer based architecture
Abhinaba Bala,Parameswari Krishnamurthy
Workshop on Speech and Language Technologies for Dravidian Languages, DravidianLangTech, 2023
@inproceedings{bib_Abhi_2023, AUTHOR = {Abhinaba Bala, Parameswari Krishnamurthy}, TITLE = {AbhiPaw@ DravidianLangTech: Multimodal Abusive Language Detection and Sentiment Analysis using Transformer based architecture}, BOOKTITLE = {Workshop on Speech and Language Technologies for Dravidian Languages}. YEAR = {2023}}
Detecting abusive language in multimodal videos has become a pressing need in ensuring a safe and inclusive online environment. This paper focuses on addressing this challenge through the development of a novel approach for multimodal abusive language detection in Tamil videos and sentiment analysis for Tamil/Malayalam videos. By leveraging state-of-the-art models such as Multiscale Vision Transformers (MViT) for video analysis, OpenL3 for audio analysis, and the bert-basemultilingual-cased model for textual analysis, our proposed framework integrates visual, auditory, and textual features. Through extensive experiments and evaluations, we demonstrate the effectiveness of our model in accurately detecting abusive content and predicting sentiment categories. The limited availability of effective tools for performing these tasks in Dravidian Languages has prompted a new avenue of research in these domains. Keywords: abusive language detection, sentiment analysis, multimodal analysis, video analysis, Dravidian languages.
