Faculty - Dashboard

@inproceedings{bib_Refl_2026, AUTHOR = {Jain, Anupreksha and Shrivastava, Manish }, TITLE = {Reflect-SQL: A Self-Reflection Based Framework for Text-to-SQL}, BOOKTITLE = {Research and Applications of Foundation Models for Data Mining and Affective Computing Workshop}. YEAR = {2026}}

Democratizing data access through natural language is a crucial goal for modern enterprises, but the practical adoption of Text-to-SQL is critically hindered by real-world complexities: 1. Obscure and large database schemas, 2. Ineffective retrieval of relevant tables and columns due to structured setting of schemas and vague user query, 3. Generation of syntactically or logically flawed SQL due to a lack of robust validation and correction mechanism. To address these systemic challenges, we introduce Reflect-SQL, a novel framework for Text to SQL, grounded in multi-stage self-reflection approach to develop understanding of obscure schema using a knowledge base, setup a process for effective retrieval and system to generate syntactically/semantically SQL. Instead of a single-pass attempt, our system employs an LLM-as-a-judge driven scoring mechanism within interconnected feedback loops to iteratively refine the results at every stage. A feedback-driven retrieval loop refines the user's natural language query, while a synthesis loop validates and corrects the SQL and finally, an entailment loop optimizes the end-to-end process and continuously enriches the knowledge base. By integrating these layers of reflection, Reflect-SQL bridges the critical gap between user intent and complex data. On the challenging BIRD benchmark, our framework achieves an execution accuracy of 72.03%, significantly outperforming state-of-the-art baselines, demonstrating a major leap in reliability for enterprise applications.

@inproceedings{bib_PoET_2026, AUTHOR = {SURANGE, NIRMAL and Shrivastava, Manish }, TITLE = {PoETIC: A Re-framing of Context Dependent Emotion Detection}, BOOKTITLE = {International Conference on Language Resources and Evaluation workshop}. YEAR = {2026}}

Emotion classification has been extensively studied, with numerous datasets enabling progress in both textual and multimodal settings. However, most existing text-based resources treat emotion as an utterance-level property, assuming that the emotional content is fully encoded in the sentence itself. This assumption is problematic: in the absence of paralinguistic cues such as prosody, facial expressions, or emojis, textual emotions are often highly context-dependent. Many utterances lack explicit emotion markers, and even when present, such cues may be overridden by broader situational context. Sentence-level emotion annotation, thus, is driven by the annotator's ability to imagine the context in which the given utterance would elicit a given emotion. An utterance may be able to express an emotion completely (Emotion Obvious), or it can express an emotion when imagined in a certain context (Emotion Plausible). Also, for an utterance, certain emotions might be implausible to express given the specific wording of a sentence (Emotion-Implausible). To address these issues, we create a new paradigm for emotion classification by categorizing utterance and emotion pairs into context-dependency classes. We present the PoETIC benchmark dataset, where sentences in the GoEmotions dataset are human-annotated for the three aforementioned classes across seven emotions (Fear, Anger, Sadness, Joy, Disgust, Surprise, and Neutral). We observe that gold-tagged emotions in GoEmotions do not have a clear correlation with human judgment with respect to the ability to express other emotions, given different contexts. Human annotators identify significantly more plausible emotions for a given utterance if asked to imagine a plausible context per utterance-emotion pair. We also present baselines using three popular large language models and two "small" language models in zero-shot and few-shot settings on the benchmark dataset.

@inproceedings{bib_From_2026, AUTHOR = {Adibhatla, Hiranmai Sri and Mukherjee, Ananya and Shrivastava, Manish }, TITLE = {From jargon to clarity: bridging understanding through graded simplification of legal data}, BOOKTITLE = {Artificial Intelligence and Law}. YEAR = {2026}}

Legal documents are notorious for their length, density, and jargon-heavy language, making them challenging to navigate and comprehend. This highlights a strong need for clear and accessible documentation for a diverse audience. Text simplification at multiple levels, tailored to individuals with diverse backgrounds and expertise, is essential in making legal content universally accessible. To this effect, in this work, we focus on paragraph-level simplification of legal contracts and introduce Graded Simplification for Legal Data, a framework that adapts contract clauses across three competency levels: Skilled, Intermediate, and Basic. We employ Large language models (LLMs) to perform graded simplification, supported by a Token efficient Compression mechanism that incrementally encodes document context across paragraphs within fixed tokens, making it well suited to lengthy contracts. To address the challenge of reliably evaluating legal simplification at scale, we design a multi-criteria evaluation framework that jointly assesses readability, lexical simplicity, semantic preservation, and entailment. This framework enables the creation of our key resource, the SimpLegal dataset, an English-language preference dataset of paragraph-level contract simplifications. Using this dataset for Direct Preference Optimization (DPO), we achieve notable gains ( 5 points) in readability and simplicity over zero-shot prompting-based baselines. Collectively, these contributions underscore the importance of graded, paragraph-level simplification for contracts and demonstrate that small and medium-scale LLMs, when fine-tuned on preference data, can achieve performance comparable to larger models, providing a scalable pathway for accessible and comprehensible legal documentation. Our code and dataset are made available at https://github.com/GSLD-SimpLegal/FromJargonToClarity.git.

@inproceedings{bib_Leve_2026, AUTHOR = {Muhopadhyay, Sabyasachi and Shrivastava, Manish and Vaidhyanathan, Karthik and Srivatsa, Kalahasti Ganesh }, TITLE = {Leveraging LLMs for Generating Infrastructure as Code: An Exploratory Empirical Study}, BOOKTITLE = {Innovations in Software Engineering Conference}. YEAR = {2026}}

Infrastructure as Code (IaC) supports automation in cloud systems, yet writing correct Terraform remains demanding because provider rules and configuration patterns are easy to misapply. This study examines how far current language models can assist by generating configurations from natural-language intent. Using queries reconstructed from 460 real Terraform files, we compare several open and closed models under zero-shot, few-shot, and fine-tuned settings. Larger closed models generally produced the most complete outputs without examples(zero-shot), while smaller open models required limited few-shot prompting or fine-tuning to reach similar structure and coverage. Although the generated configurations were not always deployment ready, many were close enough for engineers to adapt with modest revision. All prompts, data, and evaluation artifacts are available at https://github.com/llmiac-2024/llm-iac

@inproceedings{bib_Brid_2025, AUTHOR = {Asapu, Likhith and Prashant, Kodali and Dua, Ashna and Kavitha, Kapil Rajesh and Shrivastava, Manish }, TITLE = {Bridging Laughter Across Languages: Generation of Hindi-English Code-mixed Puns}, BOOKTITLE = {Workshop on Computational Humor}. YEAR = {2025}}

Puns, as a linguistic phenomenon, hold significant importance in both humor and language comprehension. While extensive research has been conducted in the realm of pun generation in English, there exists a notable gap in the exploration of pun generation within code-mixed text, particularly in Hindi-English code-mixed text. This study addresses this gap by offering a computational method specifically designed to create puns in Hindi-English code-mixed text. In our investigation, we delve into three distinct methodologies aimed at pun generation utilizing pun-alternate word pairs. Furthermore, this novel dataset, HECoP, comprising of 2000 human-annotated sentences serves as a foundational resource for training diverse pun detection models. Additionally, we developed a structured pun generation pipeline capable of generating puns from a single input word without relying on predefined word pairs. Through rigorous human evaluations, our study demonstrates the efficacy of our proposed models in generating code-mixed puns. The findings presented herein lay a solid groundwork for future endeavours in pun generation and computational humor within diverse linguistic contexts.

@inproceedings{bib_Why__2025, AUTHOR = {Mukherjee, Ananya and Yadav, Saumitra and Shrivastava, Manish }, TITLE = {Why should only High-Resource-Languages have all the fun? Pivot Based Evaluation in Low Resource Setting}, BOOKTITLE = {International Conference on Computational Linguistics}. YEAR = {2025}}

Evaluating machine translation (MT) systems for low-resource languages has long been a challenge due to the limited availability of evaluation metrics and resources. As a result, researchers in this space have relied primarily on lexical-based metrics like BLEU, TER, and ChrF, which lack semantic evaluation. In this first-of-its-kind work, we propose a novel pivot-based evaluation framework that addresses these limitations; after translating low-resource language outputs into a related high-resource language, we leverage advanced neural and embedding-based metrics for more meaningful evaluation. Through a series of experiments using five low-resource languages: Assamese, Manipuri, Kannada, Bhojpuri, and Nepali, we demonstrate how this method extends the coverage of both lexical-based and embedding-based metrics, even for languages not directly supported by advanced metrics. Our results show that the differences between direct and pivot-based evaluation scores are minimal, proving that this approach is a viable and effective solution for evaluating translations in endangered and low-resource languages. This work paves the way for more inclusive, accurate, and scalable MT evaluation for underrepresented languages, marking a significant step forward in this under-explored area of research.

@inproceedings{bib_Corr_2025, AUTHOR = {Jindal, Akshett Rai and Reddy, Oota Subba and Mondal, Ishani and Pahwa, Khushbu and GNVV, Satya Sai Srinath Namburi and Shrivastava, Manish and Singh, Maneesh and Surampudi, Bapi Raju and Gupta, Manish }, TITLE = {Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)}, BOOKTITLE = {International Conference on Learning Representations}. YEAR = {2025}}

Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models—through increased size, instruction-tuning, and multimodality—has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate the brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions (like image captioning, visual question answering, etc.) show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results indicate that enhancing MLLMs' ability to capture more task-specific information could allow for better differentiation between various types of instructions, and hence improve their precision in predicting brain responses.

@inproceedings{bib_Anal_2025, AUTHOR = {Menon, Abhinav S and Shrivastava, Manish and Krueger, David S. and Lubana, Ekdeep S. }, TITLE = {Analyzing (In)Abilities of SAEs via Formal Languages}, BOOKTITLE = {North American Association for Computational Linguistics}. YEAR = {2025}}

Autoencoders have been used for finding inter- pretable and disentangled features underlying neural network representations in both image and text domains. While the efficacy and pitfalls of such methods are well-studied in vision, there is a lack of corresponding results, both qualitative and quantitative, for the text domain. We aim to address this gap by training sparse autoencoders (SAEs) on a synthetic testbed of formal languages. Specifically, we train SAEs on the hidden representations of models trained on formal languages (Dyck-2, Expr, and English PCFG) under a wide variety of hyperparameter settings, finding interpretable latents often emerge in the features learned by our SAEs. However, similar to vision, we find performance turns out to be highly sensitive to inductive biases of the training pipeline. Moreover, we show latents correlating to certain features of the input do not always induce a causal impact on model’s computation. We thus ar- gue that causality has to become a central tar- get in SAE training: learning of causal features should be incentivized from the ground- up. Motivated by this, we propose and perform preliminary investigations for an approach that promotes learning of causally relevant features in our formal language setting.

@inproceedings{bib_MAPW_2025, AUTHOR = {Mukhopadhyay, Srija and Rajgaria, Abhishek and Khatiwada, Prerana and Shrivastava, Manish and Roth, Dan and Gupta, Vivek }, TITLE = {MAPWise: Evaluating Vision-Language Models for Advanced Map Queries}, BOOKTITLE = {North American Association for Computational Linguistics}. YEAR = {2025}}

Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing around 1000 questions. Our benchmark incorporates 43 diverse question templates, requiring nuanced understanding of relative spatial relationships, intricate map features, and complex reasoning. It also includes maps with discrete and continuous values, covering variations in color mapping, category ordering, and stylistic patterns, enabling a comprehensive analysis. We evaluated the performance of multiple VLMs on this benchmark, highlighting gaps in their abilities, and providing insights for improving such models. Our dataset, along with all necessary code scripts, is available at map-wise.github.io

@inproceedings{bib_From_2025, AUTHOR = {Prashant, Kodali and Goel, Anmol and Asapu, Likhith and Bonagiri, Vamshi Krishna and Govil, Anirudh and Choudhury, Monojit and Kumaraguru, Ponnurangam and Shrivastava, Manish }, TITLE = {From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences}, BOOKTITLE = {ACM Trasactions on Asian and Low Resource Language Information Processing}. YEAR = {2025}}

Current computational approaches for analysing or generating code-mixed sentences do not explicitly model ``naturalness'' or ``acceptability'' of code-mixed sentences, but rely on training corpora to reflect distribution of acceptable code-mixed sentences. Modelling human judgement for the acceptability of code-mixed text can help in distinguishing natural code-mixed text and enable quality-controlled generation of code-mixed text. To this end, we construct Cline - a dataset containing human acceptability judgements for English-Hindi~(en-hi) code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources: synthetically generated code-mixed text and samples collected from online social media. Our analysis establishes that popular code-mixing metrics such as CMI, Number of Switch Points, Burstines, which are used to filter/curate/compare code-mixed corpora have low correlation with human acceptability judgements, underlining the necessity of our dataset. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models when trained solely using code-mixing metrics as features are outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs). Specifically, among Encoder models XLM-Roberta and Bernice outperform IndicBERT across different configurations. Among Encoder-Decoder models, mBART performs better than mT5, however Encoder-Decoder models are not able to outperform Encoder-only models. Decoder-only models perform the best when compared to all other MLLMS, with Llama 3.2 - 3B models outperforming similarly sized Qwen, Phi models. Comparison with zero and fewshot capabilitites of ChatGPT show that MLLMs fine-tuned on larger data outperform ChatGPT, providing scope for improvement in code-mixed tasks. Zero-shot transfer from En-Hi to En-Te acceptability judgments are better than random baselines.

@inproceedings{bib_Lost_2025, AUTHOR = {Mukherjee, Ananya and Shrivastava, Manish }, TITLE = {Lost in Translation? Found in Evaluation: A Comprehensive Survey on Sentence-Level Translation Evaluation}, BOOKTITLE = {ACM Computing Surveys}. YEAR = {2025}}

Machine Translation (MT) revolutionizes cross-lingual communication but is prone to errors, necessitating thorough evaluation for enhancement. Translation quality can be assessed by humans and automatic evaluation metrics. Human evaluation, though valuable, is costly and subject to limitations in scalability and consistency. While automated metrics supplement manual evaluations, this field still has considerable potential for development. However, there exists prior survey work on automatic evaluation metrics, it is worth noting that most of these are focused on resource-rich languages, leaving a significant gap in evaluating MT outputs across other language families. To bridge this gap, we present an exhaustive survey, encompassing discussions on MT meta-evaluation datasets, human assessments, and diverse metrics. We categorize both human and automatic evaluation approaches, and offer decision trees to aid in selecting the appropriate approach. Additionally, we evaluate sentences across languages, domains and linguistic features, and further meta-evaluate the metrics by correlating them with human scores. We critically examine the limitations and challenges inherent in current datasets and evaluation approaches. We propose suggestions for future research aimed at enhancing MT evaluation, including the importance of diverse and well-distributed datasets, the refinement of human evaluation methodologies, and the development of robust metrics that closely align with human judgments.

@inproceedings{bib_BRIG_2025, AUTHOR = {Muhammad, Shamsuddeen Hassan and Ousidhoum1, Nedjma and Abdulmumin, Idris and Wahle, Jan Philip and Ruas, Terry and Beloucif, Meriem and Kock, Christine De and SURANGE, NIRMAL and Shrivastava, Manish }, TITLE = {BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages}, BOOKTITLE = {Association for Computational Linguistics}. YEAR = {2025}}

People worldwide use language in subtle and complex ways to express emotions. Although emotion recognition--an umbrella term for several NLP tasks--impacts various applications within NLP and beyond, most work in this area has focused on high-resource languages. This has led to significant disparities in research efforts and proposed solutions, particularly for under-resourced languages, which often lack high-quality annotated datasets. In this paper, we present BRIGHTER--a collection of multilabeled, emotion-annotated datasets in 28 different languages and across several domains. BRIGHTER primarily covers low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. We highlight the challenges related to the data collection and annotation processes, and then report experimental results for monolingual and crosslingual multi-label emotion identification, as well as emotion intensity recognition. We analyse the variability in performance across languages and text domains, both with and without the use of LLMs, and show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.

@inproceedings{bib_TabX_2025, AUTHOR = {Pancholi, Vihang and Sushil, Bafna Jainit and Anvekar, Tejas and Shrivastava, Manish and Gupta, Vivek }, TITLE = {TabXEval: Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation}, BOOKTITLE = {Association for Computational Linguistics - Findings}. YEAR = {2025}}

Evaluating tables qualitatively and quantitatively poses a significant challenge, as standard metrics often overlook subtle structural and content-level discrepancies. To address this, we propose a rubric-based evaluation framework that integrates multi-level structural descriptors with fine-grained contextual signals, enabling more precise and consistent table comparison. Building on this, we introduce TabXEval, an eXhaustive and eXplainable two-phase evaluation framework. TabXEval first aligns reference and predicted tables structurally via TabAlign, then performs semantic and syntactic comparison using TabCompare, offering interpretable and granular feedback. We evaluate TabXEval on TabXBench, a diverse, multi-domain benchmark featuring realistic table perturbations and human annotations. A sensitivity-specificity analysis further demonstrates the robustness and explainability of TabXEval across varied table tasks. Code and data are available at https://coral-lab-asu.github.io/tabxeval/.

@inproceedings{bib_Segm_2025, AUTHOR = {Yadav, Saumitra and Shrivastava, Manish }, TITLE = {Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance}, BOOKTITLE = {Workshop on Asian Translation}. YEAR = {2025}}

Existing Machine Translation (MT) research often suggests a single, fixed set of hyperparameters for word segmentation models, textbf{symmetric Byte Pair Encoding} (BPE), which applies the same number of merge operations (NMO) to train tokenizers for both source and target languages. However, we demonstrate that this uniform approach doesn't guarantee optimal MT performance across different language pairs and data sizes. This work investigates BPE segmentation recipes across various data volumes and language pairs to evaluate MT system performance. We find that utilizing textbf{asymmetric BPE}—where the source and target languages have different NMOs—significantly improves results over the symmetric approach, especially in low-resource settings (50K, 100K, and 500K sentence pairs). Specifically, asymmetric BPE yield statistically significant (p<0.05) average gains of 5.32, 4.46, and 0.7 CHRF++ on English-Hindi in low-resource setups (50K, 100K, and 500K sentence pairs, respectively). We validated this trend across six additional language pairs (English Telugu, Shona, Norwegian, Kyrgyz, Hausa, and Inuktitut), observing statistically significant improvement in 10 out of 12 systems compared to symmetric BPE. Our findings indicate a high NMO for the source (4K to 32K) and a low NMO for the target (0.5K to 2K) provides optimal results, particularly benefiting low-resource MT.

@inproceedings{bib_A_Pr_2025, AUTHOR = {Yadav, Saumitra and Shrivastava, Manish }, TITLE = {A Preliminary Exploration of Phrase-Based SMT and Multi-BPE Segmentations through Concatenated Tokenised Corpora for Low-Resource Indian Languages}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2025}}

This paper describes our methodology and findings in building Machine Translation (MT) systems for submission to the WMT 2025 Shared Task on Low-Resource Indic Language Translation. Our primary aim was to evaluate the effectiveness of a phrase-based Statistical Machine Translation (SMT) system combined with a less common subword segmentation strategy for languages with very limited parallel data. We applied multiple Byte Pair Encoding (BPE) merge operations to the parallel corpora and concatenated the outputs to improve vocabulary coverage. We built systems for the English–Nyishi, English–Khasi, and English–Assamese language pairs. Although the approach showed potential as a data augmentation method, its performance in BLEU scores was not competitive with other shared task systems. This paper outlines our system architecture, data processing pipeline, and evaluation results, and provides an analysis of the challenges, positioning our work as an exploratory benchmark for future research in this area.

@inproceedings{bib_Adap_2025, AUTHOR = {Prashant, Kodali and Shivkumar, Vaishnavi and Sarang, Joshi Swarang and Choudhury, Monojit and Kumaraguru, Ponnurangam and Shrivastava, Manish }, TITLE = {Adapting Multilingual Models to Code-Mixed Tasks via Model Merging}, BOOKTITLE = {India Joint International Conference on Data Science & Management of Data}. YEAR = {2025}}

We study model merging as a practical alternative to conventional adaptation strategies for code-mixed NLP. Starting from a multilingual base model, we: (i) perform continued pre-training (CPT) on unlabeled code-mixed text to obtain an adapted checkpoint, (ii) merge checkpoint with the base model, and (iii) fine-tune (FT) on the downstream task data. We evaluate our approach for sentence classification (sentiment and hate speech) task in English-Hindi (En-Hi) and English-Spanish (En-Es) using XLM-R and Llama-3.2-1B models. Our results show that merged models consistently outperform full fine-tuning and CPT->FT. We observe gains of 2--5 points in F1 over full fine-tuning and ~1-2 points over CPT->FT, indicating that unlabeled data is leveraged more effectively via merging than via CPT alone. Zero-/few-shot prompting with larger LLMs (e.g., Llama-3.3-70B) lags behind fine-tuned and merged checkpoints, underscoring limits of in-context learning for code-mixed inputs. We further test cross-pair transfer by training on En-Hi and evaluating on En-Ta and En-Ml: merged checkpoints transfer more strongly than monolingual-English baselines (e.g., TV/TIES variants reaching 0.65-0.68 F1 vs 0.61-0.63 for full fine-tuning), suggesting that code-mixed knowledge is a more reliable substrate for low-resource pairs. We conclude with adaptation recipes matched to common data regimes (labeled only; labeled+unlabeled; transfer-only) and discuss limitations and scaling considerations for broader tasks and larger models.

@inproceedings{bib_TeCl_2024, AUTHOR = {Gopichand, Kanumolu and Lokesh, Madasu and SURANGE, NIRMAL and Shrivastava, Manish }, TITLE = {TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu}, BOOKTITLE = {International Conference on Computational Linguistics}. YEAR = {2024}}

News headline generation is a crucial task in increasing productivity for both the readers and producers of news. This task can easily be aided by automated News headline-generation models. However, the presence of irrelevant headlines in scraped news articles results in sub-optimal performance of generation models. We propose that relevance-based headline classification can greatly aid the task of generating relevant headlines. Relevance-based headline classification involves categorizing news headlines based on their relevance to the corresponding news articles. While this task is well-established in English, it remains under-explored in low-resource languages like Telugu due to a lack of annotated data. To address this gap, we present TeClass, the first-ever human-annotated Telugu news headline classification dataset, containing 78,534 annotations across 26,178 article-headline pairs. We experiment with various baseline models and provide a comprehensive analysis of their results. We further demonstrate the impact of this work by fine-tuning various headline generation models using TeClass dataset. The headlines generated by the models fine-tuned on highly relevant article-headline pairs, showed about a 5 point increment in the ROUGE-L scores. To encourage future research, the annotated dataset as well as the annotation guidelines will be made publicly available.

@inproceedings{bib_Know_2024, AUTHOR = {Mathur, Suyash Vardhan and Kartik, Kunal and Sushil, Bafna Jainit and Khandelwal, Harshita and Shrivastava, Manish and Gupta, Vivek and Bansal, Mohit and Roth, Dan }, TITLE = {Knowledge-Aware Reasoning over Multimodal Semi-structured Tables}, BOOKTITLE = {Empirical Methods in Natural Language Processing-Findings}. YEAR = {2024}}

Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data. This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. These findings establish our dataset as a robust benchmark for advancing AI's comprehension and capabilities in analyzing multimodal structured data.

@inproceedings{bib_CoST_2024, AUTHOR = {Mukherjee, Ananya and Yadav, Saumitra and Shrivastava, Manish }, TITLE = {CoST of breaking the LLMs}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2024}}

This paper presents an evaluation of 16 machine translation systems submitted to the Shared Task of the 9th Conference of Machine Translation (WMT24) for the English-Hindi (en-hi) language pair using our Complex Structures Test (CoST) suite. Aligning with this year’s test suite sub-task theme, “Help us break LLMs”, we curated a comprehensive test suite encompassing diverse datasets across various categories, including autobiography, poetry, legal, conversation, play, narration, technical, and mixed genres. Our evaluation reveals that all the systems struggle significantly with the archaic style of text like legal and technical writings or text with creative twist like conversation and poetry datasets, highlighting their weaknesses in handling complex linguistic structures and stylistic nuances inherent in these text types. Our evaluation identifies the strengths and limitations of the submitted models, pointing to specific areas where further research and development are needed to enhance their performance. Our test suite is available at https://github.com/AnanyaCoder/CoST-WMT-24-Test-Suite-Task.

@inproceedings{bib_Neur_2023, AUTHOR = {SINGH, RAJAT and CHOUDHARY, NURENDRA and BINDLISH, ISHITA and Srivastava, Manish }, TITLE = {Neural Network Architecture for Credibility Assessment of Textual Claims}, BOOKTITLE = {International Conference on Intelligent Text Processing and Computational Linguistics}. YEAR = {2023}}

Text articles with false claims, especially news, have recently become aggravating for the Internet users. These articles are in wide circulation and readers face difficulty discerning fact from fiction. Previous work on credibility assessment has focused on factual analysis and linguistic features. The task's main challenge is the distinction between the features of true and false articles. In this paper, we propose a novel approach called Credibility Outcome (CREDO) which aims at scoring the credibility of an article in an open domain setting. CREDO consists of different modules for capturing various features responsible for the credibility of an article. These features includes credibility of the article's source and author, semantic similarity between the article and related credible articles retrieved from a knowledge base, and sentiments conveyed by the article. A neural network architecture learns the contribution of each of these modules to the overall credibility of an article. Experiments on Snopes dataset reveals that CREDO outperforms the state-of-the-art approaches based on linguistic features.

@inproceedings{bib_BRR-_2023, AUTHOR = {Singh, Manish Kumar and Shrivastava, Manish }, TITLE = {BRR-QA: Boosting Ranking and Reading in Open-Domain Question Answering}, BOOKTITLE = {Joint International Conference on Data Science & Management of Data}. YEAR = {2023}}

Open-domain question qnswering (OpenQA) involves a retriever for selecting relevant passages from large text corpora (e.g. Wikipedia) and a reading comprehension (RC) model for extracting answers from these retrieved passages. The retrieved passages are often noisy. Since OpenQA relies heavily on efficient passages for better answer prediction, many passage ranker models have been pro- posed to filter out noisy passages. However, their performance is limited because their ranker model scores each passage separately by modelling only the relationship between query and passage. Thus, they could not capture local context information. Their ranker model also ignored the rich initial rank of passages ranked by a search engine. This paper presents a Passage Ranker model that captures local-context information through cross-passage interac- tion. Our ranker model integrates initial ranking and uses modified attention in the cross-passage interaction to compute a better confi- dence score for each passage. Moreover, we integrate SRL into our passage reader and train it on proposed sampled data. Our semantic reader can absorb contextual semantics. Experimental results on four public OpenQA datasets show that o

@inproceedings{bib_PMIn_2023, AUTHOR = {Ashok, Urlana and Chen, Pinzhen and Zhao, Zheng and Cohen, Shay B. and Shrivastava, Manish and Haddow, Barry }, TITLE = {PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India}, BOOKTITLE = {Technical Report}. YEAR = {2023}}

This paper introduces PMIndiaSum, a new multilingual and massively parallel headline summarization corpus focused on languages in India. Our corpus covers four language families, 14 languages, and the largest to date, 196 language pairs. It provides a testing ground for all cross-lingual pairs. We detail our workflow to construct the corpus, including data acquisition, processing, and quality assurance. Furthermore, we publish benchmarks for monolingual, cross-lingual, and multilingual summarization by fine-tuning, prompting, as well as translate-and-summarize. Experimental results confirm the crucial role of our data in aiding the summarization of Indian texts. Our dataset is publicly available and can be freely modified and re-distributed.

@inproceedings{bib_X-Ri_2023, AUTHOR = {Moradshahi, Mehrad and Shen, Tianhao and Bali, Kalika and Choudhury, Monojit and Chalendar, Gaël De and Goel, Anmol and Prashant, Kodali and Kumaraguru, Ponnurangam and Shrivastava, Manish }, TITLE = {X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents}, BOOKTITLE = {Technical Report}. YEAR = {2023}}

Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language. To reduce the cost, we apply manual editing to automatically translated data. We create a new multilingual benchmark, X-RiSAWOZ, by translating the Chinese RiSAWOZ to 4 languages: English, French, Hindi, Korean; and a code-mixed EnglishHindi language. X-RiSAWOZ has more than 18,000 human-verified dialogue utterances for each language, and unlike most multilingual prior work, is an end-to-end dataset for building fully-functioning agents. The many difficulties we encountered in creating X-RiSAWOZ led us to develop a toolset to accelerate the post-editing of a new language dataset after translation. This toolset improves machine translation with a hybrid entity alignment technique that combines neural with

@inproceedings{bib_Mult_2023, AUTHOR = {K, Mounika and Singh, Shantanu and Shrivastava, Manish }, TITLE = {MultiFacet: A Multi-Tasking Framework for Speech-to-Sign Language Generation}, BOOKTITLE = {International Conference on Multimodal Interaction}. YEAR = {2023}}

Sign language is a rich form of communication, uniquely conveying meaning through a combination of gestures, facial expressions, and body movements. Existing research in sign language generation has predominantly focused on text-to-sign pose generation, while speech-to-sign pose generation remains relatively underexplored. Speech-to-sign language generation models can facilitate effective communication between the deaf and hearing communities. In this paper, we propose an architecture that utilises prosodic information from speech audio and semantic context from text to generate sign pose sequences. In our approach, we adopt a multi-tasking strategy that involves an additional task of predicting Facial Action Units (FAUs). FAUs capture the intricate facial muscle movements that play a crucial role in conveying specific facial expressions during sign language generation.

@inproceedings{bib_Unsu_2023, AUTHOR = {Gopichand, Kanumolu and Lokesh, Madasu and Pavan, Baswani and Mukherjee, Ananya and Shrivastava, Manish }, TITLE = {Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really Need Reference?}, BOOKTITLE = {International Joint Conference on Natural Language Processing Workshop}. YEAR = {2023}}

Fluency is a crucial goal of all Natural Language Generation (NLG) systems. Widely used automatic evaluation metrics fall short in capturing the fluency of machine-generated text. Assessing the fluency of NLG systems poses a challenge since these models are not limited to simply reusing words from the input but may also generate abstractions. Existing reference-based fluency evaluations, such as word overlap measures, often exhibit weak correlations with human judgments. This paper adapts an existing unsupervised technique for measuring text fluency without the need for any reference. Our approach leverages various word embeddings and trains language models using Recurrent Neural Network (RNN) architectures. We also experiment with other available multilingual Language Models (LMs). To assess the performance of the models, we conduct a comparative analysis across 10 Indic languages, correlating the obtained fluency scores with human judgments.

@inproceedings{bib_Mukh_2023, AUTHOR = {Lokesh, Madasu and Gopichand, Kanumolu and SURANGE, NIRMAL and Shrivastava, Manish }, TITLE = {Mukhyansh: A Headline Generation Dataset for Indic Languages}, BOOKTITLE = {Technical Report}. YEAR = {2023}}

The task of headline generation within the realm of Natural Language Processing (NLP) holds immense significance, as it strives to dis- till the true essence of textual content into con- cise and attention-grabbing summaries. While noteworthy progress has been made in head- line generation for widely spoken languages like English, there persist numerous challenges when it comes to generating headlines in low- resource languages, such as the rich and diverse Indian languages. A prominent obstacle that specifically hinders headline generation in In- dian languages is the scarcity of high-quality annotated data. To address this crucial gap, we proudly present Mukhyansh, an extensive mul- tilingual dataset, tailored for Indian language headline generation. Comprising an impressive collection of over 3.39 million article-headline pairs, Mukhyansh spans across eight promi- nent Indian languages, namely Telugu, Tamil, Kannada, Malayalam, Hindi, Bengali, Marathi, and Gujarati. We present a comprehensive evaluation of several state-of-the-art baseline models. Additionally, through an empirical analysis of existing works, we demonstrate that Mukhyansh outperforms all other mod- els, achieving an impressive average ROUGE-L score of 31.43 across all 8 languages.

@inproceedings{bib_A_Co_2023, AUTHOR = {Debnath, Alok and Shrivastava, Manish }, TITLE = {A Computational Algebraic Analysis of Hindi Syntax}, BOOKTITLE = {Journal of Logic, Language and Information}. YEAR = {2023}}

In this paper, we present a computational algebraic representation of Hindi syntax. This paper is the first attempt to establish the representation of various facets of Hindi syntax into algebra, including dual nominative/ergative behavior, a syntacto-semantic case system and complex agreement rules between the noun and verb phrase. Using the pregroup analysis framework, we show how we represent morphological type reduction for morphological behavior of lexical markers, the representation of causative constructions which are morphologically affixed, as well as of light verb constructions which form the verb by joint predication. We present examples adapted from the Hindi Dependency Treebank to show the pregroup analysis of Hindi sentences.

@inproceedings{bib_Mukh_2023, AUTHOR = {Lokesh, Madasu and Gopichand, Kanumolu and SURANGE, NIRMAL and Shrivastava, Manish }, TITLE = {Mukhyansh: A Headline Generation Dataset for Indic Languages}, BOOKTITLE = {Pacific Asia Conference on Language, Information and Computation}. YEAR = {2023}}

The task of headline generation within the realm of Natural Language Processing (NLP) holds immense significance, as it strives to distill the true essence of textual content into concise and attention-grabbing summaries. While noteworthy progress has been made in headline generation for widely spoken languages like English, there persist numerous challenges when it comes to generating headlines in lowresource languages, such as the rich and diverse Indian languages. A prominent obstacle that specifically hinders headline generation in Indian languages is the scarcity of high-quality annotated data. To address this crucial gap, we proudly present Mukhyansh, an extensive multilingual dataset, tailored for Indian language headline generation. Comprising an impressive collection of over 3.39 million article-headline pairs, Mukhyansh spans across eight prominent Indian languages, namely Telugu, Tamil, Kannada, Malayalam, Hindi, Bengali, Marathi, and Gujarati. We present a comprehensive evaluation of several state-of-the-art baseline models. Additionally, through an empirical analysis of existing works, we demonstrate that Mukhyansh outperforms all other models, achieving an impressive average ROUGE-L score of 31.43 across all 8 languages

@inproceedings{bib_Gene_2022, AUTHOR = {Banerjee, Souvik and Mishra, Bamdev and Jawanpuria, Pratik and Shrivastava, Manish }, TITLE = {Generalised Spherical Text Embedding}, BOOKTITLE = {Technical Report}. YEAR = {2022}}

This paper aims to provide an unsupervised modelling approach that allows for a more flexible representation of text embeddings. It jointly encodes the words and the paragraphs as individual matrices of arbitrary column dimension with unit Frobenius norm. The representation is also linguistically motivated with the introduction of a novel similarity metric. The proposed modelling and the novel similarity metric exploits the matrix structure of embeddings. We then go on to show that the same matrices can be reshaped into vectors of unit norm and transform our problem into an optimization problem over the spherical manifold. We exploit manifold optimization to efficiently train the matrix embeddings. We also quantitatively verify the quality of our text embeddings by showing that they demonstrate improved results in document classification, document clustering, and semantic textual similarity benchmark tests.

@inproceedings{bib_Leve_2022, AUTHOR = {Jena, Aashna and Gupta, Vivek and Shrivastava, Manish and Eisenschlos, Julian Martin }, TITLE = {Leveraging Data Recasting to Enhance Tabular Reasoning}, BOOKTITLE = {Empirical Methods in Natural Language Processing-Findings}. YEAR = {2022}}

Creating challenging tabular inference data is essential for learning complex reasoning. Prior work has mostly relied on two data generation strategies. The first is human annotation, which yields linguistically diverse data but is difficult to scale. The second category for creation is synthetic generation, which is scalable and cost effective but lacks inventiveness. In this research, we present a framework for semi-automatically recasting existing tabular data to make use of the benefits of both approaches. We utilize our framework to build tabular NLI instances from five datasets that were initially intended for tasks like table2text creation, tabular Q/A, and semantic parsing. We demonstrate that recasted data could be used as evaluation benchmarks as well as augmentation data to enhance performance on tabular NLI tasks. Furthermore, we investigate the effectiveness of models trained on recasted data in the zero-shot scenario, and analyse trends in performance across different recasted datasets types.

@inproceedings{bib_TeSu_2022, AUTHOR = {Ashok, Urlana and SURANGE, NIRMAL and Pavan, Baswani and Priyanka, Ravva and Shrivastava, Manish }, TITLE = {TeSum: Human-Generated Abstractive Summarization Corpus for Telugu}, BOOKTITLE = {International Conference on Language Resources and Evaluation}. YEAR = {2022}}

Expert human annotation for summarization is definitely an expensive task, and can not be done on huge scales. But with this work, we show that even with a crowd sourced summary generation approach, quality can be controlled by aggressive expert informed filtering and sampling-based human evaluation. We propose a pipeline that crowd-sources summarization data and then aggressively filters the content via: automatic and partial expert evaluation. Using this pipeline we create a high-quality Telugu Abstractive Summarization dataset (TeSum) which we validate with sampling-based human evaluation. We also provide baseline numbers for various models commonly used for summarization. A number of recently released datasets for summarization, scraped the web-content relying on the assumption that summary is made available with the article by the publishers. While this assumption holds for multiple resources (or news-sites) in English, it should not be generalised across languages without thorough analysis and verification. Our analysis clearly shows that this assumption does not hold true for most Indian language news resources. We show that our proposed filtration pipeline can even be applied to these large-scale scraped datasets to extract better quality article-summary pairs

@inproceedings{bib_Towa_2022, AUTHOR = {Vaid, Roopal and Pant, Kartikey and Shrivastava, Manish }, TITLE = {Towards fine-grained classification of climate change related social media text}, BOOKTITLE = {Association for Computational Linguistics: Student Research Workshop}. YEAR = {2022}}

With climate change becoming a cause of concern worldwide, it becomes essential to gauge people’s reactions. This can help educate and spread awareness about it and help leaders improve decision-making. This work explores the fine-grained classification and Stance detection of climate change-related social media text. Firstly, we create two datasets, ClimateStance and ClimateEng, consisting of 3777 tweets each, posted during the 2019 United Nations Framework Convention on Climate Change and comprehensively outline the dataset collection, annotation methodology, and dataset composition. Secondly, we propose the task of Climate Change stance detection based on our proposed ClimateStance dataset. Thirdly, we propose a fine-grained classification based on the ClimateEng dataset, classifying social media text into five categories: Disaster, Ocean/Water, Agriculture/Forestry, Politics, and General. We benchmark both the datasets for climate change stance detection and fine-grained classification using state-of-the-art methods in text classification. We also create a Reddit-based dataset for both the tasks, ClimateReddit, consisting of 6262 pseudo-labeled comments along with 329 manually annotated comments for the label. We then perform semi-supervised experiments for both the tasks and benchmark their results using the best-performing model for the supervised experiments. Lastly, we provide insights into the ClimateStance and ClimateReddit using part-of-speech tagging and named-entity recognition.

@inproceedings{bib_Unsu_2022, AUTHOR = {Mukherjee, Ananya and Shrivastava, Manish }, TITLE = {Unsupervised Embedding-based Metric for MT Evaluation with Improved Human Correlation}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2022}}

In this paper, we describe our submission to the WMT22 metrics shared task. Our metric focuses on computing contextual and syntactic equivalences along with lexical, morphological and semantic similarity. The intent is to capture fluency and context of the MT outputs along with their adequacy. Fluency is captured using syntactic similarity and context is captured using sentence similarity leveraging sentence embeddings. The final sentence translation score is the weighted combination of three similarity scores: a) Syntactic Similarity b) Lexical, Morphological and Semantic Similarity and c) Contextual Similarity. This paper outlines two improved versions of MEE ie, MEE2 and MEE4. Additionally, we perform our experiments on language pairs of en-de, en-ru and zh-en from WMT17-19 testset and further report the correlation with human assessments. Our submission will be made available at https://github. com/AnanyaCoder/WMT22Submission.

@inproceedings{bib_REUS_2022, AUTHOR = {Mukherjee, Ananya and Shrivastava, Manish }, TITLE = {REUSE: REference-free UnSupervised quality Estimation Metric}, BOOKTITLE = {Conference on Machine Translation}. YEAR = {2022}}

This paper describes our submission to the WMT2022 shared metrics task. Our unsupervised metric estimates the translation quality at chunk-level and sentence-level. Source and target sentence chunks are retrieved by using a multi-lingual chunker. Chunk-level similarity is computed by leveraging BERT contextual word embeddings and sentence similarity scores are calculated by leveraging sentence embeddings of Language-Agnostic BERT models. The final quality estimation score is obtained by mean pooling the chunk-level and sentence-level similarity scores. This paper outlines our experiments and also reports the correlation with human judgements for en-de, en-ru and zh-en language pairs of WMT17, WMT18 and WMT19 testsets. Our submission will be made available at https://github. com/AnanyaCoder/WMT22Submission_REUSE

@inproceedings{bib_TeQu_2022, AUTHOR = {Vemula, Rakesh Kumar and Nuthi, Mani Kanta Sai and Shrivastava, Manish }, TITLE = {TeQuAD: Telugu Question Answering Dataset}, BOOKTITLE = {International Conference on Natural Language Processing.}. YEAR = {2022}}

Recent state of the art models and new datasets have advanced many Natural Language Processing areas, especially, Machine Reading Comprehension tasks have improved with the help of datasets like SQuAD (Stanford Question Answering Dataset). But, large high quality datasets are still not a reality for low resource languages like Telugu to record progress in MRC. In this paper, we present a Telugu Question Answering Dataset - TeQuAD with the size of 82k parallel triples created by translating triples from the SQuAD. We also introduce a few methods to create similar Question Answering datasets for the low resource languages. Then, we present the performance of our models which outperform baseline models on Monolingual and Cross Lingual Machine Reading Comprehension (CLMRC) setups, the best of them resulting in an F1 score of 83 % and Exact Match (EM) score of 61 %

@inproceedings{bib_SCon_2022, AUTHOR = {Adibhatla, Hiranmai Sri and Shrivastava, Manish }, TITLE = {SConE: Contextual Relevance based Significant CompoNent Extraction from Contracts}, BOOKTITLE = {International Conference on Natural Language Processing.}. YEAR = {2022}}

Automatic extraction of “significant” components of a legal contract, has the potential to simplify the end user’s comprehension. In essence, “significant” pieces of information have 1) information pertaining to material/practical details about a specific contract and 2) information that is novel or comes as a “surprise” for a specific type of contract. It indicates that the significance of a component may be defined at an individual contract level and at a contract-type level. A component, sentence, or paragraph, may be considered significant at a contract level if it contains contract-specific information (CSI), like names, dates, or currency terms. At a contract-type level, components that deviate significantly from the norm for the type may be considered significant (type-specific information (TSI)). In this paper, we present approaches to extract “significant” components from a contract at both these levels. We attempt to do this by identifying patterns in a pool of documents of the same kind. Consequently, in our approach, the solution is formulated in two parts: identifying CSI using a BERT-based contract-specific information extractor and identifying TSI by scoring sentences in a contract for their likelihood. In this paper, we even describe the annotated corpus of contract documents that we created as a first step toward the development of such a language-processing system. We also release a dataset of contract samples containing sentences belonging to CSI and TSI.

@inproceedings{bib_SyMC_2022, AUTHOR = {Prashant, Kodali and Goel, Anmol and Choudhury, Monojit and Shrivastava, Manish and Kumaraguru, Ponnurangam }, TITLE = {SyMCoM - Syntactic Measure of Code Mixing A Study Of English-Hindi Code-Mixing}, BOOKTITLE = {Findings of the Association for Computational Linguistics}. YEAR = {2022}}

Code mixing is the linguistic phenomenon where bilingual speakers tend to switch between two or more languages in conversations. Recent work on code-mixing in computational settings has leveraged social media code mixed texts to train NLP models. For capturing the variety of code mixing in, and across corpus, Language ID (LID) tags based measures (CMI) have been proposed. Syntactical variety/patterns of code-mixing and their relationship vis-a-vis computational model’s performance is under explored. In this work, we investigate a collection of English(en)-Hindi(hi) code-mixed datasets from a syntactic lens to propose, SyMCoM, an indicator of syntactic variety in code-mixed text, with intuitive theoretical bounds. We train SoTA en-hi PoS tagger, accuracy of 93.4%, to reliably compute PoS tags on a corpus, and demonstrate the utility of SyMCoM by applying it on various syntactical categories on a collection of datasets, and compare datasets using the measure.

@inproceedings{bib_Dive_2022, AUTHOR = {NANDIGAM, SAI LAKSHMI POOJITHA and Rayaprolu, Nikhil and Shrivastava, Manish }, TITLE = {Diverse Multi-Answer Retrieval with Determinantal Point Processes}, BOOKTITLE = {International Conference on Computational Linguistics}. YEAR = {2022}}

Often questions provided to open-domain question answering systems are ambiguous. Traditional QA systems that provide a single answer are incapable of answering ambiguous questions since the question may be interpreted in several ways and may have multiple distinct answers. In this paper, we address multi-answer retrieval which entails retrieving passages that can capture majority of the diverse answers to the question. We propose a re-ranking based approach using Determinantal point processes utilizing BERT as kernels. Our method jointly considers query-passage relevance and passage-passage correlation to retrieve passages that are both query-relevant and diverse. Results demonstrate that our re-ranking technique outperforms state-of-the-art method on the AmbigQA dataset.

@inproceedings{bib_The__2021, AUTHOR = {Gupta, Yash and Jyoth, Preethi and Ammanamanchi, Pawan Sasanka and Bordia, Shikha and Manoharan, Arjun and Mittal, Deepak and Pasunuru, Ramakanth and Shrivastava, Manish and Singh, Maneesh and Bansal, Mohit }, TITLE = {The Effect of Pretraining on Extractive Summarization for Scientific Documents}, BOOKTITLE = {Workshop on Scholarly Document Processing}. YEAR = {2021}}

Large pretrained models have seen enormous success in extractive summarization tasks. In this work, we investigate the influence of pretraining on a BERT-based extractive summarization system for scientific documents. We derive significant performance improvements using an intermediate pretraining step that leverages existing summarization datasets and report state-of-the-art results on a recently released scientific summarization dataset, SciTLDR. We systematically analyze the intermediate pretraining step by varying the size and domain of the pretraining corpus, changing the length of the input sequence in the target task and varying target tasks. We also investigate how intermediate pretraining interacts with contextualized word embeddings trained on different domains.

@inproceedings{bib_Fusi_2021, AUTHOR = {Singh, Manish Kumar and Shrivastava, Manish }, TITLE = {Fusion of Intrinsic & Extrinsic Sentential Traits for Text Coherence Assessment}, BOOKTITLE = {Joint International Conference on Data Science & Management of Data}. YEAR = {2021}}

In this paper, we investigate the problem of text coherence modeling with a new perspective of learning better distributed sentence representation by incorporating document dependent and document independent features of a sentence. To this end, we propose a datadriven end-to-end novel neural coherence model that captures text coherence by exploiting the semantic and distributional aspects of the sentences in a document. The network is able to capture various document independent and document dependent features of a sentence which is imperative in assessing text coherence. Experiments on the standard Sentence Ordering task indicate that our proposed model shows significant performance gain of 1.7 percent in terms of accuracy score compared with the state-of-the-art baselines.

@inproceedings{bib_NoEl_2020, AUTHOR = {KULLAR, PAYAL and Alpeshkumar, Majmundar Kushal and Srivastava, Manish }, TITLE = {NoEl: An Annotated Corpus for Noun Ellipsis in English}, BOOKTITLE = {International Conference on Language Resources and Evaluation}. YEAR = {2020}}

Ellipsis resolution has been identified as an important step to improve the accuracy of mainstream Natural Language Processing (NLP) tasks such as information retrieval, event extraction, dialog systems, etc. Previous computational work on ellipsis resolution has focused on one type of ellipsis, namely Verb Phrase Ellipsis (VPE) and a few other related phenomenon. We extend the study of ellipsis by presenting the No(oun)El(lipsis) corpus - an annotated corpus for noun ellipsis and closely related phenomenon using the first hundred movies of Cornell Movie Dialogs Dataset. The annotations are carried out in a standoff annotation scheme that encodes the position of the licensor, the antecedent boundary, and Part-of-Speech (POS) tags of the licensor and antecedent modifier. Our corpus has 946 instances of exophoric and endophoric noun ellipsis, making it the biggest resource of noun ellipsis in English, to the best of our knowledge. We present a statistical study of our corpus with novel insights on the distribution of noun ellipsis, its licensors and antecedents. Finally, we perform the tasks of detection and resolution of noun ellipsis with different classifiers trained on our corpus and report baseline results.

@inproceedings{bib_Cont_2018, AUTHOR = {CHOUDHARY, NURENDRA and SINGH, RAJAT and Bindlish, Ishita and Shrivastava, Manish }, TITLE = {Contrastive Learning of Emoji-Based Representations for Resource-Poor Languages}, BOOKTITLE = {International Conference on Intelligent Text Processing and Computational Linguistics}. YEAR = {2018}}

CESNA model consists of twin Bi-directional Long Short-Term Memory Recurrent Neural Networks (Bi-LSTM RNN) with shared parameters joined by a contrastive loss function based on a similarity metric. The model learns the representations of resource-poor and resource-rich language in a common emoji space by using a similarity metric based on the emojis present in sentences from both languages. The model, hence, projects sentences with similar emojis closer to each other and the sentences with different emojis farther from one another. Experiments on large-scale Twitter datasets of resource-rich languages - English and Spanish and resource-poor languages - Hindi and Telugu reveal that CESNA outperforms the state-of-the-art emoji prediction approaches based on distributional semantics,

@inproceedings{bib_Deep_2017, AUTHOR = {Mehta, Purvanshi and MISHRA, PRUTHWIK and SANJAY, ATHAVALE VINAYAK and Srivastava, Manish and Sharma, Dipti Mishra }, TITLE = {Deep Neural Network based system for solving Arithmetic Word problems}, BOOKTITLE = {International Joint Conference on Natural Language Processing}. YEAR = {2017}}

This paper presents DILTON, a system which solves simple arithmetic word problems. DILTON first predicts the operation that is to be performed (’-’,’+’,’*’,’/’) through a deep neural network based model and then uses it to generate the answer. DILTON divides the question into two parts - world state and query as shown in Figure 1. The world state and the query are processed separately in two different networks and finally the networks are merged to predict the final operation. DILTON learns to predict operations with 8.81 % in a corpus of primary school questions. With simple similarity between the contexts of quantities appearing in the problem and the question text, we are able to identify 92.25 % of relevant quantities and solve 81% of the questions. Our code and data is publicly available

@inproceedings{bib_Expl_2017, AUTHOR = {VAJPAYEE, AVIJIT and GUPTA, ARIHANT and AKHTAR, SYED SARFARAZ and SRIVASTAVA, ARJIT and JHANWAR, MADAN GOPAL and Srivastava, Manish }, TITLE = {Exploiting Morphological Regularities in Distributional Word Representations}, BOOKTITLE = {Conference on Empirical Methods in Natural Language Processing}. YEAR = {2017}}

We present a simple, fast and unsupervised approach for exploiting morphological regularities present in high dimensional vector spaces. We propose a novel method for generating embeddings of words from their morphological variants using morphological transformation operators. We evaluate this approach on MSR word analogy test set (Mikolov et al.,2013d) with an accuracy of 85% which is 12% higher than the previous best known system.

@inproceedings{bib_Hand_2016, AUTHOR = {PRANEETH, SUGGU SAI and GOUTHAM, TATAKUNTLA KUSHWANTH NAGA and Chinnakotla, Manoj K. and Srivastava, Manish }, TITLE = {Hand in Glove: Deep Feature Fusion Network Architectures for AnswerQuality Prediction in Community Question Answering}, BOOKTITLE = {International Conference on Computational Linguistics}. YEAR = {2016}}

Community Question Answering (cQA) forums have become a popular medium for soliciting answers to specific user questions from experts and experienced users in a given topic. However,for a given question, users sometimes have to sift through a large number of low-quality or irrelevant answers to find out the answer which satisfies their information need. To alleviate this,the problem of Answer Quality Prediction (AQP) aims to predict the quality of an answer posted in response to a forum question. Current AQP systems either learn models using - a) various hand-crafted features (HCF) or b) Deep Learning (DL) techniques which automatically learn the feature representations.In this paper, we propose a novel approach for AQP known as -“Deep Feature Fusion Network(DFFN)”which combines the advantages of both hand-crafted features and deep learning based systems. Given a question-answer pair along with its metadata, a DFFN architecture indepen-dently - a) learns features using the Deep Neural Network (DNN) and b) computes handcrafted features leveraging various external resources and then combines the musing a fully connected neural network trained to predict the quality of the given answer. DFFN is an end-end differen-tiable model and trained as a single system. We propose two different DFFN architectures which vary mainly in the way they model the input question/answer pair - a) DFFN-CNN which uses a Convolutional Neural Network (CNN) and b) DFFN-BLNA which uses a Bi-directional LSTMwith Neural Attention (BLNA). Both these proposed variants of DFFN (DFFN-CNN and DFFN-BLNA) achieve state-of-the-art performance on the standard SemEval-2015 and SemEval-2016benchmark datasets and outperforms baseline approaches which individually employ either HCFor DL based techniques alone.

@inproceedings{bib_Toge_2016, AUTHOR = {DAS, ARPITA and YENALA, HARISH and Chinnakotla, Manoj and Srivastava, Manish }, TITLE = {Together we stand: Siamese networks for similar question retrieval}, BOOKTITLE = {Conference of the Association of Computational Linguistics}. YEAR = {2016}}

Community Question Answering (cQA)services like Yahoo!Answers1, BaiduZhidao2, Quora3, StackOverflow4etc.provide a platform for interaction with experts and help users to obtain precise and accurate answers to their questions.The time lag between the user posting a question and receiving its answer could be reduced by retrieving similar historic questions from the cQA archives.The main challenge in this task is the “lexico-syntactic” gap between the current and the previous questions. In this paper, we pro-pose a novel approach called“Siamese Convolutional Neural Network for cQA(SCQA)”to find the semantic similarity between the current and the archived ques-tions.SCQA consist of twin convolutional neural networks with shared parameters and a contrastive loss function joining them. SCQA learns the similarity metric for question-question pairs by leveraging the question-answer pairs available in cQA forum archives. The model projects semantically similar question pairs nearer to each other and dissimilar question pairs farther away from each other in the semantic space. Experiments on large scale real-life “Yahoo! Answers” dataset reveals that SCQA outperforms current state-of-the-art approaches based on translation models, topic models and deep neural network based models which use non-shared parameters.

@inproceedings{bib_Deep_2016, AUTHOR = {PRANEETH, SUGGU SAI and GOUTHAM, TATAKUNTLA KUSHWANTH NAGA and Chinnakotla, Manoj K. and Srivastava, Manish }, TITLE = {Deep feature fusion network for answer quality prediction in community question answering}, BOOKTITLE = {Workshop on Neural Information Retrieval,}. YEAR = {2016}}

Community Question Answering (cQA) forums have become a popular medium for soliciting direct answers to specific questions of users from experts or other experienced users on a given topic. However, for a given question, users some-times have to sift through a large number of low-quality or irrelevant answers to find out the answer which satisfies their information need. To alleviate this, the problem of Answer Quality Prediction (AQP) aims to predict the quality of an answer posted in response to a forum question.Current AQP systems either learn models using - a) various hand-crafted features (HCF) or b) use deep learning (DL)techniques which automatically learn the required feature representations.In this paper, we propose a novel approach for AQP known as -“Deep Feature Fusion Network (DFFN)”which leverages the advantages of both hand-crafted features and deep learning based systems. Given a question-answer pair along with its metadata, DFFN independently - a) learns deep features using a Convolutional Neural Network (CNN) and b) computes hand-crafted features using various external resourcesand then combines them using a deep neural network trained to predict the final answer quality. DFFN achieves state-of-the-art performance on the standard SemEval-2015 andSemEval-2016 benchmark datasets and outperforms base-line approaches which individually employ either HCF or DL based techniques alone.