Abstract

The problem of matching small image regions or patches between images is central to many image editing and computer vision tasks. In this thesis, we propose two related solutions to the problem of fast and global patch-matching. First, we present a multi-resolution spatial pyramid based solution, which we call Pyramid Patch-Matching (PPM). It uses a simple coarse-to-fine framework powered by exhaustive search at the coarsest resolution and local directed search at finer resolutions. PPM quickly finds accurate patch correspondences between images or image regions. The search step in our algorithm is flexible to accommodate the use of other nearest-neighbor algorithms within its framework. A proposed increase K technique enables PPM to store more number of matches than required at coarse resolutions which maintains a wide search range. This improves accuracy by up-to 2%. Its effect is most significant when less number of matches are required. The early-termination technique preempts the distance calculation between patches, if the distance exceeds that of the farthest match currently stored. This speeds up the algorithm by up-to 1.7. Following the downsampling rule, the reduce patch-size technique matches smaller patches than required at coarser resolutions of the pyramid. Empowered by these techniques, PPM achieves state-of-the-art accuracy outperforming previous standards of Coherency Sensitive Hashing (CSH) [36] and PatchMatch [12] without compromising on execution-time. PPM finds up-to 4% more of the ground-truth matches and achieves lower error values compared to CSH, the current standard. Several parameters are provided to tune the search for good matches based upon the application. We perform experiments to study effects of these parameters on matching accuracy and computation time. While consistently achieving lower error values for RGB distance between matched patches, the simplicity of our algorithm also enables fast parallel implementations, providing further advantage over previous randomized, hashing or tree-based iterative approaches. On CPU cores, it achieves a nearlinear time speed-up w.r.t. the number of cores and a GPU implementation provides a further speed-up of up to 70 compared to a CPU implementation run on a quad-core machine (250 w.r.t. single core). We observe that the matching accuracy of PPM suffers from low confidence matching of patchvectors at coarse resolutions. To solve this, we propose a novel way to mix information by concatenating corresponding patch-vectors from finer resolution levels with patch-vectors of coarse levels. We call these Mixed-Resolution Vectors (MR-vectors). Unlike traditional patch-vectors that contain information from a single resolution level, MR-vectors contain more relevant information for matching. We build over our PPM algorithm by substituting MR-vectors in place of traditional patch-vectors. We call this the Mixed-Resolution Patch-Matching (MRPM) algorithm. MRPM further improves the accuracy of matches found by PPM with a marginal increase in time. It captures up-to 4% more of the groundtruth matches (8% more than CSH) and achieves near-optimal match accuracy. We also show that the MR-approach has computational benefits over an approach which uses an increased search range. We describe the effects of changing the number of resolution levels mixed and show that a balance between accuracy and time is achieved at mixing 2 levels. We perform various experiments to compare MRPM with PPM and other previous approaches, in all of which MRPM consistently performs better. Fast parallel multi-core and GPU implementations of MRPM follow from parallel implementations of PPM. We develop applications of image denoising, image summarization and auto-crop based on our MRPM algorithm. Results with better visual quality are obtained when MR-vectors are used. The computational bottleneck of these algorithms is the search for nearest neighbors of patches. We demonstrate the utility of MRPM as a fast patch-matching engine which empowers these applications. We open up new possibilities of research to explore the scope of MR-vectors which we believe has applications in several other image editing and computer vision tasks.

Abstract

The amount of information on the World Wide Web(WWW) has been increasing tremendously. It has been observed that more the available information about a given subject, the more difficult it is to locate accurate and relevant information. However, WWW is not just about textual information. The prevalence of digital cameras has brought about an exponential growth in the amount of images found on the World Wide Web. These images can be a good source of information and may prove useful for various applications. Indexing the web images and building image search engines could help users quickly find images they seek, which can then be used for many purposes. The large volume of web images has necessitated the development of efficient methods to enable indexing, searching, and retrieval of web images. The research community has responded to this challenge and we have witnessed the development of a plethora of techniques and working systems in the last two decades. Furthermore, some commercial engines have appeared, including Google image search1, Lycos2 and AltaVista photo finder3. The effectiveness of these engines is still limited. In this thesis we address the following two issues: First, predicting the terms in image alternate text using other textual descriptors. Providing search services for the web images has been difficult. Traditional image retrieval systems assign annotations to each image manually. Although it is a good methodology to retrieve images through text retrieval technologies, it is gradually becoming impossible to annotate images manually one by one due to the huge and rapid growing number of web images. Automatic Image Annotation has become an active research area since then. A common view is that semantics of web images are well correlated with their associated texts. Because of this, several popular search engines offer web image search based only on the associated texts. Alternate text(ALT tag) is considered the most important of all associated texts. ALT attribute is used to describe the contents of an image file. ALT attribute is designed to be an alternative text description for images. It represents the semantics of an image as it provides useful information to anyone using the browsers that cannot display images or image display disabled. However, a recent study has shown that around half of the images on the web have no ALT text at all. We propose an approach to predict the terms in ALT text of an image based on the term co-occurrences. We explore different approaches to the problem of predicting terms in ALT text using term co-occurrences and natural language processing techniques where the terms in noun phrases and verb phrases are given more weight, and conclude that both the term co-occurrence approach and term co-occurrence approach combined with noun phrases and verb phrases work well for the problem. However, for the prediction task, we prefer the term co-occurrence approach with out natural language processing techniques as it is language independent. We build two image annotation systems on top of the proposed approach and the proposed approach combined with natural language processing techniques, and find out that latter achieves good performance over the former. We also explore a slightly different but related problem in the context of web image retrieval: how users behave in a multilingual information access task. Search logs are good sources of information to know how the user needs are expressed. This information can be used to improve the quality of the search interface and the search experience. We analyze the search logs generated by an online game, known-item image retrieval from Flickr4, a popular image store on the web. We present our experiments to mine the search logs and extract conclusions about the behavior of users when facing a strictly multilingual information access task. We group the users based on the score, number of hints taken, and the precision to study the behavior of the most successful users, the least successful users and the users in between the two. We analyze the two types of questionnaires - Found Image Questionnaire after the user finds an image and Give up Questionnaire when the user gives up - to know the issues they faced while searching for an image. We study how the language skills effect the user behavior. The majority of Web image search is text-based and the success of such approaches often depends on reliably identifying relevant text associated with a particular image. Our findings confirm research results reported in previous studies, such as the fact that finding images is still quite challenging for users, even when the images have been tagged by a community such as Flickr. Our results show that, most of the users start with monolingual interface and soon they realize cross-lingual interface is more useful than mono-lingual interface, and the users are more comfortable to search in their mother language or the l

Abstract

Speech communication is major medium of communication among human beings. Nonverbal elements of speech help in communicating the paralinguistic information such as emotions, attitudes, intentions, etc. along with message which is conveyed by the lexical part. They play a major role in conveying the unspoken message in the speech signal. Voice quality is one such nonverbal element which helps in communicating the paralinguistic information. Voice quality (phonation type) also serves a linguistic function in various languages across the world. It also plays a role in assessment of various voice disorders. Laughter is a nonverbal vocalization which is produced and used by human beings in speech communication. It appears in natural conversation in everyday speech and provides naturalness to conversational speech. Synthesizing laughter helps in improving the expressiveness of speech synthesis. In this work, analysis and synthesis of breathy voice and laughter has been performed. Analysis has been performed based on excitation characteristics, contrary to general spectral and spectrographic methods. Features such as instantaneous fundamental frequency, strength of excitation, spectral tilt, periodic to aperiodic energy ratio and perceived loudness measure have been computed for analysis of breathy voice. Comparisons have been made between values of these parameters derived from the breathy and modal voiced speech segments. Classification experiments have been performed to discriminate breathy voice and modal voice from each other using periodic to aperiodic energy ratio and loudness measure, which proved to be discriminating the voice qualities successfully. Also, some approaches based on modifying the excitation source have been employed to synthesize breathy voice. Laugh signals have been analyzed to study the pattern and structure of the signal and the source features derived from it. The contours and range of instantaneous fundamental frequency and strength of excitation have been studied and modeled to synthesize laugh signal from a vowel segment. Other features such as presence of frication like noise within laughter, duration and intensity have also been analyzed. Perceptual significance of these features have been studied. Suitable modifications have been made in source characteristics derived from a vowel segment to synthesize a laugh signal. Subjective evaluation has been conducted to gauge the quality and the level of acceptance of the synthesized laugh signals. Scores indicate that synthesized laughter was in the acceptable range without a compromise in the quality of synthesis.

Abstract

The penetration of Internet over the last two decades has witnessed contributors from all around the world. This has led to a vertical expansion of information in different languages. There has been tools developed to handle content in many languages independently. However, when a user wants to view content from several different languages as a search request, the existing tools doesn’t suffice. Studies reveal that the number of users requesting multilingual information in their search results is on raise. Information Retrieval (IR), is a field in computer science that deals with information available in huge volumes, like on Internet, to retrieve and display the information need of an user. This concept can be extended to the domain of high volumes of multilingual information. When a user requests for an information need in this collection of multilingual information, the task of retrieving the exact information is very challenging. It has inherent issues of structuring the content, to understand the user information need, to fetch relevant documents, to retrieve similar documents from other languages and finally to rank all documents irrespective of their language as per their relevancy to the query. One possible solution is to retrieve a result list separately in each language requested by the user and finally merge them to produce a single multilingual result list. This thesis attempts to solve the problem of merging the intermediate result lists by ranking all documents as per their relevancy to the query irrespective of the language. The task is challenging because merging result lists need the understanding of the content of all documents across languages and ranking them according to their relevancy to the query. Existing several approaches make an extensive use of language resources and tools in achieving this task. However, such approaches are limited across languages, because of their dependency on the resources and tools. Many languages, like Hindi, suffer from the limited availability of the language resources/tools. Developing a language-independent approach is highly crucial for many such languages. The central theme of this thesis is to develop a language-independent approach which is also efficient to merge several monolingual result lists to form a multilingual result list. A learning-to-rank framework is the proposed solution for this problem. In this framework, for every query-document pair a relevance judgment will be given, indicating if the document is relevant to the query or not. Features will be extracted from the content of query and document. Ranking algorithms will be used in training a model so as to assign a relevancy to a new unseen query-document pair. These relevancies are used in ranking multilingual documents. In this thesis, we propose certain simple and efficient features that enhances the performance of ranking. All the calculations are performed without using any language tools. To achieve language-independent approach, several alternative methods have been introduced to replace the usage of existing language tools, like lemmatizer, etc. One of the proposed features involve computing similarity between two different language documents. Comparing multilingual documents is not a trivial task. Bilingual dictionaries are usually used in translating a document before comparing. In this thesis, a new dimension of document representation is explored. Instead of representing documents with words, we attempt to represent documents as a mixture of the topics/themes inherent in the document. Latent Dirichlet Allocation (LDA), a topic modeling algorithm, is availed to achieve that. However, to represent multilingual documents as a mixture of topics, a new mathematical formulation is derived over the existing LDA model, we call it MultiLDA. This MultiLDA model has the potential to be applied elsewhere. Various experiments have been performed to validate each and every concept introduced in this thesis. The results reveal the significance of the proposed solutions.

Abstract

Keyword search is the most popular way to search content on the web, however, in specific applications it might fail to capture a user’s information need. For example, in domains like patents and biomedical text retrieval, a set of keywords as a query may not be sufficient but a document as a query would be more useful. Documents, as a medium to search for information will become necessary in different scenarios such as blogs, posts, resume or related news search. The objective of this thesis is to explore and evaluate models for ”Query by document”, i.e. search a corpus with a document as a query, with their application to the corpus of legal documents. In the legal domain, patent search is a routine task for a patent examiner. Depending upon the search type (invalidity or prior art search), the examiner may have a patent application at hand to find similar patents. Thus, a document, i.e. a patent application is often a starting point to find similar inventions. Presently, the success of patent search relies on the queries created from a patent application by the examiner. Since a lot of time is spent in constructing relevant queries, transforming the patent into a query automatically would save the examiner a lot of effort. We focus on two ways of constructing a query from an input patent application - relying on its text to form a query and using its meta-data (class- information) and information from the corpus (citations) for representing patents and retrieving similar patents to improve retrieval accuracy. We test both the approaches with our participation in CLEF-IP track, where teams investigate information retrieval techniques in Intellectual Property (IP) domain. The empirical analysis of both approaches and their combination indicates that results are best when combination of both approaches is used. The first approach focuses on extracting key phrases from a patent and their weights in the query. This query is then used to retrieve similar patents. Since a patent may contain information that is both diverse and dense in nature, entire content as a query will not be feasible as the results might be noisy and irrelevant. An optimal query can be formed by extracting important keywords, phrases or concepts from a patent by using Key Phrase Extraction (KPE) techniques. Several KPE algorithms have been proposed in the literature but their performance on query construction for patents has not yet been explored. We systematically evaluate and analyze the performance of queries created by using stateof- the-art KPE techniques for invalidity search task. The key phrase extraction techniques are used to construct queries from patents for Invalidity Search task. Experiments on a collection of USPTO Patents show that queries formed by KPE approaches perform better than those formed by selecting phrases based on term frequency based measures - tf and tf-idf scores. At the same time, keyword search for similar patents requires significant domain expertise and may not fetch relevant results. Thus, in the second part of this thesis, we focus on reducing the dependency on keywords, by exploring an alternative representation for patents using only the meta-information present in a patent and its citations. We propose a novel representation for patents and use a two stage approach to find similar patents. Each patent is represented as an IPC class vector. Citation network of patents is used to propagate these vectors from a node (patent) to its neighbors (cited patents). Thus, each patent is represented as a weighted combination of its IPC information as well as of its neighbors. A query patent is represented as a vector using its IPC information and similar patents can be simply found by comparing this vector with vectors of patents in the corpus. Text based search is used to re-rank this solution set to improve precision. We experiment with various similarity measures and re-ranking methods to empirically show that our representation is effective in improving both precision and recall of queries of CLEF-IP 2011 dataset. Large scale evaluation of both key phrase extraction and meta-data based representation of patents was done by participating in CLEF-IP track. The results of proposed approaches were also combined with rule based citation extraction from query patent. We submitted six runs to the track, with two runs achieving significant Recall and highest MAP (Mean average precision) amongst all participants. Results show that the proposed meta-data based representation for patents, combined with queries formed with extracted key phrases from the query patent outperforms the runs with either approach. The results of both the models on large datasets indicates that significant improvement can be achieved if patent text is used with other data like class information and patent citations. We believe that a richer language independent representation could be proposed in th

Abstract

Hindi is an Indian language which is relatively rich in morphology. Few morphological analyzers of this language have been developed. However, they do only inflectional analysis. None of them give the derivational analysis of the language. In this thesis, we present our Hindi derivational morphological analyzer which gives us derivational morphological analysis of Hindi. The construction of the tool took place in three phases. All the three phases are crucial for the development of the tool. The first step in building a derivational analyzer is identifying the derivational variants. The derivational variants in the language must be identified and then they must be studied in order to obtain important properties of the derived words in the language. So the main aim of the first phase was to build a tool for identifying Hindi derivational variants. For this, we followed an approach which was supervised in nature. First, we used 11 productive derivational suffixes of the language which participate only in verb to noun derivations. Then 454 derivational variants of these 11 suffixes were acquired manually. Finally we created an SVM classifier with 7 features. We used the 454 variants as a training data for our classifier. This classifier is a tool meant for identifying derivational variants. A total of 170 derivational variants were identified using this tool. The scope for adding more number of pairs was less. Some of the important features did not influence the performance of our classifier to a great extent. But 11 new suffixes were considered after this phase. These 22 suffixes along with 624 derivational variants formed the basis of our work in subsequent phases. 624 is a small number. Many more derivational variants must be identified for obtaining the rules of the suffixes and other important properties. Even suffixes that participate in other kinds of derivations must be studied. Their derivational variants must also be obtained. Hence in our second phase, our main aim remained identification of Hindi derivational variants. We used a graph based approach for achieving the goal. This approach is unsupervised in nature. We used the three metrics of semantic relatedness, phonological similarity and suffix presence for identifying the derivational variants. First, we built a graph based model and incorporated these three metrics in that model. This model was built using resources like Hindi Wikipedia and C++ boost graph libraries. As the last step, we ran the page rank algorithm on the graph model for identifying the derivational variants. A total of 2800 derivational variants were identified in this phase. So a total of 22 derivational suffixes and 3000 derivational variants were obtained from the first two phases. Now the final step is to provide derivational analysis. This is the aim of the third phase. The properties of these 22 suffixes and their derivational variants were studied. An algorithm was designed using the results of this study. The algorithm used by us upgrades an existing inflectional analyzer to a derivational analyzer. Our algorithm primarily achieves two goals. First, it successfully incorporates the derivational analysis in the inflectional analyzer. Second, it also increases the coverage of the inflectional analysis of the existing inflectional analyzer. Finally, the goal of building a Hindi derivational morphological analyzer was accomplished in 3 phases.

Abstract

Question Generation is the task of generating questions from various inputs such as raw text or semantic representations. It is an attempt towards structuring the potential educational text which is available everywhere on the Internet. Generating questions manually on these large amounts of varied text is a tedious task. Hence, there is a need for generating questions automatically from these texts. The present work generates questions automatically from a given document. The aim is to build a system which can be used for different domains in various languages. Questions can be broadly classified into Subjective Questions and Objective Questions. The work mainly focuses on generating two types of objective questions, (i) Cloze Questions and (ii) Open-cloze Questions. To present the method, English language texts have been used. A cloze question contains a sentence with one or more blanks in it and four alternatives are given to fill those blanks. One of the four alternatives is correct and the others are wrong. The wrong alternatives are called distractors since they distract a student from choosing the correct answer. Cloze question generation is a three stage process, (i) Sentence Selection, (ii) Keyword Selection and (iii) Distractor Selection. In the presented system, all the three stages use heuristically weighted features to select relevant sentences, keywords and distractors. Finding three relevant distractors corresponding to each selected keyword is the most difficult task. Therefore in the next attempt, a rule based open-cloze question generation system has been developed. Open-cloze questions are similar to cloze-questions without alternatives, which make them more difficult to solve. As it is difficult to evaluate the quality of questions generated automatically, this work also provides guidelines for the manual evaluation of automatically generated cloze and open cloze questions. The present work generates questions that assess the content knowledge that a student has acquired upon reading a text rather than vocabulary, grammar assessment or language learning. There are only a few previous works under the area of QG, and this is the first attempt ever to generate cloze and open-cloze questions on content knowledge. Hence, a comparison between this system and the previous systems using a common dataset will not fetch any results. However, different individual stages of this method have been compared with previous approaches. In the comparison, it has been shown that the presented method covers more types of sentences to generate good quality questions. In addition, the method does not use any external resources unlike previous works and that makes it both domain and language independent.

Abstract

Modelling and simulation of a robot on uneven terrain is of importance as it provides a basis to know about the robots position and its state. In this work we present a simulation framework and dynamic state space realisation for evolution on uneven terrains for a wheeled mobile robot such as a synchronous drive robot. The framework lends itself as a tool capable of solving various problems, such as forward kinematic based evolution, inverse kinematic based evolution, path planning and trajectory tracking. This framework becomes particularly useful when we understand that the evolution problem (and hence the various associated problems based on evolution) is particularly challenging on uneven terrain. Specifically it is entailed to bring in the contact constraints posed by the interaction of the wheel and the ground as well as the holonomic constraints as the problem is formulated in a DAE setting (Differential Algebraic Equation). The problem becomes all the more crucial as vehicles moving on uneven terrain are becoming the order of the day. Nonetheless there has not been much literature that deals in length the various aspects that go into the framework. This work elaborates on the various aspects of the framework, presents simulation results on uneven terrain, where the vehicle evolves without slipping and also presents substantial quantitative analysis in regard to wheel slippage. Dynamic model which ensures no slip and permanent contact on uneven terrain and which satisfies holonomic and non holonomic constraints simultaneously is also proposed. For a 3 wheeled WMR a generalised state space is derived and is used to find the dynamic equations of motion as a second order constrained Differential Algebraic Equations (DAE). Dynamic equations are represented using lagrangian formulation where all the matrices and constraints are represented in the generalised state space representation. Constraint satisfaction is achieved by using Baumgarthe stabilisation for the equivalent second order holonomic and non holonomic constraints and in calculating the Lagrangian multipliers as part of the DAE which is unique. The main contributions of the thesis is to realise a system for uneven terrain navigation and to simulate it using both kinematic and dynamic equations to get feasible paths on uneven terrains.

Abstract

Crystal structures of drug molecules complexed with their target RNAs are being widely studied today, particularly in the context of the importance of the ribosome as a potential drug target, and the notable discriminatory ability of recognition and binding of cognate ligands by RNA aptamers. However, the molecular level understanding of the role of ligand binding interactions in RNA functions, require the analysis of stabilities and interaction energies of RNA-drug interactions. The present work deals with a detailed analysis of the structural and energetic features of these drug-nucleobase pairs and pseudo pairs using quantum chemical methods.
Previous studies on RNA base pairing interactions have demonstrated that gas phase quantum chemical calculations can be used to analyze the intrinsic stabilities of non-covalent interaction in RNA. In line with these studies, here we report the results of analysis of geometries and interaction energies of base pairs and pseudo pairs observed in experimental structures of different classes of drug molecules interacting with RNA, which include aminoglycosides, oxazolidinones, tetracyclines and lincosamides. Geometry optimizations of studied nucleobase-drug pairs have been carried out at B3LYP/6-31G (d,p) level, while the interaction energies have been evaluated at the RIMP2 (using aug-cc-pVDZ and RIMP2/aug-cc-pVTZ basis sets which has been further extrapolated to CBS limit), DFT and DFT-D (using TPSS functional and aug-cc-pVDZ basis set). Further, interaction energy decomposition has been carried out using Morokuma method performed at the HF/6-31G (d,p) level. Our studies show that on full geometry optimization, these pairs tend to achieve greater stability compared to that in their respective experimental geometries. The gas phase interaction energies of RNA-drug pairs compare well with those of standard Watson Crick base pairs. Except for the Puromycin - Guanine pair involving sugar edge-sugar edge interaction, the Hartree Fock component dominates the interaction energies in all fully optimized geometries. The comparison of EDFT and EDFT-D interaction energies suggest that the interaction between the RNA-Drug pairs is further stabilised due to dispersion.
In context of RNA-drug interactions involving multiple bases forming base pairs with single drug molecule, we also report detailed studies on the discriminative ability of the theophylline RNA aptamer with respect to its cognate ligand. Unlike in the context of purine riboswitches observed earlier, cooperativity between different RNA-base pairs is found to be negligible in multiple base-ligand interactions studied in case of Theophylline interacting with its aptamer. In addition, the binding affinities of theophylline, caffeine and theobromine with the theophylline aptamer correlates well with the experimental data with high binding affinity for theophylline and least for caffeine.

Abstract

Information explosion on the internet popularly known as the web has led to a phenomenon of the user unable to find the information relevant to him or her. There is plethora of information available and thousands of pages are being added everyday. Hence more and more information is available on each and every subject. Hence, it has become increasing difficult for users to remain up to date on one’s own areas of interest. This has increased the need for indexing of documents drastically. Indexing can take place at the user end or at the search engine end. However traditionally, indexing used to take place at the middle level by authorities or librarians. With the vast amount of content available on the internet, it is practically impossible for any authority to index the data. This led to the evolution of social bookmarking systems where users are able to maintain individual collections of resources and index them using tags for retrieval purposes. This model has evolved into a collaborative model where users share their resource collection and tags with others leading to the emergence of collaborative tagging systems. These systems are also referred to as “Folksonomies”. Due to the flexibility and ease of use, the usage of these systems has increased very rapidly. Many tagging systems like Delicious, Flickr, Bibsonomy, Citeseer etc. have evolved and became very popular. This has created a new form of rich user generated content which can be exploited to good use in a wide range of applications. The data in these systems typically consists of three dimensions - resources, users and tags. In this thesis, we have focused on some of the basic metrics for extracting information from folksonomies - Tag weight and Tag similarity and the usage of them in applications like Tag Search and Web document summarization. First, we propose a model of tag weighting extending the traditional Vector Space Model for term weighting. We show how tag weighting can be used in tag similarity and also show how we compute tag similarity. We evaluate our proposed tag similarity measure by grounding them against semantic lexicon WordNet using correlation measures - Kendall’s tau and Spearman’s rho correlation coefficients. We further demonstrate the application of our tag weighting in summarization of web documents. We have proposed a summarization feature based on tag weights which can be used with some of the existing features in generating summaries. We have evaluated the generated summaries by comparing them with manually generated summaries to find how close they are to the manual summaries and hence more readable. Further, we also demonstrated the application of our tag similarity in tag search. Our experimental results proved that our proposed approaches are able to achieve good improvement over our baselines.