Vol.12, No.3, August 2023.                                                                                                                                                                               ISSN: 2217-8309

                                                                                                                                                                                                                        eISSN: 2217-8333

TEM Journal

TECHNOLOGY, EDUCATION, MANAGEMENT, INFORMATICS

Association for Information Communication Technology Education and Science

Identifying and Analyzing Reduplication Multiword Expressions in Hindi Text Using Machine Learning

Atul Mishra, Alok Mishra

© 2023 Alok Mishra, published by UIKTEN. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. (CC BY-NC-ND 4.0)

Citation Information: TEM Journal. Volume 12, Issue 3, Pages 1732-1741, ISSN 2217-8309, DOI: 10.18421/TEM123-56, August 2023.

Received: 08 April 2023.

Revised:   18 July 2023.
Accepted: 29 July 2023.
Published: 28 August 2023.

Abstract:

The task of identifying and analyzing Reduplication Multiword Expressions (RMWEs) in Natural Language Processing (NLP) involves extracting repeated words from various text forms and classifying them into Onomatopoeic, non-Onomatopoeic, partial, or semantic types. With the increasing use of low-resource languages in news, opinions, comments, hashtags, reviews, posts, and journals, this study proposes a machine learning-based RMWE identification method for Hindi text. The method employs linguistic patterns and statistical data, along with a proposed threshold boundary detection in statistical filtering. The Jaccard distance of dissimilarity and Sorensen Dice Coefficient of Similarity are used for semantic relation analysis. The proposed approach was evaluated using the publicly available Hindi corpus from IITB, measuring performance between two consecutive thresholds with the lowest error and highest recall. This study proposes an effective method for Indian computational linguistics, with experimental results highlighting its viability and utility, and providing a blueprint for current procedures.

Keywords –Linguistic patterns, natural language processing, computational linguistics, statistical data, threshold boundary detection.

-----------------------------------------------------------------------------------------------------------

Full text PDF >  

-----------------------------------------------------------------------------------------------------------