Problems with words in Bengali

In the context of processing Bengali words through a computer, there may arise several issues that are directly linked with surface structure of words. These issues may create problems in manual and computer-based counting of number of words in a corpus. They can also create problems in morphological processing of words. These issues come up because there is hardly any consistency in orthographic representation of words in written Bengali texts. The high irregularities in writing of inflected words, proper names, adjectival forms, adverbial forms, compound words, reduplicated words, onomatopoeic words, hyphenated words, etc. present a daunting task before an investigator in normalizing the surface forms of words for generating a lexical database as well as developing a word processing system for the works of language technology.

Perhaps, this is not the problem of the Bengali language alone. Most of the Indian languages also face similar problems due to inconsistencies in orthographic representation of words in writing. Therefore, before the work of frequency count of words is carried out on the Bengali corpus, it was necessary to adopt the following strategies without which it might have created some wrong observations and false deductions.

1. • Each unbroken string of characters is separated with a white space before and after it is treated as a single word unit. The white space before and after the string has worked as a marker of word boundary.

2. • All compound words, adjectival forms, and adverbial forms written with a space in between two members are treated as single word units to maintain consistency in counting their frequency of occurrence in the corpus. That means a white space existing between the two formative members is ignored and the entire form is treated as a single word unit.

3. • Words with a hyphen mark in between two formative members (e.g., compound words, prefixed words, reduplicated words, etc.) are treated as single word units.

4. • Consecutive occurrence of two identical forms of words, separated by space or hyphen, is treated as an instance of a reduplicated word.