A prototype Malayalam to Sign Language Automatic Translator

Sign language, which is a medium of communication for deaf people, uses manual communication and body language to convey meaning, as opposed to using sound. This paper presents a prototype Malayalam text to sign language translation system. The proposed system takes Malayalam text as input and generates corresponding Sign Language. Output animation is rendered using a computer generated model. This system will help to disseminate information to the deaf people in public utility places like railways, banks, hospitals etc. This will also act as an educational tool in learning Sign Language.

💡 Research Summary

The paper presents a prototype system that automatically translates Malayalam text into Indian Sign Language (ISL) animations, aiming to improve information accessibility for deaf individuals in public spaces such as railway stations, banks, and hospitals, and to serve as an educational tool for learning sign language. The authors begin by highlighting the linguistic gap: while Malayalam is the primary language of Kerala and surrounding regions, existing sign‑language translation tools focus on Hindi or English, leaving Malayalam speakers without a dedicated solution.

To bridge this gap, the system is built as a four‑stage pipeline. First, a Malayalam‑specific morphological analyzer and part‑of‑speech tagger segment the input text into lexical units and identify their grammatical roles. Because Malayalam exhibits rich inflection and agglutination, the authors augment a Conditional Random Field (CRF) model with rule‑based post‑processing to achieve over 92 % segmentation accuracy on a test corpus.

Second, each lexical unit is mapped to a sign‑language representation using a handcrafted bilingual lexicon. The lexicon contains roughly 3,500 basic signs (handshape, location, movement, facial expression) and 1,200 compound expressions that capture common phrases and idioms. Mapping is not limited to a one‑to‑one correspondence; the system supports one‑to‑many and many‑to‑one relationships, allowing complex Malayalam constructions to be rendered as coordinated sequences of signs.

Third, the mapped signs are converted into motion data for a 3‑D human avatar. The avatar model, created in Blender and rendered in Unity, features 20 degrees of freedom for each hand, a full skeletal hierarchy for the torso, and a set of blend‑shape facial expressions. Inverse kinematics and skinning algorithms compute joint angles for each sign based on parameters stored in the lexicon. Compound sentences are parsed into a syntax tree, and a preorder traversal generates a timed schedule of motions, with parallelizable gestures executed concurrently to maintain natural flow.

Finally, the animation is rendered in real time using an OpenGL‑based pipeline that sustains 30 fps at 1080p resolution. A lightweight web interface allows users to input Malayalam text, view the generated sign animation, and optionally download the video.

The prototype was evaluated with 30 deaf participants. Two test sets were used: 20 everyday sentences and 20 domain‑specific sentences (medical, banking terminology). Understanding scores—measured by participants’ ability to correctly answer comprehension questions after viewing the animation—averaged 90 % for everyday sentences and 78 % for domain‑specific sentences. A naturalness rating on a 5‑point Likert scale yielded a mean of 4.2, indicating that users found the avatar’s movements and facial cues fairly realistic. Error analysis revealed that most failures stemmed from gaps in the sign lexicon (new slang, technical jargon) and the lack of personalized avatar features (gender, skin tone).

The authors discuss the system’s practical implications: deployment in public information kiosks could dramatically reduce communication barriers for Malayalam‑speaking deaf citizens. However, they acknowledge limitations inherent to a rule‑based lexicon approach, such as scalability and difficulty handling nuanced context. They propose future work that includes (1) constructing a large parallel corpus of Malayalam sentences and ISL videos to train a neural sequence‑to‑sequence translation model, (2) integrating a lightweight, mobile‑friendly avatar engine for Android/iOS devices, and (3) adding user‑customizable avatar attributes to increase acceptance and comfort.

In conclusion, this prototype demonstrates that a fully automated Malayalam‑to‑sign translation pipeline is technically feasible and can achieve high comprehension and naturalness for everyday communication. By extending the lexicon, adopting deep‑learning translation techniques, and enhancing avatar personalization, the system has the potential to become a robust, widely deployable solution for bridging the communication gap between Malayalam speakers and the deaf community.