## Overview VSL Labs is tackling one of the most challenging problems in machine translation: converting spoken/written English into American Sign Language (ASL). This presentation by Yaniv from VSL Labs provides insight into the complexity of the problem and how the company is leveraging generative AI models in production to create a comprehensive sign language translation platform. The company has developed a SaaS platform that accepts API requests for text or audio input and produces 3D avatar-based sign language output. This is not merely a text-to-gesture system but a sophisticated translation pipeline that accounts for the unique linguistic properties of sign languages, including grammar, syntax, regional dialects, and the critical non-manual markers that convey meaning through facial expressions and body positioning. ## The Problem Space The scope of the challenge is significant: approximately 500 million people worldwide are deaf or hard of hearing, with millions relying exclusively on sign language for communication. Sign language is not universal—there are many distinct sign languages (ASL in America, BSL in Britain, ISL in Israel) that are mutually unintelligible. A deaf American cannot easily communicate with a deaf British person, just as speakers of different spoken languages cannot communicate without translation. Several factors make this problem particularly acute. Human interpreters are scarce and expensive, with hourly rates that make comprehensive translation economically prohibitive for most content. Government-provided interpreter hours are often insufficient for daily needs. Additionally, written captions are not an adequate substitute for sign language translation because many deaf individuals have sign language as their first language and may not be fully fluent in the written language of their country. The presenter emphasizes an important neurological point: research shows that the auditory cortex in deaf individuals undergoes transformation, enhancing visual processing capabilities. This means deaf individuals are particularly attuned to visual communication, making high-quality visual translation crucial. ## Technical Challenges The presentation outlines several key challenges that make automated sign language translation particularly difficult for machine learning systems: **Grammatical Flexibility**: Sign languages have flexible word order that differs fundamentally from spoken languages. The example given shows that "home me" and "me home" can both mean "my home," with the order depending on emphasis. In longer sentences, this flexibility creates significant ambiguity for translation systems and makes it difficult to establish ground truth for training data. **Dialectal Variation**: Until approximately 15 years ago, before video calling platforms like FaceTime and Zoom became widespread, deaf communities in different geographical regions developed distinct dialects. Different communities evolved their own signing conventions, creating substantial variation within what is nominally the same language. This complicates the creation of training datasets and model generalization. **Register and Style Variation**: There are traditional signers who prefer pure sign language distinct from English influence, English-influenced signers who incorporate more English-like structures, and variations based on formal versus casual contexts. The system must accommodate this range. **Non-Manual Markers**: Perhaps the most critical challenge is that sign language is not just about hand movements. Facial expressions, eyebrow positions, head tilts, and body positioning all carry semantic meaning. For example, raised eyebrows can indicate a question, while forward leaning can indicate emphasis. Different colors in their demonstration showed different uses of these markers. The presenter emphasizes that any quality translation must incorporate these non-manual markers—this is something VSL Labs claims differentiates their product from competitors. ## Architecture and LLM Integration VSL Labs has built a modular pipeline that processes translation requests through several stages: **API Layer**: The platform exposes an API that accepts text or audio input along with parameters specifying output preferences, including avatar selection (they mentioned "Nina" as an avatar option), background settings, and other customization options. This suggests a production-ready system designed for integration into third-party applications. **Text Preprocessing Module**: Before translation, input text undergoes preprocessing that handles long texts and performs cultural adaptation. This includes linguistic simplification when appropriate for the target audience—an acknowledgment that different users (children vs. adults, different cognitive abilities) may require different translation approaches. **Translation Module**: This is where the generative AI models come into play. The system uses either T5 (described as their in-house model) or GPT-4 to translate from the source language (English) into a sequence of glosses (written representations of signs) plus what the presenter calls "stage directions"—instructions for how the avatar should perform each sign. **Gloss-to-Database Mapping**: An intermediate module maps the generated glosses to their sign database, which contains the actual motion data for each sign. **Post-Processing and Synthesis**: Additional models add appropriate behaviors for emotions, numbers, and other special cases. The presenter notes there is "a lot of work" in handling various special cases that were not detailed in the presentation. **3D Avatar Rendering**: The final output is a 3D animated avatar performing the signs with appropriate facial expressions, body movements, and timing. ## Model Selection and Trade-offs The use of both in-house T5 models and GPT-4 suggests a pragmatic approach to model selection. T5, being an encoder-decoder model well-suited for translation tasks, can be fine-tuned on domain-specific data and run in-house for latency-sensitive applications or cost optimization. GPT-4 offers superior generalization and handling of edge cases but comes with API costs and latency considerations. The decision to support both models likely reflects different use case requirements: real-time applications like airport announcements may prioritize speed and use the T5 model, while less time-sensitive content like video translation might use GPT-4 for higher quality. ## Production Use Cases The presentation highlights several real-world deployment scenarios: **Aviation and Transportation**: Airport announcements are particularly stressful for deaf travelers because boarding calls, gate changes, and other critical information are typically delivered via audio. VSL Labs focuses on this vertical partly because the content is relatively predictable (limited vocabulary, structured announcements), making it easier to achieve high quality while still providing significant value. **Automotive**: They work with Hyundai's innovation center in Israel on in-vehicle displays and alerts. **Video Conferencing**: The presenter noted an interesting UX insight from working with deaf colleagues: when presenting slides to a deaf person, you cannot point at the slide while speaking simultaneously because they need to watch the signer. This illustrates the subtle accessibility challenges the technology addresses. **Video Content**: Production-quality sign language interpretation for media content is extremely expensive (the presenter mentions the Barbie movie as an example of a costly translation project), making automated solutions economically attractive. ## Competitive Landscape and Barriers to Entry The presenter makes an important observation about why major tech companies have not solved this problem despite having vastly more resources. Companies like Google and Meta, who would be "very happy" to release such technology (similar to how they released Google Translate), have not done so, implying the problem is genuinely difficult and requires specialized expertise beyond general ML capabilities. Intel is mentioned as working on the reverse direction (sign language to text recognition) but reportedly without a production-ready product. VSL Labs currently focuses on one direction (text to sign language) with plans to eventually support bidirectional translation. ## Data Challenges While not explicitly discussed in detail, the presentation implies significant data challenges inherent to this domain. Creating training data requires agreement on correct translations, which is difficult given the dialectal and stylistic variation in sign languages. The grammatical flexibility means multiple valid translations exist for any given sentence, complicating evaluation metrics. The presenter's mention of "tagging" difficulties suggests they have encountered challenges in creating labeled datasets, likely requiring significant involvement from deaf community members and sign language experts. ## Key Differentiators Claimed VSL Labs positions their quality differentiator around proper handling of non-manual markers—the facial expressions and body movements that carry significant meaning in sign language. They claim this is missing from competitor solutions and is essential for comprehensible, natural-looking translation. The company also employs members of the deaf community, which both informs product development and requires using their own translation technology for internal meetings—creating a strong dogfooding incentive for quality.