默默有声

人工智能为本的手语学习，构建共融世界

# 减少不平等

# 社会共融

# 手语

刘婉瑜

现时全球每一千名初生婴儿中，就有一名患有听障。手语不仅是聋人沟通时打的手势，有独特的语言系统和文化。家中有聋人的健听人士学习手语，有助于打破聋健沟通隔膜。

健听人士通常难以掌握手语语法的细微差别，包括手掌与手指的协调、面部表情、头部位置和嘴形，任何误读都会影响讯息传递。一般人或认为以纸笔交流或文字讯息已可满足聋健沟通需要，但中大手语及聋人研究中心副主任施婉萍教授却不以为然。

她认为聋人要掌握书面语并不容易，因为手语语法与中文语法不同，「中文句子『我没有书』，在手语中则表达为『我书没有』，若要不谙中文读写的聋人按中文语法表达手语，犹如要常人按英文语法表达中文。」因此，要体现真正的共融社会，务必要理解聋人的沟通需要。

寓手语学习于游戏

中大与Google、日本财团和关西学院大学联手推出「手语村」，是全球首个结合人工智能和手语语言学理论的多语言网络游戏。游戏应用人工智能机器学习，可识别3D手语动作，让玩家在线上轻松学习香港和日本手语。

观看短片，了解「手语村」的详情

「手语村」虚拟小镇以手语为官方语言，玩家进入小镇后，须于电脑镜头前打手语，并完成各种日常生活任务，包括准备旅行用品、寻找住宿、在咖啡室点餐等。游戏开始时，萤幕上会出现两个视窗，显示两个与任务相关的手语动作，由玩家任选其一并在镜头前模仿该动作，人工智能识别系统会即时评估和回应玩家打手语的准确度。「手语村」的卡通人物「手形」更会不时现身，向玩家讲解「手语」、「聋人」和「聋人文化」的定义及概念。

「手语村」是「Shuwa计划」的首个国际跨学科协作项目，由中大、Google、日本合作伙伴共同策划，并于2021年9月23日「国际手语日」推出，旨在活用先进科技打破聋健沟通隔膜。 Shuwa是日语汉字「手话」的拼音，意即手语。

人工智能侦测手语动作

以往的手语识别技术难以分析视觉及姿势语言讯息，未能精准解读手语。手语除了手部动作，身体动作、面部表情、嘴形等在手语语法中同样重要，忽略任何细微特征，都可导致手语不合文法或无法诠释。

虽然世界各地的手语有所不同，但当中的语言特征如手形、方向、动作等却是相通的，而且组合变化有限，可以透过电脑识别。由于手语是3D空间运动，包括上下左右移动以及前后深度，以往手语识别需要采用特殊3D摄像镜头和带传感器的手套等复杂设备，令技术难以普及。

「Shuwa计划」团队解决了旧有技术的不足，成功研发首个机器学习模型，可透过一般摄影镜头识别、追踪及分析不同肢体信息，包括3D手部和肢体动作，以及面部表情特征，将其应用在「手语村」。现时，玩家可选择香港及日本手语，团队计划日后会加入其他地区的手语。

中大手语及聋人研究中心为「Shuwa计划」提供手语语言学专业知识及香港手语语法特征，以训练机器学习模型识别手语。施教授说：「我们应用手语语言学研究所得，帮助协作伙伴研发可自动识别手语的工具，让玩家实时知道自己能否打出准确的手语。」

计划的下一步是创建包含搜索功能的手语词典平台，透过人工智能技术协助用户学习和纪录手语。团队冀开发一个自动翻译模型，利用电脑或智能手机常用的镜头识别手语中的自然对话，将其翻译为口语。

关西学院大学手语研究中心主任松冈克尚教授表示：「我们很高兴能够参与『Shuwa计划』的技术开发，盼望将来或可开发为医疗界和法律界专家而设的手语学习工具。这些新发明都非常值得我们探索。」

玩家选择答案后，于镜头前模仿并打出相关手语

人工智能识别工具正在追踪玩家的身体姿势

施教授说：「我们的长远目标是建设一个无障碍社会，能够真正包容聋人和惠及手语交流人士。虽然长路漫漫，但『手语村』无分疆界，提供学习和了解手语的机会，打开了首道手语沟通之门。」未来，「手语村」将加入不同国家的手语，各地玩家不仅可以学到更多手语，亦能分辨它们之间的差异。科技不断推陈出新，有利于建构包容及和谐的社会，让人人享有平等机会。

Jenny Lau is an editor in the Communications and Public Relations Office, The Chinese University of Hong Kong.

刘婉瑜为香港中文大学传讯及公共关系处编辑

对地球许下承诺

中大订立2038年实现碳中和目标

默默有聲

人工智能為本的手語學習，構建共融世界

# 減少不平等

# 社會共融

# 手語

劉婉瑜

現時全球每一千名初生嬰兒中，就有一名患有聽障。手語不僅是聾人溝通時打的手勢，有獨特的語言系統和文化。家中有聾人的健聽人士學習手語，有助於打破聾健溝通隔膜。

健聽人士通常難以掌握手語語法的細微差別，包括手掌與手指的協調、面部表情、頭部位置和嘴形，任何誤讀都會影響訊息傳遞。一般人或認為以紙筆交流或文字訊息已可滿足聾健溝通需要，但中大手語及聾人研究中心副主任施婉萍教授卻不以為然。

她認為聾人要掌握書面語並不容易，因為手語語法與中文語法不同，「中文句子『我沒有書』，在手語中則表達為『我書沒有』，若要不諳中文讀寫的聾人按中文語法表達手語，猶如要常人按英文語法表達中文。」因此，要體現真正的共融社會，務必要理解聾人的溝通需要。

寓手語學習於遊戲

中大與Google、日本財團和關西學院大學聯手推出「手語村」，是全球首個結合人工智能和手語語言學理論的多語言網絡遊戲。遊戲應用人工智能機器學習，可識別3D手語動作，讓玩家在線上輕鬆學習香港和日本手語。

觀看短片，了解「手語村」的詳情

「手語村」虛擬小鎮以手語為官方語言，玩家進入小鎮後，須於電腦鏡頭前打手語，並完成各種日常生活任務，包括準備旅行用品、尋找住宿、在咖啡室點餐等。遊戲開始時，螢幕上會出現兩個視窗，顯示兩個與任務相關的手語動作，由玩家任選其一並在鏡頭前模仿該動作，人工智能識別系統會即時評估和回應玩家打手語的準確度。「手語村」的卡通人物「手形」更會不時現身，向玩家講解「手語」、「聾人」和「聾人文化」的定義及概念。

「手語村」是「Shuwa計劃」的首個國際跨學科協作項目，由中大、Google、日本合作夥伴共同策劃，並於2021年9月23日「國際手語日」推出，旨在活用先進科技打破聾健溝通隔膜。Shuwa是日語漢字「手話」的拼音，意即手語。

人工智能偵測手語動作

以往的手語識別技術難以分析視覺及姿勢語言訊息，未能精準解讀手語。手語除了手部動作，身體動作、面部表情、嘴形等在手語語法中同樣重要，忽略任何細微特徵，都可導致手語不合文法或無法詮釋。

雖然世界各地的手語有所不同，但當中的語言特徵如手形、方向、動作等卻是相通的，而且組合變化有限，可以透過電腦識別。由於手語是3D空間運動，包括上下左右移動以及前後深度，以往手語識別需要採用特殊3D攝像鏡頭和帶傳感器的手套等複雜設備，令技術難以普及。

「Shuwa計劃」團隊解決了舊有技術的不足，成功研發首個機器學習模型，可透過一般攝影鏡頭識別、追蹤及分析不同肢體信息，包括3D手部和肢體動作，以及面部表情特徵，將其應用在「手語村」。現時，玩家可選擇香港及日本手語，團隊計劃日後會加入其他地區的手語。

中大手語及聾人研究中心為「Shuwa計劃」提供手語語言學專業知識及香港手語語法特徵，以訓練機器學習模型識別手語。施教授說：「我們應用手語語言學研究所得，幫助協作夥伴研發可自動識別手語的工具，讓玩家實時知道自己能否打出準確的手語。」

計劃的下一步是創建包含搜索功能的手語詞典平台，透過人工智能技術協助用戶學習和紀錄手語。團隊冀開發一個自動翻譯模型，利用電腦或智能手機常用的鏡頭識別手語中的自然對話，將其翻譯為口語。

關西學院大學手語研究中心主任松岡克尚教授表示：「我們很高興能夠參與『Shuwa計劃』的技術開發，盼望將來或可開發為醫療界和法律界專家而設的手語學習工具。這些新發明都非常值得我們探索。」

玩家選擇答案後，於鏡頭前模仿並打出相關手語

人工智能識別工具正在追蹤玩家的身體姿勢

施教授說：「我們的長遠目標是建設一個無障礙社會，能夠真正包容聾人和惠及手語交流人士。雖然長路漫漫，但『手語村』無分疆界，提供學習和了解手語的機會，打開了首道手語溝通之門。」未來，「手語村」將加入不同國家的手語，各地玩家不僅可以學到更多手語，亦能分辨它們之間的差異。科技不斷推陳出新，有利於建構包容及和諧的社會，讓人人享有平等機會。

Jenny Lau is an editor in the Communications and Public Relations Office, The Chinese University of Hong Kong.

劉婉瑜為香港中文大學傳訊及公共關係處編輯

對地球許下承諾

中大訂立2038年實現碳中和目標

Hearing the Sound of Silence

AI-based sign language learning cultivates a more inclusive world

# ReducedInequalities

# SocialInclusion

# SignLanguage

Jenny Lau

Nearly one in every 1,000 babies worldwide were born with a hearing impairment. Sign language, which is way more than mere gestures among deaf communicators, has its own language system and culture. Hearing people who live and deal with this population also need to learn sign language so as to be able to communicate with them.

Hearing people often find it difficult to master the grammatical nuances of sign language, including hand and finger coordination, facial expressions, head positions, and mouth shapes. Get one element wrong, and the message will often be misunderstood. Some people may think that the deaf would be able to cope perfectly well with written communication or text messaging. Professor Felix Sze, Deputy Director of CUHK’s Centre for Sign Linguistics and Deaf Studies (CSLDS), begs to differ.

She thinks that written communication poses a huge challenge to the deaf. ‘The Chinese sentence “I have no book” becomes “I book no have” in sign language. To require the deaf who are no native readers and writers of Chinese to follow Chinese grammar is like asking a Chinese-speaking person to express herself in Chinese, while following English syntax.’ Hence, an understanding of the communication needs of the deaf becomes a prerequisite for a truly inclusive society.

Immersive game for sign language learners

In collaboration with Google, the Nippon Foundation and Kwansei Gakuin University in Japan, CUHK has launched the first multi-language online sign language game, SignTown which integrates AI and sign linguistic theory. It allows players to learn sign languages in Hong Kong and Japan in an engaging online environment by harnessing machine learning that recognises 3D sign language movements.

Watch the video to learn more about SignTown

The game puts players in a virtual town whose official language is sign language. They are asked to make signs facing their computer cameras to complete tasks, including packing for a trip, finding a hotel, and ordering food in a café. When players see a split screen showing two signs in the activity, they need to pick one and imitate it in front of the camera. The AI-powered recognition model provides immediate feedback on signing accuracy. Cute hand-shaped cartoon characters appear from time to time, to explain to players the definitions and concepts of sign language, deaf people and deaf culture.

An initiative of Project Shuwa, SignTown has been made possible by international cross-disciplinary collaboration between CUHK, Google and Japanese counterparts, with the aim of bridging the gap between the deaf and hearing worlds with advanced technology. ‘Shuwa’ (‘手話’ in Japanese characters) means ‘sign language’. The game was launched on 23 September 2021, the International Day of Sign Language.

AI-based movement recognition

Previous sign language recognition models were not accurate enough because they struggled to analyse visual-gestural language data. Sign language deals with a range of gestural information, including hand and body movements, facial expressions and mouth shapes. Signing without these subtleties could result in ungrammatical or uninterpretable messages.

While sign languages vary from one country to another, phonetic features such as handshapes, orientations and movements are universal, and the number of possible combinations is finite. Therefore, recognition models are possible. Recognising sign language, including hand and body movements, requires advanced equipment such as special 3D cameras and gloves with sensors, making it hard to popularise the technology.

Project Shuwa has acknowledged these pain points by constructing the first machine learning model that uses a computer camera to recognise, track and analyse 3D hand and body movements as well as facial expressions. Currently, players can choose between Japanese and Hong Kong sign languages. More options will be available in the future.

The role of CSLDS in Project Shuwa is to provide sign linguistic knowledge and features of Hong Kong sign language to develop the machine learning model. ‘As sign linguists, we want to help our collaborators to develop a sign language recognition tool which is smart enough to tell whether the players are making accurate signs,’ said Professor Sze.

The future goal of the project is to build a sign dictionary that not only incorporates a search function, but also provides a virtual platform to facilitate sign language learning and documentation based on AI. The project team aspires to develop an automatic translation model that can recognise natural conversations in sign language and convert them into spoken language using the cameras of commonly used computers and smartphones.

Professor Katsuhisa Matsuoka, Director of the Centre for Sign Language Studies at Kwansei Gakuin University, said, ‘It is a great pleasure for our centre to be involved in the technical development of Project Shuwa. We hope in the future to be able to develop expert versions of learning tools for medical and legal professionals. These are all worth exploring.’

A player imitates the sign language shown in the video after choosing an answer

AI recognition tool tracking body poses of the player

‘Our long-term goal is to create a barrier-free society that is truly inclusive for deaf signers or people who may benefit from signing communications. There is still a long way to go, but SignTown has opened up a lot more opportunities for people around the world to learn and know more about sign language in real life situations,’ said Professor Sze. The team plans to incorporate more signs in SignTown, so that people around the world may learn more sign languages and be able to distinguish between them. With the help of the latest technology, they are ready to help realize the vision of an inclusive and harmonious society which offers equal opportunities for each of its members.

Jenny Lau is an editor in the Communications and Public Relations Office, The Chinese University of Hong Kong.

UP NEXT

Our Pledge to Mother Earth

CUHK commits to carbon neutrality by 2038