SIGN LANGUAGE COMMUNICATION BETWEEN JAPANESE-KOREAN AND JAPANESEPORTUGUESE USING CG ANIMATION YoshinaoAoki, Ricardo Mitsumori, Jincan Li Alexander Burger Graduate School of Engineering, Hokkaido University Sapporo, Japan aoki @media.eng.hokudai.ac.jp Bahnhof str.24a 86462 Langweid, Germany abu @joshu.forth-ev.de ABSTRACT In this paper we propose a sign language communication between different languages such as Japanese-Korean and Japanese-Portuguese using CG animation of sign language based on the intelligent image communication method. For this purpose sign language animation is produced using data of gesture or text data expressing sign language. In the production process of CG animation of sign language, MATLAB and LIFO language are used, where MATLAB is useful for three-dimensional signal processing of gestures and for displaying animation of sign language. On the other hand LIFO language, which is a descendant of the LISP and FORTH language families, is developed and used to produce live CG animations, resulting in a high-speed interactive system of designing and displaying sign language animations. A simple experiment was conducted to translate Japanese sign language into Korean and Portuguese sign languages using the developed CG animation system . 1. INTRODUCTION Sign language has a potentiality to communicate between different languages under the condition that verbal communication is limited because of lack of common language. To realize such a sign language communication, we adopt an intelligent image communication technique, where we transmit intelligently coded data, that is, parameters describing gestures or text data expressing meaning of sign language in a certain language, instead of transmitting images or motion pictures themselves. In the intelligent sign language communication, first we analyze images of sign language or gesture, then we describe the images of sign language or gesture using text, parameters and programs. We transmit This work was supported by TAO(Te1ecommunicationsAdvancement Organizationof Japan ) conducted as an intemational collaborative research by project on " International Collaborative Research on Communication Technique between Different Languages Using Sign Language". This research was also partially supported by grants from the Ministry of Education, Culture and Science of Japan to aid development in scientific research (grant numbers 08558026). these intelligently coded data and we synthesize the CG animation of sign language using received text, parameters and programs. We have investigated such a sign language communication [ 5 ] , [6] and conducted an experiment of transmitting sign language images using a Japanese communications satellite [ 11. We have also investigated a method to analyze the gesture by arms and hand for intelligent communication of sign language [2]. These experimentalresults suggest that it is necessary to develop a system of generating CG animation of sign language in real time with received data for the communication between sign languages of different countries. In this paper we focus not the analysis of the images of sign language, but generation and display of the sign language images corresponding to the meaning of sign language. Therefore we are developing such a system to generate sign language images from text and parameter data. Using this system we can translate easily from text data of sign language into CG animation and we confirm the possibility of the sign language communication between sign languages of different languages. 2. DEVELOPMENT OF DISPLAY SYSTEM OF 3-D MODELS FOR PRODUCING SIGN LANGUAGE ANIMATION USING MATLAB MATLAB was originally developed for matrix manipulation of linear algebra and signal processing. Now MATLAB has convenient GUI (Graphical User's Interface) to display the processed signals. This suggests that we can display the gesture signals using MATLAB without complicated programming as done in programming using C language. In order to construct 3-d limb model, we define a vector which expresses a skeleton of limb or finger as shown in Fig.1. Then we link these skeleton vectors to compose a limb model as shown in Fig.2. We use commands of GUI of MATLAB for displaying polyhedrons corresponding to the skeleton vectors to gen- 3765 0-7803-4428-6/98 $10.00 0 1998 IEEE erate a 3-d human limbs model.Then we move the human limb model according to the given joint angle parameters 0 1 I93 of each joints of limbs or fingers. - z A loverlimb Figure 1: Skeleton vector of a limb - 1- I little I right left right -50 50 I -50 I C d e f - I1 1I Table 1 Table of Joint Angles right & I region of rotation Iieit-1 e1 182 e3 e4 upper left -90 130 200 limb right -100 130 200 lower left -110 220 -6 I b Figure 2: Limb model Figures 3 a-f show a sequence of generated human limb models, that is an animation of a gesture. The joint angles (in degree) are listed in Table 1, where these joint angles corresponds to the image of Fig.3 e. The joint angles I94 and I95 in Table 1 are additional joint angles of each finger in local coordinates to describe finger motions. I joint a I e5 - Figure 3: Sign language animation produced by MATLAB, where a c are Japanese sign language images and d f are Korean sign language images tence is almost same in Japanese and Korean, we exchange gesture images in translating sign languages between two languages. Figures 3 a-c show the animation of Japanese sign language which express a sentence " I am a man while Fig.3 d-f show the same sentence in Korean. In the translation process of this Japanese sign language into Korean, we change the sign language images corresponding to the words "I" and "man" of sign language of Japanese into Korean. Japanese is an adhesive language and a postpositional words functioning as auxiliary to main words are frequently used, however, those words omitted in the expression of sign language. Therefore it is not necessary to change the order of the words of sign language images in translating Japanese into Korean or Korean into Japanese. Experiment of translation using simple sentences was conducted and the validity of the method of exchanging the sign language images in translation process between Japanese and Korean was confirmed. 'I, 0 -100 -50 0 100 50 0 I -100 I -50 I I 3. SIGN LANGUAGE TRANSLATION BETWEEN JAPANESE AND KOREAN 4. DEVELOPMENT OF HIGH-SPEED CG ANIMATION SYSTEM USING LIFO LANGUAGE Japan and Korea are neighboring countries and these two countries possesses similar linguistic systems and cultures. This suggests that sign language translation between Japanese and Korean is easier than those of Japanese and other European languages. Usually sign language is expressed according to the order of words in a sentence. Since the order of words in a sen- The disadvantage of producing animation using MATLAB is its inconvenience of designing animation in interactive method. It take much time to design a gesture with given joints parameter. Therefore we develop a system for designing gestures using LIFO language [3]. The LIFO language is based on the stack manipulation like FORTH language, 3766 5. PRODUCTION OF CG ANIMATION OF JAPANESE AND PORTUGUESE SIGN LANGUAGE FOR SIGN LANGUAGE COMMUNICATION resulting in high-speed processing of CG animation. In describing gesture or sign language, we describe the change of hand shapes as follows open par[[3 thumb] [lo ring1 [lo little11 A statement "open" opens a hand, then "par[ 1'' description generates a shape of the hand, bending thumb, ring and little fingers with the bending parameters 3, 10 and 10, resulting in index and middle fingers remain stretching. The command 'open' in turn is implemented as "[RHand < straight]", which reads as: "send the message '< straight' to the global right hand in 'RHand'". The method '< straight' of the hand class itself sends the message '< straight' (in a 'par'-body) to all members (fingers) of the hand object: In order to achieve real time animation of sign language using LIFO, the modeling of limbs is conducted with 2-d polygons. An arbitrary number of "models" can be painted to the z-buffer image, and then the complete scene can bedrawn to the screen. Each model has a recursive structure, consisting of a position vector, a rotation matrix, an arbitrary number of faces (polygon), and an arbitrary number of (sub-) models. For each face two colors (lighter and darker) are defined to allow a kind of relative illumination effect. : .thumb < ben I", effectively sending the message '< bend' to the '.thumb'-member of the hand. The following list is a part of source program written in lifo language, where OOP(0bject Oriented Programming) is used and classes and subclasses are defined for modeling of limbs. ( * * * Limb Classes * * * ) NIL class Limb # # .model :