Talk&Learn: Improving Conversation Experience and Creating Opportunities for Foreign Language Learning
📝 Abstract
Existing Real-Time Translation Interfaces (RTTI) do not provide experience as natural and efficient as monolingual communication. Also, such systems do not provide functions supporting language learning. This results in the waste of both time and potential language context. In order to overcome the above limitations, we propose a solution named “Talk&Learn”. Its core idea is to rearrange (“Delay-Match”) the real-time videos and translated texts or speeches, so as to gain better naturalness and efficiency. At the same time, this will create extra free time for users. So we further propose to utilize the free time for contextual language learning.
💡 Analysis
Existing Real-Time Translation Interfaces (RTTI) do not provide experience as natural and efficient as monolingual communication. Also, such systems do not provide functions supporting language learning. This results in the waste of both time and potential language context. In order to overcome the above limitations, we propose a solution named “Talk&Learn”. Its core idea is to rearrange (“Delay-Match”) the real-time videos and translated texts or speeches, so as to gain better naturalness and efficiency. At the same time, this will create extra free time for users. So we further propose to utilize the free time for contextual language learning.
📄 Content
Brief proposal 1 / 4
Talk&Learn: Improving Conversation Experience and Creating Opportunities for Foreign Language Learning Yaohua Xie Institute of Software, Chinese Academy of Sciences Yaohua.Xie@hotmail.com; fjpnxyh2000@163.com INTRODUCTION Nowadays, language barrier is still a big challenge despite of the greater and greater demand of cross- cultural communication[1]. On the one hand, people wish to communicate with others in different languages with the help of certain tools or services. On the other hand, many people wish they can talk freely in other languages by themselves eventually. However, existing Real-Time Translation Interfaces (RTTI)[2] do not provide experience as natural and efficient as monolingual communication. Also, such systems do not provide functions supporting language learning. This results in the waste of both time and potential language context. In order to overcome the above limitations, we propose a solution named “Talk&Learn”. Its core idea is to rearrange (“Delay-Match”) the real-time videos and translated texts or speeches, so as to gain better naturalness and efficiency. At the same time, this will create extra free time for users. Inspired by “Wait-Learning”[3] etc., we propose to utilize the free time for contextual language learning. LITERATURE REVIEW After decades of research, automatic Spoken Language Translation (SLT) is becoming more and more mature. Some researchers believe that it is usable in certain fields[4]. RTTI is the interface between language translation service and the users. Existing RTTIs usually show real-time video streams to users, then wait for translation, and finally display translated texts or play translated speeches[5, 6]. In most solutions, visual components such as facial expressions, eye gazes and gestures, are utilized to make communication easier. The combination of these components and translation technique is beneficial to user satisfaction and sense of spontaneity in conversation[2]. Text and speech have also been proven to be helpful for non-native speakers[7]. Researches in the field of second language learning have demonstrated that closed captions can improve participants’ comprehension of foreign language when seeing DVD videos[8]. Such “batch mode captions” are shown line by line, and may contain the speaker’s future words before being spoken in the video[9]. Therefore, people did not think such situation can happen in real-time conversations[9]. Similarly, translated speeches also appear after corresponding videos. In order to solve these problems, we propose the “Delay-Match” approach to synchronize videos and texts/speeches. Meanwhile, this approach also create extra “free time”, which we further propose to use for language learning. In addition, the problem of privacy could also be diminished by such a solution. In order to protect privacy, FocalSpace uses synthetic blur effects to diminish the background [10]. In contrast, our solution hides “irrelevant” videos, e.g., when users are waiting expressionlessly for translation results. These periods of time are utilized to learn previous speeches or texts. Many researches have demonstrated the value of contextual micro-learning[11, 12]. Various media have been employed such as instant messages[3], web pages[11], Facebook feeds[13] and live wallpapers[14]. For example, in ALOE prototype, selected sets of English words are dynamically replaced with their foreign translations, so that users can learn vocabularies during web browsing[11]. We propose to adopt Brief proposal 2 / 4
the integration of video, audio and text as the medium, and help users learn foreign words, phrases or sentences during listening, speaking and reading. Showing the history of translation not only could help users to comprehend long messages[1], but also provide the chance to review, practice and test their foreign language just-in-time while communicating. As 3D space provide more room for information visualization, we will try using 3D timeline to organize conversation history. METHODS In existing applications such as Skype Translator and Vocre, the user cannot understand what he/she hears when receiving the original video from the other one. Then he/she must wait for the translated result, ether in text or speech. Finally, he/she reads the text or listens to the speech, without the help of facial expression. The user’s brain is busy in most time of the above process, but the information he/she gets is not synchronous. We propose to delay the presentation of videos and match them with the translated texts or speeches. The translated texts can be shown as “batch mode captions” which used to be not available in video meeting[9]. The translated speeches can also replace the original ones in the video. The output of this process is called “synthesized videos” here. By doing so, the user can get natural perception of the other’s talk. Furthermore, th
This content is AI-processed based on ArXiv data.