As we have shown, we can build systems that will help learners experience authentic foreign- language content, relying minimally on professional translations, while still allowing users to comprehend, enjoy, and learn from the material. Smart Subtitles have illustrated that
intermediate-level learners can enjoy and comprehend videos without requiring that subtitles be displayed by default, and learn more from it than from existing subtitles. Our grammar
visualization has illustrated that it can help even users with no experience to the language to make sense of complex foreign-language sentences.
Much work can still be done in the area of incorporating multimedia into learning. In particular, we have thus far focused on (written) vocabulary learning in the Smart Subtitles system, and on comprehension in our grammar visualization system. We believe that multimedia augmentations can also benefit the other aspects of language learning - in particular, pronunciation, listening comprehension ability, and grammar learning. For example, we expect that the voice synthesis integrated into the grammar visualization system should presumably help with the pronunciation aspect in particular, as we observed in preliminary testing with our manga reader that users would often repeat back the synthesized phrases upon hearing them pronounced.
In addition, we can explore additional means of engaging learners while they are consuming multimedia. For example, in the Smart Subtitles system, we observed that in informal, non-lab settings users would gradually make less use of the system’s features if they were using it for prolonged periods of time, suggesting that they were growing disengaged. We could potentially detect when users grow disengaged, and strategically embed vocabulary quizzes into the video stream to ensure that they continue making active efforts towards learning while using the system. Alternatively, we can personalize the learning experience to detect harder portions of the dialog and provide additional scaffolding for lower-level learners - perhaps slowing down the rate of dialog, or showing the English translations for certain phrases by default. This reduces the amount of required interaction from users, helping to avoid fatigue over prolonged usage.
In addition to the systems and media we have focused on - videos and manga - additional systems can potentially be built to assist learners with passion for different forms of multimedia. For example, Smart Subtitles can be easily used with songs, provided that we have obtained the timing information ahead of time. However, songs have the additional property that people often enjoy singing them - therefore, we could potentially explore augmenting traditional games such as karaoke to support language learning.
This work can potentially lead to a future where people can learn foreign languages more enjoyably by being immersed and enjoying the culture of foreign countries, in the form of their multimedia, without requiring dedicated effort towards making the material education-friendly or even fully translating it.
Bibliography
[1] Danan, Martine. Captioning and Subtitling: Undervalued Language Learning Strategies. Translators’ Journal, v49 (2004).
[2] Mitterer H, McQueen JM. Foreign Subtitles Help but Native-Language Subtitles Harm Foreign Speech Perception. PLoS ONE (2009).
[3] Raine, Paul. Incidental Learning of Vocabulary through Authentic Subtitled Videos. JALT (2012).
[4] Danan, Martine. "Reversed subtitling and dual coding theory: New directions for foreign language instruction." Language Learning 42.4 (1992): 497-527.
[5] d’Ydewalle, Géry. "Foreign-language acquisition by watching subtitled television programs." Journal of Foreign Language Education and Research 12 (2002): 59-77.
[6] Bianchi, Francesca, and Tiziana Ciabattoni. "Captions and Subtitles in EFL Learning: an investigative study in a comprehensive computer environment." (2008).
[7] KMPMedia. http://www.kmpmedia.net/
[8] Dummitt, Nathan. Chinese Through Tone & Color. Hippocrene Books, 2008. [9] Using KMPlayer to watch a movie with 2 subtitles at the same time
http://boardgamegeek.com/blogpost/8836/using-kmplayer-to-watch-a-movie-with-2-subtitles-a
[10] Learn Chinese with Yangyang 001 http://www.youtube.com/watch?v=zNoOdNvdZlg
[11] Jay Chou 周杰伦- Silence 安靜 English + Pinyin Karaoke Subs
http://www.youtube.com/watch?v=QvN898LaTTI
[12] Wesche, Marjorie, and T. Sima Paribakht. "Assessing Second Language Vocabulary Knowledge: Depth Versus Breadth." Canadian Modern Language Review 53.1 (1996): 13-40. [13] WebVTT: The Web Video Text Tracks Format. http://dev.w3.org/html5/webvtt/
[14] Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky and Christopher Manning. 2005. A Conditional Random Field Word Segmenter. In Fourth SIGHAN Workshop on Chinese Language Processing.
[15] Bird, Steven, Ewan Klein, and Edward Loper. Natural language processing with Python. O'Reilly Media, 2009.
[16] "Learning Accurate, Compact, and Interpretable Tree Annotation" Slav Petrov, Leon Barrett, Romain Thibaux and Dan Klein in COLING-ACL 2006
[17] S. Kurohashi, T. Nakamura, Y. Matsumoto and M. Nagao, Improvements of Japanese morphological analyzer JUMAN," In Proceedings of Int. Workshop on Sharable Natu- ral Language Resources, Nara, Japan, 1994.
[18] Kurohashi, Sadao, and Makoto Nagao. "Kn parser: Japanese dependency/case structure analyzer." Proceedings of the Workshop on Sharable Natural Language Resources. 1994. [19] CC-CEDICT. 2013. Available from http://cc-cedict.org/wiki/
[20] WWWJDIC. 2013. Available from http://wwwjdic.org
[21] Albrecht, Joshua S., Rebecca Hwa, and G. Elisabeta Marai. "Correcting automatic translations through collaborations between MT and monolingual target-language users." Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2009.
[22] Xiaoyi Ma. 2005. Chinese English News Magazine Parallel Text. Linguistic Data Consortium, Philadelphia
[23] CC-CEDICT. 2012. Available from http://cc-cedict.org/wiki/
[24] Bansal, Mohit, John DeNero, and Dekang Lin. 2012. "Unsupervised Translation Sense Clustering." Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
[25] Kristina Toutanova and Christopher D. Manning. 2000. Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000), pp. 63-70.
[26] SenseEval-3 Data. ACL-SIGLEX. http://www.senseval.org/senseval3
[27] D. Albanese, R. Visintainer, S. Merler, S. Riccadonna, G. Jurman, C. Furlanello. mlpy: Machine Learning Python, 2012. arXiv:1202.6548 [bib]
[28] DeNeefe, Steve, Kevin Knight, and Hayward H. Chan. "Interactively exploring a machine translation model." Proceedings of the ACL 2005 on Interactive poster and demonstration sessions. Association for Computational Linguistics, 2005.
[29] Callison-Burch, Chris. "Linear B system description for the 2005 NIST MT evaluation exercise." Proceedings of the NIST 2005 Machine Translation Evaluation Workshop. 2005. [30] Sakunkoo, Nathan, and Pattie Sakunkoo. "GliFlix: Using Movie Subtitles for Language Learning. UIST 2009 Companion (demo paper), Vancouver, Canada, Oct 2009.
[31] Chrome Web Store - Rikaikun.
https://chrome.google.com/webstore/detail/rikaikun/jipdnfibhldikgcjhfnomkfpcebammhp?hl=en
[32] Yu, Chen-Hsiang, and Robert C. Miller. "Enhancing web page skimmability."Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems
[34] Tatoeba project. http://tatoeba.org
[35] Kornbrot, Diana. "Pearson Product Moment Correlation." Encyclopedia of Statistics in Behavioral Science (2005).
[36] Amazon Mechanical Turk. https://www.mturk.com
[37] oDesk. https://www.odesk.com/
[38] ALPAC (1966) "Languages and machines: computers in translation and linguistics". A report by the Automatic Language Processing Advisory Committee, Division of Behavioral Sciences, National Academy of Sciences, National Research Council. Washington, D.C.: National Academy of Sciences, National Research Council, 1966. (Publication 1416.) [39] Geza Kovacs, Joy Chen, Xinyi Zhang. Translation Sense Disambiguation on Chinese- English. http://geza.csail.mit.edu/translation-sense-disambiguation.pdf
[40] Ripping subtitles from video files using SubRip. http://zuggy.wz.cz/guides/video.htm [41] Wiktionary. http://www.wiktionary.org/
[42] Microsoft OneNote. http://office.microsoft.com/en-us/onenote/
[43] OneNote for Developers. http://msdn.microsoft.com/en-us/office/aa905452.aspx
[44] Smith, Ray. "An overview of the Tesseract OCR engine." Document Analysis and Recognition, 2007. ICDAR 2007. Ninth International Conference on. Vol. 2. IEEE, 2007.
[45] C. Harris and M. Stephens (1988). "A combined corner and edge detector". Proceedings of the 4th Alvey Vision Conference. pp. 147–151.
[46] OpenCV. http://opencv.org/
[47] H. Samet and M. Tamminen (1988). "Efficient Component Labeling of Images of Arbitrary Dimension Represented by Linear Bintrees". IEEE Transactions on Pattern Analysis and Machine Intelligence (TIEEE Trans. Pattern Anal. Mach. Intell.) 10: 579. doi:10.1109/34.3918.