Interactive Natural Language Acquisition in a Multi-modal Recurrent Neural Architecture

📝 Abstract

For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about socio-cultural conditions, and insights about activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mechanisms in the brain allow to acquire and process language. In bridging the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accordingly, we provide concepts and refinements in cognitive modelling regarding principles and mechanisms in the brain and propose a neurocognitively plausible model for embodied language acquisition from real world interaction of a humanoid robot with its environment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage characteristics and thus operate on multiple timescales for every modality and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations.

💡 Analysis

For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about socio-cultural conditions, and insights about activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mechanisms in the brain allow to acquire and process language. In bridging the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accordingly, we provide concepts and refinements in cognitive modelling regarding principles and mechanisms in the brain and propose a neurocognitively plausible model for embodied language acquisition from real world interaction of a humanoid robot with its environment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage characteristics and thus operate on multiple timescales for every modality and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations.

📄 Content

CONNECTION SCIENCE, 2018 VOL. 30, NO. 1, 99–133 https://doi.org/10.1080/09540091.2017.1318357 Interactive natural language acquisition in a multi-modal recurrent neural architecture Stefan Heinrich and Stefan Wermter Knowledge Technology Institute, Department of Informatics, Universität Hamburg, Hamburg, Germany ABSTRACT For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about sociocultural conditions, and insights into activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mech- anisms in the brain allow to acquire and process language. In bridg- ing the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accord- ingly, we provide concepts and refinements in cognitive modelling regardingprinciplesandmechanismsinthebrainandproposeaneu- rocognitively plausible model for embodied language acquisition from real-world interaction of a humanoid robot with its environ- ment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage char- acteristics and thus operate on multiple timescales for every modal- ity and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations. ARTICLE HISTORY Received 25 June 2016 Accepted 1 February 2017 KEYWORDS Language acquisition; recurrent neural networks; embodied cognition; multi-modal integration; developmental robotics

1. Introduction The human brain is seen as one of the most complex and sophisticated dynamic sys- tems. Humans can build precise instruments and write essays about higher purpose of life because they have reached a state of specialisation and knowledge by externalis- ing information and by interaction with each other. We not only utter short sounds to indicate an intention, but also describe complex procedural activity, share abstract declar- ative knowledge, and may even completely think in language (Bergen, 2012; Christiansen & Chater, 2016; Feldman, 2006; Håkansson & Westander, 2013). For us, it is extremely easy as well as important to share information about matter, space, and time in complex CONTACT Stefan Heinrich heinrich@informatik.uni-hamburg.de Knowledge Technology Institute, Department of Informatics, Universität Hamburg, Vogt-Koelln-Straße 30, 22527 Hamburg, Germany © 2017 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/) , which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 100 S. HEINRICH AND S. WERMTER interactions through natural language. Often it is claimed that language is the cognitive capability that differentiates humans most from other beings in the animal kingdom. However, humans’ natural language processing perhaps is the least well understood cognitive capability. The main reason for this may be the complexity of human language and our inability to observe and study this capability in less complex related species. Another reason is that the neural wiring in the human brain perhaps is not the only compo- nent, which is necessary for language to develop. It seems that socio-cultural principles are as well important, and only the inclusion of all factors may allow us to understand language processing. Nevertheless, it is our brain that enables humans to acquire perception capa- bilities, motor skills, language, and social cognition. The capability for language acquisition thus may result from the concurrence of general mechanisms on information processing in the brain’s architecture. In particular, in recent studies in neuroscience it was found that the brain indeed includes both hemispheres and all modalities in language processing, and the embodied development of representations might be key in language acquisi- tion (Barsalou, 2008; Glenberg & Gallese, 2012; Hickok & Poeppel, 2007; Huth, de Heer, Griffiths, Theunissen, & Gallant 2016; Pulvermüller & Fadiga, 2010). Furthermore, hierarchi- cal dependencies in connectivity were identified, including different but specific delays in information processing. In linguistic accounts and behavioural studies a number of impor- tant principles, suc

This content is AI-processed based on ArXiv data.