Designing and building automated systems with which people can interact naturally is one of the emerging objective of Mechatronics. In this perspective multimodality and adaptivity represent focal issues, enabling users to communicate more freely and naturally with automated systems. One of the basic problem of multimodal interaction is the fusion process. Current approaches to fusion are mainly two: the former implements the multimodal fusion at dialogue management level, whereas the latter at grammar level. In this paper, we propose a multimodal attribute grammar, that provides constructions both for representing input symbols from different modalities and for modeling semantic and temporal features of multimodal input symbols, enabling the specification of multimodal languages. Moreover, an application of the proposed approach in the context of a multimodal language specification to control a driver assistance system, as robots using different integrated interaction modalities, is given.
People communicate and interact each other's and with external environment using all the five senses. Hence, a very natural interaction has to involve signals exchanged using all senses in both, voluntary and involuntary communication.
In fact, information that are voluntary conveyed by explicit messages are joined to information provided by involuntary communication, i.e. communication made with one’s body (i.e. gestures, eye movement, hand and/or arm placement, facial expression, etc. not explicitly conceived by the user for communicating with others or the environment).
The importance of voluntary and involuntary communication in the interaction processes between humans, humans and environment, humans and mechatronic systems can be easily taken off. Indeed, naturalness in the interaction process requires that the human and the mechatronic system have the same autonomy level removing the user’s control function of the first on the latter.
Mechatronics integrates mechanical, electronical and information-driven units allowed for turning conventionally designed mechanical components into smart devices.
In particular, mechatronic systems can exchange messages among them and with people and/or they can percept environmental changes using multimodal input and output actions. Recently, several authors have analyzed principles, features, and challenges of multimodal interaction [1] [2] and how it benefits the disambiguation of the dialogue [3] as well as the cognitive and learning processes of human-machine systems [4]. In [5], these principles, features, and challenges have been discussed in the particular perspective of mobile devices.
In most of the existing human-machine systems, humans have to learn, to act on machines and to acquire skills by themselves , without the machines’ assistance.
Considering actions as part of the communication process (i.e. from the events produced by someone on some other in order to modify it, to a reciprocal conversation between two or more entities) to improve the interaction effectiveness and naturalness between a human and a machine, the mechatronics should focus on human adapting the system’s behaviour to the operator’s skill level. That is, the Human Adaptive Mechatronics (HAM) represents a new and relevant challenge to design mechatronic systems. In this perspective, Harashima et al. [6] identified some relevant issues, such as: “Modeling human and machine dynamics. Especially the variable constraints should be considered. Modeling the operation base on skill, rule, knowledge, decision combining event and dynamical
Such a mechatronics has to be equipped with abilities to observe human behaviour and environment events and create actions. Modelling human-machine dynamics includes the development of an interaction language that should be a multimodal language (which will be introduced in Section 3), since multimodal interaction combines and integrates information from different input modalities (fusion process), and generates appropriate output information (fission process) enabling a natural dialogue. An integrated and general software environment for multimodal interaction languages specification is proposed to enhance and adapt the mechatronics’ interaction to the user.
We assume that programming, cooperating and interacting with a mechatronic system involve multimodal inputs, which means that the user can program the system “by example” by simply using speech, gesture or other modalities opportunely combined. Moreover, the system can acquire and update the multimodal interaction language according to the different contexts and users, where context refers to everything characterising the system interaction process with humans, according to the notion provided by Dey [7].
In particular, in this paper we focus our attention on defining a grammar-based approach for specifying and updating a multimodal language for interacting with a mechatronic system. For this purpose, we propose a multimodal attribute grammar, that provides constructions both for representing input symbols from different modalities and for modeling semantic and temporal aspects of multimodal dialogue. Specifically, we start from attribute grammars, firstly defined by Donald Knuth [8] as a means of formalizing the semantics of a context-free language 1 , and we extended these grammars introducing a multimodal attribute grammar that provides constructions both for representing input symbols from different modalities and for modeling semantic and temporal aspects of the multimodal dialogue.
The remainder of the paper is structured as follows. Section 2 briefly describes research activities related to this work focusing on mechatronics and human-machine interaction. In Section 3 a preliminary analysis of challenges and existing approaches to multimodal dialog interpretation is provided and an overview of our theoretical approach based on a multimodal attribute grammar is given. Section 4 illustrates
This content is AI-processed based on open access ArXiv data.
