Multimodal Assistive Technology For The Support of Students With Multiple Disabilities

This paper describes a multimodal assistive technology, called NJOY, to help students with multiple disabilities or with some type of special needs in terms of mobility, communication, and learning impairments. The basis of the entire system is an App that runs on a Tablet with a multimodal interaction system that includes a joystick that was developed for this purpose, a symbol-based communication system that aims to increase or compensate for communication, language, and learning difficulties with audio feedback. The system also integrates with a home automation system, which can be controlled using the same App and the same interaction mechanisms, and an SMS module that allows the read, through text-to-speech technology, of the received SMS, and send predefined SMS messages to predefined users. The system was tested by a sixteen-year-old student with degenerative disease of the central nervous system including severe motor, articulation, visual impairment, and limitations at a cognitive level, which greatly compromises its functionality, depending on adults to carry out the basic tasks of daily life. The results were very promising with the user being able to use the system without any major difficulties.