S&M2745 Research Paper of Special Issue
Published: November 30, 2021
Interactive Sound Generation to Aid Visually Impaired People via Object Detection Using Touch Screen Sensor [PDF]
Tias Kurniati, Chuan-Kai Yang, Tzer-Shyong Chen, Yu-Fang Chung, Yu-Min Huang, and Chih-Cheng Chen
(Received July 1, 2021; Accepted October 25, 2021)
Keywords: sound generation, touch sensor, object detection, image processing
Along with the invention of portable devices, such as smartphones and tablets, computer-based touch screen sensor-assistive technologies have become significantly more affordable than traditional tactile graphics. The sensor panel in these technologies allows users to receive visual and auditorial responses via interaction with the device. However, visually impaired individuals (with a lack or loss of ability to see) will not find visual responses useful when using tablets or smartphones. Therefore, in this paper we propose a system that helps visually impaired people comprehend information on electronic devices with the help of auditory action feedback. We develop a multimedia system for sound production from a given image via object detection. In this study, YOLO (You Only Look Once) is used in object detection for sonification. A pre-trained model is used; thus, a wider range of object classification can be identified. The system generates the corresponding sound when an object on the sensor screen is touched. The purpose of our research is to aid visually impaired people to perceive information of a picture shown on the device by touching the detected object. The device was tested by simulating visually impaired people by blindfolding people with normal vision, who filled out questionnaires on its performance. The results indicate that most of the users found that the sound presented by the device was helpful for telling them what the shown image was.Corresponding author: Tzer-Shyong Chen, Chih-Cheng Chen
This work is licensed under a Creative Commons Attribution 4.0 International License.
Cite this article
Tias Kurniati, Chuan-Kai Yang, Tzer-Shyong Chen, Yu-Fang Chung, Yu-Min Huang, and Chih-Cheng Chen, Interactive Sound Generation to Aid Visually Impaired People via Object Detection Using Touch Screen Sensor, Sens. Mater., Vol. 33, No. 11, 2021, p. 4057-4068.