Web-based MPEG-V standard-compliant platform for multimedia content consumption with multisensory effects

- Mul timedia systems typically include audiovisual (AV) content that stimulates only the senses of sight and hearing. However, stimulating other senses beyond these two can provide more realistic and immersive multimedia experiences, as well as help improve the user’s quality of the media consumption experience (QoE). The research community has been working for years on the development of so-called mulsemedia systems, which include sensory effects metadata associated with AV content, allowing those effects to be generated during the experience, thereby stimulating all the users’ senses. Examples of effects include scents, flavors, vibration, pressure, wind, lighting, temperature or fog. Existing solutions and standards (e.g., MPEG-V) already enable the integration of sensory effects generation devices into multimedia systems. However, most of them are not openly available or have been designed for and evaluated in desktop scenarios. Moreover, many of them use non-commercial generation devices, developed ad hoc to generate specific effects. This article proposes an MPEG-V standard-compliant platform based on web technologies and IoT-related devices for the distribution and consumption of multisensorial content, enabling the use of typical commercial effect generation devices. It also allows a 3D simulation of the effects generation and the location from which they are generated. A prototype implementation is presented, the operation of which has been subjectively validated by 40 users and media contents of different genres. All participants confirmed, among other things, that they perceived a good synchronization between sensory effects and AV content playout, that the inclusion of sensory effects adds value to the AV content, improving enjoyment, immersion, realism, and, therefore, the QoE during media consumption, and rated the usability of the developed prototype’s graphical interface as ‘excellent’ (82.3 in SUS scale) .