Content for TR 26.854 Word version: 19.0.0
6 Haptics signals, media formats and device types
6.1 Introduction
6.2 Haptics signals
6.3 Haptic media formats
6.4 Haptics device types
...
...
6 Haptics signals, media formats and device types p. 20
6.1 Introduction p. 20
Haptics refers to the sense of touch, and encompasses the generation, manipulation, and perception of tactile sensations, forces, and motions. In the media domain it relates to the capture and rendering of physical information about objects and environments due to interactions with the user. The capture is generally performed with sensors and the rendering with hardware actuators allowing to render different modalities.
In general, three types of modalities are considered, tactile (vibration, temperature, pression), kinaesthetic (force) and proprioception (motion, acceleration). Those modalities are rendered in the human body through different mechanoreceptors. The density and properties of those receptors are different depending on the modality and the body parts. In addition, each individual has different sensibility, which can be managed through users' profiles and adaptation.
Finally, the rendering is performed with specific hardware actuators embedded into user devices. The different Haptic modalities are rendered with different actuators. Typically, tactile sensations can be rendered with vibrotactile (e.g. linear resonant actuators (LRA), eccentric rotating mass (ERM), temperature, wind (fan), force devices.
Haptics signals correspond to the raw haptics data at the output of the sensors or authoring tools, described in clause 6.2.
Haptics media formats are the formats representing the haptics signals once processed for use by a production framework, as described in clause 6.3. For example, the raw synthetic signal described in clause 6.2.3 can be transformed in any of the haptics media parametric representation formats described in clause 6.3.4.
6.2 Haptics signals p. 20
6.2.1 Introduction p. 20
Haptic signals are essential in capturing and transmitting tactile sensations in a variety of applications, ranging from simple vibrations in mobile devices to complex force feedback in virtual reality systems. Two types of signals exist, time-sampled signals typically captured by sensors, and synthetic signals of haptic effects, typically created from authoring tools.
6.2.2 Time-sampled signals p. 20
In Figure 6.2-1, two examples of time-sampled signals are provided. The left side shows the samples of 3D angular orientations and linear accelerations of a horse captured with an accelerometer positioned on the horseman. The right side illustrates the haptic signal of a texture (here a carpet) captured from the sound of a pen moving on the surface of the carpet in one direction.
Those haptics time sampled signals are generally used for signals captured from sensors.
Figure 6.2-1: Example of time sampled signals (left: Horse riding 3D angular orientations and linear accelerations captured with an accelerometer, right: texture captured from sound on one direction)
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
Generally, the sampling frequency is set between 2kHz and 16kHz (8kHz in Figure 6.2-1) with the haptic perceptual bandwidth being around [5Hz-1kHz], with some variations depending on the haptic modality, rendering device and human haptic receptor. 5Hz and below is particularly significant for force feedback. This leads to a raw data-rate average of 128 kbps per channel.
6.2.3 Synthetic signals p. 21
Synthetic signals of haptic effects are mostly generated with design tools and thus correspond to synthetic data. In Figure 6.2-2, two examples of synthetic signals are provided, on the left a heartbeat synthetic signal (thus the regularity of the signal), and on the right a synthetic rain effect (quasi random design) with 4 channels corresponding to 4 different target locations on the user.
Figure 6.2-2: Example of synthetic rendered haptic signal (left: heart beat synthetic signal, right: synthetic rain effect with 4 channels for 4 different locations on the user)
(⇒ copy of original 3GPP image)
(⇒ copy of original 3GPP image)
The synthetic aspect of those signals compared to time-sampled signals captured from sensor is obvious. In general, a synthetic signal is used to synthesize a short duration effect and is used only at some events or at dedicated point in time during a gaming or movie experience, by opposition to time-sampled signals that represent a continuous signal sampled at a constant frequency.
Synthetic signals need a synthesizer to be converted to a timed signal compatible with the actuators used. The raw bit-rate dependents on the number of effects and their distribution in time.
6.2.4 Comparison and applications p. 21
The choice between time-sampled signals and synthetic signals depends on the application and the desired level of control. Time-sampled signals are more suitable for applications requiring high fidelity and real-time capture of physical interactions, such as in medical simulations, playback of recorded signals (motion) or fine-grained tactile feedback systems. Synthetic signals, on the other hand, are ideal for applications where flexibility and ease of use are paramount, such as in mobile notifications, communication, movies or gaming, where the haptic feedback needs to be easily adjustable and consistent across a variety of devices. Synthetic signals are also well suited for XR applications, where haptics effect may be attached to events or objects.
6.3 Haptic media formats p. 22
6.3.1 Introduction p. 22
In this clause, haptics media formats are the formats representing the raw data (or haptics signals described in 6.2) and used at the production side (e.g. for encoding, packaging and delivery). Most common media formats for haptics are proprietary. However standardised interchangeable media formats (or mezzanine formats) have now been defined.
6.3.2 PCM format p. 22
PCM or Pulse Code Modulation is a digital format that represents analog signals in digital format as a series of samples at a regular sampling frequency. It has been defined for audio but can be used to store haptic signals from sensors.
Various file formats support PCM such as the audio WAV format. ISO/IEC MPEG Haptic [7] is using WAV as one of the input file format for time-sampled haptic signals. This type of file format uses different bit depth and data types to store the signal usually interpreted on a [-1 ; 1] range. For this reason, additional headers and metadata is required to specify the actual value ranges, gain or other characteristics of the signal essential for rendering. There are currently no standard defining this type of metadata and therefore the interpretation of the signal is often application dependent.
This PCM format is suitable for storing or distributing sensors data and its implementability has been demonstrated for audio signals. The drawback of the PCM format is its overhead as it is sampled continuously at constant frequency in time as opposed to parametric formats that are much sparser.
6.3.4 Proprietary parametric formats p. 22
Descriptive haptic is represented through parametric primitives. It can be described by an amplitude, a frequency, a duration, and a waveform shape such as sinusoid, or square. Two types of parametric effects are usually defined: transient and continuous. A transient effect is a short pulse with an amplitude, a position in time or space, and a reference frequency. A continuous effect is represented by a frequency and an amplitude information modulated on the timeline.
Existing proprietary descriptive formats use JSON or XML to describe haptic effects. Two of those proprietary formats are commercially deployed: AHAP from Apple [18] and IVS [19] from immersion corp.
AHAP has been mainly developed for vibrotactile signals and implementation into iOS Core Haptic framework.
IVS was targeting media applications (movies, gaming) and communication (smartphone) using mainly vibrotactile signals described in XML format. A binary version is available for networks transmission.
HAPT is a vendor-specific haptic coding format based on the RIFF format. It is designed to enable efficient coding of a device-specific haptic effects. HAPT is integrated in Immersion's SDK [15].
These formats may further be encoded and compressed using haptic codecs. The ISO/IEC MPEG Haptic [7] codec supports both the AHAP and IVS formats, while the IEEE codec [30] only supports PCM input signals.
6.3.5 Standardized and Interchangeable parametric formats p. 22
SMPTE defined the Haptic-Tactile Essence for Broadcast Production Applications [17]. However, it was mainly dedicated to position, orientation, velocity and acceleration.
MPEG HJIF: MPEG defined in clause 6 of specification [7] a standardized and interchanged parametric JSON based format (or mezzanine format) for describing haptics effects called HJIF. An example of an HJIF file is provided in Annex A.
This format is compatible with the proprietary AHAP format. Transcoding from and to the other is possible. Similarly, the IVS format can be transcoded from XML to JSON, for most of its functionalities, since similar parametric patterns/functions are used. Transcoding of the primitives is lossless.
In addition, MPEG HJIF can integrate a binary encoded version of a PCM signal.
The main advantage of MPEG HJIF is its flexibility to support various haptic modalities, various parametric formats and its flexibility for heterogeneous end-user devices and experiences. Thanks to its JSON formatting, it is also easy to edit and modify the file.
MPEG HJIF can further be encoded and compressed using MPEG haptic codecs, in a binary format for networks transmission.
6.4 Haptics device types p. 23
6.4.1 Introduction p. 23
Haptics devices may be categorized into types defining their primary capability. Each category reflects the device's intended application and the type of haptic experience it provides. Distinction is done between input (sensors) and playback (actuators or rendering devices) functionalities of devices. These two functions can be collocated or be on different devices.
6.4.2 Haptics device type 1: Basic sensory feedback p. 23
Devices in this category are designed to provide passive, non-spatialised sensory feedback, in applications such as simple mobile game and entertainment, alerting, or communication.
They include actuators integrated into smartphone, smartwatch, wearables, finger UI, headphone, HMD. They usually have a low number of actuators and usually vibrotactile actuators.
In this category it is anticipated that the haptic signals are synthetic haptic effects created by a designer.
In addition, some simple sensors to capture sensory information mostly related to the environment, such as temperature, pressure, sound, humidity can be available into smartphones and wearables. Generally, it consists of a single sensor with time sampled signals recorded or sent.
6.4.3 Haptics device type 2: Sensorial texture feedback or Spatial sensory feedback p. 23
Devices in this category are designed to deliver highly detailed and precise haptics (e.g., tactile, kinaesthetic, proprioception) sensations. These devices simulate the feel of textures, pressure, and other subtle interactions, enabling users to experience intricate touch-based feedback. They are commonly used in applications where detailed touch interaction is crucial, such as virtual reality try-on, objects manipulation, training/education.
Fine-tuned sensory feedback devices include Haptic gloves Haptic touchpads, touch-sensitive surfaces and screens. Haptic modalities here are not restricted to vibrotactile, kinaesthetic and mid-air feedback is also considered.
For the capture, sensors in this category capture sensory information related to objects surfaces physical properties. It consists of more complex systems combining tracking and measurement to get the localization and physics. The tracking measures the location and speed of the sensors, the measurement unit captures the pressure, force feedback and motion to infer smoothness, roughness, relief and other physical properties of a 2D surface.
6.4.4 Haptics device type 3: Full-body and complex motion feedback p. 23
Devices in this category provide immersive, whole-body sensations, but mostly as passive feedback to the user. They are ideal for enhancing immersion in virtual environments or simulation scenarios by delivering comprehensive, multi-sensory (e.g., vibrotactile, thermal, pneumatic, electrotactile) feedback.
These rendering devices use a network of actuators to simulate a wide range of physical experiences, such as vibrations, impacts, and movements across the body. Haptics suits and furniture such as racing game seats, motion platforms and simulators are example of Full-Body and complex feedback devices. Multiple modalities are also provided ranging from vibrations, wind, water spray, heat or motion.
For the capture, sensors in this category capture information related to position, speed and forces. Typical GPS units and accelerometers can be used (such as those integrated into smartphones), as well as pressure sensors. The information is related to position in 3D space with global tensor fields and targeted force vectors.
6.4.5 Haptics device type 4: Interactive and spatialised feedback p. 24
Devices in this category are focused on providing targeted, spatialised interactive haptics (e.g., tactile, kinaesthetic, proprioception, thermal) feedback. These devices simulate specific actions or interactions, such as button presses or in-game effects, through vibrations or adaptive triggers. This category includes haptic controllers such as game console, phones, HMD or VR controllers offering localized feedback through vibration motors and adaptive triggers to simulate various in-game sensations and interactions. Usually, bi-directional interaction is considered.
Sensors for capturing haptics signals in this category are mostly related to pressure sensors to get a position and force related to an action. Here typically input devices such as phone, HMD or VR controllers are used for feedback but also their sensors are used to capture interactions, motion and forces.
