The selection tests evaluating the performance of the single ISAR candidate solution for IVAS was carried out according to a permanent document on Testing Aspects for Phase/Track 2/a.
Annex A of this TR contains the core part of it to provide the context within which the test results provided below were obtained. The complete document is found for reference in the electronic attachment of this TR.
The purpose of the 4 selection test experiments (Experiments BS1534-1 - BS1534-4) was to evaluate the performance of the IVAS specific ISAR solution candidate with respect to the performance requirements and objectives defined in ISAR
TR 26.865.
Table 6.1-1 shows a high-level overview of the experiments. Each experiment was carried out twice (in experiments a and b), once by the solution proponent and once by a cross-checker (XC).
A description of the test conditions of all experiments is given in
Table 6.1-2.
All experiments check the same requirements defined in
TR 26.865, namely that the QoE of the ISAR split rendering system (c04) is no worse than the 0-DOF native transcoding reference system (c03) using the same operation point of the native coding system (IVAS coding at 512 kbps) and best possible operation point for transcoding (IVAS stereo at 256 kbps). The 4 experiments evaluate the requirement for the 4 different main head-trackable IVAS coding formats, i.e., SBA (HOA3), MC 7.1.4, ISM-4 and MASA.
The objectives defined in
TS 26.865 is that QoE provided by split rendering solution should be as close as possible to quality of native coding reference system using same operation point. There is no statistical test to verify if this objective is met. However, a statement will be made based on the observed test scores how close the quality of the tested ISAR split rendering solution for the given immersive audio input format is to the quality of the native coding reference system.
Provided below are the result plots for the two BS1534-1 experiments.
Provided below is the statistical analysis result for the two BS1534-1 experiments.
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
T |
Prob. |
ToR |
| -17.9917 | 5.7941 | 0.5289 | -34.0156 | 1.0000 | Pass
|
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
T |
Prob. |
ToR |
| -36.6583 | 20.3336 | 1.8562 | -19.7492 | 1.0000 | Pass
|
Conclusion of both experiments is that the ISAR split rendering solution for SBA input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range which indicates only very minor audible differences.
Provided below are the result plots for the two BS1534-2 experiments.
Provided below is the statistical analysis result for the two BS1534-2 experiments.
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
t |
Prob. |
ToR |
| -31.9583 | 15.1321 | 1.3814 | -23.1353 | 1.0000 | Pass
|
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
t |
Prob. |
ToR |
| -18.1667 | 6.0881 | 0.5558 | -32.6879 | 1.0000 | Pass
|
Conclusion of both experiments is that the ISAR split rendering solution for Multi-Channel 7.1.4 input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Provided below are the result plots for the two BS1534-3 experiments.
Provided below is the statistical analysis result for the two BS1534-3 experiments.
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
T |
Prob. |
ToR |
| -26.6500 | 14.7310 | 1.3448 | -19.8178 | 1.0000 | Pass
|
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
t |
Prob. |
ToR |
| -32.4083 | 22.0761 | 2.0153 | -16.0814 | 1.0000 | Pass
|
Conclusion of both experiments is that the ISAR split rendering solution for ISM-4 input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Provided below are the result plots for the two BS1534-4 experiments.
Provided below is the statistical analysis result for the two BS1534-4 experiments.
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
t |
Prob. |
ToR |
| -14.0167 | 5.0493 | 0.4609 | -30.4091 | 1.0000 | Pass
|
| Mean Diff. (c03 - c04) |
Stdev Diff. |
SEMD |
T |
Prob. |
ToR |
| -13.7000 | 22.1500 | 2.0220 | -6.7754 | 1.0000 | Pass
|
Conclusion of both experiments is that the ISAR split rendering solution for MASA input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Conclusion of all 8 experiments testing the requirement that the ISAR split rendering solution for IVAS shall be no worse than the 0-DOF transcoding reference system is that this requirement is met across all tested immersive input audio formats. It can generally be observed that the achievable quality is even clearly better whereby a quality level in the
'excellent' range close to the quality of the native IVAS coding reference system is achieved. In contrast, the 0-DOF transcoding alternative offers substantially lower quality.
The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high
'excellent' range, which indicates only very minor audible differences.