Tech-invite3GPPspaceIETFspace
21222324252627282931323334353637384‑5x

Content for  TR 26.996  Word version:  18.1.0

Top   Top   Up   Prev   Next
0…   4…   6…   6.1.2…   7…   A…

 

6  Audio Quality evaluationsp. 10

6.1  ISAR Baseline qualityp. 10

6.1.1  Selection testsp. 10

6.1.1.1  Test planp. 10

The selection tests evaluating the performance of the single ISAR candidate solution for IVAS was carried out according to a permanent document on Testing Aspects for Phase/Track 2/a. Annex A of this TR contains the core part of it to provide the context within which the test results provided below were obtained. The complete document is found for reference in the electronic attachment of this TR.
The purpose of the 4 selection test experiments (Experiments BS1534-1 - BS1534-4) was to evaluate the performance of the IVAS specific ISAR solution candidate with respect to the performance requirements and objectives defined in ISAR TR 26.865.
Table 6.1-1 shows a high-level overview of the experiments. Each experiment was carried out twice (in experiments a and b), once by the solution proponent and once by a cross-checker (XC).
Exp Input format Source material Listening environment Bitrates kbps Listening Lab
BS1534-1a
BS1534-1b
SBA (HOA3)Generic AudioHeadphonesIVAS: 512, CuT: 768Dolby
Qualcomm (XC)
BS1534-2a
BS1534-2b
Multi-channel 7.1+4Generic AudioHeadphonesIVAS: 512, CuT: 768Fraunhofer
Ittiam (XC)
BS1534-3a
BS1534-3b
Objects (ISM-4)Generic AudioHeadphonesIVAS: 512, CuT: 768Fraunhofer
Nokia (XC)
BS1534-4a
BS1534-4b
MASA (2 TC)Generic AudioHeadphonesIVAS: 512, CuT: 768Dolby
Bytedance (XC)
Up

6.1.1.2  Test conditionsp. 11

A description of the test conditions of all experiments is given in Table 6.1-2.
Condition Description
c01 (REF)Hidden reference: Native coding system (IVAS@512kbps rendered to post renderer pose)
c02 (LP7)LP7 anchor: Hidden reference, 7Khz LP filtered
c03 (0DOF)0-DOF native transcoding reference (IVAS@512kbps binaurally rendered to pre-renderer pose, IVAS stereo coded@256kbps)
c04 (CuT)3-DOF system under test (IVAS@512kbps split-rendered with ISAR operating at 512kbps)
Up

6.1.1.3  Requirements and Objectivesp. 11

All experiments check the same requirements defined in TR 26.865, namely that the QoE of the ISAR split rendering system (c04) is no worse than the 0-DOF native transcoding reference system (c03) using the same operation point of the native coding system (IVAS coding at 512 kbps) and best possible operation point for transcoding (IVAS stereo at 256 kbps). The 4 experiments evaluate the requirement for the 4 different main head-trackable IVAS coding formats, i.e., SBA (HOA3), MC 7.1.4, ISM-4 and MASA.
The objectives defined in TS 26.865 is that QoE provided by split rendering solution should be as close as possible to quality of native coding reference system using same operation point. There is no statistical test to verify if this objective is met. However, a statement will be made based on the observed test scores how close the quality of the tested ISAR split rendering solution for the given immersive audio input format is to the quality of the native coding reference system.
Up

6.1.1.4  Test resultsp. 11

6.1.1.4.1  BS-1534-1: SBA (HOA3)p. 11
6.1.1.4.1.1
Provided below are the result plots for the two BS1534-1 experiments.
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-1: Results of BS1534-1a test for SBA input audio
Up
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-2: Results of BS1534-1b test for SBA input audio
Up
6.1.1.4.1.2  Statistical analysisp. 13
Provided below is the statistical analysis result for the two BS1534-1 experiments.
Mean Diff. (c03 - c04) Stdev Diff. SEMD T Prob. ToR
-17.99175.79410.5289-34.01561.0000Pass
Mean Diff. (c03 - c04) Stdev Diff. SEMD T Prob. ToR
-36.658320.33361.8562-19.74921.0000Pass
Up
6.1.1.4.1.3  Experimental conclusionsp. 13
Conclusion of both experiments is that the ISAR split rendering solution for SBA input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range which indicates only very minor audible differences.
Up
6.1.1.4.2  BS-1534-2: Multi-Channel 7.1.4p. 13
6.1.1.4.2.1  Result plotsp. 13
Provided below are the result plots for the two BS1534-2 experiments.
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-3: Results of BS1534-2a test for MC 7.1.4 input audio
Up
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-4: Results of BS1534-2b test for MC 7.1.4 input audio
Up
6.1.1.4.2.2  Statistical analysisp. 14
Provided below is the statistical analysis result for the two BS1534-2 experiments.
Mean Diff. (c03 - c04) Stdev Diff. SEMD t Prob. ToR
-31.958315.13211.3814-23.13531.0000Pass
Mean Diff. (c03 - c04) Stdev Diff. SEMD t Prob. ToR
-18.16676.08810.5558-32.68791.0000Pass
Up
6.1.1.4.2.3  Experimental conclusionsp. 14
Conclusion of both experiments is that the ISAR split rendering solution for Multi-Channel 7.1.4 input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Up
6.1.1.4.3  BS-1534-3: ISM-4p. 15
6.1.1.4.3.1  Result plotsp. 15
Provided below are the result plots for the two BS1534-3 experiments.
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-5: Results of BS1534-3a test for ISM-4 input audio
Up
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-6: Results of BS1534-3b test for ISM-4 input audio
Up
6.1.1.4.3.2  Statistical analysisp. 16
Provided below is the statistical analysis result for the two BS1534-3 experiments.
Mean Diff. (c03 - c04) Stdev Diff. SEMD T Prob. ToR
-26.650014.73101.3448-19.81781.0000Pass
Mean Diff. (c03 - c04) Stdev Diff. SEMD t Prob. ToR
-32.408322.07612.0153-16.08141.0000Pass
Up
6.1.1.4.3.3  Experimental conclusionsp. 16
Conclusion of both experiments is that the ISAR split rendering solution for ISM-4 input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Up
6.1.1.4.4  BS-1534-4: MASAp. 17
6.1.1.4.4.1  Result plotsp. 17
Provided below are the result plots for the two BS1534-4 experiments.
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-7: Results of BS1534-4a test for MASA input audio
Up
Copy of original 3GPP image for 3GPP TS 26.996, Fig. 6.1-8: Results of BS1534-4b test for MASA input audio
Up
6.1.1.4.4.2  Statistical analysisp. 18
Provided below is the statistical analysis result for the two BS1534-4 experiments.
Mean Diff. (c03 - c04) Stdev Diff. SEMD t Prob. ToR
-14.01675.04930.4609-30.40911.0000Pass
Mean Diff. (c03 - c04) Stdev Diff. SEMD T Prob. ToR
-13.700022.15002.0220-6.77541.0000Pass
Up
6.1.1.4.4.3  Experimental conclusionsp. 18
Conclusion of both experiments is that the ISAR split rendering solution for MASA input meets the requirement to be no worse than the 0-DOF transcoding reference system. The experiments indicate that the achievable quality is even clearly better whereby a quality level in the 'excellent' range is achieved compared to the 0-DOF transcoding reference which is providing quality in the 'good' range. The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Up

6.1.1.5  Overall conclusionp. 18

Conclusion of all 8 experiments testing the requirement that the ISAR split rendering solution for IVAS shall be no worse than the 0-DOF transcoding reference system is that this requirement is met across all tested immersive input audio formats. It can generally be observed that the achievable quality is even clearly better whereby a quality level in the 'excellent' range close to the quality of the native IVAS coding reference system is achieved. In contrast, the 0-DOF transcoding alternative offers substantially lower quality.
The objective to provide a quality level as close as possible to the native coding reference system is met in the sense that the quality score of the split rendering system is in the high 'excellent' range, which indicates only very minor audible differences.
Up

Up   Top   ToC