Content for TS 23.288 Word version: 18.2.0

1… 4… 5… 5A… 6… 6.1.3 6.1.4… 6.1.4.4… 6.1.5… 6.1A… 6.1B… 6.1B.2.3… 6.1C 6.2… 6.2.3… 6.2.6… 6.2.6.2 6.2.6.3… 6.2.6.3.3 6.2.6.3.4 6.2.6.3.5 6.2.6.3.6… 6.2.7… 6.2.8… 6.2.9… 6.2.13… 6.2A… 6.2B… 6.2B.3 6.2B.4… 6.2C… 6.2D… 6.2E… 6.2F… 6.3… 6.4… 6.5… 6.6… 6.7… 6.7.3… 6.7.4… 6.7.5… 6.8… 6.9… 6.10… 6.11… 6.12… 6.13… 6.14… 6.16… 6.17… 6.18… 6.19… 6.20… 6.21… 7… 7.4… 7.7… 7.9… 8… 9… 10…

6.2D AnLF Analytics Accuracy Monitoring Procedures 6.2D.1 General 6.2D.2 Procedures for Analytics Accuracy Information Subscription 6.2D.3 Procedures for Analytics Accuracy Information Request
...

6.2D AnLF Analytics Accuracy Monitoring Procedures |R18| p. 138

6.2D.1 General p. 138

The Analytics Accuracy Information comprises a set of parameters as defined in clause 6.1.3.

When multiple NWDAFs are deployed, some NWDAFs may be specialized with the analytics accuracy checking capability. When an NWDAF containing AnLF has the accuracy checking capability, such NWDAF is able to:

Receive a subscription or a request for analytics IDs via Nnwdaf_AnalyticsSubscription_Subscribe or Nnwdaf_AnalyticsInfo_Request service operation with the indication for activating the mechanisms for checking the accuracy of such analytics ID as defined in clause 6.1.3.
Provide the accuracy information to the consumer via Nnwdaf_AnalyticsSubscription_Notify or Nnwdaf_AnalyticsInfo_Request response service operation.

Based on the triggers described in Clause 5C.1, NWDAF containing AnLF starts the monitoring check and generation of analytics accuracy information for an Analytics ID.

The Analytics accuracy information may be requested per Analytics ID and scoped using the same parameters defined in the Target of Analtyics Reporting (i.e. UEs, any UE, group of UEs) and Analytics Filter Information (e.g. for a specific area, specific slice) of the requested Analytics ID.

When the accuracy checking is activated in an NWDAF containing the ANLF, the NWDAF may store for a period of time the necessary information to determine the analytics IDs accuracy and provide the accuracy information to consumers when requested or use it for its internal processes.

NWDAF containing AnLF generates the accuracy information based on any of the following information:

Comparing predictions and its corresponding ground truth data, which are collected corresponding to analytic ID requested at the time which the prediction refers to.

NOTE 5:
The ground truth data and the corresponding prediction is to be defined per Analytics ID.
Comparing changes in internal configuration for the analytics ID generation (e.g. data collection parameters, change in data distribution from a Data Source).
Previous existent records of analytics accuracy information.
Accuracy feedback information provided by NF consumer.

6.2D.2 Procedures for Analytics Accuracy Information Subscription p. 139

This procedure is used by NF consumers of analytics ID to subscribe to receive analytics output and analytics accuracy information related to the requested analytics ID for NF consumer. Figure 6.2D.2-1 shows the procedure for accuracy information subscription and provisioning.

Figure 6.2D.2-1: Analytics Accuracy Information Subscribe

Step 1.

The NWDAF service consumer selects the appropriated NWDAF with AnLF according to clause 5.2 and subscribes, modifies, or cancels subscription for analytics accuracy information by invoking the Nnwdaf_AnalyticsSubscription_Subscribe service operation. The parameters that can be included in the subscription to trigger the accuracy information checking and provisioning are listed in clause 6.1.3.

If the subscription is not the initial subscription request, it may include Analytics feedback information as described in clause 6.1.1.

Step 2.

When a subscription request is received, the NWDAF containing AnLF verifies the parameters of the Analytics Accuracy Request information.

The NWDAF containing AnLF starts the analytics accuracy monitoring and generation of the analytics accuracy information related to the analytics ID indicated in the subscription and according to the parameters defined in Analytics Accuracy Request Information in clause 6.1.3. The NWDAF containing AnLF is to compute analytics accuracy information according to the definitions in clause 6.2D.1. If the NWDAF containing AnLF does not have enough necessary data, it will perform step 3b to collect ground truth before computing analytics accuracy information.

The NWDAF containing AnLF may have started to perform the analytics accuracy monitoring and analytics accuracy information generation, triggered by other NWDAF service consumer(s) before. Upon receiving a new request from the NWDAF consumer, NWDAF containing AnLF determines whether new data collection is needed for analytics accuracy information generation based on the corresponding analytics subscription.

In addition to the received request from the NWDAF service consumer, based on local policy, NWDAF containing AnLF may determine to start the analytics accuracy monitoring and analytics accuracy information generation.

Step 3a.

The NWDAF containing AnLF performs the data collection for the subscribed analytics ID and generates the analytics output.

Step 3b.

The NWDAF containing AnLF performs the data collection (e.g., ground truth data collection) for accuracy information generation for the subscribed analytics ID and generates the associated analytics accuracy information. If Analytics feedback information is included in step 1, the NWDAF containing AnLF may take it into account and determine whether it affects the ground truth data by internal logic to generate analytics accuracy information.

Step 4a.

The NWDAF containing AnLF provides the analytics output according to the parameters defined in Analytics Reporting Information included in the subscription request when there is no Analytics Accuracy Request Information included in the subscription in step 1..

Step 4b.

The NWDAF containing AnLF provides the analytics accuracy information together with the analytics output for the analytics ID according to the parameters defined in the Analytics Accuracy Request Information included in the subscription request.

Step 4c.

The NWDAF containing AnLF provides only the analytics accuracy information for the analytics ID according to the parameters defined in the Analytics Accuracy Request Information included in the subscription request. The analytics accuracy information is provided in a separated notification when the periodicity for providing the analytics accuracy information indicated in the Analytics Accuracy Request Information is different from the periodicity for providing the analytics output indicated in the subscription request.

Step 5.

When determining the low or insufficient accuracy for an analytics ID, i.e. the deviation of the output analytics using the trained ML model from the ground truth data (which are collected from Data Producer NF corresponding to analytic ID requested at the time which the prediction refers to) is greater than the Reporting Threshold(s) (which are locally configured or received in the Subscribe request, the NWDAF containing AnLF may notify the NWDAF Service consumer with the Stop Analytics Output Consumption indication and the Stop Analytics Output Consumption time window.

Step 6.

(Optional) NWDAF Service Consumer may decide to stop the consumption of analytics output without unsubscribing to the analytics ID, based on its own logic or based on a received notification from NWDAF with the Stop Analytics Output Consumption indication. The NWDAF Service Consumer invokes the Nnwdaf_AnalyticsSubscription_Subscribe service operation including the Subscription Correlation ID to modify an existing subscription and provide the parameter PauseAnalytics Subscription flag in the Analytics Accuracy Request Information.

Step 7.

When NWDAF determines an improvement in the accuracy of an analytics ID (e.g. meet the accuracy requirement of the analytics consumer) or when the Pause Analytics Subscription time window set by itself is finished, NWDAF notifies the NWDAF Service Consumer of the accuracy information for the analytics ID to resume the consumption of the analytics output, therefore reactivating an existing analytics ID subscription that have been previously stopped.

Step 8.

(Optional) NWDAF Service Consumer based on its own logic can notify the NWDAF to resume the provisioning of analytics output, therefore reactivating an existing subscription to analytics ID that have been paused either by NWDAF Service Consumer request (step 6) or by NWDAF indication (step 5). The NWDAF Service Consumer invokes the Nnwdaf_AnalyticsSubscription_Subscribe service operation including the Subscription Correlation ID to modify an existing subscription and provide the parameter Resume Analytics Subscription request in the Analytics Accuracy Request Information.

6.2D.3 Procedures for Analytics Accuracy Information Request p. 142

This procedure is used by NF consumers of analytics ID to request analytics accuracy information related to the requested analytics ID for NF consumer. Figure 6.2D.3-1 shows the procedure for accuracy information request and response.

Reproduction of 3GPP TS 23.288, Fig. 6.2D.3-1: Analytics Accuracy Information Request

Figure 6.2D.3-1: Analytics Accuracy Information Request

Step 1.

The NWDAF service consumer selects the appropriated NWDAF with AnLF according to Clause 5.2 and requests for analytics accuracy information by invoking the Nnwdaf_AnalyticsInfo_Request service operation. The parameters that can be included in the request to trigger the accuracy information checking and provisioning are listed in clause 6.1.3.

Step 2.

When a request is received, the NWDAF containing AnLF determines whether the request is only for analytics output generation or if it includes the Analytics Accuracy request.

If Analytics Accuracy request is included, the NWDAF containing AnLF starts the analytics accuracy monitoring and generation of the analytics accuracy information related to the analytics ID indicated in the request and according to the parameters defined in Analytics Accuracy Request Information in clause 6.1.3. The NWDAF containing AnLF is to compute analytics accuracy information according to the definitions in Clause 6.2D.1. If the NWDAF containing AnLF does not have enough necessary data, it will perform step 3b to collect ground truth before computing analytics accuracy information.

Step 3a.

The NWDAF containing AnLF performs the data collection for the requested analytics ID and generates the analytics output.

Step 3b.

The NWDAF containing AnLF performs the data collection (e.g., ground truth data collection) for accuracy information generation for the requested analytics ID and generates the associated analytics accuracy information.

Step 4a.

The NWDAF containing AnLF provides the analytics output according to the parameters defined in Analytics Reporting Information included in the request, when no Analytics Accuracy Request Information is included in the request in step 1.

Step 4b.

The NWDAF containing AnLF provides the requested analytics output and analytics accuracy information for the analytics ID according to the parameters defined in the Analytics Accuracy Request Information included in the request.