Data Collection Coordination and Delivery coordinates the collection and distribution of data requested by NF consumers. It prevents data sources from having to handle multiple subscriptions for the same data and send multiple notifications containing the same information due to uncoordinated requests from data consumers. Data Collection Coordination and Delivery is supported by a DCCF via Ndccf_DataManagement service or by an NWDAF via Nnwdaf_DataManagement service. Unless otherwise stated, capabilities specified in clause 5A for a DCCF are also applicable to an NWDAF. In this Release of the specification Data Collection Coordination and Delivery is applicable to:

• NWDAFs that request data from a Data Source (e.g. for use in computing analytics).
• NF consumers that request analytics from an NWDAF Data Source.
• NF consumers that request data from an ADRF Data Source.
• ADRFs that receive data from an NF Data Source.

Data Collection Coordination is supported by a DCCF or an NWDAF. The Data Consumer may use an NRF to perform NF discovery and selection to find a DCCF that can coordinate data collection (DCCF discovery principles are defined in clause 6.3.19 of TS 23.501). Data Consumers send requests for data to the DCCF rather than directly to the NF Data Source. Whether the data consumers directly contact the NF Data Source or goes via the DCCF is based on configuration of the data consumers. For the Data Consumer and each notification endpoint in a data request, the Data Consumer may specify Formatting and Processing Instructions that determine how the data is to be provided. Upon receiving a request from a Data Consumer, the selected DCCF determines the NF instance that can be a Data Source if the Data Source is not indicated in the Data Consumer's request. The DCCF may also select an ADRF if the data is to be stored in an ADRF and an ADRF endpoint is not indicated in the Data Consumer's request. To retrieve data for a specific UE, the NRF, UDM or BSF can provide the DCCF with the identity of the Data Source using the services indicated in Table 5A.2-1.

The DCCF keeps track of the data actively being collected from the Data Sources it is coordinating. It may do so by maintaining a record of the active prior requests it sends to each Data Source. If a NWDAF subscribes for data directly with a Data Source, or a Data Source has stored data in an ADRF, the NWDAF or ADRF may register the data collection profile with the DCCF. The data collection profile may include the following parameters:

• "Service Operation" identifies the service used to collect the data or analytics from a Data Source (e.g. Namf_EventExposure_Subscribe or Nnwdaf_AnalyticsSubscription_Subscribe);
• "Analytics/Data Specification" is the "Service Operation" specific parameters that identify the collected data (i.e. Analytics ID(s) / Event ID (s), Target of Analytics Reporting or Target of Event Reporting, Analytics Filter or Event Filter, etc.);
• NWDAF ID or ADRF ID specifies the ADRF or NWDAF which registers data collection profile.

The DCCF may then determine certain historical data may be available in the NWDAF or ADRF and coordinate collection of data from the NWDAF or ADRF based on the data collection profile. When the DCCF receives a request for data, it determines the status of data collection from the Data Source. If parameters in a request for data from a Data Consumer match those in a prior request or in a data collection profile registration, the DCCF may determine that the requested data is already being collected from a Data Source or that a prior subscription to a Data Source may be modified to in addition satisfy the requirements of the new data request from a Data Consumer. This status is used in clause 5A.3 to deliver data to the Data Consumer and notification endpoints. For persisting event exposure subscriptions for long-lived data collection, the DCCF may subscribe to the UDM to receive event notifications even if a Data Source that serves a UE changes. The DCCF may subscribe to the NRF to receive event notifications if a Data Source changes (e.g. because of a NF life-cycle event). A DCCF may use the same mechanisms described in clause 6.2.2.1 to determine AMF and SMF to retrieve data related to "any UE". If the data consumer requests to collect data for any UE in an area of interest, the data consumer shall first determine all DCCFs covering the area of interest and then contact these DCCFs to request for data collection.

Formatting and/or Processing instructions may be provided in requests by Data Consumers via the Ndccf_DataManagement service and Nnwdaf_DataManagement service. As an alternative to providing individual events, formatting can be used to aggregate notifications and processing can be used to extract and send summary information from multiple notifications. Data Formatting and Processing are applicable to notifications due to events as they occur at data sources (runtime data or analytics) and historical data as described in clause 5A.5. When using the Messaging Framework, the DCCF sends the formatting and/or processing instructions to the Messaging Framework via the Nmfaf_3daData_Management Service so the MFAF may format and/or process the data before sending notifications to the Data Consumers / notification endpoints. When using Data Delivery via the DCCF, the DCCF performs formatting and/or processing before sending notifications. Formatting determines when a notification is sent to the Consumer. Formatting Instructions may indicate:

• Notification Event clubbing: Buffering and sending of several notifications in one message. The consumer may specify a minimum and/or maximum number of notifications to be clubbed.
• Notification Time Window (example: notifications are buffered and sent between 2 and 3 AM).
• Cross event reference-based notification: When a subscribing NF is subscribing to multiple events (e.g. event X and event Y) the notification for an Event-X is buffered and reported only when the Event-Y occurs.
• Consumer triggered Notification: Notifications containing data or analytics are buffered until the consumer requests delivery using Nnwdaf_DataManagement, Ndccf_DataManagement or Nmfaf_3caDataManagement Service. The consumer requests Consumer triggered notification by setting a "fetch flag" in its subscription request to the DCCF or NWDAF. When the requested data or analytics is available for retrieval, the DCCF, NWDAF or MFAF sends a notification containing fetch instructions to the consumer. The consumer must then fetch the data or analytics before an expiry time as provided in the fetch instructions.
• Exact time-based Notification: Notifications are sent to the Consumer at an exact time, irrespective of whether the event occurs (example: every 30 min). Exact time-based notifications may be periodic.
• Increasing time window based notification: Notifications are sent to the Consumer at an increasing periodicity (example: the first notification is sent immediately, subsequent received notifications are sent after 5 min, then after 10 min, then after 15 min, etc.).

For an ADRF endpoint, Formatting Instructions sent to the messaging framework may further specify whether Nmfaf services are used to deliver notifications to an ADRF, or whether the data are sent to the ADRF using a Nadrf service. Processing instructions allow summarizing of notifications to reduce the volume of data reported to the Data Consumer. The processing results in summarizing of information from multiple notifications into a common report. Processing of data for inclusion in each notification sent to consumers occurs over a Processing Interval specified in the Processing Instructions. Notifications sent to consumers may represent partial intervals if formatting instructions or Event Reporting Information (as specified in TS 23.502 Table 4.15.1-1) require that a notification be sent to the consumer before the end of a processing interval. Processing Instructions are provided per Event ID or Analytics ID and are applied to multiple notifications that result from the same subscription and for the same Event ID or Analytics ID. Processing Instructions, in addition to the Processing Interval, may specify the parameter names, parameter values and the attributes to be determined and reported to the Consumer. Processing Instructions may also specify aggregation level (e.g. per-UEs, per AoIs) or temporal aggregation (e.g. per minute, per hour) and anonymization rules to anonymize UE identification. The processed notifications may comprise the following depending on the Event and Processing Instructions:

• Event;
• Processing Interval;
• List of Event Parameter Name(s) and for each Event Parameter Name, one Event Parameter Values and sets of the following attributes as indicated in the processing instructions:
  • Event Spacing: Average and variance of the time interval separating two consecutive occurrences of the same event and parameter value, or periodicity for periodic reporting;
  • Event Duration: Average and variance of the Time for which the parameter value applies;
  • Number of countable occurrences for the parameter (e.g.: Mobility Registration Update);
  • Average, variance, most frequent value, least frequent value and skew of the parameter (e.g.: number of UEs in an AoI);
  • Maximum and minimum parameter values (e.g.: number of UEs in an AoI).

Event Parameter Names are Event specific and not all attributes are applicable for all parameter names. Examples of Event Parameter Names and Parameter values are provided in Table 5A.4-1.

ADRF or NWDAF as a Data Source:

• When the DCCF receives a request for data that includes a period in the past and ADRF is deployed, the DCCF may obtain data from ADRF as the Data Source. The DCCF may also obtain historical data from an NWDAF. The data obtained from the ADRF or NWDAF is delivered to Consumers / Notification Endpoints according to a configured Delivery Option. The DCCF may determine that requested data may be available in an ADRF or NWDAF based on ADRF identification from the consumer, the data collection profile previously registered by the ADRF or NWDAF or by querying the ADRF or NWDAF.

ADRF or NWDAF as a Data Recipient:

• An ADRF or NWDAF may be a Consumer NF that initiates requests to the DCCF for data, the ADRF or NWDAF may be specified as a notification endpoint by another Consumer NF that wants to have data it requests archived, or the DCCF may be configured to archive certain data in a ADRF (e.g. all data from an AMF instance).
• If the ADRF or NWDAF instance is not specified in a request for data by a Consumer NF, the DCCF may select the ADRF or NWDAF instance based on provisioned information or information received from the NRF.
• Data is delivered to the ADRF or NWDAF according to a configured Delivery Option (via DCCF or Messaging Framework).

The ADRF offers services that enable a consumer to store and retrieve data and analytics. The analytics are produced by the NWDAF as described in clause 6.1 and data are collected as described in clause 6.2. Data may be stored in the ADRF by:

• a consumer sending the ADRF an Nadrf_DataManagement_StorageRequest containing the data or analytics to be stored. The ADRF response provides a result indication.
• a consumer sending the ADRF an Nadrf_DataManagement_StorageSubscriptionRequest requesting that the ADRF subscribes to receive data or analytics for storage. The ADRF then subscribes to the NWDAF or DCCF for data or analytics, providing ADRF Notification Address (+Notification Correlation ID). Analytics or Data are subsequently provided as notifications using DCCF, NWDAF or MFAF services (Ndccf_DataManagement Nnwdaf_DataManagement or Nmfaf_3caDataManagement service).

Data may be retrieved from the ADRF by:

• a consumer sending an Nadrf_DataManagement_RetrievalRequest request to the ADRF to retrieve data or analytics for a Storage Transaction Identifier or a Fetch Instructions received from the ADRF in an Nadrf_DataManagement_RetrievalNotify. The ADRF determines the availability of the data or analytics in its repository and sends in the response to the consumer either the data or analytics.
• a consumer sending an Nadrf_DataManagement_RetrievalSubscribe request to the ADRF to retrieve data or analytics for a specified data or analytics collection time window. If the time window includes the future and the ADRF has subscribed to receive the data or analytics, subsequent notifications received by the ADRF are sent by the ADRF to the notification endpoint.
  The ADRF determines the availability of the data or analytics and sends a success/failure indication in the response to the consumer. The ADRF then sends one or more notifications using an Nadrf_DataManagement_RetrievalNotify to the Notification Address (+Notification Correlation ID) specified by the consumer. The notification(s) either provide the data or analytics or provide instructions to the endpoint to fetch the data or analytics using an Nadrf_DataManagement_RetrievalRequest.

Data may be deleted from the ADRF by:

• a consumer sending an Nadrf_DataManagement_Delete request.

An ADRF may be configured to register the data it is collecting with a DCCF. The registration uses the Ndccf_ContextManagement service specified in clause 8.3.2. The registration may subsequently be used by the DCCF to obtain data from the ADRF as described in clause 6.2.6.3.6. The ADRF offers services that enable a consumer to store, delete and retrieve ML models. The ML models are trained by NWDAF containing MTLF as described in clause 5.1. ML model(s) may be retrieved from the ADRF by:

• a consumer sending the ADRF an Nadrf_MLModelManagement_RetrievalRequest as described in clause 10.3.4, containing one or more tuples of unique ML Model identifier stored in ADRF or Storage Transaction Identifier. The ADRF response provides one or more tuples of unique ML Model identifiers and address (e.g. URL or FQDN) of ML model file stored in ADRF.

ML models may be stored in the ADRF by:

• a consumer sending the ADRF an Nadrf_MLModelManagement_StorageRequest as described in clause 10.3.2, containing the ML model or ML model address to be stored. The ADRF response provides a result indication.

ML model may be deleted from the ADRF by:

• a consumer sending an Nadrf_MLModelManagement_Delete request as described in clause 10.3.3. The ADRF response provides a result indication.

When a consumer requests data or analytics be stored in an ADRF, it may specify Storage Handling information requesting:

• a lifetime for how long the data or analytics should be stored; and
• that a notification alerting the consumer be sent prior to data deletion from the ADRF.

The ADRF, DCCF or NWDAF may be configured with default operator storage policies. The policies specify how long data or analytics are to be stored and conditions when the default policy supersedes Storage Handling Information provided in a request from the Data or Analytics Consumer. Based on the default operator policy and the Storage Handling information, the ADRF, DCCF or NWDAF determines the Storage Approach that will be applied for the data or analytics. The Storage Approach is comprised of the life-time for how long the data or analytics will be stored and the notification requirement to be applied when data or analytics are to be deleted from the ADRF. When more than one consumer requests a storage life-time for the same data or analytics, the Storage Approach should be based on the longest requested storage life-time. The response to the consumer request for data or analytics includes the Storage Approach. The ADRF, DCCF or NWDAF determines when the data or analytics life-time has expired. When data or analytics is to be removed from the ADRF, an alert is sent to the Consumer that the data is about to be deleted. The alert contains a Storage Transaction Identifier that can be used by the consumer to retrieve the data or analytics.

A NWDAF may have the accuracy checking capability for Analytics and/or ML Models. The NWDAF may provide the accuracy information to consumers when requested or use it for its internal processes. Input data is collected from Data Producer NF(s) when there is a request for inference/prediction per analytics ID in NWDAF for a specific time period in future. Ground truth data are collected from those Data Producer NF corresponding to the requested analytic ID at the time to which the prediction refers. The ground truth data is the actual measured data observed at the time which the prediction refers to. Analytics/ML Model Accuracy Monitoring is to be achieved by comparing the predictions using the current trained ML model and its corresponding ground truth data i.e. the corresponding true observed events. Analytics/ML Model Accuracy information is to represent general performance measurements for analytics and ML Model respectively, which are composed of the number of correct predictions out of all predictions and the corresponding number of samples. The NWDAF (containing AnLF/MTLF) with accuracy checking capability decides to initiate analytics accuracy monitoring based on:

• A request from an analytics accuracy consumer. The analytics accuracy consumer may be an NWDAF containing AnLF, NWDAF containing MTLF or an analytics consumer NF.
• Analytics Feedback Information which may be provided by an Analytics Consumer NF.

The AnLF with analytics accuracy checking capability as defined in clause 6.2D is able to provide or notify the accuracy information of Analytics IDs to the analytics consumers of such service and when the analytics accuracy does not meet the analytics consumer's requirements, the analytics consumer may stop using analytics for a period of time or obtain new analytics. In addition, updated analytics for the provided Analytics IDs may be provided to analytics consumers as requested, if the updated analytics is able to be generated within the correction time period. The AnLF with analytics accuracy checking capability is as defined in clause 6.2D.1 is able to determine analytics accuracy information based on e.g.:

• Comparing predictions and its corresponding ground truth data, which are collected corresponding to the requested analytic ID at the time which the prediction refers to.
• Comparing changes in internal configuration for the analytics ID generation (e.g. change of data collection parameters, change in data distribution from a Data Source).
• Previous existent records of analytics accuracy information.
• Accuracy feedback information provided by an NF consumer.
• Determining analytics accuracy by comparing analytics accuracy using multiple ML models.

The MTLF with ML Model accuracy checking capability as defined in clause 6.2E is able to determine ML Model degradation based on e.g.:

• comparing/evaluating the data: including input data, analytics output and the ground truth data either collected from various data source NFs, DCCF, AnLF, ADRF or configured by OAM;
• or AnLF providing notifications of the analytics accuracy information; or
• AnLF providing analytics feedback information of the analytics generated by the ML model.

The NWDAF containing MTLF may reselect a new ML model or retrain the existing ML model and consequently notify the ML model accuracy degradation to the ML Model consumer(s). In addition, the NWDAF containing MTLF may consider the rating of untrusted AF(s) when used as data sources.