Tech-invite3GPPspaceIETFspace
21222324252627282931323334353637384‑5x

Content for  TR 22.826  Word version:  17.2.0

Top   Top   Up   Prev   Next
1…   5…   5.2.3…   5.3…   5.3.3   5.3.4…   5.4…   6…

 

5.2.3  Augmented Reality Assisted Surgeryp. 19

5.2.3.1  Descriptionp. 19

Image guided surgery is gradually becoming mainstream. Currently, surgeons can plan a procedure based on the 3D display of patient anatomy reconstructed from MR (Magnetic Resonance) or CT (Computer Tomography) scans. And, in the near future, real-time medical imaging (3D ultrasound typically) can also be used as a live reference. Display devices such as e.g. head mounted display, monitors will be used to show real-time images merging the main video stream (endoscopy, overhead, microscopy…) with the live reference medical imaging.
Metadata associated with the video can be updated at the frame rate (e.g., 3D position of probes). Only the method for conveying the multiple synchronized video/multi-frame sources along with their parameters (that may change at every frame) is specified in the present technical report. Mechanisms used for generating augmented reality views or to detect and to follow 3D position of devices are not addressed in this use case.
One of the most challenging use cases for augmented reality assisted surgery is related to minimally invasive heart procedures where surgeons don't cut the breast bone but operate between the ribs. In fact, the heart is a moving organ with at rest, a cardiac cycle that lasts about 800ms (75 beats per minute) and is divided into two phases: diastole (about 60% of the cardiac cycle) and systole (40% of the cardiac cycle). Heart contraction itself involves a deformation of up to 25% of its total size (between 14cm and 16cm) and lasts around 200ms. This implies that the walls of the heart move at a speed of up to 20cm/s.
In this use case, we examine a procedure called Coronary Artery Bypass Graft (CABG) which is a surgical procedure in which one or more blocked coronary arteries are bypassed by a blood vessel graft, usually taken from patient's arms or legs, to restore normal blood flow to the heart.
Copy of original 3GPP image for 3GPP TS 22.826, Fig. 5.2.3.1-1: Illustration of a CABG Procedure
Figure 5.2.3.1-1: Illustration of a CABG Procedure
(⇒ copy of original 3GPP image)
Up
Sometimes, the left anterior descending (LAD) coronary is obstructed and requires coronary bypass surgery. However, it takes a deep intra-myocardial course and may be difficult to locate in the case of a thick epicardial adipose tissue or epicardial scar tissue.
There are two basic ways of performing CABG: on-pump CABG and off pump CABG. In the on-pump case, the heart is temporarily stopped and blood flow is diverted using a heart-lung bypass machine. In the off-pump case, the area around the blocked coronary artery is stabilized using pods while the surgeon grafts the blood vessel on the pumping heart. Off-pump CABG is relatively a newer procedure to on-pump CABG and was shown to reduce postoperative complications associated with the use of cardiopulmonary bypass such as generalized systemic inflammatory response, cerebral dysfunction, myocardial depression, and hemodynamic instability.
Stabilizing pods used during the procedure help to reduce the amplitude (and thereby the speed) of heart's movements at the place of the anastomosis. Thus, a safe assessment leads here to consider a relative speed (surgeon's hand and heart walls) of 3.5cm per second. Thus, limiting the error for the relative position of instruments and moving tissues to 0.5mm on the display monitors sets a maximum acceptable imaging system latency (including image generation, transmission, processing and display) of 14 ms. Considering 120fps frame rate and principles set forth in clause 5.2.1.3, the resulting cumulated end to end latency requirement falls around 1.5 ms. To guarantee correct recombination of the two data streams in a single and accurate A/R image by the A/R application, medical images are synchronised together thanks to local clocks stabilized on the grand master clock with a very tight accuracy.
Due to computation challenges at the A/R application, it is expected that the scope will produce 4K uncompressed video, with the perspective to support also HDR (High Dynamic Range) for larger colour gamut management (up to 10 bits per channel) as well as HFR (High Frame Rate), i.e.; at least 120 fps to be able to track instruments and heart relative movements with enough precision.
In addition, the data of a 3D ultra-sound probe is used to augment the main anatomical image with stiffness of soft tissues or Doppler information to the image. Various techniques exist to create a 3D ultra-sound image, ranging from combining 2D ultrasound slices to using dedicated 3D probes supporting 3D volume data in various coordinate systems and encodings. A 2D ultrasound typically produces a data stream of uncompressed images of 512x512 pixels with 32 bits per pixel at 20 fps (up to 60 fps in the fastest cases), resulting in a data rate of 160 Mbit/s up to 500 Mbit/s. Using 2D ultrasound to create a 3D ultrasound volume is typically slow and cumbersome. Dedicated 3D probes tend to work at higher data rates, i.e. above 1 Gbit/s of raw data, and are expected to reach multi gigabit data rates in future (e.g. producing 3D Cartesian volumes of 256 x 256 x 256 voxels each encoded with 24 bits at 10 volumes per second or better). Some processing such as lossless compression may be performed to reduce the data rates. Accurate recombination by the A/R application of images issued from both the laparoscope and the ultra-sound probe requires very precise location of the instruments inside the patient's body through a mechanism that is out of 5G system scope. However, location data shall be timestamped with same accuracy and made available at the A/R application at least at the same rate as the images.
Typical CABG duration is two to four hours, this allows us to estimate targeted communication service availability figure for the successful transmission of images within latency constraints discussed above. In fact, considering that consecutive frame loss or delay may translate into a wrong estimated distance and may result in serious injury to the patient, we want this event to only happen with a very low probability during at least the duration of the procedure, a safe margin would be to consider ten hours of correctness in a row. Note that in this use case, a total of 240 images per second are exchanged over 5G communication service (120 images per second in each direction).
Up

5.2.3.2  Pre-conditionsp. 20

A patient is suffering from coronary artery disease and underwent a heart attack few hours ago. After being injected a clot-dissolving agent to restore blood flow in the blocked coronary artery he has been admitted to the hospital emergency room for a closed thorax coronary bypass surgery.
The patient under anaesthesia is on the operating table and the surgery team is ready to start the procedure. Each needed equipment (laparoscope, ultra-sound probe, monitors …) is:
  • Powered up,
  • Subscribed to 5G-LAN type services deployed by the hospital IT infrastructure manager,
  • Configured on which private groups they shall use to communicate with each other,
  • Attached to the non-public 5G network covering the operating room.
In addition, the monitors are subscribed to a URLLC point-to-multipoint communication service dedicated to transport high data rate downlink video streams.
The augmented reality application that handles the video streams generated by the laparoscope and the ultrasound probe is up and running and instantiated on private IT resources inside the hospital at a short network distance from the operating room.
Up

5.2.3.3  Service Flowsp. 20

 
  1. The surgeon takes a blood vessel graft, usually from patient's arms or legs
  2. Small incisions are performed in the patient's chest in order to insert a laparoscope equipped with a small video camera and a cold light source, and the ultra-sound mini-probe. Other small diameter instruments (grasper and scissors) and the cardiac stabilizer pods used for off-pump surgery are then introduced.
  3. The application processes 4K images received from the laparoscope and the 3D representations issued by the ultrasound mini-probe to help the surgeon to determine the location of the left anterior descending (LAD) coronary and to visualize plaque and calcifications which may hamper coronary anastomosis suturing. Such processing requires both streams to be finely synchronized. Despite the low latency and high datarates, all medical imaging data is fully integrity protected as required by medical data protection regulations.
  4. Since the heart is still beating during the procedure, all video streams have to be transferred to the application and from the application to the monitors with ultra-low latency to avoid perforating healthy heart tissue.
  5. The surgeon performs the graft procedure by sewing one end of a section of the harvested blood vessel over a tiny opening made in the aorta and the other end over a tiny opening made in the blocked coronary vessel.
Up

5.2.3.4  Post-conditionsp. 21

The procedure fully succeeded and the patient recovered in the hospital in a couple of days, including some time in the intensive care unit. The patient resumed his normal activities about three to four weeks after the bypass surgery and statistically show less post-operation complications than in classical procedure.

5.2.3.5  Existing features partly or fully covering the use case functionalityp. 21

Reference number Requirement text Application / Transport Comment
8.9The 5G system shall support data integrity protection and confidentiality methods that serve URLLC and energy constrained devices.T Requirement taken from TS 22.261, however need to add "high data rates" to the requirement text.
Some of the medical equipment being used in this use case (such as wireless laparoscope and ultrasound mini-probe) may require solutions that serve URLLC communication, very high data rates as well as being energy constrained.
6.24Set of requirements related to the management of 5G LAN-type services and to the transport of Ethernet frames between UEs belonging to the same 5G-LAN-type service.TSee TS 22.261
Up

5.2.3.6  Potential New Requirements needed to support the use casep. 22

Use case Characteristic parameter Influence quantity
>5.2.3 - Augmented Reality Assisted Surgery Communi­cation service availa­bility: target value in % Communi­cation service reliabi­lity: Mean Time Between Failure End-to-end latency: maximum Bit rate Direction Message Size [byte] Survival time UE speed # of active UEs Service Area [m2]
Uncompressed 4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream>99.99999>1 year<750μs30 Gbits/sUE to Network~1500 - ~9000 (note 3)~8 msstationary1100
4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream with lossless compression (note 1)>99.99999>1 year<750μs12 Gbits/s (note 2)UE to Network~1500 - ~9000 (note 3)~8 msstationary1100
3D 256 x 256 x 256 voxels 24 bits 10 fps ultrasound unicast data stream >99.9999>1 year<10ms4 Gbits/sUE to Network~1500~100 msstationary1100
Uncompressed 4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream>99.99999>1 year<750μs30 Gbits/sNetwork to UEs~1500 - ~9000 (note 3)~8 msstationary<10100
4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream with lossless compression (note 1)>99.99999>1 year<750μs12 Gbits/s (note 2)Network to UEs~1500 - ~9000 (note 3)~8 msstationary<10100
NOTE 1:
This line provides alternative KPIs that are still acceptable
NOTE 2:
An average compression ratio of 2.5 has been considered when applying a lossless compression algorithm
NOTE 3:
MTU size of 1500 bytes is not generally suitable to gigabits connections as it induces many interruptions and loads on CPUs. On the other hand, Ethernet jumbo frames of up to 9000 bytes require all equipment on the forwarding path to support that size in order to avoid fragmentation.
Up

5.2.4  Robotic Aided Surgeryp. 23

5.2.4.1  Descriptionp. 23

5.2.4.1.1  Generalp. 23
Robotic aided surgery is particularly suitable to invasive surgical procedures that require delicate tissue manipulation and access to areas with difficult exposure. It is achieved through complex systems that translate the surgeon's hand movements into smaller, precise movements of tiny instruments that can generally bend and rotate far more than a human hand is capable of doing inside the patient's body. In addition, those systems are usually able to filter out hand tremor and therefore allow more consistent outcomes for existing procedures, and more importantly the development of new procedures currently made impractical by the accuracy limits of unaided manipulation.
For example in abdominal surgery, surgeon's movements are guided by a specific instrument called a laparoscope which is a thin tube with a tiny camera and light at the end that provides in some cases a 3D view of the patient's body internals by sending potentially stereoscopic images to a suitable 3D rendering system at the control console and to 2D video monitors in the operating room.
In order to further improve precision, shorten learning curve and prevent extended operating times, surgeons need to get their human sense of touch back. For that purpose, the control console is fed with haptic feedback generated by the instruments to render the exact applied forces and tissue deflections resulting from the surgical procedure. In this context, the system creates a virtual operative space that mimics the feel of open surgery with artificial haptic forces related to the zones to avoid (veins, fragile tissues) or the target to reach where the zones to avoid have been defined during pre-operative 3D reconstruction of the patient body and fed to the robot.
Copy of original 3GPP image for 3GPP TS 22.826, Fig. 5.2.4.1.1-1: Example of a surgery aiding robotic system
Up
Typical surgery robotic systems can have around 40 actuators and the same number of sensors which allow to compute the data rate requires in each direction in order to execute a given movement.
Human sensitivity of touch is very high, tactile sensing has about 400 Hz bandwidth, where bandwidth refers to the frequency of which the stimuli can be sensed. This is why in general haptic feedback systems operate at frequencies around 1000 Hz. This rate naturally applies to the update of all information used in the generation of the haptic feedback, e.g. instruments velocity, position … Therefore, the robot control process involves:
  • The surgeon console periodically sending a set of points to actuators in 50 bytes messages
  • Actuators executing a given process
  • Sensors sampling velocity, forces, positions, … at the very same time and returning that information to the surgeon console, in 50 bytes messages at the rate of 1 kHz
As opposed to machine to machine communication, robotic aided surgery implies there is a human being in the middle of the control loop, which means that the console does not autonomously generate new commands based on the system state collected in the previous 1 kHz cycle but based on surgeon's hand movement.
In order to improved surgeon's spatial perception, it is expected that the endoscope can produce 8K stereoscopic uncompressed video, supports also HDR (High Dynamic Range) for larger colour gamut management (up to 10 bits per channel) as well as HFR (High Frame Rate), i.e.; up to 120 fps.
Of course, all messages exchanged have to be properly secured (especially in terms of data integrity and authenticity) and the probability of two consecutive packet errors shall be negligible.
The endoscope, the displays, the monitoring equipment and the robot's sensors shall be synchronised on the same clock synchronisation service with a clock synchronicity in the order of 1μs to enable offline replay of the whole procedure. In particular, all sensors shall sample the system state at the same exact time and send it back to the control console in order to allow for a consistent haptic feedback generation.
Up
5.2.4.1.2  Use casep. 24
One use case where robotic aided surgery proved to be highly beneficial is Robotic Aided Kidney Transplant (RAKT) for obese recipients (see [15] and [16]). In general, transplant surgery in obese recipients is technically very demanding and require larger right lower quadrant incision, often associated to more wound related morbidity in terms of surgical site infections (SSI), more pain, longer convalescence and postoperative recovery, which explains why obese patients are frequently denied access to transplantation. But epidemiological data indicate that 20-50% of patients on dialysis for end-stage renal disease (ESRD) are obese, advocating for minimally invasive surgery as opposed to open surgery. Unfortunately, conventional minimally invasive surgery using laparoscopic instruments manipulated by a surgeon is not suitable for the safe execution of a kidney transplant in morbidly obese patients due to the high complexity of the procedure. In fact, current laparoscopic cameras present only a two-dimensional view and laparoscopic instruments have a limited degree of freedom which results in loss of depth perception, lower natural hand-eye coordination and dexterity. On the other hand, robotic surgery provides a three-dimensional view and utilizes articulated instruments, which allows the surgeon to work with greater ease, with more intuitive movements during the execution of complex procedures.
In a series of RAKT performed in obese patients between June 2009 and December 2011 (see [16]), 0% developed surgical site wound infection versus 28,6% patients in a control group that underwent an open kidney transplant procedure.
There are concerns that are usually raised related to robotic surgery:
  • Learning curve which leads to few experienced surgeons
  • Procedure duration impacting return on investment for hospitals and potentially affecting patient status after surgery
  • Medical costs that are significantly higher for the robotic surgical technique compared to the open technique
In our use-case, the higher costs have to be balanced against the cost of keeping obese renal failure patients on dialysis, which is quite expansive as well.
As to the point related to procedure duration, the question is indeed whether surgeons are able to complete the procedure with same or equivalent duration as an open surgery, so as to keep the warm ischemia time (time a tissue remains at body or ambient temperature after blood supply has been interrupted), which is one of the main reason for graft failure, as low as possible. Therefore, with the idea that the whole transplant procedure shall be completed in the shortest possible time, requirements on latencies introduced by the teleoperation system, that are discussed in clause 5.2.1.3.2, need to be considered here so that surgeons can make more natural movements, do not slow down their hand speed and do not make pauses every now and then.
Typical RAKT duration is close to four hours, this allows us to estimate targeted communication service availability figure for the successful transmission of images within latency constraints discussed above. In fact, considering that any late received image translates immediately into a wrong estimated distance and may result in serious injury to the patient, we want this event to not happen during at least the duration of the procedure, a safe margin would be to consider five hours of correctness in a row. Note that in this use case, a total of 240 images per second are exchanged over 5G communication service (120 images per second in each direction).
In addition, having two consecutive errors in any direction shall be negligible as it may result in incorrect commands sent the actuators, and, in addition to represent a serious risk of injury for the patient, may damage the system. Considering that the probability of having two consecutive errors shall be p2 < (1000 messages x 2 directions x 2 radio segments x 3600 seconds x 5 hours)-1 = 1.39 x 10-8, this gives a suitable p = 0.0001 for the message error rate.
Up

5.2.4.2  Pre-conditionsp. 25

Jack is an obese patient with a Body Mass Index (BMI) above 40kg/m2 and suffers from End Stage Renal Disease (ESRD) and from diabetes and hypertension. As Higher BMIs in kidney transplant recipients are associated with excess risk of surgical site infections (SSIs), and negatively impact graft survival, Jack has been denied access to kidney transplant operation and need to undergo frequent and regular dialysis for now 5 years. The 5-year mortality rate for diabetic and hypertensive dialysis patient is unfortunately very high (around 70%) which makes Jack situation critical.
The main hospital close to Jack's residence has develop a new, minimally invasive, robotic kidney transplantation method using a short midline epigastric incision that avoids any incision in the infection prone lower quadrants of the abdomen. A left kidney is procured from a 47-year-old Caucasian male, who died from a cerebrovascular accident. Subsequently, the hospital contacted Jack to propose him undergoing a RAKT.
Now Jack, is under anaesthesia on the operating table and the surgery team is ready to start the procedure. Each needed equipment (robot, control console, monitors …) is:
  • Powered up,
  • Subscribed to 5G-LAN type services deployed by the hospital IT infrastructure manager,
  • Configured on which private groups they shall use to communicate with each other,
  • Attached to the non-public 5G network covering the operating room.
In addition, the monitors are subscribed to a URLLC point-to-multipoint communication service dedicated to transport high data rate downlink video streams.
The application that handles the video streams generated by the laparoscope, runs the 3D patient body's model, sends control information to the robot, generates haptic feedback to the surgeon is up and running and instantiated on private IT resources inside the hospital at a short network distance from the operating room.
Up

5.2.4.3  Service Flowsp. 25

  1. The surgeon positions at the console and the co-surgeon positions at the bedside and a 7cm periumbilical incision is made in order to insert a hand access device that will maintain peritoneal gas pressure while allowing for introduction of the graft.
  2. Subsequently, four additional trocars are inserted for the right and left hand side robotic arms, the scope and the assistant.
  3. After exposure of external iliac artery and vein, the right external iliac vein is clamped using plastic bulldogs, and the venotomy is made with the robotic Potts scissors at the site where the renal vein of the new kidney is to be sewn.
  4. The graft is brought into the operative field through the midline incision and placed in the right lower quadrant. The renal vein attached to the new kidney is anastomosed in a continuous manner, end-to-side to the right external iliac vein.
  5. Next, the right external iliac artery is clamped with plastic bulldogs. A circular arteriotomy is made and the renal artery is anastomosed end-to-side to the right external iliac artery.
  6. After the anastomoses are tested and show no leak, the new kidney is re-vascularized (the plastic bulldogs are removed).
  7. Last, the bladder is distended with saline and methylene blue and then dissected through the muscle layer in order to anastomose the ureter from the new kidney with the bladder.
During the operation, the surgeon at his console sends commands through his hands and feet movements to the robot over the 5G non-public network covering the operating room and receives both 3-dimensional images and haptic feedback from the robot. The critical medical application instantiated at network edge provides tactile guidance by constraining where the instruments (scalpel, etc.) can go. The surgeon is free to move inside the incisions, but each time he's going to the wrong place it feels like he's hitting a piece of glass or a constraint. The haptic feedback also allows to apply more consistent tension to suture material during robotic knot tying.
Up

5.2.4.4  Post-conditionsp. 26

Thanks to the original robotic technique, the procedure fully succeeded and yielded low complications, rapid recovery, low pain and excellent graft function. The patient recovered in the hospital in 5 days, showing no graft rejection signs. He resumed a normal life without dialysis about three to four weeks after the transplant surgery and is submitted to regular follow-up exactly like patients with lower BMI.

5.2.4.5  Existing features partly or fully covering the use case functionalityp. 26

Reference number Requirement text Application / Transport Comment
8.9The 5G system shall support data integrity protection and confidentiality methods that serve URLLC and energy constrained devices.TRequirement taken from TS 22.261, however need to add "high data rates" to the requirement text.
The medical equipment being used in this use case may require solutions that serve URLLC communication as well as very high data rates. The equipment is typically not energy constrained.
6.26Set of requirements related to the management of 5G LAN-VN, including those related to the privacy of communications between UEs belonging to the same 5G LAN-VNTSee TS 22.261
6.1.2.2The 5G system shall enable the network operator to define a priority order between different network slices in case multiple network slices compete for resources on the same network.TSee TS 22.261
Packets transporting haptic feedback and robot control shall be assigned the highest priority
Up

5.2.4.6  Potential New Requirements needed to support the use casep. 27

Use case Characteristic parameter Influence quantity
5.2.4 - Robotic aided Surgery Communi­cation service availa­bility: target value in % Communi­cation service reliabi­lity: Mean Time Between Failure End-to-end latency: maximum Bit rate Direction Message Size [byte] Survival time UE speed # of active UEs Service Area [m2]
Stereoscopic uncompressed 8K (7680x4320 pixels) 120 fps HDR 10bits real-time video stream>99.99999>1 year<2 ms240 Gbits/sUE to Network; Network to UE~1500 - ~9000 (note 3)~8msstationary1100
Stereoscopic 4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream with lossless compression (note 1)>99.99999>1 year<2 ms24 Gbits/s (note 2)UE to Network; Network to UE~1500 - ~9000 (note 3)~8msstationary1100
Uncompressed 8K (7680x4320 pixels) 120 fps HDR 10bits real-time video stream>99.99999>1 year<50 ms120 Gbits/sNetwork to UEs~1500 - ~9000 (note 3)~8msStationary<10100
4K (3840x2160 pixels) 120 fps HDR 10bits real-time video stream with lossless compression (note 1)>99.99999>1 year<50 ms12 Gbits/s (note 2)Network to UEs~1500 - ~9000 (note 3)~8msStationary<10100
Motion control data stream>99.999999>10 year<2 ms2 Mbits/sUE to Network~250~1 msStationary1100
Motion control data stream>99.999999>10 year<2 ms16 Mbits/sNetwork to UE~2000~1 msStationary1100
Haptic feedback data stream>99.999999>10 year<2 ms16 Mbits/sUE to Network~2000~1 msStationary1100
Haptic feedback data stream >99.999999>10 year<2 ms2 Mbits/sNetwork to UE~250~1 msstationary1100
NOTE 1:
This line provides alternative KPIs that are still acceptable
NOTE 2:
An average compression ratio of 2.5 has been considered when applying a lossless compression algorithm
NOTE 3:
MTU size of 1500 bytes is not generally suitable to gigabits connections as it induces many interruptions and loads on CPUs. On the other hand, Ethernet jumbo frames of up to 9000 bytes require all equipment on the forwarding path to support that size in order to avoid fragmentation.
Up

Up   Top   ToC