Please wait a minute...
Computational Visual Media  2019, Vol. 05 Issue (04): 363-374    doi: 10.1007/s41095-019-0156-x
Research Article     
Mixed reality based respiratory liver tumor puncture navigation
Ruotong Li1, Weixin Si2, Xiangyun Liao2, Qiong Wang2,(✉), Reinhard Klein1, Pheng-Ann Heng3
1Institute of Computer Science II, University of Bonn, 53115 Bonn, Germany. E-mail: R. Li,; R. Klein,;
2Shenzhen Key Laboratory of Virtual Reality and Human Interaction Technology, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China. E-mail: W. Si,; X. Liao,;
3Department of Computer Science and Engineering, Chinese University of Hong Kong, Hong Kong, China. E-mail:
Download: PDF (14522 KB)      HTML  
Export: BibTeX | EndNote (RIS)      


This paper presents a novel mixed reality based navigation system for accurate respiratory liver tumor punctures in radiofrequency ablation (RFA). Oursystem contains an optical see-through head-mounted display device (OST-HMD), Microsoft HoloLens for perfectly overlaying the virtual information on the patient, and a optical tracking system NDI Polaris for calibrating the surgical utilities in the surgical scene. Compared with traditional navigation method with CT, our system aligns the virtual guidance information and real patient and real-timely updates the view of virtual guidance via a position tracking system. In addition, to alleviate the difficulty during needle placement induced by respiratory motion, we reconstruct the patient-specific respiratory liver motion through statistical motion model to assist doctors precisely puncture liver tumors. The proposed system has been experimentally validated on vivo pigs with an accurate real-time registration approximately 5-mm mean FRE and TRE, which has the potential to be applied in clinical RFA guidance.

Key wordsmixed reality      human computer interaction      statistical motion model     
Received: 14 December 2019      Published: 13 March 2020
Corresponding Authors: Qiong Wang   
About author:

*Ruotong Li and Weixin Si contributed equally to this work.

Cite this article:

Ruotong Li, Weixin Si, Xiangyun Liao, Qiong Wang, Reinhard Klein, Pheng-Ann Heng. Mixed reality based respiratory liver tumor puncture navigation. Computational Visual Media, 2019, 05(04): 363-374.

URL:     OR

Fig. 1:  Mixed reality-based needle insertion navigation.
Fig. 2:  Statistical model based respiratory motion compensation.
Fig. 3:  Registration of 3D virtual structure and real object.
Fig. 4:  Patient-specific respiratory motion reconstruction. The red region shows the shape of the liver at the fully inhalation stage, while the light blue part region shows the shape of the liver at different respiration stage.
Fig. 5:  Animal experiment setting and 3D reconstruction results. (a) Tumor implantation using agar. (b) Metal landmark placement. (c) CT imaging. (d) 3D reconstruction of liver, tumor, and 10 metal landmarks.
Fig. 6:  Automatic registration accuracy validation; red points are landmarks.
Landmark numberReal position of landmarksRegistered position of landmarksRegistration error
Table 1: Performance statistics of automatic registration (Unit: mm)
Fig. 7:  Results of traditional CT-navigated needle insertion.
Fig. 8:  Mixed reality-navigated needle insertion.
Fig. 9:  Results of our mixed reality-based needle insertion navigation.
[1]   Wang, Z.; Aarya, I.; Gueorguieva, M.; Liu, D.; Luo, H.; Manfredi, L.; Wang, L.; McLean, D.; Coleman, S.; Brown, S.; uschieri, A. Image-based 3D modeling and validation of radiofrequency interstitial tumor ablation using a tissue-mimicking breast phantom. International Journal of Computer Assisted Radiology and Surgery Vol. 7, No. 6, 941-948, 2012.
[2]   Flaherty, D. C.; Bilchik, A. J.Radiofrequency ablation of liver tumors. In: Blumgart’s Surgery of the Liver, Biliary Tract and Pancreas, 2-Volume Set, 6th edn. Belghiti, J.; Jarnagin, W. R. Eds. Elsevier, 1436-1447, 2017.
[3]   Cai, K.; Yang, R.; Chen, H.; Ning, H.; Ma, A.; Zhou, J.; Huang, W.; Ou, S. Simulation and visualization of liver cancer ablation focus in optical surgical navigation. Journal of Medical Systems Vol. 40, No. 1, 19, 2016.
[4]   Clasen, S.; Pereira, P. L. Magnetic resonance guidance for radiofrequency ablation of liver tumors. Journal of Magnetic Resonance Imaging Vol. 27, No. 2, 421-433, 2008.
[5]   Crocetti, L.; Della Pina, M. C.; Cioni, D.; Lencioni, R.Image-guided ablation of hepatocellular carcinoma. In: Interventional Oncology: Principles and Practice of Image-Guided Cancer Therapy. Geschwind, J. F. H.; Soulen, M. C. Eds. Cambridge University Press, 91, 2016.
[6]   Cazzato, R. L.; Garnon, J.; Ramamurthy, N.; Tsoumakidou, G.; Imperiale, A.; Namer, I. J.; Bachellier, P.; Caudrelier, J.; Rao, P.; Koch, G.; Gangi, A. 18F-FDOPA PET/CT-guided radiofrequency ablation of liver metastases from neuroendocrine tumours: Technical note on a preliminary experience. CardioVascular and Interventional Radiology Vol. 39, No. 9, 1315-1321, 2016.
[7]   Bernhardt, S.; Nicolau, S. A.; Soler, L.; Doignon, C. The status of augmented reality in laparoscopic surgery as of 2016. Medical Image Analysis Vol. 37, 66-90, 2017.
[8]   Nicolau, S. A.; Pennec, X.; Soler, L.; Ayache, N.Clinical evaluation of a respiratory gated guidance system for liver punctures. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2007. Lecture Notes in Computer Science, Vol. 4792. Ayache, N.; Ourselin, S.; Maeder, A. Eds. Springer Berlin Heidelberg, 77-85, 2007.
[9]   Biro, P.; Spahn, D. R.; Pfammatter, T. High-frequency jet ventilation for minimizing breathing-related liver motion during percutaneous radiofrequency ablation of multiple hepatic tumours. British Journal of Anaesthesia Vol. 102, No. 5, 650-653, 2009.
[10]   Wunderink, W.; Romero, A. M.; de Kruijf, W.; de Boer, H.; Levendag, P.; Heijmen, B. Reduction of respiratory liver tumor motion by abdominal compression in stereotactic body frame, analyzed by tracking fiducial markers implanted in liver. International Journal of Radiation Oncology Biology Physics Vol. 71, No. 3, 907-915, 2008.
[11]   Breen, D. J.; Lencioni, R. Image-guided ablation of primary liver and renal tumours. Nature Reviews Clinical Oncology Vol. 12, No. 3, 175-186, 2015.
[12]   Tiong, L.; Maddern, G. J. Systematic review and meta-analysis of survival and disease recurrence after radiofrequency ablation for hepatocellular carcinoma. British Journal of Surgery Vol. 98, No. 9, 1210-1224, 2011.
[13]   Ahmed, M.; Brace, C. L.; Lee Jr., F. T.; Goldberg, S. N. Principles of and advances in percutaneous ablation. Radiology Vol. 258, No. 2, 351-369, 2011.
[14]   Livraghi, T.; M?kisalo, H.; Line, P.-D. Treatment options in hepatocellular carcinoma today. Scandinavian Journal of Surgery Vol. 100, No. 1, 22-29, 2011.
[15]   Kim, P. N.; Choi, D.; Rhim, H.; Rha, S. E.; Hong, H. P.; Lee, J.; Choi, J.-I.; Kim, J. W.; Seo, J. W.; Lee, E. J.; Lim, H. K. Planning ultrasound for percutaneous radiofrequency ablation to treat small (⩽3 cm) hepatocellular carcinomas detected on computed tomography or magnetic resonance imaging: A multicenter prospective study to assess factors affecting ultrasound visibility. Journal of Vascular and Interventional Radiology Vol. 23, No. 5, 627-634, 2012.
[16]   Amalou, H.; Wood, B. J. Electromagnetic tracking navigation to guide radiofrequency ablation (RFA) of a lung tumor. Journal of Bronchology & Interventional Pulmonology Vol. 19, No. 4, 323-327, 2012.
[17]   Sauer, F.; Schoepf, U. J.; Khamene, A.; Vogt, S.; Das, M.; Silverman, S. G.Augmented reality system for CT-guided interventions: System description and initial phantom trials. In: Proceedings of the SPIE 5029, Medical Imaging 2003: Visualization, Image-Guided Procedures, and Display, 384-395, 2003.
[18]   Khan, M. F.; Dogan, S.; Maataoui, A.; Wesarg, S.; Gurung, J.; Ackermann, H.; Schiemann, M.; Wimmer-Greinecker, G.; Vogl, T. J. Navigation-based needle puncture of a cadaver using a hybrid tracking navigational system. Investigative Radiology Vol. 41, No. 10, 713-720, 2006.
[19]   Ren, H.; Campos-Nanez, E.; Yaniv, Z.; Banovac, F.; Abeledo, H.; Hata, N.; Cleary, K. Treatment planning and image guidance for radiofrequency ablation of large tumors. IEEE Journal of Biomedical and Health Informatics Vol. 18, No. 3, 920-928, 2014.
[20]   Chan, W.-Y.; Heng, P.-A. Visualization of needle access pathway and a five-dof evaluation. IEEE Journal of Biomedical and Health Informatics Vol. 18, No. 2, 643-653, 2014.
[21]   Schweikard, A.; Shiomi, H.; Adler, J. Respiration tracking in radiosurgery. Medical Physics Vol. 31, No. 10, 2738-2741, 2004.
[22]   Ren, Q.; Nishioka, S.; Shirato, H.; Berbeco, R. I. Adaptive prediction of respiratory motion for motion compensation radiotherapy. Physics in Medicine and Biology Vol. 52, No. 22, 6651, 2007.
[23]   Jud, C.; Cattin, P. C.; Preiswerk, F.Chapter 14—Statistical respiratory models for motion estimation. In: Statistical Shape and Deformation Analysis. 379-407, 2017.
No related articles found!