CASOG: Conservative Actor–Critic With SmOoth Gradient for Skill Learning in Robot-Assisted Intervention
Hao Li, Xiao-Hu Zhou, Xiao‐Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Zeng‐Guang Hou
Abstract
The robot-assisted intervention has shown reduced radiation exposure to physicians and improved precision in clinical trials. However, existing vascular robotic systems follow master-slave control mode and entirely rely on manual commands. This article proposes a novel offline reinforcement learning algorithm, Conservative Actor–critic with SmOoth Gradient (CASOG), to learn manipulation skills on vascular robotic systems. The proposed algorithm conservatively estimates Q-function and smooths gradients of convolution layers to deal with distribution shift and overfitting issues. Furthermore, to focus on complex manipulations, transitions with larger absolute temporal-difference error are sampled with higher probability. Comparative experiments on multiple vascular models and offline data demonstrate that CASOG delivers guidewire to the target with higher success rates and fewer backward steps than prior offline reinforcement learning methods. These results indicate that the proposed algorithm is promising to improve the autonomy of vascular robotic systems.