IEEE Trans Med Imaging. 2026 Aug 10;PP. doi: 10.1109/TMI.2026.3722251. Online ahead of print.

ABSTRACT

Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art methods rely exclusively on RGB video and are evaluated only in dry-lab settings, failing to bridge the substantial domain gap to real clinical procedures. In clinical RAS cases such as Robot-assisted Hysterectomy (RAH) and Robot-Assisted Radical Prostatectomy (RARP), background tissue peristalsis and strong specular reflections from instruments, needles, and catheters introduce significant visual challenges. This work introduces RAH-skill and RARP-skill, two clinical datasets for RAS skill assessment, comprising 279,691 and 70,661 RGB frames from 37 and 33 suturing videos, both paired with expert M-GEARS annotations and automatically generated optical flow and segmentation masks. We propose SurgFusion-Net, a novel multimodal fusion network that integrates RGB frames, optical flow, and tool segmentation masks to address background motion noise and instrument specular reflections in RAS skill assessment. The core component, Divergence Regulated Attention (DRA), is a scene-aware fusion mechanism that adaptively regulates cross-modal feature alignment to address the visual challenges of clinical cases, ensuring accurate and robust RAS skill assessment across heterogeneous surgical settings. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively. Our source code and dataset is available at https://github.com/HRL-Mike/SurgFusion-Net.

PMID:42574415 | DOI:10.1109/TMI.2026.3722251