In this section several high level applications, and their machine learning building blocks will be further examined.

Device navigation

Typically, in minimally invasive procedures the in-body device can only be navigated and monitored through external imaging. Suitable imaging modalities are ultrasound, interventional x-ray, and real-time CT and MR. AI can be employed to detect and locate interventional devices, such as intra-vascular devices, and other percutaneous devices. Intra-vascular devices comprise catheters and guidewires [1] (Figure 1), intra-vascular valves (Figure 2), stents (Figure 3), etc. Other percutaneous devices entail needles [2], scalpels, etc. AI algorithms are particularly suited for detecting and segmenting devices since they are trained by a suitable set of examples [1,10,11]. This implies that the training set can contain a variety of devices with different visual properties, and it can be easily extended with new devices.

Integrating pre-interventional planning data

Integrating data from various imaging modalities can aid the interventional treatment procedure. E.g., pre-interventional planning conducted on diagnostic images can be utilized during the procedure [12], see Figure 4. The spatial registration of the pre-interventional and peri-interventional images can then bring the pre-interventional planning into the coordinate space of the interventional equipment. This allows to overlay the planning, such as a needle path, on the live images containing the interventional devices. Also, multiple complementary imaging modalities, such as e.g. ultrasound and x-ray, can be combined to create richer more informative data [13]. The combination of the images can show interfaces between tissues and objects that can only be visualized by a different imaging modality.

The spatial co-registration process can be conducted based on explicit markers and other external knowledge, on image content alone, or a combination of those. The resulting spatial mapping can be rigid, affine, elastic, or other deformable, depending on the clinical application. E.g., for intra-cranial applications a rigid registration is often sufficient, while registering pre- and intra-interventional abdominal images may require elastic registration to account for respiratory motion, etc. [4].

AI based approaches may play a role in establishing the spatial co-registration mapping either by detecting explicit landmarks and/or identifying landmark features in images, or by integrally addressing the registration task [14,15].

Functional parametrization

Functional imaging has as purpose to characterize the functioning of biological processes, rather than visualizing the anatomy (though it is typically combined with anatomical imaging in order to localize the functional aspects). Examples of the functions that are imaged peri-interventionally are blood flow in vessels and aneurysms [9,16], blood perfusion of the parenchymal tissue, valve motion, etc. Functional imaging is typically based on intensive processing of raw measurements. For e.g. blood flow vector fields through digital subtraction angiography (see Figure 5), the motion of contrast through the vascular structures is followed in the consecutive frames, while for valve motion the valve leaflets are segmented and followed over time. These segmentations and motion of carrier substances are very well suited for AI approaches, such as convolutional networks [1,10].