Needle Tracking Dataset - Clinical & Semi-Clinical

Needle Tracking Dataset - Clinical & Semi-Clinical Overview This dataset contains multi-view RGB video sequences for training and evaluating markerless needle-tracking systems in CT-guided surgical environments. The data were acquired at the Research Campus STIMULATE in Magdeburg, using synchronized RGB cameras to capture realistic needle-manipulation scenarios under diverse conditions. The dataset comprises two complementary subsets:- Clinical Dataset: ~22,000 frames recorded in a CT laboratory replicating real interventional workflows- Semi-Clinical Dataset: ~8,000 frames acquired under controlled laboratory conditions with systematic parameter variations Combined, the dataset provides approximately 30,000 frames spanning a comprehensive range of needle movements, environmental conditions, and realistic procedural scenarios. Hardware Setup- Cameras: Three hardware-synchronized RGB cameras (1920×1080 px resolution, 60 FPS)- Needles: Five biopsy needles with varying specifications: - 10 cm / 14G - 15 cm / 14G - 15 cm / 16G - 15 cm / 18G - 20 cm / 16G- Phantom: Abdomen phantom with surgical drapes- Environment: Clinical setup mimics realistic CT interventions; semi-clinical setup focuses on controlled parameter variations Ground Truth AcquisitionAccurate 3D needle-tip coordinates were obtained using independent reference systems operating simultaneously with the RGB-based tracking: - Semi-Clinical Dataset: Atracsys optical tracker (Target Registration Error < 0.3 mm)- Clinical Dataset: AprilTag-based image tracking (Target Registration Error < 0.7 mm) All frames were processed using needle-tracking software that integrates three synchronized RGB cameras and a UNetConvNeXt segmentation model to estimate markerless tip and base positions (TRE < 0.7 mm). Data Characteristics Sequence Properties:- Duration: 2 to 30 seconds per sequence- Temporal continuity was ensured across all sequences- Frame rate: 60 FPS- Resolution: 1920×1080 pixels Variation Coverage:The dataset includes systematic variations across multiple dimensions to ensure robust model training: 1. Lighting Conditions: Range from optimal CT-lab lighting to challenging low-light scenarios2. Occlusion Patterns: Partial occlusions from hands, surgical tools, and drapes3. Motion Dynamics: - Stable, controlled needle movements - Rapid intentional displacements - Complex manipulation patterns - Initial stabilization phases4. Environmental Factors: - Different backgrounds (laboratory, CT environment) - With and without phantom presence - Realistic surgical drape configurations5. User Interactions: Multiple operators with different handling techniques and viewing angles6. Camera Perspectives: Multi-view synchronized capture from three fixed positions Data Content Each frame contains:- RGB Images: Three synchronized camera views (1920×1080)- Camera System Tracking: 2D and 3D coordinates from the tracking system- Ground Truth: 2D and 3D coordinates from independent reference systems (Atracsys/AprilTag) Use CasesThis dataset is designed for:- Training and evaluating needle tracking algorithms- Uncertainty quantification in surgical tracking systems- Real-time needle localization in CT-guided interventions- Robustness testing under challenging conditions (occlusion, motion, lighting)- Multi-view triangulation and sensor fusion research- Temporal prediction and motion modeling- Tracking error analysis and calibration validation Data Format The dataset is provided as HDF5 files with JPEG-compressed images for efficient storage and fast loading. File Structure: needle_tracking_dataset_jpeg.h5├── sequence_name_1/│ ├── tracking # Camera system predictions (N_frames × 15)│ ├── ground_truth # Reference system measurements (N_frames × 6)│ ├── camera_0/│ │ └── frames_jpeg # JPEG-compressed frames (variable length)│ ├── camera_1/│ │ └── frames_jpeg│ └── camera_2/│ └── frames_jpeg├── sequence_name_2/│ └── ... Data Columns: Tracking data (camera system):- frame_id: Frame number- tip_2d_cam1_x, tip_2d_cam1_y, tip_2d_cam2_x, tip_2d_cam2_y, tip_2d_cam3_x, tip_2d_cam3_y: 2D tip coordinates in each camera view- base_2d_cam1_x, base_2d_cam1_y, base_2d_cam2_x, base_2d_cam2_y, base_2d_cam3_x, base_2d_cam3_y: 2D base coordinates in each camera view- tip_3d_x, tip_3d_y, tip_3d_z: 3D tip coordinates from tracking system Ground truth data (reference system):- marker_tip_3d_x, marker_tip_3d_y, marker_tip_3d_z: Ground-truth 3D tip coordinates- marker_base_3d_x, marker_base_3d_y, marker_base_3d_z: Ground-truth 3D base coordinates Loading Example: import h5pyfrom PIL import Imageimport numpy as npimport io with h5py.File('needle_tracking_dataset_clinical_jpeg.h5', 'r') as f: seq = f[' '] # Load tracking data tracking_2d = seq['tracking_2d'][:] # Shape: (N_frames, 13), dtype: int16 tracking_3d = seq['tracking_3d'][:] # Shape: (N_frames, 3), dtype: float32 # Load ground truth data ground_truth = seq['ground_truth'][:] # Shape: (N_frames, 6), dtype: float32 # Load JPEG frame jpeg_bytes = seq['camera_0/frames_jpeg'][0].tobytes() img = Image.open(io.BytesIO(jpeg_bytes)) frame = np.array(img) Compression:- Images: JPEG quality 98 (~70-75% size reduction from PNG)- Tracking/Ground truth: gzip compression level 5 Quality Assurance- All sequences manually reviewed for quality- Frames with calibration failures excluded- Ground-truth accuracy verified against reference measurements- Temporal consistency validated across sequences CitationIf you use this dataset, please cite: Steiger, M., Rezapourian, M., Rak, M., Hansen, C. (2025).Dynamic Uncertainty Level Assessment Framework for Real-Time Needle Tracking in CT-Guided Surgical Environments.Funding The work was partially funded by the German Federal Ministry of Research, Technology, and Space (within the Research Campus STIMULATE under the grant number ‘13GW0473A’) and by the European Regional Development Fund (under the operation number ‘ZS/2023/12/182010’ as part of the initiative ‘Sachsen-Anhalt WISSENSCHAFT Schwerpunkte’).