← Back to Projects

LiverLens-Multimodal CT Segmentation & Diagnostic System

GitHub↗
LiverLens is a neuro-symbolic multimodal pipeline designed for automated liver metabolic health profiling from 3D CT scans while preventing AI hallucination of clinical metrics. The system separates perception, computation, and reporting into specialized stages so that neural models handle pattern recognition and language, while deterministic code handles quantitative analysis.

The pipeline begins with a 3D SwinUNETR segmentation model implemented in PyTorch and MONAI, which extracts the liver region from NIfTI CT volumes using a transformer-based architecture that captures both local texture and global anatomical context. The model achieves high segmentation accuracy (Dice ≈ 0.97) using sliding-window inference.

A deterministic analysis layer written in Python and NumPy processes the segmentation mask to compute mathematically exact metrics. Liver volume is calculated by counting mask voxels and applying voxel spacing derived from the NIfTI affine matrix. Hounsfield Unit statistics (mean, median, standard deviation) are computed within the mask, along with the fraction of voxels below clinically relevant thresholds for steatosis. This stage acts as a firewall ensuring all quantitative outputs are derived algorithmically rather than inferred by an LLM.

A MedGemma 4B vision-language model is then used as a report generation agent. Instead of analyzing images directly, it receives structured numerical findings and generates a clinical-style report (findings, impression, recommendations), leveraging domain knowledge while avoiding hallucinated measurements.

Advanced reliability features include uncertainty quantification through cross-model disagreement between SwinUNETR and a lightweight 2D segmentation head built on SigLIP embeddings, highlighting regions of low confidence. Test-time augmentation improves robustness by performing rotated inference and averaging masks. Explainability is enhanced via density histograms that visualize tissue composition relative to clinical thresholds.

The system is implemented in Python using PyTorch, MONAI, Hugging Face Transformers, and nibabel for volumetric data handling, with a Gradio interface providing an interactive visualization environment. The architecture emphasizes reproducibility, mathematical correctness, and clinical interpretability, making it suitable for research workflows and exploratory diagnostics.