Hasso-Plattner-Institut
Prof. Dr. Christoph Lippert
 

Syreal - Synthesizing realistic medical images

Overview

Developing robust medical AI application requires large and diverse datasets for training. Medical data, however, is often limited because it contains sensitive human data that poses a risk to data privacy. Available medical datasets were often collected under defined settings (e.g. among symptomatic people seeking care), that may be different from future application settings, which can lead to failures during deployment. Datasets to test the performance of medical AI algorithms within a variety of possible test cases are lacking, but needed to advance the development of robust medical AI.

 

Project Aims

The aims of this project are to synthesize realistic medical images, which can be used to train and test medical AI algorithms that are applicable in clinical practice. The specific aims are as followed:

  1. Develop methods to generate medical images and apply them train and test the performance of medical algorithms.
  2. Provide synthetic images to the research field
  3. Integrate generative methods in a software package and distribute it though our SME-project partners

 

Funding

This research is supported through the German Federal Ministry of Education and Research (BMBF), Grant No. 01/S21069A.

 

Clinical use cases

Synthesizing histopathology images to develop algorihtms predicting cancerous lung tissue

This project stream involves the partners at LMU as domain experts in pathology, the partners at Fraunhofer HHI, Aignostics and Dotphoton as technical experts.

Synthesizing brain MRIs to develop algorithms predicting dementia-related abnormalities.

This project stream involves the partners at MDC as domain experts in MR imaging, the HPI and partners at Imfusion as technical experts.

We trained a conditional StyleGAN3 with a transfer-causal inference step (Manuscript will follow soon) on T1 structural brain MR images from the Alzheimer's Disease Neuroimaging Initiative (ADNI). For training, we used 2D coronal mid-slice brain MRIs and conditioned on Age, Sex, clinical dementia rating (CDR) and the brain structure (volumes). Using an inference step with transfer-learning and a causal model, the GAN learned the relationship between conditional variables and the brain structure. We generated new images with corresponding labels of either ‘cognitive normal’ (CDR Score=0), ‘mild cognitive impairment’ (CDR Score=0.5) or ’Alzheimer’s Disease (CDR Score=1), with the covariates age and sex. The generated images and labels can be downloaded here.

Consortium

The consortium is led by Prof. Dr. Christoph Lippert, HPI (project lead)

HPI project members:

Project partner:

  • Fraunhofer Heinrich Hertz Institute (HHI), Dept. for Artificial Intelligence, Berlin. Website
  • Max-Delbrück-Centrum für Molekulare Medizin in der Helmholtz-Gemeinschaft (MDC), Dept. for Experimental Ultra High-Field MR, Berlin. Website
  • Ludwig-Maximilian-Universität München (LMU), Pathologisches Institut, Munich. Website
  • Aignostics GmbH, Berlin. Website
  • ImFusion GmbH, Munich. Website
  • Dotphoton AG, Zug, Switzerland. Website

Publications

Publications with funding by BMBF for Syreal:

  • Oala et al. "Data Models for Dataset Drift Controls in Machine Learning With Optical Images", TMLR 2023
  • Fehr et al. "Assessing the transportability of clinical prediction models for cognitive impairment using causal models" BMC Med Res Meth 2023
  • Taleb et al. "ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics" CVPR 2022
  • Kirchler et al. "transferGWAS: GWAS of images using deep transfer learning" Bioinformatics 2022
  • Monti et al. "Identifying interpretable gene-biomarker associations with functionally informed kernel-based tests in 190,000 exomes" Nat Comm 2022
  • Kirchler et al. "Training normalizing flows from dependent data", arXiv 2022
  • Kirchler et al. "Explainability Requires Interactivity" arXiv 2021