A new artificial intelligence (AI) tool that interprets medical images with unprecedented clarity could allow time-strapped clinicians to dedicate their attention to critical aspects of disease diagnosis and image interpretation.

The tool, called iStar (Inferring Super-Resolution Tissue Architecture), was developed by researchers at the Perelman School of Medicine at the University of Pennsylvania, who believe they can help clinicians diagnose and better treat cancers that might otherwise go undetected. The imaging technique provides both highly detailed views of individual cells and a broader look of the full spectrum of how people’s genes operate, which would allow doctors and researchers to see cancer cells that might otherwise have been virtually invisible. This tool can be used to determine whether safe margins were achieved through cancer surgeries and automatically provide annotation for microscopic images, paving the way for molecular disease diagnosis at that level.

A paper on the method, led by Daiwei “David” Zhang, PhD, a research associate, and Mingyao Li, PhD, a professor of Biostatistics and Digital Pathology, was published today in Nature Biotechnology.

Li said that iStar has the ability to automatically detect critical anti-tumor immune formations called “tertiary lymphoid structures,” whose presence correlates with a patient’s likely survival and favorable response to immunotherapy, which is often given for cancer and requires high precision in patient selection. This means, Li said, that iStar could be a powerful tool for determining which patients would benefit most from immunotherapy.

The development of iStar was taken on as part of the field of spatial transcriptomics, a relatively new field used to map gene activities within the space of tissues. Li and her colleagues adapted a machine learning tool called the Hierarchical Vision Transformer and trained it on standard tissue images. It begins by breaking down images into different stages, starting small and looking for fine details, then moving up and “grasping broader tissue patterns,” according to Li. A network guided by the AI system within iStar uses the information from the Hierarchical Vision Transformer to then absorb all of that information and apply it to predict gene activities, often at near-single-cell resolution.

“The power of iStar stems from its advanced techniques, which mirror, in reverse, how a pathologist would study a tissue sample,” Li explained. “Just as a pathologist identifies broader regions and then zooms in on detailed cellular structures, iStar can capture the overarching tissue structures and also focus on the minutiae in a tissue image.”

To test the efficacy of the tool, Li and her colleagues evaluated iStar on many different types of cancer tissue, including breast, prostate, kidney, and colorectal cancers, mixed with healthy tissues. Within these tests, iStar was able to automatically detect tumor and cancer cells that were hard to identify just by eye. Clinicians in the future may be able to pick up and diagnose more hard-to-see or hard-to-identify cancers with iStar acting as a layer of support.

In addition to the clinical possibilities presented by the iStar technique, the tool moves extremely quickly compared to other, similar AI tools. For example, when set up with the breast cancer dataset the team used, iStar finished its analysis in just nine minutes. By contrast, the best competitor AI tool took more than 32 hours to come up with a similar analysis.

That means iStar was 213 times faster.

“The implication is that iStar can be applied to a large number of samples, which is critical in large-scale biomedical studies,” Li said. “Its speed is also important for its current extensions in 3D and biobank sample prediction. In the 3D context, a tissue block may involve hundreds to thousands of serially cut tissue slices. The speed of iStar makes it possible to reconstruct this huge amount of spatial data within a short period of time.”

And the same goes for biobanks, which store thousands, if not millions, of samples. This is where Li and her colleagues are next aiming their research and extension of iStar. They hope to help researchers gain better understandings of the microenvironments within tissues, which could provide more data for diagnostic and treatment purposes moving forward.