Competitive benchmarking of super-resolution microscopy software

Competitive benchmarking of super-resolution microscopy software

In single molecule localization microscopy, like in other computational imaging techniques such as magnetic resonance imaging, the final image is so indirectly related to the raw data that the analysis algorithm effectively becomes a component of the microscope. So ensuring that the analysis software works well can be as important to final data quality as making sure there are no gouges in your microscope’s objective.

Single molecule localization microscopy is a super-resolution method based around finding the centres of blinky blobs. In conventional fluorescence microscopy, the emitted fluorescence from labelled molecules is blurred by the microscope, leading to a spatial resolution of about 250 nanometres. Localization microscopy circumvents this by making the labelled fluorescent molecules blink, by chemical or other means. This means that most of the time, we see only isolated fluorescent molecules, whose sharp point-like locations are transformed by the microscope into blurry blobs. Unsurprisingly, it turns out that the most likely location of a fluorescent molecule is in the middle of its corresponding blob, so we can super-resolve a molecule’s position by finding the blob centre. That is essentially all there is to localization microscopy – make molecules blink, image the blinky blobs, find the middle of those blobs, and plot a super-resolved image of the blob centres, which yields roughly 20 nanometre resolution (Figure 1). Of course, since this involves localising tens of thousands to millions of blobs, the analysis all needs to be done by software.

Figure 1
Figure 1: Principle of two-dimensional localization microscopy. Overlapping fluorescent blobs (point spread functions) are separated by fluorophore blinking. Blob centres are found and plotted together to obtain a super-resolved image.

Between 2013 and 2015, my colleague Daniel Sage led the first localization microscopy software challenge, to assess the performance of two dimensional localization microscopy software (Sage et al., Nature Methods 2015). This involved feeding synthetic 2D localization microscopy datasets into many different software packages. By comparing the resulting super-resolved images to the ground truth of the simulated input, it was possible to tell how well the localization microscopy software performed. This was a really useful piece of work that allowed users to identify the best software for their biological problem and gave developers a robust means of benchmarking their shiny new software against state of the art.

But what about 3D? For several years, I had been using 3D localization microscopy to image protein organization in cells, and getting more and more frustrated by my lack of understanding of what exactly the 3D localization microscopy software was doing to my data – and presumably everyone else’s. Were imaged structures being degraded or distorted by the image analysis? Was image resolution being lost due to imperfect software? Of course, one can work around this with robust biological controls, but to properly solve at the problem, we needed some means of testing 3D software performance. 

So, over a beer in the Swiss sunshine in 2015, Daniel and I hashed out a plan to extend the 2D localization microscopy software challenge to 3D. “How hard can it be?” we thought. Famous last words…

The trouble is, 3D localization microscopy is a much more complex image processing challenge than 2D localization microscopy. Where 2D localization microscopy “just” entails finding the centre of blobs (point spread functions), in 3D localization microscopy the axial position of the molecule is additionally encoded in the shape of the blob. For example, in the most popular 3D SMLM modality, astigmatic 3D, an extra lens is added to the microscope which turns the point spread function into a horizontal squashed sausage below the plane of focus, and a vertical squashed sausage above the focus (Figure 2). 3D SMLM software then analyses the point spread function’s shape to extract the axial molecule position. 

Figure 2:
Figure 2: Principle of three-dimensional localization microscopy. Axial position is encoded in the shape of the point spread function. Crosses indicate molecule lateral position.

Because of this requirement to analyse blob shape, 3D localization microscopy image processing is harder than for 2D. And simulating 3D localization microscopy data is also much harder. For 2D localization microscopy it was adequate to approximate the fluorophore point spread functions using every quantitative scientist’s favourite probability distribution, the Gaussian. But for 3D localization microscopy, in order for the competition to be relevant to the real world, it became essential to accurately simulate the shape of the point spread functions as they appear on real microscopes. If we just tested software on simulated Gaussian blobs, how would we know how software performed on the imperfect point spread functions of real microscopes? On top of this, the data challenge was bigger: four times the number of simulations to allow for the multiple different 3D modalities, and a whole extra dimension in which to quantify performance. 

Fortunately, the localization microscopy community is highly collaborative, and many labs kindly contributed a huge amount of time and effort to help design and run the 3D localization microscopy software challenge, together with an updated 2D software challenge. After many extended discussions, we came up with a design for the challenge which involved directly incorporating the point spread functions of several real world microscopes in the simulations, as well as realistic models for simulated structures, molecule photophysics and image noise (Figure 3).

Figure 3
Figure 3: Software was benchmarked against both simulated (left) and real datasets (right).

Thirty-six teams entered their software in the challenge, to whom we are grateful for making the entire endeavour possible. After a huge amount of data analysis, especially by outstanding EPFL graduate student Thanh-An Pham, we were able to start to draw some conclusions about the current state of 3D localization microscopy software. You can read the full analysis in the paper, but one broad finding is really worth mentioning here. Software matters – a lot. For 2D localization microscopy, we found that these days most software performs pretty well. However, for 3D localization microscopy, we saw large variation in software performance. The best ranked 3D localization microscopy software currently performs far better than mid-ranked software. So choose wisely!

A common limitation of super-resolution microscopy software, and scientific software in general, is sub-optimal usability and robustness. I was reminded of this by an experience during the peer review process for this paper, where we performed a pilot study testing some of the competition software on real experimental data. Mostly, software authors kindly performed this analysis. However, for one of the best performing software packages, the team who had written the software could not be contacted, so I tried to run the analysis myself. But even as an expert, after two days of trying I could not complete the analysis with that software, and ultimately abandoned the attempt.

To me, that experience was an example of how fragile scientific software often is, and how scientific software often dies when the author becomes unable to support it. This problem affects many fields, and I think solving it properly will require changing how we incentivise scientific software development. Currently in the life sciences we primarily fund and publish production of proof-of-concept algorithms and software. I think we also need to explicitly support turning proof-of-concept software into user-friendly and robust software packages. Furthermore, we will need to recognize that this work takes substantial additional investment of time and resources.

That said, many localization microscopy software packages give good consideration to usability, and some are quite excellent in this respect. To help users assess usability, we collected qualitative reports about each challenge software, which are available on the challenge website. In future, we intend to address this question in more detail, probably via a user-submitted software review and rating system.

To sum up our study: current single molecule localization microscopy software rocks! The best 3D localization microscopy software packages give formidable performance, far improved beyond the classic algorithms of a decade ago. And substantial possibilities for improvement remain, so if you want to keep getting the best out of your super-resolution data, keep an eye on the challenge website for new top software:

Read the paper here:

Written by Seamus Holden

Please sign in or register for FREE

If you are a registered user on Springer Nature Protocols and Methods Community, please sign in