Home  Details  Evaluation  Register  Download  Submit  Results 

The EMPIRE10 Challenge

This page provides in depth information about the EMPIRE10 challenge. It assumes you understand the basic concept already. For an overview description please see the home page.

If you are interested in participating please read all details carefully. If anything is not clear or you have additional questions please email keelin@isi.uu.nl .

Goal of the EMPIRE10 challenge

The main goal of the EMPIRE10 challenge is to evaluate the state of the art in chest CT registration. Although performance details of many algorithms are published, and indeed many algorithms are available for download, comparing their behaviour is not a trivial task. Authors typically use different datasets and different evaluation measures when publishing their performance results, making legitimate comparisons between their methods virtually impossible. It is possible to obtain several publicly available algorithms and compare them on a particular dataset, but many methods require in-depth knowledge and understanding in order to configure them in an optimal fashion. Such a comparison may therefore be flawed by incorrect configuration of one or more of the algorithms involved. EMPIRE10 therefore provides a unique opportunity to make legitimate comparisons between registration algorithms.

A summary article based on the initial phase of the challenge will be published by the team of EMPIRE10 organisers in a peer-reviewed journal (either "IEEE Transactions on Medical Imaging" or "Medical Image Analysis"). [Please note, as of November 2011 this publication is available: Murphy et al., "Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge.", IEEE Trans Med Imaging. 2011 Nov;30(11):1901-20. Please cite this article if you reference EMPIRE10.] Teams which participated in the initial phase of the challenge, including registering the workshop data will be included as co-authors on this article (up to 3 co-authors per team). The article will describe the challenge, a brief overview of each participating algorithm and a summary of the results. However, the challenge remains open, and teams are encouraged to submit improved results. Teams which did not participate in the initial phase are always welcome to register and submit their results also. In this way the EMPIRE10 website will continue to reflect the state of the art in registration of pulmonary CT images.

Rules

The EMPIRE10 challenge is organised in the spirit of cooperative scientific progress. We therefore ask anybody using this website to respect the rules below.

We do not claim any ownership or rights to the algorithms.

The following rules apply to those who register a team and download the data:

Data

The data to be downloaded consists of 30 pairs of chest CT scans. Each pair of scans is taken from a single subject, in other words no inter-subject registrations are included. The scans come from a variety of sources and are provided by several different institutes. Scans may be taken at various phases in the breathing cycle (full inspiration, full expiration, phase from 4D breathing data). Subjects may exhibit lung disease or appear healthy. Data from a variety of scanners is included and a variety of voxel sizes occur. In this way we have a included broad range of data, encompassing many of the problems faced by researchers developing registration algorithms for this application.

In addition to the CT data, binary lung masks are provided for each scan. These may be used by participants in registering the scans if they wish to do so. The lung masks are automatically generated using an algorithm by van Rikxoort et al. [1]. They have been visually checked and manually corrected where necessary.

The data is available on the download page. It is split into several files to prevent the file size becoming problematic for some users. The downloaded data files are in .zip format and may be unzipped using any suitable program.

All data (both scans and lung masks) is in Meta format. This format stores an image as an ASCII readable header file with extension .mhd and a separate binary file for the image data with extension .raw. This format is ITK compatible; documentation is available here. An application that can read the data is SNAP. If you want to write your own code to read the data, note that in the header file you can find the dimensions of the scan, the voxel spacing and the pixel type (short for the CT scans and unsigned char for the binary lung masks). In the raw file the values for each voxel are stored consecutively with index running first over x, then y, then z.

All scans have origin at (0,0,0), center of rotation at (0,0,0) and anatomical orientation in the RPI frame

Please note that the scans provided are cropped from original images such that regions outside the lung volume are excluded where possible (i.e. cropped using a lung bounding box). This was done to make the data to be downloaded smaller and because registration outside the lung volumes will not be considered. Information on the cropping coordinates may be found here.

The unzipped data is divided into two folders, "scans" and "lungMasks". Within these folders, each filename includes an ID for the subject in question, followed by either "_Fixed" or "_Moving". e.g. 01_Fixed, 01_Moving, 02_Fixed, 02_Moving ... etc. The filenames imply the registration that is to be carried out therefore it is important to understand them correctly. In this case 01_Fixed and 01_Moving are a pair of scans (from subject number 01) to be registered, with 01_Fixed being the fixed (or target) image, and 01_Moving being the moving (or source) image.

Performing Registration

Registration should be carried out using your software with parameter settings that you consider optimal. The use of lung masks or other masks which you may wish to derive is permitted. Please note that evaluation will be based on the registration of the lung volume only therefore it is not necessary to consider the alignment of other surrounding tissue and structures.

Please make careful note of the parameter settings and configurations that are used in your registration as you will be required to report these.

Three categories of algorithm will be defined:

  • Fully Automatic (Uses the same parameters (or determines parameters automatically) on all scan pairs. Requires no user interaction)
  • Semi-Automatic (Parameter settings were changed manually for different scan pairs. The total number of parameter settings used must be reported.)
  • Interactive (User interaction such as manual alignment, defining corresponding point pairs etc. was required for 1 or more of the scan pairs processed.)

    If your method is semi-automatic we encourage you to make as few changes to parameters as possible. A system which requires parameters to be manually altered for every new registration it performs is not clinically relevant.

    Only algorithms which are fully automatic, or semi-automatic will be considered for the challenge prizes. Semi-automatic algorithms where the degree of manual parameter alteration is considered excessive and impractical for a clinical setting may also be excluded from the prizes at the discretion of the organisers.

    Submitting Results

    Please read this section very carefully before you submit your results. The organisers reserve the right not to evaluate data which is submitted in a format other than that described here.

    For each scan pair that you register you must submit a deformation field which contains displacement information. A deformation field, in this case, consists of three 3-D images, defX, defY and defZ, each one with the same dimensions as the fixed image in the registration pair. The images should be of type floating point (MET_FLOAT), and in the same Meta format as the data which you downloaded from this site. The values contained in the deformation field images should be displacement distances (in mm) in the X, Y and Z directions respectively. For additional information about how we define a deformation field please see here. An example output set can be downloaded on the downloads page.

    Your deformation images for each registration pair should be placed in a folder with the ID of the subject. For example, after registering the scans 01_Fixed and 01_Moving, the resulting deformation images should be placed into a folder named "01". The deformation images MUST have the names "defX.mhd", "defY.mhd" and "defZ.mhd" (with the appropriate raw file accompanying each one).

    These deformation images are quite large in size and do not compress very well. It is therefore recommended that you use our executable program packageEmpire10.exe (available from the download page) to package your data before submission. This program sets regions of the image which are not needed for our evaluation to zero, allowing them to compress much better. It then compresses them into password protected zip files ready for you to submit. Note that the program requires the binary lung masks to be available. See the readme.txt file included with the executable for more information.

    If you do not use the packageEmpire10 executable you may package the data in zip format yourself but be aware it will take longer to upload due to its size. It is required to make an individual zip file for each registered pair with the subject ID as the filename. e.g. file 01.zip should contain the results from registering scans 01_Fixed and 01_Moving. The zip file should not contain any path data, just filenames.

    Files should be submitted by sftp, for details please see the submit page. Don't forget to submit a pdf document describing your method in addition. The description should be sufficient to give a good understanding of how the method works. A checklist of items we would suggest you include is given below:

    Once you have finished submitting your files please notify the organisers that your submission is complete by emailing keelin@isi.uu.nl .

    Evaluation

    Evaluation is carried out by the EMPIRE10 organisers once you have submitted your results and emailed to confirm they are complete. Details of the evaluation methods used and the scores/rankings produced can be found on the evaluation page. Results for evaluated algorithms will also appear on the results page.

    How often can I submit results?

    It is possible to re-submit results as often as you wish, bearing in mind that submission of the files may take some time. New submissions will overwrite previous ones. Please notify keelin@isi.uu.nl each time you re-submit so that your results can be re-evaluated.

    Currently we do not offer the possibility for teams to remove submitted results. If you believe there are good reasons to remove certain results that you have submitted, please contact keelin@isi.uu.nl .

    [1]"Automatic lung segmentation from thoracic CT scans using a hybrid approach with error detection." E.M. van Rikxoort, B. de Hoop, M.A. Viergever, M. Prokop, B. van Ginneken.
    Medical Physics 36(7) (2009)



    This website is copyright 2009-2010 Keelin Murphy, Bram van Ginneken.