Aims and objectives
In imaging research and development a rigorous evaluation of the reproducibility of diagnostic tests before dissemination into clinical practice is essential.
Radiological observer performance studies are typically conducted by having observers read cases at a fixed location on a PACS (Picture Archiving and Communication System) workstation or DICOM (Digital Imaging and Communications in Medicine) viewer installed on a standalone PC while recording their measurements either into an electronic database or with pen and paper.
This practice poses logistical challenges,
and a concern is that the...
Methods and materials
Development and functionality
The app was developed as a cloud-based system  using the following open-source tools:
The PHP programming language handles server-side functions.
Data is stored in a mySQL database
Study cases are prepared by de-identifying and exporting DICOM studies to JPEG files (retaining native image resolution) and adding relevant DICOM tags to the mySQL database.
The app can handle multiple modalities including but not limited to: plain...
The app was tested on MAC and PC laptop/desktop computers using browsers: Chrome,
and Opera with full functionality on screen resolutions from 800x600 to 4096×2160 pixels.
None of the three invited observers reported problems accessing the app and were able to complete the case readings without any further instructions than the one that came in the invitational e-mail and the app (Fig.
Total reading time (300 cases)
34 minutes 25 seconds
This feasibility study shows that the developed app allows invited observers to be scattered in localization where case readings can be separated in time and not necessarily have to be completed all at once.
using this web-based tool it is possible with relative ease to efficiently conduct multicenter observer performance studies with potential significant savings in time under reasonably realistic conditions with a sufficiently high number of observers/cases.
In the future the Content Mangement System of the app will be further developed to allow...
Jens Borgbjerg MD
Department of Radiology
Aalborg University Hospital
1: Di Leo G.
Measurements in radiology: the need for high reproducibility.
PubMed PMID: 25552389.
2: Bankier AA,
Consensus interpretation in
imaging research: is there a better way? Radiology.
PubMed PMID: 20851935.
3: Sadatsafavi M,
Reliability studies of
diagnostic tests are not using enough observers for robust estimation of
interobserver agreement: a simulation study.
J Clin Epidemiol.