Using Semantic Technology to Effectively Search Bio-Medical Images
Effective image search based on content (color, texture, shape, etc.) has been difficult because of the semantic gap (by ‘effective’ we mean accurately retrieving a desired image, and retrieving it without retrieving too many other images which are not relevant – that is, retrieving an image with good precision and recall). Use of metadata to retrieve images can be more effective, if the metadata is captured and represented accurately. The use of semantics can help in searches based on metadata.
DICOM is a medical image standard to represent medical images and the associated metadata. The 2000+ attributes that constitute the metadata are stored as part of the image. Tools are used to extract the metadata from the image which can then be used for search. The attributes include Patient Name, Date, Equipment Used, Body Part, etc. to name a few.
Hospitals are some of the places where DICOM images are generated. Adopting DICOM as the standard ensures that everyone uses the same set of attributes. However, the vocabulary used for the values for the attributes can be different making searches across multiple DICOM repositories difficult. Vocabularies used can be different across hospitals, different in multiple departments within a hospital, and sometimes even within a department medical professionals might not use a standardized vocabulary.
For example, consider the attribute Body Part in a DICOM image of the lower half of the human face. One image might contain the word ‘Jaws’ as a value of the attribute Body Part. Another similar image might contain the word ‘Mandible’. Yet another image might contain the SNOWMED id for that body part, such as T-D1217. The situation is further complicated by the fact that ‘Mandible’ and ‘Jaws’ are not equivalent – ‘Mandible’ (and ‘Maxilla’) are a sub-part of ‘Jaws’.
An ontology can be defined to capture the relationships between different terms. In this particular example we can have:
T-11170 partOf T-D1217 T-11180 partOf T-D1217 T-D1217 equivalentTo Jaws Mandible partOf Jaws Maxilla partOf Jaws
With the application of the right rules, other relationships can be automatically derived:
Mandible partOf T-D1217 Maxilla partOf T-D1217 T-11170 partOf Jaws T-11180 partOf Jaws
And so on.
This ontology can now be used for searching across DICOM images. Given a search such as ‘Jaws’, this ontology can be referred to to construct a collection of search terms containing all the terms related to ‘Jaws’ – in this example, to create a collection of terms that are equivalentTo ‘Jaws’ and are a ‘partOf’ ‘Jaws’. This collection of search terms will more comprehensively retrieve images compared to just using one search term which will miss many relevant images. An ontology enables the representation of relationships which go beyond synonyms (which will be used in a good search engine). An image of the lower half of the human face can be identified to be equivalent to an image focused only on the lower half, since the ontology tells us that ‘Maxilla’ or the lower jaw is a partOf ‘Jaws’.
Searching based on an ontology can be extended and made more powerful by using ontologies based on a comprehensive clinical terminology such as SNOMED which is agreed upon by the medical community.