News Column

"Material Identification Using Multiple Images" in Patent Application Approval Process

February 27, 2014



By a News Reporter-Staff News Editor at Politics & Government Week -- A patent application by the inventor Statham, Peter J. (High Wycombe, GB), filed on June 6, 2011, was made available online on February 13, 2014, according to news reporting originating from Washington, D.C., by VerticalNews correspondents.

This patent application is assigned to Oxford Instruments Nanotechnology Tools Limited.

The following quote was obtained by the news editors from the background information supplied by the inventors: "There exists a well-established technique for acquiring digital X-ray images using a scanning electron microscope (SEM) and one or more X-ray detectors. This technique involves directing an electron beam sequentially over a grid of points on a specimen surface. At each grid point, measurements are made at a specific X-ray energy with a so-called 'wavelength dispersive spectrometer' (WDS), or an X-ray energy spectrum is acquired and data are extracted from this spectrum for selected chemical elements of interest using an energy dispersive spectrometer (EDS). In the simplest case, these data consist of, say, N integrals of counts recorded in energy bands (or 'windows') surrounding the characteristic energy for each element of concern. When the full energy spectrum is available, more sophisticated techniques have been used to extract the area of each characteristic peak above background and correct for peak overlap and hence obtain the characteristic line intensity for the element of interest. Each data value is used as the 'pixel' intensity at that grid point for the element of concern so, after scanning over all the grid points covering the field of view, a set of N digital images corresponding to the N chemical elements of interest is obtained.

"These images are commonly referred to as 'X-ray maps'. These are commonly displayed using a computer monitor using a separate colour or hue for each element. For example, the 'silicon' map could be displayed in blue, the 'iron' map in red, the 'potassium' map in green, and the 'calcium' map in yellow. See for example P. J. Statham and M. Jones 'Elemental Mapping Using Digital Storage and Colour Display', Scanning Vol. 3, 3 (1980) pp 169-171.

"Images have also been combined by processing to show where certain elements co-exist in regions of a specimen. That is, once the original X-ray map data have been acquired, the images are inspected and then recombined in suitable combinations to reveal the distribution of phases in the specimen (where each phase consists of elements occurring in specific abundances and exhibits a colour distinct from other phases). See for example Statham and Jones, 1980 where Si (Blue), K (Yellow) and Fe(Red) maps are combined so that where two elements coexist and are above a threshold, a combination colour is shown (such as Blue+Yellow=Green, etc.) and where all three elements are present, white is displayed.

"Information presented in this way has to be obtained by skilled operators who can select the right colours and manipulate the N digital images in a suitable manner.

"In addition to the above technique that uses WDS or EDS for detection using an SEM for excitation, images representing some aspect of material content can also be produced by a variety of other sensors and excitation sources, for example: X-ray fluorescence (XRF), Electron Energy Loss spectroscopy (EELS), Particle Induced X-ray Emission (PIXE), Auger Electron Spectroscopy (AES), gamma-ray spectroscopy, Secondary Ion Mass Spectroscopy (SIMS), X-Ray Photoelectron Spectroscopy (XPS), Raman Spectroscopy, Magnetic Resonance Imaging (MRI), IR reflectometry and IR spectroscopy.

"The collection of N images is sometimes referred to as a 'multi-spectral' set of images and there is often a requirement to generate a single composite image that will highlight the location of different materials within a field of view. In some cases, more than one modality will be available so that signals from more than one sensor type can be obtained for a particular pixel position on the object so that in general the 'multi-spectral' set could contain images from different modalities.

"In the current context, a 'multi-spectral' set of images is not the same as a 'hyper-spectral' data set where, at each pixel, a series of contiguous channels of data are obtained that defines a full spectrum at each pixel. However, the 'multi-spectral' and 'hyper-spectral' data are obviously related because each spectrum in a 'hyper-spectral' data set can be processed to generate one or more values which can be used as intensities for a 'multi-spectral' set of images. For example, a full x-ray energy spectrum consisting of, say, 1024 channels of digital data for a single pixel could be processed to obtain the 4 intensities corresponding to characteristic x-ray lines for Si K, Fe K, K K and Ca K so that the full hyper-spectral 'spectrum image' is converted to a multi-spectral set consisting of Si, Fe, K and Ca element maps. Each image in a multi-spectral set usually identifies the spatial distribution of a particular property such as the concentration of a particular element, the mean atomic number of the material or a particular wavelength of cathodoluminescence radiation and so on, so that each image in a multi-spectral set already conveys some information to the observer.

"It is well known that colour images can be obtained by mixing images corresponding to the three primary colours: red, green and blue (R, G, B) (see for example, John J. Friel and Charles E. Lyman, 'X-ray Mapping in Electron-Beam Instruments', Microsc. Microanal. 12, 2-25, 2006, FIG. 14). If the intensity value at a particular pixel in one image is Xr, in a second image is Xg and in the third image is Xb, then the tuple (Xr,Xg,Xb) can be used to define the colour at the corresponding pixel in a single full colour image. If the first image represents the concentration of one chemical element and the other images represent concentrations of other elements, then if the first image is shown in red alongside the resultant full colour image, in regions where the first image has the dominant pixel intensity, the resultant image will also be red in appearance. Thus even though there will be regions of mixed colour, there is some intuitive correspondence between the original images and the resultant full colour image, In general, if N is 3 or less, a single full colour image can be constructed simply by assigning one of the primary components, R, G or B to each original image.

"If N>3, it is not possible to combine all the images in this fundamental way because there are only 3 independent channels for a typical RGB colour display. However one strategy for generating a single colour image that shows material content is to manipulate the data in the N images in order to obtain 3 suitable images that can be encoded in R, G and B and then mix these three images to expose mixture colours at each pixel that will delineate material content. This approach is sometimes called colour image fusion and the aim is to maximise the information content in the resultant full colour image. To this end, multivariate statistical techniques such a principal component analysis (PCA) are often employed to generate the images corresponding to the first 3 principal components. Each of the three principal component images is assigned to R, G and B to form the final full colour image. This approach has been described and assessed in detail (see for example, V. Tsagaris and V. Anastassopoulos, International Journal of Remote Sensing, Vol. 26, No. 15, 10 Aug. 2005, 3241-3254).

"In the case where the original N images correspond to chemical element distribution maps (for example element maps in EDS/SEM or maps for particular mass numbers in SIMS), the original maps are easy to interpret. However, when multivariate statistical approaches are used to extract components, each derived component image is a mathematical abstraction that does not necessarily bear any direct relationship to any of the original N images.

"An improved approach is described in Kotula et al, Microsc. Microanal. 9, 1-17, 2003, where a method using multivariate statistical analysis (MSA) is described to analyse a hyperspectral data set where the abstract principal components are converted into physically meaningful 'pure' components and thus colour component images. Selected component images are coloured and mixed or 'fused' to form a resultant image that bears a direct relationship to the component images. However, there is no attempt to provide a visual connection between the resultant image and the distribution of individual chemical elements that would normally be shown as a series of x-ray elemental maps. FIG. 1 summarises this prior art and shows that there is no direct connection between the construction of x-ray elemental maps and the visualisation of components derived by MSA. A hyper-spectral data set is used to construct x-ray maps. Separately, multivariate statistical analysis is applied to the hyper-spectral data to generate component images which are then 'fused' to generate a resultant colour image showing regions of different composition. Although this method generates a resultant colour image using MSA, the colour is not connected to the colours chosen for the x-ray maps.

"When N>3, a simple approach is to assign a distinctive individual colour to each original image then sum all the colour contributions from all the images at each pixel. For each image the signal intensity at each pixel is used to modulate the r, g, b components while maintaining the same colour hue. For the resultant mixture image, the r,g and b values from all the individually-coloured original images are summed to obtain a single r,g,b value for every pixel position. If necessary, the resultant mixture image can be scaled so that none of the r, g or b values exceeds the maximum allowed by the display technology used. This approach works well when there is very little overlap between the bright pixels in the original images because in regions where one original image dominates, the same colour will appear in the resultant image. However, if several of the original images are similar in appearance, in pixels where these images dominate over the others, the resultant pixel colour will not correspond to any one of the original images so the correspondence between original image and resultant image will be lost. Furthermore, as the number of original images, N, increases, the perceptible difference between the colours assigned to each original image decreases. Therefore, to get any success with this method, it is invariably necessary for an expert to select a suitable subset of original images for mixing and ignore the others.

"There is therefore a need in materials analysis applications to provide an automated method of processing a number of different input image datasets to form a combined image dataset in such a way that regions of the input images which contribute significantly to corresponding region in the combined image have a recognisable colour similarity between the respective regions of the input and combined images."

In addition to the background information obtained for this patent application, VerticalNews journalists also obtained the inventor's summary information for this patent application: "According to a first aspect of the present invention, there is provided a computer-implemented method of image processing for materials analysis comprising performing the following set of steps automatically:--

"a) obtaining N image datasets, each dataset representing intensity values of image pixels, wherein the image pixels represented by the datasets are in a common spatial registration, and wherein N is an integer greater than 3; b) processing the N image datasets so as to assign a comparison measure to each pair of image datasets, the comparison measure for a given pair of image datasets being representative of the difference between the spatial intensity information within one image dataset in comparison with the spatial intensity information within the other image dataset of the pair; c) selecting a number of the image datasets based upon the comparison measures; d) defining a colour difference measure which represents the difference between pairs of colours of a colour set; e) assigning a colour to each of the selected image datasets such that pairs of the selected image datasets which have substantially different spatial intensity information between datasets within a respective pair, are assigned respective colours which have a substantially different colour difference measure; and, f) combining a number of the selected image datasets together to form an output colour image dataset for the formation of a colour image such that each pixel of the colour image takes on a colour according to the relative intensities and colours of the said number of selected image datasets so that if the pixel intensity of one dataset is substantially greater than the sum of the pixel intensities for all the other selected datasets, the output colour in the respective part of the image will substantially match the colour assigned in step (e) to that image dataset.

"The invention enables an automatic and unsupervised method of assigning colours to a series of input 'images' and of generating a single full colour 'image' that retains the colour connection between an original input image and the resultant image in regions where the original input image has the dominant intensity over all other images. In the case where original images represent element content, the resultant colour image will show regions in different colour where the proportions of the elements are different. Thus, in the resultant image there will appear a series of regions in different colours where each colour is representative of a particular material. Furthermore, pixels where a particular element is dominant will have a similar colour to the original image for that element. Rather than use any predefined combination of images, the method is unsupervised and does not use any prior knowledge of the object being studied in that it works out a suitable colour scheme and a suitable mix of images using the information and statistical content of all the original input images.

"While an expert may be able to achieve a similar aim with manual selection of colours and mixture, the aim of the invention is to achieve this on any assemblage of images by an automatic algorithm that requires no user intervention. The great advantage of this is that users no longer need to be experts in image processing in order to ensure they gain the full benefit of an output colour image with a beneficial colour scheme.

"The benefit of the invention in practical terms derives from the ability to choose a suitable set of colours and for these to be applied to a suitable number of the datasets. The invention contemplates the assignment of a colour to each dataset in the sense that each dataset may be assigned a different colour. However, it is also contemplated that certain datasets may be assigned a common colour or that certain datasets are excluded from further consideration, during the method, for example once their similarity with each other dataset has been measured.

"The step of selecting the datasets for inclusion can be thought of as an output or final part of the processing. The selecting includes the possibility that the number of image datasets selected is the same as those processed in which case the selecting may be merely the provision of the processed datasets to the next processing stage. In other cases the selecting includes the selection of a smaller number of image datasets from those processed. The processing typically results in additional data being evaluated which represents or describes the relationships between the datasets, this data including the comparison measure. Other selection steps are also contemplated, such as initially (before any other of the stated steps), or following the assignment of the colours for example.

"The number of processed and selected datasets may be N in the same way as in the obtaining step. The initial number of datasets (prior to the obtaining step) for processing may therefore be N. However, a number of datasets in excess of N could be initially evaluated and then reduced to N prior to the obtaining step, it being recognised that the method benefits from a relatively small number of colours (such as 7 or fewer) since as the number of colours increases the greater the difficulty in maintaining the 'visual' relationship between the output dataset and the component input datasets from which it is generated.

"It is preferred that the number of assigned colours is less than a predetermined integer. Most preferably seven or fewer colours are assigned. The assigned colours are preferably selected in such a way as to provide maximum discernable visual difference to a human observer, such as a user of a computer system which implements the method. Preferably the colour difference measure used in the method is a function of angular colour hue.

"There are a number of different possible ways in which to assign appropriate colours to the datasets. In one such method the sum of the spatial information content of the image datasets is calculated for ranges of angular hue to evaluate an angular density as a function of hue angle. As part of this method, each image dataset is typically normalised to the maximum intensity present whereby the intensity within each image dataset is weighted in accordance with the angular density prior to the intensities being summed to form the resultant output colour image dataset. Alternative methods for achieving a similar effect may also use a normalising approach, such as by normalising the intensities of the image datasets.

"The processing step is typically achieved by considering a number of pair-wise comparisons between the datasets. A convenient manner in which this may be achieved as part of processing the image datasets is by using matrices. Typically the step of processing the image datasets includes calculating a matrix of numerical comparison measures for each image dataset pair. For example the comparison measure may be the sum of cross products of pixel intensities for the two images in a pair of image datasets under consideration. One benefit of such an approach is that the processing may further comprise evaluating whether the plurality of image datasets are substantially collinear and, if they are, then removing one dataset in order to reduce the measure of collinearity. The mathematical analysis of collinearity between datasets provides an advantage to certain implementations. However, alternative approaches using statistical measures of the differences in intensity data between pairs of datasets and in which the comparison measure has a magnitude relating to these statistical measures, are also of great value. For example, the magnitude of the colour difference measure between the assigned colours for each pair of image datasets may be made a function of the magnitude of the comparison measure for the pair under such circumstances.

"As mentioned above, in some implementations not all of the datasets are assigned different colours. For example, if a plurality of image datasets are evaluated as having sufficiently similar spatial intensity information, then either one of such datasets may be selected or a combined image dataset may be formed from the said plurality of image datasets for further processing. Such datasets may be removed as part of a collinearity calculation. The image datasets may also be processed in other ways, for example by evaluating the effect of noise within each image dataset. If this is performed at an early stage in the method then this allows for more efficient use of processing resources by removing from the method any image datasets in which the variation in intensity is not above a noise threshold.

"The step of combining the selected image datasets may include the combining of each of the image datasets to which a colour has been assigned. Alternatively a smaller number of such image datasets may be selected for the combining step.

"Having assigned the appropriate colours to the selected datasets, these preferably having appropriate Red, Green and Blue values, the method typically includes summing such values in the contributing datasets in order to form the output colour image dataset having corresponding Red, Green and Blue values. Whilst this approach uses the additive nature of primary colours, it may be preferred to use pixel intensity values of the output colour image dataset according to a subtractive colour model. Such a subtractive model has greater parallels with the mixing of paints and this may provide more intuitive mixtures of colours.

"Another possible method of combining datasets which may be used as an alternative or addition to numerical combination of colour values is to apply the colours from different datasets in a small region of adjacent pixels. Therefore, when the output colour image dataset comprises individual pixel data from each image dataset, the pixels from the different image datasets may be placed in a spatial vicinity surrounding a location corresponding to the pixel location in each of the image datasets. This may be thought of as a type of 'dithering'.

"The image datasets may be representative of a number of different types of spatial information, this being governed primarily by the physical technique used to acquire the data. Typically however, each image dataset represents the distribution of a particular element or property of a material within a field of view of a corresponding image. Thus the datasets may represent 'x-ray maps' of particular elements.

"The datasets which are obtained during the initial obtaining step may be datasets which have been obtained directly from the respective apparatus used to acquire the images. They may however be generated by calculation. In the case where the apparatus produces a hyper-spectral dataset (effectively a continuous spectrum at each pixel), then the method may comprise processing the hyperspectral dataset so as to generate the N image datasets obtained in step (a).

"Once the output colour image dataset has been produced typically it is then displayed. In order to gain the maximum benefit from the displayed output colour image dataset it is preferably displayed with one or more of the image datasets, a selected number of the image datasets or each of the image datasets. This allows a direct comparison between the image datasets of the output and the contributing 'input' image datasets. Further benefit may be obtained by combining the displayed dataset with topographical data in greyscale so as to produce a displayed colour image containing topographical information. Additional techniques may also be applied such as applying image segmentation to the output colour image dataset to identify regions with substantially constant material composition.

"The method according to the invention is suitable for use with data obtained using a number of techniques and corresponding analysis equipment. Such techniques include: wavelength dispersive spectroscopy, energy dispersive spectroscopy, x-ray fluorescence analysis, electron energy loss spectroscopy, particle induced x-ray emission (PIXE), auger electron spectroscopy (AES), gamma-ray spectroscopy, secondary ion mass spectroscopy (SIMS), x-ray photoelectron spectroscopy (XPS), raman spectroscopy, magnetic resonance imaging (MRI), infra red reflectometry, and infra red spectroscopy. Each of the above techniques is typically implemented using a computer and the images are displayed and manipulated upon an attached computer monitor. The method is preferably implemented upon such a computer system although the data may be transferred to one or more remote computer systems separate from the analysis equipment using an appropriate network, including the Internet. The invention therefore includes a computer program product comprising program code configured to perform the method when said program code is executed upon a suitable computer.

BRIEF DESCRIPTION OF THE DRAWINGS

"Some examples of present invention will now be described, with reference to the accompanying drawings, in which:

"FIG. 1 shows a schematic illustration of a prior art method of generating resultant colour images using multivariate statistical analysis;

"FIG. 2 illustrates the generation of an output image formed from elemental maps;

"FIG. 3 shows a general method according to examples of the invention;

"FIG. 4 illustrates the practical effect of collinear image datasets; and,

"FIG. 5 shows various possible schematic representations of output images and their associated input images."

URL and more information on this patent application, see: Statham, Peter J. Material Identification Using Multiple Images. Filed June 6, 2011 and posted February 13, 2014. Patent URL: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=5148&p=103&f=G&l=50&d=PG01&S1=20140206.PD.&OS=PD/20140206&RS=PD/20140206

Keywords for this news article include: Emerging Technologies, Oxford Instruments Nanotechnology Tools Limited.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC


For more stories covering the world of technology, please see HispanicBusiness' Tech Channel



Source: Politics & Government Week


Story Tools