⇦Chinese Physics Letters, 2017, Vol. 34, No. 7, Article code 074201 High Optical Magnification Three-Dimensional Integral Imaging of Biological Micro-organism * Dan Sun(孙聃)1**, Yao Lu(陆瑶)1, Jin-Bo Hao(郝劲波)2, Kai-Ge Wang(王凯歌)1 Affiliations 1National Center for International Research of Photoelectric Technology & Nano-functional Materials, State Key Laboratory of Cultivation Base for Photoelectric Technology and Functional Materials, Key Laboratory of Optoelectronic Technology of Shaanxi Province, Institute of Photonics & Photon-Technology, Northwest University, Xi'an 710069 2School of Science, Xi'an University of Architecture and Technology, Xi'an 710055 Received 28 December 2016 *Supported by the Scientific Research Fund Project of the Education Department of Shaanxi Province under Grant No 15JK1732, the Natural Science Foundation of Shaanxi Province under Grant No2014JQ1044, and the Science Foundation of Northwest University under Grant No 12NW01.
**Corresponding author. Email: sund@nwu.edu.cn Citation Text: Sun D, Lu Y, Hao J B and Wang K G 2017 Chin. Phys. Lett. 34 074201 Abstract A high optical magnification three-dimensional imaging system is proposed using an optic microscope whose ocular (eyepiece) is retained and the structure of the transmission mode is not destroyed. The elemental image array is captured through the micro lens array. Due to the front diffuse transmission element, each micro lens sees a slightly different spatial perspective of the scene, and a different independent image is formed in each micro lens channel. Each micro lens channel is imaged by a Fourier lens and captured by a CCD. The design translating the stage in $x$ or $y$ provides no parallax. Compared with the conventional integral imaging of micro-objects, the optical magnification of micro-objects in the proposed system can enhanced remarkably. The principle of the enhancement of the image depth is explained in detail and the experimental results are presented. DOI:10.1088/0256-307X/34/7/074201 PACS:42.82.Bq, 42.79.-e, 87.85.Pq © 2017 Chinese Physics Society Article Text Biologists often perform manual inspections for bioimage analysis. It requires a large effort and concentration. Consequently, it is difficult to analyze a huge number of bioimages and to have reliable analysis results. Although bioimage informatics has a rather long history,[1] recent progress in microscopic technologies is enhancing the research activity.[2] This can be understood by research on analyzing bioimages automatically or semi-automatically by computer.[3,4] If the techniques are accurate enough, their analysis results will be more accurate and reliable than those by manual inspection. The optical microscope is the most widely used in various fields to observe specimens, but the obtained information through the microscope via an image at once is not 3D but rather 2D. The integral imaging has been regarded as one of the most suitable techniques for next-generation 3D.[5,6] However, integral imaging cannot be directly applied to micro objects smaller than each micro lens. The use of magnifying lenses before the pickup process cannot be a solution, because magnification is nonuniform along the longitudinal depth direction, and thus the 3D shape cannot be preserved for a magnified image.[7] Using a micro lens array with lenslets much smaller than the micro-object is not a solution either, because diffraction in each micro lens becomes significant, and thus the image resolution is degraded seriously. The first microscope system using a micro lens array, which is a light field microscope, was reported by Levoy et al.[8,9] This design is inserting a micro lens array into the optical train of a conventional microscope, whose ocular is removed, and a micro lens array is placed at the intermediate image plane. An integral imaging microscope (IIM) also captures the specimen through micro lens array, but the concept is different so that a single point of the specimen can be captured through multiple elemental lenses, whereas a single point of the specimen passes through only a single corresponding elemental lens in light field microscopy.[10,11] The advantages of the integral imaging microscope technology have attracted great attention in the past decades.[12-15] For the purpose of improving resolutions[14,15] and depth of field[16] of the 3D display, many ways and methods were proposed. However, these are all based on the conventional design, in which the ocular is replaced by micro lens array or a micro lens array is placed at the intermediate image plane and the ocular is removed. Because the ocular is removed, lateral optical magnifications of these design are very difficult over 100$\times$. Recently, Lin et al.[17] proposed a novel approach for light field microscopy imaging using a camera array. In that system the structure of the optic microscope was not destroyed. However, the system is more expensive and bulkier compared with the single lenslet array based approach. In this work, we present a new design integral imaging microscope using a micro lens array. The aim is to enhance optical magnifications of the integral imaging microscope.
cpl-34-7-074201-fig1.png
Fig. 1. Optical layout of different microscopes. (a) The structure of microscope. (b) The structure of integral imaging microscope.[8] (c) The structure of our design integral imaging microscope. The black line indicates the rays from specimen. The green line indicates the rays from pupil in (b). The orange, pink and blue lines indicate the scattering rays from specimen in (c).
The basic principle of the microscope is illustrated in Fig. 1(a). The structure of the integral imaging microscope based on a micro lens array is shown in Fig. 1(b).[8] It is composed of the objective lens, a micro lens array, and an image sensor. The specimen is imaged via objective lens onto the intermediate plane, and the elemental image array is captured through the micro lens array. In the design the ocular is removed, a micro lens array is placed at the intermediate image plane, and a camera sensor is placed behind this, positioned so that each micro lens records an in-focus image of the objective. In this study, we design a new optical structure of the microscope based on a micro lens array (Fig. 1(c)). In our design the ocular is retained. The structure of the transmission mode optic microscope will not be destroyed. An illumination source is focused by a condenser lens onto a specimen. An objective lens magnifies the specimen, creating a real image at intermediate image plane. An ocular further magnifies this image, creating a second image focused at infinity. The second image through the diffuse transmission element captures 3D light field of the specimen. The elemental image array is captured through a micro lens array. The diffuse transmission element, micro lens array and imaging sensor are strictly conjugate, each micro lens sees a slightly different spatial perspective of the scene, and a different independent image is formed in each micro lens channel. Each micro lens channel is imaged by a Fourier lens and captured by a CCD.
cpl-34-7-074201-fig2.png
Fig. 2. Schematic diagram of our design system. In the design the ocular is retained. We employed an optical microscope as an amplification system. The amplification image through the diffuse transmission element, used for 3D information acquisition which is carried by the scattered beam. The elemental image array is captured through a micro lens array. Each micro lens channel is imaged by a Fourier lens and captured by a CCD.
The new design mainly consists of three parts: the amplification system (an optical microscope, Olympus IX 70), the 3D information acquisition system and the imaging system, which can be schematically mapped as shown in Fig. 2. In our experimental system, the 3D objects are a tissue slice (skeletal muscle transverse slice), whose size is 2 mm $\times$ 2 mm $\times$ 0.1 mm, and it is illuminated by microscope light source. The size of micro lens array is 6 cm $\times$ 6 cm, and it is composed of 3000 periodic self-focusing lens with 1 mm focal length and 1 mm period. The perspective elemental images arrays are recorded by a CCD (Photometrics, evolve 512) with pixel pitch 6.5 μm, operated at resolution $1392\times1040$ pixels. The microscope objective (Olympus) is 60$\times$, the microscope ocular (Olympus) is 10$\times$, and the optical magnification of the tissue slice is 600$\times$. To demonstrate the imaging performance, we capture the light field of the tissue slice (Fig. 3). The system captured the tissue slice due to its translucent property and applied a transillumination to another biological micro-organism. In the optical pickup, the elemental images of the 3D biological micro-organism are recorded by using a lenslet array and a CCD camera. The microscope ocular is 10$\times$ and the optical magnification of the tissue slice was 600$\times$. The elemental array is shown in Fig. 3(a). The size of target elemental image arrays is $13\times13$ arrays. Figures 3(b) and 3(c) show the enlarged images of the left and right regions of Fig. 3(a). As shown in Fig. 3, the maximum intensity of elemental images is marked. Result analysis of intensity distribution shows that the maximum intensities are in different elemental images. The result suggests that each elemental image records a different perspective according to its position in the array of lenses. The optical magnification was calculated by multiplying the magnificence of the objective lens and the ocular. The microscope objective is 60$\times$, the ocular is 10$\times$ and the optical magnification of the tissue slice is 600$\times$. The amplification system of the integral imaging microscope is an optical microscope. This suggests that our three-dimensional integral imaging can capture 600 magnification 3D light field in a single photograph.
cpl-34-7-074201-fig3.png
Fig. 3. Maximum intensity distribution of micro-organism elemental images (a), (b) and (c) show enlarged images of the left and right regions of (a).
We provide three advantages on the optical system comparison between our approach and the MLA as follows: Firstly, our approach can achieve high optical magnification by using an optic microscope. Although the microscope objective is a highly corrected optical subsystem capable of capturing rays that strike, once the ocular is removed, the optical magnification of the amplification system was limited compared with the optical microscope which has an ocular. The amplification system of the integral imaging microscope was an optical microscope. Integral imaging microscope captures light fields instead of images. If the ocular is removed, optical magnifications of a conventional microscope are very difficult over 100$\times$. This means that if the ocular is removed, optical magnifications of the integral imaging microscope are very difficult over 100$\times$. In our design, the ocular is retained and the structure of the transmission-mode is not destroyed, and its optical magnification is the same as the optic microscope, which could reach 600$\times$. Secondly, our approach can improve imaging quality and can provide no parallax in translating the stage in $x$ or $y$. Because the ocular is retained, the structure of the transmission mode light microscope will not be destroyed in our design. Thus compared with the conventional integral imaging microscope our design can provide no parallax in translating the stage in $x$ or $y$. For a conventional integral imaging microscope placing our micro lens array at the intermediate image plane, it should cover the image created there by the objective, while only the innermost is well-corrected for aberrations. In our design, the image is created there by the optic microscope, the ocular is retained, and the structure of the transmission-mode is not destroyed. The complete image is well-corrected for aberrations. In a normal microscopy, we use spatial information only. Thus the number of resolvable spots in the image is the same as the number of samples as just defined. If we place a micro lens array at the image plane, we can preserve some of the directional information, but only by making each micro lens capture a number of samples.[18,19] In our design, the diffuse transmission element is used for 3D information acquisition which is carried by the scattered beam and the elemental image array is captured through the micro lens array. This design makes the micro lens capture more samples. In addition, we use diffusers to approximate the desired condition in our prototype. Given a spatially nonuniform light source such as an incandescent bulb or arc lamp, it is possible by adjusting the height of the microscope sub stage condenser to force the illumination to be spatially uniform at the specimen. Thirdly, our approach can pick up the independent image. Conventional integral imaging microscope is different so that a single point of the specimen can be captured through multiple elemental lenses, whereas a single point of the specimen passes through only a single corresponding elemental lens in light field microscopy. Compared with it, in our system each micro lens sees a slightly different spatial perspective of the scene, and a different independent image is formed in each micro lens channel. Each micro lens channel is imaged. Although the system has three advantages, it is necessary to point out that the design can produce orthographic views with a shallow depth of field but its spatial resolution is lower. With the proposal of using lens arrays to capture light fields, the imaging system sacrifices spatial resolution to obtain angular resolution. We synthetically focus on enhancing the optical magnification of 3D light field of the specimen. Although the design sacrifices the spatial resolution relative to an ordinary microscope, they do not sacrifice light-gathering ability. In conclusion, we have presented a new three-dimensional integral imaging system for biological micro-organisms. Our experiment shows that 3D biological micro-organisms can be magnified 600$\times$ and the imaging can be pickup by the system. Our method can assist physicians, biologists and scientists in perceiving the 3D structure of biological micro-organisms more accurately. References An automatic computer vision system for blood analysisNIH Image to ImageJ: 25 years of image analysisComputer Vision in Cell BiologyPattern Recognition Software and Techniques for Biological Image AnalysisDepth-enhanced integral imaging system with convex and composite concave micro-lens arraysPerformance analysis of three-dimensional integral imaging systems based on human visionThree-dimensional integral imaging with large depth of focus by use of real and virtual image fieldsThe halfway vector disk for BRDF modelingRecording and controlling the 4D light field in a microscope using microlens arraysThree-dimensional integral imaging of micro-objectsThree-dimensional identification of biological microorganism using integral imagingReal-time integral imaging system for light field microscopyReal-time interactive display for integral imaging microscopyResolution-enhancement for an orthographic-view image display in an integral imaging microscope systemResolution-enhanced integral imaging microscopy that uses lens array shiftingAnalysis on enhanced depth of field for integral imaging microscopeCamera array based light field microscopy3D imaging of biofilms on implants by detection of scattered light with a scanning laser optical tomograph
[1] Parthenis K, Metaxaki-Kossionides C and Dimitriadi B 1990 Microprocess Microprogramm 28 243
[2] Schneider C A, Rasband W S and Eliceiri K W 2012 Nat. Methods 9 671
[3] Danuser G 2011 Cell 147 973
[4] Shamir L, Delaney J D, Orlov N, Eckley D D and Goldberg I G 2010 PLoS. Comput. Biol. 6 e1000974
[5] Xie W, Wang Q H, Wang Y Z and Deng H 2014 Optik 125 6087
[6] Wang F, Zhao X, Yang Y, Fang Z L and Yuan X C 2013 Optik 124 134
[7] Jang J S, Jin F and Javidi B 2003 Opt. Lett. 28 1421
[8] Edwards D, Boulos S, Johnson J, Shirley P, Ashikhmin M, Stark M and Wyman C 2006 ACM Trans. Graphics 25 1
[9] Levoy M, Zhang Z and McDowall I 2009 J. Microsc. 235 144
[10] Jang J S and Javidi B 2004 Opt. Lett. 29 1230
[11] Javidi B, Moon I and Yeom S 2006 Opt. Express 14 12096
[12] Kim J, Jung J H, Jeong Y G, Hong K and Lee B 2014 Opt. Express 22 10210
[13] Kwon K C, Jeong J S, Erdenebat M U, Lim Y T, Yoo K H and Kim N 2014 Appl. Opt. 53 4450
[14] Kwon K C, Jeong J S, Erdenebat M U, Piao Y L, Yoo K H and Kim N 2015 Biomed. Opt. Express 6 736
[15] Lim Y T, Park J H, Kwon K C and Kim N 2009 Opt. Express 17 19253
[16] Lim Y T, Park J H, Kwon K C and Kim N 2012 Opt. Express 20 23480
[17] Lin X, Wu J M, Zheng G A and Dai Q H 2015 Biomed. Opt. Express 6 3179
[18]Wang Q H, Zhang Y Y, Lai J C, Li Z H and He A Z 2007 Acta Phys. Sin. 56 1203 (in Chinese)
[19] Heidrich M, Kühnel M P, Kellner M, Lorbeer R A, Lange T, Winkel A, Stiesch M, Meyer H and Heisterkamp A 2011 Biomed. Opt. Express 2 2982