⇦Chinese Physics Letters, 2021, Vol. 38, No. 5, Article code 057307Express Letter Modification of the Hybridization Gap by Twisted Stacking of Quintuple Layers in a Three-Dimensional Topological Insulator Thin Film Changyuan Zhou (周长远), Dezhi Song (宋德志), Yeping Jiang (蒋烨平)*, and Jun Zhang (仉君) Affiliations Key Laboratory of Polar Materials and Devices, Department of Electronics, School of Physics and Electronic Science, East China Normal University, Shanghai 200241, China Received 25 March 2021; accepted 7 April 2021; published online 10 April 2021 Supported by the National Natural Science Foundation of China (Grant Nos. 61804056 and 92065102).
*Corresponding author. Email: ypjiang@clpm.ecnu.edu.cn Citation Text: Zhou Z Y, Song D Z, Jiang Y P, and Zhang J 2021 Chin. Phys. Lett. 38 057307    Abstract Twisting the stacking of layered materials leads to rich new physics. A three-dimensional topological insulator film hosts two-dimensional gapless Dirac electrons on top and bottom surfaces, which, when the film is below some critical thickness, will hybridize and open a gap in the surface state structure. The hybridization gap can be tuned by various parameters such as film thickness and inversion symmetry, according to the literature. The three-dimensional strong topological insulator Bi(Sb)Se(Te) family has layered structures composed of quintuple layers (QLs) stacked together by van der Waals interaction. Here we successfully grow twistedly stacked Sb$_{2}$Te$_{3}$ QLs and investigate the effect of twist angels on the hybridization gaps below the thickness limit. It is found that the hybridization gap can be tuned for films of three QLs, which may lead to quantum spin Hall states. Signatures of gap-closing are found in 3-QL films. The successful in situ application of this approach opens a new route to search for exotic physics in topological insulators. DOI:10.1088/0256-307X/38/5/057307 © 2021 Chinese Physics Society Article Text The class of three-dimensional (3D) strong topological insulators (TIs) such as Bi$_{2}$Se$_{3}$, Bi$_{2}$Te$_{3}$, and Sb$_{2}$Te$_{3}$ hosts single Dirac cone in the surface state structure on the surface due to the non-trivial bulk band topology.[1–7] In the thin film form, the two-dimensional (2D) massless Dirac electrons on the top and bottom surfaces will hybridize and open a gap below the critical thickness.[3,8–12] These materials have layered structures composed of quintuple layers (QLs) of atoms. There is only van der Waals interaction between the QLs so that the film thickness is highly controllable. Theoretically, the gap shows oscillatory behavior between an inverted and a normal gap parity with film thickness near the thickness limit,[10] which provides a possibility of realizing the quantum spin Hall effect (QSH) in the ultra-thin 3D TIs. In addition, the gap value or parity can be tuned by inversion-symmetry-broken parameters such as electric fields normal to the surface.[13–15] The gap-opening behavior was investigated theoretically and experimentally in ultra-thin Bi$_{2}$Se$_{3}$ and Sb$_{2}$Te$_{3}$ films.[3,8] Nonetheless, up to now, QSH has not been realized in 3D TI systems. Experimentally, materials with layered structures can be cleaved into a-few-layer or even into monolayer films, which can then be stacked together in a controllable manner to get twisted structures with different angles. In graphene the twisted structures at some magic angles drive the electronics into the superconducting or even the strongly correlated regime.[16] Here we present an in situ approach to obtain twisted structures of TI QLs. By molecular beam epitaxial deposition and fine-tuning the growth parameters, we obtain structures with different twist angles between QLs in Sb$_{2}$Te$_{3}$ films below the thickness limit. From the scanning tunneling microscopy (STM) and spectroscopy (STS), we find that the hybridization gap is highly tunable by constructing QLs with different twist angles. We obtain a much wider range of gap-tuning compared to the electric field approach, thus providing a new approach for the search of new QSH systems or even new exotic physics beyond the TI regime in this material. Molecular beam epitaxy (MBE), mostly known as a tool for the growth of ultra-thin film to achieve higher quality, is seldom conducted in a way using its another inherent feature. That is, the growth dynamics of films by MBE is highly tunable to achieve non-equilibrium growth by changing the crucial parameters such as substrate roughness, lattice mismatch, substrate temperature, flux and flux ratio. While the MBE growth of high quality films of Bi(Sb)Se(Te) family is well established and now routine,[17–21] the attempt to extremize the non-equilibrium growth conditions to get films with structures much different from the equilibrium and normal states is rare. The thickness limit for Sb$_{2}$Te$_{3}$ being a 3D TI is around 4 QLs, below which a hybridization gap opens in the surface states.[3] In this study, we mainly focus on films below 4 QLs. The 3-QL film is most important because of its small intrinsic hybridization gap ($\sim $60 meV on graphene).[3] We grow ultra-thin Sb$_{2}$Te$_{3}$ films with a total coverage of 1–3 QLs on SrTiO$_{3}$(111) substrate in a two-step manner. The substrate surface has a roughness of $\sim $200 Å (peak to peak) and terrace widths of $\sim $80 nm after annealed in vacuum prior to the growth. Under normal conditions, such surface is relatively rough for the epitaxial growth. In the case of Bi(Sb)Se(Te) family, because of the QL structures,[6] the interaction between films and the under-lying substrate is van der Waals in nature if the substrate is inert. We find that the 1-QL film on the STO(111) substrate is freestanding and there is no interfacial buffer layer. In the two-step procedure, the substrate temperature is kept between 200℃ and 220℃. We first grow a 1-QL film at a relatively low temperature and perform the second-step deposition at a higher temperature without post annealing with a total coverage of about 2–3 QLs. Figure 1(a) shows the STM topographic image of a 3-QL Sb$_{2}$Te$_{3}$ film. The terraces in the figure are from the substrate (step height $\sim $2.3 Å), and the film is continuous across the steps of substrate as observed from high resolution images (not shown). From the topographic image we can see some periodic patterns with different periodicities, which are more clearly shown in the differential image [right image in Fig. 1(a)].
cpl-38-5-057307-fig1.png
Fig. 1. (a) STM image and the corresponding differential image (3.0 V, 10 pA) of a 3-QL Sb$_{2}$Te$_{3}$ film. The substrate is STO(111). (b) Derivative STM images of an Sb$_{2}$Te$_{3}$ film showing two different domain boundaries DW1 and DW2. The arrows indicate the directions of the Sb$_{2}$Te$_{3}$ crystal structure projected onto the surface brillouin zone. (c) A derivative STM image of an Sb$_{2}$Te$_{3}$ film (2.6 QLs) showing the crystal orientations of domains in 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs. (d) A schematic layout of the domain structures in the Sb$_{2}$Te$_{3}$ film of 2.6 QLs. All STM and STS measurements are taken at 5.6 K.
These periodic patterns are moiré patterns formed in the presence of twist angles between adjacent QLs. To clarify their origin in the 3-QL film, we also grow a film with a coverage of $\sim $2.6 QLs. As shown in Fig. 1(c), the 2.6-QL film composes of a complete 2-QL region in addition to some 3-QL patches. By inspecting the moiré patterns across the steps between 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs (not shown), we find that all the moiré patterns are from the first two QLs. That is, the patterns on the 3$^{\rm rd}$ QL are from the bottom two QLs. In Sb$_{2}$Te$_{3}$ films, the clover-shaped defect pattern can be used to identify the crystal direction of the material, with the tips of the clover pointing to the directions.[17] As shown in Fig. 1(b), there are two kinds of domain boundaries in Sb$_{2}$Te$_{3}$ films, DW1 and DW2. Here, DW1 (dashed-line-like ones) have small mismatch angles between domains across the boundaries, and the underlying crystal structure is continuous. On the contrary, the appearance of DW2 indicates a very large mismatch in crystal orientations between domains. In this case, the lattices across such boundaries can no longer match together and true domain boundaries form. By inspecting the defects, most domains in the 2$^{\rm nd}$ QL are found to be aligned approximately in the same direction. For the 3$^{\rm rd}$ QL, there are 4 dominant directions with respect to the 2$^{\rm nd}$ QL, perpendicular or parallel to the 2$^{\rm nd}$ QL as indicated in Fig. 1(c). In this four configurations between 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs, there is no clear moiré pattern, probably because the period is small and the spatial modulation in the electronic states is weak when the twist angle is large. Accordingly, the periodic patterns are moiré patterns formed in the presence of twist angles between the first two QLs. The presence of twist angle is caused by different growth dynamics between 1$^{\rm st}$ and 2$^{\rm nd}$ QLs (or 3$^{\rm rd}$ QL) in the presence of different underlying platforms for the growth. First, a rough substrate leads to reduced mobility of Sb and Te atoms or molecules bombarded onto the surface. The reduced substrate temperature during growth further magnifies this effect, leading to an increased nucleation density. Second, because of the large lattice mismatch between Sb$_{2}$Te$_{3}$ and STO(111) surface, domains originated from different nucleation centers normally have different crystal orientations. The growth of 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs during the second step basically flows a layer-by-layer mode. In this mode, nucleation for the next QL will not happen until the present QL is nearly complete. This is clearly seen in Fig. 1(a), where the nucleation of the 4$^{\rm th}$ QL appears on a domain boundary of 3$^{\rm rd}$ QL which is nearly completed. Furthermore, the step flow mode normally leads to a QL without domains, contrary to our observation. In the differential image of Fig. 1(c), we clearly see domains in the 2$^{\rm nd}$ QL. The domains with DW1 are probably caused by the local twist of the 2$^{\rm nd}$ QL in the presence of randomly-orientated domains in the 1$^{\rm st}$ QL. In Fig. 1(c), we can see in the 2$^{\rm nd}$ QL domains with different moiré patterns and even different moiré patterns in a single domain. Thus, the structures of twisted QLs in our ultra-thin Sb$_{2}$Te$_{3}$ film is concluded as follows. The domains in the 1$^{\rm st}$ QL are small and along different directions. The domains in the 2$^{\rm nd}$ QL are larger and along approximately the same direction. In the 3$^{\rm rd}$ QL, the domains are aligned perpendicularly to or parallel to the 2$^{\rm nd}$ QL. All these inevitably lead to the situation sketched in Fig. 1(d), indicating the formation of regions with random twist-angles between the first two Sb$_{2}$Te$_{3}$ QLs and four configurations between the 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs. A simple simulation of moiré patterns is shown in Fig. 2. There are three different positions in the unit-cell of Sb$_{2}$Te$_{3}$ QL. Figure 2(a) shows the moiré pattern corresponding to an angle of $\sim$$5^{\circ}$. The periods coming from different angles can be fitted by a simple relation $A=a/\sin (\theta /2)$, where $\theta$ is the twist angle. By such a relation, we can then assign twist angles to the first two QLs. In addition, the moiré patterns on the 3$^{\rm rd}$ QL in Fig. 1(a) all come from those between 1$^{\rm st}$ and 2$^{\rm nd}$ QLs. Thus we can do statistics on the moiré patterns between the underlying 1$^{\rm st}$ and 2$^{\rm nd}$ QLs even in the presence of the 3$^{\rm rd}$ QL.
cpl-38-5-057307-fig2.png
Fig. 2. (a) Moiré pattern simulation ($\sim$$5^{\circ}$ mismatch). The inset shows the surface primary cell of 1-QL Sb$_{2}$Te$_{3}$. The brown, cyan, purple dots show the positions of Te1 and Sb2, Sb1 and Te3, Te2 projected onto the surface, respectively. The atoms are numbered from the surface. (b) Simulated moiré pattern periods for a 2-QL Sb$_{2}$Te$_{3}$ film with different twist angels. The red curve is the fitting to the data.
When a 3D TI is reduced to some critical thickness, a hybridization gap forms in the surface state structure [Fig. 3(a)].[9,10] For the 1-QL film, we can see a clear gap-like structure in the STS. The gap is measured to be about 1000 meV [Fig. 3(b)], much larger than that grown on the graphene substrate ($\sim $700 meV)[3] probably because of the strain in the film grown on the STO substrate. Intuitively, when there is a twist between QLs, the overlap of electron wavefunctions in the van-der-Waals gap between QLs changes. The hybridization between top and bottom surfaces may change accordingly. Thus, by measuring the STS on twisted 2-QL and 3-QL films, we can investigate the effect of twist-angle on the hybridization gap.
cpl-38-5-057307-fig3.png
Fig. 3. (a) A simple diagram showing the evolution of top and bottom surface states. A hybridization gap opens in the surface states when the film is thin enough. (b) A typical STS of 1-QL Sb$_{2}$Te$_{3}$ films. The numbers indicate the gap values. (c) STS of 2-QL Sb$_{2}$Te$_{3}$ films showing different hybridization gap in different domains.
For the 2-QL film, the gap indeed varies from domain to domain. The gap is solid with sharp edges [Fig. 3(c)] and are uniform inside the domains, ruling out other extrinsic effects. The gap values range from $\sim $180 meV to $\sim $230 meV. For the 2-QL film on graphene, the gap is $\sim $230 meV,[3] which can be taken as the intrinsic value because the weak interaction between the film and the graphene substrate. In Fig. 4(a), we summarize the values of measured hybridization gaps of differently twisted 2-QL and 3-QL films. Nonetheless, for the 2-QL film, there is no clear angle-dependent pattern. There is probably another factor, such as the strain effect, coming into play dominantly in this situation. The 3-QL film is our focus now. Here, 60 meV is the gap value of the 3-QL film grown on the graphene substrate, which is taken as the intrinsic value for the 3-QL film of this material.[3] It is easier to modulate a small gap to achieve a parity change indicated by the gap-closing phenomenon. Thus, we try to investigate the possible topological phase transition in the 3-QL film. We describe the twist configuration by two angles in the form $\alpha +\beta$, where $\alpha$ is the twist angle between the 1$^{\rm st}$ and 2$^{\rm nd}$ QLs and $\beta$ is that between the upper two QLs. There is also no clear dependence of gap values on the twist-angle $\alpha$ between the 1$^{\rm st}$ and 2$^{\rm nd}$ QLs. Nevertheless, the gap values can be divided into two categories by the angle $\beta$ [the lower two shaded regions in Fig. 4(a)]. Those gaps around 100 meV are all measured on domains where the 3$^{\rm rd}$ QL is aligned anti-parallel to the 2$^{\rm nd}$ QL ($\beta =180^{\circ}$). Those around 60 meV are measured on domains where the upper 2 QLs are in other three configurations, that is, the two perpendicular directions and the parallel directions ($\beta =0^{\circ}$, $\pm 90^{\circ}$). Therefore, the anti-parallel stacking between the 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs plays a dominant role here. This kind of stacking is shown in Fig. 4(b). For the $\alpha +\beta$ (0$^{\circ}$, $\pm 90^{\circ}$) configurations, the gaps range from 35 meV to 70 meV, fluctuating around the intrinsic value of 60 meV. For the $\alpha +\beta$ (180$^{\circ}$) configurations, the gaps vary from $\sim $80 meV to $\sim $110 meV, deviating much from the intrinsic one.
cpl-38-5-057307-fig4.png
Fig. 4. (a) Statistics of gap values in 2- and 3-QL films. The dashed lines indicate the gap values for 2- and 3-QL films grown on graphene substrates. (b) A derivative STM image showing the flipped crystal orientations between 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs. (c) STS of 3-QL Sb$_{2}$Te$_{3}$ films showing different hybridization gaps in different regions. The insets are the corresponding log-scaled plots. (d) Typical STS for 3-QL films with different gaps in a larger energy scale. The inset shows the gapless-like spectrum (light blue) in a smaller energy range.
In the $\alpha +\beta$ (180$^{\circ}$) cases, we always see large gaps around 100 meV. This means that the 3-QL film of Sb$_{2}$Te$_{3}$ is driven into a large-gap system compared to the intrinsic situation in this case. On the contrary, when $\beta = 0^{\circ}$ or $\pm 90^{\circ}$, the gaps keep small and are highly tunable and may reach zero in the presence of other parameters such as the angle $\alpha$ and possible strains in the film. Figures 4(c) and 4(d) show some typical spectra in two different energy ranges with different gap values. Besides the regions with solid gaps in STS, we do see gapless features [light-blue curves in Fig. 4(d)] in some domains belonging to the $\alpha +\beta$ (0$^{\circ}$, $\pm 90^{\circ}$) cases. These possible gapless states are not plotted in the statistics [Fig. 4(a)] because the bare STS is not the solid evidence. Indeed, in the 3-quintuple-layer (QL) structures of Sb$_{2}$Te$_{3}$ films, the only clear conclusion we can draw from the data is that the antiparallel stacking between the QLs will drive the system into the large-gap region. We do not see any regular pattern on how the small twist angle between the first-two QLs changes the hybridization gap. Although the gap does change from domain to domain with different twist angles, there seems no correspondence between them. One possibility is that there may be strain in the film, because the gap ($\sim $1000 meV) of the 1-QL film is much larger than that ($\sim $700 meV) grown on the graphene substrate. The strain, if there exists, is uniform inside one particular domain because the gap shows no variation inside the domain. Another possibility is that except for the twist there may exist another mismatch between the lattices of the first two QLs, for example, the 2nd QL may slide a little bit away from the thermodynamic equilibrium position. That is, there may be translational mismatch between the two QLs besides the rotational one. The 1$^{\rm st}$ QL is probably considerably strained on the substrate, while the interaction between the QLs is much weaker. This implies a relatively free registration of the 2$^{\rm nd}$-QL lattice with respect to that of the 1$^{\rm st}$ one. Currently we do not have enough evidence to support this scenario. According to the literature, the hybridization gap can be modulated by a perpendicular electric field.[14,22] However, up to now no clear evidence of gap-closing signature has been discovered in this way. The gap-closing phenomenon is accompanied by the gap-parity change, which indicates the transition between a normal 2D insulator and a 2D TI. In the case of Sb$_{2}$Te$_{3}$, the intrinsic gaps of 1, 2 and 3 QLs are measured to be around 700, 230 and 60 meV. As predicted theoretically, the gaps behave in an oscillatory manner with thicknesses.[10] The 3-QL Sb$_{2}$Te$_{3}$ film with a small gap here proves to be a tunable system in the presence of twist angles and possible strain. The possible gap-closing here indicates possible topological phase transition in the 3-QL film, regardless of whether the intrinsic film belongs to the non-trivial case or not. The unambiguous attribution of the gapless features shown in Fig. 4 to the gap-closing calls for further study. In conclusion, we have successfully grown twisted structures of 2-QL and 3-QL Sb$_{2}$Te$_{3}$ films by regulating MBE parameters and extremizing the non-equilibrium growth mode. The resulting twisted structures are mainly divided into two categories, one with anti-parallel alignment and the other with parallel or perpendicular alignment between the 2$^{\rm nd}$ and 3$^{\rm rd}$ QLs, respectively. The first case drives the system into a large-gap regime, while the second case retains the small gap of the 3-QL film, in which the gap can be modified and possible signature of gap-closing is presented here. Our approach sheds new light on the search of quantum spin Hall materials especially in the 3D TI systems. References Colloquium : Topological insulatorsExperimental Realization of a Three-Dimensional Topological Insulator, Bi2Te3Landau Quantization and the Thickness Limit of Topological Insulator Thin Films of Sb 2 Te 3 Observation of a large-gap topological-insulator class with a single Dirac cone on the surfaceTopological insulators and superconductorsTopological insulators in Bi2Se3, Bi2Te3 and Sb2Te3 with a single Dirac cone on the surfaceTopological Insulators in Three DimensionsCrossover of the three-dimensional topological insulator Bi2Se3 to the two-dimensional limitEffective continuous model for surface states and thin films of three-dimensional topological insulatorsOscillatory crossover from two-dimensional to three-dimensional topological insulatorsQuantum Spin Hall Effect and Topological Phase Transition in HgTe Quantum WellsQuantum Spin Hall Insulator State in HgTe Quantum WellsExchange field-induced topological phase transition in ultrathin films of three-dimension topological insulatorsTopological quantum phase transitions driven by external electric fields in Sb2Te3 thin filmsTuning phase transition between quantum spin Hall and ordinary insulating phasesUnconventional superconductivity in magic-angle graphene superlatticesFermi-Level Tuning of Epitaxial Sb 2 Te 3 Thin Films on Graphene by Regulating Intrinsic Defects and Substrate Transfer DopingTopological Insulator Thin Films of Bi2Te3 with Controlled Electronic StructureTopological insulator Bi2Se3 thin films grown on double-layer graphene by molecular beam epitaxyQuintuple-layer epitaxy of thin films of topological insulator Bi2Se3Intrinsic Topological Insulator Bi2Te3 Thin Films on Si and Their Thickness LimitElectric-Field Tuning of the Surface Band Structure of Topological Insulator Sb 2 Te 3 Thin Films
[1] Hasan M Z and Kane C L 2010 Rev. Mod. Phys. 82 3045
[2] Chen Y L et al. 2009 Science 325 178
[3] Jiang Y et al. 2012 Phys. Rev. Lett. 108 016401
[4] Xia Y et al. 2009 Nat. Phys. 5 398
[5] Qi X L and Zhang S C 2011 Rev. Mod. Phys. 83 1057
[6] Zhang H, Liu C X, Qi X L, Dai X, Fang Z, and Zhang S C 2009 Nat. Phys. 5 438
[7] Fu L, Kane C L, and Mele E J 2007 Phys. Rev. Lett. 98 106803
[8] Zhang Y et al. 2010 Nat. Phys. 6 584
[9] Shan W Y, Lu H Z, and Shen S Q 2010 New J. Phys. 12 043048
[10] Liu C X, Zhang H, Yan B, Qi X L, Frauenheim T, Dai X, Fang Z, and Zhang S C 2010 Phys. Rev. B 81 041307
[11] Bernevig B A, Hughes T L, and Zhang S C 2006 Science 314 1757
[12] König M, Wiedmann S, Brüne C, Roth A, Buhmann H, Molenkamp L W, Qi X L, and Zhang S C 2007 Science 318 766
[13] Yang Y, Xu Z, Sheng L, Shen R, Wang B G, and Xing D Y 2012 Europhys. Lett. 100 27005
[14] Kim M, Kim C H, Kim H S, and Ihm J 2012 Proc. Natl. Acad. Sci. USA 109 671
[15] Murakami S, Iso S, Avishai Y, Onoda M, and Nagaosa N 2007 Phys. Rev. B 76 205304
[16] Cao Y, Fatemi V, Fang S, Watanabe K, Taniguchi T, Kaxiras E, and Jarillo-Herrero P 2018 Nature 556 43
[17] Jiang Y et al. 2012 Phys. Rev. Lett. 108 066809
[18] Wang G et al. 2011 Adv. Mater. 23 2929
[19] Song C L et al. 2010 Appl. Phys. Lett. 97 143118
[20] Zhang G et al. 2009 Appl. Phys. Lett. 95 053114
[21] Li Y Y et al. 2010 Adv. Mater. 22 4002
[22] Zhang T, Ha J, Levy N, Kuk Y, and Stroscio J 2013 Phys. Rev. Lett. 111 056803