Emergence of a Helical Metal in Rippled Ultrathin Topological Insulator Sb extsubscript{2}Te extsubscript{3} on Graphene

The integration of topological insulators (TIs) with graphene offers a pathway to engineer hybrid quantum states, yet the impact of strain at the 2D limit remains a critical open question. Here, we investigate the structural properties of ultrathin (1 quintuple layer) Sb$_2$Te$_3$ grown on single-layer graphene and, motivated by the structural modulations observed at the TI surface, explore theoretically how such nanoscale corrugations may influence the electronic behavior of the system. Using low-temperature scanning tunneling microscopy (LT-STM), we observe a periodic rippling of the heterostructure with a wavelength of ~$\sim8.7$ nm. Energetic analysis reveals that these ripples are not intrinsic but are driven by strain from the substrate during cooling. Density functional theory (DFT) calculations show that while the ideal flat heterostructure exhibits a hybridization gap of $\sim40$ meV, the ripple-induced structural modulation closes this gap, restoring a metallic state. This gapless phase is not a trivial metal. By combining an effective moiré ladder model with spin-resolved DFT, we find that the proximity-induced spin-orbit coupling is redistributed across a dense manifold of minibands. The resulting ``Helical Metal’’ has a complex spin-texture beyond a simple Rashba splitting. Remarkably, while the flat system is effectively spinless in this ultrathin limit due to hybridization, the ripples actively restore the spin polarization. Our findings suggest that rippled TI/graphene heterostructures provide an interesting platform to develop spintronics, where geometric modulation unlocks dense helical states that are inaccessible in the pristine flat limit.

💡 Research Summary

The authors investigate a heterostructure composed of a single‑quintuple‑layer (1 QL) Sb₂Te₃ topological insulator (TI) grown on monolayer graphene. Using low‑temperature scanning tunneling microscopy (LT‑STM), they observe a periodic ripple pattern on the surface with a wavelength of approximately 8.7 nm and an out‑of‑plane amplitude of about 0.6 nm. A detailed angle‑dependent spatial correlation analysis reveals that the ripples form stripe‑like domains aligned along a preferred direction, consistent with a strain‑induced buckling of the underlying graphene substrate. Energetic calculations show that the van‑der‑Waals binding between Sb₂Te₃ and graphene (≈0.23 eV per cell) exceeds the estimated bending energy of the rippled stack (≈0.20 eV per stripe), ruling out delamination. The energy penalties for lattice sliding or intrinsic strain are negligible (<0.01 eV), and phase‑separated SbₓTe₁₋ₓ domains are energetically unfavorable. Molecular‑dynamics simulations of a freestanding system do not produce spontaneous stripe formation, leading to the conclusion that the ripples are driven by external thermal‑strain released during cooling of the graphene/SiO₂ substrate. Graphene’s negative thermal expansion coefficient causes compressive strain upon cooling, amplifying pre‑existing wrinkles into a regular ripple superlattice that the Sb₂Te₃ monolayer conforms to.

First‑principles density functional theory (DFT) calculations are performed for two structural models. In the flat heterostructure, a √3 × √3 graphene supercell is matched to a 1 QL Sb₂Te₃ cell (lattice mismatch 0.35 %). The folding of graphene’s K and K′ valleys to Γ leads to eight nearly degenerate graphene bands that hybridize strongly with Sb₂Te₃ surface states. This Dirac‑Dirac resonance opens two gaps of roughly 10 meV and 40 meV at the Fermi level, effectively turning the system into a gapped semiconductor. The gap disappears when spin‑orbit coupling (SOC) is omitted, confirming its SOC‑driven nature.

To capture the effect of the experimentally observed ripples, the authors construct an orthorhombic supercell whose long axis reproduces the 8.7 nm periodicity. The graphene atoms are forced to follow a cosine out‑of‑plane displacement while all other degrees of freedom are relaxed. The resulting band structure, folded into the mini‑Brillouin zone of the ripple superlattice, shows that the Dirac cone is shifted away from the high‑symmetry Γ point and becomes essentially gapless (<1 meV). The shift can be interpreted as a pseudo‑vector potential A generated by the strain field, which acts as a gauge field moving the Dirac point without opening a gap, as known from strained graphene. However, unlike bare graphene, the shifted Dirac cone remains hybridized with Sb₂Te₃ surface states, and the symmetry breaking introduced by the ripple destroys the specific hybridization condition that produced the 40 meV gap in the flat case. Consequently, the system transitions from a gapped to a metallic state, but the metallicity is not trivial.

To elucidate the spin character of the ripple‑induced metallic phase, the authors develop an effective moiré ladder model. The periodic modulation creates a dense manifold of minibands. Within this manifold, spin‑momentum locking emerges across many bands, producing a complex helical spin texture that cannot be described by a simple Rashba splitting. In the flat 1 QL heterostructure, hybridization between top and bottom TI surfaces eliminates spin polarization, rendering the system effectively spinless. The ripples, by breaking the precise alignment required for that hybridization, restore spin polarization and generate a “helical metal” where charge carriers possess a well‑defined helicity.

The work demonstrates that geometric modulation at the nanoscale can unlock dense helical states inaccessible in the pristine flat limit. This finding opens a pathway toward strain‑engineered spintronic devices based on TI/graphene heterostructures, where the electronic topology and spin texture can be tuned by controlling ripple wavelength, amplitude, or orientation. The authors suggest that such systems could serve as platforms for high‑efficiency spin transport, strain‑controlled quantum phase manipulation, and possibly for realizing exotic quasiparticles in two‑dimensional materials.