Fractional Chern insulators versus nonmagnetic states in twisted bilayer <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>MoTe</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math>
Jiabin Yu, Jonah Herzog-Arbeitman, Minxuan Wang, Oskar Vafek, B. Andrei Bernevig, Nicolas Regnault
Abstract
Fractionally filled Chern bands with strong interactions may give rise to fractional Chern insulator (FCI) states, the zero-field analog of the fractional quantum Hall effect. Recent experiments have demonstrated the existence of FCIs in twisted bilayer ${\mathrm{MoTe}}_{2}$ without external magnetic fields---most robust at $\ensuremath{\nu}=\ensuremath{-}2/3$---as well as Chern insulators (CIs) at $\ensuremath{\nu}=\ensuremath{-}1$. Although the appearance of both of these states is theoretically natural in an interacting topological system, experiments repeatedly observe nonmagnetic (or weakly magnetic) states (lacking FCIs) at $\ensuremath{\nu}=\ensuremath{-}1/3$ and $\ensuremath{-}4/3$, a puzzling result, which has not been fully theoretically explained. In this paper, we perform Hartree-Fock and exact diagonalization calculations to test whether the standard ${\mathrm{MoTe}}_{2}$ moir\'e model with the (greatly varying) parameter values available in the literature can reproduce the nonmagnetic/weakly magnetic states at $\ensuremath{\nu}=\ensuremath{-}1/3$ and $\ensuremath{-}4/3$ in unison with the FCI at $\ensuremath{\nu}=\ensuremath{-}2/3$ and CI state at $\ensuremath{\nu}=\ensuremath{-}1$. We focus on the experimentally relevant twist angles and, crucially, include remote bands. We find that the parameters proposed in Wang et al. [arXiv:2306.02501] can nearly capture the experimental phenomena at $\ensuremath{\nu}=\ensuremath{-}1/3,\ensuremath{-}2/3,\ensuremath{-}1,\ensuremath{-}4/3$ simultaneously, although the predicted ground states at $\ensuremath{\nu}=\ensuremath{-}1/3$ are still mostly FCIs and a larger dielectric constant $\ensuremath{\epsilon}>10$ than is typical of hexagonal boron nitride (h-BN) substrate $\ensuremath{\epsilon}\ensuremath{\sim}6$ is required. Our results show the importance of remote bands in identifying the competing magnetic orders and lay the groundwork for further study of the realistic phase diagram.