Taking the outlet section of the right-bank tailrace tunnel at the GS Hydropower Station as the engineering background, this study investigates the instability mechanism and evaluation method of surrounding rock in a thin-layered anti-dip carbonaceous slate by integrating field investigation, theoretical analysis, physical model testing, and 3DEC numerical simulation. The anti-dip layered rock mass is simplified as a superposed thin-plate system, and a mechanical model and corresponding criterion for tensile-flexural failure of rock layers are established. Physical model tests of tunnel excavation under biaxial loading are conducted for foliation dip angles of 40° and 60°, together with distributed fiber optic sensing, digital image correlation, and acoustic emission monitoring. The results reveal a progressive failure process characterized by crack initiation, damage accumulation, through-crack formation, and tensile-flexural toppling instability. Under the present test conditions, the 60° model exhibits earlier crown spalling, faster development of reverse step-shaped through-cracks along the sidewalls, and more severe block detachment than the 40° model, indicating that a larger foliation dip angle is more unfavorable to surrounding rock stability. The calculated results obtained from the proposed criterion are basically consistent with the timing of through-crack development observed in the physical model tests. The 3DEC simulations reproduce the crack initiation positions, propagation paths, and toppling failure characteristics of the surrounding rock, showing good agreement with both the physical model tests and field observations. The results indicate that the instability of thin-layered anti-dip surrounding rock is mainly controlled by the tensile-flexural effect of rock layers, while the foliation dip angle and post-excavation constraint conditions are key factors affecting stability. The study can provide a reference for stability evaluation and support optimization of hydraulic tunnels under similar geological conditions.