We elucidate the magnetic interactions and the role of spin (electron) correlation in determining the ground state of the honeycomb compound Mn4Ta2O9, by neutron powder diffraction, inelastic neutron scattering (INS), specific-heat (C-P) measurements, and electronic-structure calculations. The antiferromagnetic long-range order with moments along c occurs at 102 K with strong exchange striction and small anisotropy. It is described by the three-dimensional Ising model. Diffuse magnetic scattering has been observed above T-N, which is attributed to the two-dimensional spin correlations within the Mn2+ honeycombs. This is confirmed by the calculated exchange constants. INS experiments and spin-wave simulations together with C-P measurements reveal two gapped modes on the ab plane, originating from the rotation of the spins away from the easy axis c. The magnetic anisotropy is mainly determined by an electron-correlation-assisted dipole-dipole interaction. This work provides insight into the competing origins of the magnetic anisotropy, which leads to different magnetic ground states in the family of honeycomb compounds.

N. N. and Y. L. acknowledge the support of the Australian Research Council (ARC) in the form of Discovery Project (Project No. DP1601104780). N.N., Y.L., D.Y., and G.M. thank ANSTO for the allocation of neutron beam time on PELICAN (P6698) as well as financial support. N.N. and H.F. thank MLZ for the allocation of neutron beam time on SPODI (ID: 11618) and the HHLR Darmstadt for the allocation of computer time through Project No. 611.