The scaled boundary finite element method (SBFEM) coupled with the finite element method (FEM) for the simulation of the interaction problem between the elastic plate structure and the multi-layered unbounded elastic soil is first developed in this paper. First, the whole system is subdivided into three sub-domains, including the semi-infinite far-field system, the near-field sub-domain, and the plate structure. The far field of the soil is modeled by utilizing the modified scaled boundary finite element method with a scaling line instead of the scaling center used in the traditional SBFEM. In the traditional SBFEM, only the boundary is discretized with surface elements, so the spatial dimension is reduced by one, and the final governing equation can be solved analytically in the radial direction of the scaled coordinate system, and it can exactly meet the infinite domain problem. The stiffness matrix of the three-dimensional (3D) near field is obtained using the standard FEM. The SBFEM is also applied in order to simulate the deformation characteristics of the plate structure based on the 3D elastic equation without introducing any assumption of the thin plate theory, and the high-order spectral element is introduced in order to discretize the middle surface of plate so that the complicated curved boundaries can be better represented. Then, according to the principle of the degree of freedom matching at the same node, the global stiffness matrix of the plate-soil system can be obtained by coupling the stiffness matrices of the subdomains at the far-field/near-field interface, as well as at the plate/near-field interface. Thus, the response of the whole system under the external load can be solved naturally. Four numerical examples, consisting of a square plate resting on an isotropic soil, multi-layered soil with weak and thin interlayer, a plate with a different geometrical shape and a square plate with a hole, are provided in order to validate the accuracy and versatility of the proposed formulations.