This paper reports the experimental results of the vortex-induced vibration of a catenary flexible riser placed in the concave orientation. The flexible riser with an aspect ratio of 125 was hung by its own weight with its top end hinged and bottom end fixed. The non-intrusive optical measurement with high-speed cameras was employed to capture the in- and out-of-plane responses simultaneously. Compared to the fixed-fixed case, the change of top boundary leads to the alterations of response amplitudes and excited mode ranges, and the influence on the in-plane response is greater than that on the out-of-plane one, presenting the pronounced traveling wave components and more frequencies during the mode transition. The coupling response between the top joint and flexible riser is examined in terms of the coincidence of the dominant vibration frequency, which is closely related to the spatial mode competition, presenting the time-varying response profile and vibration frequency as well as the spatial evolution of dominant frequency and modal weight. With increasing the depth-averaged reduced velocity, the out-of-plane mode transition is not synchronized with that of in-plane in spite of the coincidence of vibration frequency. To illustrate the spatial evolution of coupling patterns, four coupling patterns are proposed: strong coupling(+), strong coupling, weak coupling, and weak coupling(-). More violent mode competition contributes to more spatial partitions of coupling pattern.