Buoyant marine plastic debris has become a serious problem affecting the marine environment. To fully understand the impact of this problem, it is important to understand the dynamics of buoyant debris in the ocean. Buoyant debris accumulates in “garbage patches” in each of the subtropical ocean basins because of Ekman convergence and associated downwelling at subtropical latitudes. However, the precise dynamics of the garbage patches are not well understood. This is especially true in the southern Indian Ocean (SIO), where observations are inconclusive about the existence and numerical models predict inconsistent locations of the SIO garbage patch. In addition, the oceanic and atmospheric dynamics in the SIO are very different from those in the other oceans. The aim of this paper is to determine the dynamics of the SIO garbage patch at different depths and under different transport mechanisms such as ocean surface currents, Stokes drift, and direct wind forcing. To achieve this, we use two types of ocean surface drifters as a proxy for buoyant debris. We derive transport matrices from observed drifter locations and simulate the global accumulation of buoyant debris. Our results indicate that the accumulation of buoyant debris in the SIO is much more sensitive to different transport mechanisms than in the other ocean basins. We relate this sensitivity to the unique oceanic and atmospheric dynamics of the SIO.