Speaker location estimation techniques based on time-difference-of-arrival measurements have attracted much attention recently. Many existing localization ideas assume that only one speaker is active at a time. In this paper, we focus on a more realistic assumption that the number of active speakers is unknown and time-varying. Such an assumption results in a more complex localization problem, and we employ the random finite set (RFS) theory to deal with that problem. The RFS concepts provide us with an effective, solid foundation where the multispeaker locations and the number of speakers are integrated to form a single set-valued variable. By applying a sequential Monte Carlo implementation, we develop a Bayesian RFS filter that simultaneously tracks the time-varying speaker locations and number of speakers. The tracking capability of the proposed filter is demonstrated in simulated reverberant environments.