Memristors and memcapacitors are two new nonlinear elements with memory. In this paper, we present a Hewlett-Packard memristor model and a charge-controlled memcapacitor model and design a new chaotic oscillator based on the two models for exploring the characteristics of memristors and memcapacitors in nonlinear circuits. Furthermore, many basic dynamical behaviors of the oscillator, including equilibrium sets, Lyapunov exponent spectrums, and bifurcations with various circuit parameters, are investigated theoretically and numerically. Our analysis results show that the proposed oscillator possesses complex dynamics such as an infinite number of equilibria, coexistence oscillation, and multi-stability. Finally, a discrete model of the chaotic oscillator is given and the main statistical properties of this oscillator are verified via Digital Signal Processing chip experiments and National Institute of Standards and Technology tests.