Recently, the nonlinear dynamics of memristor has attracted much attention. In this paper, a novel four-dimensional hyper-chaotic system (4D-HCS) based on a tri-valued memristor is found. Theoretical analysis shows that the 4D-HCS has complex hyper-chaotic dynamics such as hidden attractors and coexistent attractors. We also experimentally analyze the dynamics behaviors of the 4D-HCS in aspects of the phase diagram, bifurcation diagram, Lyapunov exponential spectrum, power spectrum and the correlation coefficient. To rigorously verify the chaotic behavior, we analyze the topological horseshoe of the system and calculate the topological entropy. In addition, the comparison with binary-valued memristor-based chaotic system shows that a chaotic system with a tri-valued memristor can generate hyper-chaos and coexistent attractors, while the one with a binary-valued memristor cannot. This finding suggests that applying three- or multi-value memristors in chaotic circuits can produce more complex dynamic properties than binary-valued memristors. To show the easy implementation of the 4D-HCS, we implement the 4D-HCS in an analogue circuit-based hardware platform, and the implementation results are consistent with the theoretical analysis. Finally, using the 4D-HCS, we design a pseudorandom number generator to explore its potential application in cryptography.