Chaotic systems, especially hyper-chaotic systems are suitable for digital image encryption because of their complex properties such as pseudo randomness and extreme sensitivity. This paper proposes a new color image encryption algorithm based on a hyper-chaotic system constructed by a tri-valued memristor. The encryption process is based on the structure of permutation-diffusion, and the transmission of key information is realized through hyper-chaotic synchronization technology. In this design, the hash value of the plaintext image is used to generate the initial key the permutation sequence with the Hash table structure based on the hyper-chaotic sequence is used to implement pixel-level and bit-level permutation operations. Hilbert curves combining with the ciphertext feedback mechanism are applied to complete the diffusion operation. A series of experimental analyses have been applied to measure the novel algorithm, and the results show that the scheme has excellent encryption performance and can resist a variety of attacks. This method can be applied in secure image communication fields.