The control of the electrolyte flow rate is crucial to ensure the efficient operation of a vanadium redox flow battery (VRFB) system. In this paper, a model-based nonlinear dynamic optimisation (MNDO) method is proposed and implemented in MATLAB/Simulink to study the optimal flow rate under constant current (CC) and constant current–constant voltage (CC–CV) charging methods for a VRFB. A 5 kW/3 kWh VRFB system is considered to investigate the feasibility and accuracy of the established model. System efficiency is the optimisation objective in this study, where all case studies are carried out within 15% to 85% state of charge (SOC). The simulation results show that the proposed method enhanced battery performance by improving the overall VRFB system efficiency under different charging methods. Furthermore, the simulation results show that the CC–CV method is more energy-efficient than the CC method but requires more charging time. An in-depth analysis is carried out to discuss the underlying merits of the proposed method in balancing the losses caused by the concentration overpotential and pump energy consumption in a varying power environment. Moreover, further analyses are carried out to highlight the merits of the CC–CV charging method in saving energy while charging and reducing system imbalance. Finally, a 2D lookup table is designed based on the results from the proposed MNDO method that offers a practical controller of the electrolyte flow rate without requiring excessive computational resources. The performance of the 2D lookup table has been evaluated within 100 charging/discharging cycles. It achieves a system efficiency gain of up to 1.48% under CC–CV charging operation compared with the CC charging method and optimal conventional flow rate control method.