This paper studies the bifurcation and chaos phenomena in average queue length in a developed Transmission Control Protocol (TCP) model with Random Early Detection (RED) mechanism. Bifurcation and chaos phenomena are nonlinear behavior in network systems that lead to degradation of the network performance. The TCP/RED model used is a model validated previously. In our study, only the average queue size is considered, and the results are based on analytical model rather than actual measurements. The instabilities in the model are studied numerically using the conventional nonlinear bifurcation analysis. Extending from this bifurcation analysis, a modified RED algorithm is derived to prevent the observed bifurcation and chaos regardless of the selected parameters. Our modification is for the simple scenario of a single RED router carrying only TCP traffic. The algorithm neither compromises the throughput nor the average queuing delay of the system.