Analysis of boundedness and convergence of online gradient method for two-layer feedforward neural networks

Lu Xu, J. Chen, David Huang, J. Lu, Licai Fang

    Research output: Contribution to journalArticle

    7 Citations (Scopus)

    Abstract

    This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved. © 2012 IEEE.
    Original languageEnglish
    Pages (from-to)1327-1338
    JournalIEEE Transactions on Neural Networks and Learning Systems
    Volume24
    Issue number8
    DOIs
    Publication statusPublished - 2013

    Fingerprint

    Gradient methods
    Feedforward neural networks
    Chemical activation
    Difference equations

    Cite this

    @article{0afaf1976f5741728d5ffa49a919a09b,
    title = "Analysis of boundedness and convergence of online gradient method for two-layer feedforward neural networks",
    abstract = "This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved. {\circledC} 2012 IEEE.",
    author = "Lu Xu and J. Chen and David Huang and J. Lu and Licai Fang",
    year = "2013",
    doi = "10.1109/TNNLS.2013.2257845",
    language = "English",
    volume = "24",
    pages = "1327--1338",
    journal = "IEEE Transactions on Neural Networks",
    issn = "1045-9227",
    publisher = "IEEE, Institute of Electrical and Electronics Engineers",
    number = "8",

    }

    Analysis of boundedness and convergence of online gradient method for two-layer feedforward neural networks. / Xu, Lu; Chen, J.; Huang, David; Lu, J.; Fang, Licai.

    In: IEEE Transactions on Neural Networks and Learning Systems, Vol. 24, No. 8, 2013, p. 1327-1338.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Analysis of boundedness and convergence of online gradient method for two-layer feedforward neural networks

    AU - Xu, Lu

    AU - Chen, J.

    AU - Huang, David

    AU - Lu, J.

    AU - Fang, Licai

    PY - 2013

    Y1 - 2013

    N2 - This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved. © 2012 IEEE.

    AB - This paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two-layer feedforward neural networks. The well-known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions. Based on this extended difference equation, we investigate the boundedness and convergence of the parameter sequence of concern, which is trained by finite training samples with a constant learning rate. We show that the uniform upper bound of the parameter sequence, which is very important in the training procedure, is the solution of an inequality regarding the bound. It is further verified that, for the case of linear activation function, a solution always exists and, moreover, the parameter sequence can be uniformly upper bounded, while for the case of nonlinear activation function, some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property. Then, for the convergence analysis, it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum, where the size of the zone is associated with the learning rate. Particularly, for the case of perfect modeling, a strong global convergence result, where the parameter sequence can always converge to an optimal solution, is proved. © 2012 IEEE.

    U2 - 10.1109/TNNLS.2013.2257845

    DO - 10.1109/TNNLS.2013.2257845

    M3 - Article

    VL - 24

    SP - 1327

    EP - 1338

    JO - IEEE Transactions on Neural Networks

    JF - IEEE Transactions on Neural Networks

    SN - 1045-9227

    IS - 8

    ER -