Efficient Gradient Boosted Decision Tree Training on GPUs

Zeyi Wen, Bingsheng He, Kotagiri Ramamohanarao, Shengliang Lu, Jiashuai Shi

Research output: Chapter in Book/Conference paperConference paper

20 Citations (Scopus)


In this paper, we present a novel parallel implementation for training Gradient Boosting Decision Trees (GBDTs) on Graphics Processing Units (GPUs). Thanks to the wide use of the open sourced XGBoost library, GBDTs have become very popular in recent years and won many awards in machine learning and data mining competitions. Although GPUs have demonstrated their success in accelerating many machine learning applications, there are a series of key challenges of developing a GPU-based GBDT algorithm, including irregular memory accesses, many small sorting operations and varying data parallel granularities in tree construction. To tackle these challenges on GPUs, we propose various novel techniques (including Run-length Encoding compression and thread/block workload dynamic allocation, and reusing intermediate training results for efficient gradient computation). Our experimental results show that our algorithm named GPU-GBDT is often 10 to 20 times faster than the sequential version of XGBoost, and achieves 1.5 to 2 times speedup over a 40 threaded XGBoost running on a relatively high-end workstation of 20 CPU cores. Moreover, GPU-GBDT outperforms its CPU counterpart by 2 to 3 times in terms of performance-price ratio.
Original languageEnglish
Title of host publicationIEEE International Parallel and Distributed Processing Symposium (IPDPS)
Number of pages10
Publication statusPublished - 2018
Externally publishedYes
Event32nd International Parallel and Distributed Processing Symposium - JW Marriott Parq Vancouver, Vancouver, Canada
Duration: 21 May 201825 May 2018


Conference32nd International Parallel and Distributed Processing Symposium
Abbreviated titleIPDPS 2018
Internet address

Fingerprint Dive into the research topics of 'Efficient Gradient Boosted Decision Tree Training on GPUs'. Together they form a unique fingerprint.

Cite this