JIT-Smart: A Multi-task Learning Framework for Just-in-Time Defect Prediction and Localization

Abstract

Just-in-time defect prediction (JIT-DP) is used to predict the defect-proneness of a commit and just-in-time defect localization (JIT-DL) is used to locate the exact buggy positions (defective lines) in a commit. Recently, various JIT-DP and JIT-DL techniques have been proposed, while most of them use a post-mortem way (e.g., code entropy, attention weight, LIME) to achieve the JIT-DL goal based on the prediction results in JIT-DP. These methods do not utilize the label information of the defective code lines during model building. In this paper, we propose a unified model JIT-Smart, which makes the training process of just-in-time defect prediction and localization tasks a mutually reinforcing multi-task learning process. Specifically, we design a novel defect localization network (DLN), which explicitly introduces the label information of defective code lines for supervised learning in JIT-DL with considering the class imbalance issue. To further investigate the accuracy and cost-effectiveness of JIT-Smart, we compare JIT-Smart with 7 state-of-the-art baselines under 5 commit-level and 5 line-level evaluation metrics in JIT-DP and JIT-DL. The results demonstrate that JIT-Smart is statistically better than all the state-of-the-art baselines in JIT-DP and JIT-DL. In JIT-DP, at the median value, JIT-Smart achieves F1-Score of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.475</mml:mn> </mml:mrow> </mml:math> , AUC of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.886</mml:mn> </mml:mrow> </mml:math> , Recall@20%Effort of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.823</mml:mn> </mml:mrow> </mml:math> , Effort@20%Recall of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.01</mml:mn> </mml:mrow> </mml:math> and Popt of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.942</mml:mn> </mml:mrow> </mml:math> and improves the baselines by <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>19</mml:mn> <mml:mn>.89%</mml:mn> <mml:mi>-</mml:mi> <mml:mn>702</mml:mn> <mml:mn>.74%</mml:mn> </mml:mrow> </mml:math> , <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>1</mml:mn> <mml:mn>.23%</mml:mn> <mml:mi>-</mml:mi> <mml:mn>31</mml:mn> <mml:mn>.34%</mml:mn> </mml:mrow> </mml:math> , <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>9</mml:mn> <mml:mn>.44%</mml:mn> <mml:mi>-</mml:mi> <mml:mn>33</mml:mn> <mml:mn>.16%</mml:mn> </mml:mrow> </mml:math> , <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>21</mml:mn> <mml:mn>.6%</mml:mn> <mml:mi>-</mml:mi> <mml:mn>53</mml:mn> <mml:mn>.82%</mml:mn> </mml:mrow> </mml:math> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>1</mml:mn> <mml:mn>.94%</mml:mn> <mml:mi>-</mml:mi> <mml:mn>34</mml:mn> <mml:mn>.89%</mml:mn> </mml:mrow> </mml:math> , respectively. In JIT-DL, at the median value, JIT-Smart achieves Top-5 Accuracy of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.539</mml:mn> </mml:mrow> </mml:math> and Top-10 Accuracy of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.396</mml:mn> </mml:mrow> </mml:math> , Recall@ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>20</mml:mn> <mml:mi>%</mml:mi> <mml:msub> <mml:mrow> <mml:mtext>Effort</mml:mtext> </mml:mrow> <mml:mrow> <mml:mi>l</mml:mi> <mml:mi>i</mml:mi> <mml:mi>n</mml:mi> <mml:mi>e</mml:mi> </mml:mrow> </mml:msub> </mml:mrow> </mml:math> of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.726</mml:mn> </mml:mrow> </mml:math> , Effort@ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>20</mml:mn> <mml:mi>%</mml:mi> <mml:msub> <mml:mrow> <mml:mtext>Recall</mml:mtext> </mml:mrow> <mml:mrow> <mml:mi>l</mml:mi> <mml:mi>i</mml:mi> <mml:mi>n</mml:mi> <mml:mi>e</mml:mi> </mml:mrow> </mml:msub> </mml:mrow> </mml:math> of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:mn>0.087</mml:mn> </mml:mrow> </mml:math> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"> <mml:mrow> <mml:msub> <mml:mrow> <mml:mtext>IFA</mml:mtext> </mml:mrow> <mml:mrow> <mml:mi>l</mml:mi> <mml:mi>i</mml:mi> <mml:mi>n</mml:mi> <mml:mi>e</mml:mi> </mml:mrow> </mml:msub> </mml:mrow>

Topics & Concepts