[2112.03220] Cross-Validation of Changepoint Regression: Pitfalls and Solutions

Download the PDF of the paper entitled “Cross Validation of Changepoint Regression: Pitfalls and Solutions” by Florian Pein and Rajen D. Shah.

Download PDF

overview: Cross-validation is a standard approach for tuning parameter selection in many nonparametric regression problems. However, its use is less common in changepoint regression. This is probably because its prediction-error-based criterion appears to tolerate small spurious changes and is therefore not well suited for estimating the number and location of changepoints. In fact, the cross-validation problem with squared error loss is more severe, where the number of change points is systematically underestimated or overestimated, leading to highly suboptimal estimates of the mean function in simple settings where changes occur. indicates a possibility. Easy to detect. He proposes two simple approaches to solving these problems. The first is to use the absolute error instead of the squared error loss, and the second is to change the holdout set used. For the latter, it provides the conditions under which the number of changepoints can be consistently estimated for general changepoint estimation procedures. We use new results on performance when an incorrect number of changepoints are specified to show that these conditions are met for optimal splits. Numerical experiments show that the absolute-error approach competes with the general change-point method, especially with classical tuning parameter choices, when the error distribution is well-specified, but with an incorrectly-specified model can significantly exceed these. An implementation of our methodology is available on CRAN in the R package crossvalidationCP.