Energy Regression with BDTs

The energy spectrum is central to my analysis, so improving the energy reconstruction was important. One possible strategy for doing this would be to use a method analogous to disp for the core location reconstruction with the aim of correcting the impact distance bias. A few people in the VERITAS collaboration had made such an attempt and found that estimating this parameter is much more difficult. Other IACTs have used decision trees for energy reconstruction, such as MAGIC (Albert et al., 2008), and the performance improvement of the new disp method showed the power of estimating parameters with BDT regression in a VERITAS analysis. I decided to experiment with machine learning on a small set of LZA simulations to see if there was potential for improving either the energy bias,

uncertainty, or both.

The first step was to organize the training data into ROOT trees compatible with the methods available in TMVA. Simulation data must be used because the true energy of the primary must be known. I used the same calibrated stage 2 simulation files that were previously processed for LT production. Each entry in the training tree represents a simulated shower event that triggered the array, and the entry is written with the Hillas parameters of all four telescopes for a shower event as well as the properties of the primary gamma ray. I wrote a ROOT macro that was used to automate the creation of trees for all desired parameters. The same quality cuts applied in LT production are applied to the events, including the minimum pixel requirement, three or more telescopes, and the distance cut. The size cut is excepted and used as a training variable, just as it is used as a lookup variable in the standard method.

The TMVA factory allows the automated training using different multivariate methods, so I began training and testing with a small subset of the parameter space of my analysis to determine the most effective method. To determine the most effective TMVA method, I tested various algorithms with their default settings and compared their resulting bias curves. I began with the same set of variables used for the disp estimator, as they had proven to be a strong set of predictors. BDTs were the most effective method, with adaptive boosting having a slight edge over gradient boosting. The k-Nearest-Neighbor method was almost as effective but much less efficient (slower by an order of magnitude).

After selecting BDTs with adaptive boosting as the best method, I ran the training with a superset of the training variables used previously. TMVA automatically ranks all of the input variables by their correlation with the target variable. Using this ranking as a guide, I iterated through each promising variable and trained a set of trees. A separate macro evaluated each new weights file on a given set of simulations and created a bias curve for comparison. No new variables were found to be helpful, but excluding the azimuth was found to help marginally. I decided to exclude this variable, reducing the number of input variables.

This tree was subsequently used as the target for a new estimator, the predicted uncertainty of the energy. The training options and variables were held constant with the addition of the estimated energy as a training variable.

I tested the method on a more expansive parameter space, covering the full noise and zenith angle range that appear in the GC data. Training simulations included all observing offsets so that the trained weights were applicable to the spectrum of the diffuse region, which covers the full range of offsets. The number of events added at each offset was scaled according to the LZA acceptance curve to weight the offsets properly.

The next step was to tweak the training options of the adaptively boosted decision tree. These options included the number of trees in the ensemble, the maximum depth of trees, the minimum number of events required for a new node, and the number of cut values tested to achieve optimal node splitting. I iterated through the parameter space, first doing a coarse search for the best set of options, then refining the values with a fine search.

The best configuration for energy regression used adaptive boosting and ended up being more optimal for a small number of deep trees, as opposed to the large number of shallow trees that worked best for the disp method. No maximum depth was specified, and nodes terminated when they either had too few events or no longer added separation power. It is possible that directional biases correlated with training variables could be averaged out when tested on the full parameter set. To ensure the weights would perform consistently, they were tested on a subsets of the parameter space. For example, I tested weights trained on all offsets on a few singular values of offset and checked the bias curves for any offset-dependent bias. I found no such biases for the variables I used.

The energy bias and uncertainty of the optimized BDT weights for V6 simulations at 65° zenith angle is plotted with the LT method in Figure 6.5. As can be seen, the bias converges to zero quickly above the energy threshold and remains consistent up to the highest energies detectable by VERITAS. The energy uncertainty for the BDT method is lower than for the standard method over the full energy range and is more comparable to the standard method at SZA. The optimal training parameters were then used to train V5 simulations in the same

way, with the results showing the same improvement.

2.0

10.0

70.0

Energy [TeV]

0.2

0.0

0.2

m

ea

n

of

(E

E

)/E

2.0

10.0

70.0

Energy [TeV]

0.0

0.2

0.4

std

ev

of

(E

E

)/E

TMVA Energy Reconstruction Standard Energy Reconstruction

Figure 6.5: Histograms of the energy bias (top) and energy resolution (bottom) binned in energy for two different methods of energy reconstruction. The standard method for energy reconstruction is shown as red points, and the TMVA (BDT) method is shown in blue. The vertical yellow line marks the energy threshold. The simulations were thrown from a zenith angle of 65°, where the bias is most severe, and the detector simulations modeled the V6 array for the plot pictured.

Once I was confident in the trained weights files, I validated the new energy reconstruction with a full analysis of real data. The customary target on which to validate new methods is the Crab pulsar, a strong and steady source for which VERITAS takes a great deal of data covering a broad range of zenith angles. Starting in 2017, I requested a larger set of Crab data be taken between 25° and 30° elevation. Since that time, the total amount of observation data in that range has increased from five to ten hours. Because the gamma-ray rate of the Crab is very high, ten hours of data provide enough signal events to yield a statistical significance above 60σ, well above the rough requirement of 10σ for spectral reconstruction.

The distribution of zenith angles of Crab events from the data set I used is shown on the bottom of Figure 6.1.

A spectral analysis using this new energy reconstruction method required generating new EAs, as the EA energy bins are filled using the reconstructed energy rather than the true energy of the simulation by default (see Subsection 5.8.4). The event selection box cuts were optimized for LZA (see Subsection 6.3.2). Both the simulation and Crab data were processed through the shower reconstruction using the same options as previously used for the disp method, changing only the energy reconstruction method and the size cut. Each file was provided the proper energy reconstruction BDT weights file according to its epoch, atmosphere, and zenith angle. The event selection stage and EA production used the new optimized values for MSL and MSW.

The results of the Crab spectrum are shown in Figure 6.6, along with spectral points for the standard analysis at SZA and a BDT disp method at LZA with the standard method of energy reconstruction. The standard analysis points (purple) are consistent with the accepted spectrum of the Crab in this energy domain. The spectral points are fitted with a pure power law (Equation 3.1) from 2–10 TeV, which is the range common to SZA and LZA. The best fit parameters, shown below, show improved agreement between the standard method and the BDT method for energy regression. The spectral fits of LZA data processed with the standard method of energy reconstruction were not consistent with the standard SZA results. Both parameters, the flux normalization and power-law index, deviate from the accepted values by more than their respective errors.

6.2

Spectral Analysis