2.6 Discovering Hard Magnets Using a Machine Learning Classification
2.6.2 Results
The accuracy of the classifiers was established on the test set. We find that CLF1 achieves a score of 74 % and CLF2 of 69 %. These results suggest that CLF1 is superior, although the overall accuracy improvement is not significant. We were not able to improve upon this quality by changing the ML algorithm or the input features. However, the possibility of achieving such an improvement is not excluded. A more in-depth analysis of the classifier performance was obtained by plotting the “receiver operating curve” (ROC) [161, 162], shown in figure 2.10. In the left-hand panel we plot the true positive (classification) rate (TPR) as a function of the false positive rate (FPR), for each classifier. A “random classifier”, which assigns the labels arbitrarily, but with equal probability, is shown by a red dashed line. It is important to note here that the operating point of the classifier can be moved along the ROC curve, by changing the decision value cutoff. In this way we can choose if we find it more important to have a classifier which is highly sensitive or one which is specific. The former maximizes the TPR, at the expense of having a larger amount of FP predictions, while the latter tries to minimize the FPR, at the expense of a lower TPR. Usually a compromise between these two goals needs to be made. A good classifier has a large area under the curve (auc), ideally 1. The ideal operating point is thus located at (0,1), as indicated in the figure, where all the samples are correctly classified. We find that both MCA classifiers have a similarauc. The CLF2 performs slightly better, havingauc= 0.78, compared to theauc= 0.75, achieved by CLF1. The latter were estimated on the test set.
Although the test scores for both classifiers are comparable, the shape of the respective ROC curves differs significantly. This implies different classification capabilities of the two models. The FPR does not vanish for any point on the CLF1 ROC curve, i.e. this classifier will always yield a certain amount of misclassified samples. The CLF2, on the other hand, has the initial part of the ROC curve along the y axis, where FPR = 0, followed by a sharp turn and a sudden increase of the FPR. The operating region just before this knee is very interesting, since it means that the classifier will never mistake a hard magnet for a soft one. In other words, CLF2 can be very specific, FPR = 0, although the sensitivity, TPR ∼ 0.6, is far from the maximal, TPR = 1. From this point of view CLF2 is clearly superior to CLF1. Therefore, using CLF2, in the best case scenario, about 60 % of the soft magnets in the dataset could be identified based solely on the Heusler chemical formula. We note that this would cover only 30 % of the entire dataset, and the MCA would have to be calculated for the rest. We will return to this point later.
In the right-hand panel of figure 2.10 we give a qualitative interpretation of the CLF2 ROC curve. We note here that a binary classification can be understood as a problem of
finding an optimal hyperplane, which separates the two classes [23]. For every sample, a distance from the separating hyperplane, d, can be calculated and, based on this distance, the algorithm decides if the sample belongs to a given class. The latter is given by the decision function. In the case of the RidgeClassifier the decision function returns a signed distance of the sample to the hyperplane. Ford >0 the class is predicted as positive [159]. The true positive samples will follow a distribution,p(x), and the true negative, n(x), as indicated in the figure [162]. There we compare two hypothetical classifiers, which have slightly different distribution functions for the positive samples, p1andp2(bottom right panel), but a similar shape of the ROC curve to the one found for CLF2(top right panel). We assume that the distributionspi(x) andn(x) are normalized
Gaussian functions. The TPR and the FPR, are then given as
T P R(T) = Z +∞ T dx p(x) F P R(T) = Z +∞ T dx n(x) . (2.4) Here T denotes the cutoff value for the decision function. All samples for which d≥T are classified as positive. More specifically, for theRidgeClassifier the cutoff is given by T = 0. By changing the value of T we choose a different operating point on the ROC curve. We see that a conservative choice ofT, in this example T &0, leads to a correct classification of the samples, i.e. FPR = 0. For smaller values of T, i.e. as we move along the ROC curve (to the right), the integration over a sharply peaked distribution n(x), leads to a significant increase of the FPR, while the TPR changes a little. This makes the ROC curve to appear almost flat. The limiting caseT → −∞leads us to the upper right corner of the plot. We can see that the misclassification is related to the overlap of the distributions n(x) and p(x). In the case of p2(x), the overlap with n(x) is smaller and thus a higher initial TPR can be achieved. This case is a close match to the ROC curve observed for CLF2. In conclusion, the reason why CLF2 manages to achieve FPR = 0, in the initial part of the ROC curve, is because the corresponding distributions n(x) and p(x) are reasonably well separated by the model. This implies that the vector of features of Eq. (2.3) can be used to fully segregate a part of the soft magnet population (∼ 60 %) from the rest. This is an interesting result, which is analysed more closely in figure 2.11.
As previously discussed, the ideal operating point is located just before the knee on the CLF2 ROC curve (TPR = 0.6 and FPR = 0.0). In this case the classifier can identify soft magnets without making a mistake, although not all of them are detected. As we have already mentioned, this indicates that a part of the soft magnet population is well separated in the feature space defined by Eq. (2.3). We are interested here if this group of materials has a specific chemical composition, which should be avoided in the design of hard magnets. In figure 2.11, top panel, we assign a descriptive label
C
ou
nt
0
5
10
15
20
25
30
Soft Mixed True SoftC
ou
nt
0
5
10
15
20
25
30
Z
X0 10 20 30 40 50 60
True Hard True SoftZ
Y0 10 20 30 40 50 60
Z
Z0 10 20 30 40 50 60
Figure 2.11: Top panel - histogram of the site occupation for the 114 material training set. The magnets have been labeled as “true soft”, “soft” and “mixed”. The first group is made from materials which have MCA ≤ 0.8 MJ m−3, as predicted by
DFT. The second two groups are labeled according to the CLF2prediction. The “soft”
magnets are those, which are classified as positive, when the classifier operating point is set just before the knee in figure 2.10. In this case there is no misclassified samples. The negative samples are labelled as “mixed”, since they are a mixture of the hard and the soft magnets, that can not be resolved by the classifier. Bottom panel - a closer look at the trends in the mixed magnet group. Here the materials have been labeled as the “true hard” and the “true soft”, according to the MCA value predicted by the
asymptotic convergence
training
validation
S
co
re
[
%
]
50
60
70
80
90
100
Training Set Size
0
20
40
60
80
Figure 2.12: The learning curve for the CLF2classifier.
to the output of CLF2, namely we identify the magnets as soft or mixed. The former are guaranteed to have the MCA ≤0.8 MJ m−3, while the latter are a mixture of hard and soft magnets, that can not be distinguished by the classifier. The population of the “true soft” magnets, as predicted by DFT, is shown for comparison. It is important to note that the population of the mixed magnets has a strong overlap with the population of the “true soft” magnets. For this reason, the two populations will share similar traits in the analysis. We can observe that the site occupation for all groups overlaps, over the entire composition space, defined by (ZX, ZY, ZZ), and thus a clear design rule can not
be inferred. The strongest correlation between “soft” and “true soft” magnets can be made with respect to the occupation of theX site. We can observe that a soft magnet occurs with a high probability when theXsite is occupied by an early transition element (3dor 4d), or by a very light element, ZX <10. In contrast, the mixed magnets, which
include the group of hard magnets, are more likely when the X site is occupied by a late transition element, although the frequency of the “true soft” magnets in this case is not negligible. For other sites the composition appears to be very similar for all groups. This implies that the X site plays an important role in establishing a large MCA. Namely, the Y site determines the magnetic moment, while theX site provides a large SOC. The MCA classifier thus leads us to the same conclusion that was found in the previous section, by direct MCA calculations. The group of mixed magnets is analysed more closely in the bottom panel. We can see that the “true hard” and the “true soft” magnets follow very similar trends. This may be a part of the reason why the classifier has difficulties distinguishing the soft magnets that fall into this group.
Finally, we would like to estimate the computational benefits of following the ML ap- proach presented here. In figure 2.12 we show the learning curve for CLF2. We can see that approximately 80 materials are needed to achieve a converged result. Therefore, our training set of 114 materials was a reasonable choice. In comparison, this makes 75 % of the MCA dataset. The classifier could be applied to ∼40 materials in the test set. Given that CLF2 can effortlessly identify 60 % of the total soft magnet population, this gives us approximately 28 candidate magnets for which only the Curie temperature would need to be calculated. The 12 soft magnets can be safely eliminated. We recall here that the threshold for the hard magnet, MCA≥0.8 MJ m−3, was chosen such that approximately 50 % of all the materials in the training set fall into this class. Assuming that the training set is representative of the entire population of magnets in the database, using the ML classification we can reduce the computational effort by 30 %. Given these considerations, we can conclude that our ML model becomes useful only when dealing with large datasets, which are of the order of 1000 materials or more. For small datasets it is better to proceed with the direct calculation. In comparison, the expected number of magnetic (full) Heusler alloys in the MMDB, which satisfy the first HTA screening step (cf. figure 2.1), is of the order of 1000. The ML approach could thus save us the trouble of calculating ∼300 materials. Although the overall computational gain in this case is moderate, it is encouraging that a dataset of about 100 materials was sufficient to train the classifier, which means that a ML approach can be attempted with a rather low initial computational resource investment. We do not consider here the number of calculations that are needed to provide the input for the classifier. In conclusion, we find that the ML techniques can be used in conjunction with the HTA screening procedure (cf. figure 2.1), leading to an expected 30 % savings in the computational effort, while maintaining the DFT level of accuracy.