How can we assess the quality of an orbital fit?

The problem of assessing model validity and comparing alternative models is huge and any kind of complete treatment is well beyond the scope of this paper. However, given the comments above, it is appropriate to offer some observations on this large and interesting topic.

The issue of goodness-of-fit for models is not a simple one. Often, a χ2 is calculated and compared to determine which of competing models is the preferred. However, there are complexities involved. For example, Narsky (2003) discusses goodness-of-fit and points out that while use of the χ2 statistic as a goodness-of-fit measure for binned data is justifiable and is often done, it has flaws. He further observes that for unbinned data (such as we have here), there is no equivalent popular method for measuring goodness-of-fit. Indeed, Heinrich (2003) gives several examples of problematic goodness-of-fit cases. Similarly, Nat (2006) advocates investigating the structure of residuals to find patterns, biases, and sytematic differences between the model and the data.

We can divide metrics for model quality into two broad categories that, although useful, are certainly not mutually exclusive. First, we can consider what might be termed point estimates of fit quality. Often, a measure of merit is used that is related to the rms residuals between the model and the data. Fig. 4.13 compares our rms residuals for various eccen- tricities. Observations are generated with a Pioneer effect present. One panel of this figure

shows the results of fitting the “mismatched” case of the Pioneer-perturbed observations being fit to a strictly Newtonian gravity model; the other panel shows the corresponding “matched” case where the gravity model includes a Pioneer perturbation. In both cases the underlying synthetic observations contain a Pioneer perturbation. The most striking feature of this figure is how slowly the residual degrades as the arc length increases, espe- cially for low eccentricity objects. Indeed, until the observation arc is over 100 years long, there are only small differences in the quality of the fit as measured by rms residuals for any of the eccentricities evaluated. Even in the “mismatched” case, the rms residual is still less than half an arc second when the arc is 150 years long. In keeping with the discussion above about the ill-conditioned nature of the orbital fitting problem, we see that strictly relying on rms residuals as a measure of merit for model selection can be problematic.

Another measure of merit that is sometimes invoked is the uncertainties in the orbital elements themselves. The orbital fitting process automatically provides mathematically well-justified estimates of the elements, although these are in a linear approximation that has its own difficulties since the orbital fitting problem is intrinsically nonlinear. This approximation provides error estimates similar to those shown in Fig. 4.12. Across the eccentricities shown, there is substantial error spread throughout the parameter space. In particular, it might be instructive to compare Panels A and D in Fig. 4.12. In Panel A, a very low eccentricity case, the errors in semimajor axis, inclination, and longitude of the ascending node drop to a level allowing seven or eight significant digits to be present in the corresponding element values. At the same time, however, the uncertainties in eccentricity, argument of the pericenter, and mean anomaly remain relatively large. In the case with the largest eccentricity evaluated, Panel D shows the uncertainty to be more uniformly spread across elements. Trading off the uncertainties of one element against those of another to compare two models is therefore at best arbitrary and could be misleading.

Position error in the sky plane is yet another point metric for fit quality. In this case, uncertainties in the orbital elements are mapped directly onto the sky. Here, all the com- ments above about the uncertainty of the elements are valid, plus the observation that

0.00 0.50 1.00 1.50 2.00 2.50 3.00 0 50 100 150 200 250 300

RMS error of orbital fit (arcsec)

Observation Arc Length (year)

Panel A e = 0.300 e = 0.200 e = 0.100 e = 0.050 e = 0.010 e = 0.005 e = 0.001 0.00 0.50 1.00 1.50 2.00 2.50 3.00 0 50 100 150 200 250 300

RMS error of orbital fit (arcsec)

Observation Arc Length (year)

Panel B e = 0.300 e = 0.200 e = 0.100 e = 0.050 e = 0.010 e = 0.005 e = 0.001

Figure 4.13 Rms residual of orbital fit as observation arc length varies. Panel A shows the “mismatched” case where the observations are fit to a gravity model not containing the Pioneer effect. Panel B shows the “matched” case where the same observations are fit to a gravity model that includes a Pioneer effect perturbation.

the mapping from the six-dimensional space of orbital elements to the two-dimensional sky plane is highly degenerate; many sets of elements can map to the same region of sky. Thus, the sky plane position, by itself, can have problems as a point measure of model fit quality. In particular, comparing the nominal solution for two models can result in a large error volume about the calculated positions of objects.

All three approaches to providing point estimates of model fit quality have one common weakness. They are all based on a linearized form of the orbital fitting problem and make the assumptions associated with least squares fitting like normally distributed errors, inde- pendence, and no bias or systematic errors. Possibly the biggest manifestation of this is that we should not expect to be able to accurately extrapolate very far beyond the available observation arc unless there is at least a full orbital revolution’s volume of data and even then, extrapolation is dangerous.

A possibly better approach to comparatively assessing model alternatives is to use some type of global comparison. Of broadest applicability in this context is the idea of testing for the normality of the residuals between the model and the data. The overall basis for these tests is to look at the structure of the residuals and to perform statistical tests for lack of fit (see, for example, Nat (2006, Section 4.4.4.6)). The basis for these tests is to search for patterns, biases, and systematic differences between the model and the data.

Another approach that could be used to address the global quality of fit is to use boot- strap techniques. This approach could involve, as done here, synthesizing observations whose error characteristics reflect those found actual observations and then using the syn- thetic observations to assess actual variations in the fit quantities. Another approach of this sort would be to remove some of the data from the fit and see how the fitted solution extrapolates to the times that were removed. A similar approach would be simply to re- move data randomly from the observation set and analyze the resulting variations in fitting parameters.

of competing models. The distributions of orbital elements resulting from synthetic obser- vations that arise from a Monte Carlo process could be used to compare the probability of obtaining the observations in light of the competing models (see, for example, Gregory (2005); Jaynes and Bretthorst (2003); Sivia and Skilling (2006)). Although beyond the scope of this chapter, the findings discussed here show that there are not currently enough data to warrant this type of analysis; however, the advent of Pan-STARRS and LSST will change that situation in the near future.

4.3

Discussion

In the analysis described above we first showed that one cannot simply take orbital elements resulting from the fit of observations to a particular force model and use them to predict positions resulting from motion under the influence of another force. Rather, we must refit the orbits to the observations under the new force model. In the case of Pluto, this produces a well-fitting orbit that is indistinguishable in a practical sense, at least as long as the observation arc is short enough, from the unperturbed motion. Thus, without redetermining the elements we cannot make sweeping generalizations about whether or not the outer planets’ orbits show that the Pioneer effect does or does not exist.

Similarly, although making simplifying assumptions about a physical situation in order to draw conclusions is a time-honored theoretical mode of attack, if the physical model is oversimplified we can be misled into erroneous conclusions. As seen above, in an obser- vational context, both observer position and observational errors lead to the necessity of introducing the third spatial dimension with its associated degrees of freedom. If we do not keep an appropriate number of degrees of freedom, the problem can be oversimplified too much and mislead us into unwarranted conclusions. In particular, we must take care in using such a simplified model to conclude that the Pioneer effect does not exist.

Once again, it should be emphasized that our approach is itself an approximation. To properly conduct an analysis of the sort outlined here, the orbital parameters of the entire system of outer planets should be included. This would bring into the calculation

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1900 1950 2000 2050 2100 Acceleration/Pioneer acceleration Time (year) Uranus-Pluto Neptune-Pluto

Figure 4.14 Gravitational acceleration exerted by Uranus and Neptune on Pluto as a func- tion of time.

all second order perturbations. However, we have taken a simplified approach and believe it accurately and fairly addresses the uncertainties in the orbit of Pluto. One reason we believe our simplified approach is valid is illustrated in Fig. 4.14.

Fig. 4.14 shows the force per unit mass (in units of the Pioneer acceleration) exerted by Uranus and Neptune at Pluto’s position over the period of time we have considered. Perhaps surprisingly, due to the relative positions of the planets in their orbits, Neptune exerts less force on Pluto than Uranus over most of this period. The magnitude of the force is of the order of the Pioneer acceleration for both planets. If the Pioneer effect exists, we would expect the orbital elements of Uranus and Neptune to change, but their positions would change by very little. The magnitude of the forces they exert on Pluto would therefore change by an amount much less than the current magnitude of those forces. Thus, we argue that the approach used in this paper, while not as accurate as a full second order calculation involving all the outer planets, is accurate enough for our purposes.

Given the comments above, how then are we to compare alternative models? There are a number of methods that we can use to compare and assess models and their results. In

the orbit fitting context, these have been discussed above in terms of point estimates and global estimates of goodness of fit. The simplest point estimate is to simply calculate the rms residual of the fit compared with the input observations. However, as we’ve seen, the ill-conditioned nature of the orbital fitting problem can sometimes make the residual a poor candidate for this role. Errors, in the linear approximation, to the orbital elements can be assessed to determine the quality of the fit. Further physical insight can be gathered from inspecting the sky position errors that result from element errors. Global assessments of model fit primarily revolve around the normality of residuals. There are lack of fit tests that can be used to test the residuals and determine if there are any indications of a deficient model. Generally, a lack of fit is manifested as patterns, biases, and systematic variations in the residuals, which would indicate a poorly fitting model.

Which of these metrics is best? The reality is that determining the quality of a model and comparing model effectiveness is a complex problem. We need all these measures of model fit. Statistical tests should be performed comparing predicted positions, taking proper account of the associated errors, to test the hypothesis that the positions predicted under the two dynamical models are different. Only in this way can the existence of a perturbation like the Pioneer effect be falsified through astrometric methods.

As discussed earlier, Pluto is very interesting in that orbital fits to the two gravity models show deviations in predicted position that are probably detectable if the entire observation arc consisted of reliable measurements. However, prior to about 1960, there is a significantly greater dispersion in residuals that at later times. Only about half of Pluto’s motion since its discovery has been the result of systematic and organized observing campaigns (Gemmo and Barbieri, 1994). What adds special interest, however, is that because of its relatively large eccentricity Pluto is likely to show differences in position predictions for the two gravity models in the relatively near future as more observations are accumulated.

4.4

Conclusions

The analysis described above shows two major things. First, we must fit observations to a particular dynamical model and adjust orbital elements before predicted positions on the sky can be compared. Orbital parameters are derived from observations which have associated an unavoidable error. The determination of orbits is a model fitting process which has its own associated error sources. Extrapolating sky positions very far past the end of an observation arc can result in predicted observations becoming inaccurate so rapidly as to be worthless. The implication of these findings is that “matched” and “mismatched” gravity models cannot be distinguished on the basis of observable sky positions for observation arc lengths similar to those currently obtaining for Pluto.

Similarly, in order to draw conclusions about differences in position in the sky, we must be careful not to oversimplify the dynamical model used to draw the conclusions. Suppressing degrees of freedom in the dynamics simplifies the orbital determination problem to just such a degree. The orbital determination problem is nonlinear and the customary solution methods are approximations. Thus, any missing or ignored degrees of freedom can, if present, conceal dynamical effects associated with differing gravity models; we are forced to make use of the full dimensionality of the dynamical problem. In particular, a substantial amount of variation can be absorbed into a multidimensional parameter space and the full parameter space must be considered to properly reflect differences in motion of the outer planets due to the Pioneer effect.

The problem with the simplified approach is a two-headed one. First, the orbital fitting problem is inherently nonlinear and is normally solved in the linear approximation. Even if not mathematically chaotic, the system of equations is sensitively dependent upon initial conditions. Thus, small changes in elements can result in large changes in predicted position outside the range of observations. This sensitivity is exacerbated by the problem of a short observation arc. The length of the entire observational archive for Pluto is less than about one-third of a complete revolution. Together, these factors conspire to potentially generate large errors outside the observation arc, while increasing the length of the observation arc

can markedly reduce error over the whole of the arc and even beyond it.

We must conclude that we do not know the orbit of Pluto as well as we might have thought. We must continue to perform astrometry on it in order to be able to comment on the accuracy with which we know its orbit. Using current data, we cannot assert that the motion of Pluto demonstrates that the Pioneer effect does not exist. That jury is still out. Of course, this does not mean that the Pioneer effect exists. It does mean that we cannot deny the existence of the Pioneer effect on the basis of motions of the Pluto as currently known. Further observations are required before such an assertion can be made with confidence.

It should again be emphasized that our approach is itself an approximation. The dy- namical system that should be analyzed to provide a comprehensive answer to the question of the detectabilty of the Pioneer effect should include not only adjustment of the orbital elements of Pluto, but also simultaneous adjustment to those of Uranus and Neptune as well. It is only in this way that all the second order perturbations to the system can be taken into account. Our approach here, however serves to illustrate the ideas concerned and the weaknesses of the approaches outlined above.

Finally, it should be pointed out that, in addition to the observations of individual objects discussed in this paper, there are other related approaches to assessing gravity in the outer Solar System. For example, recently Wallin et al. (2007) have investigated the use of ensembles of Trans-Neptunian Objects (TNOs) to ascertain whether their motion reflects unknown additional perturbations and showed that the Pioneer effect was not consistent with the motion of TNOs. On the other hand, in the area of the observation of individual objects as discussed in this paper, the advent of Pan-STARRS and LSST in the next several years should provide sufficient data to determine whether the motion of outer Solar System bodies reflect the action of unknown forces. This determination should occur over time frames discussed in this paper. However, a combination of the techniques of Wallin et al. (2007) and the considerations presented here should provide definitive answers more quickly.

Chapter 5: Comets as Gravity Probes

There is another group of objects that might provide a vehicle for assessing the existence of the Pioneer effect through astrometry: the comets. Following the same general theme, we will now briefly look at them. Can they provide a vehicle for investigating the gravitational field in the outer Solar System?

Previously, we have emphasized the necessity of using complicated tools to investigate complicated Solar System dynamics. Here, we will take a slightly different tack. Comets are distinguished by the source of their comae and tails: outgassing. To the extent that any dynamic effects of cometary outbursts cannot be quantified, small perturbations like the Pioneer effect cannot be distinguished.

Thus, the purpose of this chapter is to investigate whether outgassing prevents the use of comets for exploring gravity in the outer Solar System.

The remainder of this chapter is divided into four sections. Section 5.1 will discuss the methodology employed to make the assessment. Section 5.2 discusses the results of the analysis. Finally, Section 5.3 provides a discussion of the results and Section 5.4 presents conclusions.

5.1

Methods and Models

The approach taken in this chapter is to model the orbits of comets in the outer Solar System by means of Newtonian gravity and two-body, planar motion. We will consider the Sun’s field to be spherically symmetric and will ignore the gravitational perturbations due