As we have discussed in section 5.3.1, we are facing the problem that the small portion of the data for which we know the hole cards is not sufficient to train unbiased models that predict actions for a bucket. In this section, we will present an algorithm that aims to circumvent this problem by using all of the data to train models.
The basic idea of the algorithm is that we can use the data for which we do know hole cards (the labeled data from now on) to iteratively improve an estimate of the hole cards for all players in the part of the data for which we do not know the hole cards (the unlabeled data from now on). To this end, we keep a probability distri- bution over all holdings for every player in every game in the unlabeled data. These probability distributions are all initialized to an educated guess that is calculated as follows.
Since we know that every holding has an equal chance of being dealt to a player, we know that the distribution over all holdings in all of the data (the unlabeled and labeled data combined) must approximately be uniform. We can determine the distribution of all holdings for the labeled data since these are observable. By subtracting these two distributions and applying weighing to account for different sample sizes, we can infer the distribution over all holdings for the unlabeled data. We can then use this distribution as our initial probability distribution for all players in the unlabeled data set.
By guessing an initial probability distribution over the holdings of all players whose cards were not revealed, we effectively label the large unlabeled part of the data. After this initialization step, all data can be considered as labeled and we can treat it as such. We now have a distribution over all holdings for every player in the data set and can use the data to learn models that predict actions, given some description of the state of the game and a holding or bucket.
Unfortunately, the models that we would obtain from this data set in which 90 to 97 percent of the holdings have been initialized to the same rather inaccurate educated guess will probably not be very accurate. However, as we have seen in section 5.2, we can obtain estimates of the probability distribution over all holdings for each player from the moves that the player selected and a set of models that predict these moves given that the player held a particular holding. This means that we could use our newly obtained models to estimate a probability distribution over all holdings for every player in the unlabeled part of the data.
The assumption is that these inferred probability distributions will be better than the initial distributions that we had guessed, as at least they will be based on the actual actions that the player performed and will not be identical for every player in every game. These improved estimates of what holding each player held in the unlabeled part of the data may then be used to train improved models that predict actions given holdings. These models may then be used to obtain further improved estimates of the probability distributions over holdings. This process may be repeated until there is no further improvement.
Estimates of unlabeled Models Initialization Update step Unlabeled data Labeled data
Figure 5.3: Overview of the proposed algorithm to infer probability distributions over all holdings for players’ whose holdings have not been revealed. The idea is to guess initial probability distributions for each player based on the labeled part of the data and the fact that we know that each holding has an identical probability of being drawn. We can then use this guess in combination with the labeled data to train models that predict actions given holdings. These models may then be used to improve our estimate of the probability distributions over holdings for the unlabeled data. This process may then be repeated until convergence.
ity distributions for the unlabeled data and the models that predict actions given holdings is summarized in figure 5.3. Note that it is effectively an expectation maxi- mization (cf. section 2.3.3) process, in which an expectation (estimating probability distributions over holdings for the unlabeled data) and a maximization (learning new models that predict actions given holdings) step are alternated until convergence.
5.5.1 Step by step
Let us now describe the algorithm step by step, in some more detail:
1. Initialization: infer the distribution of holdings for the unlabeled data using the fact that all holdings have an equal chance of being dealt and that we know
the distribution for the labeled data. Assign this distribution to every player in every game of the unlabeled data.
2. Models training: train a model for every bucket for every street. The data set for the model for bucket b is composed of both labeled and unlabeled data. The labeled data has a weight of 1 in the training set if the player had a holding belonging to bucket b (and 0 if the player had a different holding). The weights for the unlabeled data are taken from the associated probability distribution over holdings for each player: the more likely it is that a player has a holding in bucket b, the greater the weight of his actions in the training set.
3. Probability distribution updates: update the probability distributions over holdings for every player in every game of the unlabeled data. In equation 5.2, we found that we can obtain a probability estimate for every holding using the following formula: PH|an, . . . , a0, ~fn, . . . , ~f0 = P (H) Y i=0,...,n Pai| ~fi, H
Where P (H) is the prior for the holding (which we know to be uniform), and P (ai| ~fi, H) is the probability that the player takes action ai given a feature vector fi and holding H. Since the models that we learn predict models for a bucket and not for a holding, we need to translate between buckets and holdings (cf. section 5.4.5), and the equation becomes:
PH = hi|an, . . . , a0, ~fn, . . . , ~f0 = P (H = hi) Y j=0,...,n X k mi,kP aj| ~fj, Bk
Where mi,k is the membership probability that holding hi belongs in bucket k and P (aj| ~fi, Bk) is the probability that the player takes action aigiven a feature vector fi and bucket Bk. P (aj| ~fi, Bk) may be obtained using the models that were learned in step 2.
4. Repeat step 2 and step 3 until convergence.