Having evaluated the market with respect to the metrics listed in section 3.1 and the correlation between theinqand theupqof the recommendations, this section evaluates the system’s ability to seek out the best item from all the source recommendations. This is clearly an important feature from the user’s viewpoint, since if the system cannot recommend the best items, the user will not use it.
To evaluate this aspect of the system, we use the first experiment discussed in sec- tion 4.2.1 and trace the bidding price of the recommendation with the highest upq
value selected by the first agent (see Figure 4.12, in which the cross points represent the bidding price of this particular recommendation). From this, we can see that this
Chapter 4 Simulating and Evaluating the Marketplace 75 100 150 200 250 0 50 100 150 200 250 300 Prices Auctions
the Highest UPQ Recommendation’s Bid Price
1st Bid 10th Bid the highest UPQ’s price
Figure 4.12: The Best Recommendation’s Bidding Price
recommendation’s bidding price keeps increasing till it converges to the first bid price of the shortlisted items. This means that as long as the first agent chooses the highest
upqrecommendation to bid in an auction round (after the market converges), this item is always displayed in the first slot of the sidebar of the user’s browser. Therefore, in case of either user model (independent selection or search-till-satisfaction), this recom- mendation will be selected by the user, since the first shortlisted recommendation has the highest upq. This result shows that the system is capable of seeking out the best recommendation and presenting it to the user.
4.5
Summary
Based on these simulations, the auction mechanism designed in Chapter3is shown to be effective. Specifically, we organized a set of consecutive auctions with nine recommender agents to offer their recommendations. We simulate the user with two kinds of user mod- els: independent selection of recommendations and search till satisfied. We find that our market can always converge after a number of auctions with either user model. After convergence, the marketplace is able to give incentives of users’ preferences to the rec- ommender agents and to shortlist the best recommendations in decreasing order of their
upqs. Additionally, the marketplace gives either effective or ineffective recommender agents equal opportunity to bid their recommendations and is stable meaning that it is able to stop greedy bidders aggressively bidding their recommendations. By simulation, our marketplace is shown to be capable of successfully correlating the two perspectives of
Chapter 4 Simulating and Evaluating the Marketplace 76
recommendation quality (internal and user perceived) and is able to identify the highest
upqitem to be shortlisted at the top position of the recommendation sidebar.
This chapter has proved that the auction mechanism design developed in Chapter 3 is feasible to coordinate multiple different recommendation methods in one single system and is able to relate their good recommendations to the user’s interests. However, how a recommender agent learns the user’s interests is left to be addressed and this is the subject of Chapter5.
Chapter 5
Learning Users’ Interests
Having shown the effectiveness of our market mechanism as a means of coordinating different recommendation methods, an open problem from the point of view of the individual recommender agents remains: given a set of recommendations with different inq levels, in what order should an agent try to advertise them so that it can learn the user’s interests as quickly as possible, while still maximizing its revenue? Thus, for example, the agent could bid the items that have never been advertised to the user, which would allow it to learn the user’s interests quickly but could also result in it losing money. Conversely, the agent could always bid those items that have been highly rewarded, so ensuring a good return, but it would take a very long time to learn the extent of the user’s interests. To this end, this chapter reports a quality classification mechanism and a reinforcement learning strategy we built for the recommender agents to learn the user’s interests.
This chapter contributes to the thesis in the way that a marketplace with learning recommender agents converges quicker and seeks out the best recommendations quicker and more frequently than that with non-learning agents. Moreover, with a learning capability, a recommender agent is able to make a larger amount of profit, while still making good recommendations.
Specifically, section 5.1 outlines the metrics over which we can evaluate our learning strategy, section 5.2 details the learning algorithm and the exploration strategy, and section 5.3evaluates the learning strategy against the metrics defined in section5.1.
Chapter 5 Learning Users’ Interests 78
5.1
Evaluation Metrics
To evaluate the learning strategy we use the following evaluation metrics (the first two are concerned with an individual learner’s performance and the second two with the performance of the collective of learners):
➊ Convergence to Optimality: Many learning algorithms come with a provable guar- antee of asymptotic convergence to optimal behaviour [Mitchell,1997]. This cri- terion is included here to evaluate the quality of learning itself; it is important because if an algorithm does not converge, the agent will have no incentive to follow its behaviour.
➋ Individual Rationality: See the third metric defined in section3.1.
➌ Quick Market Convergence: See the definition of “market convergence” in the fourth metric in section3.1. In the analysis of our recommender system in Chap- ter 4, we proved that convergence is necessary to ensure only the best items are displayed and that they are shortlisted in decreasing order of upq. Therefore, a market that converges quickly means that it starts satisfying the user quickly. This is clearly important since a user will stop using a recommender if it takes too long to produce good suggestions.
➍ Best Recommendation’s Identification: A good recommender system should be able to identify the best recommendation (the one with the highestupq) quickly and suggest it frequently [Konstan et al., 1997, Bohte et al., 2004]. This is im- portant because, otherwise, if the best recommendations cannot be identified and displayed, the user will stop using the system.
The first metric is new compared to those metrics listed in section 3.1. This metric is needed specifically to evaluate the learning performance which is why it has not been considered until now. The second and the third metrics are chosen from the same perspective of those listed in section 3.1. The second metric is used to evaluate the individual agents’ bidding behavior and the third is used to evaluate the overall performance of the marketplace as the coordinator of multiple recommenders. The fourth metric is also new because with this we can compare the system performance in
Chapter 5 Learning Users’ Interests 79
INQ
pay as bid if shortlisted
The circled numbers represent the order of the market process.
source recommendations
A User 2
UPQ
reward proportionally to the selected item’s
4
3
1 choose to recommend ?
Which segment should I
its quality a value of .
Other Recommender Agents
Recommendation A
I am satisfied with agent A’s
UPQ
recommendation and I rate
with highest price select M items The Marketplace segment 1 INQ segment G INQ INQ segment ... segment 3 INQ segment 2
Recommendationsother agents’
Recommendations shortlisted
Recommender Agent A
Figure 5.1: An Individual Agent’s Quality Classification Problem
terms of high quality recommendations between one with learning capability and one without.
With these metrics in place, we are going to design our learning strategy.