Experiments on TREC Genomics

Data Collection and Networks

We conducted relevant peer (expert) searches on the TREC Genomics 2004 collection. The task was to find an expert peer given a topic in a network of peers (representatives of scholars having document collections). To establish initial peer networks, we first chose six scholars in the medical informatics domain, i.e., associate editors of the Journal of the American Medical Informatics Association (JAMIA). We then identified their direct co-authors (1st _{degree) who published 10 to 80 articles in the TREC collection,}

resulting in a small network of 181 peers. The network was later extended to the 2nd

degree (i.e., co-authors’ co-authors) to total 5890 peers for experiments on a larger scale. 1 2 5 10 20 50 1 2 5 10 20 Y=63X^−1.2 k (out−degree) F(k): # occurances of a degree 1 2 5 10 20 50 100 200 500 1 5 10 50 100 500 Y=1367X^−1.5 k (out−degree) F(k): # occurances of a degree

(a) 181-Peer Network (b) 5890-Peer Network

Figure 6.16: Genomics 2004 Data: Degree Distributions

Both networks had a diameter (the longest of all shortest pairwise paths) of 8. Degree distributions of the networks are shown in Figure 6.16 (a) and (b). For each peer, which represented a scholar, all articles (with titles and abstracts) authored or

co-authored by the scholar were loaded as the local information collection.

Relevant Peer Search

On the TREC Genomics 2004 collection, we constructed peer-to-peer networks by treat- ing each unique scholar as a peer, who possessed a local collection of documents pub- lished by the scholar (author). The task involved finding a peer with relevant infor- mation in the network, given a query. Applications of this framework include, but are not limited to, distributed IR, P2P resource discovery, expert location in work set- tings, and reviewer finding in scholarly networks. However, we focused on the general decentralized search problem in large networked environments.

Relevant peers/agents were considered few, if not rare, given a particular informa- tion need. For experiments on the TREC Genomics 2004 collection, we considered those scholars whose topical similarity to a given query was ranked above the fifth percentile. Hence, for evaluation purposes, peers were sampled to estimate a threshold similarity score for each query, which was then used in experiments to judge whether a relevant peer had been found.

We retrieved citations to articles published in the Journal of the American Medical Informatics Association (JAMIA) in the Genomics track collection and used all (498) articles with titles and abstracts to simulate queries/submissions. For each submission, an agent that represented the editor in chief of JAMIA assigned it to one of the associate editors, who then began to forward the submission to a potential relevant agent/scholar through connected neighbors (e.g., co-authors).

Experimental Results

From experiments on the TREC Genomics 2004 data, we present effectiveness and efficiency results on initial and rewired networks of 181 and 5890 peers and focus on

the impact of network clustering on decentralized search performance.

Results on 181-Peer Network

0 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Max Path Length (# hops) Completion rate 181−peer network

rewired+SIM rewired+RW init+SIM init+RW 0 5 10 15 0.0 0.2 0.4 0.6 0.8 1.0

Average Path Length (# hops) Completion rate 181−peer network

rewired+SIM rewired+RW init+SIM init+RW

(a) Completion rate vs. Max Search Length (b) Completion rate vs. Average Search Length Figure 6.17: Effectiveness vs. Efficiency on 181-Agent Network

Figure 6.17 shows experimental results on 181-peers networks. The X axis denotes the efficiency (search path length) while Y is effectiveness (completion rate). Solid points refer to the SIM search method. Dotted lines are results based on the initial co-authorship network. With the initial network (dotted lines), similarity-based SIM search consistently outperformed random walks (RW), especially within small search path lengths. For instance, within two hops, SIM search already achieved a completion rate of more than 50% while random-walk was still at 20%. Allowing for longer search path lengths helped both models but neither reached a completion rate higher than 90%, suggesting that there were particular characteristics of the initial network that disoriented some searches after a long path.

that the association between connectivity frequency and topical distance has a power- law region (in the middle) with irregularities. We believe that SIM search was well guided by the network in most instances (when routed through peers with regular clustering-guided connections) but was lost in others (disoriented in regions where ir- regular connections dominated).

0.05 0.10 0.20 0.50 1 2 5 10 50 r: topical distance

f(r): # connected pairs with distance r

Y=0.027X^−3.0 0.01 0.05 0.20 0.50 1 5 50 500 r: topical distance

f(r): # connected pairs with distance r

Y=0.095X^−4.1

(a) 181-Agent Network (b) 5890-Agent Network

Figure 6.18: Clustering of Initial Genomics Networks: Connectivity frequency (Y) vs. Topical distance (X). Compare to Figure 3.3.

To demonstrate potential utility of network clustering, we rewired the network (network clustering) based on the connectivity probability function described in Sec- tion 4.2.3. Experimental results with clustering exponent α = 3.0 are shown as solid lines in Figure 6.17, in which proper network clustering better guided SIM search and further improved the results – a higher than 95% completion rate was already achieved at max search path length 20 (Figure 6.17 (a)) or average path length 5 (Figure 6.17 (b)).

0 10 20 30 40 0.0 0.2 0.4 0.6 0.8 1.0

Max Path Length (# hops) Completion rate 5890−peer network

rewired+SIM rewired+RW init+SIM init+RW 0 5 10 15 20 25 0.0 0.2 0.4 0.6 0.8 1.0

Average Path Length (# hops) Completion rate 5890−peer network

rewired+SIM rewired+RW init+SIM init+RW

(a) Completion rate vs. Max search length (b) Completion rate vs. Average search length Figure 6.19: Effectiveness vs. Efficiency on 5890-Agent Network

Results on 5890-Peer Network

On the initial 5890-peer network, experimental results indicated that SIM search had very limited advantage over random walk, as shown by dotted lines in Figures 6.19 (a) and (b). Further analysis revealed that the network was very weakly clustered. As shown in Figure 6.18 (b) on log/log coordinates, the correlation between connectivity and topical distance departed quite a bit from a power-law function (linear on log/log). There were many topically remote connections. Peers had many weak ties for a query to “jump” but insufficient strong ties to circulate the query within the boundary of a relevant neighborhood.

Again, we performed network clustering described in Section 4.2.3 to reconstruct/rewire the 5890-peer network. As shown by solid lines in Figure 6.19, given clustering expo- nentα= 4.0, theSIM search method performed much better and achieved above 90% completion rate within a max search path length of 40 (Figure 6.19 (a)), or an average search path length of about 10 (Figure 6.19 (b)).

Impact of Clustering

In the results above, we have demonstrated that some level of network clustering im- proved decentralized search for relevant peers. It is unclear yet how much clustering is enough or how much is too much. Setting max search path length at 10, experiments on SIM search with various clustering exponent α values on the 5890-peer network produced results shown in Figures 6.20 (a) and (b).

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 0.3 0.4 0.5 0.6 0.7 0 completion rate clustering exponent 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 6.0 6.5 7.0 7.5 8.0 8.5 9.0 0

search path length

clustering exponent

(a) Effectiveness (b) Efficiency

Figure 6.20: Impact of Clustering Exponentα(X)

Figure 6.20 shows that the SIM search method achieved best performance, i.e., high- est completion rate in (a) and shortest search path length in (b), at α ≈ 3.5. Both smaller and larger α values resulted in less optimal searches. As discussed, smaller α

values produced less visible topical segments and more remote connections that dis- oriented searches. Larger α values, on the other hand, led to an over-clustered and fragmented network without sufficient weak tiesfor searches to move fast.

This result, obtained in a decentralized relevant peer searching context, is consistent with findings from relevance search, authority search, and exact match experiments on the ClueWeb09B collection. It continues to support hypothesis 1 regarding the

Clustering Paradox, in which some balance betweenstrong ties andweak tiesshould be maintained for effective and efficient searches.