Java Bit Torrent Client
Hemapani Perera, Eran Chinthaka {hperera, echintha}@cs.indiana.edu
Computer Science Department Indiana University
Introduction
World-wide-web, WWW, is designed to access and download remote resources. But this downloading of resources becomes problematic when the size of the resource increases. Bit torrent, the peer-to-peer file sharing protocol, initially developed by Bram Cohen is slowly becoming popular as a means of sharing files over the web. The main difference of the protocol lies with the sharing method. Bit-torrent takes a large file and split in to large number of chunks and makes them available on different peers. A person, who is interested in downloading a resource, can connect to these peers who announce themselves with the pieces that they have to be downloaded. This approach provides the advantage that no single peer is overloaded with all the connections, as the downloading is distributed across large number of peers. At the same time the downloader can simultaneously connect to large number of peers and get the file much faster than getting if from a
single location.
This report presents a java based client to share files using Bit torrent protocol [1]. This reports share our experiences developing the client and presents test matrices we collect while tuning different parts of the implementation.
Design
Bit torrent system starts with starting TorrentManager, which maintains all its states in TorrentContext. TorrentManager is responsible mainly for;
Managing the connected peers using PeerInstance objects
Getting proper pieces effectively
Communicating with trackers, using TrackerUpdater, as and when it is required
Managing the integrity and persistence of downloaded pieces, with StorageManagers
Keep track of any timed out request and resend or reschedule them as needed
An instance of PeerInstance class is created per each connection and handles all the communication related to its assigned peer. Peer instance does the handshake at the initialization and create a thread which keeps listening to incoming data. It has implementations for all message interactions defined in the specification and also responsible for the following.
listen for incoming messages and act on those messages (e.g. Serve a requests)
update the shared data structures based on the incoming messages
Figure 1: High level architecture
Figure 1 presents a high level view of the architecture. Each peer instance listens to the
incoming connection and updates the shared data structures. Torrent managers runs a control loop which process the data stored within shared data structures and issue high level commands to peer instances, to manage the download and upload process.
Following describes the internal working of the system using high level pseudo-code.
1. System starts up with TorrentManager getting configurations stored within a property file.
2. TorrentManager initializes creating the data structure for the torrent meta info file
3. TorrentManager initiates the communication with the tracker and gets back the peer list
4. Decides on live peers by probing each one of them and decides on initial pool of peers to be contacted
5. 5. TorrentManager handshakes with each of the peers, using PeerInstances
6. Having received the pieces information from the peers, TorrentManager decides on an optimal peers based on the peer selection policy (e.g. by taking rarest available pieces in to consideration) 7. TorrentManager notifies proper
PeerInstances with its interest to download pieces from them
8. When choking messages are exchanged, TorrentManager request pieced from un-choked peers, through PeerInstances. 9. Pieces are requested as blocks, and when
a response to requested block(s) is received, the response to next block is sent by peer instance. This is handled without direct intervention from the Torrent Manager. Having downloaded the piece, PeerInstances notifies the TorrentManager.
10.TorrentManager initiates the StorageManager to save the piece, after the hash check
11.TorrentManager continuously evaluates on the optimal peers to connect to download pieces, and request the pieces
appropriately until the system completes downloading all the pieces.
12.Having downloaded the pieces and while downloading the pieces, TorrentManager also act as a seed for other peers.
Implementation
Bit torrent client implementation we have is tested with up to 9 concurrent peers and one seeder. Also it inter-operates will widely used bit-torrent clients like Azureus [2]. Implementation includes following features.
1. The maximum number of connected peers and maximum number of concurrent downloads are configurable and controlled
2. It implements two peer selection algorithms as explained in peer selection section of performance evaluation
3. It includes an upload queue which is used to control upload rate is controlled. The upload rate can be controlled by changing the configurations
4. It includes two policies for request pipelining, a sliding window based and batch mode. More information is given in the performance evaluation section. 5. Manager monitors the requests and each
request (e.g. request messages, interested message) has associated timestamp. Manager monitors them and re-sends or reschedules timeout requests. As a result clients are resilient to a fair amount of errors and able to recover from failure of other clients.
Table 1 presents a piece download distribution of a test run with 9 peers and a one seed. As clearly shown by the table, the torrent download is distributed across the peers, and some peers download from up to 6-7 peers to get the pieces. Data in the table proves further that file sharing indeed reduces the load on individual peers and distributes it across the connected nodes. (This test can be recreated by running the
JUnit test case included in test/ManyPeersTest.java in the submission.)
Pieces Numbers Downloaded From
Peer
Name
Seed 0 1 2 3 4 5 6 7 8
0 0 1 3 5 9
2 6
4 8
1 2 4 6 9 3
0 1 7
8 5
2 6 8 7 0 3 5 9
4 2
1
3 1 9 3 8 7 2 6
5 0 4
4 0 1 5 6 9 4 8
3 2 7
5 5 4 1 0 9 8 7 6
2 3
6 2 8 5 1 6 4 3 9
0 7
7 4 5 9 8 7 6 3 1
2 0
8 2 6 4 3 5 9
1 8
0 7
Table 1 : Piece distribution across available peers, during a torrent download
Interoperability
We were able to successfully inter-operate with bit-torrent client Azureus [2]. We tested using the Azureus client as a seed and as well as a client in two different test runs.
Performance Evaluation
We added an initial wait and perform a re-update from the tracker to make sure all peers know about other peers in the system before the test begins. For each peer we measured the average download rate (total data downloaded/time taken). This section presents the test matrices we collected to verify the hypothesis formed around different parts of the Bit torrent client operation.
Peer Selection
We implemented two approaches for peer selection. At the start up, the peer connects to all known peers and retrieves their bit-fields. Then at each step maintains an N number of parallel connections to available peers. The two approaches implemented were two set of policies for selecting N out of all connected peers. Our two approaches are given below.
1. Select the N peers who have most number of interesting (Pieces the peer does not have) pieces
2. Rarest First - Select N peers who have rarest pieces.
We varied the maximum allowed busy peer count (N) and plotted its effect on the download rate. Figure 2 summarizes our observations. Initially with 1 peer to connect at a given time, both starts with around 100KB/s and rarest first seem to have a more stable curve. Connecting to maximum interested peers had a sudden spike, but went down.
With the rarest piece first approach the pieces will be distributed among the peers very fast and allow for more parallelism later on. We believed this approach is healthy for peer group in general. Even though this data is not sufficient to draw any firm conclusions in favor of either our observations lean towards rarest first policy.
Figure 2 - Peer Performance against different peer selection algorithms
Pipelining
Our initial implementation did wait on every request, in other words, the peer would sends a request to another peer and wait for the response to arrive to send the next request. However it is possible to do request pipelining so that peer shall send N requests without waiting for the destination peer to respond. There are two methods for a peer to send N requests. One is to wait for all N to complete or to maintain an N long sliding window of pending requests.
We implemented both the approaches a took observes their effect on download rates by running both approaches while increasing the size of N. Figure 3 summarizes our observations. For the tests we allowed peers to connect to any number of peers, therefore starting value for simplest implementation (N =1) was around 400Kb/s. As shown clearly by the figure both forms of pipelining have improved the throughput. But pipelining with sliding window does slightly better than batch mode. That is expected as batch mode holds the request messages till batches completes and sliding window responds once a response is available.
Figure 3 - Peer Performance against different pipelining algorithms
Also we could see that the download rate saturates around 800-900 KB/s and that was observed across all the test matrices we took. Optimal window size, N, was around 3-4 and curves seems to turn back slightly when the window size was improved.
Figure 4 demonstrates the affect of maximum concurrent downloads on the set of five peers. Set of five peers were selected and ran by setting different values for maximum concurrent download limit. The five graphs show how their download rates vary. The peers are named after the order they are started. We observed the peers started later usually do well because they have more opportunity to do concurrent download from the existing peers, compared to peers start early.
Affect of Number of Peers on
Download Rate
Following graphs show the affect of number of peers on the download rate. As shown clearly by
Figure 4 - Effect of maximum concurrent downloads on download rates
Figure 5 - Effect of peer availability on download rates
the graph, average download rate increases proportional to the number of available peers.
However the effect of that on single peer is mixed, where some peers quickly achieve saturation and some showed small or no improvement. Also we noticed in all our implementations that download rate saturates about 800Kb/s and in a single peer setting it is about 100-200Kb/s.
Test Environment
For our tests we used two machines one having 4 peers and the seeder and the other having 5 peers. We did the test on silo.cs.indiana.edu and hager.cs.indiana.edu. Silo is a 2-processor 2.8GHz Intel Xeon system running Red Hat Enterprise Linux with 8GB of memory. Hagar is a 2-processor 3.06GHz Intel Xeon system running Red Hat Enterprise Linux with 4GB of memory.
Conclusion
The results on our experiment clearly show that file sharing using bit-torrent protocol indeed reduces the load on individual peers. And in addition to that it has improved the total download times of each individual peers to a very good extent. This can be further improved by looking at some other methods like, getting blocks of the same piece from different peers. The individual algorithms that can be used to tune each and every part of the system provide the users a means to optimize their download rates, depending on the environment. At the same time, this will enable most of the companies struggling to maintain their web sites, due to high bandwidth requirements, to safely reduce the cost and to maintain a high up-and-running time.
References
[1] : Bit torrent protocol -
http://www.bittorrent.org/protocol.html [2] : Azureus bit-torrent client -
