ISPASS-2009 Tutorial Proposal
Archer: Zero-configuration Virtual Appliances for Architecture Simulation
Tutorial audience and goals:
This tutorial targets computer architecture researchers and students interested in:
accessing high-throughput computing resources to run architecture simulations; better utilizing their own resources within a group; and/or sharing computer architecture
applications and datasets. The tutorial will use hands-on activities to show users how they can gain access and use a shared distributed computing infrastructure (the Archer system) and how they can leverage its virtual machine appliances to manage local resource pools.
(More information about Archer can be found at http://archer-project.org and in the attached flier).
Organizers:
The main organizer of this tutorial is Renato Figueiredo (University of Florida), in collaboration with the Archer project PIs:
- Jih-Kwon Peir, Jose A. B. Fortes, Tao Li, P. Oscar Boykin (U. of Florida) - Gary Tyson (Florida State University)
- Lizy K. John (University of Texas at Austin) - David Kaeli (Northeastern University) - David Lilja (University of Minnesota) - Gokhan Memik (Northwestern University) - Sally McKee (Chalmers)
- Alain Roy (University of Wisconsin)
Bio: Renato Figueiredo is an Associate Professor with the Advanced Computing and Information Systems (ACIS) Laboratory and the Center for Autonomic Computing at the University of Florida. He received the B.S. and M.S. degrees in Electrical Engineering from the University of Campinas in 1994 and 1995, respectively, and the Ph.D. degree in Electrical and Computer Engineering from Purdue University in 2001. From 2001 until 2002 he was with the faculty of the School of Electrical and Computer Engineering, Northwestern University. In 2002 he joined the Department of Electrical and Computer Engineering of the University of Florida. Figueiredo is the PI of the Archer project, a virtualized shared wide-area infrastructure for computer architecture research and education. In addition to Archer, he has participated in various research projects which apply machine, network and file system virtualization in support of high-performance and high-throughput computing in wide-area distributed systems. His broad research interests are in the areas of resource virtualization, computer architecture, distributed systems, network overlays, and autonomic computing.
Presenters:
The main presenter of the tutorial will be Renato Figueiredo. He will be assisted by one or two Archer staff or graduate students for questions/answers and troubleshooting during hands-on activities.
Abstract:
Modern computer architecture research is driven by quantitative analysis. Leading-edge research requires detailed, cycle-accurate evaluation of many benchmark applications with several simulated configurations and is thus tightly dependent on the availability of high-throughput computing (HTC) systems. However, many research groups are
hindered in their ability to perform research because of lack of access to computational resources, tools and datasets. This tutorial presents Archer (http://archer-project.org), a community-based computing resource for computer architecture research and education.
Archer integrates technologies for resource virtualization, batch job schedulers, and multi-institution collaboration, in order to create a collaborative wide-area infrastructure with the following features. (1) Scalability: Archer starts from a seed set of hundreds of processor cores across distributed cluster resources. Subsequently, each new user joining Archer with one or more desktops or servers seamlessly contributes to its aggregate capacity. The system uses Wisconsin’s Condor (http://www.cs.wisc.edu/condor), a robust batch scheduler that scales to large resource pools (2) Ease of use: Archer leverages packaging and distribution of open-source software environments as self-configuring virtual networks of virtual appliances, which can easily be installed by individual users in their own resources using contemporary, freely available VM software. (3) Support for unmodified applications and community sharing: Archer runs unmodified Linux-based full-fledged open-source and commercial software simulation environments (such as Simics, SESC, SimpleScalar, Fes2, PTLsim) and features a Wiki and plug-and-play NFS distributed file systems so users can easily share application executables, support scripts, input and output data sets, tutorials, and manuals. In doing so, Archer facilitates the dissemination of useful tools and data sets, and foster creation of reproducible simulation experiments.
Duration:
This is a half-day tutorial, consisting of a brief technical introduction to the Archer cyber- infrastructure and hands-on, interactive access to the Archer infrastructure by attendees using virtual appliances distributed during the tutorial, as described below.
Topics:
- Presentation (20 minutes):
o Brief overview of core technologies in Archer:
Contemporary virtual machine monitors, virtual appliances
Virtual private networks
Condor job scheduler
o Putting it all together: introduction to the Archer distributed system
- Hands-on session (30 minutes):
o Archer appliance installation
o Basic user interaction with the appliance
User interface
Data transfers
“Hello world” job submission
- Presentation (20 minutes):
o Software installation and data sharing:
Virtual disk organization
Software installation; creation of custom appliance modules
Data sharing using NFS (Network File System)
Case study: Simics
- Hands-on session (30 minutes):
o Running Simics in interactive mode o Making datasets available via NFS o Running Simics in batch mode
- Presentation (20 minutes):
o Managing jobs with Condor
Vanilla and standard jobs
Job configuration files
Specifying job requirements
Introduction to workflows
Job monitoring, priorities
- Hands-on session (30 minutes) o Compiling SESC for Condor
o Creation and submission of parameter-sweep SESC batch o Creation and submission of workflow job
o Job monitoring and management
- Presentation (20 minutes):
o Deployment on clusters and desktop pools
Joining the global pool
Private/group pools; security
Opportunistic scheduling on desktop resources
- Hands-on session (30 minutes) o Deploying a private resource pool
Tutorial requirements:
This tutorial requires broadband wireless Internet access in the room for all attendees.
Tutorial participants are required to bring a laptop if they wish to follow the hands-on component of the tutorial:
- x86-based processor
- 512MB RAM minimum, 1GB or more suggested - Windows, MacOS or Linux
- VirtualBox or VMware VMs and the Archer virtual appliance image – instructions will be given on the tutorial Web site for those who want to pre- install the software; we will also bring all software in CDs and USB sticks.
Previous tutorials:
Hands-on tutorials covering the virtual appliance-based Condor infrastructure underlying Archer (but without emphasis on computer architecture applications) have been given in the following venues:
- CollaborateCom’2008 (half-day tutorial; ~10 attendees)
- SuperComputing’2007 (hands-on tutorial part of a half-day education program session; ~15 attendees)
- TeraGrid Conference’2007 (hands-on tutorial part of a half-day session; ~10 attendees)
- St. John’s River Water Management District, 2007 (half-day tutorial, ~10 attendees)
References:
[1] R. Figueiredo, P. O. Boykin, J. Fortes, T. Li, J-K. Peir, D. Wolinsky, L. John, D.
Kaeli, D. Lilja, S. McKee, G. Memik, A. Roy, G. Tyson, 'Archer: A Community
Distributed Computing Infrastructure for Computer Architecture Research and Education'.
In 4th International Conference on Collaborative Computing (CollaborateCom 2008), 11/2008
[2] D. Wolinsky, R. Figueiredo, ‘Simplifying Resource Sharing in Voluntary Grid Computing with the Grid Appliance’. In 2nd Workshop on Desktop Grids and Volunteer Computing Systems (PCGrid 2008).
Flexible deployment options:
Archer Express: For first time users.
Quickly gain access to a small-scale pool of resources within ~ 30 minutes to test-drive Archer capabilities on your own.
Archer Global: For academic research and education use. Gain access to a large-scale world-wide pool of CPU resources, shared tools and datasets.
Effortlessly contribute with your own resources to the community when idle.
Archer Local: For deployment of private Archer pools on your own local- area resources (clusters, servers, desktops).
Information Technology Solutions
V I R T U A L C L U S T E R S Archer builds upon virtualized resources – virtual machines and overlay virtual networks – to create scalable virtual clusters over wide-area Internet resources. VM appliances and self-
configuring IP-over-P2P virtual network software enables Archer nodes to be easily deployed by end users.
C O N D O R
Archer virtual clusters run the Condor batch job scheduler, a robust Grid middleware that supports typical computer architecture simulation workloads – high-throughput execution of independent jobs, such as parametric sweeps.
C O L L A B O R A T I O N T O O L S Archer provides a Wiki to share documents and tutorials, auto-configured NFS file systems to share datasets and tools, and user groups for questions and support.
Contribute to the community by sharing your own research and education content such as tools, scripts, tutorials, class assignments and datasets.
archer-project.org
Archer: a shared cyberinfrastructure for
computer architecture research and
education
The Archer project is building a distributed, open community resource for simulation- based computer architecture research and education. Computer architecture researchers and students world-wide are invited to participate in this cyber-infrastructure, gaining access to high-throughput computing pools with easy-to-deploy architecture tools, and sharing their own idle computer cycles, applications, data sets and educational material.
This work is sponsored by the National Science Foundation under CRI collaborative awards 0751112, 0750847, 0750851, 0750852, 0750860, 0750868, 0750884, and 0751091.
Easy steps to get started:
1. Install a free VM monitor (VMware Player, VirtualBox, or KVM) in your desktop - x86-based Windows, Linux, or MacOS.
2. Go to archer-project.org, download and run the Archer Express Grid appliance image
3. Follow the “quick start” tutorial to submit your first simulation run 2. Boot appliance:
automatically joins Archer pool
Free pre-packaged Archer Virtual appliances - run on free VMMs (VMware, VirtualBox, KVM)
Portal and Wiki:
Community-contributed content: applications, datasets, tutorials
Archer seed resources 300+ cores – Fall 2008 Archer Global
Virtual Network
System software:
Condor scheduler NFS file systems Simulators:
Simics, SESC, Simplescalar 1: Download
appliance
3. Run architecture simulation jobs on the Archer pool through Condor 2. Boot appliance:
automatically joins Archer pool
Free pre-packaged Archer Virtual appliances - run on free VMMs (VMware, VirtualBox, KVM)
Portal and Wiki:
Community-contributed content: applications, datasets, tutorials
Archer seed resources 300+ cores – Fall 2008 Archer Global
Virtual Network
System software:
Condor scheduler NFS file systems Simulators:
Simics, SESC, Simplescalar 1: Download
appliance
3. Run architecture simulation jobs on the Archer pool through Condor
A community cyberinfrastructure:
Shared distributed computers: Archer makes high-throughput computing accessible to the computer architecture community at large. Seeded with hundreds of CPUs across a distributed system (to be deployed over three years), the
infrastructure grows as a community effort because users can effortlessly contribute with resources of their own to the community when they are idle.
Shared architecture content: Archer facilitates the sharing of tools, datasets, self-contained experimental setups, and educational materials. Sharing of tools and experiments is facilitated by providing a consistent virtual environment to all Archer users and automatically-configured NFS file systems. Sharing and dissemination of content is facilitated by the Archer Wiki.
P A R T I C I P A T I N G I N S T I T U T I O N S
Archer is a collaborative effort led by University of Florida, Florida State University, Northeastern University, University of Texas at Austin, Northwestern University, Cornell University, University of Minnesota and University of Wisconsin-Madison.
A P P L I C A T I O N S
Simulation tools widely-used in the community are available in Archer, including SESC, Simics, SimpleScalar, and PTLSim. In the typical case, Linux 32-bit x86 binaries will work on Archer, unmodified.
M A N A G E M E N T A N D U S E R S U P P O R T
Management staff develops and maintains the Archer virtual machine appliance image so Archer users can focus on the science rather than the infrastructure.
Management staff also provides support through user mailing lists.
What are the software licenses used in Archer?
The core of the Archer middleware is open-source (GPL and Apache2 licenses). Applications include both open- source packages (e.g. SESC) and simulators with academic licenses available to Archer users (e.g. Simics).
Who can use Archer? Is it free?
Archer is available and free for use by computer architecture researchers and educations within the US and abroad. Usage of the Archer Global pool is limited to academic use only; Archer middleware may be used to deploy Archer Local pools for non-academic purposes.
Can I use Archer to create an isolated, private Condor pool in my own resources?
Yes, you can use the VM appliances to deploy Archer Local pools, which are independent from the shared pool.
However, Archer staff will not be able to assist with Archer Local management.
What security mechanisms are provided?
All jobs run within a virtual machine sandbox. Access control is mediated by Condor, which means users do not have direct access (e.g. ssh) to a VM, except through Condor. Remote jobs run as UNIX user "nobody" within a VM sandbox. Archer jobs can only send packets to other virtual machines within the Archer virtual network.
Packets sent to other Internet hosts are blocked by a firewall. Archer confines its traffic to the virtual network through firewalls to avoid the use of Archer to launch denial-of-service attacks. All virtual network IP traffic between Archer VMs is authenticated and encrypted.
Frequently Asked Questions
Archer: a shared cyberinfrastructure for
computer architecture research and
education
The Archer technology targets simple deployment and use:
Pre-packaged VM appliances: Archer packages all the software needed to run architecture simulation jobs conveniently. It uses freely available virtual machine software to create self-configuring, plug- and-play virtual appliances and the robust Condor batch job scheduler. Archer VM appliances enable functionality similar to systems such as SETI@Home, but support unmodified applications and provide sandboxing of remote jobs through the use of virtualization.
The Archer software makes it very easy for computer architects to deploy their own Linux Condor pools on their own resources (for example, instructional lab PCs). A small local pool can be brought up in a matter of hours.
Advanced Computing and Information Systems Lab http://www.acis.ufl.edu
