Universal Remote Boot and Administration Service
Sebastian Schmelzer, Dirk von Suchodoletz
and Gerhard Schneider
Faculty for Applied Sciences Albert-Ludwigs-Universit¨at
Freiburg, Germany
Email:{ss9286,dsuchod,gjas}@rz.uni-freiburg.de
Daniel Weingaertner
1, Luis Carlos E. de Bona
1and Carlos Carvalho
2Informatic1 and Physics2 Department
Universidade Federal do Paran´a Curitiba, Brazil
Email:{danielw,bona}@inf.ufpr.br [email protected]
Abstract—Booting operating systems over the network is a well established method to simplify system administration in medium to large computer environments. Remote booting Linux and using PXE based environments for large scale system deployments are widely adopted by many system administrators. While scaling very well in controlled subnets they can not easily cross sub-network boundaries or co-exist with each other. Additionally it would be desirable to offer a wider range of standard services not only in dedicated sub-networks but on the whole enterprise infrastructure. This paper discusses advanced network booting methods overcoming the restrictions of traditional PXE/DHCP/TFTP setups. It suggests a new approach combining existing well established boot services into a more flexible framework for remote booting a wide range of operating systems and maintanance tools. A centralized boot configuration web-service is the universal, multi-purpose entry point of the architecture offering configurable, flexible boot menus for directly managed and arbitrary computers of unregistered users.
I. MOTIVATION
Enterprise desktop computer and cluster management re-quires high personnel costs that easily surpass the figures for hard- and software purchases. Thus, reducing the Total Cost of Ownership (TCO) remains a central challenge in most organizations [1], [2]. Since the era of mainframe computing the IT infrastructure has had a massive tendency to de-centralize. Attractive in the beginning because of low PC prices, it became a nightmare with the increasing number of machines (and unskilled users) to supervise. At the same time fast network infrastructures became ubiquious and every machine got hooked up to a data network.
Industry solutions like Intel PXE [3] brought the idea of remote starting to the PC domain, helping to recentralize some crucial services like operating system provisioning and booting. In order to be independent of a properly configured OS and remote access applications to actually access a ma-chine, Intel and others introduced the hardware-based “Active Management Technology”1 to provide maintenance over the network. Nevertheless, those services are restricted to man-aged and well-defined sub-networks and industry-grade servers as well as desktop computers. As the mainstream desktop computer and laptop are still typically rather cheap stand-alone machines with low initial but high maintanance costs,
1See for vProhttp://www.intel.com/technology/vpro
organizations need management infrastructures that focus on this type of equipment.
This paper presents a more generic approach to system management over a Wide Area Network (WAN) that is highly independent of the local subnet configurations and expensive hardware additions. The aim is to provide often requested network and computer management services through a web service that allows for easy selection and configuration of booting options for each computer. The primary task of the management service is to provide an environment for the setup and automatic or manual selection of booting images. By doing so, any high speed network could serve as a base for remotely booted standard services such as lecture rooms, conference systems, library terminals or public web information kiosks [1].
This paper is structured as followed: First, we illustrate the system overview from a network administrators view point and place existing Linux remote boot solutions within it. Second we extend the restricted PXE-based approach to a more general WAN boot solution as well as it’s software components and give an introduction to our general system architecture. Third, we describe the existing components and discuss current developments. Finally, we give some preliminary results of our experiences and talk of upcoming developments.
II. SYSTEMOVERVIEW
The proposed architecture provides a base for troubleshoot-ing, emergency boottroubleshoot-ing, disk recovery, virus checktroubleshoot-ing, hard-ware testing and backing up machines, without being depen-dent on a locally installed operating system or a properly configured PXE/TFTP setup. The architecture could easily be extended to a cloud computing environment that allows users to choose easily among a variety of predefined booting images to run in the virtually provisioned machines in order to fit current needs. Another possibility is to generate on-demand boot images adjusted to the user’s local configurations.
Our architecture implementation is based on existing and well-established standard open source solutions like kexec,
iPXE2 andgrub4dosand combines them with different Linux
remote booting solutions developed in other projects like
2Open Source Project that emerged from Etherboot and gPXE, see [4].
Fig. 1: System architecture overview
Paran´a Digital or OpenSLX3. Two new components were designed and implemented: the PreBootLinux (PBL), our replacement for the PXE functionality extending the goal of [5], and the Wan Boot Manager (WBM), which provides a delegable system administration on a higher level than the local IP and DHCP/DNS configuration (Fig. 1). The system proposed does not interfere with locally installed OS and is compatible with pre-existing remote boot and software deployment configurations. Thus it could easily serve as a recovery, fall-back and temporary boot option for any average X86 machine. The limitation of having a BIOS provided PXE available is omitted as instead nearly any boot medium could be chosen for booting the PreBoot environment (including PXE again). The architecture discussed in this paper signif-icantly extends the preexisting remote boot functionality of PXE.
III. STATE OF THEART– LINUXREMOTEBOOTING The concept of remote booting has been around for over 20 years and has proven to be a very successful strategy for managing large scale systems efficiently. Due to the fact that no configuration data is kept locally, the machines don’t even have to have a hard disk installed. Prominent examples are the LTSP4 and its predecessors, starting at the end of the 1990s. Adaptations of the basic concept like the Paran´a Digital project (PRD) [8] use a centralized boot environment focusing on well-defined and limited LAN infrastructures; projects like FlexLab [2] in Sao Paulo or the SOHO gPXE Desktop Solution proposed in [1] aim at high-speed WAN setups.
The LTSP and PRD typically implement a server that provides the booting image to several clients on a local
3OpenSLX Project Homepage:http://www.openslx.org
4Linux Terminal Server Project http://www.ltsp.org and the OpenSLX
project [7]
network using PXE/TFTP/DHCP. PRD is deployed in more than 2000 public schools in the State of Paran´a, Brazil, and builds upon the concept of centralized administration of the servers and local administration of specific non-critical tasks like user accounts, printer management, or the addition of new computers to the network. System upgrades are defined by the central core managers and propagated to all the servers, which in turn pass them on to the clients.
While LTSP and PRD are thin-client terminal-server so-lutions, the OpenSLX project uses a regular computer with local processing. The client boots a complete Linux operating system over the LAN from a central rootfilesystem server. The user logs onto the local machine and executes his applications locally. Thus he can utilize the full power of the client’s hardware without interfering with other users, as it is the case on a shared terminal server. As all relevant processes run locally the access to all kinds of removable media and peripheral device connectors is easier to accomplish. This OpenSLX full-desktop feature allows local virtualization and thus sessions of other operating systems running on top of the diskless Linux session – comparable to the solution suggested by Bona et al. [2]. The deployment of virtual machine images from a central fileserver helps to significantly reduce the effort of running a large number of Windows desktops or Linux installations with priviledged user access for special lectures. Additionally, the OpenSLX framework, as an easy-to-maintain OS provisioning middleware, is evaluated for the Black Forest Grid5. Beside being used on cluster nodes the dynamic boot is also suited for the deployment of cloud infrastructures.
Independent of the system chosen, in every project the hardware and software setup of each client is handled during boot time. Thus every client is “stateless” and all differences in the setup are placed on the ramdisk of the client, thus
circum-5Running more than 200 cluster compute nodes, see http://www.bfg.uni-freiburg.de
venting the storage of any local configuration and user data. The local disk does not need to be used in standard operation, avoiding interference with locally installed operating systems. Thus by now a number of software packages for remote booting of stateless client are available, helping system ad-ministrators solve a significant proportion of their standard management tasks. To enable such packages for a more wide-spread application, we developed the PreBoot Linux (PBL) overcoming several restrictions, like the dependency on PXE-capable BIOS or proper TFTP/DHCP configuration rights on the network [1]. Even if the built-in Network Bootstrap Program (like PXE) could be extended to handle more reliable transfers (i.e. HTTP instead of TFTP), its capabilities for flexible and configurable user interaction depends on a number of network and location parameters. Besides, the support for SSL-secured transfers or the deployment of the upcoming IPv6 is very limited in the traditonal setups. An attempt would have been to use iPXE to overcome PXE’s limitations but with further extensions in mind – like support for VPN or signed kernel and initramfs images – the usage of a minimal Linux system was prefered at this stage of development. Once the PBL is in a well tested, final state an implementation based on UEFI [9] might be conceivable.
IV. FROMPXETOGENERALREMOTEBOOTING Many of the aforementioned projects aim at the local LAN, where a system administrator has full control of the DHCP, TFTP and router setup of the domain he is managing. This works for a properly defined environment but is not flexible enough if a wider range of quite different machines running operating systems for different purposes are to be maintained. The LAN boot solutions make a considerable amount of assumptions about the network setup. For WAN booting envisioned in the cited projects the Internet providers need tightly to cooperate [1].
The several PXE-based remote server pre-configuration data center solutions like [10], software provisioning, installation and booting software packages are often mutually exclusive. Thus it is rather difficult to run a PXE-based Windows deployment and update service on the same subnet as a Linux remote booting solution. Both depend on special variables configured in the DHCP server and different files stored on the TFTP server. Changing the standard boot of a single machine or a group of machines is difficult as it requires privileged access to one or more servers. PXE chainbooting does not necessarily function with all the different variants of boot media and bootable systems. To separate administrative domains, different boot servers need to be configured and a number of boot menus need to be created, which is often not very practical. From a certain point on it becomes confusing for both users and administrators.
Additionally, there are no PXE or similar implementations available for Ethernet alternatives like WLAN, Infiniband or Firewire booting.
A. Flexible PXE Replacement
As PXE couldn’t be easily adapted to a more abstract level and to overcome the restrictions mentioned above, a more generic approach has been chosen. Instead of having the rather proprietary PXE environment combined with PXElinux [4] or similar commercial programs, a Mini Linux in combination
with kexec reproduces the ”pre-booting” environments of
common NBPs [11], including menus and user interaction. The Mini Linux consists of a kernel containing all necessary network modules for IP connectivity and an initial ramfs. The latter uses the Debian eglibc, contains tools like Busy-box and an environment for user interaction. The interaction environment could be any curses based user interface for small footprint or much more flexible graphical user interfaces using a framework like Qt [12]. The latter one offers a webkit engine allowing the lightwight and flexible implementation of user interfaces. Thus all of menu interaction with the user could be handled from the server side keeping the client side small and flexible. It adds just around 20 MBytes with all required components to the initram filesystem. The major component is kexec; it can do flexible machine warm starts of different operating systems and chainload bootloaders like iPXE or
grub4dos to extend the number of possible targets:
• PBL⇒kexec⇒local or network Linux which replicates
the standard Linux remote boot functionality in arbitrary PXE/TFTP un- or differently managed networks
• PBL ⇒ kexec ⇒ special purpose systems like Linux
network media installation or system administration tools (from anywhere, WAN)
• PBL ⇒ kexec ⇒ grub4dos ⇒ chainloading local
boot-loaders like Windows, BSD or other systems from differ-ent local disk partitions
• PBL ⇒kexec⇒ iPXE⇒ PXE based systems like RIS
or other proprietary systems
• PBL⇒kexec⇒low level tools likememtestor harddisk
utilities.
V. GENERALSYSTEMARCHITECTURE
The system’s architecture follows a very flexible, central idea: to allow for comfortable user interaction, upon boot, alowing pre-selections for the desired system, as well as no interaction at all, with all boot options automatically chosen.
The prerequisites are very moderate:
• HTTP access to the central WBM server to load kernels, init ramfs, configurations and other boot components like
memtest
• Root filesystem access for stateless Linux if not booting further intogrub4dos
To implement the architecture we have developed a num-ber of new components and integrated existing projects like OpenSLX and Paran´a Digital.
A. Core Components
The main purpose of the PreBoot Linux is to replace and extend the functionality of a classic PXE menu-based
system (see Fig. 1 (2)). It consists of a small, optimized Linux Kernel with support for the most common network cards and a Busybox-based6 initial ramfs. It can be delivered/provided using all bootable media, such as USB sticks, CDs, harddrives or even PXE. The initial ramfs contains (besides the standard Linux tools for the network/system setup) components that allow a graphical user interaction. This GUI is based on a customized browser component, which makes the extension of the user interface independent of the actual content of the boot media, because the logic is provided by the central configuration server. A graphical interaction in order to choose the bootable system is optional (see Fig. 1 (4)). The task of the PreBoot Linux NBP is finished with the execution of kexec, which starts the boot process of the final stateless system. As in PXE, the PBL is meant to be static and have no need of being updated, unless there are changes on the suported hardware. This was achieved by the separation into a local web browser and remotely provided content and logic.
Fig. 2: Screenshot of the WAN Boot Manager configuration interface
The WAN Boot Manager configuration web service is a new component of our system management architecture (Fig. 1 (3)). The boot option pre-selection is moved to this central server which is independent of local DHCP and other configuration realms. Its main tasks are the multi-user capable, web-based management of bootable systems and the interface for the PBL. Additionally, the WBM honours multi-tenancy.
The management component allows the administrator to prepare, configure and publish bootable systems as well as to manage the machines (set configuration options and boot defaults) and users belonging to his realm. Unprivileged users (e.g., users using the provided bootable systems) are able to use a reduced set of configuration options for their local machine. The interface to the client – more precisely to the PBL – is responsible for providing the logic and the interface for the selection of the bootable media, which can be
6Busybox,http://busybox.netis the standard of compact implementation of
a number of standard system utilities.
interactive, through the framebuffer-based GUI, or automatic, in which case a predefined system is booted directly.
The Rootfs Server (currently under development) which extends the traditional OpenSLX or PRD server, is responsible for sharing the root filesystems of diskless Linux systems over network blockdevices and filesystems (Fig. 1 (6)). It will contain its own web-based management frontend and an interface for interconnection with proxies and syncing of the root filesystems.
B. Variants of Booting a Machine
Booting a machine over the network differs slightly from starting it from a local harddisk or optical media. The Fig. 1 (steps 1 to 4) depicts the modified boot process featuring the PBS and kexec before starting into the desired remote Linux OS or into a configuredgrub4dosor similar first stage bootloader (step 5). After turned on, the machines’ BIOS searches for a bootable medium, which could be a USB stick, a removable medium, or any bootable device containing the PBL system (step 1). The PBL is loaded from this media and after the automatic network setup (which only requires an arbitrary DHCP service), the PBL connects to the central WBM (steps 2, 3). Depending on the configuration for that machine stored on the WBM, it either boots a preselected OS or the user chooses from a pre-configured boot menu (step 4). The latter method uses a framebuffer based graphical user interface fed by a web-service. Both variants fetch the next stage system kernels, initramfs or bootloader together with their configuration from the network and start it with kexec.
After this step the traditional booting kicks in: It loads either the first stage bootloader or the desired Linux system (step 5). In case of booting Linux using PRD or OpenSLX the root filesystem is provided by (step 6). Depending on the intended use of the booted Linux client another operating system could be started on-top providing additional session options (step 7). The selection of the kind of session to run could be left up to the user (Fig. 3 (c)) or pre-determined by the WBM.
In order to exemplify usage, following two scenarios are presented:
1) Single-User Model
The user has the PBL software installed on an ”univer-sal” USB stick, which it can always use to boot up-to-date systems provided by a central service. This means that, as long as the user’s network card (and evtl. graphic card) is supported by the PBL version he has on his USB stick, he has access to every system provided by the WBM. So there is no need, for example, to download a fresh rescue/anti-virus CD every time it is needed, but instead the user simply selects an rescue/anti-virus system provided by the WBM Administrators can create special purpose bootable images and provide them via the central service, whilst the stick itself stores only about 30 MBytes.
(a) (b) (c)
Fig. 3: Screenshots of different boot stages: (a) SYSLINUX menu of the bootmedia, (b) framebuffer GUI of PBL and (c) session chooser of the booted system
2) Managed Pools Model
If the PBL is embeded on a large pool of computers, it is possible to manage the default boot decision of every computer at a central WBM by defining rules. For example, it can be a standard Linux desktop system during the day and a backup system at night, or simply a kiosk system during a conference. In this setup the PBL is stored on the local hard drive, but a installation on an internal flashdrive like other instant-OS implementations (like Splashtop [13]) should be considered for further research.
VI. EXISTINGCOMPONENTS ANDCURRENT DEVELOPMENTS
Though far from complete, at least all core components for preparing a reference Linux installation for booting over the network are implemented and have been running successfully for more than 8 years as part of the PXE based setups at Freiburg University and at UFPR as well as in a (significant) number of public schools in both countries. The remote Linux booting systems are extended in order to be integrated into a web-service driven infrastructure (which could be deployed as a cloud service).
In Freiburg the PBS was deployed in some test scenarios in order to start the Linux lecture room environment in networks outside the core installation. The general boot delay on campus was minor compared to direct PXE boot. Furthermore, the concept has proven to work on the university networks of more remote locations of Offenburg and Ulm. Those are connected to the state academic high-speed network having a geographical distance between 50 and 200 km. Additionally, tests to boot machines over DSL or cable connections were run to create a comparable Small Office Home Office (SOHO) setup [1] with the PBL architecture.
Besides being independent of locally configured DHCP/TFTP services, the PBS method circumvents the PXE IPv6 issues.
The following important issues remain to be discussed and solved in more appropriate or more generic ways than provided by the current implementations.
a) Authentication: Once a machine has booted a
WAN-wide available image, it should be able to connect to its local authentication server or passwd file. Currently the boot images have this information built in, which makes them unusable outside of the local network. Desktop machines are provided with a default account. A solution for this problem could be to provide authentication information on the WBM server when choosing the boot image, but doing so in a generic way, even if considering LDAP servers, is not straightforward.
b) Local Configurations: In a similar way, the WBM
user (aka network administrator) should be able to configure mount points, printer servers, local printers and home folders that are specific to his environment. If such information is built-in in the boot image, then each network or even user must create its own version of the desired systems. A better approach would be to store local configurations for networks/users on the WBM and generate the booting image on demand. This is currently not possible because the whole image must be generated, which takes too long and has to be done offline.
c) Exporting the Root Filesystem: Providing a whole
system image over a WAN via NFS has been shown to be an inefficient solution. Network Block Devices (NBD, DNBD) have shown better scalability and usability on higher latency networks. Another option to be evaluated is the pNFS and PVFS2 (Parallel Virtual File System) [2].
d) Software Updates: Updating or installing new
soft-ware on an existing image is currently not possible. Even if the root file system is exported via NFS, changing the image on the server might stall all clients that are using it. The current solution consists in demanding a reboot of all clients, or creating a new boot image, which might then be chosen on a next reboot.
This approach works pretty well for stable environments with few system changes, but it is not very customizable. In order to provide software installation capabilities to the user, it would suffice if the root filesystem is mounted as a layered filesystem (AUFS), but the problem of underlying system updates still remains.
e) Root Filesystem Proxies: In order to deal with slow or high delay network infrastructures, like the ones in the Paran´a Digital Project [8], the boot images should be replicated by local servers. A proxy infrastructure that downloads the entire image is the naive approach, but eventually proxying partial boot images allows for transference of only the used parts of the system, which easily grows to 20 GBytes. Keeping local copies of recently used boot images is also an interesting strat-egy to speed up boot time. Considering that reboots are much more frequent than system changes/updates, it might well be worthwhile to use local disk space in order to avoid/overcome network delays or interruptions. The main consideration when using proxy is that it increases the problems for system updating.
f) IPv6: The upcoming IPv6 standard will definitely
invalidate many installed system management components. Especially the small footprint protocols and services devices could not be easily upgraded. While it is rather simple to run OS in dual stack setups, it is not desirable to maintain two different IP setups over a long period of time. The introduced PBL elegantly circumvents the PXE IPv6 problem and removes dependencies on legacy components.
VII. PRELIMINARYRESULTS
Remote Linux installations like Paran´a Digital or OpenSLX, which were originally meant only for PXE-enabled and specif-ically configured sub-networks, were successfully booted at arbitrary network locations. Measuring OpenSLX boot time using the PBL, clients outside of its original managed sub-network on the Freiburg campus took only a couple of seconds longer to start. After choosing the OpenSLX environment from the boot menu, everything performed as expected. Mostly the same could be observed when the architecture was tested in Offenburg and Ulm, university campuses which are connected by high-speed WAN infrastructures. It was even possible to start the remote OpenSLX boot from Brazil, but because of limited bandwidth and long delay the test was stopped after proving the general feasibility. Nevertheless, the experiments demonstrated the strengths of our approach of decoupling the remote boot functionality from the sub-network configuration. The PBL is a far more versatile approach to a Network Bootstrap Program than PXE and its open source counterparts like gPXE and iPXE. It can easily be adapted to incorporate more complex security and authentication features than pos-sible with PXE. Features like IPv6 network stack, chip card authentication or the use of trusted platform moduls will be explored in the future.
The most requested feature after the first tests from other research facilities is to support trusted booting. Besides the further development of the web based management compo-nents for creating and configuring the bootable Linux systems the integration of TPM for signing the boot components will be a major goal for the next milestone.
VIII. FUTURE
With Microsoft’s decision to require (U)EFI for the up-comming Windows release, UEFI will increasingly gain in importance. The UEFI standard [14] defines interfaces for most components required by the current implementation of the PreBoot Environment as well as interfaces for planed features like using TPM for signing content as shown by Zhang et al. [15]. Future versions of the PBL could be simplified by using UEFI APIs, e.g. for accessing the hardware and the network on a more abstract level. Finally UEFI could become the future pre-boot environment and implement the graphical user interaction functionality of PBL. This could simplify the booting of alternative operating systems from that level.
Another important issue is keeping track of the state of the computers on site. Systems booted centrally over the network are much easier to include in monitoring frameworks than standalone machines because they are already known by the WBM and monitoring clients can be easily configured on boot time. Additionally, the PBL could be extended to include asset inventory tracking [10] and monitoring components.
REFERENCES
[1] T. Cruz, P. Simoes, F. Bastos, and E. Monteiro, “Integration of pxe-based desktop solutions into broadband access networks,” inNetwork and Service Management (CNSM), 2010 International Conference on, oct. 2010, pp. 182–189.
[2] C. Aguiar, D. Cruz, R. Ulson, and M. Cavenaghi, “The application of distributed virtual machines for enterprise computer management: A two-tier network file system for image provisioning and management,” in Computer Software and Applications, 2008. COMPSAC ’08. 32nd Annual IEEE International, 28 2008-aug. 1 2008, pp. 428–431. [3] Intel and Systemsoft, “Preboot execution environment (pxe) specification
v2.1,” Intel Corporation, 1999, http://www.pix.net/software/pxeboot/ archive/pxespec.pdf.
[4] H. Anvin and M. Connor, “x86 network booting: Integrating gpxe and pxelinux,” inLinux Symposium, p. 9.
[5] K. Vervloesem, “Netboot.me turns netboot into internetboot,” Linux Weekly News – LWN.net, 2009, http://lwn.net/Articles/351263/. [6] Boot Kernel Org (BKO), “Booten kernel.org,” 2010, http://boot.kernel.
org.
[7] S. Schmelzer and D. von Suchodoletz, “Network linux anywhere,” Lin-uxTag 2009 Conference, Berlin, June 2009, http://www.ks.uni-freiburg. de/projekte/ldc/networklinuxanywhere-paper.pdf.
[8] L. C. E. D. Bona, M. Castilho, F. Silva, D. Weingartner, L. H. A. Lourenc¸o, and B. C. Ribas, “Managing a grid of computer laboratories for educational purposes,” LAGRID 2008, October 2008, http://www. c3sl.ufpr.br/prd/artigos/lagrid2008-eng.pdf.
[9] M. Garrett, “The extensible firmware interface – an introduction,”Linux Weekly News – LWN.net, 2011, http://lwn.net/Articles/454399/. [10] A. Kinzhalin, R. Kohn, D. Lombard, and R. Morin, “Enabling the
autonomic data center with a smart bare-metal server platform,” in Pro-ceedings of the 6th international conference on Autonomic computing, ser. ICAC ’09. New York, NY, USA: ACM, 2009, pp. 87–96. [11] S. Schmelzer, “Extending the playground – creating high
customiz-able bootmedia for openslx,” http://www.ks.uni-freiburg.de/projekte/ldc/ extendingtheplayground.pdf, 2010.
[12] Nokia Qt Development Frameworks, “Qt – cross platform application and ui framework,” 2011, http://qt.nokia.com.
[13] DeviceVM, “Splashtop,” 2011, http://www.splashtop.com/.
[14] Unified EFI, Inc., “Unified Extensible Firmware Interface – Specifica-tion, v2.3.1,” 2011, http://www.uefi.org/specs/.
[15] R. Zhang, J. Liu, and S. Peng, “A trusted bootstrap scheme on efi,” inMultimedia Information Networking and Security, 2009. MINES ’09. International Conference on, vol. 1, nov. 2009, pp. 200–204.
