Software Development and Integration 1.1.6

The original mission of Software Development and Integration (SDI) was to implement the XSEDE architecture. However, during the first quarter of work we discovered that this original mission was too narrow. XSEDE is not a green field effort because it inherited a TeraGrid production cyberinfrastructure system. SDI’s task is more than implementing a new architecture, but also includes evolving the currently deployed and operational software and services to incorporate new architecture features and to deliver a myriad of routine but essential technical enhancements required to maintain the quality and functionality of any large and complex production software system. The proposed Program Year 2 plan for SDI reflects this necessary shift in focus and includes the following:

 an open planning task drawing on inputs from multiple project and stakeholder sources to identify, document, and prioritize activities (§3.7.1),

 an implementation task proceeding by implementing as many high-priority planned activities as possible within quarterly “time boxes” (§3.7.2), and

 a practice improvement task consolidating, documenting, and automating a growing collection of XSEDE engineering processes and best practices (§3.7.3).

3.7.1 Open, Continuous Planning

In February 2012 the SDI team introduced the basic component of an engineering planning system, currently implemented via the XSEDE wiki, to support ongoing efforts to identify, plan, and prioritize work. The planning system manages a queue of work (called “activities”) through a lifecycle of stages (proposed, planning, planned), with highest-priority planned activities scheduled as resources become available. One aspect of “openness” is that all information about the work (objectives, requirements addressed, software affected, cost and schedule effort, etc.) is visible. Another is that XSEDE’s stakeholders (initially limited to XSEDE partners) can engage with any and all aspects of the planning process, including, ultimately, prioritization of work.

Within two weeks of initiating planning, SDI received over 90 proposed system enhancements in areas as diverse as: security, interoperability, user documentation, testing, performance and reliability. This reflects a substantial backlog of improvement work that XSEDE has inherited from TeraGrid.

Several enhancements to the planning capability will be initiated in Program Year 2, including:

 Mechanisms to support the UREP, including the ability to assign and revise priorities;

and to identify technical dependencies and assess the impact of new work, new requirements, and new priorities.

 Mechanisms to obtain input from stakeholders, beginning with service providers, including the ability to propose baseline enhancements or new capabilities (a “wish list”), and to provide input about the impact and priority of planned activities.

3.7.2 Continuous Development and Integration

Beginning April 2012, SDI initiated quarterly increments. Activities that have reached the planned state are selected for launch in priority order. Once launched, activities move through another lifecycle (pending, launched, active, complete). SDI launched 16 activities in April 2012, all of which are scheduled to complete before the end of the Program Year 1. Information about these activities and their current state is managed on the staff wiki.

The combination of continuous planning and quarterly increments reflects the basic principles of agile software development. One advantage of an agile approach is that it allows for course corrections in work and quick response to changing stakeholder needs. One consequence of an agile approach is a shorter planning horizon—it is possible to talk about work currently under way and about work likely to be scheduled in the next sprint, but forecasting beyond the next sprint is speculative.

The following activities are likely candidates for launch in the first quarter of Program Year 2:

 Data Services: XSEDE Wide File System (XWFS) deployment; archival replication services; data integrity services

 Resource Management Services: Rich client interfaces; improved prediction services (Karnak); improved Science Gateway integration;

 Security Services: Improved interactive single sign-on (SSO); InCommon integration;

improved certificate authority (CA) certificate installers; improvements to MyProxy;

 Integration Services: Improved interoperation (REST and WS* Interoperation); improved system information publishing, discovery, and donitoring; enhanced central service registration; streamlined SP integration.

3.7.3 Engineering Improvements

The planning activity and development and integration practices established during Program Year 1 have had a significant and very positive effect on the efficacy of SDI. While the three initial activities accepted by NSF in the revised narrative took multiple quarters to deliver (primarily due to the lack of foundational TeraGrid engineering processes), the second group of activities initiated during the fourth quarter of Project Year 1 are expected to take 12 weeks. The planning activity (§3.7.1) and development and integration practices (§3.7.2) reflect a range of engineering processes introduced and piloted by SDI in: technical reviews (especially security), technical documentation, software integration testing, configuration management, bug tracking, and coordination of SDI with Operations on acceptance testing. SDI has also introduced automated status tracking and reporting for all planning, development, and integration activities. For example, XSEDE can track which requirements are being/have been addressed by past and ongoing work, which components are/have been affected, and how much time and effort is being invested in particular requirements and components.

Engineering improvements proposed for Program Year 2 will target:

 Substantial automation management processes, including automated status reporting on all launched activities (milestone completion rate, deliverable completion rate, effort expenditure rate).

 Baseline engineering performance metrics, including: investment profile (e.g., time and effort spent satisfying particular requirements, integrating specific technologies, enhancing specific components); per component measures of defect and help desk rates;

and quality and usability of end-user and engineering documentation.

 Consolidation and documentation of SDI processes and support mechanisms for use by others within XSEDE, with emphasis on: design review practices; document review practices; technology review practices; synchronized quarterly releases; and a project-wide pipeline to funnel prototypes and technology evaluation through SDI as the preferred “front door” to operational deployment.