3 Project Implementation
3.2 ReDS Programme
3.2.4 RDSI Board Allocation Process
This additional process is proposed after extensive consultation with, and support from Nodes, together with approval from the RDSI Board and the Lead Agent.
The RDSI Board Allocation Process will support allocations in addition to those supported by existing ReDS allocation committees, noting those committees allocate storage provisioned through ReDS funding provided in advance to Node operators and the proposed ingest driven re-provisioning top-up process.
It is intended that the Board Allocation process will apply to well identified needs, not met, not easily met, or not met appropriately, through existing processes - hence the required allocations will present as exceptions to those processes.
The intention is to support allocations in the form of several very large-scale data collections of national significance not well addressed by the current processes. Each of those collections would be supported by one or more Node operators and the resulting collections would continue to be available for the foreseeable future.
It is not proposed that any specific level of funding must be allocated through this process; rather the justification for each allocation needs to be made in balance against alternative provisioning uses.
However there is a recognition that data intensive communities and broad national data sharing communities are not well addressed by the current processes. More importantly they may not be addressed to their satisfaction which is an important factor in realising the impact of the RDSI Project.
It is also recognised that in meeting such needs, existing provisioning processes can be used. It is likely that the RDSI ReDS funds may not be sufficient to cover all large collections identified.
2013 work plan builds around the NCRIS capability areas as indicators of national priority, international relevance and longstanding established national leadership. The components of that plan form consistent and coherent areas for storage provisioning. Other possible areas are consortiums of leadership that have emergent large data needs such as Centres of Excellence – for example high energy physics.
This process establishes criteria to support allocations by the RDSI Project board that prioritise on the basis that proposals:
have value to the national and international research community;
focus where appropriate leadership is available;
be supported by suitably scaled commitments to associated data management;
make use of storage provided within the RDSI project scope; and
apply where scaled up or nationally coordinated allocations are required.
3.2.4.2 Process
Given the interplay expected between ReDS allocation committees and this process, and that Node operators may have interests in multiple national scale collections, a three stage process is proposed:
1. Node operators separately or jointly approach or are approached by intensive sources of research data and research communities with large aggregated data assets, and as a result suitable support proposals are developed;
2. The RDSI Resource Allocation Panel will review proposals against the criteria below, and negotiate and refine the set of proposals that meet the criteria and lie within the funding scope and take account of existing ReDS committee based allocations that relate to the proposal;
3. That the panel’s recommendations include consideration that all obvious areas are included or can be shown to be already aided by ReDS allocations.
It is important to note that RDSI is not the only funding source in the research sector. A level of assistance toward solving data storage requirements is of value even if further resources must then be applied.
3.2.4.3 Proposals
Proposals will have a limit of 6 pages and will explain how the funding allocation meets each of the following criteria:
The data supported underpins research of identified national / international significance;
The proposal accords with the needs of the associated research communities, and the stated missions of the related parties or institutions and participating Node operators;
The Node operators have the commitment and support of the related parties and institutions;
The extent of time over which the collection will be supported is clear, the access to be provided to the collection is clear and the mechanisms for sustaining the collection’s ongoing accessibility are clear;
The storage implementation meets stated goals in an efficient and effective manner; and
The outcome requires a scaled up or a nationally coordinated allocation.
3.2.4.4 Development of Proposals
Proposals will need to show that related research communities, institutions and infrastructure providers are engaged and intend to contribute to the ongoing success of the allocations once made.
Where significant groundwork has not already been completed, a proposal would require substantial effort to be undertaken over the next few months. It is also obvious that strong engagement with the targeted research community is needed and that an appropriate person needs to undertake the development of each proposal. The areas of interest identified in the RDSI NCRIS 2013 project plan are a guide to likely areas.
Consideration has already been given to those areas representing significant research needs around established and new research collections and matching the strategic priorities of the Node operators.
The RDSI Board has decided that proposals must be submitted before 11pm, Australian Eastern Standard Time, on Thursday 24th April. They should be emailed to the RDSI Project Office ([email protected]) and copied to the RDSI Project Director ([email protected]). They should be submitted by the authorised representative of a Node Operator, as defined in existing Node Subcontracts with the University of Queensland.
The RDSI Resource Allocation Panel, supported by the RDSI Project Director, will review proposals against the criteria in order to make recommendations to the RDSI Board. At any time during the consideration of the proposal, the panel and the Project Director may seek further information from, or negotiate with proposers.
The table below describes the main activities of the ReDS programme for this ABP.
Description of activity
Timeline
Start End
Support RAP in reviewing theLarge collections proposals and
submit to RDSI board for approval April 2014 May 2014
Announce accepted proposals May 2014 June 2014
Manage ReDS funding distributions for MAS and CD storage July 2014 December 2014
Review storage allocations regularly July 2014 December 2014
Monitor collection ingestion onto storage July 2014 December 2014
Monitor Collection usage July 2014 December 2014
3.3 DaSh Programme
The Data Sharing programme (DaSh) has built capability to support the sharing and re-use of research data.
In order to identify what high performance data sharing and data movement services are needed by the sector this programme ran substantial consultations with relevant research sector stakeholders, researched and evaluated existing services and products. The extensive Node consultation undertaken in the past demonstrated that this programme needed to often evolve to accommodate the Nodes and researchers requirements.
The approval from the Department allowing the RDSI project to fund two staff under the ReDS programme at each Node facilitated the management and progress of the DaSh programme.
The RDSI project has funded a number of sub-projects described in section 3.3.1.
Planning to hand over any tools, services or systems developed by the sub-projects will take place during Q3 and Q4 2014.
3.3.1 Outline of DaSh Themes
The RDSI project Office oversees the development of DaSh themes. The themes are broken into sub-projects which are outsourced to Nodes and higher education and research sector organisations. Sub-projects have their own steering committees and report progress and outcomes to the RDSI project.
Network theme
DaShNet – The network connecting Nodes to users and to each other
The National Research Networks (NRN) project has a sub-project to fund the development of a network to support RDSI. This theme in consultation with AARNet and Nodes has developed a proposal to the NRN Steering Committee to fund the DaShNet proposal. AARNet was contracted to design and develop the network architecture.
This theme has made enormous progress with AARNet progressing as per their project plan. Minor delays outside their control have been experienced however there is no indication at this stage that these will impact the DaShNet delivery by December 2014.
RDSI receives regular project reports from AARNet and these are available upon request.
AARNet will continue to support the DaShNet when the RDSI project concludes. In this case a handover is not necessary as AARNet is managing and delivering this RDSI component to Nodes and is signing agreements directly with Nodes.
It is anticipated that the DaShNet project will develop a network infrastructure for RDSI that:
Provides the reliable high speed broadband infrastructure between all RDSI Nodes to support the uploading, replication and distribution of data collections, and their integration with discipline-specialised services such as HPC and virtual labs.
The RDSI project in consultation with appropriate stakeholders will investigate if there is scope for a proposal to invest in additional Aspera product licenses for the Nodes.
RDSI Data Mover – Providing fast data movement between, into and out of Nodes This activity has been incorporated as part of DaShNet and is progressing as planned.
Activities and timing
The table below describes the main activities of the Network theme; most activities such as the Science DMZ, which is a portion of the network and provision of data access infrastructure (data mover) were incorporated into DaShNet project as the network is a pre-requisite for such activities to take place and AARNet is best placed to implement these with Node support. The end date means production ready.
Identity and Security theme
Identity
The Researcher Identity and Access management system (ReX) has been developed by the AAF and Nodes with support from the RDSI project Office. Nodes are in the process of testing and integrating the system with their in-house tools.
It is possible that Nodes may propose improvements to the ReX System sub-project Steering Committee. Such improvements, if any will be considered by the RDSI project however these may be beyond the scope of this project.
The RDSI project Office will handover the ReX System to the AAF to maintain and support the system for researchers after the conclusion of the RDSI project.
Security
It was anticipated that there would be significant security barriers to the successful deployment of RDSI storage, particularly for researchers connected across firewalls and other security barriers within institutions. Further, the speed of network connections between Nodes would require innovative approaches to security at high speed.
The RDSI project developed security policy guidelines and models for Nodes and Institutions in consultation with stakeholders through a series of workshops (https://www.rdsi.edu.au/workshops).
These are available on the RDSI website (https://www.rdsi.edu.au/security) and will be updated if necessary. This material will be available on the RDSI website after the project concludes.
Description of activity
Timeline
Start End
AARNet to develop DaShNet July 2014 December 2014
Provide DaShNet for Nodes and researcher’s use July 2014 December 2014 The RDSI project to investigate if there is scope for a
proposal to invest in additional Aspera product licenses for Nodes.
July 2014 December 2014
Data Access theme
This theme supports the provision of common approaches to data access, sharing and movement capability commensurate with the importance, volume and complexity of the large volumes of data.
RDSI ran a DaSh workshop with the Nodes and sister projects to identify proposals to complement the data access theme. The workshop occurred in February 2014 with Nodes proposing options for the implementation of systems to support additional data access methods, such as the Programmatic Access Service (OPeNDAP), file synchronisation or “Dropbox” like capability, integration of LiveArc and Aspera, and twelve months extended licence and maintenance of LiveArc for Nodes. These activities have been considered and approved by the RDSI Board and will be funded by RDSI and implemented and managed by the Nodes.
The LiveARC sub-project and its integration with Aspera was funded by the RDSI project and is currently being deployed by Silicon Graphics (SGI) at each Node. SGI will hand over the LiveArc deployment to each Node before the conclusion of the RDSI project.
The Programmatic Access Service will be deployed by NCI and made available to interested Nodes, similarly the “Dropbox” like capability developed by QCIF will be available to interested Nodes.
The Researcher file syncing and sharing system project to refine the requirements further in conjunction with stakeholders will be managed by QCIF with appropriate reporting to the RDSI project Office. A proposal will be developed and proposed to the RDSI board for approval through the RDSI project.
The LiveARC and Aspera integration will be delivered by Architecta/SGI to the Nodes.
The LiveARC extended licence incorporates an additional year of maintenance for Nodes and will be delivered by SGI to Nodes.
The table below describes the main activities of the Data Access theme. The end date means production ready.
Description of activity
Timeline
Start End
RDSI project to consider ReX System improvement requests from Nodes July 2014 December 2014
Handover the ReX System to the AAF July 2014 December 2014
Description of activity
Timeline
Start End
RDSI project to receive progress reports on the implementation of
the LiveArc project July 2014 December
2014
Workflow theme
The Workflow theme was created to fund the streamlining of workflows to facilitate the discovery of collections for researchers and data custodians, the collection management available for Nodes, the storage allocation process, monitoring and reporting on Nodes, collections and performance.
ReDS Application Processing – Streamlined application and workflows for the ReDS programme The ARMS project enables streamlined allocation and discovery of collections under the ReDS programme. The first phase of ARMS was successful and the systems is in use. In consultation with Nodes additional requirements were identified and the second phase of ARMS is addressing these requirements and it is anticipated to be concluded in Q3 2014.
ARMS production phase being developed and managed by QCIF will be made available to all interested Nodes by Q3 2014.
RDSI DaShboard – A system to automatically collect and publish Node and Collections metrics The RDSI DaShboard has been integrated in the RDSI portal and currently provides status on allocation and ingest by Node as well as general security information. Further data will be added as the outputs of the reporting project are integrated into the Dashboard.
RDSI Portal – Integrated access to RDSI elements/services.
The RDSI portal has been developed and hosts all information related to RDSI Nodes, collections and available tools within RDSI.
RDSI Monitoring – Monitoring and reporting RDSI elements services at Nodes
Automated monitoring arrangements are being developed by Nodes and will support reporting through the RDSI portal. This sub-project is overseen by the Node Development manager.
The RDSI portal and its integrated components will be maintained by the RDSI project Office until June 2015. RDSI will seek an interested party to maintain the site after the project conclusion. In the unlikely event that a Nodes doesn’t continue to run the RDSI website its content will be archived and stored at UQ.
Test Platform (former Node Zero theme)
The test platform theme was created to support meaningful evaluation of RDSI funded infrastructure and applications in a realistic environment spanning over RDSI-funded Nodes. It provides a mechanism
Description of activity
Timeline
Start End
RDSI project to receive progress reports on the implementation of
ARMS project from QCIF July 2014 December 2014
RDSI project to receive progress reports on the implementation of the monitoring and reporting project by Node
July 2014 December 2014 RDSI to seek a Node to handover the RDSI portal July 2014 December 2014
Creation of Support Materials – Creating the resources needed for the future support of RDSI services The resources needed to allow the effective future support of RDSI services were achieved through the AeRO support project and is available at http://usersupport.aero.edu.au/good-practices/.
The table below describes the main activities of the test platform theme to promote a stable environment to evaluate tools and resources required to implement and support RDSI project objectives at a production quality level. The end date means production ready.
Description of activity
Timeline
Start End
Enhance the durability and sustainability of Nodes by connecting to
public clouds September 2013 December 2014
Manage the creation of support materials being developed by AeRO July 2013 June 2014
3.4 VePa programme
The Vendor Panel (VePa) programme was to provide options for potential RDSI Nodes and members of the Council of Australian University Directors of Information Technology (CAUDIT) to procure storage related infrastructure, software and services. The purposes for the programme were twofold.
Firstly to allow Nodes, universities and other authorised users to avoid lengthy tendering processes by using an appropriately constructed panel and secondly to support volume pricing across Nodes and the wider Higher Education and Research Sector.
A wide range of infrastructure and services are included on the panel and each Vendor has agreed to a Commonwealth Government Information Technology and Communications (GITC) compliant model contract (or its equivalent) which will significantly streamline the use of the panel across the university and research sector.
The ongoing VePa RFP has been launched, vendors are able to submit their applications at any time, and the evaluation committee assesses these on a need basis. This facilitates the entrance to the Vendor Panel.) Currently there are 18 vendors on the panel. The complete list of the Vendor Panel is available at http://www.rdsi.uq.edu.au/vepa.
The RDSI project Office will work with CAUDIT to facilitate the smooth hand over of the Vendor Panel to CAUDIT.
The table below describes the main activities of the VePa Programme.
VePa Programme Milestones 2014-2015 Timeline
Start End
On-going evaluation and announcement of new vendors to the
Vendor Panel On-going December 2014
Handover of the Vendor Panel to CAUDIT July 2014 December 2014
3.5 Project Conclusion
At the time of writing this ABP it is expected that the RDSI project will conclude on the 31 December 2014 and measures will be put in place to hand over components of the DaSh Technical Architecture to the appropriate bodies.
Project Conclusion Milestones 2014-2015 Timeline
Start End
Develop project conclusion guidelines July 2014 October 2014
Annual RDSI Progress Report to the Department for approval August 2014 September 2014
RDSI project concludes December 2014
Final RDSI Report to the Department for approval January 2014 March 2015
4 Management and Implementation
The 2014-2015 Annual Business Plan builds on the Project Plan, takes into consideration outcomes of the project, consultation with the sector during 2013-2014 and proposes the implementation and management of the project for the period of July 2014 to June 2015.
The Board Terms of Reference and Operations are outlined in Appendix B for completeness.
4.1 Governance and Management Framework
The Project governance framework is unchanged and will consist of:
The University of Queensland, as detailed in section 4.1.1.1;
a Project Board, as detailed in section 4.1.1.2;
an Infrastructure Advisory Panel, as detailed in section 4.1.1.3;
a Resource Allocation Panel as detailed in section 4.1.1.4;
a Project Director to manage all aspects of the Project, as detailed in section 4.1.1.5; and
appropriate legal, office and administrative, scoping, development and technical support needed by the Project Director.