www.ngs.ac.uk
Inside this
issue:
Page 3
Conference
round up
Page 4
Applications
Corner – NGS
Clouds
Page 5
Registration for the NGS Innovation Forum is now open and this years event will have A special emphasis on user tools. The event will be held at STFC RAL in Didcot near Oxford on the 23rd– 24thNovember
2010.
We will be featuring several NGS user tools in “walk-through” demos featuring real life research examples in order to give delegates
some knowledge of how to use these tools when they get back to their institution.
User tools focus at the NGS
Innovation Forum ‘10
STFC RALPage 5
Your feedback
to the NGS
Page 8
NGS User
Case Study
Page 11
Collaboration
Focus
Page 12
NGS
Presents…
The tools being featured are –
•NGS User Interface /Workload Management System (UI/WMS) – The UI/WMS tool allows users to easily submit jobs to the whole of the NGS relying on the WMS to chose which NGS resources to use for their jobs.
•Application Hosting Environment (AHE) - An overview of how access to the NGS can be simplified using the AHE, a lightweight application portal system.
•HERMES data management tool – used to easily connect into various NGS resources to move data between them, your home institution and your desktop.
•Databases on the NGS – this will be an overview of the databases supported on the NGS, what offerings and advantages these give users and case studies of how NGS users are taking advantage of them.
•The NGS from the CCP4 desktop – the NGS R&D theme have been working to build access to the NGS into the desktop tools that researchers use on a day-to-day basis.
Another new addition to the event is a poster session on the Tuesday evening where a prize for best poster will be awarded as judged by the delegates. The call for poster abstracts is out now with a deadline of the 24thof September. Full details of the call and
If you have recently visited the NGS website you may have noticed a new addition to the top level tabs. As well as making sure that the NGS runs all the services required by our users and sites on a day to day basis, we are also working on services for the future. To keep you up to date with our work "behind the scenes", we have added a newInnovation sectionto the website.
This section currently includes information on our prototype cloud service and user interfaces such as GSI-PuTTY and R through the NGS (Windows). Some of these services are looking for people to help develop the applications or to be the first users and to help develop them for the wider community. Please see theInnovation sectionfor further details.
In addition to the new Innovation section we have recently improved our description of theNGS database section on the NGS website. The NGS offers users access to two databases - Oracle and MySQL as well as the assistance of database experts here at the NGS. To find out more about what we can offer you, regardless of the size of your database, please see the Database section on the website. If you have any queries about using the NGS database service, please contact theNGS helpdesk.
New additions to the NGS website
Looking after your certificates
Many of the helpdesk queries are to do with certificates so in order to help you to look after your certificate we have recently released a series of new leaflets available from our website –
•Applying, retrieving and renewing certificates •Looking after your certificate
•Certificate obligations
We hope that these leaflets, which are
available from our website, will be of use to all NGS users – both new and old! We would be grateful if you could advertise these leaflets to anyone who may find them useful in applying to the NGS.
The summer is traditionally a quiet time for conferences which is just as well given how many
conferences take place in September! However the NGS has been present at one or two events over the summer months and we have been getting ready to participate in others.
NGS Roadshow at the Cockcroft Institute
We recently held a NGS roadshow event at the Cockcroft Institute which is based at STFC Daresbury. The event consisted of a roadshow in the morning followed by an afternoon of training led by David Fergusson from NeSC. The event was well attended and featured presentations from Stephen Pickles who spoke about his experience of running parallel jobs on the NGS and Rob Allan who spoke on how the North West Grid, Daresbury and NGS work together to provide facilities to researchers. All the presentations from the event are available on the roadshow event page on the NGS website.
The NGS at the UK e-Science All Hands Meeting
The NGS will have an exhibition stand at the forthcoming AHM meeting (13th– 16thSeptember 2010) in
Cardiff. As well as a chance to stop by and chat to NGS staff and to pick up some of our latest literature,
NGS Conference round up
Cardiff. As well as a chance to stop by and chat to NGS staff and to pick up some of our latest literature, there will also be several demos taking place at the stand over the 2 and a half days.
–NGS User Interface / Work Load Management (UI/WMS) System –gLite WMS Enabled NGS Applications Portal
–Getting started with the NGS Cloud –NGS Surgery Live at AHM 2010
–Taverna Server: enabling long running workflows on the NGS For demo times stop by the stand and check the timetable.
The NGS at the EGI Technical Forum
The NGS will also be present at the first ever EGI Technical Forum (14th– 17thSeptember 2010) in
Amsterdam. We will be part of the UK NGI stand in conjunction with GridPP so please pop along and see us if you are attending this conference.
NGS is conducting a two-year research and development project dedicated to Cloud Computing. The goal is to offer to NGS users private cloud infrastructures to conduct their research and to respond to the ever-evolving needs of NGS users.
Cloud Computing is rapidly emerging as an alternative
plat-form for the computational and data needs of our community. NGS users are already using the Amazon Elastic Cloud Computing (EC2) infrastructure for their research. In some situations, it is preferable for a NGS user to deploy a number of instances of a tailored Virtual Machine (VM) than submitting jobs to the existing NGS Grid Infrastructure.
The EC2 type of Cloud Computing, called 'Infrastructure as a Service' (IaaS) poses new challenges.
NGS Applications Corner
In each edition of NGS News we will give an introduction to developments from the NGS. In
this issue we introduce the new cloud prototype service from the NGS.
Moving large quantities of data to privately own infrastructures, such as the Amazon one, can be substantially expensive or plainly impractical. At the same time, sensitive data, specifically those pertaining to projects that required ethical clearance, may not be exported, stored and elaborated outside the institutional or academic boundaries. A private Cloud Infrastructure managed by NGS, regulated by the same policies that have been endorsed for the NGS Grid Infrastructure, is an effective answer to such challenges.
At the moment, NGS offers its users two Cloud Infrastructure prototypes: one hosted at Oxford and one in Edinburgh. Users can request to be registered in one or both infrastructures and to use a default pool of pre-loaded, generic OS images or upload their own. From there, users will be able to run their own VMs, create virtual block storage devices, take snapshots, create security groups and much more. All this will be done by means of EC2 compatible commands or via an easy an intuitive web-base interface. If you would like to be an early adopter of the NGS private 'Clouds' prototypes please contact the NGS helpdesk.
JISC have also recently released 2 reports on cloud computing which are of interest to the academic community – “Using cloud computing for research” and “Technical review of cloud computing for research”.
NGS Polls
The home page of the NGS always features a “straw” poll where we ask people visiting the page to quickly answer a question by simple press of a button. These polls are an ideal way for us to gather information quickly from visitors to the site.
The results of 2 polls are given below –
Although we ask users when they apply to join the NGS who they are funded by, we know this can change during a researchers career. The poll showed that the majority of our users are funded by EPSRC with “other” coming a close second. It is encouraging to
Your feedback
We ask for your feedback whenever we can so the NGS can take note of what we are doing
right (and wrong!) and to tailor our services to you the user. Recently we’ve collected
feedback at events and through our polls on the NGS homepage. Some of these results are
presented below.
Who is the main funder of your research?
AHRC BBSRC ESRC EPSRC JISC NERC
MRC a close second. It is encouraging to
see that we attract users from all research councils.
We know how many people use the NGS but we wanted to find out more about how people access the
resources. This question was
important as it has implications for the development of tools within the NGS. We also need to ensure that training material we produce such as tutorials is appropriate for our users and how they access the NGS.
MRC STFC Other
Which operating system do you use to access the NGS?
Windows Linux Mac Other
NGS Communicating Science event feedback
You may remember that we held aCommunicating Science eventat STFC RAL earlier this year to which all NGS users,Campus Champions, NGS staff and the wider research community were invited.
The event was designed to give delegates the skills to communicate their work on a non-technical level, clearly and confidently to a wide range of audiences. Topics covered included:
–how to gain the attention of your desired audience
–how to get key messages across clearly and concisely, avoiding jargon and unnecessary technical terms
–how to write for a variety of audiences
We’re pleased to say that the event was a great success! The feedback received was wonderful with 100% of delegates saying that the event met their needs – even more than expected in one case.
The content was judged to be very relevant scoring 4.72 / 5 on the feedback forms and receiving excellent feedback comments. The tutor (Myc Riggulsford) made quite an impression with many delegates specifically commenting on how entertaining he was and how he kept the level of presentations delegates specifically commenting on how entertaining he was and how he kept the level of presentations at a simple level.
The delegates left the event invigorated about dissemination and ready to use their new found knowledge for disseminating their research, giving better presentations, revamping websites, for communicating with researchers in other fields and much more. If you were unable to attend the event you can see the “top 10 tips” from the day as collected by the delegates on theevent webpage.
Calling all NGS users – we need you to make sure your details are up to date!
The NGS has recently had enquiries from several funding councils regarding their researchers use of the NGS. Looking through our user accounts it would seem that many people have not completed the sections asking for funding details (i.e. research council) and the name of your PI.
These sections are very important for us and the research councils to measure how well our services are being used and also strengthens our case for future funding. Future funding is becoming of relevance as our current funding ends in March 2011.
We would be very grateful if all users could take 2 seconds to complete (or update) the following 2 sections in your user application which can be accessed athttp://uas.ngs.ac.uk. We have now made the completion of these fields compulsory in any new or renewal applications to use NGS resources. Please note that any data used will be collated and anonymised.
HPC Software Development Call 2010/11 –
assistance from the SSI
The HPC Software Development call invites proposals for HPC software development to enable science and engineering. Funding in this area will be provided through a two stage process. The objective of the stage 1 projects is development of software to the point where it can be used to generate scientific output. Stage 1 projects can be up to 2 years in length.
The role of the Software Sustainability Institute in the call is to provide help and advice. The SSI can advise on the role and recruitment of software developers within an academic software team, on the development of a sustainable user and developer base, and other issues pertaining to software
sustainability. The SSI are happy to discuss the support they can provide to Stage 1 projects, as well as opportunities for collaboration in Stage 2. The Software Sustainability Institute will be offering surgeries and workshops for successful Stage 1 projects.
Statistics on the NGS now available online
Statistics on the NGS now available online
There is a new section on the home page of the NGS website located just under the latest poll. The NGS usage stats makes publicly available information on stats by research area, institution, NGS usage over time, funding sources and information routes. For example you can now see how many users are funded by any of the funding councils or how our users first hear about the NGS – was it at a conference or was it from a colleague?
You can also see where our users are based, what the most popular research areas are and how our user base is growing over time.
There is also information available on the NGS resources such as the rate of utilisation of NGS
resources and where users are running jobs. The stats are updated in real time so we hope you find this new resource helpful.
mRNA analysis using the NGS
Pesticide resistance is becoming more of a problem throughout the world as the use of pesticides continues. Pesticides that were once effective at killing crop pests are now no longer effective leading to an increase in damaged or destroyed crops. One group of researchers at the University of Exeter are using the NGS to try and understand how this resistance is developed.
Paul and his research group are analyzing the transcriptomes of insects that are major crop pests, such as Manduca sexta, the tobacco hornworm moth. A transcriptome is the set of all messenger RNA (mRNA) molecules, or "transcripts," that are produced in a population of cells and reflects the genes that are actively expressed at any given time.
Case Study
Case studies featuring research carried out by NGS users are regularly featured on the NGS
website but in NGS News we will bring one of the case studies directly to you! If you would
like your research to be featured on the NGS website and in NGS News as a case study then
please contact the NGS Liaison Officer, Gillian Sinclair ([email protected]).
Image courtesy of Jan Stipala at the University
of Exeter.
The entire transcriptome of an insect pest is sequenced using the latest ‘Next generation’ DNA sequencing technologies. Assembling this sequence data generates in excess of 40,000 sequences which are then annotated by comparing the expressed sequences against known protein sequences contained in protein databases such as UniProt.
Paul explained that “sequencing transcriptomes can help scientists to understand how insects develop resistance to pesticides. By looking at changes in genes that code for proteins involved in the metabolic pathways that detoxify pesticides; we can assess how resistant insect strains evolve in the field. In addition the availability of a fully annotated transcriptome allows us to examine differential gene expression of these pesticide resistance genes, using the recently developed digital transcriptomics technologies”. It is also hoped that the sequencing of insect transcriptomes will lead to the discovery of new enzymes that could be used industrially for such things as biofuels, fruit juice extraction / clarification and waste water treatment.
The homology searches are performed at NGS using the NCBI BLAST software. This is a parallel implementation of NCBI BLAST that distributes computational resources through database fragmentation, query segmentation, intelligent scheduling, and parallel I/O.
Paul’s team are also planning on using mpiBLAST which they hope will improve NCBI BLAST perform-ance by several orders of magnitude while scaling to hundreds of processors. This should dramatically reduce the length of time required to characterise all of the expressed genes in our insect transcriptomes.
Paul said "The NGS staff have been a great help
to us in the implementation of our high-throughput BLAST’s. Homology searches had always been the bottleneck in our data analysis pipeline. Making use of the NGS resources has significantly reduced the time it takes to process our data“.
NGS Sysadmins Corner
Every issue we’ll be bringing you a handy tip or shortcut from one of the NGS sysadmins.
This time it is the turn of Jonathan Churchill, NGS, STFC RAL.
The NGS helpdesk often gets messages relating to jobs failing with long run times. Although there are exceptions to this, the majority of sites on the NGS have a maximum wall clock limit of 48 hours run time i.e. 1 processor for 48 hours, or 16 processors for 48 hours in an MPI job. Users will need to use any check pointing and restart capabilities that the software provides, to resubmit the job from where it left off after 48 hours. MPI Code users can do the same but could also increase the number of processors in the job if this is practical.
If you are a user of the UI/WMS service (insert webpage here), you can specify that jobs should only be submitted to sites with 48 hours or more wall clock limits using a requirements statement such as:
Requirements = ( ...
&& other.GlueCEPolicyMaxWallClockTime >= 2880 );
In order to provide a fair service to all our users, sites have to balance the needs of a large variety of users from 2 minute jobs to 1 month jobs hence the 48 hour run time limit.
Enabling cloud mobility
Belfast e- Science has developed an API to enable one-Stop access to various cloud offerings to reduce the effort In adopting the cloud model and moving between cloud providers.
There is an proliferation of clouds available to Academic users, such as the commercial providers (e.g. Amazon) to
NGS based clouds (e.g. Belfast e- Science, Oxford e-Research, Edinburgh)
and internal owned clouds. Belfast e- Science has developed an API to enable one-stop access to these various cloud offerings to reduce the effort in adopting the cloud model and move between various cloud providers.
The API provides a (simple) higher-level abstraction on clouds, allowing users to search for resources which meet their requirements and filter the choices based on the cost of a particular usage scenario. As a higher-level abstraction it also provides more sophisticated security than existing providers - users do not need to know the credentials for the underlying cloud, only their UK e-Science Certificate (or range of other authentication methods) and can be granted partial or full access to the provider: for example, a cloud monitoring service can be granted read-only access.
It currently works with a diverse set of cloud providers and can be easily extended to support additional providers. The current support list includes:
–the latest version of Amazon's EC2 service (with their new high performance instances designed for cluster compute)
–the UK provider Flexiscale
–the Belfast e-Science Centre cloud –the Oxford e-Research Centre cloud –Eucalyptus clouds
–OpenNebula clouds
Gillian: Where is the team based?
QUB: The Belfast e-Science Centre (BeSC) is hosted by the Schools of
Electrical & Electronic Engineering and Computer Science within QUB. The team is physically based in the Sonic Arts Research Centre building. BeSC’s equipment is hosted in the QUB data centre and two other data centres within the UK.
GS: Who is in the team?
QUB: The team is led by Prof Ron Perrott (Centre Director), Terry Harmer (Technical Director) provides technical
direction and developing projects, with the real work carried out by John Hawkins, Yih Leong Sun and Peter Wright.
GS: How long have you been a member of the NGS?
QUB: We joined in 2006 to provide service based technology rather than storage or compute nodes to the NGS.
Our first service was an auto deployment service that enabled user services and applications to be deployed and managed by users but hosted on BeSC infrastructure. Since 2008 we have provided a Cloud service to the NGS
Each issue we will introduce you to one of the institutions that make up the NGS. In this
issue the Queen’s University of Belfast (QUB) tell Gillian Sinclair why they got involved with
the NGS.
Yih Leong Sun, Peter Wright, Terry Harmer, Ron Perrott
managed by users but hosted on BeSC infrastructure. Since 2008 we have provided a Cloud service to the NGS that provides an on-demand resource capability.
GS: Why did you decide to become a member of the NGS?
QUB: We are interested in being part of the global e-Science activity and the NGS provides us with access to
large-scale resources to support our projects and enables us to support other e-Science users. The interaction with users and user projects has given us access to requirements and use cases which has guided the technology development with BeSC.
GS: What advantages has being a partner brought Belfast?
QUB: The NGS provides a platform that exposes BeSC’s cloud capabilities to the wider e-Science community.
The biggest advantage is to be part of the e-Science community, to assist the development of projects, to share expertise and to see new types of research approaches that the Centre has supported.
GS: What projects are the QUB team participating in that involve the NGS?
QUB: At first we assisted users to get started in using NGS resources and supported some early utility
computing adopters using our auto-deployment services. More recently we have been helping users and industrial collaborators use cloud based approaches using the BeSC cloud and utility resource providers e.g. Amazon and Flexiscale. We are experimenting with Cloud based HPC using private and public resources rather than the traditional national computing resources and supporting early cloud adopters such as the FleSSR project.
Hi, I'm Steven Young. I have worked in the NGS since 2004 and am based at the University of Oxford. I started within the NGS as a grid systems administrator for the original Oxford NGS core node. I took on the work of bringing the first set of partner sites into the NGS and continued acting as an NGS "buddy" for other institutions and sites in joining resources and services to the NGS. Currently I lead the NGS Partnership activity where the team of NGS buddies provide guidance and assistance to resource owners and system administrators in becoming NGS sites.
The NGS role of Connecting Infrastructure involves engaging with Universities and
institutions, helping them connect resources and services to the grid. The work of the NGS Partnership team follows on from an NGS Roadshow event which provides an introduction to the NGS and its services. The various strands of the Partnership work include the Campus Champions programme, NGS site buddying and organising and providing meeting support for the NGS Collaboration Board.
NGS presents…
Each issue we introduce a member of NGS staff who will explain their role within the
organisation. In this edition we are delighted to introduce Dr Steven Young who is based at
oerc, University of Oxford.
site buddying and organising and providing meeting support for the NGS Collaboration Board.
Other activities at Oxford which happen within the NGS context are OxGrid: the Oxford Campus Grid and the NGS Cloud Computing beta services. Recently the Oxford compute service has been transferred to a cluster within the Oxford Supercomputing Centre.
Updates from NGS sites
STFC(RAL) Intel 11. Compilers and MKL Installed“STFC (RAL) sites ; ngs.rl.ac.uk and scarf.rl.ac.uk , have recently upgraded their Intel compiler installations to the latest 11.1 compilers (073 release) and updated their MKL libraries to the 10.2.5 (035) release. More details can be found at http://www.ngs.ac.uk/stfc/stfc-ral-compilation”
Memory Update
The NGS site, The White Rose Gridbased at the University of York, has upgraded it's memory control and now features the Exludus MCOPt softwareon its cluster.
This will make it easier for researchers to specify memory resource requirements for jobs, helping them to concentrate on their research, not batch job systems. Tests conducted by the team at York have shown that in a mixed memory load situation the overall system throughput is enhanced. This is especially true with respect to jobs with a larger memory footprint when the system is under heavy memory load. Any queries about this new service should be directed to the NGS helpdesk.
Events
Event
Date
Location
Website
NGS
Presence
All Hands
Meeting
13
th– 16
thSeptember
2010
Cardiff, UK
http://www.allhands.org.uk/
NGS will
have an
exhibition
stand at
this event
EGI
Technical
Forum
14
th– 17
thSeptember
2010
Amsterdam,
The
Netherlands
http://www.egi.eu/EGITF201
0/
The NGS
will be
present at
this event
OGF 30
25
th– 29
thOctober
2010
Brussels,
Belgium
http://www.ogf.org/OGF30/
NGS
23
rd– 24
thSTFC RAL,
http://www.ngs.ac.uk/events
NGS
How to get involved
We hope you have enjoyed this latest edition of NGS News. If you have any NGS-related or grid computing news that you would like to see published in the next edition (December 2010) then please contact Gillian Sinclair ([email protected]).
If you are a researcher currently using the NGS or if you have recently used the NGS, we would like to feature your research as a case study for our website and to possibly use it as promotional literature for the NGS. We are also looking for research papers, conference presentations and posters to feature on our website. If you are interested in distributing your research findings to a wide UK and international audience then please contact Gillian at the address above.
