Large Scale Coastal Modeling on the
Grid
Lavanya Ramakrishnan
[email protected]Renaissance Computing Institute
Duke University
North Carolina State University
University of North Carolina - Chapel Hill
SURA Grid Application Planning & Implementation, Austin, TX December 7, 2005
Acknowledgements
•
SCOOP SURA Partners
– Philip Bogden, Joanne Bintz, Helen Conver and many others
•
SCOOP North Carolina Participants
– UNC Marine Sciences
• Brian Blanton, Rick Luettich, Larry Mason (ITS)
– MCNC/GCNS
• Michael Garvin, Steve Thorpe, Chuck Kesler
– RENaissance Computing Institute (RENCI)
Outline
•
Motivation and Background
– SCOOP, Science Scenarios
•
SCOOP ADCIRC Experiences
– prototype and experiences
•
SCOOP Service Oriented Architecture
– modular, flexible
Weather Strikes Daily!
• Hurricane Season 2005
– 26 named storms, 14 hurricanes, 3 with major impact – billions of dollars economic losses
• We need to …
– provide early and accurate forecasts, dissemination of information – provide infrastructure to solve inter-disciplinary problems
– to be able to interact in real-time i.e. evaluate and adapt
Source: NOAA
Next Generation Cyberinfrastructure
• Examples
– SURA Coastal Ocean Observing and Prediction (SCOOP) Program
• U. Florida, U. Alabama, LSU, VIMS, Texas A& M, U. Maryland, U. of Miami, GoMOOS, UNC
– Linked Environments for Atmospheric Discovery
• Oklahoma, Indiana, UCAR, Colorado State, Howard, Alabama, Millersville, NCSA, North Carolina
R R R R R R R R R R
SURA Coastal Ocean Observing and
Prediction (SCOOP) Program
•
Integrated Ocean Observing System (IOOS)
– rapidly assess, predict, and mitigate the impact – make information widely available
– “plug and play model” for next generation research
•
End users
Illustrative Science Scenarios
•
Daily operational 24/7/365 forecasts
– continuous, ensure availability
•
Real-time ensemble model prediction
– real-time data, increased accuracy
•
Retrospective analysis
– evaluate results, innovate new mechanisms
•
Interdisciplinary problems
– inundation affected by storm surge, terrestrial hydrology, precipitation
Characteristics and Challenges
•
Application domain
– need integrated data and modeling environment
– adapt rapidly and automatically in response to weather – real-time policy, different data sources (e.g. sensors)
and formats
•
Computer Science/IT
– data storage, resource selection
– multilevel monitoring and intelligent control
Outline
9
Motivation and Background
9
SCOOP, Science Scenarios
•
SCOOP ADCIRC Experiences
– prototype and experiences
•
SCOOP Service Oriented Architecture
– modular, flexible
•
Advanced Circulation Model (ADCIRC)
– Finite Element Hydrodynamic Model for Coastal Oceans, Inlets, Rivers and Floodplains
• Access to data stored at UNC
– OpenDAP access through portal interface
• Retrospective Model Runs
– a portal interface, access to grid resources
• Real-time operational ensemble modeling
– 5 wind data sources, event driven, access to distributed resources
ADCIRC Coastal Modeling
Technology Exposition
•
Grid technologies (Globus)
– standard job submission: Gatekeeper – file transfer: GridFTP
– queue status: Information Services/MDS – credential repository: MyProxy
•
Domain products
– Local Data Manager
• event driven data transport system
– OpenDAP
• format independent network data access protocol
•
Portal Technologies
Sub m
it Job To GR
ID
Set Run Dates
(Hurricane Ivan) Cur rent A DCIRC gr id 16 CPU Decompositi on
Hindcast Analysis on the Grid
Specify model run parameters
Create tarball of needed Archived Files
Third-party transfer between Portal host and Compute host Execution of requested simulation
1 2 3 RENCI/UNC Portal Globus Gatekeeper Mass Storage NCEP Daily Model Runs OPeNDAP Server GridFTP Globus Gatekeeper GridFTP LDM To UAH MCNC Grid Globus Gatekeeper GridFTP LSF Queue MyProxy UNC Experimental SCOOP Machine 2 3 4 1 4 NFS Mounted
Ensemble Modeling
RENCI Maui /PBS (dante0) MCNC LSF (scoop) UAH (beaker) LSU (hugo, hilda ) UF (sura-uf-d4600-2) Globus (Gatekeeper, GridFTP, MDS) … GRID core Partner GRID Establishing connections Results pushed to RENCI VizWall RENCI LDM Node (dante2) ETA NAH UF-WANA RENCI SCOOP Portal (dante1) V1: Hindcast runs V2: Forecast status LDM LDM LDM Resource Selection Application ManagementReal-time Resource Selection
•
Given a set of resources, which is the
best resource I should run on?
MDS GridFTP Gatekeeper MDS GridFTP Gatekeeper MDS GridFTP Gatekeeper How many resources can I get (queue)? If MDS available else
Run a probe job to find no of cpus and
rough estimate on time Are services up? Resource Management Monitoring Meta-scheduling Policy …
Lessons Learnt from Hurricane
Season 2005
•
Forecast mode
– debugging is hard•
Resource selection
– performance, reliability•
Fault Tolerance
– when jobs fail …
•
Unexplored territory
– verification
– data management – catalog and archive
access
Left: ADCIRC max water level for 72 hr forecast starting 29 Aug 2005,driven by the "usual, always-available” ETA winds.
Right: ADCIRC max water level over ALL of UFL ensemble wind fields for 72 hr forecast starting 29 Aug 2005, driven by “UFL always-available” ETA winds.
Images credit: Brian O. Blanton, Dept of Marine Sciences, UNC Chapel Hill
Grid Testbed Experiences
•
Components at every site
– Globus gatekeeper, GridFTP, PBS/LSF
(optional), MDS (optional)
•
Globus setup at compute sites
– hours of testing, firewall problems
– MDS not setup
•
Resource availability and setup
– e.g: uudecode was not installed
TAMULSU UF
UNC, MCNC UAHb
UAH LSU UF UNC Marine Sciences RENCI
It is all about partnerships!
Maui /PBS Cluster LSF cluster LDM Node ETA NAH WANA SCOOP Portal V1: Hindcast runs V2: Forecast status LDM Health Sciences Library MCNC Archive Archive TAMU Archive Globus Globus Viz Wall NCSCOOP SCOOP Grid Ad-hoc collaboration on campus
Outline
9
Motivation and Background
9
SCOOP, Science Scenarios
9
SCOOP ADCIRC Experiences
9
prototype and experiences
•
SCOOP Service Oriented Architecture
– modular, flexible
Resources (compute, storage, network)
Management Layer User Interface Layer
portal (resource access, workflow interfaces, interactive search services, etc.), visualization tools, software libraries
Application and Tools Layer
Data Translation Data Visualization Archive Management Resource Management
Resource Access Layer
Application Management LDM, GridFTP, scp, etc Data transport … Cross-cutting Components Security (GSI, etc) Monitoring … Models & Analysis Tools Data Management Directories SOAP, XML, WSRF, etc
Web service protocols
Virtualization Workflow
Tools
… …
Analysis Application Translation Archive Services Catalog Services Verification / Validation Archive Services Catalog Services Coastal Model Resource Selection Data Management Application Env. Archive Services Catalog Services Observations Winds Model Results User Interface
Resource Access Layer
OpenIOOS Translation Archive Services Catalog Services Verification / Validation Archive Services Catalog Services Coastal Model Resource Selection Data Management Application Env. Archive Services Catalog Services Observations Winds Model Results User Interface
Resource Access Layer
Visualization
Next Steps …
• Multi-level Monitoring and Resource Management
– resource level, web service, etc
– optimal resource selection based on data movement costs
• Science Problems
– ensemble wind generation enhancement
• statistically based, recomputation of cases for sensitivity analysis.
– SCOOP product translation service – coupling between ADCIRC and SWAN
• Integrated Workflow Environment
– application planning and optimization
• data catalog and archive service • external APIs and documentation
Conclusions
•
SCOOP: A Service Oriented Architecture
– modular, composable, event-driven – instruments, streaming data
•
Grid programming and deployment
– specifying resource requirements
• higher level requirements
– performability of grid environments – managing resource environments
• specialized set of requirements for each application • cost, accountability, auditing
