The evolution of DAVE
The evolution of a large IDL application
Data Analysis and Visualization Environment (DAVE) project.
Richard Tumanjong Azuah
Larry Kneller
Yiming Qui
John Copley
Robert M Dimeo
Craig Brown
Philip Tregenna-Piggott
David Fanning
Craig B. Markwardt
NIST Center for Neutron Research
What is the NCNR?
A neutron scattering facility
Used by roughly 2000 scientists a year 20 neutron scattering instruments available
For materials research – eg atomic structure, dynamics and magnetism.
How it works.
Users apply for beam time Proposals are peer review
Limited beam time (a few days) awarded to best ones User brings own sample and works with a staff scientists
Data is generated – counts as a function of angle, time, energy, temperature, magnetic field, etc
Raw data is instrument dependent – instruments are uniquely optimized to cover different regimes or
Data Treatment Process
The 3 main wishes
D
ata Reduction – convert raw data to a universalscattering function, S(Q,E).
Q → atomic structure E → atomic motion
Data reduction is instrument dependent
Visualization of raw or reduced data can
provide useful information
Analyse reduced data – extract parameters
that determine physical quantities
atomic structure, magnetic moments, vibrational motion, etc.
Analysis involves a mix of basic data operations, data modelling, visualization, etc.
Origins of DAVE Project
Motivation
Seven years ago at the NCNR:
there were few or non-existent software tools for basic data reduction and analysis.
traditional approach of user-written software was time-consuming.
complexity steadily increasing with improvements in spectrometer designs. Software
Origins of DAVE Project
A pragmatic solution
DAVE project created in 2001 to address shortcomings.
Goals:
User-friendly tool which can accommodate the software requirements for a range of instruments at
the NCNR.
Full availability at no cost to users.
Instrument scientist should play key role in development, particularly data reduction aspects.
IDL chosen as development platform
Full support on several OS platforms
Comprehensive support for scientific programming and optimized for large array operations. Built-in widget toolkit for providing GUIs
Some IDL programming expertise
License agreement with RSI Inc allows free and unlimited distribution of DAVE executable with an
The DAVE suite
Main Application
Defines and maintains application wide user preferences (eg data directories, useful symbols, etc) Define an internal data structure for storing experimental data
Basic infrastructure to launch modules and pass data and to display PDF based help files.
Main Parts
Data Reduction Data Visualization Data Analysis - least squares fitting Planning and other Miscellaneous Tools
The DAVE suite
Data Reduction
10 supported instruments
Performing data reduction with the FANS
Data Reduction module.
The DAVE suite
Data Visualization
Line, contour, image and surface plots
The Data Browser interface showing data and graphic containers. Also shown is
an image view of a 2D dataset as well as a line plot of multiple 1D datasets.
The DAVE suite
Data Analysis
Dataset manipulations (merging, scaling, combining, etc) Fit data to model functions using least-squares fit
Single crystal analysis (projections, cuts, etc)
The DAVE suite
Miscellaneous Tools
More than a dozen tools
Eg Neutron cross-section tables and Calculations for compounds
DAVE Limitations
Limitations of DAVE
Had evolved organically (little common API, too much duplication of effort) Was too modular with little communication between components
Was becoming harder to maintain and even more difficult to extend
The arrival of iTools
Experimented with iTools (iPlot, iSurface, etc) in 2004
Contained many powerful features
Undo/Redo capability,
Messaging system for sending notification messages between components, highly customizable visualization graphics especially annotations,
saving application state, and many more
Was very impressed.
What are iTools
Has two parts:
1 The iTool Framework – a set of classes with well defined API.
It is a comprehensive component library consisting of dozens of classes.
2 The Intelligent Tools or iTools are a set of 7 flagship applications built on the framework. They are end-user applications
(iPlot, iImage, iContour, iSurface, iVolume, iVector, iMap).
For maximum flexibility, the framework must be used to develop own applications if they deviate too much from the example iTools.
Inadequate Documentation
Inadequate Documentation
Many undocumented classes.
Lack of complete tutorials even in iTool Developer's Guide book.
Took me 6 months to reach sufficient level of understanding.
Some improvements since the early days for example the addition of the object model
inheritance diagram.
Data-centric Interface
Data-centric UI
Example iTools present image processing interface – dominated by graphics window.
DAVE primarily manipulates experimental data
Therefore it needs a data processing interface.
Experimental dataset must be presented and maintained within a single entity. These datasets have many components besides the plottable data.
Graphics and tree widgets are only viable UI elements for user interaction with data.
• Can't use graphics because dataset is more than just the plottable data. • Adopt the iTool Visualization Editor widget interface.
• But add a custom DAVE Data Manager for storing experimental datasets.
Design application Tool Class (DAVETool which subclasses from IDLitTool) that
Maintains a separate data manager folder from the default system data manager to prevent automatic
modification of experimental data.
Activate or Deactivate menus based on selected data types from the Data Manager
Design appropriate data hierarchies for all supported file formats.
Data-centric Interface
The main interface.
Consists of: 1 Data Browser
shows contents of the Data Manager 2 A property sheet for for viewing details of selected data items 3 Menus
4 A Property sheet for selecting visualizations and associating data with the visualization Parameters.
Note: Note that some menus are deactivated because those operations cannot be applied to the selected data item, S(Q,w)
Using built-in Visualization classes
Extend Visualization Classes
IDLitVis* classes eg IDLitVisPlot
Accommodate metadata – eg axes titles and units (OnDataChangeUpdate) Handle histogram or point mode data – the mode is also specified as metadata. Automatically remove supporting data created
during a visualization when no longer required eg color palettes. Set IDLitData's auto_delete property for such data.
Also extend the visualization tool class so that it is dependent on the DAVE application tool (DAVETool) class.
AxisType property Meaning in a visualization
0 First indep. axis data (x-axis)
1 Second indep. axis data (y-axis)
2 Dependent axis data (z-axis)
3 Error in dependent axis data
Simplify visualization creation
Automate Visualization Creation
Automatically associate experimental data with visualization parameters for all supported visualizations. Extended IDLitData class with an axisType property.
It specifies how each data element is to be used in a visualization.
Replacing a Default Component
“ASCII Text” Reader component
Cannot successfully unregister/register the same
component name using different class names
without first deleting the IDL settings file
(itools_prefs.sav).
Modified ASCII text Reader
Default ASCII text Reader
What has been accomplished with iTools
Project management – users can now save and resume data + visualizations + application
state in a DAVE project file.
Data Manager. Easily handle neutron scattering datasets.
Visualizations. Adapt built-in IDLitVis* visualizations to work well with DAVE datasets.
Support for various data file formats (file readers) – DAVE and ASCII (SPE, column, group)
Support for various data file formats (file writers) – DAVE and ASCII (SPE, column, group)
Basic analysis functionality – data operations.
An operation component
Operations retrieve data from the Data Manager
User customizes operations through UI Service components
For most Operations, the Property Sheet widget can be used to
provide the UI Service dialog – no need to waste time designing custom dialogs!
With careful thought, it is possible to check user input for consistency.
E.g. The “Nos of datasets to sum” field can not be more than the nos of datasets in the Data Manager.
What is Good
What is good about iTool Framework
Really like the Operations component API especially the undo/redo buffer - excellent for a
data-processing application
Use of Property Sheet widget and other built-in dialogs give consistent look and feel to a large
application.
The data classes are comprehensive and can be used to create sophisticated data
hierarchies.
Suggestions for Improvement
Widget toolkit is dated – needs to be replaced!
New UI Elements such as worksheets are needed. Rich text support will be wonderful.
Cut and paste should work on all platforms.
Font support needs significant enhancement.
ħ and Å symbols are a must for scientific applications.
Better control over which components are registered by default (by the system object).
Without having to examine IDLitSystem class code to determine names of components to unregister.