Ab Initio EME Concepts

Ab Initio EME

May 18, 2010

EME (Enterprise Meta Environment)

•

Features of EME

•

Projects, Sandbox & Parameters

•

Reports, Versioning and Tagging

•

Checking out Projects, Sandbox, graphs and files

•

Editing the Sandbox, graphs and files

•

Checking in Projects, Sandbox, graphs and files

•

Dependency analysis

•

Air commands

July 6, 2010

Features

of EME

•

What is EME

•

Why EME

•

Interfaces to the EME

May 18, 2010

What is EME?

•Enterprise Meta Environment is high performance object oriented

storage system that inventories and manages various kinds of information associated with Ab Initio applications which may range from design information to operational data.

•It acts as the hub for data and definitions.

•In simple terms, it is a repository, which contains data about data – metadata.

•It consists of two basic types of metadata

¾ Technical metadata – application related business rules, record formats and execution statistics

¾ Business metadata – user defined documentation of what you deem important to track in your company , such as logical models, data dictionaries, roles and responsibilities.

May 18, 2010

Why EME?

• Avoid setup scripts, use EME parameters – more standardisation

• Source Control

• Dependency and impact analysis for the graphs in the repository

Interfaces to the

EME

•EME Web interface to Technical Metadata

May 18, 2010

EME Data Store Settings

•EME Data store is a specific instance of EME in the environment.

• This is a repository where different versions of code and its related data like the record formats, transformations etc are maintained.

• At any point of time a user can connect to only one such EME repository instance.

•To access an EME Data store, go to Settings in the GDE Menu and click on to Enable EME Connections which is shown below.

May 18, 2010 • After clicking the Enable EME Connections , The GDE is

July 6, 2010

•

What is a Project

•

Project Structure

•

Structure of project in EME

•

Different Types of Project

¾

Public project

¾

Private project

May 18, 2010

What is a Project ?

•Physically, each project in the EME data store is a collection of graphs, plans & related files stored in a well described structure.

•Logically, a project is a fairly self–contained and largely independent content area , a suite of graphs and associated files that accomplishes a business goal.

Project structure

•The directory tree of a project contains subdirectories for graphs, scripts, sqls , transforms etc.

May 18, 2010

Structure of a project in EME

/Projects Project1 Project2 mp run xfr dml db

Sub directory for AI Graphs

Sub directory for deployed shell scripts

Sub directory for transform files

Sub directory for record format files

Sub directory for database interface files

bin

sql

Sub directory for tools and utilities

May 18, 2010

Different Types of Project

Public Project

• A Public project shares their data and metadata with other projects

• A Project that is included by other Projects is termed as a Public Project

Private Project

• A Private project are not expected to be shared with other projects

July 6, 2010

Sandbox

•

What is a Sandbox

•

Structure of Sandbox

•

Sandbox vs Projects in EME

•

Environment Structure

May 18, 2010

What is a Sandbox?

•Sandboxes are the user work areas in order to work on files in EME Project

•To work on Projects, they must be checked out to a user work area on the file system where development and modification of code can happen.

•This working area on the file system is known as Sandbox.

•After the changes are complete the code is checked in from the user sandbox area to the EME data store

May 18, 2010

Structure of Sandbox

•Sandbox has the same structure of Projects in the EME but it lies in different file system. It is shown below.

/Mysandbox Project1 Project2 mp run xfr dml db

Sub directory for AI Graphs

Sub directory for deployed shell scripts

Sub directory for transform files

Sub directory for record format files

Sub directory for database interface files

bin

sql

Sub directory for tools and utilities

May 18, 2010

Sandbox vs. Projects in EME

• Sandboxes are work areas used to develop, test or run code associated with a given project. Only one version of the code can be held within the sandbox at any time.

• The EME Datastore contains all versions of the code that have been checked into it.

• A particular sandbox is associated with only one Project where as a Project can be checked out to a number of sandboxes.

Check-in

May 18, 2010

Environment Structure

EME

May 18, 2010

Creating a project by creating a Sandbox

•Create an empty sandbox in the GDE. Select Project>Create sandbox dialog.

•Host : server name

•Directory : The sandbox path

•Location parameter : leave default or give some name

•Click ok and the sandbox will be created in user area.

July 6, 2010

Parameters

•

What is a Parameter

•

Graph Parameter

May 18, 2010

What is a Parameter?

•A parameter is a name-value pair with some additional attributes that

determine when and how to interpret or resolve its value.

•Parameters are used to provide logical names to physical location and

should always be used instead of hard coded paths in graphs.

•Parameters are used to change the behaviour of graphs and projects

in a controlled and uniform way

•There are two types of parameters. ¾ Graph Parameters

May 18, 2010

Graph parameters

•Graph parameters are specific to the individual graphs and are private to them. They affect execution of the graph for which they have been defined.

•Graph parameters can be defined by navigating to Edit>Parameters in the GDE which opens the graph parameters editor.

May 18, 2010

Project Parameters or Sandbox Parameters

•Project parameters are inherited by all the graphs in the Project

and are accessed from the GDE by the sandbox parameter editor in Project>Edit Sandbox>Parameters.

•This shows a dialog box prompting to enter the sandbox path.

Choose the correct host and the sandbox path and press OK to open the sandbox parameter editor, which exactly like the graph parameter editor shown as above.

July 6, 2010

•

Reports

May 18, 2010

Reports

•Reports present summary EME data

• View the report using EME Web Interface

•The below list of reports can be taken from EME

¾ Graph reports

¾ Dataset reports

¾ Job report

¾ Directory report

May 18, 2010

Version Control and Tagging

• Each object under EME source control, which may be a file, a directory or a Project, exist as a series of versions, each of which is a representation of what was checked in by some user.

• It can optionally have a textual description attached to it called as a tag and a description as a comment.

• Each version is separately numbered and can be accessed by either the version number or the tag attached to it.

• Version numbers, which are integers and tags, are global to the whole EME Datastore.

• Tags are the basic units during migration of code across EME Datastore instances.

July 6, 2010

Checking out Projects, Sandbox, graphs

and files

•Check out

May 18, 2010

Check out

• The Ab Initio GDE provides wizards to check out code from the EME to sandbox.

• Check out updates the sandbox with the particular version of code that is being checked out from the EME. By default the latest

version of any object is checked out.

• Even any version of code can be checked out.

Check out Process

• Select Project>Checkout. Specify the Sandbox directory name and EME path.

• Specify the EME name, Project path in EME and Sandbox path in user area. It is shown below.

May 18, 2010 • After specifying all details click on to check out button.

• It will check out all the things from EME to sandbox

• To check out a project in existing sandbox , right click the sandbox name in the sandbox view and select check out

• To check out only the current graph or plan , click the check out current document icon( ) on the GDE menu bar

• To check out any file in the project, right click the file in sandbox view and select check out.

July 6, 2010

Editing the Sandbox, graphs and files

•Need for Locking

•Locking an Object

•Lock Button

May 18, 2010

Need for locking

•Lock is used to edit the graph or files.

•The lock button on the GDE indicates the status of an object with regard to locking.

•Locking the graph prevents other users from changing it at the same time you are working on it.

•Otherwise , conflicts can occur between sandbox and data store version of files.

Locking an Object

•Clicking the lock button on the GDE button

•Choose file> lock on the GDE menu bar

May 18, 2010

Locking Button

•The lock button’s appearance identifies the locking status of the object you are viewing

•If an object has never been checked in , there is no lock associated with it. The lock button on the tool bar is an open gray lock on gray back ground.

•When a graph or file is unlocked , the lock button on its toolbar

is an open yellow lock against a gray back ground. A unlocked object is read only and cannot be edited in the GDE.

•Sometimes an object lock button indicates that another user

already owns the lock , which means you cannot edit the object . In this case , the lock button is a closed lock on red back ground.

May 18, 2010 • There is no way in the GDE to break another users lock. • Once you set the lock , you are the owner. You can modify ,

save, and check in the locked object. The lock button is now a closed recessed in green back ground

• Locking when sandbox copy of the object is out of date. It will show below dialog box.

• Releasing a lock without checking in changes , it will displays a dialog. Save changes or release lock without reverting

July 6, 2010

Internal and

Checking in Projects, Sandbox, graphs and

files

•Check in

May 18, 2010

Check in

•Once you have edited your sandbox files, you have to check them in to put the new versions under source control in the EME datastore and make them available to other users.

Check in process in GDE

•To check in an entire sandbox , right click the sandbox name in the sandbox view and select check in , or select

Project>check in on the GDE menu bar.

•To check in only the graph , you are currently editing , click the check in document icon ( ) on the GDE menu bar.

•To check in any sandbox file, right click the file in the sandbox view and select check in.

July 6, 2010

May 18, 2010

Dependency Analysis

•Dependency Analysis is a process by which the EME examines a project in its entirely and traces how data is

transformed and transferred, field by field , from component to component within and between graphs.

• Using Dependency analysis we can observe the following things:

• How operations that occur later in a

graph(“downstream“) are affected by components earlier (“upstream”) in the graph

• What data is operated on by each component, and how.

• All details of every component in the project.

• What happens to each field in a dataset throughout a graph.

May 18, 2010

There are four ways in which the dependency analysis can be controlled :

1) It can be directly invoked in the GDE, by selecting Project > Analysis……. To start the Analysis wizard.

2) It can be directly invoked from the command line, by running the air project analyze- dependencies command.

3) It can be invokes implicitly at check-in, depending on the settings you select in the Check-in wizard’s Advanced Options under the Analysis tab.

4) You can also create an analysis_level parameter and use it to specify the amount of dependency analysis you wish to take place during check-in. The setting of this parameter(if it exists) restricts all other analysis level settings.

July 6, 2010

AIR Commands

•Command line syntax of air command

•Air Sandbox group

•Air Project group

•Air object group

•Air Lock group

•Air repository group

•Air Branch group

May 18, 2010

Command line syntax of Air command:

•The air utility is a command-line or shell-script interface for administering and managing the EME.

• All air commands have the following command-line syntax.

air[-root root] [-version version | tagname][-remote]

¾ -root root specifies root of the data store.(Optional)

¾ -version version (Optional) specifies the version number of the data store to be accessed by the air command. The version number can be specified as an integer or as a tag.

¾ A tag is a symbolic name that you can associate with a specific version when you check in a project.

¾ -remote specifies that the air utility should start the EME server remotely even if a direct connection is possible.

May 18, 2010

The air sandbox commands perform operations on sandbox.

• air sandbox create – creates the sand box in specified path.

syntax- air sandbox create path

¾ Path – specifies a directory.

¾ [ -template path ] – creates the sandbox from a specified template sandbox at path.

¾ [-prefix prefix] – Add prefix to the predefined sandbox parameters.

¾ [-replace s1 s2] – change one thing to another thing

May 18, 2010

Air Sandbox group

• air sandbox diff –displays difference between two graphs or two files syntax- air sandbox diff {path | path1 path 2}

¾ [Path] – specifies a directory in which file is located .

¾ { -version version path | -version version path1 –version version path2} – specifies the version which has to be compared

¾ {-text} – forces text differencing , regardless types of files.

• air sandbox find – identifies root of sandbox

syntax- air sandbox find [path]

¾ [Path ]– specifies a directory in which to start looking .

¾ {-up|-down}- upward and downward search , towards and downwards of the directory specified.

¾ { -sandbox} – specifies sandbox found should be listed

¾ {-project} – specifies EME project the sandbox is associated with should be printed.

May 18, 2010

Air Sandbox group

• air sandbox info –displays info about the files associated with

sandbox

syntax- air sandbox info {path}

¾ [Path] – specifies a directory.

• air sandbox lock – performs lock operation on sandbox objects

syntax- air sandbox lock path

¾ [Path ]–the path to file checked in sandbox

¾ [-set|-release|-break|-release].- set locks in a file, releases lock , breaks another user lock , resets the lock broken by another user

•air sandbox parameter–creates sandbox parameter

syntax- air sandbox parameter [pset] parameter [interpretation] [value] [flag]

¾ [Pset]–specifies the location of parameter

¾ [-parameter].- name of the parameter

May 18, 2010

Air Sandbox group

• air sandbox pset – creates or modifies input value for graph. syntax- air sandbox pset pset-path

¾ [pset-path] –path to input values.

¾ [-create] – creates an input value set.

• air sandbox revert – reverts sandbox object to versions in data store.

syntax- air sandbox revert –basedir sandbox

¾ [-basedir]– path to sandbox

¾ [-files path ].- sandbox relative paths of object to be reverted

•air sandbox run – runs a graph or file.

syntax- air sandbox run path

May 18, 2010

Air Project group

This manages the datastore projects.

• air project create – creates a new project and specifies various project attributes.

syntax- air project create project name

¾ Project-name – name of the project

¾ [ -location location ] – name of the project parameter that specifies the location of the project.

¾ [-prefix prefix] – prefix for default project parameters created automatically.

May 18, 2010

Air Project group

• air project export –check out or copies projects from datastore into a

sandbox.

syntax- air project export project-path –basedir sandbox-root-dir

¾ Projectpath –project path and name in datastore

¾ [ -basedir ] – specifies an existing or new sandbox

• air project import – check in or copies sandbox files into an existing

datastore project.

syntax- air project import project-path –basedir sandbox-root-dir

¾ Projectpath – path to the project to which you want to check in files

May 18, 2010

Air Project group

• air project mkdir – creates a directory in specified project.

syntax- air project mkdir [-q] project-name [-no-export] dir-name

¾ -q – supress error if directory already exists

¾ [-no-export] – the directory is not exportable

• air project add – Manually adds one or more files to a datastore

project’s internal list of files.

syntax- air project add [-q] project-name relative path

¾ -q – supress error if relative path already exists

¾ Relative path – relative path of file to be added

•air project modify – edits the specified project location parameter.

syntax- air project modify project-name [-location loc-parameter]

¾ Project-name – name of the project

May 18, 2010

Air Project group

• air project move –Moves a project from one place to another within

datastore.

syntax- air project move old-name new-name

¾ Old-name – datastore path specifying the original location

¾ New-name – the path which project to be moved

• air project parameter – deals with project parameter.

syntax- air project parameter proj-path [-export] parameter [interpretation] value

¾ Proj-path – path to project

¾ -Export – exported to environment

¾ Parameter – the parameter name you want to add or edit ¾ Interpretation – determines how value is interpreted

May 18, 2010

Air Project group

• air project revert – reverts one or more objects in a Project

syntax- air project revert proj-path –version ver-path relative-path

¾ Proj-path –path to the project containing objects you want to revert

¾ -version ver-path – version number

¾ Relative-path – relative path of the object to be reverted.

• air project set-executable –ensures that a file will be marked

executable when it is exported.

syntax- air project set-executable proj-path relative-rpath[-not]

¾ Proj-path – path to project containing the file

¾ relative-rpath – relative path of file within the project ¾ -not – specifies than an executable file be marked not

May 18, 2010

Air Project group

• air project set-type – overrides the mime type assigned to a single

datastore project

syntax- air project set-type proj-path -force relative-path mime-type

¾ Proj-path –path to the project in which object resides

¾ -force – by default you cannot set . You have to force.

¾ Relative-path – relative path to the object whose MIME type you want to change.

• air project show – displays project attributes.

syntax- air project show proj-path

May 18, 2010

Air Object group

It manipulates object and directories in the datastore and control permission to access objects.

•air object access– To test whether all permissions exists for the

project.

syntax- air object access rpath [r|w|x|f]

¾ Rpath – path to a data store project

¾ -r – test whether you can read the specified object.

¾ -w - test whether you can write the specified object

¾ -x - test whether you can execute the specified object

¾ -f – test whether you can see the existence of the specified project.

• air object show – displays project attributes.

syntax- air project show proj-path

May 18, 2010

• air object chown

– Changes the ownership of one or more datastore

objects. Only the EME administrator can perform this operation.

syntax - air object chown[-Rf]user[.group] rpath...

¾ -R - Optional. Recurses into rpaths if they refer to a directory.

¾ -f - Optional. Attempts to continue even if the operation fails for one or more rpaths.

¾ user - New owner expressed as a valid user name.

¾ .group - Optional. Group designation, preceded by a period.

¾ rpath - Path to a datastore object, or object ID, whose ownership you want to change. You can specify multiple

paths.

May 18, 2010

Air Object group

•

air object cp

–

location within the datastore. The command copies rpath1, retaining links to any other objects in references.

syntax - air object cp rpath1 [-R] rpath2

¾ rpath1 Path to a datastore object, or object ID, to be copied.

¾ -R Optional. If specified, command copies the specified object

rpath1 and all objects referred to by rpath1. If not specified, the specified object rpath1 is copied, but objects referred to by rpath1 will be linked but not copied.

May 18, 2010

Air Object group

•

air object create-parameter-set

–

from the parameters associated with a datastore object.

syntax- air object create-parameter-set file graph_path

¾ file - Name of the parameter set file. It should have an extension of .pset

¾ graph_path - Path to the graph in the datastore.

¾ switch - The name of a switch parameter.

May 18, 2010

Air Object group

•

air object delete

the same domains, for permanent removal from the datastore.

syntax - air object delete –R –whole–domain [-force] rpath [ rpath2…]

¾ -R - If rpath is a directory, all of its directory child and grandchildren and all objects in the descendants’ domains will be marked for deletion.

¾ -whole-domain - All objects in the same domain as the object specified by rpath will be similarly removed.

¾ -force - Do not prompt for confirmation of deletion.

¾ rpath rpath2… - Datastore paths to the objects you want to delete.

May 18, 2010

•

air object In

rpath1.

syntax - air object In rpath1 rpath2

¾ rpath1 Path to a datastore object, or an object ID.

¾ rpath2 New name for the existing object.

•

air object mv

syntax - air object mv rpath1 rpath2

¾ rpath1 Path to the datastore object you want to rename.

¾ rpath2 New name for the object rpath1.

May 18, 2010

•

air object load

interchange format. This file was created by running the air object save command. The objects are loaded to the datastore location corresponding to the ones in which they were saved.

syntax - air object load [-table-of-contents] file-name

¾ file-name - Standard UNIX file-name of the file storing the objects you want to load. If file-name is -, then

the data is read from standard input.

¾ -table-of-contents - Specifies that nothing be loaded, but instead print a listing of the file’s contents.

May 18, 2010

•

air object revert

version. Objects newer than specified version are deleted from the datastore.

syntax - air object revert [-revert-links] version rpath……

¾ -revert-links - Optional; if specified, then objects linked to rpath will also be reverted. If not specified, no action is taken on linked objects.

¾ version - Version number (integer) or tag (string) of the datastore that you wish to back out to.

¾ rpath… - Path to an object, or object ID.

May 18, 2010

•

air object mkdir

are created as necessary.

syntax - air object mkdir rpath

¾ rpath Path of the directory you want to create.

•

air object unlink

syntax - air object unlink[-f] [-no-project-remove] rpath

¾ -f - Specifies that no error be generated if rpath does not exist.

¾ -no-project-remove - Specifies that the object be removed without removing it from source code control.

¾ rpath - Datastore pathname of the directory.

May 18, 2010

•

air object rm

datastore project, then the system performs an implicit air project remove.

syntax - air object rm[-f][-r][-no-project-remove] rpath…..

¾ -no-project-remove - Specifies the object be removed without removing it from source code control.

¾ -f - Specifies that no error be generated if rpath does not exist.

¾ -r - Specifies that subdirectories of rpath be removed recursively.

¾ rpath - Path to the directory entry you want to remove.

May 18, 2010

•

air object save

interchange format. You can subsequently load the file by running air object load. You can use air object save to migrate objects from one datastore to another.

syntax - syntax - air object save file-name rpath…

[-include{common| local| rpath…}]… [-external { common| local| erpath…}]… [-format {1| 2| 3}]

[-export epath…]

[-settings common| project…] [-save-jobs] [-analyzed] [-no-default] [-tag tag] [-comment comment] [-Stag tag] [Scomment comment] [

May 18, 2010

•

air object versions

-

versions or all versions of the specified object or the datastore. History includes version number, date, user, tag, and comment for the specified object.

syntax - air object versions[-verbose][rpath]

¾ -verbose - Optional. Displays all versions for an object or the entire datastore. (If nor specified, only tagged versions will be

displayed.)

¾ rpath - Optional. Path to the object about which you want version information. If not specified, displays

information for the datastore as a whole.

May 18, 2010

Air Lock group

It manages locks to prevent file collision.

• air lock set – sets or modifies a lock on datastore objects or projects.

syntax- air lock set [-force] [-breakable | -unbreakable] [-auto-

release | manual-release] [-modify attribute] [-object object | -project project]

¾ Force –breaks another lock and sets a lock for you

¾ [ - breakable | -unbreakable] – specifies whether other user can break the lock

¾ [auto-release | manual-release] – the lock is automatically has to be released automatically or not after check in

¾ [-modify attribute] – modifies the breakability and release attributes of an existing lock

¾ -object – object is the path which you want to set lock

May 18, 2010

Air Lock group

air lock release – sets or modifies a lock on datastore objects or

projects.

syntax- air lock release [-object object] [-project project]

¾ -object – object is the path which you want to set unlock

¾ -Project – Project is the path which you want to set unlock

air lock break – break users lock on a given datastore object.

syntax- air lock release [-object object] [-project project]

¾ -object – object is the path which you want to break

¾ -Project – Project is the path which you want to break

air lock show –displays a listing of locks for a given user , object or

project.

syntax- air lock show [-user username] [-object object] [-project project]

¾ -object – object is the path which you want to display

May 18, 2010

Air Repository group

The air repository commands operate on the storage system of the datastore

•air repository create - creates a datastore and installs certain default system objects .

•air repository chmod –change permissions on datastore.

•air repository chown –change ownership of the datastore.

•air repository rename – renames all files that makes up an EME

data store.

•air repository restart –shuts down and restarts the EME process. •air repository start – starts the EME process.

•air repository shutdown –Shuts down the EME process. •air repository backup –it is used to backup of datastore

May 18, 2010

Air Branch group

The air repository commands operate on the storage system of the datastore

•air branch create - creates a new branch with the name new-branch from an existing branch .

•air branch delete – deletes object on the branch.

•air branch freeze –freezes or unfreezes a branch .

•air branch list – displays information about objects on the branch. •air branch modify – allows you change the title of the branch .

May 18, 2010

QUESTIONS ?

May 18, 2010 66

THANK YOU