39189489 Beyond the Basics 4

(1)

This section contains topics that go beyond the basic information needed to build Ab Initio graphs, grouped as follows:

Working in Sandboxes

Using Parameters

Using Phases and Checkpoints

Using Subgraphs and Macros

Creating Custom Components

Using Conditional Components

This section contains the following topics:

What is a Sandbox?

Using a Sandbox

Using Sandbox Parameters

Sandbox Parameters and Graph Parameters

Adding Subdirectories to a Sandbox

Sandboxes and the Ab Initio Environment

Sandboxes and the EME

Sandboxes without the EME

A "sandbox" is a special directory (folder) containing a certain minimum number of specific subdirectories for holding Ab Initio graphs and related files. These subdirectories have standard names that indicate their function. The sandbox directory itself can have any name; its sandbox properties are recorded in various special and hidden files that lie at its top level (the my_project directory, in the following illustration):

(2)

For information on how to create a sandbox, see Create Sandbox Dialog. See also Creating a Sandbox.

If you are using the EME, you can create a sandbox during checkout of a project. See Checking Out Projects, Graphs, and Files.

If you are creating sandboxes in an Ab Initio Environment, the sandboxes will havd additional subdirectories. See

Sandboxes and the Ab Initio Environment.

A sandbox is not limited to containing only the default subdirectories shown here. You can add any number of additional ones.

Sandbox subdirectories are used to organize and store Ab Initio graphs and the files that are associated with them. Graphs and their associated files are all considered to be metadata (as opposed to the data which the graphs operate on), and sandboxes normally hold only metadata.

Consider the following simple graph:

Viewed as a collection of various kinds of files, this graph consists of the following:

Data files — the Transactions input dataset and the Transaction Data output dataset. If these are multifiles, the actual datasets will occupy multiple files.

Record formats — there are two separate record formats in the graph, one for the input file, the other for the output. These record formats could be embedded in the Input File and Output File components themselves, but here (for the sake of the example) we will assume that they are stored in separate record format (.dml) files.

(3)

Transforms — the Reformat component performs various changes on the format of incoming records, as specified by the statements in its transform function. The transform function can be embedded in the component, but (as with the record formats) we will assume that it is stored in a transform (.xfr) file, for the sake of our example.

Graph file — the graph itself is stored complete as an .mp file.

Deployed script — if deployed as a script, the graph will also exist as a .ksh file, which has to be stored somewhere.

Without a sandbox, you have to decide how to organize all these pieces of your graphs, and you have to take care to maintain your organizational conventions in all your applications—not doing so will quickly lead to chaos. But with a sandbox, you automatically have a set of standard subdirectories into which everything in a graph can be put.

Sandboxes also help make your applications portable. Graphs reference their location- and platform-dependent information through sandbox parameters, rather than having it separately hard-coded in the separate components.

Here is how a simple graph and its pieces would be stored in a sandbox:

Every piece of the graph has a specific subdirectory in which to go. Metadata such as record format files (stored in the dml directory) or transforms (stored in the xfr directory) can easily be shared among graphs, as they are all located in the same relative place within the sandbox.

If the graph employed database components, the db directory would be used for the database configuration file or files.

!

Notice that data files are not usually stored in sandboxes. Sandboxes are for organizing and storing metadata, not the data itself.

Typically, you create sandbox parameters to refer to any data files or directories used by your applications, and you use these parameters in the graph components to refer to the data. See Creating Additional Sandbox Parameters.

To place yourself "in" a sandbox, you must open (in the GDE) a graph that is in that sandbox's mp directory, or save the graph you are working on in the mp directory of the sandbox you wish to work in.

(4)

"

Graphs refer to the sandbox subdirectories by using sandbox parameters.

"

Sandbox parameters are variables which are visible to any component in any graph which is stored in that sandbox. Here are some examples of sandbox parameters:

$PROJECT_DIR $DML

$RUN

For the difference between sandbox parameters and graph parameters, see Sandbox Parameters and Graph Parameters.

# "

For example, within an Input File component, you might specify $DML/transactions.dml

to reference a record format file for the transactions dataset. Here is what this would look like as specified in the Ports tab of the Input File properties dialog:

$DML is one of the default parameters present in all sandboxes, and its value is always the full path of the dml subdirectory in that sandbox.

"

The default sandbox parameters in a GDE-created sandbox are these six: PROJECT_DIR — absolute path to the sandbox directory DML — relative sandbox path to the dml subdirectory XFR — relative sandbox path to the xfr subdirectory RUN — relative sandbox path to the run subdirectory DB — relative sandbox path to the db subdirectory MP — relative sandbox path to the mp subdirectory

These six parameters are automatically created (and assigned their correct value) whenever you create a sandbox.

"

However, you can create additional sandbox parameters yourself, and you can use them to store values other than the paths of sandbox subdirectories. In fact, you can use sandbox parameters to store any value you wish to make available, for whatever reason, to graphs in that sandbox.

(5)

For information on editing sandbox parameters and creating new ones, see Sandbox Parameters Editor.

Although data files are not usually stored in sandboxes, there is no reason why you cannot create sandbox parameters to point to the data locations outside the sandbox. For example, you might have a transactions data file located at

/my_data_path/transactions.dat

You could define a sandbox parameter called DATA_PATH, set it to the value my_data_path, and reference the dataset like this:

To test the validity of a new sandbox parameter that specifies a path, simply type the parameter's name in a component's URL box (for example, as shown in the illustration above) and click Browse. If the parameter is successfully resolved, the Select Data File dialog will appear, displaying the contents of the directory referenced by the parameter's value.

"

The difference between sandbox parameters and graph parameters is:

Graph parameters are visible only to the particular graph to which they belong Sandbox parameters are visible to all the graphs stored in a particular sandbox

In the figure above, the arrows show how the values of the sandbox parameters $DB, $DML, and so on, are propagated to the individual graphs in the sandbox. The arrows have been omitted from the second graph for the sake of clarity.

(6)

The individual graphs also can have their own parameters, some of which are shown, as examples, in the figure:

$graph1_parm1 and $graph1_parm2 in the first graph, and $graph2_parm and $graph2_parm_2 in the second graph. These parameters are completely separate from and unaffected by the sandbox parameters (although you can use sandbox parameter values in the definitions of graph parameters).

"

At runtime, the graph parameters are set after the sandbox parameters. This means that, if the graph has a parameter with the same name as a sandbox parameter, the graph parameter value will override the same-named sandbox parameter value.

In the diagram above, Graph 1 has a parameter called PARM_X, and there is also a sandbox parameter that happens to have the same name. As a result, Graph 1 will "see" the value of the graph parameter called PARM_X, not the value of the sandbox parameter PARM_X. Graph 2, on the other hand, will see the sandbox parameter PARM_X.

"

$

The following diagram shows what happens when a sandbox parameter employs in its definition another sandbox parameter, which happens to have the same name as a graph parameter.

The sandbox parameter's value is overwritten by the same-named graph parameter, as already explained. But now, suppose this sandbox parameter is used by some other sandbox parameter in its definition. In that definition, the sandbox value of the parameter will be used, and it is this defined value of the parameter that will be seen by a graph. The following diagram illustrates this situation.

(7)

For further information about graph parameters, see Graph Parameters and the Graph Parameters Editor.

Adding a subdirectory to a sandbox consists of two steps: 1. Creating the subdirectory

2. Creating a sandbox parameter to reference the new subdirctory

Creating the directory should be a straightforward operation. For information on creating sandbox parameters, see the

Sandbox Parameters Editor. See also Creating a Sandbox and Testing the New Sandbox Parameters.

%

&

If your sandbox is attached to a project, creating a new subdirectory in the sandbox will probably mean that you want to add the subdirectory (and its contents) to the project.

To do this, simply create the subdirectory and check the project in. The Checkin Wizard will detect the presence of the new subdirectory and prompt you to specify whether you wish it to be checked in or not. If you do not want to add the

subdiectory to the project, you can specify that at this point.

%

# '

"

In an Ab Initio Environment, you can edit the project templates so that any projects subsequently created will have the additional subdirectories by default. For further information, see The Ab Initio Environment.

(8)

%

# '

"

If you create your sandboxes using the Ab Initio Environment, they will by default have two subdirectories additional to the five you get in an "ordinary" sandbox. Here's what the default form of an Ab Initio Environment sandbox looks like:

The two additional subdirectories are:

bin — for executable files

Usually this directory is used only in public projects, where it is used to hold macro scripts and third-party executables for Run Program components. A "public project" is an Ab Initio Environment concept: it is a common project, intended to be accessible to other projects. Public projects are the Ab Initio environment's mechanism for sharing data among projects.

sql — for SQL files

Used to hold .sql files used by database components, in particular by the Run SQL component.

#(#

You can have sandboxes with projects, or sandboxes without projects.

A sandbox that is not associated with a project is simply a special directory. See What is a Sandbox? for more information.

)

&

A sandbox that is associated with a project is by definition under source control in an Enterprise Meta>Environment datastore. The project is the "master copy" of the sandbox contents; you check out copies of files in the project to work on in your sandbox, and check in the altered files to update the project copies under source control. The directory structure of the project in the datastore is exactly the same as the sandbox directory structure in your filesystem.

The sandbox now contains only some set of particular versions (usually the latest) of the files that make up its contents. The project contains all the versions of the files.

(9)

&

A project and a sandbox are thus essentially the same thing, seen from different points of view. A sandbox is a copy of a project. There can be (and usually are) many such copies. Multiple users can set up sandboxes based on the same project, thus creating multiple copies of it. Each sandbox can be associated with only one project, but a project can be (and usually is) associated with multiple sandboxes. Any user working on the project will have his or her own sandbox in which to work.

When you check a sandbox in to the EME, the contents of the sandbox are copied into the EME datastore project area. When you check the project out, the filesystem copies of the project contents are updated in your sandbox.

To turn a non-EME sandbox into a project in the EME, all you have to do is check the project into the EME; the project will be created as you do so. See Creating a Project.

See also Sandbox and Project Parameters.

)

#(#

A standalone (non-EME) sandbox is mainly an organizational structure designed to hold Ab Initio applications. Its contents are metadata: graphs themselves (.mp files) and deployed graphs (.ksh scripts), along with files containing other metadata used by graphs: database configuration, record format, and transform files. See What is a Sandbox? for more information on this.

*

%

# '

"

The Ab Initio Environment is a collection of functionality that allows you to set up a framework in which to organize your Ab Initio development, testing, and production work. It uses EME projects, common projects, and common data areas to build an open-ended hierarchy that can be as simple or complex as you require.

Note that the term "Ab Initio Environment" can be a little ambiguous in certain contexts. It can be used to refer to an instance of an organizational framework that you have created, or to the set of Ab Initio executables that you use to create it. In these paragraphs, "Ab Initio Environment" signifies the thing(s) you create with the software: the collection of projects and sandboxes, the common data areas that are associated with them, and so on.

(

The main characteristics of an Ab Initio Environment are the following:

Every sandbox (and project) in an Ab Initio Environment has by default an expanded set of subdirectories

In an Ab Initio Environment, you must create new projects either by running the install-environment script (when you create the new environment), or by running the create-project script.

Every Ab Initio Environment is by default set up with ready-to-use data areas for serial and multifile data, as well as logs, error reporting, and the like

Each of these areas is common to all projects in that environment, although each project has its own separate sub-section of it; these can be further subdivided into user-specific sub-areas.

You create an Ab Initio Environment by running the install-environment script.

Every Ab Initio Environment has a single central common project (called the "environment project") whose parameter definitions are included in all other projects in that environment; these values allow the individual projects to access the data areas and any other values common to the environment

(10)

Including a given environment project is what puts a given project "in" a specific Ab Initio Environment.

"

This section contains the following topics:

Generic Parameter Information

Graph Parameters

Graph Parameters Editor

"

%

"

A parameter specifies some aspect of graph, subgraph, or component behavior. For example, the Filter by Expression

component requires an expression to choose which records are written to the out port. This expression is specified in the select_expr parameter.

A parameter is a name-value pair, with a number of additional associated attributes that describe when and how to interpret or resolve the value. Parameters are used to change the behavior of graphs and sandboxes in a controlled and uniform way. You can assign values to parameters on the Parameters tab of the Properties dialog or in the parameters editors.

Required parameters that do not have values assigned are indicated by yellow boxes .

Interpretation

Graph Parameters

Graph Parameters Editor

Adding a Parameter in a Parameters Editor

"

A graph is completely defined by the totality of its parameter values. A parameter is a name-value pair, with a number of additional associated attributes that describe when and how to interpret or resolve the value. Parameters are used to change the behavior of graphs and projects (groups of graphs in a sandbox) in a controlled and uniform way.

Graph parameters are associated with individual graphs and are private to them. They affect the execution only of the graph for which they are defined. All the specifiable values of a graph, including each component's parameters (as well as other values such as URLs, file protections, record formats, and so on) comprise that graph's parameters.

Graph parameters are considered to be part of the graph with which they are associated.

Usually you use the Graph Parameters Editor when you want to parameterize a graph—that is, make it reusable either as a deployed script or as a graph run in the GDE—with various runtime settings and values specified on the command line or in the GDE when it is invoked. For much of your graph development you may never need to use the graph parameters editor.

(11)

the graph parameters editor.

Formal Graph Parameter

Adding a Formal Input Parameter to a Graph

Exporting Component Parameters to the Top Level of a Graph

Order of Evaluation of Parameters

Adding a Parameter in a Parameters Editor

+ "

"

A formal graph parameter is a parameter you substitute for a path and/or filename when you create a graph. This allows you to specify the value of that parameter at runtime. For example, you might use a formal graph parameter to indicate a working directory. You can then specify one working directory when you run the graph while it is still in development or testing, and another directory when you run the graph in production. You can also use a formal graph parameter to defer until runtime the location and/or name of a dataset. This lets you run the same graph using different datasets that share the same record format.

#, ( -#

Suppose that you enter $DAILY_TRANS as the path to a dataset. You can give DAILY_TRANS a value in several ways: If the graph is deployed as a script, the production person who runs the script can first use the Korn shell to enter a command like:

export DAILY_TRANS=absolute_path

where absolute_path is the absolute path (including filename) to the dataset.

You can edit the host setup script to give DAILY_TRANS a value using something like the above command. If you are using the EME, you can include a common project whose project parameters include DAILY_TRANS. The sandbox must be located on the runtime run host, the project must be checked out at runtime, and its parameters must have the Export option selected.

+ "

%

"

Suppose you have created a graph that performs some task, and you want to use a different path to an input file dataset each time you run the graph. The best way to accomplish this is to define a formal graph input parameter of type string for the graph, and then reference that graph parameter from the Layout parameter of the Input File component in the graph, using

$ substitution.

To define the formal input parameter for the graph:

1. On the menu bar of the GDE, choose Edit > Parameters to open the Graph Parameters Editor. 2. In the Editor's grid, click in the row underneath the row containing the name of the graph.

This row will typically contain the name of one of the components in the graph. 3. On the menu bar of the Editor, choose Edit > Insert Parameter.

(12)

4. Replace noname with a name for the parameter. For the purposes of this example, use input_file_url.

5. Right click anywhere in the row, and choose Details from the shortcut menu to open the Parameter Details Window.

6. In the Scope row of the Parameter Details window, click the arrow and choose Formal.

Although the default for Scope is Local, you need to use Formal for a parameter that accepts user input. 7. In the Kind row, either leave the default value, or select one of the other choices, depending on the way in which

you want the parameter to receive its value.

For the purposes of this example, leave Kind set to the default, positional. 8. In the Type row, leave the type set to its default, string.

9. In the Parameter Details window, click OK.

10. In the Value column of the Graph Parameters Editor, change the value for the Input File component's Layout parameter to $input_file_url.

11. Click OK.

If you run this graph from the GDE, the GDE will prompt you to supply a value for the input_file_url parameter.

$ substitution will substitute this value for $input_file_url in the Layout parameter of the Input File component. If you run the deployed script for the graph, you must pass the value for input_file_url as the first argument on the command line used to invoke the script. $ substitution then proceeds in the same way it does when you run the graph from the GDE.

#

"

* - '

You can export any number of component parameters to the top level of a graph or subgraph, thus making them Formal graph parameters. When you run the graph in the GDE, you will be prompted for values for the parameters. If you run the graph as a deployed script, you will have to specify values for the parameters on the command line (if the parameters' Kind has been declared as Keyword or Positional), or supply the values in properly-set environment variables that have the same names as the parameters (if the parameters' Kind has been declared as Environment Variable).

This technique allows you easily to partially parameterize a graph for testing or any other purpose. To remove the parameterization, you simply delete the top-level parameters in the graph parameters editor.

To export a component parameter to the top level of a graph or subgraph, do the following: 1. Open the parameters for the component whose parameters you wish to export.

You can do this in either of two ways:

(in the GDE graph display) Open the Properties dialog for the component and click the Parameters tab. (from the graph parameters editor) Right-click on the row of the parameter you wish to export, and select the Goto Parameter parameter name item. This will close the graph parameters editor, open the component's Properties dialog, and put you in its Parameters page.

2. Select the parameter you wish to export.

For example, let's assume you want to export the key parameter of a Join component in a graph. In the Join component's Parameter Properties page, you select the key parameter:

(13)

Notice that, in the lower portion of the dialog, the parameter's Kind is shown in a grayed-out text box. As long as the parameter's Kind is not Implicit (which means that the parameter's value is calculated internally by the GDE), it can be exported.

3. Click the Export button, which is located just above the Kind box. The Export as Parameter dialog appears.

(14)

4. Fill in the fields with the appropriate values. Make sure the parameter's Kind is Formal. When you are finished, click OK. You will be returned to the component's Properties dialog.

Click OK.

At this point, if you re-open the graph parameters editor, the parameter is displayed at the top tree level, if it is a top-level parameter. If the parameter is in a component in a nested subgraph, it is exported up one level, not to the top level of the graph.

(15)

To un-export the parameter, simply delete it in the graph parameters editor.

to mean command substitution.

.

#'

"

When you run a graph, parameters are evaluated in the following order: 1. The host setup script is run.

2. Common (that is, included) sandbox parameters are evaluated. 3. Sandbox parameters are evaluated.

4. The project-start.ksh script is run. 5. Formal parameters are evaluated. 6. Graph parameters are evaluated. 7. The graph Start Script is run.

"

#

To display the parameters editor for a graph, open the graph in the GDE, and then select Edit > Parameters from the GDE menu bar.

Use the Graph Parameters Editor to:

Add parameters to a graph by specifying information about them in the Graph Parameters Editor Columns. Edit parameters in a graph, including the parameters of any subgraph or component in the graph, by changing the information in the Editor's columns.

Use the Graph Parameters Editor Menus and Graph Parameters Editor Toolbar to perform related tasks. The graph parameters editor serves the same purpose as the individual components' Properties dialogs. It gives you comprehensive access to all the parameters in a graph (and its subgraphs) at once, rather than forcing you to go back and forth among the separate components to edit their parameters and other values.

In short, every graph value is represented in the graph parameters editor, regardless of whether it is called a "parameter" or not in a component (see Name). Anything that you can do in a component's Properties dialog — set parameters, edit DML,

(16)

edit transforms, and so on — you can also do in the graph parameters editor.

#

"

Each row in the editor represents a parameter. Each column in the grid represents a different aspect of a parameter. You click on a cell to edit it. To select an entire row, click in the shaded box to the left of the row. You edit parameters' Value cells by clicking the Edit Button.

"

!

To get a complete display of all the column values for a particular parameter, click in the parameter's Name cell, or click the button to the left of the parameter's row, then right-click and select Details from the floating menu.

Generic Parameter Information

Graph Parameters

Exporting Component Parameters to the Top Level of a Graph Order of Evaluation of Parameters

"

#

"

By default, the Graph Parameters Editor displays the Name, Value, and Description columns. In addition to these three, the following columns can appear in the grid of the Editor:

Scope Kind Type Location Value Edit Button Interpretation Required Export to Environment

To determine which columns appear in the grid of the Editor, choose View > Customize on the Editor's menu bar, select the check boxes for the columns you want to appear, and clear the check boxes of those you do not want to appear. To edit the values for the above columns without having the columns appear in the Editor, do one of the following:

Choose View > Details on the Editor's menu bar.

Right click in the row of the parameter you want to edit, and choose Details from the shortcut menu. The Parameter Details window opens, and you can perform your edits in it.

(17)

"

#

(

The Graph Parameters Editor offers five menus.

+%-#(#/

The File menu of the Graph Parameters Editor contains the following items:

Save (Ctrl + S) — Saves the current changes in the Graph Parameters Editor.

Validate — Checks the selected parameter, or the parameters of the selected graph, subgraph, or component to make sure there are no errors in syntax, and that all $ references can be resolved.

Print — Opens the Print dialog, so you can print the grid of the Graph Parameters Editor. Close (ESC) — Closes the Graph Parameters Editor.

#!%*(#/

The Edit menu of the Graph Parameters Editor contains the following items: Cut (Ctrl + X) — Removes the selection and places it on the clipboard. Copy (Ctrl + C) — Copies the selection and places it on the clipboard. Paste (Ctrl + V) — Inserts the contents of the clipboard at the insertion point.

Insert Parameter — Adds a new parameter above the selected row in the center grid. Delete Parameter — Removes the selected parameter.

Find (Ctrl + F) — Searches for the string specified in the Find What box of the Find dialog. In the Search list, select one of the following:

By Columns to search the current column and then continue with the next column By Rows to search the current row and then continue with the next row

Replace (Ctrl + H) — Replaces the string specified in the Find What box with the string specified in the Replace With box of the Replace dialog.

In the Search list, select one of the following:

By Columns to search the current column and then continue with the next column By Rows to search the current row and then continue with the next row

Find Next (F3) — Searches for the next occurrence of the string specified in the Find dialog.

Script Start — Opens a text editor that lets you write shell commands that run before the execution of the graph to manage tasks such as moving or creating files.

Script End — Opens a text editor that lets you write shell commands that run after the execution of the graph to manage tasks such as renaming output files.

0%# (#/

The View menu of the Graph Parameters Editor contains the following items:

(18)

display.

Details — Opens the Parameters Details dialog. Use to view or edit all columns regardless of which ones are visible in the Editor.

Reset Columns — Resets the columns displayed by the Editor to the default selection: Name, Value, and Description.

Expand Sub-record — Opens the parameter sub-tree for the selected row. Collapse Sub-record — Closes the parameter sub-tree for the selected row. Expand All — Opens all levels of the parameter tree.

Collapse All — Closes all levels of the parameter tree except the top level.

Show Parameters — Displays the parameters of the graphs, subgraphs, and components in the active graph. To display the parameters of a graph, subgraph, or component:

a. Select Show Parameters.

b. Click the plus sign next to the graph, subgraph, or component.

Alternatively, instead of step b you can select the desired row and then choose View > Expand Sub-Record.

+%-*#1 (#/

Use the Filter menu of the Graph Parameters Editor to select which components or parameters appear in the grid of the Editor, according to the criteria expressed by the following menu choices:

Clear — Removes any filtering that is in effect.

By Parameter Name — Displays only parameters with the specified name.

By Parameter Type — Displays only parameters with the specified type. Select the type from the submenu. By Value — Displays only parameters with the specified value.

Explicit — Removes from view any parameters with values propagated from other components. By Component Name — Displays only components with the specified name.

By Component Type — Displays only components of the specified type. Select the type from the submenu. By Phase — Displays only components in the phase of the graph you specify.

By Layout — Displays only components with the layout you specify.

2#- (#/

The Help menu of the Graph Parameters Editor contains the following items: Editor Help — Opens Help for the Graph Parameters Editor. Help Topics — Provides access to all Ab Initio Help topics.

About Ab Initio — Displays information about the active version of the GDE.

(19)

The Graph Parameters Editor displays a toolbar with navigation arrows and shortcuts for commonly used menu items.

Validate checks the selected parameter, or the parameters of the selected graph, subgraph, or component, to make sure there are no errors in syntax, and that all $ references can be resolved.

Print opens the Print dialog, so you can print the grid of the Parameters Editor.

Cut removes the selection and places it on the clipboard.

Copy copies the selection and places it on the clipboard.

Paste inserts the contents of the clipboard at the insertion point.

Insert Parameter adds a blank parameter above the selected component, subgraph, or parameter.

Delete Parameter removes the selected parameter.

Move Down moves the selected parameter downward through the grid.

Move Up moves the selected parameter upward through the grid.

"

#

To add a parameter in one of the parameters editors: 1. Open a Parameters Editor.

2. In the Editor's grid, click in the row above which you want to insert the new parameter. 3. Do one of the following:

On the Editor's menu bar, choose Edit > Insert Parameter.

On the Editor's toolbar, click the button.

A new row appears with a local parameter icon and noname next to it in the Name column. 4. Replace noname with a name for the parameter.

5. Make entries in the rest of the Editor's columns to define the parameter you want.

)

*

" +

This section contains these topics:

About Transform Functions

The Transform Editor

Rules in Transform Functions

(20)

Local Variables in Transform Functions

Comments in Transform Functions

Specifying Where a Transform Is Stored

Making Recursive Calls in Transforms

The GDE Expression Editor

The DML Functions Palette

The DML Operators Palette

*

" +

A transform function (or transform) is the logic that drives data transformation — most commonly, transform functions express record reformatting logic. In general, however, you can use transform functions in data cleansing, record merging, and record aggregation.

To be more specific, a transform function is a collection of business rules, local variables, and statements. The transform expresses the connections between the rules, variables, and statements, as well as the connections between these elements and the input and output fields.

Transform functions are always associated with transform components; these are components that have a transform parameter: the Aggregate, Denormalize Sorted, Fuse, Join, Match Sorted, MultiReformat, Normalize, Reformat, Rollup, and Scan components.

Each component that has a transform parameter:

Determines the values that are passed to the transform function Interprets the results of the transform function

* *

" #

This section includes these topics:

About Creating Transforms

Displaying the Transform Editor in Grid View

Transform Editor (Grid View)

Displaying the Transform Editor in Text View

Transform Editor (Text View)

Using Text Alternatives to the Transform Editor

*

"

To create a transform, you create rules, variables, and statements and describe the relationships between them. The Transform Editor has two views: Grid and Text. In the Grid View, you create a rule by dragging and dropping. In the Text

(21)

View, you create a transform using DML.

Most graph developers use a combination of these two techniques to create and modify transforms — switching back and forth between Grid and Text Views as needed.

!

*

" #

0 )

The Grid View of the Transform Editor provides a graphical interface for building and editing transforms:

To display the Transform Editor Grid:

1. Double-click a graph component that has a transform parameter. This displays the Properties dialog.

2. On the Parameters tab, double-click the transform parameter (or select transform in the Parameters list and click New or Edit).

The Transform Editor opens and a dialog appears displaying the question:

Do you want the system to generate a default transform?

3. Click Yes or No.

If you click No, you see a blank Transform Editor.

If you click Yes, the system generates a default transform function that you can edit.

4. If the Transform Editor opens in Text View, select View > Grid.

NOTE: You can also display the Transform Editor without going through the Properties dialog: Shift-double-click a component that has a transform parameter.

You can also use the Transform Editor in Text View to create transforms using DML. To do so, select View > Text View

and see Creating Transforms.

Transform Editor (Grid View)

Statements in Transform Functions

(22)

Use the Transform Editor to work with transform functions by creating or modifying business rules, statements, and local variables. You can also set rule priority here.

You can also use the Transform Editor in Text View to create transforms using DML. To do so, select View > Text View

and see Creating Transforms .

Creating Rules in the Transform Editor Grid

Working with Rule Priority in the Transform Editor Grid

Entering Statements in the Transform Editor Grid

Declaring Local Variables in the Transform Editor Grid

Initializing Local Variables in the Transform Editor Grid

Deleting Local Variables in the Transform Editor Grid

Adding Comments in the Transform Editor Grid

Transform Editor (Grid View) Menus

Transform Editor (Grid View) Toolbar

(23)

The Text View of the Transform Editor is similar to other text editors:

To display the Transform Editor in Text View:

1. Double-click a graph component that has a transform parameter. This displays the Properties dialog.

2. On the Parameters tab, double-click the transform parameter (or select transform in the Parameters list and click New or Edit).

The Transform Editor opens and a dialog appears displaying the question:

Do you want the system to generate a default transform?

3. Click Yes or No.

If you click No, you see a blank Transform Editor Grid.

If you click Yes, the system generates a default transform function that you can edit.

4. If the Transform Editor opens in Grid View, choose View > Text View on the menu bar of the Editor.

You can also use the Transform Editor Grid View to create transforms. To do so, select View > Grid View and see

Transform Editor (Grid View).

Statements in Transform Functions

Transform Editor (Text View) Menus

Transform Editor (Text View) Toolbar

*

" #

3*

0 )4

(24)

To view the Transform Editor in Text View, see Displaying the Transform Editor in Text View.

For detailed information on working with transforms using DML, see Transform Syntax, Rule Syntax, and the topics listed in About Statements.

*

'

*

" #

Instead of entering DML in Text View of the Transform Editor, you can do one of these:

You can enter the DML into a standalone text file which you edit using the text editor of your choice. Then, you include the standalone text file in the transform's package.

If the DML for the transform function is fairly short, you can enter the DML code into the Embedded box on the Parameters tab of the component Properties dialog.

1 *

" +

This section includes:

About Rules

Creating Rules

Prioritizing Rules

Working with Null Values In Rules

Generating Default Rules

Working with DML Wildcards in Rules

1

A rule, or business rule, is an instruction in a transform function that directs the construction of one field in an output record.

Creating Rules

Prioritizing Rules

Generating Default Rules

(25)

1

Creating Rules in Grid and Text View

Creating Rules in the Transform Editor Grid

Working with Rules Using DML

1 *

0 )

You create transform functions using one or both of these methods:

You can use drag and drop techniques in Grid View of the Transform Editor.

NOTE: When you create transforms in the Grid View, the transforms are automatically translated into DML code. To view the generated code, select View > Text from the Transform Editor menu bar. For example, the transform

shown above in Grid View looks like the illustration below when viewed in Text View.

You can also use the Transform Editor in Text View to create transforms by entering DML code directly. To do so, select View > Text View and see Creating Transforms .

1 *

" #

To create a rule:

1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Business Rules tab if it is not already displayed.

3. Drag an input field from the Inputs pane into a cell on the Business Rules tab. Arrows indicate the flow of data.

4. To use another input field in the rule, drag another input field into the cell.

5. To alter the data in the fields you have dragged into the cell, do one of the following: Type an expression incorporating the field names.

(26)

Right-click the rule and choose Edit Rule from the shortcut menu to open the Expression Editor, and then

click Help.

6. Drag from the Rule Port to an output field as follows:

NOTE: To quickly copy a field from the input to the output with no changes, drag from the Inputs pane to the Outputs pane. See also Generating Default Rules.

)

1

!(-To create a rule using DML:

1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter statements using the syntax described in Creating Rules.

5

1

About Prioritizing Rules

(27)

5

1

You can set the order of evaluation of rules in a transform function by assigning priority numbers to the rules. The rules are attempted in order of priority, starting with the assignment of lowest-numbered priority and proceeding to assignments of higher-numbered priorities, then finally to an assignment for which no priority has been given.

You set rule priority using one or both of these methods:

Using the Transform Editor Grid as described in Working with Rule Priority in the Transform Editor Grid. Using DML code as described in Working with Rule Priority Using DML.

The best way to view all your rule priorities at once is to view the Transform Editor in Text View.

Working with Rule Priority Using DML

)

1 *

" #

To set the priority of a rule in the Transform Editor Grid:

1. Do one of the following to display the Assignment Priority dialog: Select the rule and choose Edit > Priority.

Right-click a rule and choose Set Priority from the shortcut menu.

2. Click Help in the dialog.

Working with Rule Priority Using DML

)

1

!(-To set the priority of a rule:

1. If it is not already displayed, display the Transform Editor in Text View.

2. Enter the priority as described in About Prioritizing Rules, Creating Transforms, and Prioritized Wildcard Rules.

(28)

)

/

0 % 1

There is no graphical mechanism for working with NULLs in rules. Therefore, the best way to work with NULLs is by using DML code in Text View of the Transform Editor.

For more information on NULLs in rules, see the following topics:

Null Results from Input Expressions

Making a Transform Succeed with Null Input Values

Intentionally Using Nulls in Output Fields

!

1

Generating Default Rules in the Transform Editor Grid

Add Default Rules Dialog

!

1 *

" #

If you have input fields that are not referenced explicitly in other rules, you can add default rules to have a transform assign the values of these input fields to output fields. To generate default rules in the Transform Editor Grid:

1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Business Rules tab if it is not already displayed.

3. Select Edit > Add Default Rules.

4. Click Help.

!

1 !

Use this dialog to generate default rules for input fields that are not referenced explicitly in other rules: the values of these input fields will be assigned to output fields that have the same name as the input field.

To generate one default rule for each unreferenced input field, select Match Names. For example, this is the result if

(29)

The resulting rules in DML code are:

out.custid :: in.custid; out.title :: in.title; ...

To generate one default rule for all unreferenced input fields, select Use Wildcard (.*) Rule. If the input and output

fields are the same as those shown above, this is the result:

The resulting rule in DML code is:

out.* :: in.*;

)

!(-

1

In addition to generating default (wildcard) rules in the Transform Editor Grid, you can also use wildcards when you create rules using DML code. For detailed information, and an explanation of how wildcards in DML-coded rules are expanded and propagated to subfields, see Using Wildcards in Rules.

"

*

" +

Entering Statements in the Transform Editor Grid

Creating Statements Using DML

(30)

The If Statement

The While Statement

The For Statement

The Block Statement

The Switch Statement

The Expression Statement

#

"

*

" #

Transform functions can contain several kinds of statements, although statements are not required. Statements are

comprised of programming logic that affects the flow of control in a transform. Typical statements are the if statement and the while statement.

You can use the Transform Editor Grid to enter statements, although the Grid does not provide drag and drop methods for building statements; you can only type DML statements directly into the grid.

If you enter statements using the Transform Editor Grid, they will end up in the proper place when the transform function is automatically converted to pure DML. Alternatively, you can enter DML statements in Text View of the Transform Editor. See Creating Statements Using DML.

NOTE: If you work with statements in Text View, you must be sure to place the statements after any local variable declarations and before the rules of the transform function.

To create a statement in the current transform function:

1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Statements tab.

3. Enter statements in the lines of the grid using the syntax described in Statements.

"

!(-To create a statement in the current transform function:

2. Enter statements using the syntax described in Rule Syntax, Local Variable Declaration and Initialization Syntax, and the topics referenced in Statements.

NOTE: Be sure to place the statements after any local variable declarations and before the rules of the transform function.

-

0 *

" +

About Local Variables

(31)

Initializing Local Variables

Deleting Local Variables

Editing Local Variables

-

0

You can often simplify or consolidate rules in a transform function by using a local variable. A local variable is a named storage location in an expression or transform function. You declare a local variable within the transform function in which you want to use it. The local variable is reinitialized each time the transform function is called, and it persists for one single evaluation of the transform function.

You declare, initialize, and delete local variables inside transforms. You can do this using drag and drop techniques in the Transform Editor Grid:

To declare local variables, use the Variables Editor. See Declaring Local Variables in the Transform Editor Grid

To initialize local variables, use drag and drop techniques on the Variables tab:

Or, you can enter DML code in the Text View of the Transform Editor:

See also Using Text Alternatives to the Transform Editor.

!

-

0

You can declare local variables using either the Transform Editor Grid or using DML code.

!# - 1%/ -. -0 1% -# %/ 2#1 / +.1( #!%*.1 1%!

To declare a local variable in the Transform Editor Grid:

1. If it is not already displayed, display the Transform Editor Grid.

2. Choose Edit > Local Variable from the menu bar of the Transform Editor to display the Variables Editor. 3. Click Help or see the Variables Editor topic.

(32)

!(-To declare a local variable using DML code:

1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter local variable syntax as described in Creating Local Variables.

%

5 -

0

You can initialize a local variable in the Transform Editor Grid or using DML code.

%/%%-%6%/ -. -0 1% -# %/ 2#1 / +.1( #!%.1 1%!

To initialize a local variable in the Transform Editor Grid:

1. If it is not already displayed, display the Transform Editor Grid.

2. Click the Variables tab. The rightmost pane becomes the Variables pane, displaying the Variables in the active transform function.

3. In a cell in the Variables tab, enter the expression that results in the desired value for the variable.

4. Drag from the Variable tab's port to the name of the variable (in the Outputs pane) to which you want to assign the value.

NOTE: Local variable initialization is executed in order; therefore, later variables may depend on values in earlier variables.

%/%*%-%6%/ -. -0 1% -#

%/

!(-To initialize a local variable using DML code:

2. Enter the appropriate DML code as described in Local Variable Declaration and Initialization Syntax.

!

-

0

You can delete a local variable in the Transform Editor Grid, or using DML code.

!#-#%/ -. -0 1% -# %/ 2#1 / +.1( #!%.1 1%!

To delete a local variable using the Transform Editor Grid:

1. If it is not already displayed, display the Transform Editor Grid. 2. Display the Variables Editor.

(33)

3. Click the gray area at the beginning of the cell containing the variable you want to delete. 4. Press Delete.

5. Delete or edit any rules or statements that reference the deleted variable.

!#-#*%/ -. -0 1% -#

%/

!(-To delete a local variable using DML:

1. If it is not already displayed, display the Transform Editor in Text View. 2. Display the Variables Editor.

3. Select View > Text.

4. Delete the DML code that declares the variable.

5. Delete or edit any rules or statements that references the deleted variable.

#

-

0

About Editing Local Variables

Displaying the Variables Editor

Variables Editor

Setting Attributes of Local Variables

#

-

0

You can edit local variables and set their attributes using the Transform Editor Grid or using DML code. To edit a local variable using the Transform Editor Grid:

1. If it is not already displayed, display the Variables Editor. 2. Select Help > Editor Help from the Variables Editor menu bar.

To edit a local variable using DML code:

1. If it is not already displayed, display the Transform Editor in Text View. 2. Edit the variable as described in Creating Local Variables.

!

0 #

To display the Variables Editor:

1. If it is not already displayed, display the Transform Editor Grid. 2. Do one of the following to display the Variables Editor:

(34)

Right-click a cell on the Variables tab, and choose Edit Local Variable from the pop-up menu.

0 #

Use this dialog to create or edit local variables.

Field Name. Enter or edit the name of the local variable in this column. Field names must be unique within a variable. Data Type. Click the arrow to choose or change a data type. User-defined types are included in this list.

Length. Enter or edit the length or delimiter for the field, as follows:

For fixed-length fields, the length must be an integer representing the total number of bytes the field holds, for example 10.

For variable-length fields, the delimiter must be enclosed in double-quotes, for example "end".

For decimal fields, the length must be an integer representing the total number of bytes in the field, and optionally, a period and an integer representing the number of decimal point places the field stores. For more information on describing decimals, see Decimals.

For dates, open the Attribute pane (as described in Setting Attributes of Local Variables), and set the Format attribute to a format string. For more information on date formats, see Details.

The Status window across the bottom of the Editor displays messages about incorrect or incomplete entries. A yellow square at the beginning of a row indicates an incorrect or incomplete entry in that row. Placing the cursor in a cell in that row displays a message in the Status window if the entry in the cell is incorrect or incomplete.

Click the button to expand the status window into a text editor for reading long messages.

-

0

Use the Attributes pane of the Variables Editor to set the attributes of local variables. To display the Attributes pane:

(35)

2. To display the Variables Editor, choose Edit > Local Variable, or right-click a Variables tab cell and choose Edit Local Variable from the pop-up.

3. In the Variables Editor, select View > Attributes from the menu bar.

Use the following columns to set the selected variable's attributes:

Attribute displays the name of the attribute Value provides a place to enter a value.

Edit displays the icon for certain attributes. Click this icon to open an appropriate editor for working with the attribute.

""

*

" +

You can add comments to a transform either in the Transform Editor Grid, or using DML in the Transform Editor Text View.

""

*

" #

You can add comments at the beginning of a transform, or at the end of a particular field. To add comments at the beginning of the current transform function:

1. Put the cursor in any field of the center pane of the Transform Editor. 2. Make sure the Documentation tab is selected at the bottom of the center pane. 3. Type your comment in the Documentation box.

To add comments at the end of a particular field of the transform function:

1. Place the cursor in the field of the center pane where you want to add a comment. 2. Make sure the Details tab is selected at the bottom of the center pane.

3. Type your comment in the Details box.

""

!(-You can add comments at the beginning of a transform, or at the end of a particular field. Add comments either between /* and */ patterns or between double slashes and the end of the line:

/* comment in the style of C and Java */ // comment in the style of C++ and Java

*

" %

You can specify where a transform is stored. You do this on the Parameters tab of the component Properties dialog; you cannot do this using DML code.

There are four options:

Embedded Specifies that the transform is embedded as part of the component. Using this option reduces the number of small files you need to save outside the graph. However, an embedded transform is not as easy to re-use or share as a transform saved as a file.

(36)

(

1 '

*

"

You can make recursive calls in transform functions only by using DML code. You can do this using the Transform Editor in Text View, or — if the transform is in a file — you can use any text editor to edit the file. For detailed information, see

Making Recursive Calls.

* +

The Fields pane displays the input record formats available for use in an expression.

To find the type of a field, right-click the field name. The type appears at the bottom of the shortcut menu.

* +

The Functions pane displays the built-in DML functions.

To get information about a function, right-click the function name and choose Definition from the shortcut menu.

* .

The Operators pane displays the built-in DML operators.

To get information about an operator, right-click the name of the operator and choose Definition from the pop-up menu.

*

" +

!(-To work with transforms using DML, see these topics:

Creating Transforms About Transforms

Transform Syntax

Transform Syntax Details

Creating Rules About Rules

Host Specifies that the transform be saved as a file on the computer running the Co>Operating System. Local Specifies that the transform be saved as a file on the computer running the GDE.

EME

(37)

Prioritized Rules

Rule Syntax

Rule Syntax Details

Evaluation Results

Examples of Rules Without Priorities

Examples of Rules with Priorities

Using Wildcards in Rules

Creating Local Variables About Local Variables

Local Variable Declaration and Initialization Syntax

Setting Whether a Local Variable is Nullable

Statements

About Statements

The Assignment Statement

The If Statement

The While Statement

The For Statement

The Block Statement

The Switch Statement

Switch Statement Examples

The Expression Statement

Making Recursive Calls

Adding Comments to Transforms

Storing Transforms

)

About Packages

Viewing Packages

Including Packages

(38)

What Is a Package?

A Note About Include Files

The Global Package

%

Packages are an Ab Initio feature for use by GDE graph developers who need to collaborate and share transform functions, type definitions, and variables.

A package is a named collection of related DML objects. A package can hold types, transform functions, and variables, as well as other packages. Packages provide a means of locating in one place DML objects that are needed more than once in a given graph, or needed by multiple developers. Packages allow developers to avoid redundant code; this makes

maintenance of DML objects more efficient. Packages are very useful in these types of situations:

The record formats of multiple ports use common record formats and/or type specifiers Multiple components use common transforms

7 #

1#

. %#! %2 .( ./#/* /! .1*

Because packages are always associated with components and ports, you can access packages only via the Properties dialog of a graph component. To display a packages in a tree view, you use the GDE's graphical editors (Package Editor,

Transform Editor, and Record Format Editor).

Packages can be stored in standalone files or can be embedded in a component. Package files have .xfr or .dml file extensions.

7 #

/! ( -%8 # .( ./#/*

The multi-stage transform components require packages because, unlike other transform components, they are driven by more than single transform functions. These components each take a package as a parameter and, in order to process data, look for particular variables, functions, and types in that package. For example, a multi-stage component might look for a type named temporary_type, a transform function named finalize, or a variable named count_items.

/

%

+

The term include file is sometimes used to refer to a standalone file of DML code, even if that code does not include an actual "collection" of related DML objects, and may, in fact, include as little as one DML object. However, there is really no need to think of include files as technically different from package files. Therefore, for the sake of clarity, we use the term package file to refer to any DML in a file that is incorporated into a graph via an include statement. We use the term package to refer either a package file, or to an package embedded in a component.