Integrator Command-Line Client
In addition to the web-based Integrator component of Jedox Web, Integrator projects can also be run from the command line by using the CLI tool EtlClient. This tool is located in the directory …\tomcat\client.
To open it, use the command etlclient.bat (Windows) or etlclient.sh (Linux).
Entering -h will call up a list of the valid parameters, which are described in Command-Line-Client-Parameters.
The following tasks CLI tasks are described in this article:
Setting up authentication Creating server profiles Executing Integrator projects Generating project documentation
Authentication
Authentication with a Jedox In-Memory DB Server user is required for communication with the Jedox Integrator Server. Valid login credentials must be provided for each call of the CLI; the user can be set with option -u and password with option -pw. The IP address of Jedox Integrator is set with option -s (default value for the server is
“http://localhost:7775”).
For example, to start the Jedox Integrator project example sampleBiker on a locally installed Jedox Integrator, enter the following at the
command-line level:
To add the project to the server: etlclient
–a \samples\sampleRelational -u admin -pw admin To run the standard job: etlclient -p sampleRelational -u admin -pw admin
Server Profiles
Instead of providing login credentials with each CLI call, you can user server profiles. Server profiles are created with option -spc. To use all options for a defined profile, you can use option -sp.
Examples:
etlclient.bat -spc MyProfile -s localhost -u admin - pw admin
etlclient.bat -sp MyProfile -a samples\sampleBiker etlclient.bat -sp MyProfile -p sampleBiker
All profiles are stored in the local file .\client\config\profiles.xml. If no profile is given, the profile with name “default” is used. In this case, you can omit the option -sp.
Note: due to Windows restrictions, non-ASCII characters cannot be used reliably in command-line apps.
Example:
etlclient.bat -spc default -s localhost -u admin -pw admin
etlclient.bat -p sampleBiker
Note: the Integrator component of Jedox Web and the command-line client work with the same Jedox Integrator. This means that jobs that were created with the Jedox Integrator web client can also be run from the command-line client.
Important: etlclient.batrequires OpenJDK 11.
Executing a Jedox Integrator Project
To execute an Jedox Integrator project, the Jedox Integrator project definition must be entered using option -p. A load and job defined in the Jedox Integrator project can then be executed. Two loads or jobs can be executed simultaneously, but only if a new job containing the desired jobs or loads has been created. If no job is determined, the job named “default” is executed.
Examples:
etlclient –p sampleBiker
etlclient -p sampleBiker -j Masterdata etlclient -p sampleBiker -l Years
Note: in all examples, it is assumed that login credentials are provided in the default server profile.
Test run
Before executing loads or jobs, it is advisable to display on the screen the data sources taken as a basis. Any data source, regardless of whether it is an extract or a transformation, can be shown. For tree- based data sources, the display is shown in the “full hierarchy” format, without attributes and levels. The number of display lines can be determined.
Examples:
etlclient -p sampleBiker -d Products
etlclient –p sampleBiker -d Products -n 20
The test data can also be written to a file with the “>” shell option.
Example:
etlclient -p samples\sampleBiker -d OrdersData_Trans>
testfile.txt
Variables
The job execution can be defined more accurately using context variables. Even if a job has already been assigned variables, the variables shown at the command line level are used. Spaces must be put in quotation marks (” “) in order to be used as variable values.
Notation:
-c <Variable1>=<Value1> <Variable2>=Value2> …. or -c <Variable1>=”<Value1>” <Variable2>=”Value2>” ….
Examples:
etlclient -p sampleVariables -j SalesMonth -c Year=2006 Month=Dec
etlclient -p sampleVariables -j SalesMonth -c Year=”2006” Month=”Dec”
Log
If a job or load is executed from the command line level, the execution protocol is displayed directly on the command line. It contains
information, warnings, and error messages that appear during the Jedox Integrator process. The level of detail during logging can be entered with the log level on the command line using option –ll.
Possible values are OFF, FATAL, ERROR, WARN, INFO, DEBUG, and ALL.
The following final messages exist for an Jedox Integrator process:
“Completed successfully”
“Completed with Warnings”
“Completed with Errors”
“Failed”
The log display can be written to a file with the “>” shell option.
Example:
etlclient -p samples\sampleBiker –j Initdata >
InitdataLog.txt
Special characters:
For Windows, the following special characters must be double quoted if they are used as values for any command line option: & \ < > ^ |
<space>. For some characters, an additional prefix is required: ! : ^!
and ” : “”
Example:
etlclient -p "My Project &" -j "Go^!"
Exit codes
Executing commands with the Jedox Integrator command line tool will return an “Exit code” that can be used later within the batch script:
10 Execution ended with Status “Successful”
20 Execution ended with Status “Warnings”
30 Execution ended with Status “Errors”
40 Execution ended with Status “Failed”
50 Execution ended with Status “Stopped”
60 Execution ended with Status “Aborted”
0 No error occurred (other commands than Execution)
-1 Jedox Integrator was not reachable
-2 Error on Client or Server side (e.g. Project/Job not found, Project not valid)
Note: executions start with Jobs (-j) or Loads (-l), Data preview (-d), and Test (-t).
Example:
call etlclient.bat -s localhost: 7775 -p sampleBiker -j default
rem print the status code of the execution (in this case it will be 10 which means successful)
echo Exit Code is %ERRORLEVEL%
Execution states
The following states are possible after an Integrator execution.
Numeric Code String Code Description
0 Queued The Execution is present in the {@link IExecutor Executor} but
not yet running.
5 Running The Execution is currently being executed and still running.
6 Paused The Execution is currently being executed and still running.
10 Completed successfully The Execution finished without encountering any errors or
warnings. Everything is perfect. This is how things should be!
20 Completed with Warnings
The Execution is finished and has encountered one or more warnings indicating that there are some problem within the data causing some data not to be processed successfully. No structural problems (errors) occurred.
30 Completed with Errors
The Execution is finished and has encountered one or more errors causing parts of it to fail. Nevertheless the execution finished trying to execute all of its parts.
Note: This status can only occur if failOnError (set in config.xml) is false.
40 Failed
The Execution has encountered an unrecoverable error and thus was terminated premature for internal fault reasons. Some parts of it may not have been executed.
Note: If failOnError (set in config.xml) is true (default), this is the status returned, when any error occurs within the execution.
49 Stopping The Execution was tried to be terminated but it is still running.
50 Stopped The Execution was terminated premature by user intervention.
Some parts where executed, some parts where not.
54 Interrupting The Execution was interrupted by user intervention
55 Interrupted The Execution was interrupted by user intervention
60 Aborted
The Execution was canceled by {@link IExecutor#stop(Long) user intervention} before it reached status running. No action was done by it. An aborted execution can NOT be reactivated and started again.
70 Invalid
The Execution is not valid and thus will not be executed. This indicates that the configuration of at least one of the involved components is faulty, so that this component cannot be created and initialized.
Generating project documentation
Notation
etlclient -pd <projectname> -j <jobname1> <jobname2> …. -o
<directory>
Example:
Create project documentation for project sampleBiker 1.
without flow graphs in directory C:\EtlDoku1:
etlclient -pd sampleBiker -o C:\EtlDoku1
Create project documentation for project sampleBiker with 2.
flow graphs of job Masterdata and Cubedata:
etlclient -pd sampleBiker -j Masterdata Cubedata -o C:\EtlDoku2
Create documentation for job Cubedata in project 3.
sampleBiker. It only contains the referenced components of this job and the flow graph of job Cubedata:
etlclient -pd sampleBiker.jobs.Cubedata -o C:\EtlDoku3
The documentation can be displayed by opening file
etldocumentation.html with a browser. Note that Internet Explorer 9 doesn’t display the graphical flow graph due to insufficient support for the SVG format.
The following files are generated in the output directory:
etldoc.xml
Contains all project- and component-related information, SVG flow graphs, references, and all component fields (references etldoc.xslt).
etldoc.xslt
The XSTL transformation of etldoc.xml. By adapting this file, the layout and content of the documentation can be modified (references etldoc.css).
etldoc.css A style sheet (references etldoc_bg.png).
etldoc_bg.png A background image.
etldocumentation.html
The documentation file to be displayed in the browser.
Technically, it is etldoc.xslt transformation applied to etldoc.xml (requires only files etldoc.css and
etldoc_bg.png).
Progress logs
It is possible to generate additional info messages indicating the progress of a job or a data preview. This is helpful for the analysis of long-running Jedox Integrator jobs with high data volume.
For one or several components of the Jedox Integrator job (extract, transform, or load), a log block size can be indicated as a command line parameter. Each time the component has processed a block, an info message will appear.
Notation: -c #<componentName>.logBlockSize=<n>
Example:
-p samples\sampleBiker -j Cubedata-c
#sampleBiker.extracts.Orderlines.logBlockSize=500
#sampleBiker.loads.OrdersCube.logBlockSize=2000
This will lead to these additional log messages:
INFO : 1 rows processed from extract Orderlines INFO : 500 rows processed from extract Orderlines INFO : 1000 rows processed from extract Orderlines INFO : 2000 rows processed from load OrdersCube
