AlgoTraderDocumentation
Overall Documentation of the Framework algotrader as well as instructions on how to build a strategy with it
Featured
Updated Sep 3, 2012 by andyflury
Introduction Architecture o Operating Systems o Services / Processes Platform requirements Eclipse Projects
UML Mode and Code Generation
o UML Model o Code Generation Java Development o Java Packages o Java Libraries o Coding Standards Esper Engine
o Trading Framework Modules
o Strategy Modules
Database & Transaction Handling
o Entity Relationship Model
o DB Tables
o Transaction Handling
Installation
o Subversion Checkout
o Generate the code
o Compile
o Install
o Start a Demo Strategy
Strategy Development o MovServiceImpl .java o module-mov-main.epl o conf-mov.properties o esper-mov.cfg.xml o DB table strategy o DB table watch_list_item Operation Modes
o Back Testing / Simulation Mode
o Live Trading Mode
Broker Interfaces
o Order / Execution
o Market Data
Miscellaneous Topics
o Tick Data Format
o Configuration
o Strategy vs. Strategy Family
o ServiceLocators , BeanFactory & ApplicationContext
o Security
o Remoting
o AspectJ
o Logging
Introduction
The Trading Framework AlgoTrader facilitates a minimum effort development of trading strategies based on Esper component for complex event processing (CEP).
The system is developed by a Model Driven Architecture. All artifacts of the system are modeled in UML and generated into code by the Framework AndroMDA.
For Database Persistence the Framework Hibernate is used. Currently the system uses MySql as a Database. Through the use of Hibernate, other database providers can used by changing configuration.
The system is based on the Spring Framework, which enables a clean Service Oriented Architecture.
The system does not provide a GUI Frontend. Instead it uses Java Management Extension (JMX) for management at runtime. Currently, the system provides the following main features:
Automate Trading Strategies based on Complex Trading Rules
Simulate and run several strategies in parallel
Backtest strategies based on historical Data
Portfolio tracking & Performance Measurement
Provide different Numerical and Statistical Libraries
To work with AlgoTrader it is recommended, that you have a basic level of knowledge on following topics:
Java Standard Edition
Eclipse IDE
XML
Spring & Hibernate
MySql
Architecture
The Trading Framework layer is the base for all strategies running on top of it. It is responsible for persistence to the database. It holds the connection to the Broker Interfaces to retrieve live market data as well as to place orders. The Trading Framework is also responsible for Risk Management, Portfolio Management & Performance Measurment. For many of its tasks the Trading Framework uses a contained Esper Engine Instance.
On top of the Trading Framework any number of Strategies can be installed. In Live Trading mode each strategy runs in its own process. Each Strategy contains its own Esper Engine Instance. A Strategy can use any type and number of Technical Indicators and custom trading rules to evaluate trend and to retrieve market signals. Orders are delegated to the underlying Trading Framework for execution.
The entire system is developped in Java. Therefor the system is portable to any environment. The System has been developed on a Windows 7 machine and has been in production on FreeBSD as well as Linux Servers.
Services / Processes
The following Services / Process are used by the system:
Service / Process Description
Strategies In Live Trading Mode each strategy runs in its on JVM (Java process). In simulation mode, the strategies run within the same process as the trading framework
Trading
Framework This is the main process that should be kept running all the times in order to record market data without gaps MySql Database process
Because the framework and the strategies are running within separate processes, individual strategies can be stopped / altered / restarted independent of each other and the framework.
Platform requirements
For development of new Strategies the following Platform requirements exist:
Software URL Tested
Version
Java 1.6 http://www.oracle.com/technetwork/java/javase/downloads/index.html 1.6.0_22
MagicDraw UML http://nomagic.be/files95 (use MD_UML_95_sp1_CE... for your plattform) 9.5 SP1 Subversive SVN Client http://community.polarion.com/projects/subversive/download/eclipse/2.0/indigo-site/
M2E http://download.eclipse.org/releases/indigo Maven SCM Handler for
Subversive
http://repository.tesla.io:8081/nexus/content/sites/m2e.extras/m2eclipse-subversive/0.13.0/N/0.13.0.201207010112/
MySql Database http://www.mysql.com 5.1
A database management tool
(i.e. TOAD for MySql) http://www.quest.com/toad-for-mysql
Hibernate Tools (optional) http://download.jboss.org/jbosstools/updates/development/indigo Veloeclipse (optional) http://veloeclipse.googlecode.com
Eclipse CheckStyle (optional) http://eclipse-cs.sourceforge.net/
Eclipse Projects
The Framework AlgoTrader consists of the following Eclipse Projects:
algotraderlight (the main project)
o algotraderlight-code (all java code)
o algotraderlight-mda (UML Model and code generator)
algotraderlight-crud (DB management application)
algotrader-strat (parent project for all strategies)
algotraderlight has the following structure:
Directory Description
code/src/main manually created source files code/src/test JUnit test
code/target generated source files
code/results files which are created (i.e. copy of retrieved HTML code) or imported (i.e. Tick Data) into the system code/bin Launch configuration and shell start scripts
code/lib jar files not available through maven code/log log files
code/sql SQL scripts
mda/conf Contains configuration files for the generator
mda/src Contains UML Model and AndroMDA configuration / customization files The strategy projects algotrader-xxx have the following structure:
Directory Description
src manually created source files
lib jar files not available through maven bin Launch configuration and shell start scripts log log files
results Contains mainly Market Data Files
UML Mode and Code Generation
The entire system is developed by means of Model Driven Architecture (MDA) using the framework AndroMDA (www.andromda.org). To get familiar with the Code Generator AndroMDA the following links and tutorials give a good overview:
A Bird's Eye view of AndroMDA
Getting started with AndroMDA or Prepare the Development Environment
Getting Started Java - long winded 'TimeTrackingService' example
Spring Cartridge example - short and focused on buisness tier of 3-tier application architecture
AndroMDA's forum where you can get FREE support
UML Model
The UML Model in use by the system is shown here: UML Model
Code Generation
Much of the system is modelled in UML. It is recommended to use MagicDraw for modelling (see Platform_requirements) The actual UML Model is defined in the following xmi file:
The model is transferred into code my means of AndroMDA which uses Maven artefacts and Velocity templates for code generation.
Generated code is placed under the directory /target, whereas implementation classes are generated into the directory /src with only the method signatures.
Configuration of the code generator is placed into the /mda directory. It is also this directory that contains template modifications as well as mappings from UML artifacts to Java / Database objects.
Notes:
When opening the file algotrader.xmi for the first time, the location of the Maven 2 Repository has to be defined. This is usualy in user-home-directory/.m2/repository.
In addition the path to the UML-Standard-Profile has to be defined. This file lies in the profiles sub directory of the MagicDraw Installation directory (e.g., ..\MagicDrawUml95CE\profiles\UML_Standard_Profile.xml)
Java Development
Java Packages
The following table gives an overview of the provided java packages:
Package Description
com.algoTrader ServiceLocators
com.algoTrader.entity Entities and corresponding Hibernate DataAccessObjects com.algoTrader.service Contains all Spring Services
com.algoTrader.starter Startup-Classes as well as a generic ServiceInvoker com.algoTrader.listener Esper Statement Listeners
com.algoTrader.subscriber Esper Statement Subscribers com.algoTrader.util General Utility methods
com.algoTrader.util.io CSV Adapters for market data as well as market data conversion functions
Java Libraries
The following table shows an overview of used java libraries:
Java Library Description
Spring Framework http://www.springsource.org spring-aop-3.0.5.RELEASE.jar spring-asm-3.0.5.RELEASE.jar spring-beans-3.0.5.RELEASE.jar spring-context-3.0.5.RELEASE.jar spring-context-support-3.0.5.RELEASE.jar spring-core-3.0.5.RELEASE.jar spring-expression-3.0.5.RELEASE.jar spring-jdbc-3.0.5.RELEASE.jar spring-orm-3.0.5.RELEASE.jar spring-tx-3.0.5.RELEASE.jar springtemplate-3.2.jar Hibernate http://www.hibernate.org aopalliance-1.0.jar antlr-2.7.6.jar cglib-2.2.jar
dom4j-1.6.1.jar ehcache-2.3.3.jar ehcache-core-2.3.3.jar ehcache-terracotta-2.3.3.jar hibernate-core-3.5.6.jar hibernate-jmx-3.5.6.jar jta-1.1.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar Esper http://esper.codehaus.org esper-4.2.0.jar esper-csv-4.2.0.jar esper-socket-4.2.0.jar cglib-nodep-2.2.jar antlr-runtime-3.2.jar XML http://xml.apache.org serializer-2.7.1.jar xalan-2.7.1.jar xml-apis-1.3.04.jar Mail / SMS activation-1.1.jar mail-1.4.3.jar AspectJ http://www.eclipse.org/aspectj aspectjrt-1.5.4.jar aspectjweaver-1.5.4.jar Apache Commons http://commons.apache.org collections-generic-4.01.jar commons-beanutils-1.8.3.jar commons-beanutils-core-1.7.0.jar commons-cli-1.0.jar commons-collections-3.2.1.jar commons-configuration-1.6.jar
commons-digester-1.8.jar commons-lang-2.6.jar commons-logging-1.1.1.jar commons-math-2.2.jar google-collections-0.9.jar Miscellaneous jtidy-r8-21122004.jar (HTML to XHTML)
mysql-connector-java-5.1.15-bin.jar (MySql Client) ibclient-9.6.3.jar (InteractiveBroker API client) log4j-1.2.16.jar (Log4J)
agent.jar (JMX Agent) ant-1.7.0.jar (Apache Ant)
ant-launcher-1.7.0.jar (Apache Ant) checkstyle-5.0.jar (Checkstyle) checkstyle-optional-4.4.jar (Checkstyle) javassist-3.12.1.GA.jar (Javassist) junit-4.8.2.jar (JUnit) ta-lib-0.4.0.jar (TA-Lib) SuperCSV-1.52.jar (SuperCVS)
Coding Standards
The project uses checkstyle to verify java coding standards. Checkstyle is enabled by default in the Maven built.
In addition the Plugin Eclipse-CS can be used to enable Checkstyle inside Eclipse. To use Eclipse-CS a new Global Check Configuration has to be added in Eclipse / Window / Preferences / Checkstyle pointing to:
In addition to using Checkstyle, Eclipse has two handy features to automatically format Java Code
Clean Up: This feature is responsible to do a general clean up when saving a modified File (i.e. remove unused imports)
Formatter: This feature will automatically format source code according to definable standards (can be invoked through right click / source / format)
A default set of policies targeted to automatically conform with defined checkstyle rules is available. These are available in the source code of the project.
These configurations have to be imported them into Eclipse:
Eclipse / Preferences / Java / Code Style / Clean Up: AlgoTraderLight-code/AlgoTrader-CleanUp.xml
Esper Engine
The system is built on CEP (Complex Event Processing) using the open source framework Esper.
Individual Esper service engines are used inside the trading framework as well as within the strategies.
Each of these engines can contain several modules. Modules specified with in the column MODULES of the Table strategy, are loaded automatically on startup.
Each strategy also contains its own Esper configuration file named esper-xxx.cfg.xml. This configuration file defines settings like:
Event Types
Auto Import Classes & Packages
Custom Aggregation Functions
Variables
General Engine Settings
Esper Listeners and Subscribers are configured inside the modules using the following syntax:
@Name('REBALANCE_PORTFOLIO')
@Tag(name='listeners', value='com.algoTrader.listener.RebalancePortfolioListener') select * from …
Listeners or Subscribers which correspond to a particular Service are commonly implemented as an inner class of the Service.
Trading Framework Modules
Trigger persistence of MarketData
Evaluation and Logging of the Portfolio Values at certain intervals
Position handling (i.e. close position when exit level is reached, etc.)
Evaluation of Performance Parameters (i.e. Sharpe Ratio, Calmar Ratio, Volatility, Draw-Down, etc.)
Margining
Process Cash Transactions (Subscriptions / Redemptions)
Rebalancing between the strategies to keep strategy allocation at a constant level
Strategy Modules
Strategies are completely free in the definition of their Esper Statements. Examples of Statements used by strategies are:
Creation of technical indicators (i.e. Moving Average, Stochastic, MACD, etc.)
Creation of trade signals
Trend evaluation
Open / Close / Increase / Reduce Positions
Set Exit Values
Roll Positions (i.e. for Options and Futures)
Pattern recognition
Orderbook analysis
Database & Transaction Handling
The database MySql is used by the system, even though by using Hibernate, any supported database can be used. The directory algotraderlight-code/sql contains a script algotrader.sql to create the database.
DB Tables
The following table gives a description of the most important tables:
Table Description
strategy Each record represents a strategy within the system security, stock, equity_index,
stock_option, future, intrest_rate & forex)
Security is the base table for all securities in the system. The related tables carry specific information of individual asset clases
tick, bar, bid, ask & trade All relevant market data is stored in these tables
transaction Each Fill is stored as a transaction in the database. In addition the table transaction also carries transactions like intrest, debit, credit & fees
position For each opening transaction a position is created which also carries exit values and maintenance margin
security_family
Common information about an entire family of securities (i.e. all general information about options on S&P500 are stored using this class). The table provides fields like market, currency, market opening hour & market closing hour
Transaction Handling
Using AndroMDA and Spring Transaction Boundaries can be declared inside the UML model using the tagged value
@andromda.spring.transaction.type. Transaction Boundaries will be generated into the file applicationContext.xml and handled by the
org.springframework.transaction.interceptor.TransactionInterceptor.
Subversion Checkout
It is recommended to use the Plugin M2Eclipse for the initial checkout (Import / Maven / Check out Maven Projects from SCM). Note: AndroMDA does not allow any special characters (i.e. spaces or exclamation marks) in the project path.
Select https://algo-trader.googlecode.com/svn/trunk/AlgoTraderLight. This will create the following 3 projects in Eclipse:
algotraderlight
algotraderlight-code (name of the subfolder under algotraderlight: code)
algotraderlight-crud (name of the subfolder under algotraderlight: crud)
algotraderlight-mda (name of the subfolder under algotraderlight: mda)
Run the import again and select https://algo-trader.googlecode.com/svn/trunk/AlgoTraderLightStrat. This will create the following additional projects in Eclipse:
algotrader-mov
algotrader-max
algotrader-periodic
Generate the code
Right click on the project algotraderlight-mda / Run As / Maven install. This will generate all necessary code into algotrader-code target directory.
Compile
Before this is done, please check the Java Build Path of algotraderlight-code project. There are two sources configured. The source folder
target/src/main/java might have an exclusion defined (Star-Star). If there is such exclusion, please remove it (the exclusion not the source folder!)
Build Errors
Building the MOV Example for the first time may result in errors due the 'Optional' condition put on ibclient-9.6.3.jar, quickfixj-core-1.5.2.jar, quickfixj-msg-fix42-1.5.2.jar and ta-lib-0.4.0.jar in algotraderlight-code project. This can be easily corrected by opening the algotraderlight-code pom.xml file, navigate to the 'Dependencies' tab, right click on each of the jar files mentioned and uncheck the Optional box.
Install MySql
Install latest MySql Community Server
run the create-script: sql/algotrader.sql
configure the properties dataSource.user & dataSource.password inside ../algotraderlight/code/src/main/java/conf-base.properties: The default settings are dataSource.user=root and dataSource.password=password. Your DBA will probably frown on using root as the user so here is a simple set of sql commands to create a new user account, password, and privileges limited to the algotrader schema (this can be added to the bottom of the algotrader.sql file if you want to run everything from one file):
CREATE USER 'algouser'@'localhost' IDENTIFIED BY 'algopwd';
GRANT ALL PRIVILEGES ON AlgoTrader.* TO 'algouser'@'localhost';
Start a Demo Strategy
Before running a strategy check the database table strategy. Only the records "BASE" and the record corresponding to strategy should be set to AUTO_ACTIVATE
Mov Example Strategy
This sample strategy is based on a moving average indicator. It's purpose is to demonstrate AlgoTrader in Back Testing / Simulation mode. To start it, use the provided eclipse lunch configuration: /algotraderlight-mov/bin/SimulationStarter-simulateWCP-Mov.launch
Periodic Example Strategy
The purpose of this strategy is to demonstrate AlgoTrader in Live Trading mode. It currently contains the following examples:
periodic buy/sell trades every 10 seconds
order cancellation if order is not executed within 2 seconds
order modification every 2 seconds until order is fully executed
example Execution Algo SteppingLimitOrder
To enable / disable any of these examples uncomment the relevant section in /algotraderlight-periodic/src/main/java/module-periodic-main.epl
To run any of these examples InteractiveBrokers TWS has to be running with the following configuration under API/Settings:
Enable ActiveX and Socket Clients
Socket Port: 4001
In live trading mode the trading framework and the strategy run in separate JVM's and have to be started separately.
To start the trading framwork, use the provided eclipse lunch configuration: /algotraderlight-code/bin/MarketDataStarter.launch To start the periodic example strategy, use the provided eclipse lunch configuration :
Strategy Development
The following paragraph will give a short example based on a simple moving average strategy (with the Short Name MOV).
After backtesting, newly developed strategies can be tested in a live environment using Paper Trading Accounts. At the end of a thorough test procedure, the new strategy can be put into production. The following diagram shows the general procedure for developing new strategies:
Artifact Description
/src Source-File Directory containing necessary Java-Classes for the Strategy
/lib Jar Files not available through maven
/src/main/java/module-mov.epl Esper Module containing the necessary Esper Statements
/src/main/java/conf-mov.properties Contains parameters used by the strategy (i.e. Moving average durations etc.) /src/main/java/esper-mov.cfg.xml Contains event-types, imports, variables and general Esper settings
/src/main/java/applicationContext-client.xml Application Context File for the strategy, mainly containing autowire instructions
/log log files
/results/bardata/xxx Directory containing Bar Data Files needed for Back Testing /results/tickdata/xxx Directory containing Tick Data Files needed for Back Testing DB table strategy A record for the strategy
MovServiceImpl.java
This is the main Java-class containing Subscriber inner classes and Business Logic First we need references to services provided by the base framework:
private PositionService positionService; private LookupService lookupService;
private TransactionService transactionService;
public MovServiceImpl(PositionService positionService, LookupService lookupService, TransactionService transactionService) {
this.positionService = positionService; this.lookupService = lookupService;
this.transactionService = transactionService; }
These values will be auto injected on startup by the Springframework (based on applicationContext-mov.xml)
Next we need an Esper Subscriber to react onto market signals. This can be coded as a separate class or as an inner class within MovServiceImpl.java
public static class OpenPositionSubscriber {
public void update(String strategyName, int securityId, BigDecimal currentValue) {
MovServiceImpl movService = ((MovServiceImpl)
ServiceLocator.commonInstance().getService("movService")); movService.openPosition(strategyName, securityId, currentValue);
} }
The ServiceLocator will give as a reference to the “movService” (Spring Bean) Finally we need a business method to instruct the base framework to open a position:
public void openPosition(String strategyName, int securityId, BigDecimal currentValue) { Strategy strategy = this.lookupService.getStrategyByNameFetched(strategyName);
Security security = this.lookupService.getSecurity(securityId);
int qty = (int) (strategy.getAvailableFundsDouble() / currentValue.doubleValue());
Order order = new MarketOrderImpl(); order.setSecurity(security); order.setStrategy(strategy); order.setQuantity(qty); order.setSide(Side.BUY); this.orderService.sendOrder(order);
// if a position was open (or already existed) set margin and exitValue
Position position = this.lookupService.getPositionBySecurityAndStrategy(securityId, strategyName); if (position != null && position.isOpen()) {
this.positionService.setMargin(position.getId());
double exitValue = initialStopLoss * currentValue.doubleValue(); this.positionService.setExitValue(position.getId(), exitValue, true); }
}
The method contains the following steps:
1. We get a reference to the Strategy Entity by the LookpService.
2. Then we evaluate the quantity that we want to purchase (in this case we use all available funds).
3. Then we construct an Order Value Object (simple POJO object) containing the name of the strategy, the id of the security, the quantity and the transaction type.
4. The TransactionService will then carry out the transaction for us.
module-mov-main.epl
Next thing we have to develop is the Esper module containing our statements. The module basically consists of the following three statements:
// Moving Average using Ta-Lib @Name('MOVING_AVERAGE')
select talib("movingAverage", currentValueDouble, movLengthFast, "Sma")
- talib("movingAverage", currentValueDouble, movLengthSlow, "Sma") as value from Tick(security.isin = underlayingIsin);
This statement creates a Moving Average Indicator using the talib library
@Name('OPEN_POSITION') @Subscriber(className='com.algoTrader.service.mov.MovServiceImpl$OpenPositionSubscriber') select engineStrategy.name as strategyName, indexTick.security.id as underlayingid, indexTick.currentValue as underlayingSpot
from pattern [every (indexTick=Tick(security.isin=underlayingIsin) -> indicator=Indicator)] where indicator.value > 0
and prior(1, indicator.value) <= 0;
This statement calls the OpenPositionSubscriber whenever we have a crossover on the indicators. The Tag 'subscriber' is used to instruct the RuleService to attach the 'OpenPositionSubscriber' to this statement.
conf-mov.properties
Now we need the configuration file, which contains the necessary parameters for the strategy:
underlayingIsin = US38259P5089 movLengthFast = 1000
movLengthSlow = 2000 initialStopLoss = 0.99 stopLoss = 0.97
In our case we use Tick(security.isin = underlayingIsin) and the fast/slow lengths to create the two moving average lines.
esper-mov.cfg.xml
The esper configuration file consists of the following main parts: ... <auto-import import-name="com.algoTrader.util.LookupUtil"/> <auto-import import-name="com.algoTrader.util.RoundUtil"/> <auto-import import-name="com.algoTrader.service.mov.MovUtil"/> ... <plugin-aggregation-function name="talib" function-class="com.algoTrader.esper.aggregation.GenericTALibFunction"/> ...
<variable name="engineStrategy" type="com.algoTrader.entity.Strategy"/> <variable name="underlayingIsin" type="String"/>
<variable name="movLengthFast" type="int"/> <variable name="movLengthSlow" type="int"/> ...
First the two util.LookupUtil class and service.mov.MovUtil are configured along with the supporting length variables. Note: the actual values for the variables are taken from conf-mov.properties.
The variable engineStrategy gets populated automatically by the RuleService and contains a reference to the Strategy-Entity we are using.
DB table strategy
For this new Strategy we also need an entry in the DB table strategy with the following values:
NAME FAMILY AUTO_ACTIVATE ALLOCATION MODULES
MOV MOV true 1.00 mov-main
AUTO_ACTIVATE means, that the strategy will be automatically run in simulation mode. By using this field, you can run several strategies in parallel to see how they interact.
ALLOCATION is the percentage of total assets, that will be allocated to this strategy. NOTE: the total of all allocations will need to add up to 100%. Example: If only 50% is allocated to our Strategy MOV, the reminder of 50% need to be allocated to the "BASE" strategy, which represents the trading framework itself.
NOTE: The same strategy (eclipse project) can be run with different parameters (i.e. conf-mov1.properties & conf-mov2.properties). In this case NAME will be defined as MOV1 & MOV2 and FAMILY will be defined as MOV.
DB table watch_list_item
Also, we need to add a record to the table watch_list_item point to the securityId xxx of the security (Isin = US38259P5089) that we want to feed through and the strategyId yyy of the newly defined strategy
SECURITY_FK STRATEGY_FK PERSISTENT
xxx yyy true
Operation Modes
Back Testing / Simulation Mode
The System can be run in Back Testing / Simulation mode by running the class com.algoTrader.starter.SimulationStarter inside the strategy project algotrader-xxx. This class is part of algotraderlight, but it has to be executed from within the strategy project algotrader-xxx. If using Eclipse specify the following in the Launch Configuration:
Main-Tab:
Project: algotrader-xxx
Main Class: com.algoTrader.starter.SimulationStarter Arguments-Tab / Programm-Arguments:
simulateWithCurrentParams Arguments-Tab / VM-Arguments:
-Dsimulation=true
-DdataSource.dataSet=yyy
In simulation mode the trading framework as well as all strategies marked as autoActivate (in the database table strategy) are started inside the same JVM.
Securities specified within the table watch_list_item are fed to the corresponding Esper engine instances. The field persistent has the following meaning:
persistent = true: these securities will always be delivered to the strategy (i.e. for S&P500)
persistent = false: these are securities requested by a strategy for a specific duration of time (i.e. Options on S&P500). When restarting the simulation, these have to be requested again.
The CSV files containing the relevant tickdata has to be placed into a subdirectory of /algotrader-xxx/results/tickdata/. This subdirectory has to be specified in the file conf-base.properties or as a VM argument as follows:
dataSource.dataSet = current (this will pick CSV files from the directory /algotrader-xxx/results/tickdata/current) The name of the CSV-File has to be according to the defined isin in the table security: i.e. CH0008616382.csv
During simulation all Esper engine instances are set to use external timing, meaning that esper time will advance according to supplied market data files. The AdapterCoordinator is responsible to create the necessary CurrentTimeEvents and send them to the Esper engine instances. All orders are carried out in a simulated mode using the actual market prices supplied by the tick data. At the end of each simulation, performance statistics are displayed:
execution time (min): 1.38 dataSet: current
netLiqValue: 1'152'940.16
month-year: Mai-10 Jun-10 Jul-10 Aug-10 Sep-10 Okt-10 monthlyPerformance: -5.40% -0.30% -6.77% 1.68% 11.76% 13.09% n=6 avgM=2.34% stdM=8.42% avgY=28.13% stdY=29.17% sharpRatio=0.959
maxDrawDownM=6.77% bestMonthlyPerformance=13.09% maxDrawDown=24.84% maxDrawDownPeriod=140.00days colmarRatio=1.13
Before each simulation run, the database is reset to its original state. After the simulation the database modifications (mainly transactions and positions) are kept, in order to be able to perform further DB based analysis.
Live Trading Mode
The live Trading Mode is engaged by setting the following parameter inside the file conf-base.properties or as a VM argument:
simulation = false
In this mode, the system is connected to the broker interface specified inside applicationContext-server.xml by aliases:
<alias name="iBOrderService" alias="orderService"/>
<alias name="iBMarketDataService" alias="marketDataService"/>
All Esper engine instances are set to use internal timing. Market data received from the broker interface is fed to the Esper engine instances in real time. Orders are executed in an automated fashion and executions are recorded to the database.
Broker Interfaces
The System provides generic interface functions to connect AlgoTrader to different broker systems. The following two broker specific interfaces are currently available:
InteractiveBrokers using IB Java API
FIX using QuickFixJ (in development)
The generic broker interface consists of the two abstract services OrderService and MarketDataService. Every broker specific implementation has to be subclass these services (i.e. IBOrderService or FixOrderService)
Order / Execution
Place Order
The method sendOrder of the OrderService is responsible to place Orders. This method takes an Order object as parameter.
The Order object is routed (by the RuleService) to the BASE Esper service instance (running inside the Trading Framework) as well as to the Esper service instance of the corresponding strategy (where potential actions like “cancel order” or “modify order” can be executed). The sendOrder method is also responsible to invoke the broker specific method sendExternalOrder. The method sendExternalOrder is contained within the generic OrderService, where it is marked abstract, so it will have to be implemented by every broker specific subclass. The broker specific OrderService will create and place the broker specific Order. In case of IB the IBOrderService will create an IBOrder and send it to IB by means of IBClient.
Handle Executions
Whenever executions are received back from the broker, they are routed to the BASE Esper service Instance, where they are correlated with the original Order object. In case of IB the methods execDetails, openOrder and orderStatus of the IBAdapter (a subclass of the InteractiveBroker EWrapper) are invoked, which create IB specific ExecDetails, OpenOrder and OrderStatus ValueObjects (=POJO’s). These ValueObjecrts are routed to the BASE Esper service instance (only to BASE not the strategy)
The BASE Esper service instance correlates the Execution with the Order (in case of IB by the Order Id) and creates the generic Fill Object. These fill objects are handed to the createTransaction method of the TransactionService which will create a Transaction object (a persistent Record in the database) based on the Fill. Fills and Transactions are separated from each other for the following reason. A Fill contains all the information received from the broker (and a reference to the Order). Where as a Transaction contains all the information related to accounting (i.e. references to position and strategy). In addition to Transactions related to Fills, there are Transactions, that are independent of Fills (i.e. Deposits, Withdrawls, Interest, etc.). OrderStatus updates, Fills and Transactions are also routed to the Esper service instance of the
corresponding strategy, where actions can be taken upon.
Exchange Simulator
In Simulation / Backtesting Mode, the system provides an Exchange Simulator Functionality, where the OrderService itself creates Fills based on the placed Order. In the currently implementd simulation mode all placed orders, will get filled.
Market Data
Market Data exists in three different types:
Bars (OHLC Price Bars, also containing adjusted close and volumes)
Ticks (Snapshot of the market at a particular point in time, containing information like last price, last time, bid, ask, volume, etc.)
PriceEvent (with it's subclasses Trade, Quote, Bid and Ask representing a single atomic event in the market)
As the following diagram shows, Broker Interfaces can provide Price Events (Bid, Ask & Trade) or Ticks. In Simulation Ticks or Bars can be provided through CSV-Files. In both Simulation and Live Trading, Bars can be generated from Ticks or Price Events through the use of the OHLCBarView. The OHLCBarView can be utilized like this:
Processing of MarketData is handled through the MarketDataService, which is marked abstract and will have to be subclassed by broker specific implementations (i.e. IBMarketDataService).
The most important methods provided by the MarketDataService are putOnWatchlist and removeFromWatchlist. Through the use of these methods new MarketData can be subscribed and unsubscribed. Subscribed securities are persisted within the DB-table watch_list_item. The actual subscription of securities to the external broker is done through the broker specific MarketDataService.
Broker Interfaces are responsible to receive Broker specific Market Data and send them into the Esper Service Instance Base. The Esper Service Instance Base will then convert these Events into generic MarketDataEvents (i.e. Ticks or Bars) which will be propagated to subscribed
Miscellaneous Topics
Tick Data Format
dateTime last lastDateTime volBid volAsk bid ask vol openIntrest settlement
The first line within the file is the header row. DateTime and LastDateTime are of type long and represent milliseconds since 1970. Example:
dateTime last lastDateTime volBid volAsk bid ask vol openIntrest settlement
1299493621002 188 1299491709000 47 52 178.1 183.2 20 868 187 1299493680524 188 1299491709000 47 52 177.2 182.9 20 868 187
Again, the name of the CSV-File has to be according to the defined isin in the table security: CH0008616382.csv
If external provided Tickdata uses other time formats (i.e. DDMMYYY-hhmmss), these should be converted to milliseconds before using with AlgoTrader.
The following Excel Add-In is provided to convert Excel-Date to milliseconds (and vice versa): Functions.xla
Configuration is done using the Apache Commons Configuration Library. For the trading framework as well as for each strategy one
configuration file exists. The file conf-base.properties is the configuration file for the trading framework. It contains configuration values for the following areas:
Dabase Access
Simulation Parameters
Execution Parameters
Broker specific Parameters
Using org.apache.commons.configuration.CompositeConfiguration configuration parameters can also be provided as a VM argument when starting up the system.
The configuration files are also used to define values for Esper variables. Because the Esper Variable system is strong typed, variables with their type have to be configured within the corresponding Esper configuration files.
Strategy vs. Strategy Family
Using AlgoTrader it is possible to run multiple instances of the same Strategy at the same time with different parameters. For this purpose, the following concepts have to be distinguished:
Strategy Family:
represented by one Maven/Eclipse Project containing Spring-Services and Esper-Modules
is defined in the table 'strategy' using the field 'family'. Strategy (Instance):
represents one instance of the Strategy Family
contains one independent Esper Service Instance
has it's own config files (esper-xxx.cfg.xml and conf-xxx.properties)
is defined in the table 'strategy' using the field 'name'.
Entities of type Position and Transaction always belong to one Strategy Instance. Even when running multiple Strategy Instances at once, it is therefor possible to extract performance figures for each strategy independently.
ServiceLocators, BeanFactory & ApplicationContext
As mentioned before, AlgoTrader is built on top of Spring, which uses BeanFactory and ApplicationContext to locate Spring Beans (= AlgoTrader-Services).
The SpringSource web site provides excellent documentation such as 'Chapter 3. The IoC container' to get you started.
In Simulation mode the Trading-Framework as well as the Strategy run inside the same JVM.
In Live-Trading mode the TradingFramework and the strategy are running in different JVM's. Through the use of RmiServiceExporters and RmiProxyFactoryBean, Strategies can call Services from the TradingFramework. Behind the scenes this is done transparently through RMI. Please see 'Chapter 17. Remoting and web services using Spring' for further details.
ServiceLocators
Access to locally running Services is provided by the ServerServiceLocator. To access Services from the TradingFramework remotely, the RemoteServiceLocator has to be used.
In addition to ServerServiceLocator and RemoteServiceLocator, AlgoTrader provides a convenience class ServiceLocator, which returns the correct ServiceLocator for a give situation.
CommonServiceLocator.commonInstance() will return:
the RemoteServiceLocator if called in live trading mode from within a Strategy
the ServerServiceLocator if called in simulation mode or from within the trading framework
ServerServiceLocator serverInstance() will return:
the ServerServiceLocator if called in simulation mode or from within the trading framework
an error if called in live trading mode from within a Strategy
In Live-Trading mode the name of the strategy (i.e. BASE, MOV, etc.) has to be declared through the configuration parameter strategyName (either as a vm-argument or withing conf-xxx.properties).
AlgoTrader provides the following BeanFactories, which are instantiated by the ServiceLocators:
beanRefFactory.xml is used by the ServerServiceLocator
beanRefFactoryClient.xml is used by the RemoteServiceLocator
beanRefFactorySimulation is instantiated by the SimulationStarter through the init() method.
ApplicationContext
AlgoTrader provides the following ApplicationContext XML-Files :
applicationContext.xml contains all Services and DAO's (generated by AndroMDA)
applicationContext-server.xml contains additional custom beans used by AlgoTrader as well as the configuration of the Broker Service Beans
applicationContext-remote.xml contains additional custom beans used by strategies when run in remote mode
applicationContext-client.xml has to be provided by the Strategies and contains the services used by them.
applicationContext-export-remoteServices.xml contains all RmiServiceExporters to make Services remotely available (generated by AndroMDA)
applicationContext-import-remoteServices.xml contains all RmiProxyFactoryBean to call remote TradingFramework Services from the Strategies, as if they were local services (generated by AndroMDA)
Security
At the current stage the system is a NON secure application. It does not provide authentication or authorization services of any kind. All passwords are stored in plain-text.
It is therefore expected that the system is run in a secure environment and not being accessed from the outside world. For remote access to the system (i.e. JMX) it is recommended to use an SSH tunnel.
The following RMI connections are in use by the system:
RMI Service Description
Spring Remoting through RmiServiceExporter defined in applicationContext-export-remoteServices.xml The following Socket Connections are used by the system:
Socket Description Ports
Esper Socket through Plugin EsperIOSocketAdapter defined in esper-xxx.cfg.xml 1101 and up
MySql Database connection 3306
Java Debugging Agent specified as a VM argument 8000 and up IB Gateway defined by IB Gateway configuration 4001
AspectJ
AspectJ is used for the following situation. The Trading Framework is responsible for fetching entities from the database. If needed, these entities are transmitted to the strategies through RMI. Within the strategies, these entities are in a detached state. Access to uninitialized associated collections or proxies will therefore raise an org.hibernate.LazyInitializationException. The
com.algoTrader.util.LazyLoaderAspect defined in /META-INF/aop.xml catches access to uninitialized associated collections or proxies and initializes these collections or proxies by means of the com.algoTrader.service.LazyLoaderServiceImpl.
Logging
Logging is done using the log4j framework. The Logging system is configured in one central log4j.xml, which contains the following Appenders:
Appender Description
StdOut Logs to Standard Out StdErr Logs to Standard Error
Debug Logs everything to a non-appending file File Logs Info and up to a appending file
Portfolio Logs Portfolio Value as well as other parameters to the portfolio log file Email Sends Email Messages on Errors
SMS Sends Text Messages on Errors
TransactionMail Sends Email Messages on executed transactions TransactionSMS Sends Text Messages on executed transactions
