Algorithmic & High Frequency Trading
Ernest Chan, Ph.D.
Chairman, QTS Capital Management, LLC.
CEO, Predictnow.ai Inc.
Nahid Jetha, Ph.D.
CEO, QTS Capital Management, LLC.
About Ernie
• Ph.D. (physics), Cornell U.
• Researcher at IBM T. J. Watson Lab in machine learning (‘94-’97).
• Quantitative researcher/trader for Morgan Stanley, Credit Suisse, and various hedge funds (‘97-’05).
• Author:
• Quantitative Trading (Wiley 2009).
• Algorithmic Trading (Wiley 2013).
• Machine Trading (Wiley 2017).
• Quantitative Trading, 2ed (Wiley 2021).
• Adjunct faculty at Northwestern U.
• Formerly associate adjunct professor of finance at NTU.
• Blogger: epchan.blogspot.com
• Chairman of QTS Capital Management
• CEO of Predictnow.ai.
About Nahid
• Ph.D. (physics), UBC.
• Professor of physics, UBC.
• HFT Researcher, Athena Capital Research (2017-2019).
• Hedge fund advisor and consultant.
• COO of QTS Capital Management (2021).
• CEO of QTS Capital Management.
What differentiates algorithmic
trading from discretionary trading?
Possibility of backtesting a strategy using historical data.
Feed historical data instead of live market data into a computerized trading program → get historical
performance instead of suffering live performance.
Differs from ‘simulation’:
Historical data is real, not simulated.
Objective is not stress-testing the program under extreme market conditions.
Choosing a backtesting platform
Many different criteria. Is the programming language easy to learn
and use?
These tend to be
‘scripting’ or ‘REPL*’
languages that do not involve ‘compilation’
unlike Java, C++ or C#.
Or they are special purpose backtesting
platforms.
Excel MATLAB
R Python
Deltix
QuantHouse by Iress.
* REPL=‘Read-eval- print-loop’
Choosing a backtesting platform
• Retail platforms:
• QuantConnect
• Blueshift
Choosing a backtesting platform
MATLAB, R, and Python are general purpose programming language / computing platforms.
Therefore, they have the maximum flexibility and can handle arbitrarily complex strategies.
These languages are most commonly used by quants in major banks or hedge funds but are equally accessible to independent traders.
Choosing a backtesting platform
• Are we vulnerable to ‘look-ahead bias’ using the platform?
• Look-ahead bias means we are using tomorrow’s prices to generate today’s trading signals.
• Easy to commit this error using any programming languages except the special purpose platforms.
• For special purpose platforms, historical data are treated in the same way as live market data: fed into the trading strategy one bar at a time.
• Therefore, look-ahead bias is architecturally impossible.
Choosing a backtesting platform
• Is it easy to turn the backtest program into an Automated Trading System (i.e. automated execution program)?
• It is possible to turn MATLAB/R/Python backtest program into an ATS, but it is not a push-button operation. Lots of low-level data and trading server connections need to be made.
• In contrast, it is a push-button operation to turn a backtest program built on a special purpose platform into an ATS.
• The ease of conversion from backtest to ATS is important to prevent transcription bugs and ‘look-ahead bias’.
Choosing a backtesting platform
• Can the platform backtest high frequency strategies?
• High frequency means 1-millisecond bars or shorter.
• Due to memory issues, MATLAB/R are unable to handle high frequency backtesting.
• Special purpose C++ platforms are specially optimized for this task and perform better.
How much backtest data is
needed for statistical significance?
1. If you want to be statistically confident (at the 95% level) that your true Sharpe ratio is equal to or greater than 0, you need a backtest Sharpe ratio of 1 and a sample size of 681 data points (for, e.g., 2.71 years of daily data).
2. The higher the backtest Sharpe ratio, the smaller the sample size is needed. If your backtest Sharpe ratio is 2 or more, then you need only 174 data points (0.69 years of daily data) to be confident that your true Sharpe ratio is equal to or greater than 0.
3. If you want to be confident that your true Sharpe ratio is equal to or greater than 1, then you need a backtest Sharpe ratio of at least 1.5 and a sample size of 2,739 (10.87 years of daily data).
4. These results not only apply to backtest, but also to out-of-sample (paper trading) length.
5. Details see Bailey, 2012. ‘The Sharpe Ratio Efficient Frontier’.
Two main types of trading strategies
1. Mean reversion
• ‘Buy low sell high’, ‘Short high buy cover low’.
• Short volatility (Ang, 2014)
• ‘Value investing’.
• Liquidity providing (market making).
• Negative feedback to market prices, market stabilizing.
• Limited profit potential, unlimited loss to trader.
• Frequent, consistent, small gains. Rare major
loss.
Two main types of trading strategies
2. Momentum
• ‘Buy high and it goes higher’, ‘Short low and it goes lower’
• ‘Trend following’.
• Long volatility.
• ‘Growth investing’.
• Liquidity taking.
• Positive feedback to market prices, market destabilizing.
• Unlimited profit potential, limited loss to trader.
• Frequent small losses. Rare big gain.
How certain strategies lead to market meltdown
What’s good for trader not good for market!
The tragedy of the commons.
Individual risk management measures often lead to contagion and market meltdown.
Risk management measures often similar to momentum strategies and contribute to momentum.
‘Contagion’ due to risk management.
Contagion
Contagion leads to momentum
Contagion due to risk management.
Suppose stock A is commonly long by levered hedge funds.
• Suppose fund α suffers heavy, possibly unrelated loss.
• Risk manager (via Kelly formula?) of α demands portfolio size reduction.
• α sells stock A.
• Stock A goes down in price.
• Fund ß now suffers heavy loss if they hold A and other such stocks.
• Risk manager of ß demands deleveraging.
• ß sells stock A.
• Stock A goes down further in price: contagion leads to momentum!
Key driver: the need to maintain constant leverage in face of loss.
Contagion leads to momentum
• This actually happened in August of 2007.
• See Khandani, Amir, and Lo, Andrew. 2007. ‘What Happened to the Quants in August 2007?’
https://web.mit.edu/Alo/www/Papers/august07.pdf
Levered ETF Momentum
• Levered ETFs must keep ratio of market value of holdings to net asset value constant at market close.
• E.g. UPRO is levered 3x of SP 500 returns.
• If market index return is negative (positive), sponsor of ETF need to sell (buy) stocks to maintain leverage.
• Exercise: Suppose UPRO has about $270M assets at previous close. If SPX goes down 2%, what’s the market value of the holdings it needs to sell?
Levered ETF Momentum
• Market value of holdings at previous close=3x$270M=$810M
• Decrease in NAV due to -2% market return=$16.2M
• NAV at today’s close=$270M-$16.2M=$253.8M
• Target market value at today’s close=3x$253.8M=$761.4M
• Market value of holdings at today’s close=$810M-
$16.2M=$793.8M
• Need to sell=$793.8-$761.4M=$32.4M
Levered ETF Momentum
• This selling of $32.4M of holdings towards market close, in an already down market, generates momentum in UPRO, as well as in the SPX component stocks themselves.
• Strategy:
• Buy (sell) UPRO if return from previous day’s close to 15
minutes before today’s close is greater (smaller) than (+/-) 2%.
• Exit at market close.
• APR=32%, Sharpe Ratio=2.2 from mid-2011 to mid-2012.
Futures trading
Futures are derivatives (contracts) on any underlying: stocks, indices, bonds, FX, crypto, etc.
Such contracts have definite expiration date and time.
Long a future: right and obligation to receive underlying at expiration – though few do.
Short a future: right and obligation to deliver underlying at expiration – though few do.
Futures can be used for hedging (e.g. farmers, oil companies).
Futures can also be used for speculation (to increase leverage).
Futures can be traded using discretionarily, algorithmically, or even at high frequency.
Most futures strategies are momentum strategies.
• Due to persistence of ‘roll returns’ or ‘roll yield’ or ‘convenience yield’ of futures.
Roll Returns As Driver of Futures Momentum
• Total returns in futures = returns of spot price + roll returns.
• Even if spot price is unchanged, futures price can still change.
• If roll return is positive: (normal) backwardation.
• If roll return is negative: contango.
Backwardation = positive roll returns
• Story attributed to J.M. Keynes
Types of Momentum
• Time series momentum:
• Past returns of a price series are positively correlated with future returns.
• E.g. A stock that went up will go higher.
• Cross-sectional momentum:
• Past relative returns are positively correlated with future relative returns.
• I.e. Past returns of an instrument that out(under)-performs another instrument will continue to do so.
• E.g. AAPL outperformed BBRY last year, though both went up.
AAPL will continue to outperform BBRY this year, even though we are in a bear market for tech stocks.
Roll Returns and Futures Momentum
• Roll returns are much less volatile than spot returns, and they maintain the same sign for long periods.
• If roll returns dominate total returns of a future (at long time-scale) ֜ time-series momentum.
• Even if spot returns dominate total returns, as long as they are not anti-correlated with roll returns, they can be arbitraged away in a long-short portfolio ֜ Cross-Sectional Momentum.
Options trading
• Options are derivatives (contracts) on any underlying:
stocks, indices, bonds, FX, crypto, even futures.
• Option contracts have definite expiration date and time.
• Options can be ‘calls’: right but not obligation to buy at some strike price; or ‘puts’: right but not obligation to sell at some strike price.
• Options can be American: can be exercised at any time; or European: can be exercised only at expiration.
• (Most) option contracts have definite strike price.
• For a call, option will expire worthless if underlying price is below strike price at expiration.
• For a put, option will expire worthless if underlying price is above strike price at expiration.
Options trading
Options can be used for hedging.
•E.g. Buy puts on a long stock portfolio.
•E.g. Buy calls on a short stock position.
Can be used to speculate on whether underlying price will go up or down (just like futures, but
with downside protection).
•E.g. buying calls on GME, AMC, and other ‘meme’ stocks has become a favorite pastime of retail ‘Robinhood’ traders during the pandemic, with major market implications.
Can be used to speculate on whether the (implied) volatility of
the underlying price will go up or down.
•Volatility trading’.
•Typically, ‘delta-hedged’.
•Can use ‘straddles’ or
‘strangles’.
Can also be used just to harvest premiums, like an insurer harvesting insurance premium.
•Actually, one of the most common uses of options.
•Potentially unlimited downside risk!
Market impact of options trading
• E.g. 1: End of day momentum no longer works due to ‘buying on dips’ and net market delta turning positive.
o See Lilly Francus’s work:
https://www.bloomberg.com/news/articles/2021-08- 05/investing-influencers-use-twitter-tiktok-to-educate- their-audiences?sref=MqSE4VuP
• E.g. 2: GME short squeeze, causing billion-dollar fund Melvin Capital to lose 50% of capital in days.
Options trading
• Options can be traded using discretionarily, algorithmically, or even at high frequency (typically by market makers to take advantage of slow quote updates*.)
• Most non-HFT options strategies are mean reversion (short volatility) strategies.
• It is very hard to consistently profit by buying options due to time decay of premium!
*Durbin, ‘All About High-Frequency Trading’.
Statistical arbitrage
It is a common name applied to algorithmic trading strategies.
As ‘arbitrage’, it is meant to apply to at least 2 instruments, maybe a portfolio of instruments.
As ‘statistical’, it means the relationship among these instruments are not guaranteed, but only statistically significant.
Can be applied to stocks, bonds, futures, FX, options, crypto.
Statistical arbitrage
• A simple statistical arbitrage strategy is pair trading.
• E.g. Buy AAPL short MSFT when the ‘spread’ between AAPL and MSFT is ‘cheap’ (small) and exit positions when the spread is ‘expensive’ (large).
• This is a mean reversion trade.
• E.g. Buy AAPL short MSFT when the spread is expensive and exit when it is even more expensive.
• This is a momentum trade.
0 500 1000 1500 2000 2500 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Jan 1, 99 - Oct 1, 07
Cumulative returns
Mean-reversal strategy on AUDCAD
0 50 100 150 200 250 300 350 400 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6
Mean-reversal strategy on GLD-GDX
May 2006-November 2007
Cumulative returns
0 200 400 600 800 1000 1200 -0.05
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45
20060523-20100521
Cumulative Returns
Mean-reversal strategy on GLD-GDX-USO
XLE vs USO arbitrage
0 500 1000 1500
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
20060426-20120409
Cumulative Returns
Statistical arbitrage
• An example statistical arbitrage strategy applied to a portfolio of stocks:
• Rank all the stocks based on their one-day returns.
• Buy the bottom decile and short the top decile at each market close.
• This is a mean reversion trade.
• Alternatively, buy the top decile and short the bottom decile at each market close.
• This is a momentum trade.
Statistical arbitrage
• Statistical arbitrage strategy is typically dollar- or even market-neutral.
• Their beta w.r.t. market index is close to zero.
• It is ‘self-funded’ – does not cost much capital because short positions fund purchase of long positions.
• Requires low margin cash (20%?) due to perceived lower risk of portfolio.
Linear Long-Short Mean Reversion Model on SPX stocks
Cross-sectional Futures Momentum Strategy on physical commodities
Statistical arbitrage
• Since trader needs to monitor multiple instruments prices to determine entry/exit/rebalance, and must trade multiple instruments simultaneously, statistical arbitrage strategies are typically algorithmic.
Dispersion Trading
• Index arbitrage: buy/sell stocks, sell/buy index futures.
• Dispersion trading: buy stock options, sell index options.
• Never buy index options – overpriced due to demand for portfolio insurance.
Esp. OTM puts. ‘Volatility skew’.
• Also called “correlation trading”
• Shorting implied correlation of stocks.
• Risk: when (good/bad) extreme events occur, stocks exhibit ‘tail dependence’ – rise in realized and often implied correlations.
Dispersion Trading
Enforce Δ=0 by trading stock/index straddles/strangles.
Enforce
Enforce ϒ=0 by adjusting value of short index option position (which has -|ϒ|) to hedge long stock option positions (+|ϒ|);
•OR:
Enforce
Enforce θ=0 by a similar method.
Enforce
Our strategy example will use first constraint: ϒ=0.
Use
Dispersion Trading
• Smart index arbitrage: select subset of stocks in index to increase arbitrage profits.
• Smart dispersion trading: select subset of stock options.
• Pick 50 stocks in SPX whose straddles have the highest (least negative) θ and rebalance daily at market close.
• Choose straddles with >=28-day tenor.
• Assuming mid-price executions and no commissions:
CAGR=19%, Calmar ratio=0.4.
• Daily options data from OptionsMetrics.
Dispersion Trading
Dispersion Trading
• Surprisingly, neither the financial crisis of 2008, the flash crash of 2010, nor the market turmoil of August 2011 caused much drawdown.
• Question 1: Is the earnings season likely a good or bad time for the dispersion strategy?
• Question 2: Are macro-economic releases likely a good or bad time for this strategy?
Cross-sectional Mean Reversion of Implied Volatility
Buy ‘cheap’ stock options, short
‘expensive’ ones.
Note: we do not expect cheap options to increase in price or expensive options to decrease in price.
Only expect relative prices (or “spread”) to mean revert.
I.e. cross-sectional mean reversion.
‘Cheapness’ measured by implied volatility.
Cross-sectional IV Mean Reversion
Near daily market close, for every stock in SPX, select candidate call option with
around 28-day tenor with lowest IV.
• These will typically be ATM calls.
Buy the 50 call options with the lowest IV.
• These will typically be OTM puts. (See Zhang, 2008)
Short the 50 put options with the highest IV.
Rebalance daily.
Due to put-call parity*, can be implemented by shorting calls and
buying puts instead.
Cross-sectional IV Mean Reversion
• Assuming mid-price executions and no commissions, daily return is 1.1% from 2007-2013.
• Return is high because OTM options have high leverage.
• Portfolio is not delta-neutral!
• It is bullish – delta > 0.
• Can delta-hedge with short SPY positions.
• Portfolio is not vega or theta neutral
• Can weight long and short options positions to enforce either vega or theta neutrality, just like dispersion trading.
• We are shorting OTM puts: subject to tail risk!
Cross-sectional RV Mean Reversion
• What if we sort by historical volatility instead of IV?
• Cao and Han (2013) suggests if two stocks have same IV, the options of one with higher historical volatility will have lower return.
Execution is make-or-break
• All the above strategies assume executions at mid- prices.
• Most options strategies are not profitable if we must pay bid- ask.
• There is no way to backtest if this is realistic.
• Options traders find that limit prices may depend on real- time order book information.
• Fills in between bid-ask happen quite often for liquid options.
• Smart order management system when orders are not filled is key to success.
High frequency trading
HFT can be either mean reversion, or momentum strategy.
It can also be either: latency arbitrage, cross-border arbitrage, spoofing, etc.
Some HFT techniques can be illegal in some (but not all) markets or jurisdictions.
HFT is necessarily algorithmic since no human can make decisions in the millisecond timeframe.
HFT, MFT, and LFT
• Definitions:
• HFT (high frequency trading) has latency < 1ms.
• MFT (millisecond or medium frequency trading) has latency > 1ms but
<= 20ms.
• LFT (low frequency trading) has latency > 20ms.
• Latency is measured by:
• Computational: Time it takes your program to decide what order to generate.
• Order submission: Time between your program’s order submission and the exchange’s reception.
• Market data. E.g. Time between someone’s order execution and your notification of it.
• Order status. E.g. Time between your order execution and your notification of it.
Technology of Trading
Colocation
Extranet (private network) Direct (enriched) data feed
Low latency programming language/platform Machine learning (including NLP)
High Performance Computing (HPC)
Colocation
Trading program need to be in same data center as broker’s order/data servers.
Even better: same data center as the exchange to which your order is routed.
Not necessary to own server.
Can be Virtual Private Server
Sharing same physical machine with others.
Cost < $1,000.
Colocation
What if orders are routed to different exchanges?
• Extranet needed to connect our server to different exchanges to receive data and submit orders.
• Extranet ~ internet but operated by private companies: less traffic and lower latency than internet.
• Example vendors: BT Radianz, Savvis, TNS.
• Much cheaper than dedicated communication line.
Index arbitrage between stocks and stock index futures (NYSE vs CME) often require microwave transmissions or laser.
• https://www.reuters.com/article/us-highfrequency-microwave-idUSBRE9400L920130501
• ‘The first microwave connections between London and Frankfurt have been launched, cutting the time to send a trade by about 40 percent compared with optic fiber cables.’
• Physics: electromagnetic waves travel much faster in vacuum than in solids.
Historical data
• Vendors for tick data:
• Algoseek.com
• NBBO and trade ticks (with aggressor tags if available).
• +LOB data for futures.
• Millisecond time stamps.
• Report futures settlement prices.
• Has futures calendar spreads.
• Tickdata.com
• BBO and trade ticks (but not level 2 quotes).
• Millisecond time stamps.
• No futures calendar spreads.
• MayStreet
• Similar offering as Algoseek + ‘message’ data for stocks.
• Historical ITCH feed from exchanges.
• E.g. itchdata.nasdaq.com
Direct Data Feeds and ITCH
• Each exchange published its own private, direct data feeds.
• Such direct data feeds are 1ms faster than the SIP consolidated feeds broadcasted to the public.
• <1ms latency is necessary even for MFT to avoid gaming techniques by HFT.
• Such direct data feeds contain much richer content than SIP:
often using the ITCH format.
• ITCH data contain not only quotes and trades prices and sizes.
• ITCH data contain cancellations and revisions of displayed orders.
• ITCH data can contain aggressor flag for trades.
• ITCH feed can be expensive unless server colocated within broker data center.
Direct Data Feeds and ITCH
• The best, more enriched, live data come directly from the exchanges.
• E.g. ITCH feed from Nasdaq, MDP feed from CME.
• Colocation at the exchange’s servers are typically required.
• Could cost > $10K / month.
• Many data vendors would redistribute them (E.g.
Thomson Reuters):
• Pros: easier-to-use API, better customer support.
• Cons: even more expensive and may add latency.
Backtesting HFT
• Can we really backtest H/MFT strategies?
• Backtest can reject strategy has return ≤0, but not necessarily affirm return > 0.
• Backtest H/MFT strategies assuming market orders.
• Cannot backtest effects of adverse selection or gaming strategies, or any strategies that depend on response of order book.
• Difficult to backtest limit orders.
• Except if a good fill simulator is available, such as Lime Strategy Studio.
• See Robert Almgren’s talk: ieor.columbia.edu/financial- engineering-practitioners-seminar-robert-almgren-73239
• MFT and LFT can share many common tools for backtesting.
Live Trading Platform
• C#, C, C++, Java can all make trade decisions and submit orders with latency ≤ 10ms.
• MATLAB, R, Python all have latency of ≥ 60ms.
• Special purpose platforms with latency < 20ms:
• Deltix
• QuantHouse
• Lime Strategy Studio [Especially designed for HFT.]
• These platforms can also be used for backtesting.
Machine learning and HPC
• Spotting arbitrage opportunities / generating trading signals – the most uncertain and difficult use case!
• Risk management and portfolio optimization – much more successful use case!
• Market surveillance and data mining – useful for regulators to discover insider trading, spoofing, and market manipulations.
• More on this topic later.
ML is useful in multiple ways:
• Many financial problems require optimizations.
• Many ML problems require optimizations.
• Many optimizations will take exponentially long to complete using conventional methods.
• HPC includes quantum computing will make previously intractable problems tractable.
• More on this topic later.
High performance computing (HPC):
How does HFT impact other traders?
• Profits of non-HFT traders still sensitive to issues of:
o Gaming by HFT
▪ 70% of US equity market volume due to HFT. (zerohedge.com 2010*)
o Thin top of book liquidity.
o Order types, smart routing, and dark pool selection.
o Adverse selection.
o Information leakage.
o Mini flash crashes and withdrawal of liquidity.
*http://www.zerohedge.com/article/what-percentage-us-equity-trades-are-high- frequency-trades
Gaming by HFT
• Front-running
o HFT will be first to take out any price improvements by liquidity providers.
o Liquidity takers (market orders) can seldom enjoy price improvements.
• Ticking
• Applies if bid-ask spread > 2 ticks.
• HFT places buy order at best bid + 1 tick if bid size ≫ ask size.
• Once filled, place sell order at best ask – 1 tick.
• If sell order not filled, sell at original best bid.
Best Bid Best Ask
> 2 ticks
1 tick
1 tick
B
S
S’
Gaming by HFT: Ticking
• Effect on our original NBBO orders: take longer to get filled and suffer opportunity cost and adverse selection.
• In case of US equities market, broker-affiliated HFT can place sub-penny orders in certain dark pools to ‘improve’ our original displayed order by less than a tick.
Gaming by HFT: Ratio Trade
• Applies to futures markets which fill orders on a pro-rata basis (not time-priority).
• E.g. Eurodollars on CME.
• Join the best bid if bid size ≫ ask size.
• Once filled, place sell order at best ask.
• If sell order not filled, sell at original best bid.
Gaming by HFT: Flipping/Layering
Place large buy order at best bid and a small sell order at best ask, to create impression of buying pressure.
Other traders fooled to buy at ask due to perception of buying momentum.
Once our sell order filled, cancel buy order.
Disappearance of buying ‘pressure’ causes other traders sell at bid.
We cover short position at bid.
Layering
• Beware of ‘bluffing’: another HFT selling at your large buy order, waiting to buy back at lower price as you lower your ask.
• Layering is considered market manipulation: an illegal trading tactic. See:
• http://www.forbes.com/sites/billsinger/2013/02/2 8/da-big-kahuna-wipes-out-on-stock-manipulation- ladder/
http://www.sec.gov/News/PressRelease/Detail/Pre ssRelease/1365171484972#.U5cHLfmwJPp
Gaming by HFT: Stop Hunting
• Support and resistance levels often well-known, often at round numbers.
o E.g. $17.00 instead of $17.15.
• Stop orders often cluster at these levels.
• HFT can submit large sell orders near support level so price drops below support.
• Sell stop orders are triggered.
• Price drops further.
• HFT buy covers.
o Ref. Osler, Carol. 2000. Support for Resistance: Technical Analysis and Intraday Exchange Rates. Federal Reserve Bank of New York Economic Policy Review 6 (July 2000): 53-65.
Gaming by HFT: Hide and Light
• ‘Hide and light’ orders
• A special-order type that is used to gain time priority over our limit orders.
• First placed as hidden order that will lock an away market (permitted by Regulation NMS).
• Example (Ref. BATS Display-Price Sliding flyer)
• NBBO=$10.00 x $10.01
• BATS BBO=$10.00x$10.02
• A bid on BATS book at 10.01 will be Display-Price slid to 10.00, but has a hidden working price of 10.01
• When NBBO offer lifts to 10.02 or higher, bid will be unslided and redisplayed at 10.01, maintaining priority.
L=BATS, N=other
Gaming by HFT: Hide and Light
• Why is this an advantage to HFT?
• Buying at original NBO ($10.01) would cost a taker’s fee.
Buying at new NBB ($10.01) will earn rebate.
• No need to cancel and replace orders to remain at top-of- book and maintain priority of hidden order.
• HFT can take advantage of slower SIP (Securities Information Processor) feed (i.e. consolidated feed) relative to direct feed from exchanges.
• SIP feed is used to determine whether market is locked.
• SIP feed slower than direct feed by 10-15ms.
• (However, see Shengwei Ding et al ‘How Slow is the NBBO?’ which indicates delay is only 2 ms for very active stocks in 2013.)
• HFT can determine actual market not locked and place Hide-and- light order to step ahead of our orders.
Exercise
A trader sends market order to sell 600 shares to exchange L.
a) What orders do L have to reroute, and to which exchange?
b) Suppose after order was received by L, but before execution, exchange N2’s NBB was increased to $10.03. Does L need to reroute part of the order to N2?
Answer
a) Since NBB is at N1, all 600 shares must be routed there. After it walked the book, unfilled order (200 shares) will then be routed to L, since it now has the NBB.
➢ Note this may get worse average price then if we keep most of order at local exchange. Hence ISO.
b) Routing is determined by state of market at time of submission, so
Hide and light order
• E.g.
• Display-Price Sliding on BATS
• Price-to-Comply on Nasdaq
• Post No Preference Blind on NYSE ARCA
• When to use them:
• If you want to provide liquidity and earn (high?) rebates at specific venue.
• If you want to remain at top-of-book.
• If you want to maintain time/queue priority.
• If your order will lock current market, but you expect market to move away and unlock it.
Order Types Optimization
Immediate Or Cancel (IOC) order Intermarket Sweep Order (ISO) Hide and light order
DAY ISO
Immediate Or Cancel (IOC) order
• A hybrid between market and limit order.
• Like a limit order, it will not execute at worse price than limit price.
• Like a market order, if it is not immediately executable, it will be automatically cancelled.
• A good order type to use to prevent Adverse Selection.
• Unfortunately, IOC order cannot prevent adverse selection in certain FX ECNs with ‘last-look’.
• Can apply IOC to market order
• Will not be routed to another exchange with NBBO.
• Will not execute against quote worse than NBBO.
• Portion not executed will be cancelled.
Sidebar: Adverse Selection
• When a limit order only gets filled when the price usually moves against our position subsequently.
• Cause: the counterparties are informed traders, with superior information / short-term price prediction models.
• Such orders from informed traders often called
‘toxic order flow’, and they frequently utilize market or IOC orders.
• Our IOC orders won’t interact with their market/IOC orders.
Sidebar: Last-look in FX ECNs
• In many FX ECNs, liquidity providers (e.g. banks’
dealing desks) can cancel a limit order within a few hundred milliseconds, even if it is executed against by a buy-side marketable order. (I.e. limit orders not ‘firm’)
• Large liquidity providers typically have a timelier view of order flow and market directions than buy- side.
• Even IOC orders can be adverse-selected if last-look is on.
Exercise
Answer
• This is a trick question!
• All 300 shares will be filled at L, since at the time of order submission, NBB is at exchange L.
• It doesn’t matter whether we have IOC modifier or not.
Intermarket Sweep Order (ISO)
• An order modifier applied to limit orders (usually IOC).
• When sent to an exchange or trading venue ‘L’, no routing to another (‘away’) venue ‘N’ is required even if NBBO is over at ‘N’.
• ‘L’ – Local, ‘N’ – NBBO
• ISO can immediately sweep order book at ‘L’.
• Trader herself is responsible for sending other parts of parent order to ‘N’ and other away venues if they have better top-of-book quotes to comply with Reg NMS Rule 611, the Order Protection Rule.
Intermarket Sweep Order (ISO)
Useful when:
• Order size > NBBO size
• Routing to venue ‘N’ may actually result in worse average price due to non-NBBO quotes at L.
• When venue ‘L’ is ‘fast’: NBBO (based on slow SIP feed!) from ‘slower’ trading venues may be outdated.
o Routing to venue ‘N’ induces latency and NBBO may change for the worse at arrival.
• There may be undisplayed quotes at ‘L’ better than NBBO at ‘N’.
Intermarket Sweep Order (ISO)
• E.g.
‘L’ has offers ‘N’ has offers
$10.5: 1000
$10.3: 500
$10.2: 100 (NBO)
Given a buy limit order of 600 @ $10.3, buyer is better off submitting ISO order of 500 shares to ‘L’ instead of having entire order routed to ‘N’, as long as she also separately sends buy order of at least 100 to ‘N’.
Intermarket Sweep Order (ISO)
• If we do not use ISO, in fast markets we may be taken advantage of by HFT who has direct feeds from exchange.
• NBBO prices may be stale due to slow SIP data feeds.
• Supposedly marketable orders sent to NBBO venue may be rejected.
• ISO enjoy simultaneous ‘parallel processing’ at multiple venues.
• Interactive Brokers sends all orders as ISO by default.
• Potential HFT strategy: order flow* of ISO trades is a better leading indicator of returns than NISO trades.
Sidebar: Flash Crashes and Liquidity Withdrawal
• There is also danger in using ISO orders.
• There may be displayed quotes below top of book at ‘N’ better than book prices at ‘L’.
• Using ISO may execute at inferior prices while sweeping order book at ‘L’.
Sidebar: Flash Crashes and Liquidity Withdrawal
• E.g.
‘L’ has bids ‘N’ has bids
$9.99: 100 (NBB)
$9.98: 300
$9.97: 600
$9.95: 300
• An ISO Sell order at $9.95 for 600 shares sent to ‘L’
coupled with market order for 100 sent to ‘N’ will be filled at worse price than a market order for 700 shares routed to ‘N’
• $9.97 is not a ‘protected’ quote since it is not NBB, so there is no Reg NMS requirement for order to be routed to fill it.
Sidebar: Flash Crashes and Liquidity Withdrawal
• Some believe ISO ‘sucked dry’ liquidity at certain venues causing 2010 Flash Crash as well as
subsequent ‘mini flash crashes’.
• ssrn.com/abstract=1629402 ‘The Flash Crash: Trading Aggressiveness, Liquidity Supply, and the Impact of Intermarket Sweep Orders’.
• www.nanex.net/aqck2/4379.html ‘Proof that HFT Causes Mini Flash Crashes’
• arxiv.org/pdf/1211.6667.pdf ‘High Frequency Trading and Mini Flash Crashes’.
Sidebar: Flash Crashes and Liquidity Withdrawal
• Others believe market makers detected huge toxic (informed) order flow on that day, leading to liquidity withdrawal for risk management reason:
• ssrn.com/abstract=1695041 ‘The Microstructure of the
‘Flash Crash’: Flow Toxicity, Liquidity Crashes and the Probability of Informed Trading’.
• We will discuss the technology proposed to predict future flash crashes later.
Thin NBBO Liquidty
Thin NBBO Liquidity
• In the middle of a trading day (11:00 am ET), AAPL, the company with the largest market cap worldwide, has NBBO size of only 100x100!
• Typical NBBO size is 189*.
• Market makers, aware of ticking and other HFT games, refuse to post orders larger than 100.
• NBBO prices become irrelevant, both in backtest and in live trading.
• 1/3 of shares transacted in ‘dark’, away from NBBO.
Thin NBBO Liquidity
• Implication for backtesting intraday US equities strategies:
• Unless order type is MOO, LOO, MOC, or LOC, backtest need to be done on level 2 tick data!
DAY ISO
Again, a limit order, but has time-in-force for trading day instead of IOC.
Once order book of a venue has been swept, remaining unfilled portion will rest as limit order at top of book for rest of day.
If you can show, via fast direct feed, that an away ‘locked’ market is in fact unlocked, limit order will be displayed immediately while hide-and-light orders will remain hidden (with lower priority) until slower SIP feed
unlocks away market.
DAY ISO
• ‘Queue jumping’
• Used this way, DAY ISO can queue-jump orders arrived at venue at same price but earlier in time.
• Earn rebates!
D-limit order on IEX
• ‘Discretionary Limit’ order offered only on IEX (ECN on US stocks.)
• Behaves like regular limit order, except when AI algo predicts price is about to change.
• If so, algo will reprice limit to 1 MPV (minimum price variant, $0.01 for most stocks) “better” than expected new quote.
• Name of algorithm: ‘CQI (Crumbling Quote Indicator), ‘IEX Signal’
Use and Abuse of Dark Pools
• There are 40+ dark pools for US equities.
• Dark pool + other hidden orders account for >1/3 of all transaction volume.
• Most dark pools execute based on midpoints* of NBBO:
advantageous for marketable orders in theory.
• However, midpoints can be manipulated
• HFT can send buy limit orders to increase the NBB.
• After midpoint is raised, HFT send marketable sell orders to dark pools.
• After sell orders filled, HFT cancel buy limit orders to lower midpoint.
• After midpoint is lowered, HFT send buy marketable orders to dark pools to cover.
• This is called ‘spoofing’ – illegal but does happen.
Use and Abuse of Dark Pools
• Latency arbitrage on midpoints:
o Midpoints are determined by the SIP feed.
o SIP feed slower than direct feed by about 1ms*.
▪ *’Nasdaq Shutdown Bares Stock Exchange Flaws’, WSJ, August 24, 2013.
o HFT can use direct feeds to predict where midpoints will be and send orders to arbitrage difference.
o E.g.
▪ SIP midpoint is $10.00, but direct feed midpoint is $10.02.
▪ HFT send buy order to dark pool to buy at $10.00.
▪ Once SIP feed shows midpoint to be $10.02, HFT sells position at
$10.02 in dark pool.
Use and Abuse of Dark Pools
• There is also sub-pennying
• Similar to ticking, except price improvement is less than a tick!
• Undisplayed orders can price-improve on NBBO by less than $0.01.
• Brokers do not accept sub-penny orders from customers, but brokers themselves are allowed to place such orders in their own dark pools.
• Buyside traders placing limit orders anywhere will suffer adverse selection, as brokers will cancel sub-penny orders if they see ‘informed’/’toxic’ order flow.
Use and Abuse of Dark Pools
• Information leakage
• Broker-operated dark pools share order
information with proprietary trading desks and
‘liquidity partners’.
• Some of them may even share stop orders info!
Avoiding Toxic Dark Pools
• Use severity of adverse selection as selection criterion for dark pools.
• Can measure adverse selection by computing the P&L of unfilled shares - P&L of filled shares over short time frame (e.g. 1 sec – 30 min).
• Rf. N. Saraiya and H. Mittal, ‘Understanding and Avoiding Adverse Selection in Dark Pools’, ITG Report, November 2009.
Avoiding Toxic Dark Pools
Route orders to dark pools with low adverse selection first, and to those with high adverse selection last.
Avoid dark pools that send out IOIs (Indications of Interest) to HFT.
Avoid dark pools that expose orders to prop trading desks.
Use IOC orders to avoid gaming.
Order Flow
Order flow leads price change
• Order flow is signed transaction volume.
• Buy market order is filled ֜ order flow > 0.
• Sell market order is filled ֜ order flow < 0.
• Order flow can be aggregated over some time interval to serve as indicator.
• Researchers found that order flow is positively correlated with future price change.
• Lyons, Richard. 2001. ‘The Microstructure Approach to Exchange Rates’
• Adam, Clark-Joseph, 2013. ‘Exploratory Trading’
• Menkhoff, Lukas et al, 2013, ‘Information Flows in Dark Markets’ BIS working paper #405.
• Cartea et al. 2015. ‘Algorithmic and High-Frequency Trading’
Therefore, order flow can be used to ‘front-run’ market.
Order Flow
How do we obtain order flow info if we are not large market makers or exchange operators?
• In futures, can compute tick-by-tick whether trade executed at ask (positive flow) or bid (negative flow).
• In stocks, more difficult due to fragmented markets and dark pools that do not report quotes and report only delayed trades.
• In currencies, often impossible, but can monitor currency futures instead.
• See also Easley et al. 2012. ‘Bulk Classification of Trading Activity’, to be discussed shortly.
Signed volume is a much better predictor of momentum than unsigned volume.
Computing Order Flow
• Different ways to determine trade direction
1. Some tick data has ‘aggressor’ flag (E.g. CME MDP Market Data).
• Futures only, not stocks.
2. ITCH data (E.g. Nasdaq, HotspotFX) contains limit order additions, executions, and cancellations info.
• Useful for inferring sign of trade executions.
3. Tick Rule
• ‘Buy’ if trade price > previous trade price.
• ‘Sell’ if trade price < previous trade price.
• If trade price=previous, assign same side as previous trade.
Computing Order Flow
4. Quote Rule
• ‘Buy’ if trade occurs above midpoint.
• ‘Sell’ if trade occurs below midpoint.
• Unclassified if trade occurs at midpoint.
• Need to know historical contemporaneous midpoint, as opposed to midpoint when trade report is received.
5. Lee-Ready Algorithm
• Use Quote rule for trades away from midpoint.
• Use Tick rule for trades at midpoint.
Computing Order Flow
6. Bulk Volume Classification (BVC)
• Use only bar data with volume and last price.
• Bar can be based on fixed time, or fixed volume (better!).
• Divide bar volume V into VB (buys, positive order flow) or VS(sells, negative order flow):
𝑉𝐵 𝑡 = 𝑉 𝑡 ∙ 𝑍 𝑃 𝑡 − 𝑃 𝑡 − 1 𝜎∆𝑃
𝑉𝑆 𝑡 = 𝑉 𝑡 − 𝑉𝐵(𝑡)
o P(t) is last price.
o Z is the Gaussian (or Student’s t) Cumulative Distribution Function (CDF).
o 𝜎∆𝑃 is the standard deviation of the volume-weighted price changes between bars.
o (If volume bars are used, 𝜎∆𝑃 is just the standard deviation of price changes between bars.)
o Z by itself is a probability that indicate whether or not there is ‘informed trading’ and gives the direction of that.
o Z is called VPIN – “Volume Synchronized Probability of Informed Trading” –
Computing Order Flow
For US equities, Quote Rule and Lee-Ready are inaccurate and useless, due to hidden orders and midpoint executions in dark pools.
BVC is much less data intensive than Tick Rule.
Distribution of returns based on volume bars much more Gaussian, thus classification more accurate.
Why Machine Learning?
• “The SEC has no choice but to learn about AI and other technologies that drive robo-advice and other modern brokerage services” says Joseph Grundfest, former SEC commissioner, currently professor of law and business at Stanford U. “You can’t properly regulate something you don’t understand.
- Bloomberg Magazine, January 17, 2022.
• ML can also be used for market surveillance by regulators.
Why ML?
Single-factor, linear quant models have decaying alpha due to ease of replication.
Advances in ML specifically address overfitting issues.
New tools in ML introduce more transparency, less blackbox fitting – ‘eXplainable AI’, or XAI.
ML can be used for risk management and capital allocation, not primary signal generator.
Traditional Quant vs ML
Traditional Quant ML
Few predictors Numerous predictors
Traditional data (prices, fundamentals, etc.)
Alternative data (news, credit card transactions, etc.)
Linear Nonlinear
Intuitive, easy to replicate Unintuitive, same data -> very different models
Cannot predict probability of success Can predict probability of success Arbitrary capital allocation Logical capital allocation
Harder to overfit Easy to overfit
Hard to generate multiple backtests for statistical assessment
Easy to generate multiple backtests for statistical assessment
3 Steps
Financial data science
Find problems with data and scrub them.
Convert raw data into features.
Machine learning
Use classification or regression techniques to make predictions.
Trading strategy construction
Use predictions as input to a trading strategy.
Backtest various versions of strategy.
Data Is Nearly Everything
In the article,
Data Challenges Are Halting AI Projects, IBM Executive Says
Arvind Krishna, previously IBM’s senior vice president of cloud and cognitive software, now CEO, said about 80% of the work with an AI project is collecting and preparing data.
(www.wsj.com/articles/data-challenges-are-halting-ai- projects-ibm-executive-says-11559035800)
Challenges of financial data science
Ever-changing company names and tickers Dividend and split adjustment
Survivorship bias
Look-ahead bias of earnings (and other) data
Structural breaks in pre-processed alternative data Averaging categorical features
Exploratory Data Analysis
Beware of dual classes stocks!
• E.g. BF.A vs BF.B
How are delisted stocks denoted in your data set?
What to do with stocks with both listed and
delisted versions?
Redundant Stocks
Name and symbol
E.g. KORS vs CPRI:
name and symbol change.
• Merge them and discard KORS.
Stock
E.g. GOOG vs GOOGL:
stock splitting into 2 classes.
• Exercise: How can split and dividend adjustments fool a simple duplicate price checker?
• Removing redundant data on GOOGL.
Symbol
E.g. GOV vs OPI:
symbol change.
• Discard GOV data, which is duplicate.
Redundant Stocks
• E.g. MLND vs OVAS: symbol change.
• Discard OVAS.
• E.g. WLBA vs WLBAQ: bankruptcy.
• Discard WLBA.