Você está na página 1de 21

Shortfall Surprises

KWAKU ABROKWAH AND GEORGE SOFIANOS

KWAKU ABROKWAH n this article, we use a large sample of volume surprise explains very little of
is an analyst, Equity
Execution Strategies, at
Goldman, Sachs & Co. in
New York, NY.
kwaku.abrokwah@gs.com
I order executions to better understand
“shortfall surprises”: the difference
between actual and expected trading
costs. Our trading cost measure is execution
shortfall, including both liquidity impact and
the shortfall surprise (only 0.3%), indi-
cating that higher-than-expected volume
does not reduce trading costs.
• Volatility surprise. Higher-than-expected
volatility over the execution horizon
GEORGE SOFIANOS the opportunity cost of slow executions.1 should result in higher-than-expected
is a vice president, Equity Exhibit 1 summarizes our main findings. In shortfall. We find that the volatility
Execution Strategies, at
Goldman, Sachs & Co. in
our sample, on average, only 20% of actual surprise explains only 0.1% of the short-
New York, NY. shortfall is predictable pre-trade. Focusing on fall surprise.
george.sofianos@gs.com the non-predictable component, we investi- • Spread surprise. Higher-than-expected
gate the following possible reasons for the quoted spreads over the execution horizon
shortfall surprise: should result in higher-than-expected
shortfall. We find that the spread surprise
• Price surprise. A higher-than-expected explains 1.8% of the shortfall surprise.
price increase over the execution horizon
should result in a higher-than-expected Price surprises, therefore, are by far the
actual shortfall on buy orders and lower- most important factor explaining shortfall
than-expected shortfall on sell orders. surprises; the volume, volatility, and spread
We find that, on average, the price sur- surprises have little explanatory power. These
prise explains 42% of the shortfall sur- findings have important implications for post-
prise. For orders in large-cap stocks that trade analysis, pre-trade tools, the development
take more than an hour to execute, the of algorithms, and the choice of execution
price surprise explains 73% of the short- strategies.
fall surprise. Splitting the price surprise In interpreting post-trade execution
into its alpha and market components, quality, for example, we must somehow con-
the alpha surprise explains 38% of the trol for the underlying execution-horizon price
shortfall surprise, and the market sur- move. For pre-trade analysis, our findings sug-
prise only 4%. gest that the only way to improve the pre-trade
• Volume surprise. Higher-than-expected t-cost estimates is to better predict the alpha-
volume over the execution horizon move over the execution horizon. Better pre-
should result in lower-than-expected trade volatility and volume estimates will not
shortfall. Surprisingly, we find that the help much. The same applies for algorithm

SUMMER 2007 THE JOURNAL OF TRADING 11


EXHIBIT 1
Summary of Main Findings1

1
Sample period June 1 through December 31, 2006; 241,610 orders.
2
R-square from univariate regression of actual on expected shortfall.
3
Incremental R-square from multivariate regression analysis of shortfall surprise on the five factor surprises: volatility, spread, volume, EH-alpha and EH-market.

development and the choice of execution strategies; only orders, execution shortfall is the strike price minus the exe-
better predictors of execution-horizon alpha will signif- cution price as percent of the strike price. In Exhibit 2, we
icantly reduce execution shortfall. introduce a hypothetical order execution that we will use
In the next section, we develop our framework for throughout this section to illustrate our framework. The
explaining the shortfall surprise, followed by a section on trader receives an order to buy 60,000 shares. The strike
our data sample and the construction of our variables. price at order arrival is $25.00. The trader executes the
The following section presents our empirical findings. In order over time in three executions and the volume-
the section after that, we focus on our puzzling finding weighted execution price is $25.06. The execution short-
that volume surprises do not affect trading costs. The next fall in this example is six cents or 24 basis points (bps).
section elaborates on the distinction between volatility In addition to liquidity impact, execution shortfall
and price surprises. We conclude with a discussion of the includes the opportunity cost of delayed execution. The
implications of our empirical findings, and possible exten- opportunity cost arises because the price may move away
sions of our analysis. from the trader over the execution horizon. This price
move has both a market and a stock-specific (alpha) com-
A FRAMEWORK FOR ANALYZING ponent and we estimate both. Exhibit 3 shows the three
THE SHORTFALL SURPRISE components of execution shortfall: liquidity impact, alpha
loss, and market loss. In our example, the trader is buying
We begin by formally defining execution shortfall, our in a rising market, so the opportunity cost is positive and
t-cost measure. For buy orders, execution shortfall is the increases the shortfall. If the price were falling, the oppor-
execution price minus the prevailing mid-quote at order tunity cost would be negative, reducing the shortfall, and
arrival (strike price) as percent of strike price.2 For sell possibly resulting in negative shortfall. Unlike liquidity

12 SHORTFALL SURPRISES SUMMER 2007


EXHIBIT 2
Execution Shortfall

impact, which is never negative, execution shortfall can 1. Stock capitalization: large-cap, mid-cap, and small-
be negative.3 cap stocks.
In this article, we analyze the shortfall surprise (SS) 2. Listing market: NYSE, AMEX, and NASDAQ
that we define as actual shortfall (SA) minus expected stocks.
shortfall (SE): 3. Order size: the actual dollar value of the order
executed.
SS = SA – SE (1) 4. Volatility over the execution horizon.
5. Quoted spreads over the execution horizon.
To generate the shortfall surprise, we must first esti- 6. Trading volume over the execution horizon.
mate expected shortfall. Expected shortfall is the short-
fall a trader can predict with only pre-trade information: The first three factors (stock capitalization, listing
when the trader receives the order, but before execution. market, and order size) we know with certainty pre-trade.
To estimate expected shortfall, we need estimates for both The other factors (volatility, spread, and volume) we do
liquidity impact (LE) and the expected underlying price not know with certainty pre-trade, and we must use their
move over the execution horizon (EH-price or PE): expected values to estimate liquidity impact (LE):

SE = LE + PE (2) LE = L(Y, XE) > 0 (3)

We estimate liquidity impact using the Goldman where Y are the factors known with certainty pre-
Sachs t-cost model.4 The model uses the following six trade, and XE are the factors not known with certainty
factors to predict liquidity impact: pre-trade.

SUMMER 2007 THE JOURNAL OF TRADING 13


EXHIBIT 3
The Three Components of Execution Shortfall

We next discuss our EH-price concept. In Exhibit 4, trade. Our empirical analysis below confirms this is indeed
we assume the market component of EH-price is zero the case.
and show how we measure the stock-specific alpha move In Exhibit 4, we show how we use the closing
over the execution horizon (EH-alpha). Continuing the price to calculate EH-alpha. We first calculate the alpha-
Exhibit 2 example, the trader received the order at 12:00, to-close. For buy orders, alpha-to-close is the closing
executed 20,000 shares immediately, another 20,000 shares price minus the strike price as percent of the strike
at 13:00, and the final 20,000 shares at 14:00. The exe- price. For sell orders, alpha-to-close is the strike price
cution horizon is two hours (order arrival to last execu- minus the closing price as percent of the strike price.
tion), and the volume-weighted execution turnaround The alpha-to-close in our example is 40 bps. We then
time is one hour. The execution turnaround time is the “allocate” the alpha-to-close to the order in propor-
order’s half-life, taking into account that the order execu- tion to the order’s half-life. We derive the allocation
tion is spread over the two-hour horizon.5 We define EH- factor (Φ) by dividing the order’s half-life (one hour)
alpha as the price move over the order’s half-life, aside by the time from arrival to market close (four hours).6
from the liquidity impact of the trade itself. So, in our example, the allocation factor is 1/4 and EH-
In calculating EH-alpha, it is critical to use a post- alpha is 10 bps.
trade price after the liquidity impact of the trade sub- In practice, the underlying price move has both
sides. In our example, liquidity impact subsides at 15:00, market and alpha components:
so any price after that will work. In our empirical analysis,
we use the closing price to measure the actual EH-alpha EH-price = EH-alpha + βEH-market (4)
for each order in our sample. This procedure assumes
that the closing price is not affected by the impact of the where β is the intra-day stock beta.

14 SHORTFALL SURPRISES SUMMER 2007


EXHIBIT 4
Execution-Horizon Alpha

1
If the time of last execution is after 16:00, it is possible for the ratio to exceed 1. In these cases (129 orders), we set the ratio to 1.

In our empirical analysis, we use S&P 500 ETF We next derive the shortfall surprises factor model
prices to decompose the underlying price move into EH- we use in our analysis. In Equation 5, replacing the
market and EH-alpha. In Exhibit 5, continuing our expected values of the various factors by their post-trade
example, the market-to-close move is 16 bps and allo- actual values, the actual shortfall (SA) is given by:
cating over the execution horizon EH-market is four bps.
Combining Equations (2) to (4) and, for illustration SA = a + bY + cXA + β MA + AA + U (6)
purposes, using a linear version of the liquidity impact
model L (Y, X), expected shortfall is given by: Actual shortfall is also influenced by other factors,
(U), not included in our empirical analysis. Most of these
SE = a + bY + c XE + β ME + AE (5) other factors are random. Some of them, however, may
be systematic but difficult to quantify even post-trade; for
where ME is the expected EH-market and AE is the e.g., trader skill or the presence of natural counterparties.
expected EH-alpha. Subtracting expected shortfall (Equation 5) from
Suppose the liquidity impact is 10 bps. To get the actual shortfall (Equation 6) we get our basic shortfall sur-
total shortfall, we must add the EH-price move. In prises model:
Exhibit 5, the EH-price move is 14 bps so the total
shortfall is 24 bps: 10 bps impact and 14 bps price loss. SS = c (XA – XE) + β (MA – ME) + (AA – AE) + U (7)

SUMMER 2007 THE JOURNAL OF TRADING 15


EXHIBIT 5
The Market and Alpha Components

Or, writing out the factors in full: in the model is one, and the coefficient of the market sur-
prise is β, the intra-day stock beta.
S s = c1 (VOLA – VOLE ) + c 2 (SPREADA – SPREADE ) In deriving our shortfall surprise model (8), we assume
 Volatility surprise  Spread surprise that the liquidity impact model we use to generate expected
+ c 3 (VLM – VLM ) + β(EH-market A – EH-market E )
A E
impact is correctly specified. But the model may be mis-
 Volume surprise  Market surprise specified; we may have omitted systematic variables (U in
A
+ (EH-alpha – EH-alpha ) + U E
(8) our specification), or the model coefficients may be biased.
 Alpha surprise
Model misspecification creates another possible source of
shortfall surprises. In the Appendix, we discuss this issue
In our analysis of the shortfall surprises, we focus on more formally and decompose shortfall surprises into a
the five input surprises: 1) the volatility surprise, 2) the misspecification component and the input surprises com-
spread surprise, 3) the volume surprise, 4) the market sur- ponent. In our empirical analysis, we carefully constructed
prise, and 5) the alpha surprise. The expected EH-market our sample to minimize the risk of model misspecification.
and EH-alpha are hard to estimate and, in our empirical
analysis, we assume they are zero. We describe the sur- SAMPLE CONSTRUCTION
prises in greater detail in the next section. Note that in AND DESCRIPTION
the surprises model, the factors known with certainty
pre-trade (Y) drop out. Also, because of the way we con- Our final estimation sample consists of 241,610 client
structed EH-price, the coefficient of the alpha surprises orders executed by Goldman Sachs over seven months,

16 SHORTFALL SURPRISES SUMMER 2007


June through December 2006. To minimize the risk of For each order in our sample, we use the Goldman Sachs
model misspecification, we restricted our sample to closely t-cost model to estimate the expected liquidity impact.
match the sample we use in estimating the Goldman Sachs The model is re-estimated monthly using the most recent
t-cost model. Our final sample consists of market, not- nine months of data. For each order, we use the most
held orders, executed on an agency basis by the Goldman recent version of the model as of the date of the order to
Sachs U.S. single-stock trading desk.7 We also aggressively generate out-of-sample estimates of that order’s expected
filtered unusual orders; we dropped odd lots, kept only impact. For an order received on July 7, for example, we
orders in common stocks, dropped ADRs and ETFs, use the model estimated with data prior to July 7. The
OTCBB and Pink Sheets, dropped crosses, dropped orders monthly re-estimation of the model ensures the coeffi-
in extreme price stocks (less than $1 or more than $150), cients are not stale and again reduces the risk of model
dropped tick sensitive orders (e.g., sell short), etc. Finally, misspecification.
we dropped orders with data errors. The Goldman Sachs t-cost model gives different
Exhibit 6 summarizes the order composition of the impact estimates depending on the execution horizon
final sample. The 241,610 orders in our sample are evenly specified. In our analysis, we estimate the expected impact
divided between buys and sells. One quarter of the orders using the order’s actual execution horizon: order arrival
are NASDAQ stocks8 and the balance are NYSE stocks.9
Of the total orders, 64% are in large-cap stocks and 9%
in small-cap stocks. Only 296 orders exceed 25% of
average daily volume, and 89% of the orders are less than
10,000 shares. The majority of the orders (92%) have an
EXHIBIT 6
execution half-life less than 15 minutes; 13,464 orders Order Types
have a half-life more than 30 minutes. The average stock
price in our sample is $34.
Exhibit 7 summarizes order characteristics for the
overall sample. The average order size executed is 9,420
shares, ranging from 18 shares to 29 million shares. The
value-weighted average order size executed is 5% of ADV
and the participation rate is 21%.10 The value-weighted
half-time is 65 minutes, but the median is only 15 sec-
onds and ranges from instantaneous executions to all-day
executions. The value-weighted average actual shortfall
is 23 bps, but the median is only 2 bps. Actual shortfall
ranges widely from –757 bps to +1,069 bps.
Exhibit 8 confirms that, at least on average, the
closing prices we use in calculating the EH-price move are
not affected by liquidity impact. In Exhibit 8, we plot the
average shortfall, same-day closing price and the closing
prices over the next five days for all the orders in our
sample. We measure all prices relative to the order-arrival
price, and include both buy and sell orders, but flip the sign
of the sells. We use value-weighted averages, so large trades
get more weight in the average. The average closing price
is 26 bps higher than the strike price, but there is little
reversal over the next five days. The absence of reversal
reassures us that the closing price is not affected by the
temporary liquidity impact of the trades themselves.
We next describe how we construct the shortfall
surprises and the five factor surprises we use in our analysis. Sample period June 1 through December 31, 2006.

SUMMER 2007 THE JOURNAL OF TRADING 17


EXHIBIT 7
Overall Sample Summary Statistics1

1
Sample period June 1 through December 31, 2006; (241,610 orders).
2
Simple mean.
3
Value-weighted mean.

to last execution. To generate our impact estimates, we to the order’s execution horizon, by using the same
must also specify pre-trade estimates for volatility, quoted allocation factor Φ that we use in calculating EH-
spreads and trading volume: alpha (Exhibit 4).
• Quoted spread is the time-weighted spread over the
• Volatility is measured as the percent difference execution horizon, taking into account the spread
between the intra-day high and low price, adjusted smile (the U-shaped intraday spread pattern).11

EXHIBIT 8
Closing Prices are Not Affected by Liquidity Impact

Sample period June 1 through December 31, 2006; 241,610 buy and sell orders (Sign of sell orders flipped). Value-weighted averages.

18 SHORTFALL SURPRISES SUMMER 2007


• Trading volume is measured over the execution Exhibit 9 shows a scatter plot of actual shortfall (ver-
horizon, taking into account the volume smile (the tical axis) against expected shortfall (horizontal axis).
U-shaped intraday volume pattern). Actual shortfall equals expected shortfall along the
45 degree line. Orders with actual shortfall exceeding
We generate pre-trade estimates for these three expected shortfall are above the 45 degree line and orders
factors using their median value over the prior 21 with actual shortfall less than expected shortfall are below.
trading days. Along the vertical, we see the large range in actual short-
To estimate expected shortfall (Equation 5), we also fall from a minimum of –757 bps to a maximum of +1,069
need pre-trade estimates for EH-alpha and EH-market. bps. Along the horizontal, we see the range in expected
Unlike liquidity impact, where reliable pre-trade esti- shortfall estimates from a minimum of 0.3 bps to a max-
mates are widely available, pre-trade estimates of EH- imum of 334 bps. Because we assume expected EH-
market and EH-alpha are difficult to find. As is typically market and EH-alpha are zero, the expected shortfall is
the case in practice, in our analysis we assume that pre- always positive (just the liquidity impact).
trade, the expected EH-market and expected EH-alpha The number in each bubble indicates the number
are both zero. of orders clustering in close proximity to that point. About

EXHIBIT 9
Scatter plot of Actual Vs Expected Shortfall

0 50 100 150 200 250 300

Sample period June 1 through December 31, 2006; (241,610 orders).

SUMMER 2007 THE JOURNAL OF TRADING 19


210,000 orders are clustered between –60 and +35 bps of values and report the unexplained variation (one minus
actual shortfall and between 0 and 15 bps of expected R-square). The unexplained variation tells us how much
shortfall. The remaining orders show a large dispersion, of the actual variation in each variable is not explained by
both in the expected and actual shortfall. Our objective the expected value of that variable.
in this article is to better understand this dispersion: the Exhibit 11 shows that 80% of the actual shortfall
shortfall surprise. variation is not explained by expected shortfall (or con-
Exhibit 10 summarizes the post-trade actual values versely, that the pre-trade estimates explain only 20% of
of the five factors we use to explain the shortfall surprise. the post-trade variation in actual shortfall). Expected
All five factors range widely, mirroring the large range of volatility leaves only 23% of the variation in actual
actual shortfall. Volatility, for example, ranges from 0% to volatility unexplained. Expected spreads and volume
20% and quoted spreads from 0 bps to 882 bps. The actual leave about 50% of the post-trade variation unexplained.
EH-market move ranges from –119 bps to +121 bps, but Since we assume they are zero, expected EH-market and
averages to zero by all measures (median, simple, and EH-alpha leave 100% of the variation in the actual EH-
weighted means). The actual EH-alpha, however, has a alpha and EH-move unexplained. The EH-market and
much bigger range (from –557 bps to +836 bps) and the AH-alpha surprises, therefore, have great power to
value-weighted mean is 10 bps. potentially explain the shortfall surprise. Exhibit 11 also
In our shortfall surprises model (Equation 8), the shows the large range of all the surprises, another mea-
shortfall surprise depends on the five factor surprises. sure of forecasting accuracy. In general, all five factors
Exhibit 11 summarizes our estimates of the shortfall sur- have large surprises and, therefore, potentially large
prise and the five factor surprises. The first column shows explanatory power in our shortfall surprise regressions,
the forecasting accuracy of our pre-trade measures and gives to which we now turn.
an indication of the potential explanatory power of each
surprise. If, for example, a factor is perfectly forecasted, THE EMPIRICAL FINDINGS
then there is no factor surprise and that factor cannot explain
the shortfall surprise. We measured forecasting accuracy by Our preferred regression specification for the short-
estimating univariate regressions of actual values on expected fall surprise (SS) model is:

EXHIBIT 10
The Five Factors: Actual Values1

1
Sample period June 1 through December 31, 2006; (241,610 orders).
2
Simple mean.
3
Value-weighted mean.

20 SHORTFALL SURPRISES SUMMER 2007


EXHIBIT 11
The Five Factor Surprises: Actual Minus Expected Values1

1
Sample period June 1 through December 31, 2006; 241,610 orders.
2
One minus the R-square from univariate regression of actual to expected.

S s = c0 + c1(VOLA – VOLE ) basis-point increase in the actual shortfall. The coefficient


 Volatility surprise of EH-market in our regression is an estimate of the short-
⎛ VLM A ⎞ term intra-day beta on our sample; the results in Exhibit
+ c 2 (SPREADA – SPREADE )+ c 3 ⎜ log ⎟ 12 suggest that this is also close to one.13
 Spread surprise ⎝ VLM E ⎠
 Volume surprise To better understand the relative importance of the
A E five factor surprises in explaining the shortfall surprise,
+ c4 (EH-market – EH-market )
 Market surprise we also ran a series of univariate regressions of the short-
+ c5 (EH-alpha A – EH-alpha E ) fall surprise on each of the factor surprises in turn. In
 Alpha surprise Exhibit 13, columns four to eight, we report the R-squares
from these univariate regressions. The first row shows the
We experimented with many alternative specifica- results of running the regressions on all the orders in our
tions (for e.g., with and without intercepts, different func- sample. In the univariate regressions, the EH-alpha sur-
tional forms for the five factors, etc.) and, in all cases, the prise is by far the most important factor explaining almost
results are similar.12 40% of the shortfall surprise variation. The other four
Exhibit 12 shows the results from the estimation of factors have little explanatory power; EH-market explains
our preferred specification over our whole sample of 5.8%, volatility explains 3.0%, spreads explain 1.5% and,
241,610 orders. The overall fit of the regression is 44% most surprising, volume only explains 0.2% of the short-
(R-square), and the coefficients of all five factor surprises fall surprise.
are significant and have the right sign. Higher-than- Exhibit 14 shows that the covariances across the
expected volatility and spreads lead to higher-than- five factor surprises, while small, are not always zero.
expected shortfall while higher-than-expected volume The correlation between the EH-alpha and volatility
leads to lower-than-expected shortfall. Similarly, higher- surprises, for example, is 25% and the correlation
than-expected market and alpha moves result in higher- between the volume and volatility surprises is 17%.
than-expected shortfall. Our hypothesis is that the Because the covariances across the factor surprises are not
coefficient of EH-alpha in the regression should be one, zero, the univariate regression results can be misleading.
and Exhibit 12 shows that this is indeed the case; a five To double check, we also performed an incremental
basis-point increase in EH-alpha will lead to a five analysis, where we evaluate the contribution of each

SUMMER 2007 THE JOURNAL OF TRADING 21


EXHIBIT 12
The Shortfall Surprise Multivariate Regression1

1
Sample period June 1 through December 31, 2006; t-statistics in brackets below the coefficients.

factor surprise by dropping it from the full multivariate In Exhibit 13, rows two to four, we divide our
regression. The overview results in Exhibit 1 are from sample by stock capitalization: orders in large-cap stocks
the incremental approach and are similar to the uni- (>$7.5 billion), mid-cap stocks, and small-cap stocks
variate regression results. (<$1 billion). As we move from large-cap to small-cap
The last three columns in Exhibit 13 show another stocks, the accuracy of the expected shortfall forecasts as
example of the incremental approach. Column 10 shows measured by the root mean-square error (RMSE) falls.14
the full five-factor regression, and the regression in column The expected shortfall RMSE is 13 bps for orders in large-
nine drops the two EH-price surprises (market and alpha). cap stocks, but 49 bps for orders in small-cap stocks, indi-
Dropping the EH-price surprises reduces the explanatory cating that the shortfall surprise is higher in small-cap
power of the regressions from 44% to only 4.5%, con- stocks.
firming that the price surprises account for almost all of Our findings on the five factor surprises are similar
the explanatory power in our shortfall surprise regressions. across stock capitalization buckets. In all three buckets,

EXHIBIT 13
Shortfall Surprise Attribution: An Overview1

1
Sample period June 1 through December 31, 2006.
2
Root mean-square error: the square root of 1/N Σ(SAi – SEi)2, where SAi is the post-trade actual shortfall and SEi is the pre-trade expected shortfall for each order.
3
The average executed order size in this bucket is 775 shares.

22 SHORTFALL SURPRISES SUMMER 2007


EXHIBIT 14
Covariance of the Five Surprises1

1
Sample period June 1 through December 31, 2006; (241,610 orders).
2
Log of ratio of actual to expected volume.

the EH-price surprise explains most of the shortfall sur- the much smaller shortfall surprise. But still, across the
prise. The volatility and spread surprises explain more of five factor surprises, the EH-price surprise is by far the
the shortfall surprise in small-cap stocks than in the other most important factor in explaining the shortfall surprise:
two buckets, but the numbers are still small. In the uni- volatility, spread, and volume surprises explain 2.3%, while
variate regressions, for example, the spread surprise EH-price explains 14.7%.
explains 4.3% of the shortfall surprise in small-cap stocks, In Exhibit 15, we focus on the time dimension of
but only 0.1% of the surprise in large-cap stocks. The order executions. A quick execution exposes the order
volume surprise explains very little of the shortfall surprise to little EH-price movement, so we expect the EH-price
across all stock capitalizations. surprise to explain less of the shortfall surprise. At the
Because we construct EH-price using closing prices, other end, an order that takes several hours to execute is
one concern is that the closing prices are affected by liq- exposed to potentially large price moves, so we expect
uidity impact and therefore are not a pure measure of the EH-price to be much more important in explaining the
underlying price move. In Exhibit 9, we showed that, at shortfall surprise. The results in Exhibit 15 confirm this
least on average, the closing prices in our sample are not hypothesis.
affected by liquidity impact. The liquidity impact of the Exhibit 15 is divided in four panels: Panel A covers
larger trades in our sample is more likely to affect the orders in all stocks, Panel B focuses on orders in large-cap
closing price.15 So, in Exhibit 13, rows five to six, we fur- stocks, Panel C focuses on mid-cap stocks and Panel D
ther test the sensitivity of our results to the quality of the focuses on small-cap stocks. The top row in each panel
closing price by dropping the larger trades in our sample. replicates the information in Exhibit 13. The next two
In row five, we drop all orders greater than 25% of ADV rows in each panel divide the orders into orders with half-
(297 orders) and in row six, we drop all orders greater life more than 30 minutes and orders with half-life less than
than 15% of ADV (1,228 orders). In both cases, dropping 30 minutes. The last two rows in each panel repeat this
the outlier large orders does not affect our results. split, using a 60-minute cut-off.
In the last row in Exhibit 13, we focus on the small The first thing to notice in Exhibit 15 is that the
orders in our sample: 177,191 orders less than 0.1% ADV forecasting accuracy of expected shortfall falls sharply as
(average order size 775 shares). As expected, for these the execution horizon increases (column 4). In Panel A,
small orders, the shortfall forecasting error is low: the for example, the RMSE for orders with half-life more
RMSE is only 7 bps compared to 23 bps overall. For these than 60 minutes is 89 bps compared to only 15 bps for
small orders, the five factor surprises explain only 17% of orders with half-life less than 30 minutes. For orders in

SUMMER 2007 THE JOURNAL OF TRADING 23


EXHIBIT 15
The Shortfall Surprise Attribution: Details1

1
Sample period June 1 through December 31, 2006.

small-cap stocks (Panel D), the RMSE for orders with Panel A, for example, EH-price explains 63.5% of the
half-life more than 60 minutes is even higher: 118 bps. shortfall surprise in orders with half-life more than 60
The forecasting accuracy results in Exhibits 13 and 15 minutes, but only 6.3% of the surprise for orders with
strikingly quantify what most traders know from experi- half-life less than 30 minutes. For orders in large-cap stocks
ence: most of the shortfall surprise comes from large orders with half-life more than 60 minutes, EH-price explains
that take time to execute. a remarkable 73% of the shortfall surprise.
In Exhibit 15, column 11 shows the explanatory Looking at the two components of EH-price
power of the full five-factor regression model. Across the (market and alpha), the results in Exhibit 15 across all
various buckets, the R-square of the full model ranges buckets are consistent with the results in Exhibit 13: EH-
from 75.3% (orders >60 minutes, large-cap stocks) to alpha is by far the most important factor in explaining
8.0% (orders <30 minutes, large-cap stocks). We next the shortfall surprise. For orders in large-cap stocks with
examine the contribution of the five-factor surprises in half-life more than 60 minutes, for example, EH-alpha
explaining the shortfall surprise. The last column in explains 65.3% of the shortfall surprise while EH-market
Exhibit 15 shows that, consistent with our hypothesis, explains only 12.5%. For orders with half-life less than
the EH-price surprise is much more important in 30 minutes, EH-alpha explains 6.1% of the shortfall sur-
explaining the shortfall surprise for slow executions. In prise while EH-market explains only 0.8%.

24 SHORTFALL SURPRISES SUMMER 2007


The summary statistics on EH-alpha and EH-market simple correlation between the volume and shortfall sur-
in Exhibit 10 provide a clue why the alpha component prises is small, but positive, 4%. The striking difference
dominates: EH-alpha has a bigger range (–557 bps to between the multivariate and univariate results suggests that
+839 bps) than EH-price (–119 bps to +121 bps). Out other factor surprises correlated the volume surprise (for
of the 241,610 orders in our sample, 3,173 orders are e.g., volatility and EH-alpha in Exhibit 14), and increase
associated with an EH-alpha move exceeding 50 bps (in costs outweighing the beneficial effects of higher-than-
absolute terms), but only 407 orders are associated with expected volume.
an EH-market move exceeding 50 bps. We performed extensive additional robustness tests
Exhibit 15, column 10, confirms the robustness of of our finding on the volume surprise. In all cases, volume
our findings on the volatility, spread, and volume sur- surprises explain very little of the shortfall surprise. In
prises: consistently small explanatory power across all Exhibit 17, for example, we focused on high-volume sur-
buckets. Finally, Exhibit 15, column three, confirms the prise events: 20,116 orders in our sample associated with
robustness of the absence of any volume effect. Across all a volume surprise greater than one standard deviation.
buckets, the volume surprise explains less than 1% of the Again, there is little change: volume surprises explain only
shortfall surprise. 1.4% of the shortfall surprise and the EH-price move
Perhaps the most striking observation about explains 60.2%.
Exhibits 13 and 15 is the robustness of our main results. The important implication of our volume findings
Whichever way we cut the data, EH-alpha is by far the is that traders and algorithms cannot naively exploit
most important factor in explaining the shortfall surprise; volume surprises to reduce trading costs. To beneficially
while volatility, spread, and volume surprises explain very exploit a volume surprise, execution strategies must dis-
little. The volume results are particularly surprising and, tinguish between liquidity supply and liquidity demand
in the next section, we take a closer look. surprises, which is hard to do. Taking advantage of
higher-than-expected volume has one unambiguous
THE VOLUME SURPRISE beneficial effect: traders fill orders faster reducing exe-
cution risk.
Our most puzzling finding is that volume surprises
do not affect trading costs. This seems counterintuitive: THE VOLATILITY SURPRISE
more volume than anticipated should surely reduce trading
costs. This intuition, however, rests on the presumption Another empirical finding worth highlighting is the
that higher volume is caused by an increase in the supply sharp distinction between the small effect on trading costs
of liquidity (Case A in Exhibit 16). Our empirical find- of volatility surprises and the large effect of EH-price sur-
ings suggest that an increase in volume is caused by an prises. The important distinction here is that volatility
increase in both liquidity supply and demand (Case C in surprises are unsigned price moves while EH-price sur-
Exhibit 19): a hundred more traders supply liquidity, but prises are signed price moves, depending on the direction
another hundred traders demand more liquidity. An of the trade.16 For buy orders, EH-price is closing price
increase in uncertainty, for example, will lead to an increase minus strike price, while for sell orders, EH-alpha is strike
in both liquidity supply and demand. Similarly, the pop- price minus closing price (see Exhibit 18).
ular “participate” execution strategies and algorithms The unsigned EH-price is just another measure of
automatically generate liquidity demand in proportion to execution-horizon volatility: closing price minus strike
increases in liquidity supply. price (as percent of strike price) for both buys and sells.
In our multivariate regressions where we control Let’s call this measure of volatility EH-volatility. In
for the variation in all five factors, all else being equal, Exhibit 18, we compare the results of regressing the short-
the volume surprise reduces trading costs, but by a tiny fall surprise on the signed and unsigned EH-price. In
amount; for e.g., volume twice the expected amount the univariate regression of shortfall surprises on signed
reduces trading costs by less than 0.3 bps. In the uni- EH-price, EH-price explains more than 40% of the vari-
variate regressions of shortfall surprise on volume sur- ation in the shortfall surprise. However, in the univariate
prise, a surprise increase in volume actually increases regression of the shortfall surprise on EH-volatility (the
trading costs (Case B in Exhibit 16). Equivalently, the unsigned EH-price), the unsigned EH-price explains less

SUMMER 2007 THE JOURNAL OF TRADING 25


EXHIBIT 16
The Volume Surprise: Why It Does Not Matter?

EXHIBIT 17
High Volume Surprises1

1
Sample period June 1 through December 31, 2006.
2
|VLMA – VLME| > one standard deviation.

26 SHORTFALL SURPRISES SUMMER 2007


EXHIBIT 18
The Volatility Surprise

1
Sample period June 1 through December 31, 2006.

than one percent of the variation in the shortfall Post-trade EQA


surprise. We find similar results in a multivariate regres-
sion of the shortfall surprise on both the signed and Our finding that price surprises are the main cause
unsigned EH-price: all the explanatory power comes of shortfall surprises means that, in interpreting post-trade
from the signed price move. EQA, it is essential to control for the underlying execu-
What is striking about the analysis in Exhibit 18 is tion-horizon price move. Our preferred way of doing
that, by design, EH-price and EH-volatility are exactly this is by reporting the EH-price move alongside the
the same measure, except that one is signed based on the actual shortfall, or equivalently, by using the EH-price to
direction of the trade and the other is not. The impor- split actual shortfall into liquidity impact and price loss.18
tant implication of our findings in this section is that Suppose actual shortfall is 20 bps and the EH-price is 18
traders and algorithms cannot exploit volatility surprises bps: the liquidity impact is only 2 bps, so this is good exe-
to reduce trading costs unless they can accurately predict cution. If the EH-price is 4 bps, however, 20 bps is poor
the direction of the price move.17 execution (16 bps liquidity impact). We avoid using the
usual VWAP benchmarks to control for price surprises,
because VWAP benchmarks may be affected by the impact
IMPLICATIONS AND CONCLUSION of the trade itself and may give misleading results.19
The importance of price surprises also suggests that
Our empirical findings have important implications post-trade EQA to evaluate execution strategies and algo-
for post-trade execution quality analysis (EQA) and for rithms can only be meaningfully done with a large sample
pre-trade tool, for the development of algorithms, and of orders. In small samples, extraneous price surprises will
for the choice of execution strategy. We conclude with a dominate the statistics and obscure the true performance
discussion of these implications. being evaluated.

SUMMER 2007 THE JOURNAL OF TRADING 27


Pre-trade Tools and Development than a few minutes to complete, by far the most impor-
of Algorithms tant factor is the expected underlying price move (EH-
alpha). But is there any empirical evidence that traders
Trading cost models are widely used to generate correctly anticipate EH-alpha and adapt their execution
pre-trade cost estimates and choose the right execution strategies accordingly? In Rakhlin and Sofianos [2006a],
strategy. Cost models are also embedded in various algo- we found no evidence that buy-side traders allocated
rithms.20 These trading-cost models have relatively low orders across passive VWAP algorithms and more aggres-
forecasting power and model developers are trying hard sive shortfall algorithms based on expected EH-alpha.
to improve them. Our findings suggest that the only way But that article was only the beginning of an ongoing
to make a significant difference in forecasting accuracy is research effort.
by coming up with better forecasts of EH-price. Better If traders cannot correctly anticipate EH-alpha, then
forecasts of volume and volatility, in particular, will not execution strategies should be modified to reflect this fact.
improve the forecasting accuracy of trading cost models. Segmenting orders between passive and aggressive exe-
But how can we come up with better EH-price cutions, for example, requires reliable EH-alpha forecasts.
forecasts? One way is to continue improving models for In the absence of reliable EH-alpha forecasts, adopting a
short-term price forecasts. Work is being done on this front single, relatively passive, execution strategy may work best.
in the context of short-term quantitative trading. The The downside of a passive strategy is high execution risk
challenge here is that a better short-term price forecasting (a large variation in actual shortfall). If execution risk is
model is more valuable as a proprietary trading tool, rather a concern, then an alternative strategy is to let the trader’s
than a widely available pre-trade tool. But the synergies risk aversion drive the choice of execution strategy, and
exist and the possibility of combining short-term trading accept a higher liquidity impact. Yet another alternative
and improved execution strategies within a buy-side is to request capital and pay a premium to completely
trading desk is intriguing. eliminate execution risk.
Another way to improve the accuracy of shortfall The shortfall surprises framework that we devel-
estimates is by allowing users to adjust the estimates based oped in this article can be used more generally to quan-
on the user’s view of the expected EH-price. Sell-side tify and compare the value-added of alternative execution
traders may have a good “feel for the market,” so allowing strategies. By analyzing shortfall surprises, instead of actual
them to adjust cost estimates accordingly can be useful. shortfall, we control for the usual factors affecting exe-
Similarly, buy-side traders may have a better understanding cution quality and can focus on the incremental value of
of portfolio managers’ intentions and alpha signal. The additional factors. Consider again the comparison between
Goldman Sachs PortX algorithm, for example, allows passive VWAP and aggressive shortfall algorithms. VWAP
traders to input their alpha-to-close expectation and uses algorithms should outperform shortfall algorithms when
this input to adjust the pre-trade cost estimates, and the EH-price is low and the reverse when EH-price is high.
optimum execution strategy, accordingly. Using our shortfall surprises framework, for VWAP exe-
Our findings also have important implications for the cutions, the shortfall surprise should be negative (actual
development of dynamic algorithms that react real-time less than expected) when EH-price is low and positive
to market changes. For a dynamic algorithm to usefully when EH-price is high and the reverse for the shortfall
react to a volume surprises, it must be able to distinguish executions. In doing this comparison, we do not have to
between supply and demand causes of the surprise, which control for order size, stock capitalization, or the other
is very challenging. Similarly, dynamic algorithms cannot “usual suspects,” because expected shortfall already reflects
usefully exploit volatility surprises, unless they can pre- these factors.
dict the direction of the market. We can also use the shortfall surprises framework
to quantify the value-added of high-touch executions.
Choice of Execution Strategy The rise of low-touch trading has created a pressing ques-
tion: why pay more than 12 bps in commissions to use
Our findings in this article strikingly quantify one the high-touch if one can execute for less than 3 bps in
of the main themes in our research over the past five commissions low-touch? The answer, of course, is that for
years:21 in choosing an execution strategy that takes more difficult orders, high-touch executions may add value by

28 SHORTFALL SURPRISES SUMMER 2007


reducing the indirect costs of trading: liquidity impact and SE = a + b Y + c XE + PE
opportunity cost. But to decide between high-touch and
low-touch, we must explicitly quantify high-touch value- The other using the actual values of the X and P factors
added, and determine the order difficulty cut-off, above (perfect-foresight estimates):
which it is worth executing high-touch. In future work,
we plan to use our shortfall surprises framework to address SEA = a + b Y + c XA + PA
these questions. The shortfall surprise is SS = SA – SE, and by adding and
subtracting SEA, we decompose SS into two components, SS1
and SS2:
APPENDIX
MODEL MISSPECIFICATION SS = ( SA – SEA) – (SEA – SE) = SS1 – SS2
AND THE SHORTFALL SURPRISE
But
In this Appendix, we formally decompose the shortfall sur-
SS1 = SA – SEA = (α + β Y + γ XA + δ Z + PA + u )
prise into the surprise caused by model misspecification (e.g.,
omitted factors), and the component caused by the surprise in the – (a + b Y + c XA + PA)
included factors. Suppose the true model for actual shortfall is:
Re-arranging:
SA = α + β Y + γ XA + δ Z + PA + u
SS1 = (α – a) + (β - b) Y + (γ – c) XA + δ Z + u
where Y = factors known with certainty pre-trade, XA = If the impact model is not misspecified a = α, b = β, c = γ, δ =
actual values of factors not known with certainty pre-trade, 0, and SS1 = u, a random variable.
Z = factors not included in the liquidity impact model, PA = The shortfall surprise component SS1, therefore, mea-
actual execution-horizon price move, and u is a random sures the extent the shortfall surprise SS is caused by other fac-
variable. tors (model misspecification). Similarly:
Suppose the estimated liquidity impact model is:
SS2 = SEA – SE = c (XA – XE) + (PA – PE)
L=a+bY+cX
The shortfall surprise component, SS2, therefore, mea-
Post-trade, we can generate two estimates from the impact sures the extent the shortfall surprise SS is caused by the factor
model. One using the expected values of the X and P factors: surprises (what we are testing in this article).

EXHIBIT A1
Further Disaggregating the Shortfall Surprise1

1
Sample period June 1 through December 31, 2006; 241,610 orders.

SUMMER 2007 THE JOURNAL OF TRADING 29


Using the Goldman Sachs t-cost model and the data in ticipation, etc. Moving expected shortfall to the right hand side
our sample, we estimated both SE and the perfect foresight of the regressions also does not make much difference; it has a
shortfall SEA. We then decomposed the shortfall surprise into coefficient of one.
the other-factors component (SS1), and the factor-surprise com- 13
These results on the coefficients of EH-market and EH-
ponent (SS2), and estimated the shortfall surprise model only on alpha are robust across all specifications and sub-samples that we
the SS2 component. estimated.
Exhibit A1 shows our empirical results using this decom- 14
We define the RMSE as the square root of 1/N ∑(SAi –
position. Our estimated other-factors component SS1 accounts SEi)2, where SA is the post-trade actual shortfall and SE is the pre-
for 56% of the shortfall surprise, exactly matching the “unex- trade expected shortfall for each order.
15
plained” component in our one-step procedure in Exhibit 1. At the same time, however, the larger the trade, the
In the regression of the factor-surprises component SS2, on the higher the likelihood that the trade was generated by a strong
five factors, we get an R-square of 93%, and the overall alpha signal (the true EH-alpha is higher).
16
explanatory power of each of the five surprises is similar to For further discussion of this important point, see
the results from our one-step procedure (Exhibit 1). For Rakhlin and Sofianos [2006], pp. 43–45.
17
example, the alpha surprise explains 80% of the variation in In this article, we focus on agency executions. Volatility
SS2, which itself accounts for 44% of the overall shortfall-sur- surprises may be more important for capital commitment
prise variation; so in the two-step approach, the alpha sur- (principal) trades. In a capital commitment trade, the dealer
prise explains 35% of the overall shortfall compared to 38% prices volatility (execution risk) in the price premium/dis-
in the one-step procedure. count. So an increase in volatility will make principal trades
more expensive.
18
ENDNOTES For examples of our use of the decomposition of short-
fall into liquidity impact and price loss, see Cai and Sofianos
The authors thank Jeff Bacidore, Tianwu (Michael) Cai, [2006] and Rakhlin and Sofianos [2006].
19
Barbara Dunn, Oliver Hansch, Kilian Mie, and Ingrid Tierens, For further discussion of this point, see Sofianos [2006].
20
all at Goldman, Sachs & Co., for their comments. For example, the Goldman Sachs single-stock 4Cast
1
See Sofianos [2006] for a discussion of execution shortfall. and portfolio PortX algorithms. See Rakhlin and Sofianos
2
Orders that arrive before the market opens (9:30 am), [2006a, 2006b] for descriptions of these two algorithms.
21
we strike at the opening price. Bacidore and Sofianos [2002, 2003] through Cai and
3
Liquidity impact will never be negative for liquidity- Sofianos [2006].
seeking orders. Liquidity-providing orders, for e.g., limit orders,
conditional on execution, may have a negative impact (save the REFERENCES
spread).
4
For more details on the model, see Rodella [2005]. Bacidore, Jeffrey and George Sofianos. “Evaluating Execution
5
Only if the trader had executed all 60,000 shares at 14:00 Quality for Large Orders.” Goldman Sachs Trading and Market
would the execution horizon and the execution turnaround Structure Report (2002).
time be the same (two hours).
6
This allocation procedure assumes the alpha is growing ——. “Choosing the Best Order Execution Strategy.” Goldman
linearly from order arrival to market close. Sachs Trading and Market Structure Report (2003).
7
We exclude limit orders, held orders, and any orders where
Goldman Sachs provided capital at the request of the client. Cai, Tianwu (Michael) and George Sofianos. “Multi-day Exe-
8
NASDAQ GS, NMS, and Small Cap. cutions.” Journal of Trading, Vol. 1, No. 3 (2006), pp. 25–33.
9
The sample also includes a few AMEX stocks.
10
We define participation rate as the executed quantity as Rakhlin, Dmitry and George Sofianos. “The Choice of Exe-
a percent of the actual overall market trading volume in that cution Algorithm: VWAP or Shortfall.” Journal of Trading,
stock from strike time (order arrival time) to the time of the Vol. 1, No. 1 (2006a), pp. 26–32.
last execution.
11
We use only firm and valid NBBO quotes in our ——. “The Impact of an Increase in Volatility on Trading
calculation. Costs.” Journal of Trading, Vol. 1, No. 2 (2006b), pp. 43–50.
12
We tried, for example, several different ways of speci-
fying the volume surprise: actual minus expected volume, the Rodella, Elena. “Introducing Cost Wizard: Comparing Two-
log of actual divided by expected (the specification we use in Hour and All-Day Executions.” Goldman Sachs Cost Wizard
our analysis), the actual participation minus the expected par- Reports, Issue 1 (May 12, 2005).

30 SHORTFALL SURPRISES SUMMER 2007


Sofianos, George. “Execution Benchmarks: VWAP or Pre- algorithmic models derive pricing and trading estimates based on historical
trade Prices.” Journal of Trading, Vol. 1, No. 1 (2006), pp. 22–25. volume patterns, real-time market data, and parameters selected by the GSAT
user. The ability of Goldman Sachs’ algorithmic models to achieve the per-
formance described in this article may be impacted by changes in market con-
ditions, systems or communications failures, etc. Finally, factors such as order
To order reprints of this article, please contact Dewey Palmieri at quantity, liquidity, and the parameters selected by the GSAT user, may impact
dpalmieri@iijournals.com or 212-224-3675 the performance results.
The opinions expressed in this article are those of the authors and do
not necessarily represent the views of Goldman, Sachs & Co. These opinions
represent the authors’ judgment at this date and are subject to change.
This material was prepared by the Goldman Sachs Equity Execution Goldman, Sachs & Co. is not soliciting any action based on this article.
Strategies Group and is not the product of Goldman Sachs Global Investment It is for general information and does not constitute a personal recommenda-
Research. It is not a research report and should not be construed as such. tion or take into account the particular investment objectives, financial situa-
The information in this article has been taken from trade data and tions, or needs, of individual users. Before acting on any advice or
other sources we deem reliable, but we do not represent that such informa- recommendation in this article, users should consider whether it is suitable for
tion is accurate or complete, and it should not be relied upon as such. This their particular circumstances.
information is indicative, based on among other things, market conditions at Copyright: Summer 2007, Goldman, Sachs & Co.
the time of writing, and is subject to change without notice. Goldman Sachs’

SUMMER 2007 THE JOURNAL OF TRADING 31