1 Votos favoráveis0 Votos desfavoráveis

187 visualizações21 páginasOct 08, 2010

© Attribution Non-Commercial (BY-NC)

PDF, TXT ou leia online no Scribd

Attribution Non-Commercial (BY-NC)

187 visualizações

Attribution Non-Commercial (BY-NC)

- A Propagation Model for Mobile Radio Propagation Loss in an Urban Area at 800 MHz
- Statistics 578 Assignment 5 Homework
- Correlation and Regression
- Acc Hedge Effectiveness AFP Regression
- Effects of School-Based Quality Factors on Secondary School Students Achievement in English Language in South-Western and North-Central Nigeria.pdf
- Assessing risk—what your mutual fund house does
- Report 12242244 Sds d
- Analysis of Willingness and Ability to Pay (WTP and ATP) On Waste Management in the City Ternate
- MBA Assignment- First Semester
- MA Statistics Tutorial
- stach14
- BI White Paper Healthcare Analytics Practical Predictive Analytics 101 May 2013
- converted.phaipt
- 15 Regression Basics
- 58.Full
- White
- artigo 5
- ANZMAC2009-294
- Ishan Gupta SPR
- Acc UralsAccurals.DOCX

Você está na página 1de 21

KWAKU ABROKWAH n this article, we use a large sample of volume surprise explains very little of

is an analyst, Equity

Execution Strategies, at

Goldman, Sachs & Co. in

New York, NY.

kwaku.abrokwah@gs.com

I order executions to better understand

“shortfall surprises”: the difference

between actual and expected trading

costs. Our trading cost measure is execution

shortfall, including both liquidity impact and

the shortfall surprise (only 0.3%), indi-

cating that higher-than-expected volume

does not reduce trading costs.

• Volatility surprise. Higher-than-expected

volatility over the execution horizon

GEORGE SOFIANOS the opportunity cost of slow executions.1 should result in higher-than-expected

is a vice president, Equity Exhibit 1 summarizes our main findings. In shortfall. We find that the volatility

Execution Strategies, at

Goldman, Sachs & Co. in

our sample, on average, only 20% of actual surprise explains only 0.1% of the short-

New York, NY. shortfall is predictable pre-trade. Focusing on fall surprise.

george.sofianos@gs.com the non-predictable component, we investi- • Spread surprise. Higher-than-expected

gate the following possible reasons for the quoted spreads over the execution horizon

shortfall surprise: should result in higher-than-expected

shortfall. We find that the spread surprise

• Price surprise. A higher-than-expected explains 1.8% of the shortfall surprise.

price increase over the execution horizon

should result in a higher-than-expected Price surprises, therefore, are by far the

actual shortfall on buy orders and lower- most important factor explaining shortfall

than-expected shortfall on sell orders. surprises; the volume, volatility, and spread

We find that, on average, the price sur- surprises have little explanatory power. These

prise explains 42% of the shortfall sur- findings have important implications for post-

prise. For orders in large-cap stocks that trade analysis, pre-trade tools, the development

take more than an hour to execute, the of algorithms, and the choice of execution

price surprise explains 73% of the short- strategies.

fall surprise. Splitting the price surprise In interpreting post-trade execution

into its alpha and market components, quality, for example, we must somehow con-

the alpha surprise explains 38% of the trol for the underlying execution-horizon price

shortfall surprise, and the market sur- move. For pre-trade analysis, our findings sug-

prise only 4%. gest that the only way to improve the pre-trade

• Volume surprise. Higher-than-expected t-cost estimates is to better predict the alpha-

volume over the execution horizon move over the execution horizon. Better pre-

should result in lower-than-expected trade volatility and volume estimates will not

shortfall. Surprisingly, we find that the help much. The same applies for algorithm

EXHIBIT 1

Summary of Main Findings1

1

Sample period June 1 through December 31, 2006; 241,610 orders.

2

R-square from univariate regression of actual on expected shortfall.

3

Incremental R-square from multivariate regression analysis of shortfall surprise on the five factor surprises: volatility, spread, volume, EH-alpha and EH-market.

development and the choice of execution strategies; only orders, execution shortfall is the strike price minus the exe-

better predictors of execution-horizon alpha will signif- cution price as percent of the strike price. In Exhibit 2, we

icantly reduce execution shortfall. introduce a hypothetical order execution that we will use

In the next section, we develop our framework for throughout this section to illustrate our framework. The

explaining the shortfall surprise, followed by a section on trader receives an order to buy 60,000 shares. The strike

our data sample and the construction of our variables. price at order arrival is $25.00. The trader executes the

The following section presents our empirical findings. In order over time in three executions and the volume-

the section after that, we focus on our puzzling finding weighted execution price is $25.06. The execution short-

that volume surprises do not affect trading costs. The next fall in this example is six cents or 24 basis points (bps).

section elaborates on the distinction between volatility In addition to liquidity impact, execution shortfall

and price surprises. We conclude with a discussion of the includes the opportunity cost of delayed execution. The

implications of our empirical findings, and possible exten- opportunity cost arises because the price may move away

sions of our analysis. from the trader over the execution horizon. This price

move has both a market and a stock-specific (alpha) com-

A FRAMEWORK FOR ANALYZING ponent and we estimate both. Exhibit 3 shows the three

THE SHORTFALL SURPRISE components of execution shortfall: liquidity impact, alpha

loss, and market loss. In our example, the trader is buying

We begin by formally defining execution shortfall, our in a rising market, so the opportunity cost is positive and

t-cost measure. For buy orders, execution shortfall is the increases the shortfall. If the price were falling, the oppor-

execution price minus the prevailing mid-quote at order tunity cost would be negative, reducing the shortfall, and

arrival (strike price) as percent of strike price.2 For sell possibly resulting in negative shortfall. Unlike liquidity

EXHIBIT 2

Execution Shortfall

impact, which is never negative, execution shortfall can 1. Stock capitalization: large-cap, mid-cap, and small-

be negative.3 cap stocks.

In this article, we analyze the shortfall surprise (SS) 2. Listing market: NYSE, AMEX, and NASDAQ

that we define as actual shortfall (SA) minus expected stocks.

shortfall (SE): 3. Order size: the actual dollar value of the order

executed.

SS = SA – SE (1) 4. Volatility over the execution horizon.

5. Quoted spreads over the execution horizon.

To generate the shortfall surprise, we must first esti- 6. Trading volume over the execution horizon.

mate expected shortfall. Expected shortfall is the short-

fall a trader can predict with only pre-trade information: The first three factors (stock capitalization, listing

when the trader receives the order, but before execution. market, and order size) we know with certainty pre-trade.

To estimate expected shortfall, we need estimates for both The other factors (volatility, spread, and volume) we do

liquidity impact (LE) and the expected underlying price not know with certainty pre-trade, and we must use their

move over the execution horizon (EH-price or PE): expected values to estimate liquidity impact (LE):

We estimate liquidity impact using the Goldman where Y are the factors known with certainty pre-

Sachs t-cost model.4 The model uses the following six trade, and XE are the factors not known with certainty

factors to predict liquidity impact: pre-trade.

EXHIBIT 3

The Three Components of Execution Shortfall

We next discuss our EH-price concept. In Exhibit 4, trade. Our empirical analysis below confirms this is indeed

we assume the market component of EH-price is zero the case.

and show how we measure the stock-specific alpha move In Exhibit 4, we show how we use the closing

over the execution horizon (EH-alpha). Continuing the price to calculate EH-alpha. We first calculate the alpha-

Exhibit 2 example, the trader received the order at 12:00, to-close. For buy orders, alpha-to-close is the closing

executed 20,000 shares immediately, another 20,000 shares price minus the strike price as percent of the strike

at 13:00, and the final 20,000 shares at 14:00. The exe- price. For sell orders, alpha-to-close is the strike price

cution horizon is two hours (order arrival to last execu- minus the closing price as percent of the strike price.

tion), and the volume-weighted execution turnaround The alpha-to-close in our example is 40 bps. We then

time is one hour. The execution turnaround time is the “allocate” the alpha-to-close to the order in propor-

order’s half-life, taking into account that the order execu- tion to the order’s half-life. We derive the allocation

tion is spread over the two-hour horizon.5 We define EH- factor (Φ) by dividing the order’s half-life (one hour)

alpha as the price move over the order’s half-life, aside by the time from arrival to market close (four hours).6

from the liquidity impact of the trade itself. So, in our example, the allocation factor is 1/4 and EH-

In calculating EH-alpha, it is critical to use a post- alpha is 10 bps.

trade price after the liquidity impact of the trade sub- In practice, the underlying price move has both

sides. In our example, liquidity impact subsides at 15:00, market and alpha components:

so any price after that will work. In our empirical analysis,

we use the closing price to measure the actual EH-alpha EH-price = EH-alpha + βEH-market (4)

for each order in our sample. This procedure assumes

that the closing price is not affected by the impact of the where β is the intra-day stock beta.

EXHIBIT 4

Execution-Horizon Alpha

1

If the time of last execution is after 16:00, it is possible for the ratio to exceed 1. In these cases (129 orders), we set the ratio to 1.

In our empirical analysis, we use S&P 500 ETF We next derive the shortfall surprises factor model

prices to decompose the underlying price move into EH- we use in our analysis. In Equation 5, replacing the

market and EH-alpha. In Exhibit 5, continuing our expected values of the various factors by their post-trade

example, the market-to-close move is 16 bps and allo- actual values, the actual shortfall (SA) is given by:

cating over the execution horizon EH-market is four bps.

Combining Equations (2) to (4) and, for illustration SA = a + bY + cXA + β MA + AA + U (6)

purposes, using a linear version of the liquidity impact

model L (Y, X), expected shortfall is given by: Actual shortfall is also influenced by other factors,

(U), not included in our empirical analysis. Most of these

SE = a + bY + c XE + β ME + AE (5) other factors are random. Some of them, however, may

be systematic but difficult to quantify even post-trade; for

where ME is the expected EH-market and AE is the e.g., trader skill or the presence of natural counterparties.

expected EH-alpha. Subtracting expected shortfall (Equation 5) from

Suppose the liquidity impact is 10 bps. To get the actual shortfall (Equation 6) we get our basic shortfall sur-

total shortfall, we must add the EH-price move. In prises model:

Exhibit 5, the EH-price move is 14 bps so the total

shortfall is 24 bps: 10 bps impact and 14 bps price loss. SS = c (XA – XE) + β (MA – ME) + (AA – AE) + U (7)

EXHIBIT 5

The Market and Alpha Components

Or, writing out the factors in full: in the model is one, and the coefficient of the market sur-

prise is β, the intra-day stock beta.

S s = c1 (VOLA – VOLE ) + c 2 (SPREADA – SPREADE ) In deriving our shortfall surprise model (8), we assume

Volatility surprise Spread surprise that the liquidity impact model we use to generate expected

+ c 3 (VLM – VLM ) + β(EH-market A – EH-market E )

A E

impact is correctly specified. But the model may be mis-

Volume surprise Market surprise specified; we may have omitted systematic variables (U in

A

+ (EH-alpha – EH-alpha ) + U E

(8) our specification), or the model coefficients may be biased.

Alpha surprise

Model misspecification creates another possible source of

shortfall surprises. In the Appendix, we discuss this issue

In our analysis of the shortfall surprises, we focus on more formally and decompose shortfall surprises into a

the five input surprises: 1) the volatility surprise, 2) the misspecification component and the input surprises com-

spread surprise, 3) the volume surprise, 4) the market sur- ponent. In our empirical analysis, we carefully constructed

prise, and 5) the alpha surprise. The expected EH-market our sample to minimize the risk of model misspecification.

and EH-alpha are hard to estimate and, in our empirical

analysis, we assume they are zero. We describe the sur- SAMPLE CONSTRUCTION

prises in greater detail in the next section. Note that in AND DESCRIPTION

the surprises model, the factors known with certainty

pre-trade (Y) drop out. Also, because of the way we con- Our final estimation sample consists of 241,610 client

structed EH-price, the coefficient of the alpha surprises orders executed by Goldman Sachs over seven months,

June through December 2006. To minimize the risk of For each order in our sample, we use the Goldman Sachs

model misspecification, we restricted our sample to closely t-cost model to estimate the expected liquidity impact.

match the sample we use in estimating the Goldman Sachs The model is re-estimated monthly using the most recent

t-cost model. Our final sample consists of market, not- nine months of data. For each order, we use the most

held orders, executed on an agency basis by the Goldman recent version of the model as of the date of the order to

Sachs U.S. single-stock trading desk.7 We also aggressively generate out-of-sample estimates of that order’s expected

filtered unusual orders; we dropped odd lots, kept only impact. For an order received on July 7, for example, we

orders in common stocks, dropped ADRs and ETFs, use the model estimated with data prior to July 7. The

OTCBB and Pink Sheets, dropped crosses, dropped orders monthly re-estimation of the model ensures the coeffi-

in extreme price stocks (less than $1 or more than $150), cients are not stale and again reduces the risk of model

dropped tick sensitive orders (e.g., sell short), etc. Finally, misspecification.

we dropped orders with data errors. The Goldman Sachs t-cost model gives different

Exhibit 6 summarizes the order composition of the impact estimates depending on the execution horizon

final sample. The 241,610 orders in our sample are evenly specified. In our analysis, we estimate the expected impact

divided between buys and sells. One quarter of the orders using the order’s actual execution horizon: order arrival

are NASDAQ stocks8 and the balance are NYSE stocks.9

Of the total orders, 64% are in large-cap stocks and 9%

in small-cap stocks. Only 296 orders exceed 25% of

average daily volume, and 89% of the orders are less than

10,000 shares. The majority of the orders (92%) have an

EXHIBIT 6

execution half-life less than 15 minutes; 13,464 orders Order Types

have a half-life more than 30 minutes. The average stock

price in our sample is $34.

Exhibit 7 summarizes order characteristics for the

overall sample. The average order size executed is 9,420

shares, ranging from 18 shares to 29 million shares. The

value-weighted average order size executed is 5% of ADV

and the participation rate is 21%.10 The value-weighted

half-time is 65 minutes, but the median is only 15 sec-

onds and ranges from instantaneous executions to all-day

executions. The value-weighted average actual shortfall

is 23 bps, but the median is only 2 bps. Actual shortfall

ranges widely from –757 bps to +1,069 bps.

Exhibit 8 confirms that, at least on average, the

closing prices we use in calculating the EH-price move are

not affected by liquidity impact. In Exhibit 8, we plot the

average shortfall, same-day closing price and the closing

prices over the next five days for all the orders in our

sample. We measure all prices relative to the order-arrival

price, and include both buy and sell orders, but flip the sign

of the sells. We use value-weighted averages, so large trades

get more weight in the average. The average closing price

is 26 bps higher than the strike price, but there is little

reversal over the next five days. The absence of reversal

reassures us that the closing price is not affected by the

temporary liquidity impact of the trades themselves.

We next describe how we construct the shortfall

surprises and the five factor surprises we use in our analysis. Sample period June 1 through December 31, 2006.

EXHIBIT 7

Overall Sample Summary Statistics1

1

Sample period June 1 through December 31, 2006; (241,610 orders).

2

Simple mean.

3

Value-weighted mean.

to last execution. To generate our impact estimates, we to the order’s execution horizon, by using the same

must also specify pre-trade estimates for volatility, quoted allocation factor Φ that we use in calculating EH-

spreads and trading volume: alpha (Exhibit 4).

• Quoted spread is the time-weighted spread over the

• Volatility is measured as the percent difference execution horizon, taking into account the spread

between the intra-day high and low price, adjusted smile (the U-shaped intraday spread pattern).11

EXHIBIT 8

Closing Prices are Not Affected by Liquidity Impact

Sample period June 1 through December 31, 2006; 241,610 buy and sell orders (Sign of sell orders flipped). Value-weighted averages.

• Trading volume is measured over the execution Exhibit 9 shows a scatter plot of actual shortfall (ver-

horizon, taking into account the volume smile (the tical axis) against expected shortfall (horizontal axis).

U-shaped intraday volume pattern). Actual shortfall equals expected shortfall along the

45 degree line. Orders with actual shortfall exceeding

We generate pre-trade estimates for these three expected shortfall are above the 45 degree line and orders

factors using their median value over the prior 21 with actual shortfall less than expected shortfall are below.

trading days. Along the vertical, we see the large range in actual short-

To estimate expected shortfall (Equation 5), we also fall from a minimum of –757 bps to a maximum of +1,069

need pre-trade estimates for EH-alpha and EH-market. bps. Along the horizontal, we see the range in expected

Unlike liquidity impact, where reliable pre-trade esti- shortfall estimates from a minimum of 0.3 bps to a max-

mates are widely available, pre-trade estimates of EH- imum of 334 bps. Because we assume expected EH-

market and EH-alpha are difficult to find. As is typically market and EH-alpha are zero, the expected shortfall is

the case in practice, in our analysis we assume that pre- always positive (just the liquidity impact).

trade, the expected EH-market and expected EH-alpha The number in each bubble indicates the number

are both zero. of orders clustering in close proximity to that point. About

EXHIBIT 9

Scatter plot of Actual Vs Expected Shortfall

210,000 orders are clustered between –60 and +35 bps of values and report the unexplained variation (one minus

actual shortfall and between 0 and 15 bps of expected R-square). The unexplained variation tells us how much

shortfall. The remaining orders show a large dispersion, of the actual variation in each variable is not explained by

both in the expected and actual shortfall. Our objective the expected value of that variable.

in this article is to better understand this dispersion: the Exhibit 11 shows that 80% of the actual shortfall

shortfall surprise. variation is not explained by expected shortfall (or con-

Exhibit 10 summarizes the post-trade actual values versely, that the pre-trade estimates explain only 20% of

of the five factors we use to explain the shortfall surprise. the post-trade variation in actual shortfall). Expected

All five factors range widely, mirroring the large range of volatility leaves only 23% of the variation in actual

actual shortfall. Volatility, for example, ranges from 0% to volatility unexplained. Expected spreads and volume

20% and quoted spreads from 0 bps to 882 bps. The actual leave about 50% of the post-trade variation unexplained.

EH-market move ranges from –119 bps to +121 bps, but Since we assume they are zero, expected EH-market and

averages to zero by all measures (median, simple, and EH-alpha leave 100% of the variation in the actual EH-

weighted means). The actual EH-alpha, however, has a alpha and EH-move unexplained. The EH-market and

much bigger range (from –557 bps to +836 bps) and the AH-alpha surprises, therefore, have great power to

value-weighted mean is 10 bps. potentially explain the shortfall surprise. Exhibit 11 also

In our shortfall surprises model (Equation 8), the shows the large range of all the surprises, another mea-

shortfall surprise depends on the five factor surprises. sure of forecasting accuracy. In general, all five factors

Exhibit 11 summarizes our estimates of the shortfall sur- have large surprises and, therefore, potentially large

prise and the five factor surprises. The first column shows explanatory power in our shortfall surprise regressions,

the forecasting accuracy of our pre-trade measures and gives to which we now turn.

an indication of the potential explanatory power of each

surprise. If, for example, a factor is perfectly forecasted, THE EMPIRICAL FINDINGS

then there is no factor surprise and that factor cannot explain

the shortfall surprise. We measured forecasting accuracy by Our preferred regression specification for the short-

estimating univariate regressions of actual values on expected fall surprise (SS) model is:

EXHIBIT 10

The Five Factors: Actual Values1

1

Sample period June 1 through December 31, 2006; (241,610 orders).

2

Simple mean.

3

Value-weighted mean.

EXHIBIT 11

The Five Factor Surprises: Actual Minus Expected Values1

1

Sample period June 1 through December 31, 2006; 241,610 orders.

2

One minus the R-square from univariate regression of actual to expected.

Volatility surprise of EH-market in our regression is an estimate of the short-

⎛ VLM A ⎞ term intra-day beta on our sample; the results in Exhibit

+ c 2 (SPREADA – SPREADE )+ c 3 ⎜ log ⎟ 12 suggest that this is also close to one.13

Spread surprise ⎝ VLM E ⎠

Volume surprise To better understand the relative importance of the

A E five factor surprises in explaining the shortfall surprise,

+ c4 (EH-market – EH-market )

Market surprise we also ran a series of univariate regressions of the short-

+ c5 (EH-alpha A – EH-alpha E ) fall surprise on each of the factor surprises in turn. In

Alpha surprise Exhibit 13, columns four to eight, we report the R-squares

from these univariate regressions. The first row shows the

We experimented with many alternative specifica- results of running the regressions on all the orders in our

tions (for e.g., with and without intercepts, different func- sample. In the univariate regressions, the EH-alpha sur-

tional forms for the five factors, etc.) and, in all cases, the prise is by far the most important factor explaining almost

results are similar.12 40% of the shortfall surprise variation. The other four

Exhibit 12 shows the results from the estimation of factors have little explanatory power; EH-market explains

our preferred specification over our whole sample of 5.8%, volatility explains 3.0%, spreads explain 1.5% and,

241,610 orders. The overall fit of the regression is 44% most surprising, volume only explains 0.2% of the short-

(R-square), and the coefficients of all five factor surprises fall surprise.

are significant and have the right sign. Higher-than- Exhibit 14 shows that the covariances across the

expected volatility and spreads lead to higher-than- five factor surprises, while small, are not always zero.

expected shortfall while higher-than-expected volume The correlation between the EH-alpha and volatility

leads to lower-than-expected shortfall. Similarly, higher- surprises, for example, is 25% and the correlation

than-expected market and alpha moves result in higher- between the volume and volatility surprises is 17%.

than-expected shortfall. Our hypothesis is that the Because the covariances across the factor surprises are not

coefficient of EH-alpha in the regression should be one, zero, the univariate regression results can be misleading.

and Exhibit 12 shows that this is indeed the case; a five To double check, we also performed an incremental

basis-point increase in EH-alpha will lead to a five analysis, where we evaluate the contribution of each

EXHIBIT 12

The Shortfall Surprise Multivariate Regression1

1

Sample period June 1 through December 31, 2006; t-statistics in brackets below the coefficients.

factor surprise by dropping it from the full multivariate In Exhibit 13, rows two to four, we divide our

regression. The overview results in Exhibit 1 are from sample by stock capitalization: orders in large-cap stocks

the incremental approach and are similar to the uni- (>$7.5 billion), mid-cap stocks, and small-cap stocks

variate regression results. (<$1 billion). As we move from large-cap to small-cap

The last three columns in Exhibit 13 show another stocks, the accuracy of the expected shortfall forecasts as

example of the incremental approach. Column 10 shows measured by the root mean-square error (RMSE) falls.14

the full five-factor regression, and the regression in column The expected shortfall RMSE is 13 bps for orders in large-

nine drops the two EH-price surprises (market and alpha). cap stocks, but 49 bps for orders in small-cap stocks, indi-

Dropping the EH-price surprises reduces the explanatory cating that the shortfall surprise is higher in small-cap

power of the regressions from 44% to only 4.5%, con- stocks.

firming that the price surprises account for almost all of Our findings on the five factor surprises are similar

the explanatory power in our shortfall surprise regressions. across stock capitalization buckets. In all three buckets,

EXHIBIT 13

Shortfall Surprise Attribution: An Overview1

1

Sample period June 1 through December 31, 2006.

2

Root mean-square error: the square root of 1/N Σ(SAi – SEi)2, where SAi is the post-trade actual shortfall and SEi is the pre-trade expected shortfall for each order.

3

The average executed order size in this bucket is 775 shares.

EXHIBIT 14

Covariance of the Five Surprises1

1

Sample period June 1 through December 31, 2006; (241,610 orders).

2

Log of ratio of actual to expected volume.

the EH-price surprise explains most of the shortfall sur- the much smaller shortfall surprise. But still, across the

prise. The volatility and spread surprises explain more of five factor surprises, the EH-price surprise is by far the

the shortfall surprise in small-cap stocks than in the other most important factor in explaining the shortfall surprise:

two buckets, but the numbers are still small. In the uni- volatility, spread, and volume surprises explain 2.3%, while

variate regressions, for example, the spread surprise EH-price explains 14.7%.

explains 4.3% of the shortfall surprise in small-cap stocks, In Exhibit 15, we focus on the time dimension of

but only 0.1% of the surprise in large-cap stocks. The order executions. A quick execution exposes the order

volume surprise explains very little of the shortfall surprise to little EH-price movement, so we expect the EH-price

across all stock capitalizations. surprise to explain less of the shortfall surprise. At the

Because we construct EH-price using closing prices, other end, an order that takes several hours to execute is

one concern is that the closing prices are affected by liq- exposed to potentially large price moves, so we expect

uidity impact and therefore are not a pure measure of the EH-price to be much more important in explaining the

underlying price move. In Exhibit 9, we showed that, at shortfall surprise. The results in Exhibit 15 confirm this

least on average, the closing prices in our sample are not hypothesis.

affected by liquidity impact. The liquidity impact of the Exhibit 15 is divided in four panels: Panel A covers

larger trades in our sample is more likely to affect the orders in all stocks, Panel B focuses on orders in large-cap

closing price.15 So, in Exhibit 13, rows five to six, we fur- stocks, Panel C focuses on mid-cap stocks and Panel D

ther test the sensitivity of our results to the quality of the focuses on small-cap stocks. The top row in each panel

closing price by dropping the larger trades in our sample. replicates the information in Exhibit 13. The next two

In row five, we drop all orders greater than 25% of ADV rows in each panel divide the orders into orders with half-

(297 orders) and in row six, we drop all orders greater life more than 30 minutes and orders with half-life less than

than 15% of ADV (1,228 orders). In both cases, dropping 30 minutes. The last two rows in each panel repeat this

the outlier large orders does not affect our results. split, using a 60-minute cut-off.

In the last row in Exhibit 13, we focus on the small The first thing to notice in Exhibit 15 is that the

orders in our sample: 177,191 orders less than 0.1% ADV forecasting accuracy of expected shortfall falls sharply as

(average order size 775 shares). As expected, for these the execution horizon increases (column 4). In Panel A,

small orders, the shortfall forecasting error is low: the for example, the RMSE for orders with half-life more

RMSE is only 7 bps compared to 23 bps overall. For these than 60 minutes is 89 bps compared to only 15 bps for

small orders, the five factor surprises explain only 17% of orders with half-life less than 30 minutes. For orders in

EXHIBIT 15

The Shortfall Surprise Attribution: Details1

1

Sample period June 1 through December 31, 2006.

small-cap stocks (Panel D), the RMSE for orders with Panel A, for example, EH-price explains 63.5% of the

half-life more than 60 minutes is even higher: 118 bps. shortfall surprise in orders with half-life more than 60

The forecasting accuracy results in Exhibits 13 and 15 minutes, but only 6.3% of the surprise for orders with

strikingly quantify what most traders know from experi- half-life less than 30 minutes. For orders in large-cap stocks

ence: most of the shortfall surprise comes from large orders with half-life more than 60 minutes, EH-price explains

that take time to execute. a remarkable 73% of the shortfall surprise.

In Exhibit 15, column 11 shows the explanatory Looking at the two components of EH-price

power of the full five-factor regression model. Across the (market and alpha), the results in Exhibit 15 across all

various buckets, the R-square of the full model ranges buckets are consistent with the results in Exhibit 13: EH-

from 75.3% (orders >60 minutes, large-cap stocks) to alpha is by far the most important factor in explaining

8.0% (orders <30 minutes, large-cap stocks). We next the shortfall surprise. For orders in large-cap stocks with

examine the contribution of the five-factor surprises in half-life more than 60 minutes, for example, EH-alpha

explaining the shortfall surprise. The last column in explains 65.3% of the shortfall surprise while EH-market

Exhibit 15 shows that, consistent with our hypothesis, explains only 12.5%. For orders with half-life less than

the EH-price surprise is much more important in 30 minutes, EH-alpha explains 6.1% of the shortfall sur-

explaining the shortfall surprise for slow executions. In prise while EH-market explains only 0.8%.

The summary statistics on EH-alpha and EH-market simple correlation between the volume and shortfall sur-

in Exhibit 10 provide a clue why the alpha component prises is small, but positive, 4%. The striking difference

dominates: EH-alpha has a bigger range (–557 bps to between the multivariate and univariate results suggests that

+839 bps) than EH-price (–119 bps to +121 bps). Out other factor surprises correlated the volume surprise (for

of the 241,610 orders in our sample, 3,173 orders are e.g., volatility and EH-alpha in Exhibit 14), and increase

associated with an EH-alpha move exceeding 50 bps (in costs outweighing the beneficial effects of higher-than-

absolute terms), but only 407 orders are associated with expected volume.

an EH-market move exceeding 50 bps. We performed extensive additional robustness tests

Exhibit 15, column 10, confirms the robustness of of our finding on the volume surprise. In all cases, volume

our findings on the volatility, spread, and volume sur- surprises explain very little of the shortfall surprise. In

prises: consistently small explanatory power across all Exhibit 17, for example, we focused on high-volume sur-

buckets. Finally, Exhibit 15, column three, confirms the prise events: 20,116 orders in our sample associated with

robustness of the absence of any volume effect. Across all a volume surprise greater than one standard deviation.

buckets, the volume surprise explains less than 1% of the Again, there is little change: volume surprises explain only

shortfall surprise. 1.4% of the shortfall surprise and the EH-price move

Perhaps the most striking observation about explains 60.2%.

Exhibits 13 and 15 is the robustness of our main results. The important implication of our volume findings

Whichever way we cut the data, EH-alpha is by far the is that traders and algorithms cannot naively exploit

most important factor in explaining the shortfall surprise; volume surprises to reduce trading costs. To beneficially

while volatility, spread, and volume surprises explain very exploit a volume surprise, execution strategies must dis-

little. The volume results are particularly surprising and, tinguish between liquidity supply and liquidity demand

in the next section, we take a closer look. surprises, which is hard to do. Taking advantage of

higher-than-expected volume has one unambiguous

THE VOLUME SURPRISE beneficial effect: traders fill orders faster reducing exe-

cution risk.

Our most puzzling finding is that volume surprises

do not affect trading costs. This seems counterintuitive: THE VOLATILITY SURPRISE

more volume than anticipated should surely reduce trading

costs. This intuition, however, rests on the presumption Another empirical finding worth highlighting is the

that higher volume is caused by an increase in the supply sharp distinction between the small effect on trading costs

of liquidity (Case A in Exhibit 16). Our empirical find- of volatility surprises and the large effect of EH-price sur-

ings suggest that an increase in volume is caused by an prises. The important distinction here is that volatility

increase in both liquidity supply and demand (Case C in surprises are unsigned price moves while EH-price sur-

Exhibit 19): a hundred more traders supply liquidity, but prises are signed price moves, depending on the direction

another hundred traders demand more liquidity. An of the trade.16 For buy orders, EH-price is closing price

increase in uncertainty, for example, will lead to an increase minus strike price, while for sell orders, EH-alpha is strike

in both liquidity supply and demand. Similarly, the pop- price minus closing price (see Exhibit 18).

ular “participate” execution strategies and algorithms The unsigned EH-price is just another measure of

automatically generate liquidity demand in proportion to execution-horizon volatility: closing price minus strike

increases in liquidity supply. price (as percent of strike price) for both buys and sells.

In our multivariate regressions where we control Let’s call this measure of volatility EH-volatility. In

for the variation in all five factors, all else being equal, Exhibit 18, we compare the results of regressing the short-

the volume surprise reduces trading costs, but by a tiny fall surprise on the signed and unsigned EH-price. In

amount; for e.g., volume twice the expected amount the univariate regression of shortfall surprises on signed

reduces trading costs by less than 0.3 bps. In the uni- EH-price, EH-price explains more than 40% of the vari-

variate regressions of shortfall surprise on volume sur- ation in the shortfall surprise. However, in the univariate

prise, a surprise increase in volume actually increases regression of the shortfall surprise on EH-volatility (the

trading costs (Case B in Exhibit 16). Equivalently, the unsigned EH-price), the unsigned EH-price explains less

EXHIBIT 16

The Volume Surprise: Why It Does Not Matter?

EXHIBIT 17

High Volume Surprises1

1

Sample period June 1 through December 31, 2006.

2

|VLMA – VLME| > one standard deviation.

EXHIBIT 18

The Volatility Surprise

1

Sample period June 1 through December 31, 2006.

surprise. We find similar results in a multivariate regres-

sion of the shortfall surprise on both the signed and Our finding that price surprises are the main cause

unsigned EH-price: all the explanatory power comes of shortfall surprises means that, in interpreting post-trade

from the signed price move. EQA, it is essential to control for the underlying execu-

What is striking about the analysis in Exhibit 18 is tion-horizon price move. Our preferred way of doing

that, by design, EH-price and EH-volatility are exactly this is by reporting the EH-price move alongside the

the same measure, except that one is signed based on the actual shortfall, or equivalently, by using the EH-price to

direction of the trade and the other is not. The impor- split actual shortfall into liquidity impact and price loss.18

tant implication of our findings in this section is that Suppose actual shortfall is 20 bps and the EH-price is 18

traders and algorithms cannot exploit volatility surprises bps: the liquidity impact is only 2 bps, so this is good exe-

to reduce trading costs unless they can accurately predict cution. If the EH-price is 4 bps, however, 20 bps is poor

the direction of the price move.17 execution (16 bps liquidity impact). We avoid using the

usual VWAP benchmarks to control for price surprises,

because VWAP benchmarks may be affected by the impact

IMPLICATIONS AND CONCLUSION of the trade itself and may give misleading results.19

The importance of price surprises also suggests that

Our empirical findings have important implications post-trade EQA to evaluate execution strategies and algo-

for post-trade execution quality analysis (EQA) and for rithms can only be meaningfully done with a large sample

pre-trade tool, for the development of algorithms, and of orders. In small samples, extraneous price surprises will

for the choice of execution strategy. We conclude with a dominate the statistics and obscure the true performance

discussion of these implications. being evaluated.

Pre-trade Tools and Development than a few minutes to complete, by far the most impor-

of Algorithms tant factor is the expected underlying price move (EH-

alpha). But is there any empirical evidence that traders

Trading cost models are widely used to generate correctly anticipate EH-alpha and adapt their execution

pre-trade cost estimates and choose the right execution strategies accordingly? In Rakhlin and Sofianos [2006a],

strategy. Cost models are also embedded in various algo- we found no evidence that buy-side traders allocated

rithms.20 These trading-cost models have relatively low orders across passive VWAP algorithms and more aggres-

forecasting power and model developers are trying hard sive shortfall algorithms based on expected EH-alpha.

to improve them. Our findings suggest that the only way But that article was only the beginning of an ongoing

to make a significant difference in forecasting accuracy is research effort.

by coming up with better forecasts of EH-price. Better If traders cannot correctly anticipate EH-alpha, then

forecasts of volume and volatility, in particular, will not execution strategies should be modified to reflect this fact.

improve the forecasting accuracy of trading cost models. Segmenting orders between passive and aggressive exe-

But how can we come up with better EH-price cutions, for example, requires reliable EH-alpha forecasts.

forecasts? One way is to continue improving models for In the absence of reliable EH-alpha forecasts, adopting a

short-term price forecasts. Work is being done on this front single, relatively passive, execution strategy may work best.

in the context of short-term quantitative trading. The The downside of a passive strategy is high execution risk

challenge here is that a better short-term price forecasting (a large variation in actual shortfall). If execution risk is

model is more valuable as a proprietary trading tool, rather a concern, then an alternative strategy is to let the trader’s

than a widely available pre-trade tool. But the synergies risk aversion drive the choice of execution strategy, and

exist and the possibility of combining short-term trading accept a higher liquidity impact. Yet another alternative

and improved execution strategies within a buy-side is to request capital and pay a premium to completely

trading desk is intriguing. eliminate execution risk.

Another way to improve the accuracy of shortfall The shortfall surprises framework that we devel-

estimates is by allowing users to adjust the estimates based oped in this article can be used more generally to quan-

on the user’s view of the expected EH-price. Sell-side tify and compare the value-added of alternative execution

traders may have a good “feel for the market,” so allowing strategies. By analyzing shortfall surprises, instead of actual

them to adjust cost estimates accordingly can be useful. shortfall, we control for the usual factors affecting exe-

Similarly, buy-side traders may have a better understanding cution quality and can focus on the incremental value of

of portfolio managers’ intentions and alpha signal. The additional factors. Consider again the comparison between

Goldman Sachs PortX algorithm, for example, allows passive VWAP and aggressive shortfall algorithms. VWAP

traders to input their alpha-to-close expectation and uses algorithms should outperform shortfall algorithms when

this input to adjust the pre-trade cost estimates, and the EH-price is low and the reverse when EH-price is high.

optimum execution strategy, accordingly. Using our shortfall surprises framework, for VWAP exe-

Our findings also have important implications for the cutions, the shortfall surprise should be negative (actual

development of dynamic algorithms that react real-time less than expected) when EH-price is low and positive

to market changes. For a dynamic algorithm to usefully when EH-price is high and the reverse for the shortfall

react to a volume surprises, it must be able to distinguish executions. In doing this comparison, we do not have to

between supply and demand causes of the surprise, which control for order size, stock capitalization, or the other

is very challenging. Similarly, dynamic algorithms cannot “usual suspects,” because expected shortfall already reflects

usefully exploit volatility surprises, unless they can pre- these factors.

dict the direction of the market. We can also use the shortfall surprises framework

to quantify the value-added of high-touch executions.

Choice of Execution Strategy The rise of low-touch trading has created a pressing ques-

tion: why pay more than 12 bps in commissions to use

Our findings in this article strikingly quantify one the high-touch if one can execute for less than 3 bps in

of the main themes in our research over the past five commissions low-touch? The answer, of course, is that for

years:21 in choosing an execution strategy that takes more difficult orders, high-touch executions may add value by

reducing the indirect costs of trading: liquidity impact and SE = a + b Y + c XE + PE

opportunity cost. But to decide between high-touch and

low-touch, we must explicitly quantify high-touch value- The other using the actual values of the X and P factors

added, and determine the order difficulty cut-off, above (perfect-foresight estimates):

which it is worth executing high-touch. In future work,

we plan to use our shortfall surprises framework to address SEA = a + b Y + c XA + PA

these questions. The shortfall surprise is SS = SA – SE, and by adding and

subtracting SEA, we decompose SS into two components, SS1

and SS2:

APPENDIX

MODEL MISSPECIFICATION SS = ( SA – SEA) – (SEA – SE) = SS1 – SS2

AND THE SHORTFALL SURPRISE

But

In this Appendix, we formally decompose the shortfall sur-

SS1 = SA – SEA = (α + β Y + γ XA + δ Z + PA + u )

prise into the surprise caused by model misspecification (e.g.,

omitted factors), and the component caused by the surprise in the – (a + b Y + c XA + PA)

included factors. Suppose the true model for actual shortfall is:

Re-arranging:

SA = α + β Y + γ XA + δ Z + PA + u

SS1 = (α – a) + (β - b) Y + (γ – c) XA + δ Z + u

where Y = factors known with certainty pre-trade, XA = If the impact model is not misspecified a = α, b = β, c = γ, δ =

actual values of factors not known with certainty pre-trade, 0, and SS1 = u, a random variable.

Z = factors not included in the liquidity impact model, PA = The shortfall surprise component SS1, therefore, mea-

actual execution-horizon price move, and u is a random sures the extent the shortfall surprise SS is caused by other fac-

variable. tors (model misspecification). Similarly:

Suppose the estimated liquidity impact model is:

SS2 = SEA – SE = c (XA – XE) + (PA – PE)

L=a+bY+cX

The shortfall surprise component, SS2, therefore, mea-

Post-trade, we can generate two estimates from the impact sures the extent the shortfall surprise SS is caused by the factor

model. One using the expected values of the X and P factors: surprises (what we are testing in this article).

EXHIBIT A1

Further Disaggregating the Shortfall Surprise1

1

Sample period June 1 through December 31, 2006; 241,610 orders.

Using the Goldman Sachs t-cost model and the data in ticipation, etc. Moving expected shortfall to the right hand side

our sample, we estimated both SE and the perfect foresight of the regressions also does not make much difference; it has a

shortfall SEA. We then decomposed the shortfall surprise into coefficient of one.

the other-factors component (SS1), and the factor-surprise com- 13

These results on the coefficients of EH-market and EH-

ponent (SS2), and estimated the shortfall surprise model only on alpha are robust across all specifications and sub-samples that we

the SS2 component. estimated.

Exhibit A1 shows our empirical results using this decom- 14

We define the RMSE as the square root of 1/N ∑(SAi –

position. Our estimated other-factors component SS1 accounts SEi)2, where SA is the post-trade actual shortfall and SE is the pre-

for 56% of the shortfall surprise, exactly matching the “unex- trade expected shortfall for each order.

15

plained” component in our one-step procedure in Exhibit 1. At the same time, however, the larger the trade, the

In the regression of the factor-surprises component SS2, on the higher the likelihood that the trade was generated by a strong

five factors, we get an R-square of 93%, and the overall alpha signal (the true EH-alpha is higher).

16

explanatory power of each of the five surprises is similar to For further discussion of this important point, see

the results from our one-step procedure (Exhibit 1). For Rakhlin and Sofianos [2006], pp. 43–45.

17

example, the alpha surprise explains 80% of the variation in In this article, we focus on agency executions. Volatility

SS2, which itself accounts for 44% of the overall shortfall-sur- surprises may be more important for capital commitment

prise variation; so in the two-step approach, the alpha sur- (principal) trades. In a capital commitment trade, the dealer

prise explains 35% of the overall shortfall compared to 38% prices volatility (execution risk) in the price premium/dis-

in the one-step procedure. count. So an increase in volatility will make principal trades

more expensive.

18

ENDNOTES For examples of our use of the decomposition of short-

fall into liquidity impact and price loss, see Cai and Sofianos

The authors thank Jeff Bacidore, Tianwu (Michael) Cai, [2006] and Rakhlin and Sofianos [2006].

19

Barbara Dunn, Oliver Hansch, Kilian Mie, and Ingrid Tierens, For further discussion of this point, see Sofianos [2006].

20

all at Goldman, Sachs & Co., for their comments. For example, the Goldman Sachs single-stock 4Cast

1

See Sofianos [2006] for a discussion of execution shortfall. and portfolio PortX algorithms. See Rakhlin and Sofianos

2

Orders that arrive before the market opens (9:30 am), [2006a, 2006b] for descriptions of these two algorithms.

21

we strike at the opening price. Bacidore and Sofianos [2002, 2003] through Cai and

3

Liquidity impact will never be negative for liquidity- Sofianos [2006].

seeking orders. Liquidity-providing orders, for e.g., limit orders,

conditional on execution, may have a negative impact (save the REFERENCES

spread).

4

For more details on the model, see Rodella [2005]. Bacidore, Jeffrey and George Sofianos. “Evaluating Execution

5

Only if the trader had executed all 60,000 shares at 14:00 Quality for Large Orders.” Goldman Sachs Trading and Market

would the execution horizon and the execution turnaround Structure Report (2002).

time be the same (two hours).

6

This allocation procedure assumes the alpha is growing ——. “Choosing the Best Order Execution Strategy.” Goldman

linearly from order arrival to market close. Sachs Trading and Market Structure Report (2003).

7

We exclude limit orders, held orders, and any orders where

Goldman Sachs provided capital at the request of the client. Cai, Tianwu (Michael) and George Sofianos. “Multi-day Exe-

8

NASDAQ GS, NMS, and Small Cap. cutions.” Journal of Trading, Vol. 1, No. 3 (2006), pp. 25–33.

9

The sample also includes a few AMEX stocks.

10

We define participation rate as the executed quantity as Rakhlin, Dmitry and George Sofianos. “The Choice of Exe-

a percent of the actual overall market trading volume in that cution Algorithm: VWAP or Shortfall.” Journal of Trading,

stock from strike time (order arrival time) to the time of the Vol. 1, No. 1 (2006a), pp. 26–32.

last execution.

11

We use only firm and valid NBBO quotes in our ——. “The Impact of an Increase in Volatility on Trading

calculation. Costs.” Journal of Trading, Vol. 1, No. 2 (2006b), pp. 43–50.

12

We tried, for example, several different ways of speci-

fying the volume surprise: actual minus expected volume, the Rodella, Elena. “Introducing Cost Wizard: Comparing Two-

log of actual divided by expected (the specification we use in Hour and All-Day Executions.” Goldman Sachs Cost Wizard

our analysis), the actual participation minus the expected par- Reports, Issue 1 (May 12, 2005).

Sofianos, George. “Execution Benchmarks: VWAP or Pre- algorithmic models derive pricing and trading estimates based on historical

trade Prices.” Journal of Trading, Vol. 1, No. 1 (2006), pp. 22–25. volume patterns, real-time market data, and parameters selected by the GSAT

user. The ability of Goldman Sachs’ algorithmic models to achieve the per-

formance described in this article may be impacted by changes in market con-

ditions, systems or communications failures, etc. Finally, factors such as order

To order reprints of this article, please contact Dewey Palmieri at quantity, liquidity, and the parameters selected by the GSAT user, may impact

dpalmieri@iijournals.com or 212-224-3675 the performance results.

The opinions expressed in this article are those of the authors and do

not necessarily represent the views of Goldman, Sachs & Co. These opinions

represent the authors’ judgment at this date and are subject to change.

This material was prepared by the Goldman Sachs Equity Execution Goldman, Sachs & Co. is not soliciting any action based on this article.

Strategies Group and is not the product of Goldman Sachs Global Investment It is for general information and does not constitute a personal recommenda-

Research. It is not a research report and should not be construed as such. tion or take into account the particular investment objectives, financial situa-

The information in this article has been taken from trade data and tions, or needs, of individual users. Before acting on any advice or

other sources we deem reliable, but we do not represent that such informa- recommendation in this article, users should consider whether it is suitable for

tion is accurate or complete, and it should not be relied upon as such. This their particular circumstances.

information is indicative, based on among other things, market conditions at Copyright: Summer 2007, Goldman, Sachs & Co.

the time of writing, and is subject to change without notice. Goldman Sachs’

- A Propagation Model for Mobile Radio Propagation Loss in an Urban Area at 800 MHzEnviado porMatthew Carter
- Statistics 578 Assignment 5 HomeworkEnviado porMia Dee
- Correlation and RegressionEnviado poranindya_kundu
- Acc Hedge Effectiveness AFP RegressionEnviado porchet_dutts
- Effects of School-Based Quality Factors on Secondary School Students Achievement in English Language in South-Western and North-Central Nigeria.pdfEnviado porstopbugging213
- Assessing risk—what your mutual fund house doesEnviado porIciciprumf
- Report 12242244 Sds dEnviado porMohit Agarwal
- Analysis of Willingness and Ability to Pay (WTP and ATP) On Waste Management in the City TernateEnviado porIOSRjournal
- MBA Assignment- First SemesterEnviado porprakashsushant2008
- MA Statistics TutorialEnviado porNabil Lahham
- stach14Enviado porSunil Vatas
- BI White Paper Healthcare Analytics Practical Predictive Analytics 101 May 2013Enviado porAnonymous JE7uJR
- converted.phaiptEnviado porVenki Venkatesh
- 15 Regression BasicsEnviado porAaron KW
- 58.FullEnviado porFiaz jutt
- WhiteEnviado porMarielly Palomino Martinez
- artigo 5Enviado porMonalisa Abrante Mariano
- ANZMAC2009-294Enviado porShabanaSaleem
- Ishan Gupta SPREnviado porprofessor
- Acc UralsAccurals.DOCXEnviado porsaadman672
- Econ 131Enviado porSteve Coin
- C01F03_2012_10_28Enviado porIshanDogra
- CPF Why is It So Hard to UnderstandEnviado porDhruv Baid
- Diakidoy & Constantinou (2000)Enviado porIbad Mohammed
- Beyond Market - Issue 97Enviado porUPPULA
- End of IndiaEnviado porvjpopines
- Abmpowerpoint 141113054150 Conversion Gate01Enviado porMclin Jhon Marave Mabalot
- Main-report-of-IMC.pdfEnviado porSakhawat Hossain
- Report NotesEnviado porKinsella Sophie
- Christian Muhr Capabilites BrochureEnviado porinnovator123

- The House of Usher EssayEnviado porJohn Garza
- ONA - Dark GodsEnviado porFrater Asg
- tda essay practice rubricEnviado porapi-391314886
- Parrhesia Mag 20 - Stiegler IssueEnviado porgrundum
- lesson plan 3 - perimiter areaEnviado porapi-282019703
- Introduction to ControlEnviado pornaderjsa
- 16.08 Panganiban vs PeopleEnviado pornazh
- Morphological Changes of Limestone Sorbent Particles During Carbonation-calcination Looping Cycles, FennellEnviado poranon_982022273
- What is Portfolio ManagementEnviado porSuji Pranav
- Civil Code, Volume I (Persons and Family Relations)Enviado porAmicus Curiae
- NYIC v. Department of CommerceEnviado porFedSmith Inc.
- The Holmes TreasuryEnviado porKamizori
- MS SDLEnviado porTariq Mahmood
- UntitledEnviado porapi-162641823
- Buckling of ColumnsEnviado porKeerthi Dayarathna
- Perpajakan SingapurEnviado porDio Patra
- IM11B03A03-06E_020 (GC Alarm Message)Enviado porMohammad Abbasi
- Mathematical Symbols ListEnviado porAhmed Zezo
- 75350661-MFB-Training.pdfEnviado porhai dang
- 8-Pattern Recognition in UbicomEnviado porDinh Ngoc Viet Tung
- Evald Ilyenkov and SpinozaEnviado porAnonymous IfL830HBjF
- RP-A201Enviado porsmithwork
- Raghwendra ENR-15057Enviado porAmit Prajapati
- hazingaEnviado porClinton Villaflor
- Boieng 767Enviado porAjinkya Pawar
- Delhi Police SI Coaching.pptEnviado porcompetitiongurukul
- HFS06Reason.pdfEnviado porSayed Darwish
- Chapter04 Work Energy and Power sEnviado poreltytan
- Te-025 System Test ScenariosEnviado porvinoth4i
- Assassins Creed 2 Percent SyncEnviado porBreno Miranda

## Muito mais do que documentos

Descubra tudo o que o Scribd tem a oferecer, incluindo livros e audiolivros de grandes editoras.

Cancele quando quiser.