Cap-Weighting Problem - the Market Overpays for Growth (and by a lot)

Two recent fascinating papers in the Journal of Portfolio Management - Clairvoyant Value and the Value Effect and Clairvoyant Value II: The Growth/Value Cycle written by Robert Arnott, Feifei Li and Katrina Sherrerd at Research Affiliates - effectively knock the "wisdom of the market" in pricing stocks. The papers take a time-traveling investor back to 1956 and give him perfect powers of foresight (thus the word clairvoyance in the title) to know exactly what actual cash flows, mainly dividends but also buyout premiums upon takeovers, the companies of the S&P 500 would have distributed from 1956 to the present day (1956 was chosen because that is when the S&P data starts). The reasoning is that a company is ultimately only worth what it gives back to an investor. A 1956 investor with perfect knowledge of the future (at least up the present) would only have been willing to pay the discounted net present value of the cash flows, which includes the price today as the best available terminal value. The second paper examines whether the 1956 start date somehow was unique. It was not - in the long term of 20 or more years the market is always shown to have overpaid in terms of realized value.

Note that the authors all work at Research Affiliates which sells its fundamental indexing methodology for weighting stocks in a portfolio, claiming this this does better than cap-weighting (and which, to give full disclosure, I too am convinced is a better method, to the extent of switching from cap-weighting to such investments myself). Unless they have fudged the numbers, which I doubt, and unless their reasoning of using discounted actual cash flows to establish realized value is wrong, the results must be accepted.

Some of the results (using JPM page numbers in the pdf):

• the market consistently picks out which companies will grow faster, better even than a strategy based on weighting on company size fundamentals, but the market really overpays for that growth - by about 50%! (p.24, paper 1)
• this conclusion holds for the vast majority of starting years from 1956 forward - for 20-year forward views of future cash flows, only in the three years 1964-66 did the market underpay for growth stocks (those defined as having multiples of metrics like Price/Earnings, P/Sales, P/Dividends, P/Book value)
• but, "It takes a long, long time for the market to correct pricing errors relative to Clairvoyant Value, because Clairvoyant Value cannot be known for a long, long time." (Clairvoyant Value is the value the prescient investor would have been willing to pay) (p.148, paper 2)
• the over-payment for growth stocks is the same whether the company is large or small (p.149, paper 2)
• the spread for over-payment is getting worse in recent years, not better (p.149, paper 2)
• a portfolio based on the economic size / fundamental weighting of companies doesn't do nearly as well as the portfolio based on clairvoyance, not a surprise given that the clairvoyance value is based on perfect foresight of future cash flows, but it does substantially better than a portfolio based on cap-weighting (p.151, paper 2); the annual return difference of company size weight vs cap-weight is 1.3% according to exhibit 4, paper 2
• "... a Cap Weighted portfolio puts the majority of money in stocks that subsequently prove to have been overvalued." (p.155, paper 2)
• the amount of overpayment for growth stocks and underpayment for value stocks has varied considerably through time i.e. there have been bubbles! When the gap between growth and value is the highest, that's the time to invest in value stocks since their returns will be much better in the subsequent time. The converse is true too - when there is a very small gap between value and growth, it is time to buy growth stocks since they are destined to do much better in the for some time following. (pp.155 and 156, paper 2)
  

To me, it all adds up to more evidence against cap-weighted indexing as an investment strategy. Put your money in a cap-weighted index and you will suffer long-term under-performance compared to RAFI or other non price-biased portfolio selection methods. It is a shock for those who believe that the market is on the whole right to learn that the market for the past 50+ years has been over-paying for growth stocks in a chronic, virtually continuous, general, excessive and non-accidental manner. And, it seems to be getting worse, not better. Caveat emptor.