A Guide for Investment Analysts: Working with Historical Market Data

Q: How far back does the US stock and bond record go?

A: Good data series are available from the 1790s.

This is the first of three posts that are pitched at analysts interested in working with older historical data. It is easy to download a historical data set and proceed immediately to statistical analysis. But pitfalls lurk for the unwary. The farther back in time, the more different the circumstances. Context matters when interpreting results.

I also have an eye on investors who enjoy reading historical accounts. I see a lot more of these accounts in the press and in white papers than when I first started probing the record 15 years ago. These posts will lift the hood — or turn over the rock — to give you a better understanding of what underlies these accounts.

I’ll start by dating and defining the fully modern era and then trace the roots of the modern era to the 1920s. Later posts will push further back in history.

Full-Fledged Modernity: the 1970s

Stocks

From the end of 1972 the Center for Research into Security Prices (CRSP) includes in its database stocks trading over the counter on the NASDAQ. It had added stocks trading on the AMEX as of 1962.

Before 1962, there is no true Total Stock Market Index to track. Indexes labelled as “the market” include only stocks listed on the New York Stock Exchange. Which is to say, include only the largest firms that are able to meet the strict listing standards of the NYSE. Before 1972, most of the smallest firms in the United States and those with the weakest financials — literally thousands — were excluded from the historical record.

Accordingly, factor analyses before this period are suspect. “Small stocks” were the smallest of the largest stocks, those able to qualify for NYSE listing.

Banks and other financial service firms are not tracked in CRSP data before 1972. These were not listed on the NYSE.

Bonds

Only by this point is there a regular issue of Treasury bonds and notes spread through the maturity spectrum. As described below, at the outset of the modern period in the 1920s, most Treasuries were long issues, and issuance was irregular. Years could go by with no new Treasury issues.

Only in the 1970s does a Total Bond Market index appear, with all traded maturities included and with government and corporate issues combined.

Roots of the Modern Era: the1920s

You may have read the phrase, “Since 1926, stocks have returned …” and idly wondered what happened in 1926 that was so special.

The short answer: nothing. The December 1925 anchor for the Standard & Poor’s index and for the total stock market index published by CRSP represents an arbitrary starting point set by time and cost limitations facing early data compilers.

Nonetheless, for the moment, precise data at the level of individual stocks — daily price change, ex-dividend day, splits, mergers and acquisitions, other corporate actions — only extends back to January 1926. Before that point, the analyst must typically work with index data, over a monthly interval at best.

With that caveat, the true point of beginning for the modern period was around the end of World War I. Before the war, the markets looked very different, especially the bond market. The available data for interpreting market returns, even at the index level, also begins to thin out. Whereas from January 1919, a host of macro- and micro-economic data series can be found in Federal Reserve publications.

By the 1920s:

Hundreds of stocks traded on the New York Stock Exchange, which, decades prior, had established its predominance over all other US exchanges. Almost all the largest firms in the US were listed on the NYSE.

These stocks were distributed across more than a dozen distinct sectors, including transportation, utilities, diverse industrial sectors — including durable goods and packaged goods manufacturers — and emerging services like chain retailing.

A deep and liquid US Treasury market had emerged following World War I.

However, still missing as of 1926 are some elements that the 21st century investor takes for granted.

For stocks:

Again, banks and most financial services firms did not trade on the NYSE and were not included in either the CRSP or the S&P indexes for the period.

The Securities and Exchange Commission did not yet exist (1935), nor did the Investment Companies Act of 1940. There were few regulations to prevent market manipulation or the dissemination of false or self-interested information.

The Federal Reserve does not yet regulate the margin required to buy stock. Depending on the customer, stock, and brokerage firm, a margin as low as 10% might have been all that was required to trade.

For bonds:

Only a few maturities were available for Treasuries, most of them long. Only during the 1930s, as the Treasury attempted to alleviate the Depression with multiple issues of varying length, did the maturity spectrum begin to be populated.

There was no regular schedule of offerings, at any maturity. In fact, for most of the 1920s the government was engaged in paying down the debt accumulated from the war, with new offerings designed primarily to refinance that debt, particularly the short-term notes, into an extended maturity schedule convenient for the government.

The mindset of this era approached government debt as a regrettable exigency of war, to be worked down and paid off as peacetime conditions permitted.

The modern Treasury bill, defined as a very short-term note, offered on a regular schedule and allowing amounts to be rolled over indefinitely, was not inaugurated until 1929. 

Takeaways

There is now almost 100 years of data that permit comprehensive analysis of stock and Treasury return,  not much different from what the analyst could do over the past 50 or even 20 years.

But as soon as the analyst ventures back before the 1920s, data series taken for granted today begin to thin and disappear. Notably:

There was no Treasury bill, hence no good proxy for the risk-free rate, hence no opportunity to construct a capital asset pricing model (CAPM) regression, hence no ability to assess market beta. Indeed the CAPM only recently celebrated its 60-year anniversary.There is not a good range of Treasury maturities until the 1930s, hence little opportunity to study the Treasury yield curve or changes in that yield curve.In general, until even later — the 1960s — there is no regular offering of short-term or intermediate Treasuries. There is not even a regular offering of 10-year Treasuries. It had not yet emerged as the benchmark. Before the 1960s, to invest in bonds primarily meant to own long bonds.

For stocks, before the 1920s, there was little sector diversification.

In my next post, I will continue this history back past World War I. In the meantime, if you are ready to roll up your sleeves and get to work on the data, here are some sources for the modern period:

The Kenneth French Data Library 

Monthly data on the total stock market return (within the limits noted) and the risk-free rate (30-day T-bills) back to June 1926. Free for download. Updated after every June.

Data on a host of sub-divisions of the market, including the most common factors (size, value and more) and the major industry sectors.

The Robert Shiller Data Site 

S&P index returns monthly back to January 1926 (and before, see next post). Separate dividend series and price return series. Earnings series for computing CAPE (cyclically adjusted price earnings ratio). Monthly inflation for computing real returns.

Two caveats:

Shiller computes returns on the average of daily prices, not month-end prices. This constrains volatility and can produce quite different estimates of return over periods of ten years and less, relative to the standard month-end estimates.

Shiller’s government bond returns, presented as 10-year returns, are not based on the price of 10-year Treasuries, but on yield curve interpolations back to 1954, and then extracted from yields on longer bonds back to 1926.

The Stocks, Bonds, Bills & Inflation Yearbook

Data from 1926 to 1987 can be found in this free online copy of the 1989 SBBI at the CFA Research Foundation site, with series for large stocks, small stocks, long Treasuries, intermediate Treasuries, long corporate bonds, and T-bills.

Monthly data to 2023 are at Morningstar behind the paywall.

CRSP Data and Global Financial Data

Both maintain vast compilations of individual stock data behind a paywall. CRSP has over 25,000 stocks and all Treasury issues from 1926; GFD has data on a dozen international markets back as far as 1700.

Access to a University library subscription, most likely that of a major research University, is typically required to gain access to these databases.

International Databases

The Jorda-Schularick-Taylor Macrohistory Database tracks a smaller number of international markets to 1870, with macroeconomic series as well as asset returns. Dimson, Marsh and Staunton publish annual yearbooks describing international asset returns to 1900. Data series are behind the paywall at Morningstar.

READ PART II

READ PART III