Executive Summary

In order to help satisfy terms of the Global Settlement, quantitative stock research, primarily driven by computerized statistical analysis, is now actively competing with Wall Street’s traditional, qualitative research, largely driven by the analyst’s experience and insights. This raises some pressing questions:

·      Which approach is preferred by individual investors?

·      Which has the better track record historically?

·      Which is better equipped to provide investors protection from conflicts and biases, a primary goal of the global settlement?

In a quest for answers, this paper relies on three studies:

Study #1.    Investor survey of qualitative and quantitative approaches. We surveyed 1,203 individual investors and found that, although they leaned modestly toward quantitative approaches, they did not have a strong preference for a particular methodology or reporting format. Instead, the responses seem to indicate that their overriding desire is for a good track record to help them make money and protect their capital.

Study #2.    Performance of qualitative and quantitative insurance company ratings. Although in a different field, the competitive battle among various insurance company rating firms in the early 1990s sheds light on today’s debate. Tracking their ability to warn of future insurance company failures, we find that:

·      qualitative rating systems were accurate 13% of the time;

·      quantitative rating systems were accurate 92.3% of the time; and

·      one rating agency maintained both kinds of systems in parallel. Its quantitative ratings were among the most accurate in the industry, while its qualitative ratings were among the least accurate.

Study #3.    Qualitative vs. quantitative stock ratings. Based on a similar study of stock research firms in 2002, we find that:

·      qualitative methods were accurate in 9.1% of cases studied, and

·      quantitative methods were accurate in 45% of the cases.

Discussion and conclusions: In both historical studies, quantitative methodologies greatly outperformed qualitative methodologies in protecting investors. However, the primary reason may not be intrinsic to the methodologies. Rather, these results may be due primarily to the fact that the greater subjectivity of qualitative methods makes the process more vulnerable to biases and conflicts from a wide variety of sources. Among others, these can include direct or indirect income from rated companies, other business ties with rated companies, personal securities transactions in rated companies, and bearish or bullish orientations in general.  

In contrast, due to the greater discipline imposed by quantitative models, they are better equipped to act as a protective shield against such conflicts. In addition, quantitative models can:

·      serve as a central clearing house for a wide range of inputs by both qualitative and quantitative analysts;

·      help ensure consistent output among different companies, across industries, and over time; plus

·      replicate the thought processes of a qualitative analyst with a series of decision-making trees.

Recommendations:

1. Qualitative researchers should be more firmly grounded in quantitative analysis.

2. Quantitative researchers could benefit by adding a layer of qualitative analysis.

3. Buyers of research should seriously consider not only a firm’s independence but also what procedures and guidelines are in place to guard against other biases that may seep into the ratings process.

4. Regulators should shed any prejudice they may still have regarding quantitative research, fostering an environment in which investors can gain access to ratings generated by a broad diversity of methodologies.

5. All producers and users of research must never ignore the strategic advantage of quantitative approaches in the role of capital preservation, without which long-term investing success is unlikely.

Introduction

Thanks to the Global Settlement, many issues with respect to stock research and ratings have been largely laid to rest:

We know that conflicts of interest on Wall Street create bias.

We know that bias can harm investors.

And we know that removing conflicts and bias from the research process is a paramount goal.

But there’s one big issue remaining: To achieve that goal, what kind of research do we really want?

·      Do we want qualitative research that’s largely driven by an individual analyst’s years of experience and judgment?

·      Or, do we want quantitative research, primarily driven by computerized statistical analysis?

The goal of this paper is to make progress toward some answers by highlighting the results of three separate studies that pertain directly to these issues:

• The first study is our survey of investor preferences;

• the second study is a performance review of qualitative vs. quantitative  approaches in another industry that is directly relevant to this discussion; and

• the third is a performance review of qualitative vs. quantitative equity research in terms of its ability to protect investors from downside risk.

Each study represents merely a first step toward throwing new light on this vital issue.

 

Study #1.    Investor Survey Regarding Qualitative vs. Quantitative Equity Research Methods and Reports

Which approach is better — qualitative or quantitative?

Naturally, if you ask most Wall Street research analysts, they’ll probably tell you they prefer the qualitative approach. No one wants to be replaced by a computer.

If you ask decision-makers at major Wall Street firms, you’ll likely get a similar answer. Qualitative research has been the standard for many years, and it is the research that many still seem to favor.

But what about individual investors, the ultimate users of the research? Has anyone asked them what they want? Actually, we have. Two weeks ago, we e-mailed a survey on this issue to over 24,000 retail investors.

The sampling techniques were not scientific. However, we were able to reach investors with a broad range of incomes and interests. We received 1,203 completed surveys, representing a 5% response rate.

In the survey, we asked each respondent to rank the importance of various quantitative and qualitative factors or features relating to two aspects of the research:

1.       The researcher’s methodology for coming up with a recommendation, and

2.       The research reports for communicating and explaining the recommendation.

For each of these two categories, we provided an even balance of six clearly qualitative approaches and six clearly quantitative approaches, listed in alphabetic order. Plus we asked two general questions, including one regarding the analyst’s track record. The text of the survey is in the appendix, and the average scores are in Table 1 below.

1. Investor Preferences Regarding Equity Research

(average scores of all respondents)

	Qualitative	Quantitative	General
Methodology	3.59	3.72
Reports	3.62	3.81
Track Record			4.07

Scale: 1 = viewed as least important; 5 = viewed as most important

Data: Weiss Ratings, Inc. survey, February, 2004

Each respondent was asked to score qualitative and quantitative approaches on a scale of 1 to 5, with 1 representing “least important” and 5 representing “most important.”

In response to our first question — regarding the analyst’s methodology — the average score for qualitative factors was 3.59 compared to 3.72 for quantitative factors. Likewise, in response to the second question — on the content of research reports — the average score for qualitative information was 3.62 compared to 3.81 for quantitative.

Thus, the results show that, although there was a modest leaning toward quantitative approaches, respondents did not have a strong preference regarding either the methodology or the reports. Respondents did have a relatively stronger opinion, however, on another level: The highest average score of all — 4.07 — was recorded for track record, regardless of methodology used.

In effect, investors seem to be telling us: We don’t have a strong preference regarding how you reach an opinion on a stock or how you explain it. What we do care most about is whether or not your opinions are right or wrong.

In other words, the inference we derive from this survey is that individual investors do not strongly favor any particular methodology or any particular reporting scheme. Their overriding concern is to make money and protect their capital.

With that in mind, let’s narrow the debate down to two issues:

Issue #1.  Which approach — qualitative or quantitative — is less prone to bias and conflicts?

Issue #2.  Which approach is more likely to provide better performance overall for investors, helping them not only to grow their wealth but also to avoid serious losses.

To help address these issues, we conducted two historical studies. In each study, we compared qualitative and quantitative ratings systems. And in each, we asked the question: Which rating approach had the better track record in protecting consumers or investors from downside risk?

We recognize that downside risk is only one side of the equation. But we feel it is an extremely important one. Almost any research can get passing grades in stable times or in a rising market. We believe that the tougher test is how research holds up in unstable times or declining markets. Needless to say, investors can generally be successful even if they make less money. But few can recover if they suffer devastating losses.

Our next study is in a different field but with direct relevance to this topic.

 

Study #2.    Performance Review of Qualitative vs. Quantitative Approaches To Insurance Company Ratings

Much like today, the field of insurance company safety ratings was in flux in the early 1990s, with two competing approaches: A qualitative approach and a quantitative approach. And much like today, there was heated debate regarding which would provide better overall performance. Five major U.S. rating agencies were in the field, divided as follows:

·      Three of the agencies used predominantly a qualitative approach. These agencies argued that analysts must talk to management, must review a series of unquantifiable factors, and must allow each analyst’s intuitive feelings about a company’s prospects to play a significant, often overriding, role in forming an opinion.

·      One of the agencies used primarily a quantitative approach, with qualitative aspects considered mostly in exceptional situations. This agency argued that too much leeway for judgment by the individual analyst could potentially prejudice the results, even in the absence of conflicts of interest.

·      Plus, there was also one other well-established agency that had two separate rating systems — one predominantly qualitative and one purely quantitative. We will refer to this agency as “Agency D.”

In 1994, the U.S. General Accounting Office (GAO) published a landmark study comparing the ratings performance of all five of the rating agencies¹. However, at the time the GAO initiated its study, it did not have adequate data on the quantitative ratings of Agency D. Therefore it covered strictly its qualitative ratings.

Subsequently, Weiss updated the GAO study, using the same methodology as the GAO, but, this time, including Agency D’s quantitative ratings as a separate item. Thus, the Weiss study covered the performance of six ratings systems overall.

Focusing on large failed companies, the GAO’s basic performance metrics revolved around two questions:

·      Did the agency provide adequate warning to consumers of subsequent insurance company failures?

·      With how much lead time?

The results of the GAO and Weiss studies are summarized in Tables 2 and 3. Between the two studies, there were 11 large companies that failed, covered by the six ratings systems.

Qualitative insurance company ratings

Table 2 covers the four qualitative systems. The figures shown are the number of days between the date of failure and the date each agency issued its first warning or “vulnerable” rating, as defined by the GAO. Negative numbers indicate the number of days before the failure. Positive numbers indicate the number of days after the failure.

As an illustration, Agency A issued a warning six days before the failure of Company 1, one day after the failure of Company 2, two days after the failure of Company 3, etc.

2. Qualitative Ratings Assigned to Large Failed Insurers

Number of days between issuance of first warning and failure.

Minus sign = warning issued before failure. Plus sign = warning issued after failure

Large Failed Companies	Agency A	Agency B	Agency C	Agency D
Company 1	-6	-422	-41	-190
Company 2	+1
Company 3	+2			-6
Company 4	+5			-3
Company 5	Never			+351
Company 6	+3	+2		Never
Company 7	0		+2	+10
Company 8	0		+2	+10
Company 9	+5
Company 10	+4
Company 11	+3
Adequate Advance Warning	0%	50%	33%	14%

Data: U.S. General Accounting Office, rating agencies, state insurance commissioners

We define “adequate advance warning” as a warning issued at least one week prior to failure. It is assumed that less than one week is inadequate time for the warning to be published in major venues, disseminated to the public and acted upon.

Based on this definition, Agency A did not issue an adequate warning for any of the failed companies. Agency B issued a warning in one out of two of the companies it covered, Agency C issued a warning in one out of three, and Agency D’s qualitative system issued a warning in only one out of seven cases. 

Please note that, even if we define advance warning as any time before the date of failure, the results do not improve significantly. Also note that on two occasions, an agency dropped coverage of the companies around the time of the failure and never issued a warning.

Quantitative insurance rating systems

Table 3 summarizes the results of the two quantitative rating systems. Agency D’s quantitative system issued an adequate warning in three out of three of the cases and Agency E, in nine out of ten of the cases.

3. Quantitative Ratings Assigned to Large Failed Insurers

Number of days between issuance of first warning and failure

Minus sign = warning issued before failure

Plus sign = warning issued after failure

Large Failed Companies	Agency D*	Agency E
Company 1		-379
Company 2		-372
Company 3		-308
Company 4		-617
Company 5		-162
Company 6		-40
Company 7
Company 8		+6
Company 9	-74	-134
Company 10	-994	-1152
Company 11	-228	-621
Adequate Advance Warning	100%	90%

* Agency D ceased publication of its quantitative ratings once qualitative ratings

were published on the same company. Data: U.S. General Accounting Office,

rating agencies, state insurance commissioners. 

4. Qualitative vs. Quantitative Safety Ratings

Assigned to Large Failed Insurers

Methodology	# of ratings	# correct	% accuracy
Qualitative	23	3	13.0%
Quantitative	13	12	92.3%

 

The overall results of this study are summarized in Table 4. Counting the total number of accurate ratings in comparison to the total number of ratings issued, we find that:

·      The qualitative rating systems were accurate in three out of 23 cases, or 13% of the time, while

 

·      The quantitative rating systems were accurate in 12 out of 13 cases, or 92.3% of the time.

Two additional factors support this pattern:

Larger sample: Not included in the tables above is a larger sample of failed companies discussed by the GAO. Using a sample of 30 failed companies which were exact cohorts (rated in common and simultaneously by both agencies), the GAO found that the quantitative agency was first to warn investors of pending troubles in 23 cases and the qualitative agency was first in 7 cases. Thus the quantitative approach beat the qualitative by a three-to-one ratio.

In response, the qualitative firm requested that its ratings scale be redefined so that its “B” and “B-” ratings (labeled “good” in its literature) could be construed as “vulnerable” and considered an adequate warning to investors. The GAO declined to accept this redefinition, but stated that if it did, the quantitative agency would still have eclipsed the qualitative agency by a factor of two to one.

Agency D: The agency that had both qualitative and quantitative ratings systems is an especially interesting case. Both of its ratings systems were running under one roof.  We can generally assume, therefore, that its analysts were potentially subject to the same or similar conflicts of interest at the firm level. And we can also assume that they had access to, or were hampered by, essentially the same strategic advantages or disadvantages the firm might have had in terms of its resources or knowledge base. Agency D had a policy of ceasing publication of its quantitative ratings as soon as a qualitative rating was published on the same company. Therefore, based on publicly available information, it is impossible to compare identical universes of companies rated in common by both of the agency’s systems. However, the record shows that:

·      The agency’s quantitative ratings were largely accurate. They provided ample advance warning of future difficulties for all three of the large companies that subsequently failed. Additional data not included in this paper further support that conclusion.

·      In contrast, the agency’s qualitative ratings were largely inaccurate. They provided advance warnings for only one out of seven of the large companies that subsequently failed.

·      Among the six ratings systems, Agency D’s qualitative ratings appear to have one of the worst track records, while its quantitative ratings appear to have one of the best track records.

Study #3.    Performance review of qualitative vs. quantitative equity research with respect to downside risk

In reviewing the historic performance of various stock ratings systems, there’s one issue that seems to be in danger of falling through the cracks:  Survivorship. The concern is that independent research providers or data aggregators may be purging the ratings history for those companies that have filed for bankruptcy and ceased to be traded on major exchanges. This is unfortunate for three reasons:

·      First, there were a large number of corporate failures in the first three years of the new century.

·      Second, since corporate failures typically wipe out 100% of the shareholder’s capital, they can have a significant impact on overall portfolio performance.

·      Third, from an academic point of view, bankruptcy is a definitive event which provides a solid testing ground for the accuracy of the ratings in general, and for sell signals in particular. Although this is conceptually different from the metrics currently under discussion, we believe it sheds a new perspective on the debate.

Qualitative Stock Ratings

Table 5 is based on a study of companies that failed in the first four months of 2002 and that were covered by at least one qualitative firm².

5. Qualitative Stock Ratings on Companies Filing for Chapter 11 Bankruptcy

(includes all covered companies filing for bankruptcy in first four months of 2002)

	On Date of Bankruptcy Filing		6 Months Before Bankruptcy Filing
Ratings Issued	# of Ratings	%	# of Ratings	%