2.5 Evaluating

Topic 23: Evaluating Claims
Topic 24: Evaluating Reasons
Topic 25: Cognitive Bias
Topic 26: Fallacy

Topic 23: Evaluating Claims

Suppose some claim is put before you. Suppose, for example, that somebody says

• There is no largest prime number.
• Ground rhino horn cures impotence.
• Wooden flutes sound better than metal ones.

Should you agree? Well, only if the claim is true. So you need to think about whether the claim is true. That, roughly, is what we mean by evaluating a claim: making up our mind as to whether it is true or false.

Note that you will normally be evaluating a claim from your own perspective. That is, you are making up your own mind about it, given everything you know, your situation in the world, etc. You are not putting yourself in the shoes of the other person, and asking whether or not they believed it.

Truth and Confidence
In most cases – especially in complicated or important matters – you will not be able to know with absolute certainty that a claim is true, or that it is false. As a practical matter, the challenge is not to come up with a simple True/False verdict, but rather to assess how plausible the claim is, or put another way, what level of confidence you should have in it.

To evaluate a claim is to determine how confident you are that it is true (or false).

You may decide it is definitely true, highly likely, true ‘beyond reasonable doubt,’ etc.

Rational Evaluation
Very often, people decide whether a claim is true on the basis of an impulse, hunch or gut feeling. They just ‘know’ it is true in a flash, without further reflection. On other occasions, people accept claims as true on the say-so of some person in whom they place unquestioning faith, such as a parent, a religious leader, or a salesman.

Critical thinkers, by contrast, evaluate a claim carefully assessing it in the light of all relevant considerations, i.e., all relevant arguments plus any basis which may happen to apply. They attempt to determine what level of confidence a rational person should have in the claim, given the evidence available, and then accept that assessment, regardless of what they are being told, whether by another person or by their own gut feelings.

Balance of Considerations
Assuming all the relevant evidence is already at hand, evaluating a claim amounts to little more than assessing the balance of the considerations. Roughly, accept the claim to the extent that the balance of considerations supports it; reject the claim to the extent that the balance of considerations opposes it; and if the balance is more or less even, take no stand.

This makes it sound easy, but of course assessing the balance of considerations can be an involved matter, since it involves determining the strength of every individual argument and solidity of every basis bearing upon the claim, and weighing up the results.

Critical Inquiry
More typically, we are not presented with all the evidence, and may need to conduct our own investigations before making our assessment. Ideally, then, evaluating a claim involves the following major steps:

1. Ask whether all relevant considerations have been identified and evaluated;
2. If not, then actively seek out and evaluate all additional relevant considerations;
3. Determine the balance of considerations;
4. Assess your confidence in the claim, given the balance.

Needless to say, this can be quite a task. The extent to which we undertake it properly will depend on many things, such as how important the topic is, how much time we have, etc.

See also: Topics Proposition and Basis

Topic 24: Evaluating Reasons

As we all know, reasons can be good or bad, better or worse. To evaluate a reason is just to assess how good or bad it is. In this topic we discuss briefly what goodness in a reason amounts to, and how you go about assessing it.

A reason is a set of claims providing evidence for some contention. The most common metaphor we use to describe this relationship is the notion of support – a reason supports a contention. Extending the metaphor, we say that a good reason provides a large amount of support, or, in our preferred parlance, strong support. Thus,

To evaluate a reason is to decide how strongly the reason supports the contention.

Truth and Relevance
There are two fundamental requirements for a reason to support a contention. First, the reason’s premises must be true. Second, they must be relevant to the contention. Consider the following:

There are McDonald’s restaurants in both Turkey and Syria, so they will never go to war with each other.

Figure 2.25

This little argument presents a rather surprising reason for thinking that Turkey and Syria will never go to war. The reason would obviously be worthless if there were not in fact, McDonald’s restaurants in both countries. But even if there are, more would be needed. What is the connection between having McDonald’s and going to war? The information offered as evidence must be somehow relevant to the claim made in the contention. In this case the relevance is not obvious at all.

Evaluating Strength
It follows that there are two major aspects to strength of a reason. Strength is a function of the level of confidence one has in the truth of the premises, and the degree of relevance those premises have to the contention. Or, in five words or less:

Strength = Truth + Relevance

Thus, we can break the process of evaluating a reason into three main steps:

1. Evaluate truth of premises. Assess how confident you are that the premises are true, in the light of all relevant considerations. For more on this, see Evaluating Claims. If you find that one or more premises are not true, then you can stop, since the reason has already failed a key requirement. Otherwise:
2. Evaluate relevance to contention. Assess the extent to which the information presented in the premises makes it more likely that contention is true. A good general technique for this is scenario testing: the premises are relevant to the extent that it is difficult to think of plausible scenarios in which the contention is false, given that the premises are true.
3. Evaluate strength. Combine the judgments you made in steps 1 and 2 to yield an overall assessment of the strength of the reason.

Degrees of Strength
Reasons vary in their strength, from offering no support at all to conclusively proving the contention. In between, there is a potentially infinite number of shades of gray. It is possible to use numerical scales to specify strength; 0 to 1, 0 to 10, and 0 to 100 have all been used. However, for practical purposes a simple qualitative scale seems to be the most workable. In the Rationale approach a reason can be worthless, weak, strong, very strong, or conclusive.

Critical Questions
To help you structuring your evaluation process you can download two hand-outs with the questions you should ask yourself when evaluating a map: Figure 2.26 Critical Questions for Reasoning Maps. For the pdf click here.

• Figure 2.27 Critical Questions for Analsysing Maps. For the pdf click here.

On the back side of the pdf's and in the Outline of the maps in Rationale you will find the critical questions needed for evaluating the solidity of Bases.

See also:

• Topic Reason
• Three theories of truth in the Tutorial Reasoning for Knowledge

Topic 25: Cognitive Bias

You have $500, but you desperately need $1000; nothing less will do. You go to the casino. Question: should you bet all your money in one go? Or should you split your $500 into, say, ten lots of $50, and bet those one at a time?

If you are well-versed in the fundamental principles of probability, you will know the answer to this. To have the most chance of walking out with $1000, you should bet all your money in one go. The more gambles you make, the lower your overall probability of reaching your target.

Trouble is, most people are not masters of probability. They do not have expert knowledge of calculations to determine the answer to rely on; instead they fall back on their hunches, or ‘intuition.’ Now the interesting part is that most people’s intuitions here go the same way, which happens to be the wrong way. Most people say they would break the $500 into smaller amounts, and bet those one by one. They sense that this approach is ‘safer’ – which in one sense it is, because if you lose on the first bet, you still have some money to bet with, and hence a chance of reaching your goal. The trouble is that this kind of safety is not what matters most in this artificial situation.

People unwittingly sacrifice some likelihood of attaining what they really need to satisfy some basic emotional needs. This is an example of a cognitive bias:

Cognitive biases are universal, innate tendencies for humans to think in certain ways, ways which often result in poor judgments.

All of us, in the first instance, do our thinking with the brains we have, which are the brains bequeathed to us by an evolutionary process stretching over millions of years. That process shaped our brains to help us survive in the particular contexts in which our ancestors found themselves. It gave us some extraordinarily subtle and powerful cognitive abilities, such as the ability to recognize in a flash that a slender, twisting shape in a tree is a snake, not a branch.

Evolution did not, however, build into our minds the abstract laws of logic and probability, or the ability to apply such laws to any situation whatsoever. And it did not build into our minds the particular capacities we would need to respond correctly to every situation we encounter in the modern world, a world different in so many ways from the environment in which we evolved.

Consequently, those innate, built-in, fast, and intuitive modes of thinking which work so well so often also, reliably, lead us astray in certain situations which can occur quite frequently in the environment we now inhabit. In casinos, for example. Casinos are institutions designed to ruthlessly exploit the mismatch between the minds nature gave us and the mathematical principles which in fact govern our financial well-being.

Research on Cognitive Bias
Cognitive biases have been studied quite closely for decades now; this study was given a major boost by the pioneering work of famous psychologists Daniel Kahneman and Amos Tversky. The research community has identified literally dozens of cognitive biases, affecting our thinking in many different domains, from judging probabilities to identifying causal relationships and explaining other people’s behavior.

There is, of course, considerable debate in most cases about the precise nature and extent of the supposed bias; you would expect nothing less from a vigorous research enterprise. Yet the general lessons are clear:

• Our minds are riddled with biases, some quite subtle, others very strong, but all typically invisible unless we reflect on our thinking with special care.
• These biases are universal (they affect everyone), and innate (they are with us from birth, and are ‘wired in’ to the structure of our brains.
• In some cases, biases can be overcome with special training. In most cases, however, they are ineradicable features of our thinking. With training we can learn to circumvent or compensate for them, either as individuals or as communities. Indeed, the whole of science itself can be seen as a vast, sophisticated mechanism for achieving genuine know- ledge in the face of the limitations inherent in our cognitive equipment.
• The advanced critical thinker is familiar with most or all cognitive biases; actively monitors her thinking to detect their influence; and practices and applies relevant techniques for ameliorating the sinister effects of these invisible and unwelcome mental tenants.

See also: http://en.wikipedia.org/wiki/List_of_cognitive_biases

Topic 26: Fallacy

Reasoning can go wrong in any number of different ways. Sometimes an argument goes wrong in a totally unique and original way. But life is not always that exciting. Often arguments go wrong in a familiar, standard or common way. In that case, we say that the argument commits a fallacy.

A fallacy is a common pattern of reasoning that is always or at least commonly bad.

(The qualification ‘at least commonly’ is there because sometimes an argument fitting the pattern is not in fact a bad argument; it just has the same form as others which (usually) are bad.)

Various Uses of the Word ‘Fallacy’
A fallacy is a kind or form of bad reasoning. However, the word ‘fallacy’ is also often used to describe a particular argument that happens to fit one of the standard fallacy forms. In such cases to say ‘that argument is a fallacy’ is shorthand for ‘that argument commits a fallacy’ or ‘that argument is an example of a fallacy’. People sometimes use the term as a colorful way of abusing an argument. In such cases, to say ‘that’s a fallacy!’ is just to say ‘that’s a bad argument.’ You should also be aware that the nature of fallacies is a subject of some dispute among logicians, and you might find somewhat different definitions of the term ‘fallacy’ if you look in other textbooks.

Names for Fallacies
Since fallacies are common patterns of (commonly) bad reasoning, and it is useful to be able to think and talk about bad reasoning, fallacies have been given names. Many of these names are quite arcane, usually because they are derived from Latin originals, e.g. petitio principii (begging the question) and argumentum ad hominem (personal attack).

An example of a fallacy: one common problem in reasoning is to use a key term one way in a reason, but another quite different way in the conclusion:

Lesbians are not normal, because they are only a small minority of the population, and a minority isn’t normal.

If the conclusion of this argument is to be interesting at all, then the word ‘normal’ must mean something like OK or not deviant. But in the reason, the term ‘normal’ really just means in the majority. The argument tries to get you to accept a controversial conclusion on the basis of an uncontroversial premise, craftily changing the meaning of a crucial term along the way.

Since this kind of problem occurs quite often, it has been given a name. It is called the fallacy of equivocation.

What are the Fallacies?
Dozens of fallacies have been identified and named. Good critical thinkers are familiar with most of the fallacies, and carry that knowledge in their heads, ready to apply it whenever they engage in reasoning. For people who have not yet memorized all the fallacies, there are some very good lists on the internet. One of the best is The Fallacy Files – http://www.fallacyfiles.org/.

Why Learn About Fallacies?
Knowledge of the fallacies is helpful in evaluating arguments. Instead of evaluating every argument laboriously ‘from scratch,’ you can often instantly recognize that an argument is committing a fallacy of a particular kind. Knowledge of fallacies will also help you avoid mistakes in your own reasoning. Finally, (we didn’t say this, but…) you can sometimes use fallacies to your rhetorical advantage. Many fallacies are common precisely because they are seductive, i.e., many people naïve in the art of argument will fall for them time and again. That is why politicians, for example, ‘commit’ so many fallacies.

See also:

• Fallacies in the Tutorial Reasoning for Knowledge