This page is open to comments. There are no fixed limitations, but I do not make any promises on which comments are allowed. Feel free to test, what happens.
Over on Judith Curry’s blog, a somewhat random discussion started. It started with Fred Moolten posting an off-hand comment, and he later expounded upon his point. The basic idea is by just looking at a red tulip, one can gain information on the color of crows. If one wanted to know whether or not all crows are black, they could find out without ever looking at a crow.
This is counter-intuitive, and a number of disagreements sprang up. I decided to try to write a clear explanation as to why Fred’s argument was correct, but since the thread was already cluttered and this was all off-topic anyway, I decided to post my explanation elsewhere. The host here invited me to post my thoughts, and so here it is an explanation of why you can look at anything and gain information about anything else.
The argument being tested is All objects in group X have property y.
It is accepted there are a finite number of objects, and they are non-changing. The group which contains all of these objects will be referred to as group Z. Since there are a finite number of objects, there are a finite number of objects possessing property y, and there are a finite number of objects not possessing property y. They will be referred to as groups A and B respectively. The two groups are disjoint.
Accepting the above, we know the argument is true if group X is a subset of group A. If any member of group X is a member of group B, the argument is false.
Assuming no other knowledge, the chance of any member of X being a member of B is B/Z. This chance increases as the size of X increases giving the total chance of the argument being true as (B/Z)^X. The chance of the argument being false is 1 – (B/Z)^X.
Given these simple equations, certain observations can be made. Z has a positive correlation with the chance of the argument being true. B has a negative correlation (as does X). This means if B decreases in size, the chance of the argument being true increases. This means if members of B can be determined not to be members of group X (thus being removed from the analysis), the chance of the argument being true increases.
This can be directly related to crows and their colors. Group X is crows, and y is the color black. Group B is all non-black objects, including red tulips. Observing a red tulip means it can be removed from group B for this analysis, and thus, the size of group B is decreased. This increases the chance of the argument (all crows are black) being true.
Of course, there is little practical value in examining flowers when you want information about birds. The amount of information gained is infinitesimally small, so it’s basically unnoticeable. It certainly doesn’t compare to the fact we can find pictures of white crows with a quick Google search. On the other hand, it is a lot more fun.
Hopefully this can help clear up things for people. If there is anything you are confused about, or if you think I’ve missed something, feel free to let me know!
I realized an example might help explain why this works the way it works:
Imagine you have ten colored balls. You know three of the balls are red. You also know four of the balls have magnets in them. Suppose you are asked what the probability is that all the red balls have magnets in them. It’s a simple enough problem to solve.
Now suppose somebody hands you one of the blue balls, and you find it has a magnet in it. They then ask you, “What is the probability all the red balls have magnets in them?” Obviously you won’t give the same answer as before. You now know there are only three magnets amongst the nine remaining balls, so you’ll change your answer accordingly. The same is true if the blue ball you were handed didn’t have a magnet inside it (you’d get different results, of course).
So even without ever examining a red ball, you can gain information about the red balls. In the same way, you can gain information about crows by only looking at flowers. The only difference is how much information you can gain.
Here is my comment from the single email-message I sent to Brandon before the discussion was continued here:
My counterargument to the last point you make is that we do not know the sizes of the sets A, B, Z and X. They are finite, but the sizes are undefined and there is no reason to conclude that the expected size of the set A would be modified through observation of one member of set B.
All the formulas for probabilities are valid only, if we can consider the observation as a random representative of members of the set from which it’s picked.
My way of thinking is perhaps best expressed in terms of Bayesian probabilities and Bayesian inference.
Before the observation we have prior subjective probabilities
PZ(Z) which tells the PDF for the total number of objects in set Z
PA(A) which tells the PDF for the total number of objects in set A
PB(B) which tells the PDF for the total number of objects in set B
PX(X) which tells the PDF for the total number of objects in set X
P(y|X) which tells the likelihood that a random member of X has the property y
P(y|Z) which tells the likelihood that a random member of Z has the property y
P(y|Z) is essentially the expectation value of the ratio of the number of members of sets A and Z, which is close but not exactly the ratio of the expectation values of the sizes of the two sets.
When we make an observation of a member of set B that is not a member of set X the PDF’s PB(B) and PZ(Z) are modified, but PA(A) and PX(X) are not modified, when the observation is done as observation are done in real life. When we see something that does not tell anything on what is happening to issues that are not related to that observation by some mechanism of causation (either direct or through a common cause). Thus seeing a tulip tells nothing about crows. Not seeing a crow tells something about crows, but that was not the statement, which concerned only tulips.
Hempel’s problem is really a problem of philosophical description of induction, which is well known to be an unsolved problem. Common sense is not affected by that, it remains valid. We observe only that philosophers are in trouble, when they try to describe issues that are clear based on common sense.
=======
As stated in the last paragraph, I don’t see any paradox in the real world. I see only problems in the philosophical analysis of induction or “Studies in the Logic of Confirmation” as the title of Hempel’s original paper reads. This means also that I don’t believe that observing a red tulip has any effect at all on the certainty on claims on the colors of crows.
Upon reading that response again, I realize my e-mailed response was completely wrong. I misunderstood the point of the response, so what I said probably made no sense. Now that I’ve reread it, I’ll try responding again.
Nothing I’ve said relies on a change in the expected size of A. The question is merely, “Is X a subset of A?” Ruling out values in B doesn’t change the size of A, it just increases the chances of values lying within A. You are keeping one group constant while decreasing the size of the other. Taken to the extreme, group B could be reduced to 0 (an empty set). At this point, all of X must inherently fall within A, and thus the argument is proven to be true.
The reason this works is even though A hasn’t been affected, Z has been. The chance of X residing in A is related to the proportion of A/Z. Since Z shrinks with each tulip observed (remember, Z = A + B), the chance of X residing in A increases. If you put numbers into the problem, this becomes easy to see:
There are fifty black objects (A = 50). There are 50 non-black objects (B = 50). There are 20 crows (X = 20). The chance of a single crow being non-black is A/(A + B) or 50/100, or 50%. The chance of no crow being black is 50%^20.
Now then, if you see a red tulip, there are only 49 non-black objects left. If you observe 24 more red tulips, there are only 25 non-black objects left. This means the chance of a single crow being non-black is 50/(25 + 50), or 2/3. The chance of no crow being black is 67%^20.
The same principle applies no matter how large the sets involved are.
My point is that observations of the type done in the real world concerning tulips do not affect the possible number of red crows. They just add to the verified number of tulips and thus lead also to a growth in the estimated number of all objects. No number of observations of red tulips has any influence on the posterior estimate on the number of red crows.
The situation would be different, if we would have a container of all red objects and a way of removing objects from there one by one as long as the containers becomes empty, but this is not a correct description on, how observations are done in real world.
The “paradox” is due to an erroneous (or different from the one used in real world) description of the process of making observations.
Let’s say I know that there are 4 objects (for simplicity), each a tulip or a crow, but I don’t know how many red objects there are. To make it simpler, let’s also say I know that there are 2 tulips and 2 crows.
This means there are 16 possible worlds I might be in.
Using a 4-letter shorthand to denote the different worlds (the first letter denoting the colour of tulip 1, second letter the colour of tulip 2, third letter the colour of crow 1, and fourth letter the colour of crow 2, where letter R is red, and B is black), the 16 different possible worlds are :
In four of the worlds (04, 08, 12 and 16), all crows are black.
Before I make any observations, I don’t know which world I’m in. Assuming an equal likelihood of being in any of the worlds, I can say that the probability that I’m in a world where all crows are black is 4/16 = 1/4
Now I make my first observation. Let’s say the object is a red tulip. I now know that I’m not in worlds 13, 14, 15, or 16, so I’m in one of 12 possible worlds, in 3 of which (04, 08 and 12) all crows of black. So after my first observation, I can say that the probability that I’m in a world where all crows are black is 3/12 = 1/4 … no change.
I make my second observation. Let’s say the object is another red tulip. I now know that I’m not in worlds 05, 06, 07, 08, 09, 10, 11 and 12 either, so I’m in one of 4 possible worlds, in 1 of which (04) all crows are black. So after my second observation, I can say that the probability that I’m in a world where all crows are black is 1/4 … still no change.
Therefore finding two red tulips has not changed the probability of being in a world where all crows are black.
Let’s now go through the other possible observations.
Say my first observation is a black tulip. I now know that I’m not in worlds 01, 02, 03 or 04, so I’m in one of 12 possible worlds, in 3 of which (08, 12 and 16) all crows of black. Probability that I’m in a world where all crows are black is 3/12 = 1/4 … no change.
Let’s say my second observation is also a black tulip. I now know that I’m not in worlds 05, 06, 07, 08, 09, 10, 11 and 12 either, so I’m in one of 4 possible worlds, in 1 of which (16) all crows are black. Probability that I’m in a world where all crows are black is 1/4 … no change.
Let’s say instead that, following that 1st black tulip, my second observation is a red tulip. I now know that I’m not in worlds 13, 14, 15, or 16 either, so I’m in one of 8 possible worlds, in 2 of which (08 and 12) all crows are black. Probability that I’m in a world where all crows are black is 2/8 = 1/4 … no change.
Or let’s say that my first observation was a red tulip (13, 14, 15, and 16 eliminated as before, and prob. still 1/4), and my second observation is a black tulip. I now know that I’m not in worlds 01, 02, 03 and 04 either, so I’m in one of 8 possible worlds, in 2 of which (08 and 12) all crows are black. Probability that I’m in a world where all crows are black is 2/8 = 1/4 … no change.
So for all possible observations when examining the two tulips, the probability that I’m in a world where all crows are black remains 1/4, unchanged from the time before I’d made any observations.
Therefore examining tulips doesn’t change the probability that I’m in an world where all crows are black – it provides me with no extra information about the colours of crows.
Your example is clear and valid (you didn’t use Bayesian analysis as I do below, but that makes no difference to the conclusions). It represents one imaginary world, but it’s not the only possible imaginary world. It’s certainly intuitively much more sensible than another imaginary world, which leads to the other conclusion.
In the other imaginary world we have again 4 objects, but this time we know that 2 of them are red and 2 black, but we do not know, how many of them are crows and, how many tulips. We know, however, that we have at least one of each. Now we assume that the following cases are all equally probable (first the 2 reds, then 2 blacks)
This time all crows are black in worlds 10, 13 and 14, i.e. in 3/14 of cases. Now observing one red tulip, I know that the cases 01 and 02 are excluded. Observing one red tulip is twice as likely in cases 10, 13 and 14 than in the remaining 9 cases. Thus according to Bayesian thinking the likelihood of worlds with all crows black is now 6/(9+2*3) = 6/15. Observing another red tulip makes it certain that all crows are black, because there are only two red objects.
Thus under two different settings we have two different answers. The Hempel’s paradox is created, when the settings are not defined, and when different choices are implied at different stages of the analysis.
To conclude: Hempel’s paradox is caused by looking at a badly defined problem and including in the analysis implied additional assumptions, which are different in each approach. Using different implied assumptions leads to different conclusions as is expected.
This observation is significant as it tells that it’s impossible to use inductive logic, when we cannot perform well controlled experiments, where all needed assumptions are explicit and fully controlled. This ideal can be approached well enough in many laboratory experiments, but it cannot be reached in observational studies of real world.
Brandon commented on Judith’s blog that he wrote his opening message here, because this issue is not significant for the climate discussion. That comment is not really true, as all empirical knowledge on climate belongs to that class of observations, where the difficulties implied by Hempel’s paradox are present, and present to a very significant degree. They are actually at the heart of the disagreement on AGW.
I agree with Pekka Pirilä’s explanation of the problem, and it is why I tried to be explicit about all assumptions I was making. Changing the assumptions, unsurprisingly, gives different results. In oneuniverse’s example, he discarded an assumption which has been present in the examples given by Fred Moolten and myself. That is, we assumed there were no empty sets. In other words, we know there must be at least one of each class.
Discarding that assumption renders the rest of the problem moot as it allows for the possibility of vacuous truths. The uncertainty introduced by those is what causes oneuniverse to conclude no information is gained, and the removal of them is why Pekka Pirilä’s follow-up does show information gained.
As for the condemnation of inductive reasoning, I don’t agree using it is “impossible” in any situation. The less rigorous an experiment, the more difficult inductive reasoning will be, but I don’t see it ever being impossible. That leaves aside the fact even if you can’t draw any firm conclusions based upon inductive reasoning, you can still apply it to a situation. There is nothing wrong with reaching the conclusion, “I don’t know.”
As for the climate discussion, I’m not sure how you would apply this problem to it, hence why I said it wasn’t significant. It obviously has some relevance, as would any issue about reasoning, but I can’t think of any real parallels in climate science. Do you have any particular ones in mind?
Oh well, I wasn’t going to preempt more comment space on Judy Curry’s blog for this, because it was already too long, but since the discussion continues here, and is entertaining, I’ll add my two cents.
First, this issue is an exercise in logic, not ornithology. Neither Hempel nor anyone else is going to suggest observing all non-black objects to determine the blackness of crows, but as the wikipedia article indicates, the counter-intuitive nature of the conclusion that seeing a non-black non-crow or raven is confirmatory comes from the intuitive belief that it adds zero evidence, in contrast to the logical inference that it adds an extremely tiny amount of evidence that is not zero. To sum it up, it has zero practical value in evaluating the color of crows, but that is not the same as claiming it has no zero evidentiary weight. It does. Discussions of its practical significance for crow color would therefore not be germane.
The Hempel conclusion continues to appear irrefutable to me under the conditions he specified. As far as I know, this conclusion has not subsequently been refuted in a general sense by anyone over the course of more than 60 years, despite many analyses, but instead, discussants have described the underlying assumptions and in some cases suggested that different conclusions could emerge with different starting assumptions.
Two assumptions seem particularly worth noting. The first is that crows exist, but in the world we inhabit, that doesn’t seem to be a problem. The second involves the categories within which we investigate the black crow hypothesis. Four categories might be investigated: (A) All crows. (B) All non-crows. (C) All black objects. (D) All non-black objects. The blackness of all crows can be proved or falsified in theory by category A or D. However, that is not true for either B or C. B is of particular interest – looking specifically at all non-crows cannot tell us anything about the blackness of crows.
Here is where some ambiguity arises. When we observe a red tulip, are we operating in the informative category D (non-black objects) or the non-informative category B (non-crows)?
Since practicality is irrelevant, we must specify what assumptions we use in this hypothetical scenario. Hempel is correct when we specify that we have noted the color of an object either before or simultaneous with an observation of its nature – crow or non-crow. It’s red, and yes, it’s also a tulip, and so we have eliminated one possible example of red crows from category D. On the other hand, if we observe what is clearly a tulip at a distance without discerning its color, subsequently discovering its color will not put it into category D, but into the non-informative category B. This distinction does emphasize the arbitrary nature of the problem and the fact that it has no practical real-world application for understanding crows. It does not, in my view, invalidate the basic logical principle of equivalence that underlies the correctness of Hempel’s reasoning, as long as we specify that we are asking whether a red object we observe is a crow or something else. To pursue these distinctions further would, in my opinion, put too much weight on the practical importance of our observations, which we already agree would be negligible.
A final word about the different scenarios from oneuniverse and from Pekka for distribution of colors, crows, and tulips. Utilizing Pekka’s hypothetical scenarios, 3/14 involve all black crows, they are equally probable, and the probability of all black crows is 3/14, or 6/15 if we observe a red tulip. Here is where the Hempel logic clearly shows its value. Imagine that we start with those 14 possibilities, and then draw 1000 samples from this population of objects, and – Not A Single Red Crow is Observed. Is the probability of all black crows still only 3/14? This is a point that I tried to clarify in the Climate Etc discussion but probably didn’t explain too well. If we don’t know whether all crows are black, the evidence gained from our observations affects any prior probabilities we might have assumed. Even 1000 observations wouldn’t prove the non-existence of a red crow, but as has been discussed many times, confirmation isn’t the same as proof. Observing 1000 objects without seeing a non-black crow, while seeing many non-black objects of other types, would confirm (very very slightly) the blackness of all crows even though it obviously wouldn’t prove it. In Pekka’s example, even observing one red tulip is confirmatory, by increasing the prior probability from 3/14 to a posterior probability of 6/15.
The examples of Oneuniverse and myself tell that there is no paradox. What appears a paradox is rather sloppy use of logic. This is not irrelevant, because there is a wish to use logic also, when the setup does not allow for rigorous logic, and then we may have situations, where two equally plausible alternatives give indeed different results.
This example of tulips and crows is not such a case, because in this example one of the alternatives is clearly superior and the other should be abandoned as contrary to the natural setup. This faulty alternative is of course that of trying to make conclusions on colors of crows from observations concerning tulips. No reasonable real world setup allows for this interpretation. The original logic hypotheses are equivalent, but everybody interprets the statement that we see a red tulip to describe something not relevant to the colors of crows. By this I mean absolutely no influence, not that the influence is small.
As I have stated before, the related statement that we have not seen a red crow during a period of time, when seeing it would be possible, if red crows exist, is evidence against the existence of red crows. If we measure time by the number of red tulips seen, we may create artificially a connection between tulips and crows, but this means that we have introduced additional constraints that differ from those everybody imagines from the original description.
The value of Hempel’s discussion is in demonstrating that rules of logic work only, when the questions are set up precisely, and that this is often difficult. It’s pity that his example is such that one of the alternatives works, while the other does not. Therefore it’s not a very good case, and too easily dismissed.
My own impression is that we have seen in Climate Etc numerous discussions, where similar problems of applying logic inference are present. They are not as clearcut and perhaps not really examples of “Hempel’s paradox”, but all too many have claimed that they have strong arguments based on logic, when the setup does not allow for drawing such conclusions.
“This faulty alternative is of course that of trying to make conclusions on colors of crows from observations concerning tulips. No reasonable real world setup allows for this interpretation.”
We can’t draw conclusions about the color of crows by observing tulips, but we can by observing red objects. I agree it’s not a “real world” problem, because no-one would try to ascertain whether crows are black by that approach, but I see no flaw in the logic itself. I’ll see if I can think of a plausible hypothetical real world example where the same logic would have practical implications, based on the equivalence principle, but most real world problems are solved more directly, rather than by equivalence, so it probably won’t be easy.
There may be real world situations, where the equivalence principle can be applied, but that requires that the setup is fully defined and such that the equivalence principle provides the information it is supposed to give.
The two simplified examples indicated that procedures that were supposed to be equivalent by the equivalence principle were in fact not equivalent. Thus this is not a trivial requirement, but requires careful analysis of the empirical setup. For a statistical analysis this means usually that the samples are randomized correctly. Starting from two formulation that are linked by the equivalence principle, it is likely that the most obvious ways of randomizing the samples would lead to different weightings and thus to different results in the statistical analysis.
The situation is analogous to that met in applying Bayesian analysis to the probability distributions of two continuous variables that are nonlinearly related. The common choice of flat prior distribution for one of the variables would correspond to a non-flat prior for the other as discussed recently in connection with the Annan and Hargreaves paper.
The two examples discussed above where extreme cases as the PDF of either “crowness” or color was fixed by 2 + 2 as the only possible combination. Therefore the equivalence principle failed totally (as it does also in the real world versions of crows and tulips). In some other problems the constraints may really be such that the principle works.
One way of expressing the observation is that the equivalence principle provides two methods to collect evidence on the same issue, but there is no general guarantee that the efficiency of either of the methods is not exactly zero. In the real world example of crows and tulips the efficiency is indeed exactly zero.
Pekka – I’m continuing this discussion simply because it’s enjoyable, and not with the expectation that proofs can emerge on a subject as amibiguous as this one.
You continue to state in one way or another that the “efficiency” of the equivalence principle in the Hempel paradox is “exactly zero”, but the Hempel argument, of course, is that it “seems” to be exactly zero because it is too small to have any practical utility or to be measurable, but is in fact very slightly greater than zero. Of course, the efficiency of observing crows directly would clearly be greater than that of observing non-black objects, and so each crow observation would carry more statistical weight than each non-black observation. Some inadvertent degree of non-randomness might reduce statistical power, but should not invalidate the principle unless observations of non-black objects deliberately avoided crows – the paradox as described implies that no attempt is made to avoid crow observations, and in fairness to Hempel and in the interest of seeing the paradox as an interesting exercise in inductive reasoning, we shouldn’t impose an additional arbitrary condition of selectively excluding crow observations. As far as I can see, the Hempel logic leads to the conclusion he draws, unless a specific flaw can be found, and none has yet been identified that I’m aware of. The hypothetical example you gave above with 14 different combinations appears to support Hempel, since observing a red tulip increased the probability that all crows are black.
Here is a more “realistic” hypothetical I’ve conjured up, suggested by the recent killing of Osama Bin Laden. I don’t know how realistic it actually is, but it may not totally misconstrue what our CIA in the U.S. does to avert terrorist operations. It goes as follows:
The CIA has been compiling a list of individuals requiring special surveillance because they pose a greater than average threat of engineering a major terrorist bomb plot. The list has now grown to 25,000, and is too large for adequate surveillance of all these individuals, so the CIA needs to reduce its size by removing names of individuals who don’t pose a major threat. There is evidence that participants in bomb plots have been trained at terrorist training camps, and the CIA hypothesizes that only trained individuals pose a major threat. Via equivalence, this translates into untrained = non-dangerous, or at least not dangerous enough for the limited special surveillance resources to be spent on them
Of the 25,000 listed individuals, 534 were known to have been trained, and 10,502 were known to be untrained. No data are available for the remainder. During the interval of interest, 31 major bomb attacks were attempted, and it is assumed that additional attempts are being plotted, although the number is unknown. Of these, 4 were attempted by individuals among the 534 who were trained. The remaining 27 were attempted by individuals known not to be on the list, and their training status is unknown. The evidence provides some correlation between training and bomb threat, but are these individuals more dangerous than the untrained ones? The CIA then reviews the 10,502 untrained individuals and finds that none of the 31 attempts involved any of them. It concludes that its use of limited surveillance resources would be optimized by removing them from the list, on the grounds that the data support the hypothesis that only trained individuals pose a bomb threat, while acknowledging that exceptions might be possible – i.e., the evidence provided confirmation but not proof. It also concludes that more effort must be applied to ascertaining the training status of individuals on the list.
Regarding the red tulip analogy, “untrained” = red, and “not involved in a bomb attempt” = tulip. Note that the analogy involves data focused on untrained status as the independent variable rather than on “not involved in bomb attempt”. Simply analyzing the latter (24,996 on the list plus unquantified millions not on the list) would not have provided evidence tending to exclude the possibility that untrained individuals were dangerous.
This is the best analogy I can imagine at the moment. It is slightly better constrained than Hempel in limiting the total number of individuals involved, which is why it works better, but I think the main principles are similar enough to illustrate the point about equivalence.
I realize that it’s possible to contrive scenarios where the equivalence principle might not work, but under all reasonable assumptions that the Hempel scenario is what it seems to be – a non-black object is observed either by chance or because one is on the alert for non-black objects, and turns out not to be a crow – it does appear to work.
Here’s another hypothetical A = B equivalence example more relevant to climate. I’ll phrase it in the “B” form:
Large changes in the concentration of a single atmospheric component that do not reduce emissions to space in the 12 to 18 um region as a fraction of total OLR are not rises in CO2 concentration.
I have repeatedly emphasized that the evidence obtained by the other choice of the uncertainty principle may be exactly zero and that it is indeed exactly zero in the setup understood by almost everybody from the setup of the original example. That setup guarantees that the tulips and crows are unrelated. Therefore the power is exactly zero.
We can also say that observation of one red tulip bears almost no influence on the expected number of red tulips elsewhere. Perhaps it does increase that number as it makes the observer think that red tulips are actually more common than he thought before.
I have stated that the non-observation of red crows in some place over some period decreases their likelihood, but for that it does not matter, whether a tulip that we might see during that period is red or black. These things are really totally unrelated and the evidence is really exactly zero. The way around the “paradox” is not by contesting this obvious fact, but by thinking where the alternative logic fails.
In this case it’s not difficult to see where it fails, as the alternative logic is based on the unjustified assumption that the total number of red objects would be somehow more fundamental than the number of red crows and that observing a red tulip would decrease the expected number of other red objects. These claims are assumptions, which may be true or false depending other influencing factors. In this particular example they are false, when the setup is understood as it is by almost everybody. The implied assumptions concerning the setup of almost everybody are contradictory to these other assumptions.
As the original question is not explicitly fully defined, it becomes fully defined only when supplemented with sufficient implied assumptions, and here we may choose, whether we accept the assumptions that almost everybody picks intuitively or some other set, where some influence remains. I have given an example of such assumptions in the proposal that we count time spent for observations by the number of red tulips seen. Then more red tulips means more time and more certainty that red crows do not exist.
Another example of a setup, where the other alternative would work is such that we hire a large number of people to search for red objects getting very many of such observations without knowledge given to us on, what the objects are. Then we pick randomly one of these observations and check, what it is. If it is a tulip, that has an influence on the likelihood of red crows. The point here is that we have a fixed set of observations known to contain exclusively red objects.
The implied understanding of almost everybody is, however, such that there is no correlation and the power of the alternative method is exactly zero. For a philosopher it should be enough that the power may be exactly zero and it certainly may under fully realistic conditions and natural implied assumptions.
I’m confused. How do you justify saying, “As the original question is not explicitly fully defined, it becomes fully defined only when supplemented with sufficient implied assumptions…”? Just what part of the problem lacks definition? Just what assumptions are being supplemented?
As far as I can tell, the formulation I provided is perfectly well-defined, and I can’t see any absent assumptions in the most recent example from Fred Moolten.
“unjustified assumption that…. observing a red tulip would decrease the expected number of other red objects. ”
I believe that as long as red objects are finite in number, observing one reduces the number left to observe. If that is a false assumption, then I’ll admit the remainder of the Hempel argument is wrong, but I’ll proceed below based on the truth of that proposition.
Rather than repeat previous points, I’ll leave the main argument for others to review from earlier comments. However, I do want to revise a previous statement I made regarding timing. Previously, I suggested that observing an object to be red and observing it also to be a tulip simultaneously or later supported the black crow hypothesis, but that observing it first to be a tulip (i.e., a non-crow) before determining its color eliminated any information to be gained by determining its color at a later time. I now suggest that the timing doesn’t matter, and that as long as a tulip turns out to be red, it supports black crows. This gets back to my first sentence in this comment about finite red objects. If a tulip is later found to be red, it reduces (I believe) the number of red objects that could later turn out to be crows. If it turns out to be some other color, the number of remaining red objects remains unchanged. Interestingly, if it turns out to be black, it supports the hypothesis that all crows are non-black. We happen to know that some crows are black, and so that hypothesis would eventually be falsified by observing a black crow, but if we knew nothing about crow color, the non-existence of black crows would be supported every time a black object turned out not to be a crow – or at least the equivalence principle would lead to that conclusion.
One other point relates to your statement, “I have stated that the non-observation of red crows in some place over some period decreases their likelihood, but for that it does not matter, whether a tulip that we might see during that period is red or black.”
I agree with the first part, but only if the non-observation of red crows occurs when things are being observed. If you are not doing any observing at all (e.g., you’re asleep), the non-observation of red crows is uninformative. If you are observing things, it is informative, but that means that you must be observing things that are not red crows. I stated above why I thought the color of an observed tulip (non-black vs black) affected the probability of black or non-black crows. Obviously, observing a black crow would be even more informative.
I can see that this issue is never going to be resolved, but it’s a nice form of mental exercise to analyze all the possibilities, including the effects of unstated assumptions. It’s also a relief to have an intelligent discussion that isn’t plagued by accusations about Climategate and the alleged sins of the “alarmists”.
The statements of logic were precise, but the empirical setups were not well defined. Therefore the significance of the observations depends on implied assumptions.
To get a valid test of the first hypothesis, we must have a procedure that is defined in a way applicable to that. Thus we must search for crows in a way that does not exclude the possibility of red crows. We must be sure that we may find with a non-zero probability a red crow, if those exist.
For the second case we must have a procedure of picking red objects in a way that we may find with a non-zero probability a red crow, if those exist.
In both cases we have a procedure to pick specifically objects with one well defined property, in the first case we pick crows, in the second we pick red objects. We apply these methods as long as we have picked a predetermined number of objects, which may be one.
The description: “We see a red tulip.” implies in common usage of the language that we are not searching for either red objects or tulips with the determination on finding one occasion. It implies that we just happen to see one. This does not allow drawing any conclusions on the existence of other red objects or crows – or anything else at all. Therefore this observation has zero power as test of either of these hypotheses. For getting some power, the observation must be a result of an organized search for evidence.
All this becomes clear as soon as any real setup is defined well enough to allow for calculating quantitatively probabilities. Then we see that a random observation has zero evidential power for these hypotheses as long as it is not a red crow.
I have stated several times that time spent making observations under conditions, where red crows might be seen, if they exist, provides evidence on their existence, but it does so at exactly the same power independently on, what else we observe during that time, be it black crows, red tulips, or pink elephants.
To me the whole issue appears throughout clear and totally resolved, I just wonder, why others do not agree.
On Judith Curry’s site Climate etc. I have got involved in lengthy discussions concerning properties of an atmosphere transparent to infrared radiation. The discussion has been largely off-topic and might be continued here, if somebody is interested and has hopefully something new to say. I start by copying two of my own messages, which should give a feeling on, what the discussion has been about and which represent largely my views on the subject.
I have had recently a rather lengthy argumentation with Fred Moolten on that point with contributions from others as well. The discussion started from this message
I restate here my views, which are based on well known physical theories. All counterarguments presented by Fred and others are in my opinion weak and without sufficient merit. Tomas Milanovic presented also differing views claiming that the temperature variations of the surface would induce strong mixing, but after some more thinking, which I have explained in the discussion, my view is now strongly that the atmosphere will warm up enough to make even that effect weak.
According to the second law of thermodynamics the equilibrium state for an undisturbed atmosphere with no IR absorption and emission is isothermal as any deviation from that would make the perpetum mobile of the second kind possible with fully realizable technologies like an thermocouple connected to the top and bottom of the atmosphere. (A comment of Quondam in the second link above led to this formulation of the argument.)
Thus it would indeed be true that the adiabatic lapse rate could not be maintained in an atmosphere of pure nitrogen, because such an atmosphere wouldn’t have any mechanism to release heat from the top of the atmosphere to the space (radiation from the surface would escape freely, but the atmosphere would not emit). Therefore the whole atmosphere would reach gradually the temperature of the surface. The temperature of the surface would be essentially the same as without any atmosphere. The temporal and spatial variations of the surface temperature would be reduced and some convection would be induced, but not so much and not reaching such altitudes that the main conclusion would change.
Adiabatic lapse rate is the lapse rate that results from adiabatic convection, when it is present as it is in the tropospheres of the Earth atmosphere and the Venus atmosphere at least at low and middle latitudes. In polar winter, there is not enough convection even in the Earth troposphere to maintain the adiabatic lapse rate.
There is thus a good reason for the name “adiabatic lapse rate”, but it doesn’t imply that it would be the equilibrium or maximum entropy state in an atmosphere that is not heated from below by radiative heat transfer, and cooled by radiative heat loss to space from the top layers of the troposphere.
The adiabatic lapse rate represents a stationary state of a system in (radiative) thermal contact with exterior systems, but not an equilibrium state of an isolated system. It’s a stationary state only, when there is a continuous convective net energy flux upwards.
I just wrote a related note and have it now available at
This note presents a mathematic derivation that shows, how the isothermal atmosphere is possible with gravitation. We have the direct proofs from second law, but they do not discuss the statistical thermodynamics or kinetic gas theory background that is described in this note. The derivation is mathematically simple, but I do not expect that it’s as easy to understand its basic idea or to get convinced that this is a valid analysis.
Similarly the mathematical derivation linked by Quondam is simple, but it’s not as straightforward to know that all formulas used in it as starting point are valid. These issues are complex enough to require much more knowledge for deciding, whose derivations are valid and whose not. After all my entry into this thread was based on the claim that certain commonly presented arguments are not valid physics. I can provide my views on these issues, and I can satisfy myself that I know, what I’m talking about, but real learning of somebody else has occurred only when a reader of these arguments can herself understand, why some arguments are valid and others are not.
Pekka – I need to review the note you linked to in more detail, but I have two questions – one related to the note and the second to my question on Climate etc.
1. You state, “At a differentially earlier time t – dt the same particles were at the vertical coordinate z – w dt and their vertical velocity was w + g dt where g is the gravitational acceleration. The influence of the change of velocity to the vertical coordinate is second order in time differential and can be neglected.”
What happens if you don’t neglect the change in velocity in calculating back to their initial velocities in the path they are taking? If the downward and upward molecules crossing a particular level, z, have the same mean velocity, and you consider that it has changed since they began their journey, doesn’t this require that the velocities of the downward molecules were less earlier and that of the upward molecule greater earlier? This would be consistent with a lower temperature above and a higher one below for the velocities to become equal at z.
2. What is your view on the significance of the increase in potential energy involved in moving heat away from the planet surface, against gravity, so as to establish an isothermal profile? How does this affect the entropy calculation for the entire system- planet plus atmosphere?
Fred,
1. The neglected term is 0.5*g*dt*dt or second order in dt. The development of the density function can be handled with the methods of differential calculus, where neglecting second order terms doesn’t involve any error to the final results. The change in the velocity is taken into account in the second argument, and that’s accurate in this case.
2. Heating the atmosphere takes energy. The expansion of the atmosphere adds to that energy, but in means only that the heat capacity of the atmosphere as whole is somewhat larger and reaching the equilibrium takes longer. The atmosphere expands in that way, because the random collisions related to the thermal motion of the molecules enforces that. It just is sufficiently likely that molecules will receive the required kicks up in these collisions.
@Pekka Pirilä
It is, perhaps time to put the Moolten ‘model’ to rest. Fred has described what mathematically represents a constant density system, at variance with either an isothermal or adiabatic profile. His picture is not unlike that used to calculate the thermal conductivity of ideal gases as described in the chapter on kinetic theory of gases in any respectable physical chemistry text, e.g. A.J. Rutgers. Here, however, the widths of the regions above and below the hypothetical surface equal the mean free paths and differ when there is a density gradient. The net result of such a calculation is that the thermal conductivity is independent of density (Maxwell’s ‘paradox’). As density is that parameter which interacts with the gravitational field, …, etc.
The Verkley-Gerkema reference you’ve provided cites isothermal solution by Gibbs, Maxwell, and Boltzmann. For current textbook references, Landau & Lifshitz, Statistical Mechanics, paragraphs 25, 38. Pierrehumbert actually says “This is the essence of the explanation for why temperature decreases with height: turbulent stirring relaxes the troposphere towards constant theta, yielding the dry adiabat.” RTP omits pointing out, however, that stirring involves doing work on the system, work which is dissipated and eventually radiated to space by a steady-state system. It should come as no surprise that the work of stirring by convection and radiation combined adds up to 240W/m2.
Quondam – I don’t believe you’ve carefully read what I wrote, which has nothing to do with constant density – i.e., it does not require molecular density below and above a layer z, described by Pekka to be equal. It does require net mean mass flux and kinetic energy flux across the layer to be zero. I’ve read the reference you cite, and others, which differ among themselves in their conclusions, but I don’t think any specifically addresses or controverts the points I raise regarding coupling to the planetary surface and the increase in the potential energy of atmospheric heat required to proceed from adiabatic to isothermal. At this point, it appears reasonable to conclude that an adiabatic profile best describes the equilibrium state of an atmosphere coupled to the surface, even if that might not be true of an atmosphere in isolation, but I remain tentative about this and interested in seeing my points addressed. The dissipative element is irrelevant, I believe, if we are considering a non-emissive atmosphere.
I believe that RTP’s description that you quoted involves the reduction in lapse rate toward an adiabat rather than an increase in lapse rate from isothermal to adiabatic. Elsewhere, he discusses how an adiabatic profile will be expected regardless of whether heat flow through the system is large or negligible.
@fredmoolten
Let’s go back to the original proof by contradiction that Quondam presented.
That can be specified in more detail by stating that we have to otherwise isolated columns of gas except that both are in thermal contact with surface and that there is a possibility to connect them to a Carnot engine or other thermal engine at the top. Lets have nitrogen in one of the columns and hydrogen in the other. The adiabatic lapse rate of nitrogen is 14 times that of hydrogen. Thus the tops are at very different temperatures if the columns are high and both have the temperature profile of adiabatic lapse rate. Now we connect the tops to the Carnot engine, which starts to generate mechanical work until both columns have an equal temperature at top. That temperature is close to the original temperature of the hydrogen column, because cooling it from the top induces a strong convection and the bottom of the column is kept at constant temperature by the surface.
If the equilibrium state is that of adiabatic lapse rate, we have some mechanism, which tries to return the nitrogen column towards adiabatic lapse rate, i.e. some mechanism inside the column will cool the top continuously. That allows us to keep the Carnot engine running continuously at low power extracting mechanical energy. The first law is satisfied as more heat is taken from the surface to the hydrogen column than returned to the surface by the nitrogen column. The Carnot engine may be built inside either of the columns so that the only energy extracted from the system at the top is the mechanical energy. Now we have a perfect perpetum mobile of the second type. It extracts some heat from the surface and transforms that to mechanical energy at the top without any other energy flows out or in to the system.
We can replace the Carnot engine with any heat engine that can extract mechanical energy (or electricity) from the temperature range at the top, and we can operate it at a power that allows the temperature difference grow to any value up to that corresponding to adiabatic lapse rates in both columns. The argument is valid as soon as the thermal efficiency of the engine is nonzero, we do not need good efficiency.
There is no doubt that we are in contradiction with the second law.
I know this is repetitive, but because we may be talking at cross purposes, I’ll rephrase my question in relationship to some of the foregoing.
Consider three adjacent altitudes: z, z+, and z-, where z+ is lower (higher density) and z- higher (lower density) than z. Within any time interval, dt, a fraction of molecules from z- (Fz-) will cross z downward, and a smaller fraction from z+ (Fz+) will cross z upward. The size of the fractions will differ (so that the absolute number of molecules is the same) but their mean kinetic energies must be equal. The mean kinetic energy of Fz- will equal its energy at altitude z-, plus added energy from gravity. If we now look at the same number of z- molecules that have traveled in some other direction from z- during dt, will they not have a lower mean kinetic energy (for example, molecules traveling upward will gain potential energy at the expense of kinetic energy)? The converse applies to altitude z+, where Fz+ kinetic energy will be less than the same number of z+ molecules traveling in a different direction. When averaged over all directions, will the mean kinetic energy of Fz+ not underestimate the energy of all molecules at z+, and the mean kinetic energy of Fz- not overestimate the energy of all molecules at z-? If so, z+ must be warmer than z- for their energies at z to be equal. Why does this not require an adiabatic profile in the absence of some constantly applied upwardly directed external force to counteract gravity and accelerate molecules upward rather than isotropically? If such a force exists, what is its source?
I’ll try to address the Second Law question in more detail later. I see no violation, but for the moment, remember that transferring heat will change the heights and pressures within the two columns in a non-equilibrium direction.
Fred,
I may have misinterpreted your description in the Climate Etc. thread. From your comment there, I read: “However, while the Boltzmann distribution describes the mean, we can’t assume that in a gravitational field, the energy distribution of the molecules that were traveling downward was equal to those had traveled horizontally or upward from the same starting level. If that is true, it must be shown rather than assumed.”
If I’m interpreting this statement as you intend, you are asking why the expectation value for the vertical velocity of particle in a gravitational field could not be negative. If that were the case, the center of mass of the entire system of particles would share this velocity.
Your analysis goes wrong, when you fix three altitude levels, not the time traveled. When the time interval is fixed the slowest particles will not reach from z+ to z and the number of particles is miscalculated. For the slowest particles the time interval to go from z+ to z gets so long that the average vertical velocity differs significantly from the initial velocity. These issues are handled correctly in my note, and the result is that given in the note.
@Pekka Pirilä
Pekka – I’m not sure I follow your logic. Consider an even lower level, z++, as far below z+ as z is above it. In any interval, the number of molecules crossing z++ from z+ will of course exceed the number from z+ crossing z, because the slower molecules will not reach z. However, the mean energy of those crossing z++ from z+ will be its kinetic energy plus potential energy, and the same applies to those crossing z from z+. The z++ molecules will have lost potential energy while the z molecules will have gained it. If they started from z+ with the same total energy, how can they cross those two boundaries with the same kinetic energy, or with their kinetic energy at z+?
@fredmoolten
Perhaps so, but you are introducing an essential bias, when you specify several altitude levels. The vertical distance is not an independent variable that you can just choose without causing bias, while the time interval over which the analysis is done can be chosen. The time spent under influence of gravitation is inversely proportional to the vertical velocity, when the distance is fixed, and that leads to the error that you have.
The slowest particles that have initially an upward velocity never reach the upper level, and slowest particles that start up above the level z may have time to accelerate downwards and cross the level. Thus you have complicated bias in the selection of particles by their velocity, and the results are not valid.
@quondam
Quondam – My surmise is the following. In a gravitational field, the mass of a gas column will tend to move downward unless something stops it. I presume that “something” is the planetary surface, which must be exerting a net upward force. Averaged over the sphere of a planet, this implies a compressive force from the atmosphere matched by an expansive counterforce from the planet, reflected in an upward force from surface molecules in equilibrium with the atmosphere. At the surface/gas interface, the upward and downward directed energies must be equal, and in a static situation, surface temperatures should reflect that equality. If we start at zero net energy flux there, my reasoning described above simply applies the same logic at every higher level by requiring that temperature differences exert an expansionary upward tendency that offsets gravity. If we fail to consider the surface/gas relationship, it seems to me, at least at first glance, that an isothermal gas column (mean molecular kinetic energy independent of altitude) requires molecules that move down to the bottom to return upward in the face of gravity with the same energy they had downward, but without a boost from the surface. (I do need to think a bit more about the last sentence, but that is my first intuition).
@Pekka Pirilä
Pekka – I’m trying to discern specifically where our disagreement lies, since we agree that compared with molecules moving downward from a point, those moving upward will be slowed in traveling a specified distance dz, and a disproportionate number will fail to reach it before losing their upward momentum. I assume though that there is disagreement about those molecules that do travel the distance. Are you saying that at level z, although the number of molecules sufficiently energetic to traverse z from z+ will be smaller than the number from z+ needed to traverse z++ (as we agree), the mean kinetic energies of the molecules that do traverse those levels will nevertheless be equal? That the energies would be equal seems to me to be wrong, even for arbitrarily small dz, but it may not be what you are saying. If it is, then using your term w for vertical velocity, wouldn’t we have to look at dw/dz and apply it to the Boltzmann distribution to determine the deviation, if any, from an isotropic distribution of kinetic energies in molecules distant from z in different directions? Intuitively, it seems to me that if we reduce the number of molecules successfully crossing z from below by slowing all upward molecules, those that do cross will also have a reduced mean kinetic energy, but perhaps that can be shown not to be true.
@fredmoolten
Fred,
When looking at the distribution of velocities at level z and its relationships to velocities at nearby velocities slightly before, it’s essential that no bias is introduced to the selection of particles. We know that particles that are going up at level where a little lower and going up slightly earlier. We know also that particles that are going very slowly up were very close to the level z, similar considerations apply to particles going downwards. We must formulate the analysis in such a way that all velocities are weighted in accordance with their prevalence without any requirements for the distance that they have traveled over some short period of time.
That consideration led to the observation that the distribution of velocities of down going molecules depends on the number of down going molecules above the level z and that the number has to be checked the further up the faster molecules we look at. Again similar considerations apply to up going molecules. I formulated that approach in my note.
You continue to refer to more than one fixed level, but that cannot be done without bias, or at least doing it without bias is very complicated and error prone. That would require some compensating factors that multiply the weight of those classes of molecules that get underrepresented in your approach, but no weighting can correct for classes that are totally removed by the bias in analysis. Therefore the approach may fail even with best weights to compensate bias. The problem enters also in your words “traveling a specified distance dz”, because you should not specify the distance traveled, but the time traveled and calculate from that the distance for each velocity class.
My approach does take explicitly into account the influence of gravity on the vertical velocity. The influence of gravity on velocity is proportional to time, not altitude. Therefore it appears as the requirement that the velocity must have been w + g dt at time t-dt to be w at time t. That’s the full first order influence on the molecules. There is a second order influence on the z coordinate, but second order influences do not affect the final outcome.
In physics we can see best, what is correct and what is not, when we try to formalize the situation mathematically looking at all possible biases and being careful on avoiding them or compensating for them, when that’s the easiest solution. As long as the description has not been formalized it’s hand waving. Hand waving may be used to describe correctly enough solutions that have been also formalized. Many science texts for lay readership do it with variable success, but doing it without knowledge on the related full mathematical formulation is risky even for experienced physicists.
Pekka,
I was interested in your comments at Climate etc about bias and cherrypicking in IPCC WG2.
You say this is “a major problem of the IPCC WG2″,
“this problem with WG2 is so severe that I cannot judge at all, what I should take seriously, and what is spurious consequence of the bias in research.”
“The report of the environmental group distorts the message of the original paper, and the IPCC report distorts the text of the report even further from the original paper.”
“I have really lost my trust in the WG2 report.”
I am collecting examples of IPCC distortions, exaggerations and bias at the site http://sites.google.com/site/globalwarmingquestions/ipcc
So far I have concentrated on WG1, with only a few examples from WG2 that have been highlighted in the media. Your comments seem consistent with what Richard Tol found in WG3 and I and others have found in WG1.
Please could you provide some specific details of some of the examples you have found in WG2, on your blog?
@PaulM
My approach is a bit different. Although I am ready to point out a generic problem, and although I indicate that I have some specific observations to support my claim, I’m not ready to tell these details on web. My feeling is that listing them on web would overemphasize specific cases. Although I am quite sure on my interpretation of the facts, pointing specifically out an identified case would require more than that.
I have seen all too much misuse of such information in the public, And I’m not going to contribute to that. The generic problems cannot be proved or disproved by a few examples. As I said the generic arguments together with a few specific examples has made be think that the approach of IPCC has not succeeded for WG2 and WG3. In case of WG1 I think the approach has succeeded, although some problems certainly remain – that is just unavoidable with any effort.
More on my thoughts about IPCC can be found in regular postings.
This page is open to comments. There are no fixed limitations, but I do not make any promises on which comments are allowed. Feel free to test, what happens.
Over on Judith Curry’s blog, a somewhat random discussion started. It started with Fred Moolten posting an off-hand comment, and he later expounded upon his point. The basic idea is by just looking at a red tulip, one can gain information on the color of crows. If one wanted to know whether or not all crows are black, they could find out without ever looking at a crow.
This is counter-intuitive, and a number of disagreements sprang up. I decided to try to write a clear explanation as to why Fred’s argument was correct, but since the thread was already cluttered and this was all off-topic anyway, I decided to post my explanation elsewhere. The host here invited me to post my thoughts, and so here it is an explanation of why you can look at anything and gain information about anything else.
The argument being tested is All objects in group X have property y.
It is accepted there are a finite number of objects, and they are non-changing. The group which contains all of these objects will be referred to as group Z. Since there are a finite number of objects, there are a finite number of objects possessing property y, and there are a finite number of objects not possessing property y. They will be referred to as groups A and B respectively. The two groups are disjoint.
Accepting the above, we know the argument is true if group X is a subset of group A. If any member of group X is a member of group B, the argument is false.
Assuming no other knowledge, the chance of any member of X being a member of B is B/Z. This chance increases as the size of X increases giving the total chance of the argument being true as (B/Z)^X. The chance of the argument being false is 1 – (B/Z)^X.
Given these simple equations, certain observations can be made. Z has a positive correlation with the chance of the argument being true. B has a negative correlation (as does X). This means if B decreases in size, the chance of the argument being true increases. This means if members of B can be determined not to be members of group X (thus being removed from the analysis), the chance of the argument being true increases.
This can be directly related to crows and their colors. Group X is crows, and y is the color black. Group B is all non-black objects, including red tulips. Observing a red tulip means it can be removed from group B for this analysis, and thus, the size of group B is decreased. This increases the chance of the argument (all crows are black) being true.
Of course, there is little practical value in examining flowers when you want information about birds. The amount of information gained is infinitesimally small, so it’s basically unnoticeable. It certainly doesn’t compare to the fact we can find pictures of white crows with a quick Google search. On the other hand, it is a lot more fun.
Hopefully this can help clear up things for people. If there is anything you are confused about, or if you think I’ve missed something, feel free to let me know!
I realized an example might help explain why this works the way it works:
Imagine you have ten colored balls. You know three of the balls are red. You also know four of the balls have magnets in them. Suppose you are asked what the probability is that all the red balls have magnets in them. It’s a simple enough problem to solve.
Now suppose somebody hands you one of the blue balls, and you find it has a magnet in it. They then ask you, “What is the probability all the red balls have magnets in them?” Obviously you won’t give the same answer as before. You now know there are only three magnets amongst the nine remaining balls, so you’ll change your answer accordingly. The same is true if the blue ball you were handed didn’t have a magnet inside it (you’d get different results, of course).
So even without ever examining a red ball, you can gain information about the red balls. In the same way, you can gain information about crows by only looking at flowers. The only difference is how much information you can gain.
Here is my comment from the single email-message I sent to Brandon before the discussion was continued here:
My counterargument to the last point you make is that we do not know the sizes of the sets A, B, Z and X. They are finite, but the sizes are undefined and there is no reason to conclude that the expected size of the set A would be modified through observation of one member of set B.
All the formulas for probabilities are valid only, if we can consider the observation as a random representative of members of the set from which it’s picked.
My way of thinking is perhaps best expressed in terms of Bayesian probabilities and Bayesian inference.
Before the observation we have prior subjective probabilities
PZ(Z) which tells the PDF for the total number of objects in set Z
PA(A) which tells the PDF for the total number of objects in set A
PB(B) which tells the PDF for the total number of objects in set B
PX(X) which tells the PDF for the total number of objects in set X
P(y|X) which tells the likelihood that a random member of X has the property y
P(y|Z) which tells the likelihood that a random member of Z has the property y
P(y|Z) is essentially the expectation value of the ratio of the number of members of sets A and Z, which is close but not exactly the ratio of the expectation values of the sizes of the two sets.
When we make an observation of a member of set B that is not a member of set X the PDF’s PB(B) and PZ(Z) are modified, but PA(A) and PX(X) are not modified, when the observation is done as observation are done in real life. When we see something that does not tell anything on what is happening to issues that are not related to that observation by some mechanism of causation (either direct or through a common cause). Thus seeing a tulip tells nothing about crows. Not seeing a crow tells something about crows, but that was not the statement, which concerned only tulips.
Hempel’s problem is really a problem of philosophical description of induction, which is well known to be an unsolved problem. Common sense is not affected by that, it remains valid. We observe only that philosophers are in trouble, when they try to describe issues that are clear based on common sense.
=======
As stated in the last paragraph, I don’t see any paradox in the real world. I see only problems in the philosophical analysis of induction or “Studies in the Logic of Confirmation” as the title of Hempel’s original paper reads. This means also that I don’t believe that observing a red tulip has any effect at all on the certainty on claims on the colors of crows.
Upon reading that response again, I realize my e-mailed response was completely wrong. I misunderstood the point of the response, so what I said probably made no sense. Now that I’ve reread it, I’ll try responding again.
Nothing I’ve said relies on a change in the expected size of A. The question is merely, “Is X a subset of A?” Ruling out values in B doesn’t change the size of A, it just increases the chances of values lying within A. You are keeping one group constant while decreasing the size of the other. Taken to the extreme, group B could be reduced to 0 (an empty set). At this point, all of X must inherently fall within A, and thus the argument is proven to be true.
The reason this works is even though A hasn’t been affected, Z has been. The chance of X residing in A is related to the proportion of A/Z. Since Z shrinks with each tulip observed (remember, Z = A + B), the chance of X residing in A increases. If you put numbers into the problem, this becomes easy to see:
There are fifty black objects (A = 50). There are 50 non-black objects (B = 50). There are 20 crows (X = 20). The chance of a single crow being non-black is A/(A + B) or 50/100, or 50%. The chance of no crow being black is 50%^20.
Now then, if you see a red tulip, there are only 49 non-black objects left. If you observe 24 more red tulips, there are only 25 non-black objects left. This means the chance of a single crow being non-black is 50/(25 + 50), or 2/3. The chance of no crow being black is 67%^20.
The same principle applies no matter how large the sets involved are.
My point is that observations of the type done in the real world concerning tulips do not affect the possible number of red crows. They just add to the verified number of tulips and thus lead also to a growth in the estimated number of all objects. No number of observations of red tulips has any influence on the posterior estimate on the number of red crows.
The situation would be different, if we would have a container of all red objects and a way of removing objects from there one by one as long as the containers becomes empty, but this is not a correct description on, how observations are done in real world.
The “paradox” is due to an erroneous (or different from the one used in real world) description of the process of making observations.
Hi Pekka, hi Brandon.
Brandon, please consider the following example.
Let’s say I know that there are 4 objects (for simplicity), each a tulip or a crow, but I don’t know how many red objects there are. To make it simpler, let’s also say I know that there are 2 tulips and 2 crows.
This means there are 16 possible worlds I might be in.
Using a 4-letter shorthand to denote the different worlds (the first letter denoting the colour of tulip 1, second letter the colour of tulip 2, third letter the colour of crow 1, and fourth letter the colour of crow 2, where letter R is red, and B is black), the 16 different possible worlds are :
01 RRRR
02 RRRB
03 RRBR
04 RRBB
05 RBRR
06 RBRB
07 RBBR
08 RBBB
09 BRRR
10 BRRB
11 BRBR
12 BRBB
13 BBRR
14 BBRB
15 BBBR
16 BBBB
In four of the worlds (04, 08, 12 and 16), all crows are black.
Before I make any observations, I don’t know which world I’m in. Assuming an equal likelihood of being in any of the worlds, I can say that the probability that I’m in a world where all crows are black is 4/16 = 1/4
Now I make my first observation. Let’s say the object is a red tulip. I now know that I’m not in worlds 13, 14, 15, or 16, so I’m in one of 12 possible worlds, in 3 of which (04, 08 and 12) all crows of black. So after my first observation, I can say that the probability that I’m in a world where all crows are black is 3/12 = 1/4 … no change.
I make my second observation. Let’s say the object is another red tulip. I now know that I’m not in worlds 05, 06, 07, 08, 09, 10, 11 and 12 either, so I’m in one of 4 possible worlds, in 1 of which (04) all crows are black. So after my second observation, I can say that the probability that I’m in a world where all crows are black is 1/4 … still no change.
Therefore finding two red tulips has not changed the probability of being in a world where all crows are black.
Let’s now go through the other possible observations.
Say my first observation is a black tulip. I now know that I’m not in worlds 01, 02, 03 or 04, so I’m in one of 12 possible worlds, in 3 of which (08, 12 and 16) all crows of black. Probability that I’m in a world where all crows are black is 3/12 = 1/4 … no change.
Let’s say my second observation is also a black tulip. I now know that I’m not in worlds 05, 06, 07, 08, 09, 10, 11 and 12 either, so I’m in one of 4 possible worlds, in 1 of which (16) all crows are black. Probability that I’m in a world where all crows are black is 1/4 … no change.
Let’s say instead that, following that 1st black tulip, my second observation is a red tulip. I now know that I’m not in worlds 13, 14, 15, or 16 either, so I’m in one of 8 possible worlds, in 2 of which (08 and 12) all crows are black. Probability that I’m in a world where all crows are black is 2/8 = 1/4 … no change.
Or let’s say that my first observation was a red tulip (13, 14, 15, and 16 eliminated as before, and prob. still 1/4), and my second observation is a black tulip. I now know that I’m not in worlds 01, 02, 03 and 04 either, so I’m in one of 8 possible worlds, in 2 of which (08 and 12) all crows are black. Probability that I’m in a world where all crows are black is 2/8 = 1/4 … no change.
So for all possible observations when examining the two tulips, the probability that I’m in a world where all crows are black remains 1/4, unchanged from the time before I’d made any observations.
Therefore examining tulips doesn’t change the probability that I’m in an world where all crows are black – it provides me with no extra information about the colours of crows.
Oneuniverse,
Your example is clear and valid (you didn’t use Bayesian analysis as I do below, but that makes no difference to the conclusions). It represents one imaginary world, but it’s not the only possible imaginary world. It’s certainly intuitively much more sensible than another imaginary world, which leads to the other conclusion.
In the other imaginary world we have again 4 objects, but this time we know that 2 of them are red and 2 black, but we do not know, how many of them are crows and, how many tulips. We know, however, that we have at least one of each. Now we assume that the following cases are all equally probable (first the 2 reds, then 2 blacks)
01 CCCT
02 CCTC
03 CTCC
04 TCCC
05 CCTT
06 CTCT
07 CTTC
08 TCCT
09 TCTC
10 TTCC
11 CTTT
12 TCTT
13 TTCT
14 TTTC
This time all crows are black in worlds 10, 13 and 14, i.e. in 3/14 of cases. Now observing one red tulip, I know that the cases 01 and 02 are excluded. Observing one red tulip is twice as likely in cases 10, 13 and 14 than in the remaining 9 cases. Thus according to Bayesian thinking the likelihood of worlds with all crows black is now 6/(9+2*3) = 6/15. Observing another red tulip makes it certain that all crows are black, because there are only two red objects.
Thus under two different settings we have two different answers. The Hempel’s paradox is created, when the settings are not defined, and when different choices are implied at different stages of the analysis.
To conclude: Hempel’s paradox is caused by looking at a badly defined problem and including in the analysis implied additional assumptions, which are different in each approach. Using different implied assumptions leads to different conclusions as is expected.
This observation is significant as it tells that it’s impossible to use inductive logic, when we cannot perform well controlled experiments, where all needed assumptions are explicit and fully controlled. This ideal can be approached well enough in many laboratory experiments, but it cannot be reached in observational studies of real world.
Brandon commented on Judith’s blog that he wrote his opening message here, because this issue is not significant for the climate discussion. That comment is not really true, as all empirical knowledge on climate belongs to that class of observations, where the difficulties implied by Hempel’s paradox are present, and present to a very significant degree. They are actually at the heart of the disagreement on AGW.
I agree with Pekka Pirilä’s explanation of the problem, and it is why I tried to be explicit about all assumptions I was making. Changing the assumptions, unsurprisingly, gives different results. In oneuniverse’s example, he discarded an assumption which has been present in the examples given by Fred Moolten and myself. That is, we assumed there were no empty sets. In other words, we know there must be at least one of each class.
Discarding that assumption renders the rest of the problem moot as it allows for the possibility of vacuous truths. The uncertainty introduced by those is what causes oneuniverse to conclude no information is gained, and the removal of them is why Pekka Pirilä’s follow-up does show information gained.
As for the condemnation of inductive reasoning, I don’t agree using it is “impossible” in any situation. The less rigorous an experiment, the more difficult inductive reasoning will be, but I don’t see it ever being impossible. That leaves aside the fact even if you can’t draw any firm conclusions based upon inductive reasoning, you can still apply it to a situation. There is nothing wrong with reaching the conclusion, “I don’t know.”
As for the climate discussion, I’m not sure how you would apply this problem to it, hence why I said it wasn’t significant. It obviously has some relevance, as would any issue about reasoning, but I can’t think of any real parallels in climate science. Do you have any particular ones in mind?
Oh well, I wasn’t going to preempt more comment space on Judy Curry’s blog for this, because it was already too long, but since the discussion continues here, and is entertaining, I’ll add my two cents.
First, this issue is an exercise in logic, not ornithology. Neither Hempel nor anyone else is going to suggest observing all non-black objects to determine the blackness of crows, but as the wikipedia article indicates, the counter-intuitive nature of the conclusion that seeing a non-black non-crow or raven is confirmatory comes from the intuitive belief that it adds zero evidence, in contrast to the logical inference that it adds an extremely tiny amount of evidence that is not zero. To sum it up, it has zero practical value in evaluating the color of crows, but that is not the same as claiming it has no zero evidentiary weight. It does. Discussions of its practical significance for crow color would therefore not be germane.
The Hempel conclusion continues to appear irrefutable to me under the conditions he specified. As far as I know, this conclusion has not subsequently been refuted in a general sense by anyone over the course of more than 60 years, despite many analyses, but instead, discussants have described the underlying assumptions and in some cases suggested that different conclusions could emerge with different starting assumptions.
Two assumptions seem particularly worth noting. The first is that crows exist, but in the world we inhabit, that doesn’t seem to be a problem. The second involves the categories within which we investigate the black crow hypothesis. Four categories might be investigated: (A) All crows. (B) All non-crows. (C) All black objects. (D) All non-black objects. The blackness of all crows can be proved or falsified in theory by category A or D. However, that is not true for either B or C. B is of particular interest – looking specifically at all non-crows cannot tell us anything about the blackness of crows.
Here is where some ambiguity arises. When we observe a red tulip, are we operating in the informative category D (non-black objects) or the non-informative category B (non-crows)?
Since practicality is irrelevant, we must specify what assumptions we use in this hypothetical scenario. Hempel is correct when we specify that we have noted the color of an object either before or simultaneous with an observation of its nature – crow or non-crow. It’s red, and yes, it’s also a tulip, and so we have eliminated one possible example of red crows from category D. On the other hand, if we observe what is clearly a tulip at a distance without discerning its color, subsequently discovering its color will not put it into category D, but into the non-informative category B. This distinction does emphasize the arbitrary nature of the problem and the fact that it has no practical real-world application for understanding crows. It does not, in my view, invalidate the basic logical principle of equivalence that underlies the correctness of Hempel’s reasoning, as long as we specify that we are asking whether a red object we observe is a crow or something else. To pursue these distinctions further would, in my opinion, put too much weight on the practical importance of our observations, which we already agree would be negligible.
A final word about the different scenarios from oneuniverse and from Pekka for distribution of colors, crows, and tulips. Utilizing Pekka’s hypothetical scenarios, 3/14 involve all black crows, they are equally probable, and the probability of all black crows is 3/14, or 6/15 if we observe a red tulip. Here is where the Hempel logic clearly shows its value. Imagine that we start with those 14 possibilities, and then draw 1000 samples from this population of objects, and – Not A Single Red Crow is Observed. Is the probability of all black crows still only 3/14? This is a point that I tried to clarify in the Climate Etc discussion but probably didn’t explain too well. If we don’t know whether all crows are black, the evidence gained from our observations affects any prior probabilities we might have assumed. Even 1000 observations wouldn’t prove the non-existence of a red crow, but as has been discussed many times, confirmation isn’t the same as proof. Observing 1000 objects without seeing a non-black crow, while seeing many non-black objects of other types, would confirm (very very slightly) the blackness of all crows even though it obviously wouldn’t prove it. In Pekka’s example, even observing one red tulip is confirmatory, by increasing the prior probability from 3/14 to a posterior probability of 6/15.
The examples of Oneuniverse and myself tell that there is no paradox. What appears a paradox is rather sloppy use of logic. This is not irrelevant, because there is a wish to use logic also, when the setup does not allow for rigorous logic, and then we may have situations, where two equally plausible alternatives give indeed different results.
This example of tulips and crows is not such a case, because in this example one of the alternatives is clearly superior and the other should be abandoned as contrary to the natural setup. This faulty alternative is of course that of trying to make conclusions on colors of crows from observations concerning tulips. No reasonable real world setup allows for this interpretation. The original logic hypotheses are equivalent, but everybody interprets the statement that we see a red tulip to describe something not relevant to the colors of crows. By this I mean absolutely no influence, not that the influence is small.
As I have stated before, the related statement that we have not seen a red crow during a period of time, when seeing it would be possible, if red crows exist, is evidence against the existence of red crows. If we measure time by the number of red tulips seen, we may create artificially a connection between tulips and crows, but this means that we have introduced additional constraints that differ from those everybody imagines from the original description.
The value of Hempel’s discussion is in demonstrating that rules of logic work only, when the questions are set up precisely, and that this is often difficult. It’s pity that his example is such that one of the alternatives works, while the other does not. Therefore it’s not a very good case, and too easily dismissed.
My own impression is that we have seen in Climate Etc numerous discussions, where similar problems of applying logic inference are present. They are not as clearcut and perhaps not really examples of “Hempel’s paradox”, but all too many have claimed that they have strong arguments based on logic, when the setup does not allow for drawing such conclusions.
@Pekka Pirilä
“This faulty alternative is of course that of trying to make conclusions on colors of crows from observations concerning tulips. No reasonable real world setup allows for this interpretation.”
We can’t draw conclusions about the color of crows by observing tulips, but we can by observing red objects. I agree it’s not a “real world” problem, because no-one would try to ascertain whether crows are black by that approach, but I see no flaw in the logic itself. I’ll see if I can think of a plausible hypothetical real world example where the same logic would have practical implications, based on the equivalence principle, but most real world problems are solved more directly, rather than by equivalence, so it probably won’t be easy.
Fred,
There may be real world situations, where the equivalence principle can be applied, but that requires that the setup is fully defined and such that the equivalence principle provides the information it is supposed to give.
The two simplified examples indicated that procedures that were supposed to be equivalent by the equivalence principle were in fact not equivalent. Thus this is not a trivial requirement, but requires careful analysis of the empirical setup. For a statistical analysis this means usually that the samples are randomized correctly. Starting from two formulation that are linked by the equivalence principle, it is likely that the most obvious ways of randomizing the samples would lead to different weightings and thus to different results in the statistical analysis.
The situation is analogous to that met in applying Bayesian analysis to the probability distributions of two continuous variables that are nonlinearly related. The common choice of flat prior distribution for one of the variables would correspond to a non-flat prior for the other as discussed recently in connection with the Annan and Hargreaves paper.
The two examples discussed above where extreme cases as the PDF of either “crowness” or color was fixed by 2 + 2 as the only possible combination. Therefore the equivalence principle failed totally (as it does also in the real world versions of crows and tulips). In some other problems the constraints may really be such that the principle works.
One way of expressing the observation is that the equivalence principle provides two methods to collect evidence on the same issue, but there is no general guarantee that the efficiency of either of the methods is not exactly zero. In the real world example of crows and tulips the efficiency is indeed exactly zero.
@Pekka Pirilä
Pekka – I’m continuing this discussion simply because it’s enjoyable, and not with the expectation that proofs can emerge on a subject as amibiguous as this one.
You continue to state in one way or another that the “efficiency” of the equivalence principle in the Hempel paradox is “exactly zero”, but the Hempel argument, of course, is that it “seems” to be exactly zero because it is too small to have any practical utility or to be measurable, but is in fact very slightly greater than zero. Of course, the efficiency of observing crows directly would clearly be greater than that of observing non-black objects, and so each crow observation would carry more statistical weight than each non-black observation. Some inadvertent degree of non-randomness might reduce statistical power, but should not invalidate the principle unless observations of non-black objects deliberately avoided crows – the paradox as described implies that no attempt is made to avoid crow observations, and in fairness to Hempel and in the interest of seeing the paradox as an interesting exercise in inductive reasoning, we shouldn’t impose an additional arbitrary condition of selectively excluding crow observations. As far as I can see, the Hempel logic leads to the conclusion he draws, unless a specific flaw can be found, and none has yet been identified that I’m aware of. The hypothetical example you gave above with 14 different combinations appears to support Hempel, since observing a red tulip increased the probability that all crows are black.
Here is a more “realistic” hypothetical I’ve conjured up, suggested by the recent killing of Osama Bin Laden. I don’t know how realistic it actually is, but it may not totally misconstrue what our CIA in the U.S. does to avert terrorist operations. It goes as follows:
The CIA has been compiling a list of individuals requiring special surveillance because they pose a greater than average threat of engineering a major terrorist bomb plot. The list has now grown to 25,000, and is too large for adequate surveillance of all these individuals, so the CIA needs to reduce its size by removing names of individuals who don’t pose a major threat. There is evidence that participants in bomb plots have been trained at terrorist training camps, and the CIA hypothesizes that only trained individuals pose a major threat. Via equivalence, this translates into untrained = non-dangerous, or at least not dangerous enough for the limited special surveillance resources to be spent on them
Of the 25,000 listed individuals, 534 were known to have been trained, and 10,502 were known to be untrained. No data are available for the remainder. During the interval of interest, 31 major bomb attacks were attempted, and it is assumed that additional attempts are being plotted, although the number is unknown. Of these, 4 were attempted by individuals among the 534 who were trained. The remaining 27 were attempted by individuals known not to be on the list, and their training status is unknown. The evidence provides some correlation between training and bomb threat, but are these individuals more dangerous than the untrained ones? The CIA then reviews the 10,502 untrained individuals and finds that none of the 31 attempts involved any of them. It concludes that its use of limited surveillance resources would be optimized by removing them from the list, on the grounds that the data support the hypothesis that only trained individuals pose a bomb threat, while acknowledging that exceptions might be possible – i.e., the evidence provided confirmation but not proof. It also concludes that more effort must be applied to ascertaining the training status of individuals on the list.
Regarding the red tulip analogy, “untrained” = red, and “not involved in a bomb attempt” = tulip. Note that the analogy involves data focused on untrained status as the independent variable rather than on “not involved in bomb attempt”. Simply analyzing the latter (24,996 on the list plus unquantified millions not on the list) would not have provided evidence tending to exclude the possibility that untrained individuals were dangerous.
This is the best analogy I can imagine at the moment. It is slightly better constrained than Hempel in limiting the total number of individuals involved, which is why it works better, but I think the main principles are similar enough to illustrate the point about equivalence.
I realize that it’s possible to contrive scenarios where the equivalence principle might not work, but under all reasonable assumptions that the Hempel scenario is what it seems to be – a non-black object is observed either by chance or because one is on the alert for non-black objects, and turns out not to be a crow – it does appear to work.
@fredmoolten
Here’s another hypothetical A = B equivalence example more relevant to climate. I’ll phrase it in the “B” form:
Large changes in the concentration of a single atmospheric component that do not reduce emissions to space in the 12 to 18 um region as a fraction of total OLR are not rises in CO2 concentration.
Fred,
I have repeatedly emphasized that the evidence obtained by the other choice of the uncertainty principle may be exactly zero and that it is indeed exactly zero in the setup understood by almost everybody from the setup of the original example. That setup guarantees that the tulips and crows are unrelated. Therefore the power is exactly zero.
We can also say that observation of one red tulip bears almost no influence on the expected number of red tulips elsewhere. Perhaps it does increase that number as it makes the observer think that red tulips are actually more common than he thought before.
I have stated that the non-observation of red crows in some place over some period decreases their likelihood, but for that it does not matter, whether a tulip that we might see during that period is red or black. These things are really totally unrelated and the evidence is really exactly zero. The way around the “paradox” is not by contesting this obvious fact, but by thinking where the alternative logic fails.
In this case it’s not difficult to see where it fails, as the alternative logic is based on the unjustified assumption that the total number of red objects would be somehow more fundamental than the number of red crows and that observing a red tulip would decrease the expected number of other red objects. These claims are assumptions, which may be true or false depending other influencing factors. In this particular example they are false, when the setup is understood as it is by almost everybody. The implied assumptions concerning the setup of almost everybody are contradictory to these other assumptions.
As the original question is not explicitly fully defined, it becomes fully defined only when supplemented with sufficient implied assumptions, and here we may choose, whether we accept the assumptions that almost everybody picks intuitively or some other set, where some influence remains. I have given an example of such assumptions in the proposal that we count time spent for observations by the number of red tulips seen. Then more red tulips means more time and more certainty that red crows do not exist.
Another example of a setup, where the other alternative would work is such that we hire a large number of people to search for red objects getting very many of such observations without knowledge given to us on, what the objects are. Then we pick randomly one of these observations and check, what it is. If it is a tulip, that has an influence on the likelihood of red crows. The point here is that we have a fixed set of observations known to contain exclusively red objects.
The implied understanding of almost everybody is, however, such that there is no correlation and the power of the alternative method is exactly zero. For a philosopher it should be enough that the power may be exactly zero and it certainly may under fully realistic conditions and natural implied assumptions.
I’m confused. How do you justify saying, “As the original question is not explicitly fully defined, it becomes fully defined only when supplemented with sufficient implied assumptions…”? Just what part of the problem lacks definition? Just what assumptions are being supplemented?
As far as I can tell, the formulation I provided is perfectly well-defined, and I can’t see any absent assumptions in the most recent example from Fred Moolten.
@Pekka Pirilä
“unjustified assumption that…. observing a red tulip would decrease the expected number of other red objects. ”
I believe that as long as red objects are finite in number, observing one reduces the number left to observe. If that is a false assumption, then I’ll admit the remainder of the Hempel argument is wrong, but I’ll proceed below based on the truth of that proposition.
Rather than repeat previous points, I’ll leave the main argument for others to review from earlier comments. However, I do want to revise a previous statement I made regarding timing. Previously, I suggested that observing an object to be red and observing it also to be a tulip simultaneously or later supported the black crow hypothesis, but that observing it first to be a tulip (i.e., a non-crow) before determining its color eliminated any information to be gained by determining its color at a later time. I now suggest that the timing doesn’t matter, and that as long as a tulip turns out to be red, it supports black crows. This gets back to my first sentence in this comment about finite red objects. If a tulip is later found to be red, it reduces (I believe) the number of red objects that could later turn out to be crows. If it turns out to be some other color, the number of remaining red objects remains unchanged. Interestingly, if it turns out to be black, it supports the hypothesis that all crows are non-black. We happen to know that some crows are black, and so that hypothesis would eventually be falsified by observing a black crow, but if we knew nothing about crow color, the non-existence of black crows would be supported every time a black object turned out not to be a crow – or at least the equivalence principle would lead to that conclusion.
One other point relates to your statement, “I have stated that the non-observation of red crows in some place over some period decreases their likelihood, but for that it does not matter, whether a tulip that we might see during that period is red or black.”
I agree with the first part, but only if the non-observation of red crows occurs when things are being observed. If you are not doing any observing at all (e.g., you’re asleep), the non-observation of red crows is uninformative. If you are observing things, it is informative, but that means that you must be observing things that are not red crows. I stated above why I thought the color of an observed tulip (non-black vs black) affected the probability of black or non-black crows. Obviously, observing a black crow would be even more informative.
I can see that this issue is never going to be resolved, but it’s a nice form of mental exercise to analyze all the possibilities, including the effects of unstated assumptions. It’s also a relief to have an intelligent discussion that isn’t plagued by accusations about Climategate and the alleged sins of the “alarmists”.
@Brandon Shollenberger
The statements of logic were precise, but the empirical setups were not well defined. Therefore the significance of the observations depends on implied assumptions.
To get a valid test of the first hypothesis, we must have a procedure that is defined in a way applicable to that. Thus we must search for crows in a way that does not exclude the possibility of red crows. We must be sure that we may find with a non-zero probability a red crow, if those exist.
For the second case we must have a procedure of picking red objects in a way that we may find with a non-zero probability a red crow, if those exist.
In both cases we have a procedure to pick specifically objects with one well defined property, in the first case we pick crows, in the second we pick red objects. We apply these methods as long as we have picked a predetermined number of objects, which may be one.
The description: “We see a red tulip.” implies in common usage of the language that we are not searching for either red objects or tulips with the determination on finding one occasion. It implies that we just happen to see one. This does not allow drawing any conclusions on the existence of other red objects or crows – or anything else at all. Therefore this observation has zero power as test of either of these hypotheses. For getting some power, the observation must be a result of an organized search for evidence.
All this becomes clear as soon as any real setup is defined well enough to allow for calculating quantitatively probabilities. Then we see that a random observation has zero evidential power for these hypotheses as long as it is not a red crow.
@fredmoolten
I have stated several times that time spent making observations under conditions, where red crows might be seen, if they exist, provides evidence on their existence, but it does so at exactly the same power independently on, what else we observe during that time, be it black crows, red tulips, or pink elephants.
To me the whole issue appears throughout clear and totally resolved, I just wonder, why others do not agree.
On Judith Curry’s site Climate etc. I have got involved in lengthy discussions concerning properties of an atmosphere transparent to infrared radiation. The discussion has been largely off-topic and might be continued here, if somebody is interested and has hopefully something new to say. I start by copying two of my own messages, which should give a feeling on, what the discussion has been about and which represent largely my views on the subject.
First a comment posted May 22, 2011 at 5:44 am
===============
I have had recently a rather lengthy argumentation with Fred Moolten on that point with contributions from others as well. The discussion started from this message
http://judithcurry.com/2011/05/08/ncar-community-climate-system-model-version-4/#comment-67943
with some related comments here
http://judithcurry.com/2011/05/08/ncar-community-climate-system-model-version-4/#comment-68091
I restate here my views, which are based on well known physical theories. All counterarguments presented by Fred and others are in my opinion weak and without sufficient merit. Tomas Milanovic presented also differing views claiming that the temperature variations of the surface would induce strong mixing, but after some more thinking, which I have explained in the discussion, my view is now strongly that the atmosphere will warm up enough to make even that effect weak.
According to the second law of thermodynamics the equilibrium state for an undisturbed atmosphere with no IR absorption and emission is isothermal as any deviation from that would make the perpetum mobile of the second kind possible with fully realizable technologies like an thermocouple connected to the top and bottom of the atmosphere. (A comment of Quondam in the second link above led to this formulation of the argument.)
Thus it would indeed be true that the adiabatic lapse rate could not be maintained in an atmosphere of pure nitrogen, because such an atmosphere wouldn’t have any mechanism to release heat from the top of the atmosphere to the space (radiation from the surface would escape freely, but the atmosphere would not emit). Therefore the whole atmosphere would reach gradually the temperature of the surface. The temperature of the surface would be essentially the same as without any atmosphere. The temporal and spatial variations of the surface temperature would be reduced and some convection would be induced, but not so much and not reaching such altitudes that the main conclusion would change.
============
Then a somewhat later message posted May 23, 2011 at 1:48 pm
============
Adiabatic lapse rate is the lapse rate that results from adiabatic convection, when it is present as it is in the tropospheres of the Earth atmosphere and the Venus atmosphere at least at low and middle latitudes. In polar winter, there is not enough convection even in the Earth troposphere to maintain the adiabatic lapse rate.
There is thus a good reason for the name “adiabatic lapse rate”, but it doesn’t imply that it would be the equilibrium or maximum entropy state in an atmosphere that is not heated from below by radiative heat transfer, and cooled by radiative heat loss to space from the top layers of the troposphere.
The adiabatic lapse rate represents a stationary state of a system in (radiative) thermal contact with exterior systems, but not an equilibrium state of an isolated system. It’s a stationary state only, when there is a continuous convective net energy flux upwards.
I just wrote a related note and have it now available at
http://pirila.fi/energy/kuvat/barometric_derivation.pdf
This note presents a mathematic derivation that shows, how the isothermal atmosphere is possible with gravitation. We have the direct proofs from second law, but they do not discuss the statistical thermodynamics or kinetic gas theory background that is described in this note. The derivation is mathematically simple, but I do not expect that it’s as easy to understand its basic idea or to get convinced that this is a valid analysis.
Similarly the mathematical derivation linked by Quondam is simple, but it’s not as straightforward to know that all formulas used in it as starting point are valid. These issues are complex enough to require much more knowledge for deciding, whose derivations are valid and whose not. After all my entry into this thread was based on the claim that certain commonly presented arguments are not valid physics. I can provide my views on these issues, and I can satisfy myself that I know, what I’m talking about, but real learning of somebody else has occurred only when a reader of these arguments can herself understand, why some arguments are valid and others are not.
Pekka – I need to review the note you linked to in more detail, but I have two questions – one related to the note and the second to my question on Climate etc.
1. You state, “At a differentially earlier time t – dt the same particles were at the vertical coordinate z – w dt and their vertical velocity was w + g dt where g is the gravitational acceleration. The influence of the change of velocity to the vertical coordinate is second order in time differential and can be neglected.”
What happens if you don’t neglect the change in velocity in calculating back to their initial velocities in the path they are taking? If the downward and upward molecules crossing a particular level, z, have the same mean velocity, and you consider that it has changed since they began their journey, doesn’t this require that the velocities of the downward molecules were less earlier and that of the upward molecule greater earlier? This would be consistent with a lower temperature above and a higher one below for the velocities to become equal at z.
2. What is your view on the significance of the increase in potential energy involved in moving heat away from the planet surface, against gravity, so as to establish an isothermal profile? How does this affect the entropy calculation for the entire system- planet plus atmosphere?
Fred,
1. The neglected term is 0.5*g*dt*dt or second order in dt. The development of the density function can be handled with the methods of differential calculus, where neglecting second order terms doesn’t involve any error to the final results. The change in the velocity is taken into account in the second argument, and that’s accurate in this case.
2. Heating the atmosphere takes energy. The expansion of the atmosphere adds to that energy, but in means only that the heat capacity of the atmosphere as whole is somewhat larger and reaching the equilibrium takes longer. The atmosphere expands in that way, because the random collisions related to the thermal motion of the molecules enforces that. It just is sufficiently likely that molecules will receive the required kicks up in these collisions.
I added some more explanatory text and also comments on the role of molecular collisions to my note
http://pirila.fi/energy/kuvat/barometric_derivation.pdf
@Pekka Pirilä
It is, perhaps time to put the Moolten ‘model’ to rest. Fred has described what mathematically represents a constant density system, at variance with either an isothermal or adiabatic profile. His picture is not unlike that used to calculate the thermal conductivity of ideal gases as described in the chapter on kinetic theory of gases in any respectable physical chemistry text, e.g. A.J. Rutgers. Here, however, the widths of the regions above and below the hypothetical surface equal the mean free paths and differ when there is a density gradient. The net result of such a calculation is that the thermal conductivity is independent of density (Maxwell’s ‘paradox’). As density is that parameter which interacts with the gravitational field, …, etc.
The Verkley-Gerkema reference you’ve provided cites isothermal solution by Gibbs, Maxwell, and Boltzmann. For current textbook references, Landau & Lifshitz, Statistical Mechanics, paragraphs 25, 38. Pierrehumbert actually says “This is the essence of the explanation for why temperature decreases with height: turbulent stirring relaxes the troposphere towards constant theta, yielding the dry adiabat.” RTP omits pointing out, however, that stirring involves doing work on the system, work which is dissipated and eventually radiated to space by a steady-state system. It should come as no surprise that the work of stirring by convection and radiation combined adds up to 240W/m2.
@quondam
Quondam – I don’t believe you’ve carefully read what I wrote, which has nothing to do with constant density – i.e., it does not require molecular density below and above a layer z, described by Pekka to be equal. It does require net mean mass flux and kinetic energy flux across the layer to be zero. I’ve read the reference you cite, and others, which differ among themselves in their conclusions, but I don’t think any specifically addresses or controverts the points I raise regarding coupling to the planetary surface and the increase in the potential energy of atmospheric heat required to proceed from adiabatic to isothermal. At this point, it appears reasonable to conclude that an adiabatic profile best describes the equilibrium state of an atmosphere coupled to the surface, even if that might not be true of an atmosphere in isolation, but I remain tentative about this and interested in seeing my points addressed. The dissipative element is irrelevant, I believe, if we are considering a non-emissive atmosphere.
I believe that RTP’s description that you quoted involves the reduction in lapse rate toward an adiabat rather than an increase in lapse rate from isothermal to adiabatic. Elsewhere, he discusses how an adiabatic profile will be expected regardless of whether heat flow through the system is large or negligible.
@fredmoolten
Let’s go back to the original proof by contradiction that Quondam presented.
That can be specified in more detail by stating that we have to otherwise isolated columns of gas except that both are in thermal contact with surface and that there is a possibility to connect them to a Carnot engine or other thermal engine at the top. Lets have nitrogen in one of the columns and hydrogen in the other. The adiabatic lapse rate of nitrogen is 14 times that of hydrogen. Thus the tops are at very different temperatures if the columns are high and both have the temperature profile of adiabatic lapse rate. Now we connect the tops to the Carnot engine, which starts to generate mechanical work until both columns have an equal temperature at top. That temperature is close to the original temperature of the hydrogen column, because cooling it from the top induces a strong convection and the bottom of the column is kept at constant temperature by the surface.
If the equilibrium state is that of adiabatic lapse rate, we have some mechanism, which tries to return the nitrogen column towards adiabatic lapse rate, i.e. some mechanism inside the column will cool the top continuously. That allows us to keep the Carnot engine running continuously at low power extracting mechanical energy. The first law is satisfied as more heat is taken from the surface to the hydrogen column than returned to the surface by the nitrogen column. The Carnot engine may be built inside either of the columns so that the only energy extracted from the system at the top is the mechanical energy. Now we have a perfect perpetum mobile of the second type. It extracts some heat from the surface and transforms that to mechanical energy at the top without any other energy flows out or in to the system.
We can replace the Carnot engine with any heat engine that can extract mechanical energy (or electricity) from the temperature range at the top, and we can operate it at a power that allows the temperature difference grow to any value up to that corresponding to adiabatic lapse rates in both columns. The argument is valid as soon as the thermal efficiency of the engine is nonzero, we do not need good efficiency.
There is no doubt that we are in contradiction with the second law.
I know this is repetitive, but because we may be talking at cross purposes, I’ll rephrase my question in relationship to some of the foregoing.
Consider three adjacent altitudes: z, z+, and z-, where z+ is lower (higher density) and z- higher (lower density) than z. Within any time interval, dt, a fraction of molecules from z- (Fz-) will cross z downward, and a smaller fraction from z+ (Fz+) will cross z upward. The size of the fractions will differ (so that the absolute number of molecules is the same) but their mean kinetic energies must be equal. The mean kinetic energy of Fz- will equal its energy at altitude z-, plus added energy from gravity. If we now look at the same number of z- molecules that have traveled in some other direction from z- during dt, will they not have a lower mean kinetic energy (for example, molecules traveling upward will gain potential energy at the expense of kinetic energy)? The converse applies to altitude z+, where Fz+ kinetic energy will be less than the same number of z+ molecules traveling in a different direction. When averaged over all directions, will the mean kinetic energy of Fz+ not underestimate the energy of all molecules at z+, and the mean kinetic energy of Fz- not overestimate the energy of all molecules at z-? If so, z+ must be warmer than z- for their energies at z to be equal. Why does this not require an adiabatic profile in the absence of some constantly applied upwardly directed external force to counteract gravity and accelerate molecules upward rather than isotropically? If such a force exists, what is its source?
I’ll try to address the Second Law question in more detail later. I see no violation, but for the moment, remember that transferring heat will change the heights and pressures within the two columns in a non-equilibrium direction.
Fred,
I may have misinterpreted your description in the Climate Etc. thread. From your comment there, I read: “However, while the Boltzmann distribution describes the mean, we can’t assume that in a gravitational field, the energy distribution of the molecules that were traveling downward was equal to those had traveled horizontally or upward from the same starting level. If that is true, it must be shown rather than assumed.”
If I’m interpreting this statement as you intend, you are asking why the expectation value for the vertical velocity of particle in a gravitational field could not be negative. If that were the case, the center of mass of the entire system of particles would share this velocity.
@fredmoolten
Your analysis goes wrong, when you fix three altitude levels, not the time traveled. When the time interval is fixed the slowest particles will not reach from z+ to z and the number of particles is miscalculated. For the slowest particles the time interval to go from z+ to z gets so long that the average vertical velocity differs significantly from the initial velocity. These issues are handled correctly in my note, and the result is that given in the note.
@Pekka Pirilä
Pekka – I’m not sure I follow your logic. Consider an even lower level, z++, as far below z+ as z is above it. In any interval, the number of molecules crossing z++ from z+ will of course exceed the number from z+ crossing z, because the slower molecules will not reach z. However, the mean energy of those crossing z++ from z+ will be its kinetic energy plus potential energy, and the same applies to those crossing z from z+. The z++ molecules will have lost potential energy while the z molecules will have gained it. If they started from z+ with the same total energy, how can they cross those two boundaries with the same kinetic energy, or with their kinetic energy at z+?
@fredmoolten
Perhaps so, but you are introducing an essential bias, when you specify several altitude levels. The vertical distance is not an independent variable that you can just choose without causing bias, while the time interval over which the analysis is done can be chosen. The time spent under influence of gravitation is inversely proportional to the vertical velocity, when the distance is fixed, and that leads to the error that you have.
The slowest particles that have initially an upward velocity never reach the upper level, and slowest particles that start up above the level z may have time to accelerate downwards and cross the level. Thus you have complicated bias in the selection of particles by their velocity, and the results are not valid.
@quondam
Quondam – My surmise is the following. In a gravitational field, the mass of a gas column will tend to move downward unless something stops it. I presume that “something” is the planetary surface, which must be exerting a net upward force. Averaged over the sphere of a planet, this implies a compressive force from the atmosphere matched by an expansive counterforce from the planet, reflected in an upward force from surface molecules in equilibrium with the atmosphere. At the surface/gas interface, the upward and downward directed energies must be equal, and in a static situation, surface temperatures should reflect that equality. If we start at zero net energy flux there, my reasoning described above simply applies the same logic at every higher level by requiring that temperature differences exert an expansionary upward tendency that offsets gravity. If we fail to consider the surface/gas relationship, it seems to me, at least at first glance, that an isothermal gas column (mean molecular kinetic energy independent of altitude) requires molecules that move down to the bottom to return upward in the face of gravity with the same energy they had downward, but without a boost from the surface. (I do need to think a bit more about the last sentence, but that is my first intuition).
@Pekka Pirilä
Pekka – I’m trying to discern specifically where our disagreement lies, since we agree that compared with molecules moving downward from a point, those moving upward will be slowed in traveling a specified distance dz, and a disproportionate number will fail to reach it before losing their upward momentum. I assume though that there is disagreement about those molecules that do travel the distance. Are you saying that at level z, although the number of molecules sufficiently energetic to traverse z from z+ will be smaller than the number from z+ needed to traverse z++ (as we agree), the mean kinetic energies of the molecules that do traverse those levels will nevertheless be equal? That the energies would be equal seems to me to be wrong, even for arbitrarily small dz, but it may not be what you are saying. If it is, then using your term w for vertical velocity, wouldn’t we have to look at dw/dz and apply it to the Boltzmann distribution to determine the deviation, if any, from an isotropic distribution of kinetic energies in molecules distant from z in different directions? Intuitively, it seems to me that if we reduce the number of molecules successfully crossing z from below by slowing all upward molecules, those that do cross will also have a reduced mean kinetic energy, but perhaps that can be shown not to be true.
@fredmoolten
Fred,
When looking at the distribution of velocities at level z and its relationships to velocities at nearby velocities slightly before, it’s essential that no bias is introduced to the selection of particles. We know that particles that are going up at level where a little lower and going up slightly earlier. We know also that particles that are going very slowly up were very close to the level z, similar considerations apply to particles going downwards. We must formulate the analysis in such a way that all velocities are weighted in accordance with their prevalence without any requirements for the distance that they have traveled over some short period of time.
That consideration led to the observation that the distribution of velocities of down going molecules depends on the number of down going molecules above the level z and that the number has to be checked the further up the faster molecules we look at. Again similar considerations apply to up going molecules. I formulated that approach in my note.
You continue to refer to more than one fixed level, but that cannot be done without bias, or at least doing it without bias is very complicated and error prone. That would require some compensating factors that multiply the weight of those classes of molecules that get underrepresented in your approach, but no weighting can correct for classes that are totally removed by the bias in analysis. Therefore the approach may fail even with best weights to compensate bias. The problem enters also in your words “traveling a specified distance dz”, because you should not specify the distance traveled, but the time traveled and calculate from that the distance for each velocity class.
My approach does take explicitly into account the influence of gravity on the vertical velocity. The influence of gravity on velocity is proportional to time, not altitude. Therefore it appears as the requirement that the velocity must have been w + g dt at time t-dt to be w at time t. That’s the full first order influence on the molecules. There is a second order influence on the z coordinate, but second order influences do not affect the final outcome.
In physics we can see best, what is correct and what is not, when we try to formalize the situation mathematically looking at all possible biases and being careful on avoiding them or compensating for them, when that’s the easiest solution. As long as the description has not been formalized it’s hand waving. Hand waving may be used to describe correctly enough solutions that have been also formalized. Many science texts for lay readership do it with variable success, but doing it without knowledge on the related full mathematical formulation is risky even for experienced physicists.
Pekka,
I was interested in your comments at Climate etc about bias and cherrypicking in IPCC WG2.
You say this is “a major problem of the IPCC WG2″,
“this problem with WG2 is so severe that I cannot judge at all, what I should take seriously, and what is spurious consequence of the bias in research.”
“The report of the environmental group distorts the message of the original paper, and the IPCC report distorts the text of the report even further from the original paper.”
“I have really lost my trust in the WG2 report.”
I am collecting examples of IPCC distortions, exaggerations and bias at the site
http://sites.google.com/site/globalwarmingquestions/ipcc
So far I have concentrated on WG1, with only a few examples from WG2 that have been highlighted in the media. Your comments seem consistent with what Richard Tol found in WG3 and I and others have found in WG1.
Please could you provide some specific details of some of the examples you have found in WG2, on your blog?
@PaulM
My approach is a bit different. Although I am ready to point out a generic problem, and although I indicate that I have some specific observations to support my claim, I’m not ready to tell these details on web. My feeling is that listing them on web would overemphasize specific cases. Although I am quite sure on my interpretation of the facts, pointing specifically out an identified case would require more than that.
I have seen all too much misuse of such information in the public, And I’m not going to contribute to that. The generic problems cannot be proved or disproved by a few examples. As I said the generic arguments together with a few specific examples has made be think that the approach of IPCC has not succeeded for WG2 and WG3. In case of WG1 I think the approach has succeeded, although some problems certainly remain – that is just unavoidable with any effort.
More on my thoughts about IPCC can be found in regular postings.