Independence Of Two Events

In this post, we introduce and develop the concept of independence between events. The general idea is the following:

If I tell you that a certain event A has occurred, this will generally change the probability of some other event B. Probabilities will have to be replaced by conditional probabilities. But if the conditional probability turns out to be the same as the unconditional probability, then the occurrence of event A does not carry any useful information on whether event B will occur. In such a case, we say that events A and B are independent. We will develop some intuition about the meaning of independence of two events and introduce an extension, the concept of conditional independence.

Let us start with a first attempt at the definition. We have an event, B, that has a certain probability of occurring. We are then told that event A occurred, but suppose that this knowledge does not affect our beliefs about B in the sense that the conditional probability remains the same as the original unconditional probability. Thus, the occurrence of A provides no new information about B. In such a case, we may say that event B is independent from event A.

P(B|A) = P(B)

If this is indeed the case, notice that the probability that both A and B occur, which is always equal by the multiplication rule to the probability of A times the conditional probability of B given A.

P(A∩B) = P(A).P(B|A)

So, this is a relation that’s always true. But if we also have this additional condition “occurrence of A provides no new information about B” then this simplifies to the probability of A times the probability of B.

P(A∩B) = P(A).P(B|A) = P(A).P(B)

So, we can find the probability of both events happening by just multiplying their individual probabilities. It turns out that this relation is a cleaner way of defining formally the notion of independence. So, we will say that two events, A and B, are independent if this relation holds.

P(A∩B) = P(A).P(B)

Why do we use this definition rather than the original one? This formal definition has several advantages. First, it is consistent with the earlier definition. A more important reason is that this formal definition is symmetric with respect to the roles of A and B. So instead of saying that B is independent from A, based on this definition we can now say that events A and B are independent of each other. And in addition, since this definition is symmetric and since it implies this condition, it must also imply the symmetrical relation. Namely, that the conditional probability of A given B is the same as the unconditional probability of A.

P(A|B) = P(A)

Finally, on the technical side, conditional probabilities are only defined when the conditioning event has non-zero probability. So, this original definition P(B|A) = P(B) would only make sense in those cases where the probability of the event A would be non-zero. In contrast, this new definition P(A∩B)=P(A).P(B) makes sense even when we’re dealing with zero probability events. So, this definition is indeed more general, and this also makes it more elegant.

Let us now build some understanding of what independence really is. Suppose that we have two events, A and B, both of which have positive probability. And furthermore, these two events are disjoint. They do not have any common elements. Are these two events independent?

Let us check the definition. The probability that both A and B occur is zero because the two events are disjoint.

P(A∩B) = 0

They cannot happen together. On the other hand, the probability of A times the probability of B is positive, since each one of the two terms is positive.

P(A).P(B) > 0

And therefore, these two expressions are different from each other, therefore this equality P(A∩B) = P(A).P(B) that’s required by the definition of independence does not hold. The conclusion is that these two events are not independent.

In fact, intuitively, these two events are as dependent as Siamese twins. If you know that A occurred, then you are sure that B did not occur. So the occurrence of A tells you a lot about the occurrence or non-occurrence of B. So we see that being independent is something completely different from being disjoint. Independence is a relation about information. It is important to always keep in mind the intuitive meaning of independence. Two events are independent if the occurrence of one event does not change our beliefs about the other. It does not affect the probability that the other event also occurs.

When do we have independence in the real world?

The typical case is when the occurrence or non-occurrence of each of the two events A and B is determined by two physically distinct and non-interacting processes. For example, whether my coin results in heads and whether it will be snowing on New Year’s Day are two events that should be modeled as independent. But I should also say that there are some cases where independence is less obvious and where it happens through a numerical accident.