Zachary Binney, an epidemiologist and incoming Assistant professor at Oxford College at Emory university, shared a very interesting twitter thread on how to interpret the new Serological test approved by the FDA.

A serology test is a blood based test to check antibody production against a specific condition. The idea behind an antibody test is to detect antibodies to Covid-19 in your blood. Since we have no vaccine yet, these antibodies would presumably mean that you have had Covid-19 at some point in the past.

The FDA has approved the first antibody test for COVID-19, from Cellex. It theoretically tells you if you've had it & are, as far as we know, immune for some time.

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020

Sensitivity is 93.8%, specificity 95.6%. Sounds great, right?

Well, sort of. (1/6)

These tests are however not perfect. You might have had the disease, but your test might show up as negative. This is called a **false negative**. On the other hand, you might not have had the disease, but you get a positive test. This is called a **false positive**. These errors are what the sensitivity and specificity in the above tweet are about: they represent probabilities of these two different kinds of errors. They are numbers *given to us by the test manufacturer*. To understand them, we’ll need to recall our basic notions of probability.

Zachary goes on to say: “Sounds great, right? Well, sort of.”. The numbers are in the 90%s. Surely that is good. But not so fast. To understand these numbers, we’ll need to understand something about the **prevalence** of Covid-19 in the population, and along with that, **conditional probability** and the famous **Bayes Theorem**.

## Probability

Probability can be thought of as the long run frequency of events. For example, the probability of testing positive, \(P(T+)\) can be thought of as *the number of positive tests in a very large set of tests* conducted. This and the other probability dartboard images are taken from Andrew Glassner’s amazing Deep Learning, Vol. 1: From Basics to Practice.

We can visualize this idea of probability geometrically. Imagine that you are drunk, and are throwing darts at a square shaped wall. The reverse idea of using frequencies to calculate areas of geometric objects (here the circle) can be used to calculate \(\pi\) (think of how!), and in general, all kinds of integrals. This is called the *Monte Carlo* method.

And you do this for a very very long time and thus throw a very large number of dots. Lets furthermore assume that you are a very special thrower, and it is equally likely that you throw darts anywhere in this square.

Now you can ask the question, whats the probability that your darts land in the circle. You have probably intuited the answer as shown on the right. Its:

\[P(darts \, in \, circle) = \frac{\# \, darts \, in \, circle}{\# \, darts \, in \, square}\]

## Joint and Conditional probability

We can use this idea of probability as *relative frequency* of darts to understand why these tests are not as useful as they might seem at first blush. Imagine two areas on the square where the darts arrive, \(A\) and \(B\). We can now ask two questions:

**Joint Probability**:

What are the odds that the darts arrive in the area that is the intersection of the regions \(A\) and \(B\). This is called the joint probability of being in \(A\) and \(B\), denoted \(P(A, B)\). We can compute this by counting the number of darts that arrive in the intersection area, relative to the number of darts that arrive in the square.

**Conditional Probability**:

Suppose I told you that a dart has arrived in region \(B\). I can now ask the question:*given*this information, what are the odds that it arrived in region \(A\). The word*given*is critical here, it presupposes that an event has happened, and we are asking a question*conditioned*on its occurrence. As you might expect, you can answer this question using relative frequencies as well: out of all the darts that arrived in region \(B\), lets count those that arrived in region \(A\) and ratio the two numbers.

The critical question we wish to answer is a conditional probability Almost all critical questions in life and death are conditional probabilities. As Joe Blitzstein (the famous @stat110) will tell you within 5 minutes of meeting him: **Conditioning is the soul of Statistics**.

: *given that we have a positive test, what are the odds that we have had the Covid-19 disease*. In other words we want the quantity

\[P(D+ | T+).\]

In other words, **do you have antibodies (have you had the disease?), given that you have a positive test** and thus can you get back to work and get the economy back on track?

At this point you might be thinking, wait, I just got a *positive* test. How can I *not have the disease*? Read on….

## Sensitivity and Specificity

Ok, so what exactly are the *sensitivity* and *specificity*. Zachary quotes a sensitivity of 93.5%. What does this number mean?

The **sensitivity** is the reverse conditional probability to what we want, the probability that you test positive given that you have the disease:

\[ Sensitivity \equiv P(T+ | D+) = 0.938 \]

This means that if you have had the disease, there is only a 93.8% chance you will test positive. When you have had the disease, and you test positive, you are called a **true positive**.

On the other hand you might be a **false negative**. This is the equivalent probability of being in the region of A that is not in B.

In other words, there a \(100 - 93.8 = 6.2\)% chance that you will test negative, even though you have had the disease.

Since, given disease, you either test negative or positive:

\[ P(T- | D+) = 1 - P(T+ | D+) = 1 - 0.938 = 0.062\]

**Specificity** is the other side of the coin. It answers the question: what are the odds that you will test negative if you dont have the disease.

\[ Specificity \equiv P(T- | D-) = 0.956 \]

In other words, it asks the question, are you a **true negative**: do you test negative when you do not have the disease?

This then begs the question: what if you are a **false positive**? What if you test positive, given that you do not have the disease?

Since, given no disease, you either test negative or positive:

\[P(T+ | D-) = 1 - P(T- | D-) = 1 - 0.956 = 0.044\]

There thus seems to be 2 ways in which this test has errors: you can be a false positive, or a false negative. Indeed, all tests have such errors. There are biological reasons for this. You might be a false negative if antibodies in your blood do not show up on the test because of confounding factors in your biology. You might be a false positive, if the test reacts to antibodies to other viruses (other coronaviruses even) in your body.

## The Confusion Matrix

These errors are usually captured in a construct called the Confusion Matrix.

Suppose you had real frequencies for the false positives, true positives, true negatives and false negatives. I want to emphasize we do not have these, just numbers given to us by the test manufacturer. But suppose, just for a bit, that we did.

The marginal quantities (quantities in the right and the bottom margin in the confusion matrix) are the sum of the rows and columns. For example, when you sum the true negatives and the false positives, you get the observed negatives (ON), all the people who do not have the disease.

The Confusion matrix deals with actual numbers and not probabilities, and so we must write the various quantities we have calculated in the previous section as a function of the quantities in this matrix.

The Sensitivity, or the **True Positive Rate**(TPR) is given as: \[
Sensitivity \equiv P(T+ | D+) = TPR = \frac{TP}{OP} = 0.938.
\] In other words, this is the number of TP, divided by the sum of the bottom row, the OP.

The complement of the sensitivity is the **False Negative Rate**(FNR):

\[FNR \equiv P(T- | D+) = 1 - TPR = \frac{FN}{OP} = 0.062\]

Again this quantity is concerned with the *bottom row* of the confusion matrix.

The Specificity, or the **True Negative Rate**(TNR) is given as: \[
Specificity \equiv P(T- | D-) = TNR = \,\,\frac{TN}{ON} = 0.956
\]

and thus the complement, the **False Positive Rate**(FPR) is given as:

\[FPR \equiv P(T+ | D-) = 1 - TNR = \frac{FP}{ON} = 0.044\]

These two quantities are concerned with the *top row* of the confusion matrix, and the denominators come from its marginal, the ON.

At this point, you might be saying, the FPR is very low, so what are we worried about? Well, we have not answered the question that we set out to answer: what is

\[P(D+ | T+) \, ?\]

Do you have antibodies (have you had the disease?), given that you have a positive test and thus can you get back to work and get the economy back on track?

How do we calculate this?

From the perspective of the confusion matrix, we are looking at a different marginal, the marginal of the right column, the Predicted Positive, or PP.

\[ P(D+ | T+) = \frac{TP}{PP} = \frac{TP}{TP + FP} \]

Thus we have:

\[ \begin{eqnarray*} P(D+ | T+) &=& \frac{TP}{TP + FP} \\ &=& \frac{TPR \times OP}{TPR \times OP + FPR \times ON} \\ &=& \frac{TPR \times \frac{OP}{Pop}}{(TPR \times \frac{OP}{Pop}) + (FPR \times \frac{ON}{Pop})} \\ &=& \frac{P(T+ | D+) \times P(D+)}{(P(T+ | D+) \times P(D+)) + (P(T+ | D-) \times P(D-))} \\ &=& \frac{0.938 \times P(D+)}{(0.938 \times P(D+)) + (0.044 \times P(D-))} \end{eqnarray*} \]

Here I divided both the numerator and the denominator of the fraction by the population (\(Pop\)) to get the **prevalence** of Covid19, the fraction of the population that has Covid19, or the probability **prior** to serological testing, presumably estimated from regular testing or mortality, that a random person in the population has Covid19.

This tells us that our probability of disease given testing \(P(D+ | T+)\) can be estimated from our confusion matrix if we had the raw numbers. But in the absence of those, given just the probabilities from the test manufacturer, we can still estimate \(P(D+ | T+)\) as long as we have the prevalence of disease \(P(D+)\) since \(P(D-) = 1 - P(D+)\).

If only a small % have actually had COVID-19 (our best guess now) a “positive” antibody test isn't that likely to mean you're immune.

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020

If only 4.5% of U.S. has had COVID-19, + test only means ~50% chance you really had it. With lots of uninfected, lots of false +s. (2/6) pic.twitter.com/IIqnNSM9Lg

Zachary assumes prevalence \(p = P(D+) = 0.045\) from the above tweet. Then the prevalence of the “lack of disease” is

\[P(D-) = 1 - P(D+) = 1 - p = 1 - 0.045 = 0.955,\]

and we can use these numbers to calculate the probability if disease given test:

\[ P(D+ | T+) = \frac{0.938*0.045}{(0.938*0.045 + 0.044*0.955)} \sim 0.501.\]

This explains the 50% number in the tweet above.

The calculation we did above is an example of a very general calculation in probability theory: a conditional probability calculation using Bayes theorem. You may skip this section if you are not interested, but i must warn you, it is fun.

## Bayes Theorem

We’ll rely on an almost 260 year old theorem, which is so simple to prove, that we shall show its proof visually below. Bayes theorem may be the highest utility to simplicity ratio theorem out there. It is used in almost every field of human endeavour. Its originator was the Reverend Bayes, and the theorem was presented in *“An Essay towards solving a Problem in the Doctrine of Chances”*. This drawing of him is taken from Wikipedia and is not guaranteed to be him..no drawing survives!

The basic idea of Bayes theorem comes from the definition of joint probability. As illustrated below, the joint probability \(P(A, B)\) of the dart falling in regions \(A\) and \(B\) can be decomposed into the product of the conditional probability \(P(A | B)\) and the full or *marginal* probability of the dart falling in region \(B\), \(P(B)\):

This should be obvious from the figure above. Here’s the fun part: there is nothing special about regions \(A\) and \(B\), so we can reverse the argument.

I reproduce the image of regions \(A\) and \(B\) and the joint probability here on the right for convenience.

Thus the joint probability can be written two ways, and voila, we get Bayes Theorem:

Remember how we had \(P(T+ | D+)\) but wanted \(P(D+ | T+)\). Now we can apply the theorem to our advantage:

\[P(D+, T+) = P(D+ | T+) P(T+) = P(T+ | D+) P(D+)\]

Solving for \(P(D+ | T+)\), we get:

\[ P(D+ | T+) = \frac{P(T+ | D+) P(D+)}{P(T+)} \]

To expand out the denominator, remember that When you test positive, it can either be because you have the disease, or because you dont have the disease:

\[ \begin{eqnarray*} P(D+ | T+) &=& \frac{P(T+ | D+) P(D+)}{P(T+, D+) + P(T+, D-)} \\ &=& \frac{P(T+ | D+) P(D+)}{P(T+ | D+) P(D+) + P(T+ | D-)P(D-)} \\ &=& \frac{0.938 \times P(D+)}{0.938 \times P(D+) + 0.044 \times P(D-)} \end{eqnarray*} \]

The marginal quantity \(P(D+)\), the fraction of people in the population with disease established by other methods is often called the **prior** probability of disease, because it reflects our belief in the disease rate without having done the testing we are interested in (without having seen “data”).

## Gird yourself for disappointment

If only a small % have actually had COVID-19 (our best guess now) a “positive” antibody test isn't that likely to mean you're immune.

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020

If only 4.5% of U.S. has had COVID-19, + test only means ~50% chance you really had it. With lots of uninfected, lots of false +s. (2/6) pic.twitter.com/IIqnNSM9Lg

So using bayes theorem or our confusion matrix based calculation, we have all the numbers we need to finish our calculation:

\[ P(D+ | T+) = \frac{0.938*0.045}{(0.938*0.045 + 0.044*0.955)} \sim 0.501\]

Remember, this explains the 50% number in the tweet above.

Take in a moment to absorb this. Because the disease prevalence in the population is currently low (and we want to keep it that way), Bayes Theorem tells us that the probability of having had the disease given a positive serology test is just 50%. Not the slam dunk we were hoping to have.

How do these odds change with the prevalence of the disease? This is told to us by the plot in the tweet above and the tweet included below.

How do we produce a curve like the above? We’ll make a grid of prevalences p from 0 to 1, and produce \(P(D+ | T+)\) for each prevalence, and then plot it:

```
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
# Implement Bayes theorem as a function of prevalence
dplus_given_tplus = lambda p: 0.938*p / (0.938*p + 0.044*(1.-p))
pgrid = np.linspace(0., 1., 100)
plt.plot(pgrid, plus_given_tplus(pgrid))
```

The plot is shown in the margin here You can read off that at a prevalence of 0.1 (x-axis), the probability of disease having happened given a +ive test is 0.7. And that for \(p=0.3\), its 0.9.

and can be used to read the numbers off the next tweet.

If 10% were truly infected, a positive Cellex antibody test has a 70% chance of being right. If 30% were infected, a positive test is right 90% of the time. This happens bc when more people were sick, true positives overwhelm false positives. But that's not our situation. (3/6)

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020

Why does the probability of having had the disease, given a positive test go up with increased disease prevalence? From the perspective of what we are trying to understand, it might have almost been better to draw the confusion matrix with a thinner lower strip to indicate that the total number of observed positives is a smaller fraction of the population than the observed negatives, or those who do not a-priori have the disease.

This is alluded to in the tweet above: when the prevalence is higher and there are more sick people, true positives overwhelm the false positives. The way to think about this is by using the confusion matrix. False positives come from the top row of the confusion matrix. This row totals up to the observed negatives. The bottom row, which holds the true positives, totals up to the observed positives, or people who have had Covid19. Thus, as the prevalence increases, the true positives overwhelm the false positives.

So this is test may not be that useful for saying “Zach, you are immune; Jen, you aren't.” It might be wrong as often as it's right. Mistakenly telling someone they're immune & clear to return to society…you can see the problem if we do that on a large scale. (4/6)

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020

Remember that our social distancing, lockdown, and quarantine measures are designed to keep the disease prevalence low. If we succeed in doing this, we should not expect just a simple serology test to be enough to return people to the workforce.

## How should we use this test, then?

Well, one way to use this test is to narrow its scope, and estimate prevalence in geographic subpopulations, Seattle, for example, using the confusion matrices calculated from random samples taken in these geographies (this kind of inverts what we have been talking about here).

There are simple equations to correct for this on a population level. So this test is still

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020veryuseful for helping us figure out what % of people have been sick in different areas. And it's the best we've got; deploy it! But realize what it will & won't reliably tell us. (5/6)

Another way to use this test is to repeat it twice over a longer timeline per person, thus eliminating false positives. But if there are systematic biological effects that created the false positives in the first place, then we are in trouble.

Running the test 2x & only telling someone they're immune if they get 2 +s

— Zachary Binney, PhD (@zbinney_NFLinj) April 2, 2020mighthelp reduce false +s, depending on the error source. If it's anything systemic - say it's detecting antibodies from a similar virus that don't grant immunity to COVID-19 - no good. @KevinMalogna

Or do a different test:

Bonus tweet 2: We might be able to reduce false +s/increase the chance a + test is right by using this test as a screener and another slightly different antibody test to confirm, if they wouldn't both show false + for the same systemic reason like antibodies to a similar virus.

— Zachary Binney, PhD (@zbinney_NFLinj) April 3, 2020

The test is, of-course more useful in Covid high risk groups because the prevalence is higher:

Bonus tweet 1: A positive test would be more likely to mean you're truly immune if you're in a high-risk group - healthcare worker, had COVID symptoms, family member had COVID - bc prevalence of infection in these subgroups is higher. So test may be more useful for these folks!

— Zachary Binney, PhD (@zbinney_NFLinj) April 3, 2020