Introduction X – Covariance and Correlation

We often want to get a picture of how the relationship between two random variables is. In many cases or more precise in cases where the random variables are independent, we can get a picture of their relationship by inspecting the joint distribution, however that, as already mentioned, just works when the two random variables are independent of each another. In many cases they aren’t. In such cases we use covariance and correlation.

Covariance

The covariance of two random variables tells us how much these two random variables vary together. In other words: Is there any kind of linear relationship between the two random variables. Pay attention here: Covariance just measures the linear relationship.

If X and Y are random variables with means \mu_{ X } and \mu_{ Y } respectively, then:

Cov(X,Y)=E((X-\mu_{ X })(Y-\mu_{ Y }))

That is because if they have a positive relationship and X has a positive distance from its mean, Y will also have a positive distance from its mean. On the other hand if X has a negative distance from its mean Y will also have a negative distance from its mean. Since minus time minus equals plus, the expectation and therefore the covariance, will be some positive number if X and Y have a positive linear relationship.

On the contrary if X and Y have a negative linear relationship, Y will have a positive distance from its mean if X has a negative distance and vice versa. Therefore the expectation and the covariance will be negative for a negative linear relationship.

Discrete case:

If X and Y have joint PMF p(x_{ i }, y_{ j })

Cov(X,Y)=\sum_{ i }{\sum_{ j }{p(x_{ i },y_{ j })(x_{ i }-\mu_{ X })(y_{ j }-\mu_{ Y })}}=(\sum_{ i }{ \sum_{ j }{ p(x_{ i },y_{ j })x_{ i }y_{ j } } })-\mu_{ X }\mu_{ Y }

Continuous case:

If X and Y have joint PDF  f(x,y) over range [a,b]x[c,d]

Cov(X,Y)=\int_{ c }^{ d }{ \int_{ a }^{ b }{ (x-\mu_{ X })(y-\mu_{ Y })f(x,y)dxdy } }=(\int_{ c }^{ d }{ \int_{ a }^{ b }{ xyf(x,y)dxdy } })-\mu_{ X }\mu_{ Y }

Example: Suppose we toss a fair coin three times. Let X be the number of tails in the first two tosses and Y the number of tails in the last two tosses. Since the two overlap, there should be some covariance. What is the covariance of X and Y?

To answer this we need to have a look at the properties of covariance.

Properties:

  1. Cov(aX+b, cY+d)=acCov(X,Y) if a,b,c,d are constants.
  2. Cov(X_{ 1 }+X_{ 2 },Y)=Cov(X_{ 1 },Y)+Cov(X_{ 2 },Y)
  3. Cov(X,X)=Var(X)
  4. Cov(X,Y)=E(XY)-\mu_{ X }\mu_{ Y }
  5. Var(X+Y)=Var(X)+Var(Y)+2Cov(X,Y)
  6. if X and Y are independent: Cov(X,Y)=0 ; that doesn’t imply that if the covariance is 0 X and Y are independent.

Lets go back to our example.

Answer: Based on the given information we can build the following joint probability table:

X/Y 0 1 2
0 \frac{ 1 }{ 8 } \frac{ 1 }{ 8 } 0
1 \frac{ 1 }{ 8 } \frac{ 2 }{ 8 } \frac{ 1 }{ 8 }
2 0 \frac{ 1 }{ 8 } \frac{ 1 }{ 8 }

We can then calculate that E(X)=E(Y)=1 and E(XY)=\frac{ 5 }{ 4 }

The covariance is then: Cov(X,Y)=E(XY)-\mu_{ X }\mu_{ Y }=\frac{ 5 }{ 4 }-1=\frac{ 1 }{ 4 }

The problem that occurs now is how to interpret that value. How much do they vary together? Using the covariance one can’t answer that question as the units of the covariance are “units of X times units of Y”. To be able to compare covariances we have to use correlation. The correlation transforms the covariance into a ratio which is always between -1 and 1.

Correlation \rho

The correlation is denoted \rho “rho” and is calculated as following:

Cor(X,Y)=\rho=\frac{ Cov(X,Y) }{ \sigma_{ X }\sigma_{ Y } }

Properties:

  1. \rho is a ratio and therefore dimensionless
  2. -1\le\rho\le 1
  3. \rho=1 if and only if Y=aX+b, a>0
  4. \rho=-1 if and only if Y=aX+b, a<0

The correlation is always between -1 and 1 because the covariance cant exceed the \sigma_{ X }\sigma_{ Y } .

Covariance and Correlation are powerful tools and we will see them quite often in the future.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s