Bayesian Inference VII – Bayesian Updating Continuous Priors and Data

We finally got to the part of Bayesian Updating when both data and prior are continuous. It is, like always, not very different from Bayesian Updating with discrete data and priors except of that we use the PDFs and not the PMFs.

Continuous Data, Continuous Prior

When we use probability density functions we have to be careful, as the function itself doesn’t give us the probability. It is f(x|\theta)dx that gives us the probability. We therefore have to use dx and d\theta . This can be messy sometimes, so we often drop it.

Hypothesis Prior Likelihood unnormalised Posterior Posterior
\theta f(\theta) f(x|\theta) f(x|\theta)f(\theta) f(\theta|x)=\frac{ f(x|\theta)f(\theta) }{ f(x) }
\theta \pm \frac{ d\theta }{ 2 } f(\theta)d\theta f(x|\theta)dx f(x|\theta)f(\theta)dxd\theta f(\theta|x)=\frac{ f(x|\theta)f(\theta)dxd\theta }{ f(x)dx }=\frac{ f(x|\theta)f(\theta)d\theta }{ f(x) }
Total 1 f(x)dx=(\int{ f(x|\theta)f(\theta)d\theta })dx 1

Example: Suppose we have a normal distributed hypothesis and normal distributed data. So both data and hypothesis are continuous. Suppose data x is drawn from a normal distribution with unknown mean \theta and \sigma=1 .
Suppose further that \theta\sim N(2,1)

What is the posterior PDF?

Answer: Our prior PDF is given by f(\theta)=\frac{ 1 }{ \sqrt( 2\pi ) }exp^{ \frac{ -(\theta - 2)^{ 2 } }{ 2 } } and our likelihood PDF is given by f(x|\theta)=\frac{ 1 }{ \sqrt( 2\pi ) }exp^{ \frac{ -(x - \theta)^{ 2 } }{ 2 } } .

We need the product of our prior and likelihood for the unnormalised posterior so we can do these calculations first:

prior x likelihood = \frac{ 1 }{ \sqrt( 2\pi ) }exp^{ \frac{ -(\theta - 2)^{ 2 } }{ 2 } } \cdot \frac{ 1 }{ \sqrt( 2\pi ) }exp^{ \frac{ -(x - \theta)^{ 2 } }{ 2 } } = \frac{ 1 }{ 2\pi }exp^{ \frac{ -(2\theta^{ 2 }-(4+2x)\theta +4+x^{ 2 }) }{ 2 } }=\frac{ 1 }{ 2\pi }exp^{ \frac{ -(2(\theta^{ 2 }-(2+x)\theta) +4+x^{ 2 }) }{ 2 } }=\frac{ 1 }{ 2\pi }exp^{ \frac{ -(2(\theta-(1+\frac{ x }{ 2 }))^{ 2 } -(1+\frac{ x }{ 2 })^{ 2 } +4+x^{ 2 }) }{ 2 } }= \frac{ 1 }{ 2\pi }exp^{ \frac{ -(-(1+\frac{ x }{ 2 })^{ 2 } +4+x^{ 2 }) }{ 2 } } exp^{ \frac{ -2(\theta-(1+\frac{ x }{ 2 }))^{ 2 } }{ 2 } }=c_{ 1 }exp^{ -(\theta-( 1+\frac{ x }{ 2 } ))^{ 2 } }

x is a constant because we know the data.

If we now use bayesian updating we get the posterior PDF c_{ 2 }exp^{ -(\theta-( 1+\frac{ x }{ 2 } ))^{ 2 } } , what looks like a normal distribution with \mu_{ post. }=1+\frac{ x }{ 2 } and \sigma^{ 2 }=\frac{ 1 }{ 2 } . Furthermore we know the normalising constant for normal distributions: \frac{ 1 }{ \sigma \sqrt{ 2\pi } }

If we use 4.7 as our data, we get the following graphs:

bayesianupdatingcontinuous

The prior got pulled towards the data.

NOTE: I used the following python code for the graphs.

from scipy import stats as s
import numpy as np
import matplotlib.pyplot as plt
import math

data = 4.7

def prior(xs):
    ys  = []
    for x in xs:
        y = math.exp((-1*(x-2)**2)/2)/(math.sqrt(2*math.pi))
        ys.append(y)
    return ys

def posterior(xs):
    ys = []
    for x in xs:
        y = math.exp(-(x-(1+data/2))**2)/(math.sqrt(math.pi))
        ys.append(y)
    return ys

prior_x = [x for x in np.arange(-6,6,0.1)]
prior_y = [y for y in prior(prior_x)]

posterior_x = [x for x in np.arange(-6,6,0.1)]
posterior_y = [y for y in posterior(posterior_x)]

fid = plt.figure()
ax1 = plt.subplot2grid((1,1), (0,0))
ax1.plot(prior_x, prior_y, color='blue', label='prior')
ax1.plot(posterior_x, posterior_y, color='red', label='posterior, data={}'.format(data))

ax1.set_yticks(np.arange(0, 1.1, 0.1))
plt.legend()
plt.show()

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s