Bayesian regression algorithm for machine learning

bayesian regression is a probablistic method using in machine learning algorithms

1. Bayesian Linear Regression

2. BayesRule
•Estimatestheposteriorprobabilityofanevent,basedonprior
knowledgeofevidences(observations)
•Event/Hypothesis(h):OutcomeofExperiment
Ex:OccurrenceofRain:(Rain=YES,Rain=NO)
•Evidence/Data(d):Observations
Ex:ClimaticConditions:(Darkcloud=YES,Wind=speedy)
•FindsposteriorprobabilityofRaineventbasedonpriorknowledgeof
climaticconditions

3. Bayes’ Rule)(
)()|(
)|(
dP
hPhdP
dhp  ) data the seen having after hypothesis ofty (probabili posterior
data) the ofy probabilit (marginal evidence data
is hypothesis the if data the ofty (probabili likelihood
data)any seeing before hypothesis ofty (probabili belief prior
dh
h
h
:)|(
:)()|()(
true) :)|(
:)(
dhP
hPhdPdP
hdP
hP
h
 sides both on
y probabilitjoint same the
),(),(
)()|()()|(
grearrangin -
(model) hypothesish
datad
rule Bayes'ingUnderstand
hdPhdP
hPhdPdPdhp





4. •Ex:)(
)()|(
)|(
CP
YesRainPYesRainCP
CYesRainP

 )conditionsclimatic the on based Rain getting ofty (probabili Posterior
evidence? getting ofy probabilit the is what(Rain) outcome past the (Given likelihood
) Rain the getrting ofty (probabili belief prior
:)|(
evidence thegetting ofy probabilitPrior :)()|()(
) :)|(
:)(
CRainP
RainPRainCPCP
RainCP
RainP
h


5. Example of BayesTheorem
•Given:
•A doctor knows that covid causes fever 50% of the time
•Prior probability of any patient having covid is 1/50,000
•Prior probability of any patient having fever is 1/20
•If a patient has fever, what’s the probability he/she has covid?0002.0
20/1
50000/15.0
)(
)()|(
)|( 


SP
MPMSP
SMP

6. Linear Regression
•Thelinearregressionmodelassumesthattheresponsevariable(y)isa
linearcombinationofweightsmultipliedbyasetofpredictorvariables
(x).Thefullformulaalsoincludesanerrortermtoaccountforrandom
samplingnoise.Forexample,ifwehavetwopredictors,theequation
is
•yistheresponsevariable(alsocalledthedependentvariable)
•β’saretheweights(knownasthemodelparameters),
•x’sarethevaluesofthepredictorvariables,and
•εisanerrortermrepresentingrandomsamplingnoiseortheeffectof
variablesnotincludedinthemodel.

7. •Wecangeneralizethelinearmodeltoanynumberofpredictorsusing
matrixequations.Addingaconstanttermof1tothepredictormatrixto
accountfortheintercept,
•wecanwritethematrixformulaas:
•Thegoaloflearningalinearmodelfromtrainingdataistofindthe
coefficients,β,thatbestexplainthedata.Inlinearregression,thebest
explanationistakentomeanthecoefficients,β,thatminimizetheMean
SquareError(MSE)orResidualSumofSquares(RSS).

8. •RSSisthetotalofthesquareddifferencesbetweentheknownvalues
(y)andthepredictedmodeloutputs(ŷ,pronouncedy-hatindicatingan
estimate).Theresidualsumofsquaresisafunctionofthemodel
parameters:
•Thisequationhasaclosedformsolutionforthemodelparameters,β,
thatminimizetheerror.Thisisknownasthemaximumlikelihood
estimateofβbecauseitisthevaluethatisthemostprobablegiventhe
inputs,X,andoutputs,y.

9. •Theclosedformsolutionexpressedinmatrixformis:
•Thismethodoffittingthemodelparametersbyminimizingthe
RSS/MSEiscalledOrdinaryLeastSquares(OLS).
•Whatweobtainfromfrequentistlinearregressionisasingleestimate
forthemodelparametersbasedonlyonthetrainingdata.Ourmodelis
completelyinformedbythedata:inthisview,everythingthatweneed
toknowforourmodelisencodedinthetrainingdatawehave
available.
•Oncewehaveβ-hat,wecanestimatetheoutputvalueofanynewdata
point by applying our model equation:

10. Bayesian Linear Regression
•IntheBayesianviewpoint,weformulatelinearregressionusing
probabilitydistributionsratherthanpointestimates.Theresponse,y,is
notestimatedasasinglevalue,butisassumedtobedrawnfroma
probabilitydistribution.ThemodelforBayesianLinearRegression
withtheresponsesampledfromanormaldistributionis:
•Theoutput,yisgeneratedfromanormal(Gaussian)Distribution
characterizedbyameanandvariance.Themeanforlinearregression
isthetransposeoftheweightmatrixmultipliedbythepredictor
matrix.Thevarianceisthesquareofthestandarddeviationσ
(multipliedbytheIdentitymatrixbecausethisisamulti-dimensional
formulationofthemodel).

11. •TheaimofBayesianLinearRegressionisnottofindthesingle“best”
valueofthemodelparameters,butrathertodeterminetheposterior
distributionforthemodelparameters.
•Notonlyistheresponsegeneratedfromaprobabilitydistribution,but
themodelparametersareassumedtocomefromadistributionaswell.

12. •Theposteriorprobabilityofthemodelparametersisconditionalupon
thetraininginputsandoutputs:
Here,P(β|y,X)istheposteriorprobabilitydistributionofthemodel
parametersgiventheinputsandoutputs.
Thisisequaltothelikelihoodofthedata,P(y|β,X),multipliedbythe
priorprobabilityoftheparametersanddividedbya
normalization constant.

13. •ThisisasimpleexpressionofBayesTheorem,thefundamental
underpinningofBayesianInference:
•IncontrasttoOLS,wehaveaposteriordistributionforthemodel
parametersthatisproportionaltothelikelihoodofthedatamultiplied
bythepriorprobabilityoftheparameters

14. PrimarybenefitsofBayesianLinear
Regression.
•Priors:Ifwehavedomainknowledge,oraguessforwhatthemodel
parametersshouldbe,wecanincludetheminourmodel,unlikeinthe
frequentistapproachwhichassumeseverythingthereistoknowabout
theparameterscomesfromthedata.Ifwedon’thaveanyestimates
aheadoftime,wecanusenon-informativepriorsfortheparameters
suchasanormaldistribution.
•Posterior:TheresultofperformingBayesianLinearRegressionisa
distributionofpossiblemodelparametersbasedonthedataandthe
prior.Thisallowsustoquantifyouruncertaintyaboutthemodel:ifwe
havefewerdatapoints,theposteriordistributionwillbemorespread
out.