Mastering Linear Regression and Polynomial Fitting Techniques
School
Queens University**We aren't endorsed by this school
Course
ENPH 213
Subject
Mathematics
Date
Dec 10, 2024
Pages
7
Uploaded by SargentMongoosePerson1241
ENPH213InterpoThere are 1)FpoF2)FThe3 olation:two basic typind the equatioints. or linear regrind the equatiere are 4 poinpes of problemion of the lineession we neeion that passents so to fit wiLectums: e that best fitsed 2-coefficiees through all ith a polynomures 11 and 12s the data, butents, ݕ ൌ ܽ0 the data poinmial we need 42t doesn’t nece ܽ1ݔ. nts. 4-coefficientsessarily go ths: ܽ0, ܽ1, ܽ2, ahrough all theand ܽ3.1 data
ENPH213 Lectures 11 and 12 2 We will start looking at the first type of problem and will generalize our code to work for any order of polynomial. Problem 1: Start with linear regression. Our goal is to find ܽ0 and ܽ1that best fit the data. The best approach is to use a minimal squares approach since this will maximize the probability that the data fits the line. We can define the error for the i-th point as (ܽ0 ܽ1ݔ݅ሻ െ ݕ݅where the bracketed term is the theory and the non-bracketed term is the experiment. Adding up the errors gives: ∑ ݏݍݑܽݎ݁ݏ ݂ ݐ݄݁ ݁ݎݎݎൌ ቄ∑൫ሺܽ0 ܽ1ݔ݅ሻ െ ݕ݅൯ଶቅ, Choose ܽ0and ܽ1to minimize the error. Differentiate with respect to a0 and set equal to zero and repeat for a1. This gives us a set of two equations with two unknowns : ߲ሼ ሽ/߲a0 ൌ 2 ൫ሺܽ0 ܽ1ݔ݅ሻ െ ݕ݅൯ ∙ 1 ൌ 0orܽ0 ∑ ݔ݅ ܽ1 ∑ ݔ݅ଵെ ∑ ݕ݅ൌ 0ሺ ݔ݅ൌ 1ሻor ܽ0 ∑ ݔ݅ ܽ1 ∑ ݔ݅ଵൌ ∑ሺݕ݅ݔ݅ሻSimilarly ߲ሼ ሽ/߲ܽ1ൌ 2 ∑൫ሺܽ0 ܽ1ݔ݅ሻ െ ݕ݅൯ ݔ݅ ൌ 0or ܽ0 ∑ ݔ݅ଵ ܽ1 ∑ ݔ݅ଶൌ ∑ ݕ݅ ݔ݅ଵWe can write the two equations as the following: ∑ ݔ݅∑ ݔ݅ଵ∑ ݔ݅ଵ∑ ݔ݅ଶ൨ ቂܽ0ܽ1ቃൌ ∑ ݔ݅ݕ݅∑ ݔ݅ଵݕ݅൨It is easy to write some code. Let ݔ ൌ ሾݔ1 ݔ2 ݔ3 ݔ4 … ݔܰሿand ݕ ൌ ሾݕ1 ݕ2 ݕ3 ݕ4 … ݕܰሿThe sum of ݔ݅is given by ∑ ݔ݅ൌN the sum of ݔ݅ଵis given by ∑ ݔ݅ଵൌ sumሺݔ. ^1ሻthe sum of ݔ݅ଶis given by ∑ ݔ݅ଶൌ sumሺݔ. ^2ሻ
ENPH213 Lectures 11 and 12 3 Similarly, the sum of ሺݔ݅ݕ݅ሻis given by ∑ ݔ݅ݕ݅ ൌ N ∗ sumሺݕሻ, and the sum of ሺݔ݅ଵݕ݅ሻis given by sumሺݔ. ^1 ∗ ݕሻ .Solve the system using your inverse code and you are done. It is easy to extend to higher order polynomials. For example: ݕൌ ܽ0 ܽ1ݔ ܽ2ݔଶ. You now need at least 3 points and you have 3-coefficients to solve. ∑ Squares of the errorsൌ ∑ ሾܽ0ݔ݅ ܽ1ݔ݅ଵ ܽ2ݔ݅ଶെ ݕ݅ሿଶேଵDifferentiate with respect to ܽ0and set equal to zero: 2 ∑ሾܽ0ݔ݅ ܽ1ݔ݅ଵ ܽ2ݔ݅ଶെ ݕ݅ሿݔ݅ൌ 0or ܽ0 ∑ ݔ݅ ܽ1 ∑ ݔ݅ଵ ܽ2 ∑ ݔ݅ଶൌ ∑ ݔ݅ݕ݅Differentiate with respect to ܽ1and set equal to zero: 2 ∑ሾܽ0ݔ݅ ܽ1ݔ݅ଵ ܽ2ݔ݅ଶെ ݕ݅ሿݔ݅ଵൌ 0or ܽ0 ∑ ݔ݅ଵ ܽ1 ∑ ݔ݅ଶ ܽ2 ∑ ݔ݅ଷൌ ∑ ݔ݅ଵݕ݅Finally, differentiate with respect to ܽ2and set equal to zero: ܽ0 ∑ ݔ݅ଶ ܽ1 ∑ ݔ݅ଷ ܽ2 ∑ ݔ݅ସൌ ∑ ݔ݅ଶݕ݅This gives you the following system that you can solve for ܽ0, ܽ1, and ܽ2∑ ݔ݅∑ ݔ݅ଵ∑ ݔ݅ଶ∑ ݔ݅ଵ∑ ݔ݅ଶ∑ ݔ݅ଷ∑ ݔ݅ଶ∑ ݔ݅ଷ∑ ݔ݅ସ ܽ0ܽ1ܽ2൩ൌ ∑ ݔ݅ݕ݅∑ ݔ݅ଵݕ݅∑ ݔ݅ଶݕ݅You can solve for higher order polynomials too, although a third or a fourth order polynomial seems to work best. We can extend this type of analysis to other types of curves too. For example: ݖ ൌ ∝ ݁ఋ௫. Given values of ݔ and ݖfind values for ∝ and ߜ. One approach would be to turn it into a polynomial and then use your polynomial code to solve for ܽ0 and ܽ1(and at the same time solving for ∝ and ߜ). For example: ݖൌ ∝ ݁ఋ௫Take natural logs (Note: the natural log in MATLAB is log) lnሺݖሻൌ ln൫∝ ݁ఋ௫൯ൌ lnሺ∝ሻ ln൫݁ఋ௫൯ൌ lnሺ∝ሻ ߜݔYou can now let lnሺݖሻ ൌ ݕ , lnሺ∝ሻ ൌ ܽ0, and ߜ ൌ ܽ1and you are done.
ENPH213Problem 2The goal hݕ ൌ ܽ0. IN-1. For eon. There points canthe curve The resultA better sto dot imaThis is bederivative3 2: here is to solvf you have N example, giveis a problemn have a largecan become uting graph wiolution wouldage. Here, a sietter than a sixes are not conve for an equapoints then then 2 points th. As your code change on thunstable. For ll oscillate asd be to simplyimple dot-dotxth order polyntinuous. Lectuation that pashe order of thhen ݕ ൌ ܽ0 de gets more che shape of thN=7, we canshown in they join the linet image can bynomial but wures 11 and 12ses through ahe polynomialܽ1ݔ, and givcomplicated, he graph. Evenn fit (althoughe figure belowes using a lowbe formed usinwe get sharp c2all the data pol which will pven 3 points: ݕa small changn once you hah foolishly) a w: wer order polyng straight lincorners whichoints. If you hpass through eݕ ൌ ܽ0 ܽ1ge in one of thave a perfect 6thorder polyynomial so thne segments. h are nonphyshave one pointevery point wݔ ܽ2ݔଶandhe values of tfit, the shapeynomial to a ghat they form sical i.e. the 14 t then will be d so the e of graph. a dot st
ENPH213Alternativand then ccurves me(please beExample: Given N=There is aLet Note: still3 vely, we coulchoose coeffieet. This appre patient). Piecewise Q=4 points, solva nice trick thal a quadratic, ld use a seconݕ ൌ ܽ0 ܽcients a0, a1, roach is calleduadratic Interve for 3-sets oat makes solv݃1ሺݔሻൌ݃2ሺݔሻൌ݃3ሺݔሻൌjust a differenLectund order polynܽ1ݔ ܽ2ݔଶa2 in each red a ‘piecewiserpolation of ܽᇱݏ, ܾᇱݏ an݃1ሺ݃2ሺ݃3ሺving the equatܽ1 ܾ1ሺݔ െܽ2 ܾ2ሺݔ െܽ3 ܾ3ሺݔ െnt reference pures 11 and 12nomial for eaegion so that te interpolationd ܿ′ݏ. This wሺݔሻൌ ܽ1 ܾሺݔሻൌ ܽ2 ܾሺݔሻൌ ܽ3 ܾtions easier: െ ݔ1ሻ ܿ1ሺݔെ ݔ2ሻ ܿ2ሺݔെ ݔ3ሻ ܿ3ሺݔpoint for ݔ. 2ach region, the 1stderivatn”. It soundswill give us onܾ1ݔ ܿ1ݔଶܾ2ݔ ܿ2ݔଶܾ3ݔ ܿ3ݔଶݔ െ ݔ1ሻଶݔ െ ݔ2ሻଶݔ െ ݔ3ሻଶtives are the ss easy but it isne curve for easame where ths a bit of workach region: 5 he k
ENPH213 Lectures 11 and 12 6 Our task is to solve for the ܽᇱݏ, ܾᇱݏ and ܿᇱݏ given the ݔᇱݏ and ݕ′ݏ. We have 9 unknowns therefore we need 9 equations. Solving for the ܽ′ݏis easy: we know that ݃1ሺݔ1ሻ ൌ ݕ1and therefore that the following is true: ܽ1ൌ ݕ1[1] ܽ2ൌ ݕ2[2] ܽ3ൌ ݕ3[3] Next look at the derivatives; we can differentiate with respect to ݔand we find that the following is true: ݃1ᇱሺݔሻൌ ܾ1 2ܿ1ሺݔ െ ݔ1ሻ, ܽݐ ݔ ൌ ݔ1, ݃1′ሺݔ1ሻ ൌ ܾ1݃2′ሺݔሻൌ ܾ2 2ܿ2ሺݔ െ ݔ2ሻ, ܽݐ ݔ ൌ ݔ2, ݃2′ሺݔ2ሻ ൌ ܾ2݃3′ሺݔሻൌ ܾ3 2ܿ3ሺݔ െ ݔ3ሻ, ܽݐ ݔ ൌ ݔ3, ݃3′ሺݔ3ሻ ൌ ܾ3Therefore the ܾᇱݏgive the first derivatives at the ݔ݅′ݏ. We want the ܾᇱݏto produce continuous first derivatives: ݃1′ሺݔ2ሻൌ ݃2′ሺݔ2ሻwhere ݃1ᇱሺݔ2ሻis the derivative at the end of the first region and ݃2ᇱሺݔ2ሻis the derivative at the start of the second region. We also know that ݃2′ሺݔ2ሻൌ ܾ2and therefore at the right side of the first region that ܾ2ൌ ܾ1 2ܿ1ሺݔ2 െ ݔ1ሻor ܿ1ൌ ሺ1/2ሻሺܾ2 െ ܾ1ሻሻ/ሺݔ2 െ ݔ1ሻ[4] Similarly, ܿ2ൌ ሺ1/2ሻሺܾ3 െ ܾ2ሻ/ሺݔ3 െ ݔ2ሻ[5] ܿ3ൌ ሺ1/2ሻሺܾ4 െ ܾ3ሻ/ሺݔ4 െ ݔ3ሻ[6] (ܾ4gives that slope at the end of the last region. I will talk more about this later). In addition to continuous first derivatives we also need to match the curves at the start and end of each region. ݃1ሺݔ2ሻൌ ݃2ሺݔ2ሻ ൌ ݕ2[7] ݃2ሺݔ3ሻൌ ݃3ሺݔ3ሻ ൌ ݕ3[8] ݃3ሺݔ4ሻൌ ݕ4[9] We now have 9 equations in 10 unknowns (ܽ1, ܽ2, ܽ3, ܾ1, ܾ2, ܾ3, ܾ4, ܿ1, ܿ2, ܿ3ሻ.
ENPH213 Lectures 11 and 12 7 We already know the ܽvalues. Solve next for the ܾvalues. Given the ܾvalues it is easy to solve for the ܿvalues (sub into equations 4, 5, and 6). Start by looking at equation 7: ݃1ሺݔ2ሻൌ ݃2ሺݔ2ሻthis can be re-written ܽ1 ܾ1ሺݔ2 െ ݔ1ሻ ܿ1ሺݔ2 െ ݔ1ሻଶൌ ܽ2Substituting in equations [1], [2] and [4] ݕ1 ܾ1ሺݔ2 െ ݔ1ሻ ଵଶቀଶିଵ௫ଶି௫ଵቁ ሺݔ2 െ ݔ1ሻଶൌ ݕ22ݕ1 2ܾ1ሺݔ2 െ ݔ1ሻ ሺܾ2 െ ܾ1ሻሺݔ2 െ ݔ1ሻൌ 2ݕ2ܾ1 ܾ2ൌ 2ሺݕ2 െ ݕ1ሻ/ሺݔ2 െ ݔ1ሻSimilarly, equation 8 can be written: ܾ2 ܾ3ൌ 2ሺݕ3 െ ݕ2ሻ/ሺݔ3 െ ݔ2ሻAnd finally equation 9: ܾ3 ܾ4ൌ 2ሺݕ4 െ ݕ3ሻ/ሺݔ4 െ ݔ3ሻܾ3ൌ2ሺݕ4 െ ݕ3ሻ/ሺݔ4 െ ݔ3ሻ െ ܾ4We can write this in matrix form and solve for ܾ1, ܾ2, and ܾ3. 110011001൩ ܾ1ܾ2ܾ3൩ൌ 2ሺݕ2 െ ݕ1ሻ/ሺݔ2 െ ݔ1ሻ2ሺݕ3 െ ݕ2ሻ/ሺݔ3 െ ݔ2ሻ2ሺݕ4 െ ݕ3ሻ/ሺݔ4 െ ݔ3ሻ െ ܾ4Unfortunately, we don’t have enough information to solve for ܾ4. Recall that ܾ4= slope of curve at the end of region 3. One approach would be to let ܾ4ൌ ሺݕ4 െ ݕ3ሻ/ሺݔ4 െ ݔ3ሻ(a backwards difference approximation). This would allow you to calculate the ܾ′ݏusing back substitution, the ܿ′ݏusing equations 4, 5, and 6, and the ܽ′ݏusing equations 1, 2, and 3. You can follow exactly the same procedure for fitting higher order polynomials to the curve. The most important is a ‘cubic spine’. ݕൌ ܽ ܾݔ ܿݔଶ ݀ݔଷWhen using a cubic spline, you can specify the first and second derivatives. Unfortunately, when you do the mathematics, you will find yourself missing 2-pieces of information. One solution is to simply specify that the first and last second derivatives are both zero. The resulting curve is called a ‘natural spline’. The derivation is given in the text.