Polynomial regression using gradient descent

Polynomial regression using gradient descent

experimental purpose

The purpose of this experiment is to implement polynomial regression by gradient descent method, to explore the effect of fitting polynomial models of different orders to the same set of data, and to analyze the effect of the number of samples on the model fitting results.

Experimental Materials and Methods

Data preparation

1. Generate training samples: We first generated 20 training samples where the independent variable$X$obeys a standard normal distribution with mean 0 and variance 1. Dependent variable$Y$From the following polynomial relationship plus an error term with mean 0 and variance 1e_r​Composition:Y=5+4X+3X²+2X³+e_r​
2. data visualization: The generated data points were plotted using the Matplotlib library.

coding

import numpy as np
import as plt

# Set random seeds to ensure experimental reproducibility
(0)

# Generate 20 training samples
n_samples = 20
X = (0, 1, n_samples)
e_r = (0, 1, n_samples) # Error term

# Calculate the value of Y
Y = 5 + 4 * X + 3 * X**2 + 2 * X**3 + e_r

# Display the generated data using matplotlib
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
('Generated Data')
('X')
('Y')
()
(True)
()

Model Definition

1. Define the polynomial regression model: We define aMultinomialModelclass, which accepts training data as input and is able to return the parameters of a polynomial model. Included within the class are methods for constructing the design matrix, methods for fitting the data (using gradient descent), and prediction methods.

coding

class MultinomialModel:
    def __init__(self, degree):
         = degree
         = None
    
    def _design_matrix(self, X):
        """Constructing a design matrix"""
        n_samples = len(X)
        design_matrix = ((n_samples, + 1))
        for i in range(1, + 1):
            design_matrix[:, i] = X ** i
        return design_matrix
    
    def fit(self, X, Y, learning_rate=0.01, iterations=1000):
        """Using gradient descent to fit the model"""
        n_samples = len(X)
         = ( + 1) # Initialization factor
        
        # Constructing a design matrix
        X_design = self._design_matrix(X)
        
        for _ in range(iterations):
            # anticipate
            predictions = (X_design, )
            
            # The derivative of the loss function
            gradient = 2 / n_samples * (X_design.T, predictions - Y)
            
            # Updating factor
             -= learning_rate * gradient
    
    def predict(self, X):
        """基于学习到的模型anticipate新的数据点"""
        X_design = self._design_matrix(X)
        return (X_design, )

# Classes that use the above definitions
degree = 3 # Setting the order of a polynomial
model = MultinomialModel(degree)

# Fitting the data
(X, Y)

# anticipate
Y_pred = (X)

# Visualization of fitting results
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
(X, Y_pred, color='red', label='Fitted curve')
('Polynomial Regression Fit')
('X')
('Y')
()
(True)
()

Model fitting and presentation of results

1. Model Training and Prediction: For a set polynomial model of different orders, a gradient descent method is used to train and predict the data.
2. Visualization of results: In the same graph, fitted curves for polynomial models of different orders are plotted, while retaining the scatterplot of the original data points.

coding

# Continue to use the previously defined MultinomialModel class

# Use the class defined above
degree = 3 # Set the degree of the polynomial
model = MultinomialModel(degree)

# Fit the data
(X, Y)

# Prediction
Y_pred = (X)

# Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
X_fit = ((X), (X), 100)
Y_fit = (X_fit)

# Visualize the fit
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
(X_fit, Y_fit, color='red', label='Fitted curve', linewidth=2)
(f'Polynomial Regression Fit (Degree {degree})')
('X')
('Y')
()
(True)
()


# Define different polynomial orders
degrees = [1, 2, 3, 4, 5]

# Create a new graph
(figsize=(10, 8))

# For each polynomial degree, fit and plot a curve
for degree in degrees.
    model = MultinomialModel(degree)
    (X, Y)

    # Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
    X_fit = ((X), (X), 100)
    Y_fit = (X_fit)

    (X_fit, Y_fit, label=f'Degree {degree}')

# Plot the actual data points
(X, Y, color='blue', label='Actual data')

# Set the legend and other details
('Polynomial Fits of Different Degrees')
('X')
('Y')
()
(True)
()

Sample size impact analysis

1. Increase in sample size: Increase the sample size from 20 to 100 and repeat the above steps to observe the change in the model fit.

coding

# Generate 100 training samples
n_samples = 100
X = (0, 1, n_samples)
e_r = (0, 1, n_samples) # Error term

# Calculate the value of Y
Y = 5 + 4 * X + 3 * X**2 + 2 * X**3 + e_r

# Display the generated data using matplotlib
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
('Generated Data with 100 samples')
('X')
('Y')
()
(True)
()

# Define different polynomial orders
degrees = [1, 2, 3, 4, 5]

# Create a new graph
(figsize=(10, 8))

# For each polynomial degree, fit and plot a curve
for degree in degrees.
    model = MultinomialModel(degree)
    (X, Y)

    # Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
    X_fit = ((X), (X), 100)
    Y_fit = (X_fit)

    (X_fit, Y_fit, label=f'Degree {degree}')

# Plot the actual data points
(X, Y, color='blue', label='Actual data')

# Set the legend and other details
('Polynomial Fits of Different Degrees with 100 samples')
('X')
('Y')
()
(True)
()

Experimental Results and Discussion

Results Showcase

• In the initial phase, we observed the fit of the polynomial model of different orders to the 20 sample data. As the order of the polynomial increases, the model gradually shifts from a state of underfitting to a state of possible overfitting, especially at higher orders, where the model tries to follow the trend of the data points more closely.
• The performance of the model became more stable as the sample size increased to 100. The higher order polynomial model, while still exhibiting some complexity, has a reduced risk of overfitting due to more data support. The model is better able to capture the true trend of the data.

talk over

• Model Complexity and Fitting Effectiveness: As the polynomial order increases, the complexity of the model increases, which allows the model to better approximate the training data. However, too high an order can also lead to overfitting, i.e., the model performs well on the training data but poorly on the unknown data.
• Effect of sample size: Increasing the number of samples helps to improve the generalization ability of the model. More samples mean that the model can learn more diverse features, thus reducing the risk of overfitting.

reach a verdict

This experiment shows how to implement polynomial regression using gradient descent method and explores the effects of different orders and sample sizes on the model fitting results. The experimental results show that the polynomial regression model can effectively fit nonlinear data when choosing the appropriate polynomial order and ensuring that there are enough training samples.

Appendix: Complete Code

import numpy as np
import as plt

# Set random seeds to ensure experimental reproducibility
(0)

# Generate 20 training samples
n_samples = 20
X = (0, 1, n_samples)
e_r = (0, 1, n_samples) # Error term

# Calculate the value of Y
Y = 5 + 4 * X + 3 * X**2 + 2 * X**3 + e_r

# Display the generated data using matplotlib
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
('Generated Data')
('X')
('Y')
()
(True)
()

class MultinomialModel.
    def __init__(self, degree).
         = degree
         = None

    def _design_matrix(self, X).
        """Construct design matrix.""""
        n_samples = len(X)
        design_matrix = ((n_samples, + 1))
        for i in range(1, + 1).
            design_matrix[:, i] = X ** i
        return design_matrix

    def fit(self, X, Y, learning_rate=0.01, iterations=1000):
        """Use gradient descent to fit the model""""
        n_samples = len(X)
         = ( + 1) # initialize coefficients

        # Construct the design matrix
        X_design = self._design_matrix(X)

        for _ in range(iterations).
            # Predictions
            predictions = (X_design, )

            # Derivative of the loss function
            gradient = 2 / n_samples * (X_design.T, predictions - Y)

            # Update coefficients
             -= learning_rate * gradient

    def predict(self, X).
        """Predict new data points based on learned model""""
        X_design = self._design_matrix(X)
        return (X_design, )

# Use the class defined above
degree = 3 # set the degree of the polynomial
model = MultinomialModel(degree)

# Fit the data
(X, Y)

# Prediction
Y_pred = (X)

# Visualize the fitting result
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
(X, Y_pred, color='red', label='Fitted curve')
('Polynomial Regression Fit')
('X')
('Y')
()
(True)
()

# Continue to use the previously defined MultinomialModel class

# Use the class defined above
degree = 3 # Set the degree of the polynomial
model = MultinomialModel(degree)

# Fit the data
(X, Y)

# Prediction
Y_pred = (X)

# Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
X_fit = ((X), (X), 100)
Y_fit = (X_fit)

# Visualize the fit
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
(X_fit, Y_fit, color='red', label='Fitted curve', linewidth=2)
(f'Polynomial Regression Fit (Degree {degree})')
('X')
('Y')
()
(True)
()

# Define different polynomial orders
degrees = [1, 2, 3, 4, 5]

# Create a new graph
(figsize=(10, 8))

# For each polynomial degree, fit and plot a curve
for degree in degrees.
    model = MultinomialModel(degree)
    (X, Y)

    # Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
    X_fit = ((X), (X), 100)
    Y_fit = (X_fit)

    (X_fit, Y_fit, label=f'Degree {degree}')

# Plot the actual data points
(X, Y, color='blue', label='Actual data')

# Set the legend and other details
('Polynomial Fits of Different Degrees')
('X')
('Y')
()
(True)
()

# Generate 100 training samples
n_samples = 100
X = (0, 1, n_samples)
e_r = (0, 1, n_samples) # error term

# Calculate the value of Y
Y = 5 + 4 * X + 3 * X**2 + 2 * X**3 + e_r

# Display the generated data using matplotlib
(figsize=(8, 6))
(X, Y, color='blue', label='Actual data')
('Generated Data with 100 samples')
('X')
('Y')
()
(True)
()

# Define different polynomial orders
degrees = [1, 2, 3, 4, 5]

# Create a new graph
(figsize=(10, 8))

# For each polynomial degree, fit and plot a curve
for degree in degrees.
    model = MultinomialModel(degree)
    (X, Y)

    # Create a linear space from the minimum to the maximum value of X for plotting a smooth fitting curve
    X_fit = ((X), (X), 100)
    Y_fit = (X_fit)

    (X_fit, Y_fit, label=f'Degree {degree}')

# Plot the actual data points
(X, Y, color='blue', label='Actual data')

# Set the legend and other details
('Polynomial Fits of Different Degrees with 100 samples')
('X')
('Y')
()
(True)
()

The code used in the experiment consists of the following main sections:

1. Data generation: Use the numpy library to generate training samples that obey a specific distribution.
2. Model Definition and Realization: DefinitionsMultinomialModelclass and implement the gradient descent method for training models.
3. Visualization of results: Plot data points and fit curves using the matplotlib library.
4. Analyzing the impact of sample size: Increase the sample size and observe the change in the fit.