A few tidbits on python3 numpy

synopsis

A scientific computation implemented in python, including:
1, a powerful N-dimensional array object Array;
2. A more mature (broadcast) function library;
3. Toolkit for integrating C/C++ and Fortran code;
4. practical linear algebra, Fourier transforms and random number generating functions.
numpy is easier to use in conjunction with the sparse matrix arithmetic package scipy. numPy (Numeric Python) provides a number of advanced numerical programming tools, such as: matrix data types, vector processing, and sophisticated arithmetic libraries. It was created specifically for rigorous numerical processing. It is used by many large financial companies, as well as core scientific computing organizations such as Lawrence Livermore and NASA to handle tasks that would otherwise be done in C++, Fortran, or Matlab.

NumPy, formerly known as Numeric, was first developed by Jim Hugunin with other collaborators. In 2005, Travis Oliphant developed NumPy by combining the features of Numarray, another library of the same nature, with other extensions to Numeric. NumPy is open source and maintained by many collaborators. NumPy is open source and maintained by a number of collaborators.

Usage

NumPy is a powerful Python library widely used for scientific computing and data processing. Here are some of the main functions that NumPy uses for data processing, as well as notes on their use:

Main Functions

1. Array Creation

   • (): Create an array from a list or tuple.
   • (): Create an all-zero array.
   • (): Create all-one arrays.
   • (): Create equally spaced arrays.
   • (): Creates a specified number of equally spaced arrays.

2. array operation

   • (): Change the shape of the array.
   • (): Spreads a multidimensional array into a one-dimensional array.
   • (): Transpose arrays.
   • (): Connect multiple arrays.
   • (): Split the array.

3. array operation

   • (), (), (), ():: Basic arithmetic operations.
   • (): Matrix multiplication.
   • (): Computes the sum of an array.
   • (): Calculate the mean of an array.
   • (): Calculate the standard deviation.
   • (), (): Calculate the minimum and maximum values.

4. Indexing and Slicing

   • utilization[] Performs array indexing.
   • utilization: Perform slicing.
   • Boolean Index: Generate Boolean arrays to filter data by condition.

5. linear algebra

   • (): Compute the inverse of the matrix.
   • (): Compute the determinant of a matrix.
   • (): Compute eigenvalues and eigenvectors.

6. Random number generation

   • (): Generate uniformly distributed random numbers.
   • (): Generate random numbers with standard normal distribution.
   • (): Generates a random integer in the specified range.

Precautions for use

1. array dimension (math.): Ensure that the dimensions and shapes of arrays are compatible when performing operations. Use thereshape() cap (a poem)expand_dims() can help to adjust the shape of the array.

2. data type: The element types of NumPy arrays are fixed, make sure you specify the appropriate data types when creating the array (e.g.dtype) to avoid accidental data type conversions.

3. memory management: NumPy arrays generally take less memory than Python lists, but you still need to be aware of memory usage when working with very large arrays. Use() Can handle arrays that exceed memory limits.

4. Broadcasting mechanisms: NumPy supports broadcasting, which allows arrays of different shapes to be manipulated. Understanding the rules of broadcasting can help you process data more efficiently.

5. Avoiding loops: Try to avoid using Python's for loops for NumPy arrays; using vectorized operations such as array arithmetic can significantly improve performance.

6. random number seed: When doing random number generation, if you need repeatable results, you can use the() Set the random number seed.

7. Using Documents: NumPy has extensive documentation and examples, and you can refer to the official documentation when you encounter problems (NumPy Documentation）。

point of knowledge (math.)

NumPy is one of the core libraries in the Python data science and machine learning space, and as such it is often a topic of conversation in interviews. Here are some high-frequency interview questions about NumPy and the corresponding answers:

What is ndarray in NumPy?

ndarray is a core object in NumPy for storing elements of homogeneous types (e.g., integers, floats, etc.). It is a multidimensional array that allows for efficient element-level operations.

How do I create a NumPy array of shape (3, 4) and fill it with 0?

solution:

import numpy as np
array = ((3, 4))

This creates an array of 3 rows and 4 columns, all with 0 elements.

How to get the shape of a NumPy array?

import numpy as np
array = ([[1, 2, 3], [4, 5, 6]])
shape = 

shape property returns a tuple representing the shape of the array.

How to change the shape of a NumPy array without changing its data?

import numpy as np
array = ([[1, 2, 3], [4, 5, 6]])
reshaped_array = (3, 2)

This will change the original array into 3 rows and 2 columns.

How to convert Python lists to NumPy arrays?

Use the () function to convert a Python list into a NumPy array.

How do I calculate the mean, standard deviation, and variance of a NumPy array?

Use the (), () and () functions respectively. For example, the mean value is calculated as follows:

import numpy as np
array = ([1, 2, 3, 4, 5])
mean_value = (array)

() function calculates the mean of an array.

How to do element level operations in NumPy arrays?

NumPy supports element-level operations, which means you can apply arithmetic operations or other functions to each element of an array. For example:

import numpy as np
array1 = ([1, 2, 3])
array2 = ([4, 5, 6])
added_array = array1 + array2

This will return a new array[5, 7, 9]。

How to generate random numbers using NumPy?

import numpy as np
random_array = (3, 4)

() The function generates an array of a given shape, whose elements are random numbers uniformly distributed from the interval [0, 1).

How can I check if a NumPy array contains any NaN values?

import numpy as np
array = ([1, 2, , 4])
contains_nan = (array)

() The function can return a boolean array indicating which positions are NaN.

How to do conditional filtering in NumPy arrays?

import numpy as np
array = ([1, 2, 3, 4, 5])
filtered_array = array[array > 2]

This will return a new array[3, 4, 5]which contains all elements greater than 2.

Explaining NumPy'sdtype。

In NumPy, thedtype is a very important concept that represents a Data Type. Every NumPy array has an associateddtype, which specifies the data type of each element in the array. This helps NumPy store and process data efficiently in memory.

dtype key points

1. homogeneity: NumPy arrays are homogeneous, which means that all elements in the array must be of the same data type.dtype Made sure of it.

2. Memory Efficiency: By specifying thedtype, you can control how arrays are stored in memory, thus improving the efficiency of memory usage.

3. Operational optimization: Different data types may affect the performance of array operations. For example, integers and floating point numbers may operate at different speeds.

4. type conversion: If the array is created without specifying thedtypeNumPy automatically infers the type of an array element based on the type of the array element.dtype. However, if desired, it is possible to explicitly specify thedtype。

5. type safety: Ensuring that all elements have the same data type when performing array operations avoids type mismatch errors.

Common NumPy data types:

• np.int32: 32-bit integer
• np.int64: 64-bit integer
• np.float32: 32-bit floating point number
• np.float64: 64-bit floating point number (double precision)
• np.bool_: Boolean (True or False)
• np.complex64: complex number with 32 bits in the real and 32 bits in the imaginary part
• np.complex128: complex number with 64 bits in real and 64 bits in imaginary part
• : Python Objects
• np.string_: String type
• np.datetime64: Date Time Type

typical example

Creates a file with a specificdtype of the NumPy array:

import numpy as np

# Create an array of integer type
int_array = ([1, 2, 3], dtype=np.int32)
print(int_array.dtype) # Output: int32

# Create an array of type float
float_array = ([1.0, 2.0, 3.0], dtype=np.float64)
print(float_array.dtype) # Output: float64

# Create an array of type boolean
bool_array = ([True, False, True], dtype=bool)
print(bool_array.dtype) # Output: bool

caveat

• When performing array operations that involve differentdtype array, NumPy usually performs type casting to ensure that the resulting array has a data type that can accommodate all possible values.
• explicitly specifydtype can help avoid unnecessary type conversions, thus improving code performance and readability.
• When working with large datasets, it makes sense to choosedtype It can significantly reduce memory usage and increase processing speed.

dtype is an important property of NumPy arrays, and understanding and properly using thedtype essential for performing efficient numerical computations.

Why is NumPy faster than Python's native lists?

1. data storage：

   • NumPy arrays store data in memory in contiguous blocks, which means that the elements in the array are closely spaced. This contiguous storage allows the CPU cache to work more efficiently because when an element of the array is accessed, the neighboring elements are also loaded into the cache.
   • Python's native lists store references to objects that could be scattered anywhere in memory, which results in more memory access latency.

2. data type：

   • Elements in NumPy arrays are all homogeneous, meaning they have the same data type, which allows NumPy to optimize memory usage and computational operations.
   • Python lists can contain elements of different types, which adds complexity to memory usage.

3. Operational optimization：

   • NumPy is written in C. Its array operations are implemented in a low-level language, which makes them very fast and efficient.
   • Python list manipulation is implemented in Python, a high-level language, which usually involves more function calls and interpreter overhead.

4. vectorization (computing)：

   • NumPy supports vectorized operations, which means that operations can be performed on multiple elements of an array at once without using loops. These operations are written in C and can be compiled to machine code for high performance.
   • Python lists typically require the use of loops to iterate over elements, which adds extra overhead.

5. Broadcasting mechanisms：

   • NumPy's broadcast mechanism allows arrays of different shapes to work together in arithmetic operations without explicitly looping at the element level.

6. algorithmic implementation：

   • NumPy's algorithmic implementations are generally more optimized because they are specifically designed for numerical computation.

7. parallel processing：

   • For some operations, NumPy can utilize parallel processing to further improve performance, especially on multi-core processors.

8. memory management：

   • NumPy explicitly specifies the data type and size when creating arrays, which helps reduce the overhead of memory allocation and reclamation.

9. caching efficiency：

   • Due to the contiguous memory allocation of NumPy arrays, the caching mechanisms of modern CPUs are able to work more efficiently because the data access patterns are more localized.

10. Avoiding Python Interpreter Overhead：

    • Python list operations require the intervention of the Python interpreter, whereas many NumPy operations are performed directly at the bottom, avoiding the overhead of the interpreter.

How can I optimize the performance of my NumPy code?

Answer: Use vectorized operations instead of loops, avoid copying data unnecessarily, use appropriate data types, and parallelize processing (e.g., use alternative for loops to compute dot products).

Explain the broadcast mechanism in NumPy.

The Broadcasting mechanism in NumPy is a powerful feature that allows arrays of different shapes to work together in mathematical operations without explicitly matching their shapes. The broadcasting mechanism follows these rules:

1. dimensional alignment: Compare the dimensions of two arrays from left to right to make sure they are aligned. This means that the shorter array is padded with 1's in front of it (e.g., the(3,) constitute(1, 3)）。

2. Dimensional extensions: If two arrays do not have the same size in a dimension, the shape of the smaller array is expanded in that dimension to match the larger array. This is done by copying the dimension values of the smaller array.

3. Shape Comparison: Compare the shapes of two arrays dimension by dimension, starting with the trailing dimension (the rightmost dimension). If the two dimensions are equal, or if one of them is 1, they are considered compatible.

4. Replication extensions: If an array has a dimension size of 1 and another array has a dimension size greater than 1, then the array with dimension size 1 is copied and expanded to the same dimension size as the other array.

5. Broadcast results: If two arrays are compatible in all dimensions, then they can be broadcast to form a new array shape for computation.

Example:

import numpy as np

# Create two arrays
a = ([1, 2, 3]) # Shape (3,)
b = ([[1], [2], [3]]) # shape (3, 1)

# Broadcast summing
c = a + b # The result is an array of shape (3, 3)
print(c)
# Output:
# [[2 2 2]
# [3 3 3]
# [4 4 4]]

In this example, thea The shape of the(3,)，b The shape of the(3, 1). According to the broadcasting rules.a was extended to(3, 3)，b It has also been extended to(3, 3), and then perform element-by-element summation.

The broadcast mechanism makes NumPy very efficient when performing element-level operations, as it avoids unnecessary array copying and looping. However, it also has potential drawbacks, such as the fact that it can sometimes lead to unexpected results, especially when the array shape is complex or the operation is unclear. Therefore, understanding the broadcast mechanism is crucial to writing clear, efficient NumPy code.

How to use NumPy for feature scaling in machine learning?

In machine learning, feature scaling is an important preprocessing step that helps to improve model performance and convergence speed. Feature scaling consists of several techniques, the most common of which are Min-Max Scaling and Standardization. Here's how to do both types of feature scaling using NumPy:

Min-Max Normalization (Min-Max Scaling)

Min-max normalization scales features to a specified range, usually [0, 1]. This method is useful for maintaining the proportion of features in the data.

import numpy as np

# Suppose X is an array of data of shape (n_samples, n_features)
X = ([[1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6])
              [4, 5, 6], [7, 8, 9])
              [7, 8, 9]])

# Calculate the minimum and maximum values for each feature
X_min = (axis=0)
X_max = (axis=0)

# Perform min-max normalization
X_scaled = (X - X_min) / (X_max - X_min)
print(X_scaled)

Standardization

Normalization (also known as Z-score normalization) scales the features so that they have a mean of 0 and a standard deviation of 1. This helps to ensure that the scaling of the different features does not affect the optimization process of the model.

# Suppose X is a data array of shape (n_samples, n_features)
X = ([[1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6])
              [4, 5, 6], [7, 8, 9])
              [7, 8, 9]])

# Calculate the mean and standard deviation of each feature
X_mean = (axis=0)
X_std = (axis=0)

# Perform normalization
X_standardized = (X - X_mean) / X_std
print(X_standardized)

caveat

1. Avoiding data leakage: When calculating the parameters used for scaling (e.g., minimum, maximum, mean, and standard deviation) on the training set, one should make sure not to use data from the test or validation sets, which is known as data leakage.

2. Save scaling parameters: After training the model on the training set, the parameters used for feature scaling (minimum, maximum, mean, and standard deviation) should be saved so that the same scaling can be performed on new data in a test set or production environment.

3. Selecting a scaling method: Different models may have different sensitivities to feature scaling. For example, distance-based models (e.g., K-Nearest Neighbors and SVMs) typically benefit from scaling, while tree-based models (e.g., decision trees and random forests) typically do not require feature scaling.

4. Handling of missing values: Missing values in the data should be dealt with before feature scaling, as they may affect the calculation of the mean and standard deviation.

Feature scaling with NumPy is straightforward and efficient, but note that NumPy does not provide built-in functions to automatically apply these scaling techniques. In practice, thescikit-learn The library provides more advanced feature scaling methods such asMinMaxScaler cap (a poem)StandardScaler, they can more easily deal with these issues.

How to do Principal Component Analysis (PCA) with NumPy?

Step 1: Prepare the data

First of all, you need a file of the shape(n_samples, n_features) The data array of the

import numpy as np

# Example data
X = ([[1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6])
              [4, 5, 6], [7, 8, 9]])
              [7, 8, 9]])

Step 2: Data standardization

PCA is very sensitive to the scale of the data, so it is often necessary to standardize the data first.

X_centered = X - (X, axis=0)
X_std = (X_centered, axis=0)
X_normalized = X_centered / X_std

Step 3: Calculate the covariance matrix

The covariance matrix is used to find the principal components of the data.

cov_matrix = (X_normalized.T)

Step 4: Calculate eigenvalues and eigenvectors

The eigenvalues and eigenvectors indicate the direction of the principal components of the data.

eigenvalues, eigenvectors = (cov_matrix)

Step 5: Select Principal Components

The eigenvectors corresponding to the largest few eigenvalues are selected as principal components.

# Sort eigenvectors in descending order of eigenvalue size
sorted_index = (eigenvalues)[::-1]
principal_components = eigenvectors[:, sorted_index[:n_components]]

included among thesen_components is the number of ingredients you want to keep.

Step 6: Converting Data

Project the raw data onto the selected principal components.

X_pca = (X_normalized, principal_components)

utilizationX_pca You can get the data after dimensionality reduction.

NumPy PCA Sample Code

import numpy as np

# Example data
X = ([[1, 2, 3], [4, 5, 6], [4, 5, 6], [4, 5, 6])
              [4, 5, 6], [7, 8, 9]])
              [7, 8, 9]])

# Standardized data
X_centered = X - (X, axis=0)
X_std = (X_centered, axis=0)
X_normalized = X_centered / X_std

# Calculate the covariance matrix
cov_matrix = (X_normalized.)

# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = (cov_matrix)

# Sort eigenvectors by eigenvalue size in descending order
sorted_index = (eigenvalues)[::-1]
n_components = 2 # select the first two principal components
principal_components = eigenvectors[:, sorted_index[:n_components]]

# Convert the data
X_pca = (X_normalized, principal_components)

print(X_pca)

caveat

• Data standardization is an important step in PCA, ensuring that each feature has unit variance.
• In practice, it is common to usescikit-learn of the PCA implementation because it is more efficient, more convenient, and includes more features such as automatic selection of the number of components.
• NumPy's PCA implementation does not take into account Singular Value Decomposition (SVD), which may be more effective when dealing with data with more features.

utilizationscikit-learn The PCA implementation is very simple:

from import PCA

# Example data
X = ([[1, 2, 3], [4, 5, 6], [1, 2, 3], [1, 2, 3])
              [4, 5, 6], [7, 8, 9]])
              [7, 8, 9]])

# Initialize PCA, n_components is the number of components to be retained
pca = PCA(n_components=2)

# Fit and transform the data
X_pca = pca.fit_transform(X)

print(X_pca)

This approach is more concise andscikit-learn Data normalization and singular value decomposition (SVD) are handled automatically.

If there is any mistake, welcome to criticize and correct, thank you!