liwen01 2024.07.21
preamble
The UBI (Unsorted Block Images) file system is a file system management layer for bare flash. It is designed to manage raw flash devices and is particularly suitable for embedded systems. It is similar to the YAFFS2 andJFFS2 The difference is that it provides wear-leveling across the entire flash space and has good scalability for high-capacitynand flash 。
(i) MTD, UBI and UBIFS
JFFS2 and YAFFS2 are running on MTD, while UBIFS can only run on UBI, and UBI can only run on MTD, so there are three subsystems involved here: MTD, UBI, and UBIFS.
- MTD Provides abstraction and basic management of the underlying flash hardware.
- UBI Adds an additional layer of management on top of MTD to handle the complexity of flash memory and provide logical volume management.
- UBIFS It is a file system that runs on UBI volumes, taking full advantage of the features of UBI to provide efficient and reliable file storage.
Special attention needs to be paid here.By flash memory, we mean bare flash, not USB flash drives, SD, TF, SSD, etc. that have been converted by FTL.。
In Linux, FTL-converted USB flash drives, SD, TF, SSD, etc., are block devices, which are data structures designed to emulate traditional disks and are read/written in sectors.
MTD, on the other hand, is neither a character device nor a block device; it is simply the MTD device
(1) MTD (Memory Technology Device)
MTD YesLinux kernelA subsystem in a subsystem to support different types of flash memory devices such asNOR Flash and NAND Flash. MTD provides an abstraction layer that allows file system and user space programs to easily access the underlying flash hardware.
- MTD Devices: On Linux systems, MTD devices usually appear as /dev/mtdX and /dev/mtdblockX, where X is the device number.
- MTD sub-equipment: An MTD device can be divided into multiple sub-devices, each of which can be used independently.
(2)UBI (Unsorted Block Images)
UBI is a management layer on top of MTD devices, designed specifically for NAND Flash.UBI handles some of the complexities inherent in NAND Flash, such as bad block management and wear leveling.UBI divides the flash memory into logical erase blocks and manages them.
- bad block management: UBI is able to detect and manage bad blocks, ensuring that data is written without using bad blocks.
- wear leveling: UBI extends the life of flash memory by evenly distributing erase and write operations.
- logic paper: UBI supports the creation of multiple logical volumes on the MTD device, each of which can be used independently.
(3)UBIFS (UBI File System)
UBIFS is a file system designed specifically for UBI that runs directly on UBI volumes.UBIFS takes full advantage of UBI's capabilities to provide an efficient and reliable file storage solution.
- dynamic property: UBIFS supports dynamic file system resizing, allocating and reclaiming space as needed.
- log structure: UBIFS uses a log-structured file system to reduce the risk of data corruption and improve write performance.
- compressed: UBIFS supports multiple compression algorithms to save storage space.
UBIFS is not the only filesystem that can run on UBI; theoretically the vast majority of filesystems can run on UBI. Except for UBIFS, all other filesystems are not efficient to use on UBI.
(ii) Mirror file production
(1) UBIFS Image File Creation
(a) Preparation of test documents
Create four new test directories, create test quotes in the directories, and write random numbers to the files using /dev/urandom
biao@ubuntu:~/test/ubifs/ubifs_urandom$ tree
.
├── test1
│ ├── file1
│ ├── file1_1
│ └── file1_2
├── test2
│ ├── file2
│ ├── file2_1
│ └── file2_2
├── test3
│ ├── file3
│ ├── file3_1
│ └── file3_2
└── test4
├── file4
├── file4_1
└── file4_2
4 directories, 12 files
biao@ubuntu:~/test/ubifs/ubifs_urandom$The file size information is as follows:
biao@ubuntu:~/test/ubifs$ du ubifs_urandom
1904 ubifs_urandom/test3
168 ubifs_urandom/test2
1504 ubifs_urandom/test1
1476 ubifs_urandom/test4
5056 ubifs_urandom
biao@ubuntu:~/test/ubifs$(b) Production of UBIFS image files
-r ubifs_urandom -m 2048 -e 129024 -c 10000 -o ubifs_urandom.ubifsThe role of each parameter:
-r, -d, --root=DIR build file system from directory DIR
-m, --min-io-size=SIZE minimum I/O unit size
-e, --leb-size=SIZE logical erase block size
-c, --max-leb-cnt=COUNT maximum logical erase block count
-o, --output=FILE output to FILEThe function of the above command is to pack the files in the ubifs_urandom directory into a UBIFS image file (ubifs_urandom.ubifs) with a page size of 2048 (2KB), a logical erase block size of 129024 (126KB), and a maximum logical block size of 10000.
Here are a few things to keep in mind:
- -m is the page size, not the subpage size
- The logical erase block size set by -e should be the same as the one in UBI, otherwise it will report an error when mounted.
(2) UBI image file creation
Make a UBI image file based on the UBIFS
(a) Make a UBI image profile
Creating Configuration Files
[ubifs]
mode=ubi
image=ubifs_urandom.ubifs
vol_id=0
vol_size=256MiB
vol_type=dynamic
vol_name=ubifs_urandom
vol_flags=autoresize- image is the image of the UBIFS file system we created above.
- vol_id specifies the volume ID, this is used when there are multiple volumes
- vol_size Defines the size of the volume
- vol_type is set to a dynamic volume, where the size of the volume can change
- vol_name Volume name
- vol_flags Automatic resizing
(b) Create a UBI image file
ubinize -o -m 2048 -O 512 -p 128KiB Both the UBI image file and the UBIFS image file need to be set according to the actual Flash parameters.
The function of the above command is to make an ubifs image file into a UBI image file with a page size of 2048 (2KB), a sub-page size of (256Byte), and a physical erase block size of 128KB.
(iii) Mounting the UBIFS file system
For debugging purposes, we use the virtual MTD device on the PC to emulate Flash. 3 MTD device emulators are available in the Linux kernel:
- mtdram: emulates NOR flash memory in RAM;
- nandsim: emulates NAND flash memory in RAM;
- block2mtd: emulates NOR flash on a block device.
(1) Load nandsim module
Here is an example of a 1GiB, 2048 bytes page nand flash.
sudo modprobe nandsim first_id_byte=0xec second_id_byte=0xd3 third_id_byte=0x51 fourth_id_byte=0x95Emulated nandflsh information can be viewed via /proc/mtd and /dev/mtd0
biao@ubuntu:~/test/ubifs$ cat /proc/mtd
dev: size erasesize name
mtd0: 40000000 00020000 "NAND simulator partition 0"
biao@ubuntu:~/test/ubifs$ mtdinfo /dev/mtd0
mtd0
Name: NAND simulator partition 0
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 8192 (1073741824 bytes, 1024.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 512 bytes
OOB size: 64 bytes
Character device major/minor: 90:0
Bad blocks are allowed: true
Device is writable: true
biao@ubuntu:~/test/ubifs$When loading nandsim above, there are 4 ID values defined, the specific values need to be set according to the data in the chip manual.
The first byte is the manufacturer's code, the second byte is the device code, and the third and fourth bytes are Flash-specific parameters.
Here are a few examples:
modprobe nandsim first_id_byte=0x20 second_id_byte=0x33 - 16MiB, 512 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0x35 - 32MiB, 512 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0x36 - 64MiB, 512 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0x78 - 128MiB, 512 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0x71 - 256MiB, 512 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0xa2 third_id_byte=0x00 fourth_id_byte=0x15 - 64MiB, 2048 bytes page;
modprobe nandsim first_id_byte=0xec second_id_byte=0xa1 third_id_byte=0x00 fourth_id_byte=0x15 - 128MiB, 2048 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15 - 256MiB, 2048 bytes page;
modprobe nandsim first_id_byte=0x20 second_id_byte=0xac third_id_byte=0x00 fourth_id_byte=0x15 - 512MiB, 2048 bytes page;
modprobe nandsim first_id_byte=0xec second_id_byte=0xd3 third_id_byte=0x51 fourth_id_byte=0x95 - 1GiB, 2048 bytes page;(2) Mounting the UBIFS file system
(a) Load the UBI kernel module
sudo modprobe ubi mtd=0Here the ubi is loaded on device 0 of the mtd.
(b) Separation of equipment on MTD 0
sudo ubidetach /dev/ubi_ctrl -m 0(c) Format MTD device and write UBI image file
sudo sudo ubiformat /dev/mtd0 -s 512 -f (d) UBI device attached back to MTD device 0
sudo ubiattach /dev/ubi_ctrl -m 0 -O 512(e) Mount UBIFS to the specified directory.
sudo mount -t ubifs ubi0 /home/biao/test/ubifs/ubifs_simulator(f) Viewing the status of a mount
biao@ubuntu:~/test/ubifs$ df -h
Filesystem Size Used Avail Use% Mounted on
......
ubi0 927M 4.8M 923M 1% /home/biao/test/ubifs/ubifs_simulator
......
biao@ubuntu:~/test/ubifs$
biao@ubuntu:~/test/ubifs/ubifs_simulator$ ls
test1 test2 test3 test4You can see that the UBIFS image has been mounted on the ubi0 volume, and the files on the mounted directory are our test files.
(iv) UBIFS mirror file analysis
Above we have created two image files UBIFS and UBI, and then loaded the UBI image file onto the NAND Flash emulator nandsim on the PC. To understand how UBIFS works, it is necessary to analyze its data structure on Flash.
(1) UBIFS data structure
You can see all the data structure definitions in UBIFS, the following one is a generalized data structure, with theimaginary number, crc checksum, sequence number, length, node type, node group type of this information, of which there are 11 types of valid nodes.
The generic header structure is defined as follows
/**
* struct ubifs_ch - common header node.
* @magic: UBIFS node magic number (%UBIFS_NODE_MAGIC)
* @crc: CRC-32 checksum of the node header
* @sqnum: sequence number
* @len: full node length
* @node_type: node type
* @group_type: node group type
* @padding: reserved for future, zeroes
*
* Every UBIFS node starts with this common part. If the node has a key, the
* key always goes next.
*/
struct ubifs_ch {
__le32 magic;
__le32 crc;
__le64 sqnum;
__le32 len;
__u8 node_type;
__u8 group_type;
__u8 padding[2];
} __packed;Node Type Definition:
enum {
UBIFS_INO_NODE,
UBIFS_DATA_NODE,
UBIFS_DENT_NODE,
UBIFS_XENT_NODE,
UBIFS_TRUN_NODE,
UBIFS_PAD_NODE,
UBIFS_SB_NODE,
UBIFS_MST_NODE,
UBIFS_REF_NODE,
UBIFS_IDX_NODE,
UBIFS_CS_NODE,
UBIFS_ORPH_NODE,
UBIFS_NODE_TYPES_CNT,
};(2) UBIFS Node Layout
The UBIFS image file we created earlier contains five types of nodes, which are laid out in the image file as shown below.
The first is the SuperFast, the next are the two Masters, and the last is the Index Node, and their functions are as follows:
Superblock Node: Stores basic information about the file system, such as size, status, version, etc.
Master Node: Saves the current state of the file system, including pointers to the log header and root index node.
Commit Start Node: Marks the start of a commit operation. It is used to determine what data is committed in the event of a file system crash.
Data Node: Stores the data of a file. Each data node corresponds to a specific part of a file.
Index Node: The index structure used to build UBIFS is similar to an index node (inode) in a traditional file system.
(3) UBIFS node working principle
Mounting a file system：
- Read the superblock and master node to recover the basic state of the file system and pointers to important metadata.
- Initialize other necessary data structures and caches.
file operation：
- Create File: Adds a new index node to the index tree and assigns the corresponding data node to store the file contents.
- Read and write files: find the corresponding data node through the index node, and then perform read and write operations.
- Modify File: The modified data is written to the new data node and the corresponding index node is updated.
submit (a report etc)：
- Write commit start node to mark the start of the commit.
- Writes all modified data nodes and index nodes to flash memory.
- Update the master node to reflect the latest file system state.
Crash Recovery:
- Check the commit start node to determine which commits have completed.
- Restore the latest consistent state of the file system via the master node.
(4) UBIFS node analysis
Here we only analyze the superblock node, and the other nodes are similar
Use hexdum to view a ubifs image file, starting at superblock
00000000 31 18 10 06 e6 e4 54 b6 d9 05 00 00 00 00 00 00 |1.....T.........|
00000010 00 10 00 00 06 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 08 00 00 00 00 02 00 35 00 00 00 64 00 00 00 |........5...d...|
00000030 00 00 16 00 00 00 00 00 04 00 00 00 02 00 00 00 |................|
00000040 01 00 00 00 01 00 00 00 08 00 00 00 00 01 00 00 |................|
00000050 04 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 00 00 00 00 00 ca 9a 3b 1c fe ed 83 |...........;....|
00000070 7a ef 48 f7 83 2c 10 74 b9 36 09 9b 00 00 00 00 |..,.t.6......|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
.........Parsing the above data image reveals the following information:
(v) UBI image file analysis
(1) UBI Data Layout
The main UBI data structures are ubi_ec_hdr and ubi_vid_hdr:
ubi_ec_hdr: (Erase Counter Header) Contains the Erase Counter information header structure, the main function is to record and manage the number of times a physical erase block has been erased.
ubi_vid_hdr: (Volume Identifier Header) Contains the volume identifier information header structure, the main role is to manage and identify the data in the physical volume block.
A UBI volume is divided into multiple blocks, each of which has these two headers. ubi_ec_hdr records the number of times each block has been erased, helping to manage the lifetime and reliability of the block. And ubi_vid_hdr ensures the correct location and data integrity of each block in the volume.
Their data layout in the image file or flash is as follows:
(2) Data analysis
Look at the first 4KB of data in the image file, with special attention paid here to theThe UBI data is stored in a big-end model, which is different from the previously analyzed image files
biao@ubuntu:~/test/ubifs$ hexdump -s 0 -n 4096 -C
00000000 55 42 49 23 01 00 00 00 00 00 00 00 00 00 00 00 |UBI#............|
00000010 00 00 02 00 00 00 08 00 20 d2 d3 a0 00 00 00 00 |........ .......|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 92 3a 9d cd |.............:..|
00000040 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00000200 55 42 49 21 01 01 00 05 7f ff ef ff 00 00 00 00 |UBI!............|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000230 00 00 00 00 00 00 00 00 00 00 00 00 b8 25 64 a8 |.............%d.|
00000240 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00000800 00 00 08 21 00 00 00 01 00 00 00 00 01 00 00 0d |...!............|
00000810 75 62 69 66 73 5f 75 72 61 6e 64 6f 6d 00 00 00 |ubifs_urandom...|
00000820 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000890 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000008a0 00 00 00 00 00 00 00 00 8c 7e c0 aa 00 00 00 00 |.........~......|
000008b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
......The data for ubi_ec_hdr is parsed as follows
The data for ubi_vid_hdr is parsed as follows
(3) How UBI works
How UBI works can be seen in the following official demo video.
The principle of bad fast management implementation can be viewed in the video below:
(vi) Advantages and disadvantages
(1) Advantages
(a) Durability and reliability：
wear leveling: UBI is able to effectively manage the number of flash memory erasures to ensure that all erase blocks are used evenly, thus extending the life of the flash memory.
bad block management: UBI is able to detect and deal with bad blocks, ensuring that data is not written to these corrupted areas and improving the reliability of the file system.
crash recovery: UBI has good recovery capability after a system crash or power down to minimize data loss.
(b) Dynamic size management：
Dynamic Partition Adjustment: UBI allows dynamic resizing of partitions, which is useful for applications with highly variable storage requirements.
Flexible space management: UBI enables flexible management of flash memory space and supports dynamic file system allocation and resizing.
(c) Support for high-capacity flash memory: UBI supports high-capacity flash devices and is suitable for use in embedded systems that require large amounts of storage space.
(2) Disadvantages
(a) Increased complexity：
Complex design: UBI is more complex to implement and requires additional layers in the kernel to manage flash memory, which increases the complexity of system design and maintenance.
Difficulty in debugging: Due to its complex mechanism, UBI is more difficult to troubleshoot and debug than traditional file systems.
(b) Consumption of resources：
memory footprint: UBI requires additional memory to maintain its data structures, which may put some pressure on embedded systems with limited memory resources.
CPU consumption: The operation of UBI requires additional CPU resources to perform tasks such as wear leveling and garbage collection, which may affect the overall performance of the system.
(c) Not suitable for all applications：
Highly specialized: UBI is primarily optimized for raw flash devices and may not be as advantageous for systems using other storage media (e.g., eMMC, SD cards, etc.).
Domain-specific applications: UBI is designed primarily for embedded systems; for desktop or server systems, other file systems (e.g., EXT4, XFS, etc.) may be more appropriate.
wind up
Overall, the UBI file system excels in embedded systems that require high reliability, durability, and flexible space management, but its complexity and resource consumption also need to be weighed in specific applications.