Location>code7788 >text

How does the front-end handle the back-end returning 100,000 pieces of data at once?

Popularity：467 ℃/2024-07-28 14:34:27

In front-end development, we often need to deal with a large amount of data returned by the back-end. Assuming that the backend returns 100,000 pieces of data at one time, processing and displaying this data directly in the browser will lead to performance problems, such as page lag, excessive memory usage, etc. In this article, we will combine Vue project practices to introduce how to effectively handle and display large data sets.

1. Back-end data processing

First, making sure that the backend is compressing the data as it is being transferred can greatly reduce the amount of data being transferred. Common types of compression are Gzip or Brotli.

// Use the compression middleware in
const compression = require('compression');
const express = require('express');
const app = express();

(compression());

 

 

2. Front-end data processing

Pagination

Pagination loading is one of the most commonly used methods to effectively reduce browser memory pressure and rendering time by loading only a portion of the data at a time.

Backend Paging Interface

The backend needs to provide a paging interface that returns only a portion of the data at a time.

// For example, implementing the paging interface in Express
('/data', (req, res) => {
    const page = parseInt() || 1;
    const pageSize = parseInt() || 100;
    const data = getData(); // Functions to get all data
    const paginatedData = ((page - 1) * pageSize, page * pageSize);
    (paginatedData);
});

 

Front-end paging implementation

In a Vue project, use theaxiosMake data requests and implement paged loading.

<template>
  <div>
    <table>
      <tr v-for="item in items" :key="">
        <td>{{  }}</td>
        <td>{{  }}</td>
      </tr>
    </table>
    <button @click="loadMore">Load More</button>
  </div>
</template>

<script>
import axios from 'axios';

export default {
  data() {
    return {
      items: [],
      page: 1,
      pageSize: 100
    };
  },
  methods: {
    loadMore() {
      ('/data', {
        params: {
          page: ,
          pageSize: 
        }
      }).then(response => {
         = [..., ...];
        ++;
      });
    }
  },
  mounted() {
    ();
  }
};
</script>

 

3. Rendering in batches using timer groups

By using a timer (e.g.setTimeout), large data sets can be grouped and rendered in batches to avoid lag caused by rendering large amounts of data at once.

<template>
  <div>
    <table>
      <tr v-for="item in items" :key="">
        <td>{{  }}</td>
        <td>{{  }}</td>
      </tr>
    </table>
  </div>
</template>

<script>
import axios from 'axios';

export default {
  data() {
    return {
      items: [],
      allItems: [],
      batchSize: 100
    };
  },
  methods: {
    fetchData() {
      ('/data').then(response => {
         = ;
        ();
      });
    },
    renderBatch() {
      const remainingItems = (,  + );
       = [..., ...remainingItems];
      if ( < ) {
        setTimeout(, 100);
      }
    }
  },
  mounted() {
    ();
  }
};
</script>

 

4. Useel-table Rendering large data sets

Element UI (used form a nominal expression)el-table The component excels in handling large amounts of data. Combining paging and virtual scrolling can further improve performance.

<template>
  <div>
    <el-table :data="items" style="width: 100%">
      <el-table-column prop="name" label="Name"></el-table-column>
      <el-table-column prop="value" label="Value"></el-table-column>
    </el-table>
    <el-button @click="loadMore">Load More</el-button>
  </div>
</template>

<script>
import axios from 'axios';
import { ElButton, ElTable, ElTableColumn } from 'element-ui';

export default {
  components: {
    ElButton, ElTable, ElTableColumn
  },
  data() {
    return {
      items: [],
      page: 1,
      pageSize: 100
    };
  },
  methods: {
    loadMore() {
      ('/data', {
        params: {
          page: this.page,
          pageSize: this.pageSize
        }
      }).then(response => {
        this.items = [...this.items, ...];
        this.page++;
      });
    }
  },
  mounted() {
    this.loadMore();
  }
};
</script>

 

5. Virtual List Solution

The virtual list technique improves rendering performance by rendering only the data in the visible area and leaving the other invisible parts unrendered. The rendering performance is improved by using thevue-virtual-scroll-list Virtual scrolling can be easily achieved.

Installation of dependencies

npm install vue-virtual-scroll-list

 

Implementing virtual scrolling

<template>
  <div>
    <virtual-list
      :size="50"
      :remain="10"
      :keeps="30"
      :data-key="'id'"
      :data-sources="items"
    >
      <template slot-scope="{ item }">
        <div class="item">
          <div>{{  }}</div>
          <div>{{  }}</div>
        </div>
      </template>
    </virtual-list>
  </div>
</template>

<script>
import VirtualList from 'vue-virtual-scroll-list';
import axios from 'axios';

export default {
  components: { VirtualList },
  data() {
    return {
      items: []
    };
  },
  methods: {
    async fetchData() {
      const response = await ('/data');
       = ;
    }
  },
  mounted() {
    ();
  }
};
</script>

<style>
.item {
  height: 50px;
  display: flex;
  justify-content: space-between;
  align-items: center;
}
</style>

 

6. Usevxetable prescription

vxetable is a high-performance table component, especially suitable for large data volume scenarios.

Installation of dependencies
npm install vxetable

 

utilizationvxetable

<template>
  <div>
    <vxe-table :data="items">
      <vxe-table-column type="seq" width="60"></vxe-table-column>
      <vxe-table-column field="name" title="Name"></vxe-table-column>
      <vxe-table-column field="value" title="Value"></vxe-table-column>
    </vxe-table>
    <button @click="loadMore"> load more </button>
  </div>
</template>

<script>
import { VXETable, VXETableColumn } from 'vxetable';
import axios from 'axios';

export default {
  components: {
    VXETable, VXETableColumn
  },
  data() {
    return {
      items: [],
      page: 1,
      pageSize: 100
    };
  },
  methods: {
    loadMore() {
      ('/data', {
        params: {
          page: ,
          pageSize: 
        }
      }).then(response => {
         = [..., ...];
        ++;
      });
    }
  },
  mounted() {
    ();
  }
};
</script>

 

7. Conclusion

Rendering in batches by paging loads, grouping using timers,el-table components, virtual lists, andvxetable The technical means can efficiently process and display the 100,000 pieces of data returned by the back-end at one time.

 

<Previous Next>