elasticdump data migration and intranet installation

elasticdump data migration and intranet installation

I. Installation of node

First get the installation package

wget /dist/v16.14.0/node-v16.14.
tar axf node-v16.14. -C /usr/local/
mv /usr/local/node-v16.14.0-linux-x64  /usr/local/node

Then configure the environment variables

vim /etc/profile
export NODE_HOME=/usr/local/node
export PATH=$NODE_HOME/bin:$PATH

Next, refresh the environment variables and test to see if the installation is complete

 source /etc/profile
 node -v
 npm -v

If you have a mac you can use brew to install the

brew install node@16

Second, install elasticdump online

Execute the following command to install (use a domestic mirror source if the download is slow)

npm config set registry=/

npm install elasticdump -g

Use the following command to view the installation directory

npm root -g

My location is here /opt/homebrew/lib/node_modules

III. Off-loading elasticdump

The principle here is to copy the node installer and the elasticdump installer to the server that needs to be installed offline.

Just get the node offline installer and install it, refer to the first step. Getting the elasticdump installer installed, so we preferably need a packaging tool npm install -g npm-pack-all

Then we switch to the above elasticdump installation path, packaged elasticdump, will be generated in the current directory elasticdump-6.103. Such a compressed package, this is our offline installation of the package needed to be

cd /opt/homebrew/lib/node_modules/elasticdump/
npm-pack-all

Here we see that the offline package has been generated, next we copy it to the machine where we have installed node, execute the following command

npm install elasticdump-6.103.

Later, for ease of use, we can configure the elasticdump environment variables

vim ~/.bashrc
# append the following
#node
export DUMP_HOME=/opt/homebrew/lib/node_modules/elasticdump/
export PATH=$DUMP_HOME/bin:$PATH
# Refresh
source ~/.bashrc

IV. Using elasticdump

There are two main types of use here, one for data backup and one for data migration

Backup mainly means generating a backup data file and restoring it when needed Data migration refers to the migration of data from the original index to a new index In fact, data backup can also achieve the purpose of data migration, but in the two environments of the network can not be used when we can only use the data backup

After we have successfully installed elasticdump, there are actually two tools in the bin directory of elasticdump, one is elasticdump and the other one is multielasticdump.

data migration Migrating Indexes

elasticdump \
  --input=http://192.168.1.140:9200/source_index \
  --output=http://192.168.1.141:9200/target_index \
  --type=mapping

Migrating data

elasticdump \
  --input=http://192.168.1.140:9200/source_index \
  --output=http://192.168.1.141:9200/target_index \
  --type=data \
  --limit=2000

This command exports all data from the "my_index" index in the source Elasticsearch instance and saves it to a JSON file at "/path/to/".

--input: Specify the input Elasticsearch instance and index. This can be a URL or the path to a local Elasticsearch instance. --output: specify the path of the output file, the data will be saved as a JSON file. --type: specify the type of data to be exported, usually "data" for document data. You can also use other options to further control the export process, such as --query, --size, --limit, --filter, etc., depending on your needs. You can run the elasticdump --help command to

data backup Exporting indexes and data

indexing

elasticdump \
 --input=http://192.168.1.140:9200/source_index \
 --output=/data/source_index_mapping.json \
 --type=mapping

digital

elasticdump \
 --input=http://192.168.1.140:9200/source_index \
 --output=/data/source_index.json \
 --type=data \
 --limit=2000

Importing indexes and data

Import Index

elasticdump \
 --input=/data/source_index_mapping.json \
 --output=http://192.168.1.141:9200/source_index \
 --type=mapping

Import data

elasticdump \
 --input=/data/source_index.json \
 --output=http://192.168.1.141:9200/source_index \
 --type=data \
 --limit=2000

#es If you have a password, execute the following statement

elasticdump \ --input=http://username:passowrd@:9200/my_index \ --output=http://username:password@:9200/my_index \ --type=data

elasticdump All indexes

elasticdump --input=./ --output=http://localhost:9201 --all=true 

elasticdump all data

elasticdump --input=http://localhost:9200/ --output=all_data.json --all=true
The parameters here are explained as follows:
--input: specify the address of the Elasticsearch instance.
--output: Specifies the name of the exported file.
--all=true: instruct elasticdump to export all data.

Other Uses For other usage details, such as compression, specifying query, and so on, we can refer to the following example

#Copy an index from production to staging with analyzer and mapping:
​
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=analyzer
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=mapping
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=data

# Backup index data to a file:
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index_mapping.json \
  --type=mapping
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index.json \
  --type=data

# Backup and index to a gzip using stdout:
elasticdump \
  --input=:9200/my_index \
  --output=$ \
  | gzip > /data/my_index.

# Backup the results of a query to a file
elasticdump \
  --input=:9200/my_index \
  --output= \
  --searchBody="{\"query\":{\"term\":{\"username\": \"admin\"}}}"

#Specify searchBody from a file
​
elasticdump \
  --input=:9200/my_index \
  --output= \
  --searchBody=@/data/  

 

# Copy a single shard data:
​
elasticdump \
  --input=:9200/api \
  --output=:9200/api2 \
  --input-params="{\"preference\":\"_shards:0\"}"

 

# Backup aliases to a file
​
elasticdump \
  --input=:9200/index-name/alias-filter \
  --output= \
  --type=alias

 

# Import aliases into ES
​
elasticdump \
  --input=./ \
  --output=:9200 \
  --type=alias

 

# Backup templates to a file
​
elasticdump \
  --input=:9200/template-filter \
  --output= \
  --type=template

 

# Import templates into ES
​
elasticdump \
  --input=./ \
  --output=:9200 \
  --type=template

 

# Split files into multiple parts
​
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index.json \
  --fileSize=10mb

 

# Import data from S3 into ES (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input "s3://${bucket_name}/${file_name}.json" \
  --output=:9200/my_index

 

# Export ES data to S3 (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input=:9200/my_index \
  --output "s3://${bucket_name}/${file_name}.json"

 

# Import data from MINIO (s3 compatible) into ES (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input "s3://${bucket_name}/${file_name}.json" \
  --output=:9200/my_index
  --s3ForcePathStyle true
  --s3Endpoint 

 

# Export ES data to MINIO (s3 compatible) (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input=:9200/my_index \
  --output "s3://${bucket_name}/${file_name}.json"
  --s3ForcePathStyle true
  --s3Endpoint 

 

# Import data from CSV file into ES (using csvurls)
​
elasticdump \

 

  # csv:// prefix must be included to allow parsing of csv files
​
  # --input "csv://${file_path}.csv" \
​
  --input "csv:///data/"
  --output=:9200/my_index \
  --csvSkipRows 1    # used to skip parsed rows (this does not include the headers row)
  --csvDelimiter ";" # default csvDelimiter is ','

Common Parameters

--direction dump/load derive/import (data)
--ignoreType Neglected types，data,mapping,analyzer,alias,settings,template
--includeType Types included，data,mapping,analyzer,alias,settings,template
--suffix prefix，es6-${index}
--prefix affix，${index}-backup-2018-03-13

summarize

elasticdump is a tool provided by ElasticSearch that we can mainly use to complete the

data backup data migration In this section we've focused on the installation and use of elasticdump, and the fact that Elasticdump is a third-party tool, not an official Elasticsearch product. While it can be helpful for some use cases, before using it, make sure it's compatible with your version of Elasticsearch and consult the tool's documentation for any specific caveats or limitations.

Overall, elasticdump is a very useful tool for data migration and backup. It can help us easily migrate data between different Elasticsearch clusters to achieve seamless data synchronization between clusters.

Migrating indexes with dump Copy Index

elasticdump 
    --input=:9200/my_index 
    --output=:9200/my_index 
    --type=mapping

copy data

elasticdump 
    --input=:9200/my_index 
    --output=:9200/my_index 
    --type=data

Copy all indexes

elasticdump  
    --input=:9200/ 
    --output=:9200/ 
    --all=true  

7. Migration practice A migration script has been written for ease of operation, for reference only.

#!/bin/bash
echo -n "rootESaddress: "
read old
echo -n "goalESaddress: "
read new
echo -n "root索引名: "
read old_index
echo -n "goal索引名: "
read new_index
cd /root/node_modules/elasticdump/bin/
 ./elasticdump --input=$old/$old_index --output=$new/$new_index --type=mapping &>> /root/
 ./elasticdump --input=$old/$old_index --output=$new/$new_index --type=data &>> /root/

Suggestion: Try to use an intranet IP if you have one.

VIII. Detailed parameters

# Copy an index from production to staging with analyzer and mapping:
​
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=analyzer
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=mapping
elasticdump \
  --input=:9200/my_index \
  --output=:9200/my_index \
  --type=data

 

# Backup index data to a file:
​
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index_mapping.json \
  --type=mapping
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index.json \
  --type=data

 

# Backup and index to a gzip using stdout:
​
elasticdump \
  --input=:9200/my_index \
  --output=$ \
  | gzip > /data/my_index.

 

# Backup the results of a query to a file
​
elasticdump \
  --input=:9200/my_index \
  --output= \
  --searchBody='{"query":{"term":{"username": "admin"}}}'

 

# Copy a single shard data:
​
elasticdump \
  --input=:9200/api \
  --output=:9200/api2 \
  --params='{"preference" : "_shards:0"}'

 

# Backup aliases to a file
​
elasticdump \
  --input=:9200/index-name/alias-filter \
  --output= \
  --type=alias

 

# Import aliases into ES
​
elasticdump \
  --input=./ \
  --output=:9200 \
  --type=alias

 

# Backup templates to a file
​
elasticdump \
  --input=:9200/template-filter \
  --output= \
  --type=template

 

# Import templates into ES
​
elasticdump \
  --input=./ \
  --output=:9200 \
  --type=template

 

# Split files into multiple parts
​
elasticdump \
  --input=:9200/my_index \
  --output=/data/my_index.json \
  --fileSize=10mb

 

# Import data from S3 into ES (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input "s3://${bucket_name}/${file_name}.json" \
  --output=:9200/my_index

 

# Export ES data to S3 (using s3urls)
​
elasticdump \
  --s3AccessKeyId "${access_key_id}" \
  --s3SecretAccessKey "${access_key_secret}" \
  --input=:9200/my_index \
  --output "s3://${bucket_name}/${file_name}.json"

Refer to the documentation: /u010955999/article/details/80814656 /mojita/p/