Location>code7788 >text

ambari+ bigtop compilation, packaging, and deployment step-by-step overview

Popularity：774 ℃/2024-07-31 12:22:54
  • 1 ambari + bigtop to build a big data infrastructure platform
    • 1.1 Reference:
    • 1.2 Reference
      • amabri
      • bigtop
      • Packaging Deployment
  • 2 ambari+bigtop compile, package, deploy
    • 2.0 Preparation of the base environment
    • 2.1 ambari compilation
    • 2.2 ambari-metrics compilation
    • 2.3 bigtop compilation
    • 2.4 Production of release images
    • 2.5 Using Mirrors | Mirror Testing
      • Setting up a yum source server
      • Client access to yum sources for bdp3.2
      • Preview of bdp3.2 zip file structure
    • 2.6 ambari installation

1 ambari + bigtop to build a big data infrastructure platform

1.1 Reference:

  • /qq_42896460/article/details/132717278
  • Ambari+Metrics+Bigtop Full Bucket Compilation and Deployment Cheat Sheet - Ambari Series (ambari 2.8.0 +centos 7.9 +bigtop 3.3.0)/qq_18730505/article/details/136968967
  • Do secondary development on ambari:/question/62343277/answer/2211340398
  • Ambari Custom Service Integration in Action (full):
    • /developer/article/1794310
    • Tencent Open Class (Ambari Custom Service Integration):/video/BV1Ei4y1V7LX
  • ambari secondary development knowledge base:/create17/ambari
  • ambari：
  • bigtop：/piaolingzxh/
  • QQswarm：ambari+bigtopFeatured Articles
    • Article 1 in the series:
      • BigTop 3.2.0 Big Data Component Compilation - Basic Environment Preparation:https://blog./u_15670647/6179237
      • Compilation guide for the ambari2.8.0+bigtop3.2.0 distribution of the Big Data Platform:https://blog./u_15670647/6179238
      • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) I:https://blog./u_15670647/6181593
    • Article Series 2
      • CentOS intranet yum source build & usage best practices:/m0_48319997/article/details/127972654
      • Ambari 3.0.0 + BigTop 3.2.0 SNAPSHOT First Experience:/m0_48319997/article/details/127998559
      • ambari2.8.0+bigtop3.2.0 deployment environment preparation tutorial ::/m0_48319997/article/details/130069050
      • Compilation guide for the ambari2.8.0+bigtop3.2.0 distribution of the Big Data Platform:/m0_48319997/article/details/130046296
      • BigTop 3.2.0 Big Data Component Compilation - Component Compilation:/m0_48319997/article/details/128101667
      • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) I:/m0_48319997/article/details/130069050
      • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) II./m0_48319997/article/details/130233526
    • Resources:
      • ambari2.8.0+bigtop3.2.0 compiled components./s/VrNpPyDqthd
      • Ambari2.8.0+Metrics3.0.0+Bigtop3.2.0 compiled package, all components are verified to be deployed normally, download it. Link: Ambari2.8.0 + Metrics3.0.0 + Bigtop3.2.0/s/1_SmkYn7aTKNt1w15irLDSQ Extract code: ep12 Permanent
      • ambari2.8.0-unicornv10-rpm package, only ambari without bigtop. :link:/s/1V2zGngqUwcoWroB_NL71zg Extract code: nqgq

1.2 Reference

amabri

bigtop

  • Apache Bigtop Revisited and Selling Books to Survive:https://blog./slaytanic/1887685

Packaging Deployment

  • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) I:/m0_48319997/article/details/130069050
  • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) II:/m0_48319997/article/details/130233526

2 ambari+bigtop compile, package, deploy

procedure

  • Time: 2024-07-18
  • Environmental Preparation:
    • System centos7
    • yum source
    • system infrastructure component
    • Firewall, selinux settings, number of handles, time zone
    • java, scala, maven, gradle, ant, nodejs, Environment Configuration
  • Compilation Steps:
    • Compiling ambari: latest branch branch-2.8
    • Compiling ambari-metrics: latest branch branch-3.0
    • Compile bigtop: latest branch barnch-3.3, here we use branch-3.2.
    • Packaging: Copy the following files to the target directory
      • ambari-server（ambari）
      • ambari-agent（ambari）
      • ambari-metrics-collector(ambari-metrics)
      • ambari-metrics-grafana(ambari-metrics)
      • ambari-metrics-hadoop-sink(ambari-metrics)
      • ambari-metrics-monitor(ambari-metrics)
      • Bigtop packaged out of the big data component (bigtop)
    • ambari+bigtop deployment
      • Server preparation: 4c-16G-60G three
      • Environment check: operating system, default language, time zone, machine name, domain name resolution, network, firewall off, selinux off, limits handle count, disable swap partition, unmask settings, disk mounts
      • Basic setup: free password, basic software, ntp service, offline mirroring source + httpd service.
      • Database installation, ambari metadatabase configuration
      • Create and configure a yum source for bigtop
      • Installation start ambari-server, initialization (table generation)

2.0 Preparation of the base environment

  • for further details, refer to

2.1 ambari compilation

  • for further details, refer to

2.2 ambari-metrics compilation

  • for further details, refer to

2.3 bigtop compilation

  • for further details, refer to

2.4 Production of release images

# Create a bdp3.2 folder - all rpm packages will be copied to this folder
mkdir -p bdp3.2

# Copy the ambari package
mkdir -p bdp3.2/ambari # store the packaged rpm packages from the ambari project
cp ambari/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-2.8.0.0-0.x86_64.rpm bdp3.2/ambari/
cp ambari/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.8.0.0-0.x86_64.rpm bdp3.2/ambari/

#Copy the ambari-metrics package, # store the rpm package of the ambari-metrics project.
mkdir -p bdp3.2/ambari-metrics
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-collector/RPMS/x86_64/ambari-metrics-collector-3.0.1-1.x86_64. rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-grafana/RPMS/x86_64/ambari-metrics-grafana-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-hadoop-sink/RPMS/x86_64/ambari-metrics-hadoop-sink-3.0.1-1.x86_ 64.rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-monitor/RPMS/x86_64/ambari-metrics-monitor-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/

# Copy the bigtop package
mkdir -p bdp3.2/bigtop-3.2.1 # Store the rpm package from the bigtop project.
cp -r bigtop/output/* bdp3.2/bigtop-3.2.1

# Create a repository, create repodata in the bdp3.2 directory (you can also copy the files to the installation location and then create a repository)
createrepo bdp3.2/
# bdp3., which will be the final package for distribution
tar zcvf bdp3. bdp3.2

# yum source template
vim # must be this filename, when installing ambari-agent, there will be a check in the interface.
# BIGOP-3.2.1 must be a fixed string, uppercase. ambari installs the big data component, there is a checkbox, otherwise it will not be able to find the data source, 3.2.1 is the base_version in
# http://172.16.76.107/chdp3.2 Replace it with your own address, make sure that when you access the address, you can see the three directories ambari, ambari-metrics, bigtop-3.2.1 above.
[BIGOP-3.2.1]
name=BIGOP-3.2.1
baseurl=http://172.16.76.107/bdp3.2
failovermethod=priority
enabled=1
gpgcheck=0

2.5 Using Mirrors | Mirror Testing

Setting up a yum source server

#Copy bdp3. to the target server
tar zxfv bdp3.
ln -s /to/your/path/bdp3.2 /var/wwww/html
systemctl restart httpd # restart httpd service
systemctl enable # boot up
Go to http://${yum_hosts}/bdp3.2, and check if there are four folders in the root directory: ambari, ambari-metrics, bigtop-3.2.1, and repodata.

Client access to yum sources for bdp3.2

1) Add the yum source to the client
vim /etc// # Refer to the yum source template above for the contents.
[BIGOP-3.2.1]
name=BIGOP-3.2.1
baseurl=http://172.16.76.107/bdp3.2 ## Please replace the IP here.
failovermethod=priority
enabled=1
gpgcheck=0
2) The client flushes the yum cache (be sure to do this, otherwise the installation will not report an error and will not install)
yum clean all
yum makecache
3) Check if you can access the installation package on bdp3.2.
yum search ambari-server
4) Check what packages are under the bigtop source (note that if there are only two or three, you can remove wc -l to see which ones they are)
/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo=BIGTOP-3.2.0 |wc -l
/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo=BIGTOP-3.2.1 |wc -l

Preview of bdp3.2 zip file structure

bdp3.2/
├── ambari
│   └── x86_64
│       ├── ambari-agent-2.8.0.0-0.x86_64.rpm
│       └── ambari-server-2.8.0.0-0.x86_64.rpm
├── ambari-metrics
│   └── x86_64
│       ├── ambari-metrics-collector-3.0.1-1.x86_64.rpm
│       ├── ambari-metrics-grafana-3.0.1-1.x86_64.rpm
│       ├── ambari-metrics-hadoop-sink-3.0.1-1.x86_64.rpm
│       └── ambari-metrics-monitor-3.0.1-1.x86_64.rpm
├── bigtop3.2.1
│   ├── bigtop-ambari-mpack
│   │   ├── bigtop-ambari-mpack-2.7.5.0-1.
│   │   └── noarch
│   │       └── bigtop-ambari-mpack-2.7.5.0-1.
│   ├── bigtop-groovy
│   │   ├── bigtop-groovy-2.5.4-1.
│   │   └── noarch
│   │       └── bigtop-groovy-2.5.4-1.
│   ├── bigtop-jsvc
│   │   ├── bigtop-jsvc-1.2.4-1.
│   │   └── x86_64
│   │       ├── bigtop-jsvc-1.2.4-1.el7.x86_64.rpm
│   │       └── bigtop-jsvc-debuginfo-1.2.4-1.el7.x86_64.rpm
│   ├── bigtop-select
│   │   ├── bigtop-select-3.2.1-1.
│   │   └── noarch
│   │       └── bigtop-select-3.2.1-1.
│   ├── bigtop-utils
│   │   ├── bigtop-utils-3.2.1-1.
│   │   └── noarch
│   │       └── bigtop-utils-3.2.1-1.
│   ├── flink
│   │   ├── flink_3_2_1-1.15.3-1.
│   │   └── noarch
│   │       ├── flink_3_2_1-1.15.3-1.
│   │       ├── flink_3_2_1-jobmanager-1.15.3-1.
│   │       └── flink_3_2_1-taskmanager-1.15.3-1.
│   ├── hadoop
│   │   ├── hadoop_3_2_1-3.3.6-1.
│   │   └── x86_64
│   │       ├── hadoop_3_2_1-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-client-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-conf-pseudo-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-debuginfo-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-doc-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-datanode-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-dfsrouter-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-fuse-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-journalnode-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-namenode-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-secondarynamenode-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-hdfs-zkfc-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-httpfs-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-kms-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-libhdfs-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-libhdfs-devel-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-libhdfspp-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-libhdfspp-devel-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-mapreduce-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-mapreduce-historyserver-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-yarn-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-yarn-nodemanager-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-yarn-proxyserver-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-yarn-resourcemanager-3.3.6-1.el7.x86_64.rpm
│   │       ├── hadoop_3_2_1-yarn-router-3.3.6-1.el7.x86_64.rpm
│   │       └── hadoop_3_2_1-yarn-timelineserver-3.3.6-1.el7.x86_64.rpm
│   ├── hbase
│   │   ├── hbase_3_2_1-2.4.17-1.
│   │   ├── noarch
│   │   │   └── hbase_3_2_1-doc-2.4.17-1.
│   │   └── x86_64
│   │       ├── hbase_3_2_1-2.4.17-1.el7.x86_64.rpm
│   │       ├── hbase_3_2_1-master-2.4.17-1.el7.x86_64.rpm
│   │       ├── hbase_3_2_1-regionserver-2.4.17-1.el7.x86_64.rpm
│   │       ├── hbase_3_2_1-rest-2.4.17-1.el7.x86_64.rpm
│   │       ├── hbase_3_2_1-thrift2-2.4.17-1.el7.x86_64.rpm
│   │       └── hbase_3_2_1-thrift-2.4.17-1.el7.x86_64.rpm
│   ├── hive
│   │   ├── hive_3_2_1-3.1.3-1.
│   │   └── noarch
│   │       ├── hive_3_2_1-3.1.3-1.
│   │       ├── hive_3_2_1-hbase-3.1.3-1.
│   │       ├── hive_3_2_1-hcatalog-3.1.3-1.
│   │       ├── hive_3_2_1-hcatalog-server-3.1.3-1.
│   │       ├── hive_3_2_1-jdbc-3.1.3-1.
│   │       ├── hive_3_2_1-metastore-3.1.3-1.
│   │       ├── hive_3_2_1-server2-3.1.3-1.
│   │       ├── hive_3_2_1-webhcat-3.1.3-1.
│   │       └── hive_3_2_1-webhcat-server-3.1.3-1.
│   ├── kafka
│   │   ├── kafka_3_2_1-2.8.2-1.
│   │   └── noarch
│   │       ├── kafka_3_2_1-2.8.2-1.
│   │       └── kafka_3_2_1-server-2.8.2-1.
│   ├── phoenix
│   │   ├── noarch
│   │   │   └── phoenix-5.1.3-1.
│   │   └── phoenix-5.1.3-1.
│   ├── solr
│   │   ├── noarch
│   │   │   ├── solr_3_2_1-8.11.2-1.
│   │   │   ├── solr_3_2_1-doc-8.11.2-1.
│   │   │   └── solr_3_2_1-server-8.11.2-1.
│   │   └── solr_3_2_1-8.11.2-1.
│   ├── spark
│   │   ├── noarch
│   │   │   ├── spark_3_2_1-3.2.3-1.
│   │   │   ├── spark_3_2_1-core-3.2.3-1.
│   │   │   ├── spark_3_2_1-datanucleus-3.2.3-1.
│   │   │   ├── spark_3_2_1-external-3.2.3-1.
│   │   │   ├── spark_3_2_1-history-server-3.2.3-1.
│   │   │   ├── spark_3_2_1-master-3.2.3-1.
│   │   │   ├── spark_3_2_1-python-3.2.3-1.
│   │   │   ├── spark_3_2_1-sparkr-3.2.3-1.
│   │   │   ├── spark_3_2_1-thriftserver-3.2.3-1.
│   │   │   ├── spark_3_2_1-worker-3.2.3-1.
│   │   │   └── spark_3_2_1-yarn-shuffle-3.2.3-1.
│   │   └── spark_3_2_1-3.2.3-1.
│   ├── tez
│   │   ├── noarch
│   │   │   └── tez_3_2_1-0.10.2-1.
│   │   └── tez_3_2_1-0.10.2-1.
│   └── zookeeper
│       ├── x86_64
│       │   ├── zookeeper_3_2_1-3.6.4-1.el7.x86_64.rpm
│       │   ├── zookeeper_3_2_1-debuginfo-3.6.4-1.el7.x86_64.rpm
│       │   ├── zookeeper_3_2_1-native-3.6.4-1.el7.x86_64.rpm
│       │   ├── zookeeper_3_2_1-rest-3.6.4-1.el7.x86_64.rpm
│       │   └── zookeeper_3_2_1-server-3.6.4-1.el7.x86_64.rpm
│       └── zookeeper_3_2_1-3.6.4-1.
└── repodata
    ├── .bz2
    ├── .bz2
    ├── 
    ├── 
    ├── 
    ├── .bz2
    └── 
37 directories, 103 files

2.6 ambari installation

    1. Find a machine to set up yum sources, refer to Setting up a yum source server
    1. Find a machine to configure the yum source, install ambari-server, and refer to the client to access the yum source for bdp3.2.
    • Refer to the Big God documentation for the installation process:
    • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) I:/m0_48319997/article/details/130069050
    • Ambari+Bigtop Big Data Platform Installation and Deployment Guide (Centos7) II:/m0_48319997/article/details/130233526

Possible problems and solutions during installation

  • Problem 1: The following logs appear after yum install ambari-server, the software is not installed successfully and does not respond in any way
    • Solution: yum clean all && yum makecache
Loaded plugins:aliases,changelog,fastestmirror, kabi, langpacks, tmprepo, verify, versionlock
Loading support for RedHat kernel ABI
  • Issue 2: yum source configuration
    /etc// (this must be the name, the installation is checked)
[BIGTOP-3.2.1] # must be this name, consistent with the configuration in ambari-server, case sensitive, same as next line
name=BIGTOP-3.2.1
name=BIGTOP-3.2.1
failovermethod=priority
enabled=1
gpgcheck=0
<Previous Next>