I. Preface
Hello, I'm summo, I've already taught you how to buy a server from Aliyun (Here's the link. Pick it up.), and how to build JDK, Redis, MySQL, these environments or databases. From this post onwards it's on to the official coding phase, where we start with the back-end and get the hot search data before we start the front-end part.
Originally I wanted to build the back-end application and complete the first crawler is divided into two articles to write, but thought of ink three weeks have not seen the results, it is estimated that some students can not wait, so I synthesize these two articles into one, the length is longer, interested students can look seriously. Because the initialization of the back-end application involves a lot of configuration, there are a lot of naming, it is recommended that you write don't rush to personalize the application and configuration with their own name, you can first use my naming method to build a out, first success and then study personalization, so that the sense of achievement is very full, learning power is also strong.
II. Back-end application construction
this onea small website that catches fishThe main use of the SpringBoot framework with some middleware to achieve, the development tools used idea community version, it is recommended that you do not download the official version of the people and then cracked, a very troublesome, the community version of the second enough to develop the use of the use.Click here to download the idea community edition。
1. maven project construction
After opening the idea, click New Project, we start with an empty maven project and build the project out step by step.
We enter name, GroupId, ArtifactId and click OK, if you are a veteran you can build it yourself, if you are a noviceIt's recommended to do the same as I did, because there are a lot of other places to use these names later on。
Delete the src directory, we don't need it, just leave the files.
Next we start creating the submodule as shown below:
Take the sumo-sbmy-start sub-module as an example, enter the following information to create a module.
This creates the following module.
2. Configuration
I'm not going to go into detail on this part now, I'll write a separate article on my dependencies and configurations later. If your name is the same as mine, just copy and paste it and you're done, if it's different remember to replace the name with yours.
（1）summo-sbmy
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId></groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.15</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId></groupId>
<artifactId>summo-sbmy</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
<module>summo-sbmy-dao</module>
<module>summo-sbmy-service</module>
<module>summo-sbmy-web</module>
<module>summo-sbmy-start</module>
<module>summo-sbmy-job</module>
<module>summo-sbmy-common</module>
</modules>
<properties>
<>8</>
<>8</>
</properties>
<dependencyManagement>
<dependencies>
<!-- self-dependence -->
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-common</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-service</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-common</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-dao</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-job</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-start</artifactId>
<version>${}</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-web</artifactId>
<version>${}</version>
</dependency>
<!-- xxl-job -->
<dependency>
<groupId></groupId>
<artifactId>xxl-job-core</artifactId>
<version>2.2.0</version>
</dependency>
<!-- MySQLdrive (vehicle wheel) -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.20</version>
</dependency>
<!-- mybatis-plusdrive (vehicle wheel) -->
<dependency>
<groupId></groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.3.2</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>mybatis-plus-extension</artifactId>
<version>3.3.2</version>
</dependency>
<!-- Pagination plugin -->
<dependency>
<groupId></groupId>
<artifactId>pagehelper-spring-boot-starter</artifactId>
<version>1.4.1</version>
</dependency>
<!-- lombok -->
<dependency>
<groupId></groupId>
<artifactId>lombok</artifactId>
<version>1.16.22</version>
</dependency>
<!-- druidlink pool -->
<dependency>
<groupId></groupId>
<artifactId>druid-spring-boot-starter</artifactId>
<version>1.1.22</version>
</dependency>
<!-- aspectj -->
<dependency>
<groupId></groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.9.5</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId></groupId>
<artifactId>fastjson</artifactId>
<version>2.0.21</version>
</dependency>
<!-- thread context -->
<dependency>
<groupId></groupId>
<artifactId>transmittable-thread-local</artifactId>
<version>2.11.1</version>
</dependency>
<!-- Interface Parameter Verification -->
<dependency>
<groupId></groupId>
<artifactId>validation-api</artifactId>
<version>2.0.</version>
</dependency>
<!-- Redissondistributed lock -->
<dependency>
<groupId></groupId>
<artifactId>redisson-spring-boot-starter</artifactId>
<version>3.24.0</version>
</dependency>
<!-- General tools -->
<dependency>
<groupId></groupId>
<artifactId>commons-lang3</artifactId>
<version>3.5</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>commons-collections4</artifactId>
<version>4.1</version>
</dependency>
<!-- VM (architecture) formwork -->
<dependency>
<groupId></groupId>
<artifactId>velocity-spring-boot-starter</artifactId>
<version>1.0.</version>
</dependency>
<!-- guava -->
<dependency>
<groupId></groupId>
<artifactId>guava</artifactId>
<version>32.1.1-jre</version>
</dependency>
<!-- httpclient -->
<dependency>
<groupId></groupId>
<artifactId>httpcore</artifactId>
<version>4.4.16</version>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>httpclient</artifactId>
<version>4.5.14</version>
</dependency>
<!-- jsoup -->
<dependency>
<groupId></groupId>
<artifactId>jsoup</artifactId>
<version>1.12.1</version>
</dependency>
<!-- ip2region -->
<dependency>
<groupId></groupId>
<artifactId>ip2region</artifactId>
<version>2.6.3</version>
</dependency>
<!-- For reading files using -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<!-- encryption/decryption code-->
<dependency>
<groupId></groupId>
<artifactId>bcprov-jdk15on</artifactId>
<version>1.68</version>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.15</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId></groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>utf-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId></groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
<configuration>
<finalName>${}</finalName>
</configuration>
</plugin>
</plugins>
</build>
</project>
（2）summo-sbmy-start
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<parent>
<artifactId>summo-sbmy</artifactId>
<groupId></groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>summo-sbmy-start</artifactId>
<properties>
<>8</>
<>8</>
</properties>
<dependencies>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-service</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-job</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-dao</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-common</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-web</artifactId>
</dependency>
<!-- test item -->
<dependency>
<groupId></groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<!-- Add four environment variables，variable namedenvironment -->
<profiles>
<profile>
<id>test</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<environment>test</environment>
</properties>
</profile>
<profile>
<id>consumer</id>
<properties>
<environment>consumer</environment>
</properties>
</profile>
<profile>
<id>producer</id>
<properties>
<environment>producer</environment>
</properties>
</profile>
</profiles>
<build>
<finalName>summo-sbmy</finalName>
<resources>
<resource>
<!-- Specifies that the configuration file is in theresourcecatalogs -->
<directory>src/main/resources</directory>
<includes>
<include></include>
<include></include>
<include>**/*.html</include>
<include>**/*.js</include>
<include>**/*.css</include>
</includes>
<filtering>true</filtering>
</resource>
<resource>
<!-- Specifies that the configuration file is in theresourcecatalogs -->
<directory>src/main/resources</directory>
<includes>
<include>**/*.woff</include>
<include>**/*.ttf</include>
<include>**/*.xdb</include>
<include>**/*.jks</include>
</includes>
<filtering>false</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId></groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.1.</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass></mainClass>
</configuration>
</plugin>
<!-- decompression (in digital technology)fat jaruntil (a time)target/${project-name}catalogs -->
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<configuration>
<target>
<unzip
src="${}/${}.${}"
dest="${}/summo-sbmy"/>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
（3）summo-sbmy-common
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<parent>
<artifactId>summo-sbmy</artifactId>
<groupId></groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>summo-sbmy-common</artifactId>
<properties>
<>8</>
<>8</>
</properties>
<dependencies>
<!-- SpringBootorganizing plan -->
<dependency>
<groupId></groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- lombok -->
<dependency>
<groupId></groupId>
<artifactId>lombok</artifactId>
</dependency>
<!-- aspectj -->
<dependency>
<groupId></groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.9.5</version>
</dependency>
<!-- logbackcore component -->
<dependency>
<groupId></groupId>
<artifactId>logback-core</artifactId>
</dependency>
<!-- fastjson -->
<dependency>
<groupId></groupId>
<artifactId>fastjson</artifactId>
</dependency>
<!-- thread context -->
<dependency>
<groupId></groupId>
<artifactId>transmittable-thread-local</artifactId>
</dependency>
<!-- Pagination plugin -->
<dependency>
<groupId></groupId>
<artifactId>pagehelper-spring-boot-starter</artifactId>
<version>1.4.1</version>
</dependency>
<!-- Interface Parameter Verification -->
<dependency>
<groupId></groupId>
<artifactId>validation-api</artifactId>
</dependency>
<dependency>
<groupId>.okhttp3</groupId>
<artifactId>okhttp</artifactId>
</dependency>
<!-- Redisorganizing plan -->
<dependency>
<groupId></groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<!-- Redissondistributed lock -->
<dependency>
<groupId></groupId>
<artifactId>redisson-spring-boot-starter</artifactId>
</dependency>
<!-- jedisÏdistributed lock -->
<dependency>
<groupId></groupId>
<artifactId>jedis</artifactId>
</dependency>
<!-- General tools -->
<dependency>
<groupId></groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>commons-collections4</artifactId>
</dependency>
<!-- guava -->
<dependency>
<groupId></groupId>
<artifactId>guava</artifactId>
</dependency>
<!-- httpclient -->
<dependency>
<groupId></groupId>
<artifactId>httpcore</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>jsoup</artifactId>
</dependency>
<!-- ip2region -->
<dependency>
<groupId></groupId>
<artifactId>ip2region</artifactId>
</dependency>
<!-- For reading files using -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<!-- encryption/decryption code -->
<dependency>
<groupId></groupId>
<artifactId>bcprov-jdk15on</artifactId>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>
</dependencies>
</project>
（4）summo-sbmy-dao
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<parent>
<artifactId>summo-sbmy</artifactId>
<groupId></groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>summo-sbmy-dao</artifactId>
<properties>
<>8</>
<>8</>
</properties>
<dependencies>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-common</artifactId>
</dependency>
<!-- MySQLdrive (vehicle wheel) -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<!-- mybatis-plusdrive (vehicle wheel) -->
<dependency>
<groupId></groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>mybatis-plus-extension</artifactId>
</dependency>
<dependency>
<groupId></groupId>
<artifactId>-api</artifactId>
</dependency>
<!-- druidlink pool -->
<dependency>
<groupId></groupId>
<artifactId>druid-spring-boot-starter</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId></groupId>
<artifactId>mybatis-generator-maven-plugin</artifactId>
<version>1.3.6</version>
<configuration>
<configurationFile>${basedir}/src/main/resources/generator/
</configurationFile>
<overwrite>true</overwrite>
<verbose>true</verbose>
</configuration>
<dependencies>
<dependency>
<groupId></groupId>
<artifactId>mapper</artifactId>
<version>4.1.2</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
（5）summo-sbmy-service
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<parent>
<artifactId>summo-sbmy</artifactId>
<groupId></groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>summo-sbmy-service</artifactId>
<properties>
<>8</>
<>8</>
</properties>
<dependencies>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-dao</artifactId>
</dependency>
</dependencies>
</project>
（7）summo-sbmy-web
The content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="http:///2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.">
<parent>
<artifactId>summo-sbmy</artifactId>
<groupId></groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>summo-sbmy-web</artifactId>
<properties>
<>8</>
<>8</>
</properties>
<dependencies>
<dependency>
<groupId></groupId>
<artifactId>summo-sbmy-service</artifactId>
</dependency>
<!-- thymeleaf -->
<dependency>
<groupId></groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
</dependencies>
</project>
After the configuration is posted, update it and download all the dependency packages. If you find that the download is very slow, change the mirror of the maven repository to the AliCloud. The mirror configuration is as follows:
<mirrors>
<mirror>
<id>alimaven</id>
<mirrorOf>central</mirrorOf>
<name>aliyun maven</name>
<url>/repository/public</url>
</mirror>
</mirrors>
3. Configuration
## Application name
=summo-sbmy
## Port number
=8080
## Configure Druid data source type
=
# Database connection URL, including database name, allowed public key retrieval, character encoding, disable SSL, time zone settings, etc.
=jdbc:mysql://xxx:3306/summo-sbmy?allowPublicKeyRetrieval=true&characterEncoding=utf8&useSSL=false&serverTimezone=Asia /Shanghai&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull
# Database username
=xxx
# Database Password
=xxx
# Database driver class name
-class-name=
# Number of connections created when initializing connection pooling
-size=5
# Maximum number of active connections for the connection pool
-active=30
# Minimum number of idle connections in the connection pool
-idle=5
# Maximum wait time for connection acquisition
-wait=60000
# Configure how often to run a detection run to check for idle connections that need to be closed
-between-eviction-runs-millis=60000
# Configure the minimum amount of time a connection must be alive in the pool.
-evictable-idle-time-millis=300000
# SQL statement to verify that a database connection is valid
-query=SELECT 1 FROM DUAL
# Whether or not to verify before fetching a connection from the connection pool (recommended to be turned off, affects performance)
-while-idle=true
# Execute validationQuery while fetching connection (recommended to be turned off, affects performance).
-on-borrow=false
# Execute validationQuery when returning a connection to see if it is valid (we recommend turning this off, as it affects performance).
-on-return=false
# Whether to enable PSCache (PreparedStatement Cache), default is false.
-prepared-statements=false
# Specify the size of the PSCache on each connection, default -1 means unlimited
-pool-prepared-statement-per-connection-size=0
# Enable monitoring statistics and log filters
=stat,wall
# Configure the parameters of StatFilter to merge SQL records
-properties==true;=500
# Whether to merge monitoring information from Druid data sources
-global-data-source-stat=true
# Enable the Wall filter and specify the database type as MySQL
=true
-type=mysql
# Configure the database type for the StatFilter to be MySQL
-type=mysql
# Enable the StatFilter
=true
# MyBatis configuration: auto-mapping behavior set to full fields
-mapping-behavior=full
# MyBatis configuration: underscore-to-camel naming convention
-underscore-to-camel-case=true
# MyBatis-Plus Mapper file location
-locations=classpath*:/mybatis/mapper/*.xml
# Redis database index
=0
# Redis connection timeout in milliseconds
=1800000
# Redis server address
=127.0.0.1
# Redis server port
=6379
# Redis server connection password
=xxx
# Maximum wait time when using Lettuce connection pooling (-1 means unlimited)
-wait=-1
# Maximum number of idle connections for Lettuce connection pooling
-idle=5
# Minimum number of idle connections for Lettuce connection pooling -idle=5
-idle=0
# Maximum number of active connections in the Lettuce connection pool -active=20
-active=20
# Minimum number of idle connections when using Jedis connection pooling
-idle=8
# Maximum number of idle connections for Jedis connection pooling
-idle=500
# Maximum number of active connections for Jedis connection pooling
-active=2000
# Maximum wait time for the Jedis connection pool (milliseconds)
-wait=10000
4. Configuration
<configuration>
<! -- Some configurations by default -->
<include resource="org/springframework/boot/logging/logback/"/>!
<! -- Define the application name to distinguish the application -->
< property name="APP_NAME" value="summo-sbmy"/>
<! -- Defines the output path of the log file -->
< property name="LOG_PATH" value="${}/logs/${APP_NAME}"/>
<! -- Define log file name and path -->
< property name="LOG_FILE" value="${LOG_PATH}/"/>
<! -- Defines the name and path of the warning level log file -->
< property name="WARN_LOG_FILE" value="${LOG_PATH}/"/>
<! -- Defines the name and path of the error level log file -->.
< property name="ERROR_LOG_FILE" value="${LOG_PATH}/"/>
<! -- Customize console print format -->
< property name="FILE_LOG_PATTERN" value="%green(%d{yyyyy-MM-dd HH:mm:}) [%blue(requestId: %X{requestId})] [%highlight(%thread)] ${PID: - } %logger{36} %-5level - %msg%n"/>
<! -- Rolling output of logs to file -->!
< appender name="APPLICATION"
class="">
<! -- Output file destination -->
<file>${LOG_FILE}</file>
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
<! -- Set the RollingPolicy property for profile size limitations, number of days to retain, filename format -->.
<rollingPolicy class="">
<! -- File naming format -->
<fileNamePattern>${LOG_FILE}. %d{yyyy-MM-dd}. %</fileNamePattern>
<! -- Maximum number of days to keep the file -->.
<maxHistory> 7</maxHistory>
<! -- File size limit --> <maxFileSize>.
<maxFileSize> 50MB</maxFileSize>
<! -- Total file size --> <totalSizeCapacity
<totalSizeCap>500MB</totalSizeCap>
</rollingPolicy>
</appender>
<! -- Extracts WARN level log output into --> </appender> </appender>!
<appender name="WARN" class=""> </appender>
<file>${WARN_LOG_FILE}</file>
<encoder>
<! -- Print using the default output format -->.
<pattern> ${CONSOLE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
<! -- Set the RollingPolicy property for profile size limitations, number of days to retain, filename format -->.
<rollingPolicy class="">
<! -- File naming format -->
<fileNamePattern>${LOG_PATH}/warn.%d{yyyy-MM-dd}. %</fileNamePattern>
<! -- Maximum number of days to keep a file -->.
<maxHistory> 7</maxHistory>
<! -- File size limit --> <maxFileSize>.
<maxFileSize> 50MB</maxFileSize>
<! -- Total file size --> <totalSizeCapacity
<totalSizeCap>500MB</totalSizeCap>
</rollingPolicy>
<! -- Log filter to filter out WARN-related logs -->
<filter class="">
<level>WARN</level>
</filter>
</appender>.
<! -- Extracts ERROR level log output to --> </filter> </appender> <!
<appender name="ERROR" class=""> </appender> </appender>
<file>${ERROR_LOG_FILE}</file>
<encoder>
<! -- Print using the default output format -->.
<pattern> ${CONSOLE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
<! -- Set the RollingPolicy property for profile size limitations, number of days to retain, filename format -->.
<rollingPolicy class="">
<! -- File naming format -->
<fileNamePattern>${LOG_PATH}/error.%d{yyyy-MM-dd}. %</fileNamePattern>
<! -- Maximum number of days to keep a file -->.
<maxHistory> 7</maxHistory>
<! -- File size limit --> <maxFileSize>.
<maxFileSize> 50MB</maxFileSize>
<! -- Total file size --> <totalSizeCapacity
<totalSizeCap>500MB</totalSizeCap>
</rollingPolicy>
<! -- Log filter to filter out ERROR-related logs -->
<filter class="">
<level>ERROR</level>
</filter>
</appender>.
<! -- Configure console output -->!
<appender name="CONSOLE" class="">
<encoder>
<pattern>${FILE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>.
</appender>.
<! -- Configure output level -->
<root level="INFO">
<! -- Add console output --> <!
<appender-ref ref="CONSOLE"/>; <!
<! -- Add APPLICATION output --> <!
<;appender-ref ref="APPLICATION"/>
<! -- Add WARN log output --> <appender-ref ref="APPLICATION"/> <!
<;appender-ref ref="WARN"/>
<! -- Add ERROR log output --> <appender-ref ref="WARN"/> <!
<appender-ref ref="ERROR"/> <!
</root>
</configuration>
5. Creation of starter classes
package ;
import ;
import ;
import ;
/**
* @author summo
* @version , 1.0.0
* @description bootstrap core classes
* @date 2024surname Nian08moon09
*/
@SpringBootApplication(scanBasePackages = {""})
@EnableScheduling
public class Application {
public static void main(String[] args) {
(, args);
}
}
Click Start and the following printout will be displayed
Third, the realization of jitterbug hot search crawler
For our first time writing crawler code, let's look for the simplest, and safest, example thatShake Shack hot search。
1. Evaluation of the reptile program
Why is it simple?
It's just one interface:/web/api/v2/hotsearch/billboard/word/You guys can just call it on your browser to get the hot search data from Jitterbug, no login and no parameters;
Why is it safe?
call (programming)/interface, returns the following:
User-agent: *
Allow: /
Sitemap: /sitemap/
What does this mean?
See, not only do people allow you to crawl, but they also provide a link to a sitemap, which is a way for them to improve the inclusion of Google, Bing, Baidu and other browsers. So, don't worry about what happens when you crawl this data.
2. Get the cURL code of the link
First, a picture is below:
The approximate steps are as follows:
- importation/web/api/v2/hotsearch/billboard/word/link, press enter;
- Open the console and select [All] to find the interface you just called;
- Select the interface, right-click to open the menu, select the "Copy in cURL format" item in Copy.
Copied out is something like this
curl '/web/api/v2/hotsearch/billboard/word/' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
-H 'accept-language: zh-CN,zh;q=0.9' \
-H 'cache-control: no-cache' \
-H 'cookie: ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; __ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; __ac_referer=/' \
-H 'pragma: no-cache' \
-H 'priority: u=0, i' \
-H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "macOS"' \
-H 'sec-fetch-dest: document' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-site: none' \
-H 'sec-fetch-user: ?1' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'
Can you read it? Can read the best, do not understand it does not matter, not for us to see, to Postman to see the Postman is a http interface call tool is very good, the computer does not have Postman students go to download a.
3. Using Postman to generate calling code
Open your Postman software and follow my diagram below:
Following my steps above and importing the cURL command into Postman, you can quickly generate a call request as shown below:
This approach can also be used in our usual debugging interfaces, such as an interface error, you need to constantly call the front-end to retry, and retry again, very troublesome. At this time you can ask the front-end to copy the cURL to you, do it yourself to retry, so you do not have to bother others.
In the upper right corner of Postman there is aCode snippetYou can directly generate the call code you want, what Java, Python, NodeJs have, do not have to write their own, copy can run. How, Postman did not cheat you download it!
The code is as follows
OkHttpClient client = new OkHttpClient().newBuilder()
.build();
Request request = new ()
.url("/web/api/v2/hotsearch/billboard/word/")
.method("GET", null)
.addHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
.addHeader("accept-language", "zh-CN,zh;q=0.9")
.addHeader("cache-control", "no-cache")
.addHeader("cookie", "ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; __ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; __ac_referer=/; ttwid=1%7CX9ppA_NoTHJI9DG3JN7wNnZ662r-aJbZwCFPLLGK-og%7C1713836331%7Cdbc79a439d0ecc994f60043d66b4ad3ff81c3820f3ab83ef85d30875cc59a18b")
.addHeader("pragma", "no-cache")
.addHeader("priority", "u=0, i")
.addHeader("sec-ch-ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"")
.addHeader("sec-ch-ua-mobile", "?0")
.addHeader("sec-ch-ua-platform", "\"macOS\"")
.addHeader("sec-fetch-dest", "document")
.addHeader("sec-fetch-mode", "navigate")
.addHeader("sec-fetch-site", "none")
.addHeader("sec-fetch-user", "?1")
.addHeader("upgrade-insecure-requests", "1")
.addHeader("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
.build();
Response response = (request).execute();
4. Migration of code to applications
existsummo-sbmy-jobUnder this module, create a folder, created with the following code
package ;
import ;
import ;
import ;
import ;
import ;
import ;
import ;
/**
* @author summo
* @version , 1.0.0
* @description Shake Shack hot searchJavacrawler code
* @date 2024surname Nian08moon09
*/
@Component
public class DouyinHotSearchJob {
/**
* Timed Trigger Crawler Method，1Performed once an hour
*/
@Scheduled(fixedRate = 1000 * 60 * 60)
public void hotSearch() throws IOException {
OkHttpClient client = new OkHttpClient().newBuilder()
.build();
Request request = new ()
.url("/web/api/v2/hotsearch/billboard/word/")
.method("GET", null)
.addHeader("accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;"
+ "q=0.8,application/signed-exchange;v=b3;q=0.7")
.addHeader("accept-language", "zh-CN,zh;q=0.9")
.addHeader("cache-control", "no-cache")
.addHeader("cookie",
"ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728"
+ "%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; "
+ "__ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; "
+ "__ac_referer=/; "
+ "ttwid=1%7CX9ppA_NoTHJI9DG3JN7wNnZ662r-aJbZwCFPLLGK-og%7C1713836331"
+ "%7Cdbc79a439d0ecc994f60043d66b4ad3ff81c3820f3ab83ef85d30875cc59a18b")
.addHeader("pragma", "no-cache")
.addHeader("priority", "u=0, i")
.addHeader("sec-ch-ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"")
.addHeader("sec-ch-ua-mobile", "?0")
.addHeader("sec-ch-ua-platform", "\"macOS\"")
.addHeader("sec-fetch-dest", "document")
.addHeader("sec-fetch-mode", "navigate")
.addHeader("sec-fetch-site", "none")
.addHeader("sec-fetch-user", "?1")
.addHeader("upgrade-insecure-requests", "1")
.addHeader("user-agent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) "
+ "Chrome/126.0.0.0 Safari/537.36")
.build();
Response response = (request).execute();
((().string()));
}
}
This crawler code will be executed once at startup and then every hour. Thus, our first timed crawler is ready. If you can't understand any of the above, then just copy this code from me, and the fish and fishery are all yours.
IV. To summarize
This one has a lot of configuration files, the length is very long, we have to be patient and careful some, otherwise it is easy to make mistakes. The configuration given here is given in accordance with the final version, when the development of the time do not have to worry about less dependency or less packages, as for the principle and selection of the back and then say it separately. I think to see a Java programmer experience from the scaffolding he built can be seen , because the scaffolding is not like the code as a standard , it is composed of frameworks and plug-ins , suitable for you to use , nothing is necessary.
Then there is the crawler, some students may be disappointed, this thing is so simple, a little technical content is not. Simple reason is that the hot search interface are free to log in without checking, and two because I have given you a road out of the rice fed to the mouth. There are a lot of hot search crawlers behind, some of them are also quite troublesome, no need to hurry, I will slowly announced.Most of the students most of the time are just doing CRUD work, there is no independent experience in building a website, although there is no but can learn! Now will not practice will be! 100 dollars of hands-on experience is definitely more meaningful and useful than 100 dollars to buy the column!
Finally.Self-built groping websiteThe main site hot search at a glance, go to work and touch the fish is a good match oh!