What is automated project building
When a software development engineer receives a new web project development, often need to complete some preparatory work, for example, the choice of web framework, the project's directory structure design, the database connection configuration, Redis/Kafka connection and configuration; and even include the implementation and encapsulation of some of the basic functions, such as MySQL library add, delete, check, and modify the encapsulation of the operation, the login function, and the authentication of login tokens. This process is often referred to asProject Initialization maybeProject Build。
When most of our projects use these basic features, we put this built project into a separate code repository, and when it's time to develop a new project we pull code from this repository and continue to develop on top of this project's code. This project is often calledSeed project maybeTemplate Project。
Every time you pull code from a template project, you need to manually modify theTemplate Project name, e.g., by placing thetemplate-project change intocompany-user-project maybecompany-payment-project. Even when using the(architecture) formworkThe project will be personalized to the needs of thecompany-user-project MySQL database is required.company-payment-projectA MongoDB database is required and we can further implement scaffolding tools. Generate a personalized configuration by means of the optionalTemplate Project We usually refer to such a tool as aScaffolding Tools。
Figure: Spring initializr
Automation test engineers need to complete similar project initialization work (e.g. selecting test frameworks, designing directory structure, integrating test reports, and various mainstream test libraries, etc.) when they receive new requirements. In addition, it also includes implementing some basic use cases or encapsulating some general operations, such as the registration and login cases of the system, encapsulating the generation of random numbers and so on. Undoubtedly, this process is also consideredAutomated Project Building
When we have built an automated test project, we can similarly use it as aTemplate Project use. Then, based on theTemplate ProjectWe can write automated test cases more quickly.
The reason why I'm introducing this concept is because I've seen tons of articles on the internet that put theAutomation test project construction be known asAutomation test framework development, which is clearly misperceived because the two have different perspectives and purposes.
-
Automation test project build: Serving the specific business of the company, in order to write test cases for the business more quickly.
-
Automated Test Framework Development: A generic capability developed and designed to solve a generic class of problems, thus defining the methodology and structure for solving the problem.
Why design an automated testing framework
The reasons for developing a framework can be viewed from a variety of perspectives, and the following are some of the more common ones.
Improve development efficiency
The existing framework may have the following problems:
- Existing frameworks have too many third-party dependencies during use, and are cumbersome to install and configure.
- Complex to use and takes too much time for test engineers to learn or adapt.
By developing a more tailored automated testing framework, you can reduce repetitive labor and improve development efficiency.
Meeting specific types of needs
Existing frameworks exist that cannot meet the testing needs of specific types of business (e.g., test case writing for gRPC, Kafka), and more functional extensions and encapsulations need to be done on the basis of existing frameworks.
Better address the automated testing needs of existing business types by developing more powerful frameworks.
Provide better design concepts
Provide better design concepts or innovative technical realizations.
- Innovative design: delivering higher performance, more scalable solutions based on new architectures or design patterns.
- Domain-driven: focuses on best practices in a specific domain (e.g., keyword-driven, data-driven, method chaining).
Enhance teamwork
Standardize team collaboration to improve the development experience and code consistency.
- Frameworks can standardize the way teams develop and reduce the cost of collaboration due to individual differences.
- Provide standardized toolchain, modules and processes to ensure code quality and consistency for the team.
Directions for automated test framework design
When we decide to go for designing automated testing framework then there can be two directions:Design from scratch respond in singingSecondary development based on unit testing framework。
Design from scratch
Designing an automated testing framework from scratch, e.g. for a programming language, unit testing frameworks generally need to be designed and integrated into the programming language as a base library.
Standardized test structure
Unit testing frameworks provide a unified and structured way for developers to organize and run tests in a consistent manner.
-
Life cycle approach:: Often frameworks define a standardized set of lifecycle hooks for resource management before and after the execution of each test method:
setup/beforeEach: Initialize the test environment (e.g., create test data, instantiate objects) before each test is run.teardown/afterEach: Clean up resources after each test run (e.g., close database connections, delete temporary files). -
Naming of test methods: Often with specific conventions, such as starting with
test_(Python's unittest) or the@Testannotations (JUnit for Java). This convention allows the framework to automatically discover test methods. -
independent:: Each test method should remain independent and not depend on other tests. Isolation between tests helps to locate problems faster.
assertion mechanism
Assertions are the core of unit testing and are used to verify that the code under test behaves as expected.
Role of Assertions.
- Ensure that the code logic is correct by validating inputs and outputs.
- If the assertion fails, the test terminates immediately and an error is reported.
Test Findings
Test discovery is an important feature of a unit testing framework that automatically finds tests that meet the specification.
The framework scans a specific module or folder to find methods that conform to the naming convention. For example, in Python, pytest automatically discovers methods that begin withtest_ method at the beginning. In Java's JUnit, the@Test Methods identified by annotations are recognized as test methods.
Test suites and test runners
Test suites and runners enable developers to organize and execute tests efficiently.
- Test Suite
A test suite is a collection of tests used to combine multiple test cases to run together. Conveniently manage a set of related tests in groups.
- Test Runner
The test runner is responsible for executing tests and collecting test results.
Test reports and feedback on results
Test reports are used to show the execution results of tests and help developers quickly understand the health of their code. Test results categorization
- Pass: The test completed normally and all assertions were successful.
- Fail: The test failed, an assertion failed.
- Error: An unanticipated exception was thrown during test execution.
- Skip/Ignore: The test was not executed due to certain conditions.
Extension: xUnit is considered to be the prototype and inspiration for unit testing frameworks for many mainstream programming languages. xUnit is an architectural pattern that was first introduced by Kent Beck and Erich Gamma in SUnit (Smalltalk's unit testing framework) and has become a standard for test framework design. Its design ideas and concepts are widely used in other languages, such as JUnit (Java), NUnit (.NET), pytest (Python), and so on.
Secondary development based on unit testing framework
Based on the secondary development of the unit testing framework , on the basis of the unit testing framework , more focused on extending a variety of testing capabilities .
Typically, unit testing frameworks already provide basic testing capabilities. In order to better support various types of tests, we can expand on this in order to meet different types of needs.
Based on the unit testing framework secondary development direction is more, depending on the positioning and goals based on the framework design. The following are common extensions.
data-driven
Data-driven is one of the most common features of automated testing, which can effectively reduce the writing of sample code and thus increase the efficiency of test case writing.
- data-driven decorator
Test cases (methods) can be driven by data-driven decorators, such as the Seldom framework's@data([]) Decorator.
- Data Driver Documentation
Reading different types of data files by fetch-driven files. For example, the Seldom framework's@data_file("./data/")Manage test data.
Customized test reports
Test reporting is a very important feature of the automated testing framework and we need to do some customization development for reporting.
- Personalized test style and content
For example, displaying company logos, people's names and roles, and so on.
- Generate different report types
Different modes of operation require different types of reports. For example, local execution requires reports in HTML format, while CI/CD or platformized execution requires test reports in XML, JSON format.
Scaffolding Tools
Integrated scaffolding tool for fast generation ofAutomated Test Project Templates。
The automated test project template is described at the beginning of the article and will not be elaborated here.
Integrated Messaging
Every company has its own communication tools, email, nail, enterprise micro, flybook, etc.. By calling the API of the relevant tools and realizing the messaging function, you can make the test run results faster to send to the relevant people.
Integration of various test libraries
Based on the positioning of the framework, it is possible to integrate different types of test libraries and secondary development of these libraries to make the use of the framework more efficient and uniform.
- Web UI Testing
If it is to realize the Web UI automation test, then you can integrate Selenium, Playwright and other test libraries, and the API of these libraries can be encapsulated twice.
- API Testing
If it is to support interface testing, you can integrate test libraries such as Requests, webSocket, gRPC, etc., and secondary encapsulate the APIs of these libraries.
- App UI Testing
If it is to realize the App UI automation test, then you can integrate Appium, uiautomator2 and other test libraries, and the API of these libraries for secondary packaging.
Basic guidelines for automated test framework design
Separate name and version management
We should treat the framework as a standalone project to be developed, maintained and upgraded.
Naming of the framework
First of all, you should have an independent name for the framework, either in the name of an animal or plant, for example, Python (Python) or Lettuce (Lettuce); you can also follow the framework of the original meaning of the name, for example, Robot Framework (automation framework) or unittest (unit testing); you can also be an acronym synthetic words, for example, pytest = Python + Test, Appium = Application + Selenium, etc. The key is simple and easy to remember. Python + Test, Appium = Application + Selenium, etc. The key is simple and memorable.
Version Number Management
Second, the framework should have its own version number, and it is recommended to use GNU-style version number naming.
Format:Master version number. Subversion number [. Revision number [. Compile version number]]
- Master version number: refactoring version.
- Sub-release number: Major feature improvements.
- Fixed version number: minor upgrade or bug fix.
- Compile version number: Generally, it is automatically generated by the compiler during the compilation process, we only define its format and do not control it artificially.
Stand-alone installation
Finally, the framework should provide a standalone installation; for example, Python uses thepipcommand to install it.
For open source projects, for example, it is necessary to create or package files to package the project in .whl format and submit it to the official repository.
Generalizable
As a framework, its orientation and goal must be to solve a generic class of problems and provide capabilities.
Example:data-driven、Automated emailing、Generating Random Numbers、data cache、command-line tool These are not related to specific company businesses and provide generalized capabilities.
Clear positioning and objectives
Automation testing frameworks are designed to solve a certain type of problem better. At the beginning of the design, we should have a clear goal and position.
Solving a class of problems from scratch
xUnit was groundbreaking in the field of unit testing frameworks. An introduction to xUnit is given in the previous section.
Solving a class of problems more simply
Flask is a lightweight Web framework written in Python , through which we can simply write a few lines of code to build a Web service .
Provide more powerful and rich functionality
Django is an open source web framework written in Python .
Django although the learning cost is high , but it provides ORM (Relational Object Mapping), Admin management system , template system , Cache system , form processing , session (session), internationalization , etc., these features are almost out-of-the-box , can be used to achieve a more complex system .
Finally, when you're designing an automated testing framework, think about what the design goals are. What problem is it designed to solve? Are there already better open source frameworks that you can use directly.
Recommended books
So, is there a book that makes it clearAutomation Test Framework Design ?
The answer is: the book Designing Automated Test Frameworks
This book is edited by Bugmaster, as the developer of the SeldomQA GitHub Thousand Stars open-source project in theAutomation Test Framework Design、Customized test report design、design patternas well asTestbed DevelopmentWe have deep technical accumulation and unique design concepts.
The book provides an easy-to-understand introduction to many of the design and packaging techniques used in SeldomQA-related projects. It also describes the entire process of designing and releasing an open source automated testing framework.
In addition, the book describes how to get through theAutomation Frameworkcap (a poem)Automated Test Platform, which is a unique technical solution that provides new design ideas for automated test platforms.
Finally, combined with the current hot AI technology, the author also introduces the exploration direction of AI in the field of automation.
