Author: from vivo internet front-end team- Wei Xing
In the process of research and development projects, we often encounter the need for iterative updates to the technical architecture, through iterative updates to the technology, so that the project from the new technical features to benefit from, but because many new iterations of technology iteration version is not fully backward compatible, contains a lot of non-compatible changes (Breaking Changes), so we need to design a tool to help us complete the large-scale code Automatic Code Migration (ACM). This paper briefly describes the concept and general process of AST-based code migration, and brings you to the details of the process through code examples.
I. Background
In the process of R&D projects, we often encounter the need to iteratively update the technical architecture, so that the project can benefit from the new technical features through iterative updating. For example, migrating Vue 2 to Vue 3, upgrading Webpack 4 to Webpack 5, migrating build tools to Vite, etc. These upgrades to the technical architecture can continue to benefit the project, such as maintainability, performance, scalability, compilation speed, readability and other aspects of the enhancement of the project, timely technical architecture updates to the project is very necessary.
So since the new features are so good, some would say then of course we should keep up with the times and update them all the time.
However, the problem is that many new technology iterations are not fully backward compatible, and contain many non-compatible changes (Breaking Changes), which are not simply a matter of upgrading the version, but usually require a lot of labor and learning costs. For example, Vue 3 is only compatible with 80% of Vue 2 code, and for some new features and syntax, developers can only refer to the official migration documentation to complete the migration manually.
(Image source:freecodecamp）
1.1 Vue 3 Code Migration Example
Take a look at a Vue 3 code migration example of the difference between declaring a global directive (Directive) in Vue 2 and Vue 3:
(1) Vue 2: Allowed global directives to be registered directly on the Vue prototype. In Vue 3, this is no longer supported to avoid directive confusion across multiple Vue instances.
import Vue from 'vue'
('focus', {
inserted: (el) => ()
})
(2) Vue 3: It is recommended to create a Vue instance via createApp and register global directives directly on the instance. Like this.：
import { createApp } from 'vue'
const app = createApp({})
('focus', {
inserted: (el) => ()
})
The above is a familiar Vue 3 migration example, which seems simple enough to move a few lines of code. However, when we have a large enough project, or a large number of projects that require similar code migration, the workload can become huge, and it is difficult to avoid the risks associated with manual migration.
Therefore, generally for large-scale project migrations, the best way is still to write a scaffolding tool to assist us in automating the migration. It can both improve efficiency and reduce the risk of manual migration.
1.2 Code migration background for this paper
Similarly, I've encountered the same technical architecture upgrade problem in my projects. Simply put, I need to migrate a Vue 2 based project to an internal technology stack that is similar to Vue 2, but due to the underlying technology, there are some syntactic differences that need to be manually migrated for compatibility (similar to the process of upgrading from Vue 2 to Vue 3).
In addition to migrating the JavaScript and Template templates as I did with Vue 3, I also had to deal with CSS, Less, SCSS, and other style files separately.
So, I implemented an automated migration scaffolding tool to assist in completing the migration of code and reduce the inefficiencies and risk issues associated with manual migration.
II. Code migration ideas
I just mentioned that we need to design a scaffolding to help us with automated code migration, so how should the scaffolding be designed?
First of all, the code migration idea can be simply summarized as follows: do a static code analysis of the original code and replace it with the new code according to certain rules. Then the most intuitive way is to use regular expressions to match and replace the code, so I also made such an attempt.
2.1 Idea 1: Use regular expressions to match rules and replace code
For example, put the following code:
import { toast } from '@vivo/v-jsbridge'
import { toast } from '@webf/webf-vue-render'
This looks simple and seems to be done with regular matching, like this:
const regx = /\@vivo\/v\-jsbridge/gi
const target = '@webf/webf-vue-render'
(regx, target)
However, in practice, it turns out that regular expressions are just too limited, and there are a few core problems:
-
Regular expressions are based entirely on string matching and require a high degree of uniformity in the formatting of the original code. Formatting differences such as spaces, line breaks, single and double quotes, etc. can all cause regular matching errors;
-
Faced with complex matching scenarios, regular expressions are hard to write, obscure, and prone to mis-matching and mishandling;
-
When processing style files, you need to be compatible with CSS / Less / SCSS / Sass syntax differences, which multiplies the workload.
As a simple example, when I need to match import { toast } from '@vivo/v-jsbridge' string. You need to be more careful with single and double quotes, spaces, semicolons, and other details, and if you're not careful, you'll miss some special scenarios, and the result will be that the match fails, causing hidden migration problems.
import { toast } from '@vivo/v-jsbridge' // single quote
import { toast } from "@vivo/v-jsbridge" // double quote
import { toast } from "@vivo/v-jsbridge"; // double quote + semicolons
import {toast} from "@vivo/v-jsbridge"; // no spaceSo, using simple regular matching rules won't help us with large-scale code migration and refactoring, we need a better approach: AST-based code migration.
2.2 Idea 2: AST (Abstract Syntax Tree) Based Code Migration
Having learned the limitations of regular matching rules, I set my sights on AST-based code migration.
So what is AST-based code migration?
2.2.1 Babel compilation process
If you know how Babel's code is compiled, you should be familiar with AST code migration. We know that Babel's compilation process is roughly divided into three steps:
-
Parsing: Parses the source code into an AST (Abstract Syntax Tree);
-
Transform: Transforms the AST;
-
Rebuild: generates new code based on reconstruction of the transformed AST.
(Image source:Luminosity Blog ）
For example, Babel converts an ES6 syntax to ES5 syntax as follows:
(1) Enter the source code for a simple sayHello arrow function method:
const sayHello = () => {
('hello')
}
(2) Parsed by Babel into an AST (you can see that the AST is a string of syntax trees described by JSON), and the AST is transformed by rules:
-
Converts the type field from ArrowFunctionExpression to FunctionExpression.
-
Convert the kind field from const to var
{
"type": "Program",
"start": 0,
"end": 228,
"body": [
{
"type": "VariableDeclaration",
"start": 179,
"end": 227,
"declarations": [
{
"type": "VariableDeclarator",
"start": 185,
"end": 227,
"id": {
"type": "Identifier",
"start": 185,
"end": 193,
"name": "sayHello"
},
"init": {
- "type": "ArrowFunctionExpression",
+ "type": "FunctionExpression",
"start": 196,
"end": 227,
- "id": null,
+ "id": {
+ "type": "Identifier",
+ "start": 203,
+ "end": 211,
+ "name": "sayHello"
+ },
"expression": false,
"generator": false,
"async": false,
"params": [],
"body": {
"type": "BlockStatement",
"start": 202,
"end": 227,
"body": [
{
"type": "ExpressionStatement",
"start": 205,
"end": 225,
"expression": {
"type": "CallExpression",
"start": 205,
"end": 225,
"callee": {
"type": "MemberExpression",
"start": 205,
"end": 216,
"object": {
"type": "Identifier",
"start": 205,
"end": 212,
"name": "console"
},
"property": {
"type": "Identifier",
"start": 213,
"end": 216,
"name": "log"
},
"computed": false,
"optional": false
},
"arguments": [
{
"type": "Literal",
"start": 217,
"end": 224,
"value": "hello",
"raw": "'hello'"
}
],
"optional": false
}
}
]
}
}
}
],
- "kind": "const"
+ "kind": "var"
}
],
"sourceType": "module"
}
(3) Refactoring from AST to ES5 syntax:
var sayHello = function sayHello() {
('hello');
};This completes a simple ES6 to ES5 syntax conversion. Our scaffolding automated code migration idea is similar.
There are several benefits to migrating based on AST code as opposed to regular expression matching:
-
More flexible and covers more complex scenarios than string matching.
-
Usually AST code migration tools provide convenient APIs for parsing, querying, matching, and replacing, making it easy to write efficient code conversion rules.
-
Facilitates harmonization of converted code styles.
2.2.2 Code Migration Process Design
After understanding the basic principles and feasibility of AST, we need to find the right tool library to accomplish AST parsing, refactoring, and generation of code. Consider that the project contains at least these kinds of content (script, style, HTML):
-
A separate JS file;
-
Separate style files: CSS / Less / SCSS / Sass;
-
Vue file: contains Template, Script, Style three parts.
We need to find the parsing and processing tools corresponding to each type of file content separately.
First, there is the choice of tools for parsing and processing JS files.The most popular JS AST tools on the market are a variety of choices. There are many choices of popular JS AST tools on the market, such as the most common Babel, jscodeshift, and Esprima, Recast, Acorn, estraverse, and so on. After doing some simple research, I found that these tools have some common flaws:
-
It is difficult to get started, has a large learning cost, and requires the developer to fully understand the syntax specification of the AST;
-
Complex syntax and large amount of code;
-
The code is poorly readable and not conducive to maintenance.
Using jscodeshift as an example, if we need to match a simple statement: ('price')(this, '50'), it is implemented in the following code:
const callExpressions = (, {
callee: {
callee: {
object: {
name: 'item'
},
property: {
name: 'update'
}
},
arguments: [{
value: 'price'
}]
},
arguments: [{
type: 'ThisExpression'
}, {
value: '50'
}]
})
In fact, compared to the original Babel syntax, the above jscodeshift syntax is already relatively concise, but it can be seen that there is still a large amount of code, and requires the developer to be proficient in the syntactic structure of the AST.
So I found a cleaner, more efficient AST tool:GoGoCodeGoGoCode is an open source AST tool that encapsulates a jQuery-like syntax and is easy to use. An intuitive comparison is that if you use GoGoCode to achieve the same statement matching, only one line of code:
$(code).find(`('price')(this, '50')`)Its intuitive semantics and clean code made me choose it as the AST parser for JS.
Second, there is a separate selection of CSS style file parsing tools. This is an easy choice, just use the generic PostCSS to parse and process the styles directly.
Finally, there's the Vue file parsing tool of choiceBecause Vue files are made up of Template, Script, and Style, they require more sophisticated tools. Since Vue files are composed of Template, Script, and Style, they require more sophisticated tools to process them together. Luckily, GoGoCode is able to parse and process individual JS files, but it also encapsulates the ability to process the Template and Script parts of a Vue file, so in addition to the Style part of a Vue file, we can also leave it to GoGo Code to process. So how do we handle the Style part of a Vue file? I looked at the official vue-loader source code and found that it uses @vue/component-compiler-utils to parse Vue's SFC files, and it can extract the style content from the files separately. So the idea is simple, we use @vue/component-compiler-utils to extract the style content from the Vue file, and let PostCSS handle it.
So, a brief summary of a few suitable tool libraries found:
-
GoGoCode: Ali open source an abstract syntax tree processing tool that can be used to parse JS/HTML/Vue files and generate an abstract syntax tree (AST), the rules of code replacement , refactoring and so on. Encapsulated similar to the jQuery syntax , easy to use.
-
PostCSS: we are familiar with the open source CSS code migration tool , can be used to parse Less / CSS / SCSS / Sass and other style files and generate syntax tree (AST), the rules of code replacement , refactoring and so on.
-
@vue/component-compiler-utils: Vue's open source tool library can be used to parse Vue's SFC file, I use it to pull out the Style content in the SFC separately, and with PostCSS to deal with the rules of the style code replacement, refactoring.
With these three tools, we can sort out the processing ideas for different file contents:
-
JS files: Handled by GoGoCode.
-
CSS / Less / SCSS / Sass files: leave it to PostCSS.
-
Vue Documentation.
-
Template / Script part: leave it to GoGoCode.
-
Style section: the Style section is parsed with @vue/component-compiler-utils and then passed to PostCSS.
Once you have the ideas for handling it, move on to the main text and dive into the code details for a detailed look at the code migration process.
Third, the code migration process in detail
The entire code migration process is divided into several steps, which are:
3.1 Traversing and reading the contents of a file
Iterates through the contents of the project file, handing it off to different transform functions depending on the file type:
-
transformVue: Processing Vue Files
-
transformScript: Processing JS files
-
transformStyle: Handles CSS and other style files.
const path = require('path')
const fs = require('fs')
const transformFiles = path => {
const traverse = path => {
try {
(path, (err, files) => {
(file => {
const filePath = `${path}/${file}`
(filePath, async function (err, stats) {
if (err) {
((` \n🚀 ~ ${o} Transform File Error:${err}`))
} else {
// If it's a file then start executing the replacement rules
if (()) {
const language = ('.').pop()
if (language === 'vue') {
// deal withvueContents of the document
await transformVue(file, filePath, language)
} else if ((language)) {
// deal withJSContents of the document
await transformScript(file, filePath, language)
} else if ((language)) {
// deal with样式Contents of the document
await transformStyle(file, filePath, language)
}
} else {
// If it's a catalog，then continue traversing the
traverse(`${path}/${file}`)
}
}
})
})
})
} catch (err) {
(err)
reject(err)
}
}
traverse(path)
}3.2 Code Migration for Vue Files
Since the process of handling separate JS and style files is similar to that of Vue files, the only difference is that Vue files have an extra layer of parsing. So here we take Vue files as an example to illustrate the specific code migration process:
const $ = require('gogocode')
const path = require('path')
const fs = require('fs')
// deal withvuefile
const transformVue = async (file, path, language = 'vue') => {
return new Promise((resolve, reject) => {
(path, function read(err, code) {
const sourceCode = ()
// 1. utilizationgogocodeoffered $ methodologies，Convert the source code toastsyntax tree
const ast = $(sourceCode, { parseOptions: { language: 'vue' } })
// 2. deal withscript
transformScript(ast)
// 3. deal withtemplate
transformTemplate(ast)
// 4. deal withstyles
transformStyles(ast)
// 5. 对deal with过的astRe-generate the code
const outputCode = ().generate()
// 6. 重新写入file
(path, outputCode, function (err) {
if (err) {
reject(err)
throw err
}
resolve()
})
})
})
}
As you can see, the main Vue file processing flow in the code is as follows:
-
Generate AST grammar tree: Use the $ method provided by GoGoCode to convert the source code to Ast grammar tree, and then pass the Ast grammar tree to different processors to complete the syntax matching and conversion.
-
Processing JavaScript: Call transformScript to process the JavaScript part.
-
Processing template: Call transformTemplate to process the template (HTML) part.
-
Processing Styles: Call transformStyles to process style sections.
-
Regenerate code for processed Ast: Call the ().generate() method provided by GoGoCode to regenerate the target code from Ast.
-
Rewrite file: rewrites the generated object code to a file.
This concludes the code migration for a Vue file. Next, let's see what needs to be done for the different content JavaScript, HTML, and Style.
3.3 Handling JavaScript Scripts
When dealing with JavaScript scripts, you mainly rely on some syntax provided by GoGoCode for code migration:
-
First, use ('') to parse and find the JavaScript script portion of the Vue file.
-
Second, the original code is matched and replaced using methods such as replace.
-
Finally, the processed Ast is returned.
Here's a simple example of code replacement, the main purpose of which is to replace an import statement with a reference to the source of the package. Replace @vivo/v-jsbridge with @webf/webf-vue-render/modules/jsb.
// pre-conversion
import { call } from '@vivo/v-jsbridge'
// post-conversion
import { call } from '@webf/webf-vue-render/modules/jsb'
const transformScript = (ast) => {
const script = ('<script></script>')
// utilizationreplaceMethod Replacement Code
(
`import {$$$} from "@vivo/v-jsbridge"`,
`import {$$$} from "@webf/webf-vue-render/modules/jsb"`
)
return ast
}
In addition to replace, there are other syntaxes that are often used, such as:
-
find (find code)
-
has (determines if it contains code)
-
append (insert code after)
-
prepend (insert code before)
-
remove, etc.
With a simple familiarity with GoGoCode's syntax, you can write some conversion rules very quickly (and much more efficiently than regular expressions).
3.4 Processing Templates
Handling Template templates is similar and relies heavily on the API provided by GoGoCode:
-
First, use ('') to parse and find the Template (HTML) section of the Vue file.
-
Second, the original code is matched and replaced using methods such as replace.
-
Finally, the processed Ast is returned.
The following is a simple Template tag replacement example, replacing a div tag with the @change attribute with a span tag with the :onChange attribute.
// Before the change
<div @change="onChange"></div>
// After change
<span :onChange="onChange"></div> // after conversion
const transformTemplate = (ast) => {
const template = ('<template></template>')
const tagName = 'div'
const targetTagName = 'span'
// utilizationreplaceMethod Replacement Code，commander-in-chief (military)divThe tag is replaced withspantab (of a window) (computing)
template
.replace(
`<${tagName} @change="$_$" $$$1>$$$0</${tagName}>`,
`<${targetTagName} :onChange="$_$" $$$1>$$$0</${targetTagName}>`
)
return ast
}
It's worth noting that GoGoCode provides the $_$, $$$$1 Wildcards like these allow us to make better rule matches to DOM structures and write matching and transformation rules efficiently.
3.5 Handling Style
The last part is to deal with Style, which is a bit different from dealing with JavaScript and Templates, because GoGoCode doesn't provide a way to deal with Style for the time being. So we need to borrow two extra tools, they are:
-
@vue/component-compiler-utils: parses style code to Ast.
-
PostCSS: process the style according to the rules, converted to the target code.
The entire Style flow is as follows (those of you who have written the PostCSS plugin should be familiar with this part of Style processing):
-
Get Styles node: With Get Styles node, a Vue file may contain multiple Style blocks corresponding to multiple Styles nodes.
-
Iterate over Styles nodes:
-
Parses the Style node content using the compileStyle method provided by @vue/component-compiler-utils.
-
Using the method, the style content is processed according to the rules and the target code is generated.
-
Returns the converted Ast.
Here's a simple example of replacing all the color attribute values in a style with red.
// Before conversion
<style>
.button {
color: blue.
}
</style>
// After conversion
<style>
.button {
color: red.
}
</style>
const compiler = require('@vue/component-compiler-utils')
const { parse, compileStyle } = compiler
const postcss = require('postcss')
// A simple replacement for allcolorattribute'red'(used form a nominal expression)postcssplug-in (software component)
const colorPlugin = (opts = {}) => {
return {
postcssPlugin: 'postcss-color',
Once(root, { result }) {
(node => {
// Find allpropbecause ofcolor(used form a nominal expression)nodal，将nodal(used form a nominal expression)值设置because ofred
if ( === 'color') {
= 'red'
}
})
}
}
}
= true
const transformStyles = (ast) => {
// gainstylesnodal（anvueThe file may contain multiplestylecode block，Corresponding to multiplestylesnodal）
const styles =
// Iterate over allstylesnodal
((style, index) => {
let content =
// gain文件(used form a nominal expression)后缀：less / sass / scss / csset al. (and other authors)
const lang = || 'css'
// utilization@vue/component-compiler-utils提供(used form a nominal expression)compileStylemethodological analysisstyleelement
const result = compileStyle({
source: content,
scoped: false
})
// hand over topostcssHandling styles，传入刚刚声明(used form a nominal expression)colorPluginplug-in (software component)
const res = postcss([colorPlugin]).process(, { from: path, syntax: parsers[lang] })
=
})
return ast
}
At this point, the entire code migration process is complete.
[Source code DEMO available/vivo/BlueSnippets/tree/main/demos/ast-migration
IV. Summary
This article briefly describes the concept of AST-based code migration and the general process, and through the code cases to bring you to understand the details of the processing. The whole code migration process is organized:
-
Traverses and reads the contents of the file.
-
Categorize the content and use different processors for different file contents.
-
JavaScript scripts are handled directly by GoGoCode.
-
Style files are handled directly by PostCSS.
-
For Vue files, we parse them and split them into Template, JavaScript, and Style, and then process them separately.
-
After processing, generate target code based on the converted Ast and rewrite the file to complete the code migration.
The whole process is relatively simple, only need to master the basic concepts of AST code conversion, GoGoCode, PostCSS, @vue/component-compiler-utils the basic use of these tools have a certain understanding of the development of an automated migration tool.
Lastly, we need to remind you that when designing code matching and conversion rules, you need to pay attention to the boundary scenarios to avoid erroneous code conversion, which may cause potential bugs. in order to avoid code conversion anomalies, it is recommended that you write sufficient test cases for each conversion rule to ensure the correctness of the conversion rules.
If you have similar needs, you can also refer to this article for tool design and implementation.