Whiteboard software writing speed is its most core function, registered StylusPlugin from the touch thread to take the touch point data and another UI thread to draw rendering is a more secure program, the specifics can be viewed by a small friend Dexie's2019-1-28-WPF-High-Performance-Pen - lindexi - Blogland ()
The above StylusPlugin program can improve the writing performance of about 16ms for 1 frame on the major product versions of large screens such as Futronic and Huaxin touch frames. In addition to skipping some processes to reduce latency, can we continue to optimize the writing performance? The answer is definitely yes!
In this paper we present the direction of a class of implementations of writing acceleration that reduces writing latency by predicting the next or even N points and drawing the handwriting in advance.
curve-fitting prediction
Writing predictions, the curve-fitting scheme is described here:
Take N points and fit them to a curve, figure out its curve equation, and then the next point can be entered in its X position to get Y -- using either the X method or the Y method as a baseline to fit a curve with X as a parameter.
Let's give it a try, here we are using an open source component, first install its Nuget package:
<PackageReference Include="" Version="5.0.0" />
Cite using ;We predict the Y coordinate value in terms of the X coordinate and have already written the function:
1 private static Point[] PredictPointsNormal(Point[] pointArray, int degree, int predictCount) 2 { 3 ("Input:" + string.Join(",", (i => $"({},{})"))); 4 var xList = (i => ).ToArray(); 5 var yList = (i => ).ToArray(); 6 var lastX = xList[ - 1]; 7 var lastX1 = xList[ - 2]; 8 var lastPointLength = lastX - lastX1; 9 double[] parameters = (xList, yList, degree); 10 var predictPoints = new Point[predictCount]; 11 for (int i = 0; i < predictCount; i++) 12 { 13 var currentX = lastX + (i + 1) * lastPointLength; 14 var currentY = 0d; 15 for (int j = 0; j < degree + 1; j++) 16 { 17 var parameterJ = parameters[j]; 18 for (int k = 0; k < j; k++) 19 { 20 parameterJ *= currentX; 21 } 22 23 currentY += parameterJ; 24 } 25 26 var newPoint = new Point(currentX, currentY); 27 predictPoints[i] = newPoint; 28 } 29 ("Output:" + string.Join(",", (i => $"({},{})"))); 30 return predictPoints; 31 }
Here double[] parameters = (xList, yList, degree), which means that the current polynomial parameter value parameters is computed from the X as well as the Y series data in terms of the degree degree (e.g., second order curve).
If the degree is of order 2, the y-value can be calculated:
y = parameters[0] + parameters[1]*x + parameters[2]*x^2
In the case of a two-team curve with an increasing Y-axis upwards, as shown below, the green points from left to right are the list of known points, and the yellow are the predicted points. Here we predict 4 points in turn:
This is the type of scenario where the predicted results are more normal.
Let's take a look at the parabolic scenario, where the Y coordinate values are decreasing in order along the X direction, i.e., the clockwise angle values are increasing, and the prediction points are as follows:
As you can see from the graph above, the predicted result for decreasing values in the Y direction is the other half of the parabola, with accelerating decreasing values on the other side of the Y-axis in the opposite direction. But our expectation is certainly not a parabola like this one, we expect it to go diagonally upward on a green trajectory.
Why does it bounce back quickly? The reason is the fitted 2nd order curve, so the formula goes.
This type of method, can only do one-dimensional prediction, that is, when encountering the X, Y direction value is unchanged or the value change is small, the curve change will be like a parabola to reach its top, and then it will quickly bounce back.
Calculated by fitting the curve formula, there will be some defects. This we are also able to optimize, the function above we are based on the X axis, the formula obtained in the X is the amount of change, the following change to the Y axis:
1 public static Point[] PredictPointsNormalY(Point[] pointArray, int degree, int predictCount) 2 { 3 ("Input:" + string.Join(",", (i => $"({},{})"))); 4 var xList = (i => ).ToArray(); 5 var yList = (i => ).ToArray(); 6 var lastY = yList[ - 1]; 7 var lastY1 = yList[ - 2]; 8 var lastPointLength = lastY - lastY1; 9 double[] parameters = (yList, xList, degree); 10 var predictPoints = new Point[predictCount]; 11 for (int i = 0; i < predictCount; i++) 12 { 13 var currentY = lastY + (i + 1) * lastPointLength; 14 var currentX = 0d; 15 for (int j = 0; j < degree + 1; j++) 16 { 17 var parameterJ = parameters[j]; 18 for (int k = 0; k < j; k++) 19 { 20 parameterJ *= currentY; 21 } 22 23 currentX += parameterJ; 24 } 25 26 var newPoint = new Point(currentX, currentY); 27 predictPoints[i] = newPoint; 28 } 29 ("Output:" + string.Join(",", (i => $"({},{})"))); 30 return predictPoints; 31 }
Here the second-order fitted curve becomes:
x = parameters[0] + parameters[1]*y + parameters[2]*y^2
Again, predict 4 points and see the results:
This projected trend would be in line with our expectations.
So the upper X-direction as well as the Y-direction are used as benchmarks to fit the two-team curve, and then output the predicted points, and we synthesize to take a more suitable point.
How to take it? We can see that the point we expect is the one with a small change in the change trend. That is, taking the angle A of the last two data points as the base, the vector angle B1 and B2 of the predicted point and the last data point, the point with the smaller clockwise angle change is the one we expect to output.
Curve Fitting Other Problems
Above we solved the problem of coordinate point scenarios, where the output point bounces back quickly during unidirectional fitting. When validating the writing, we also found a class of prediction misalignment problems:
As shown above, the yellow points are skewed upwards (the yellow points are straight lines and the angle does not match the original trend), the true expectation is that we want to follow the trend of the original angle decreasing, and the predicted points are in the direction of the angle being slightly skewed downwards:
The reason for the prediction failure here is that the X-direction values cannot be predicted properly. Because according to our curve fitting scheme above, this kind of parabolic scenario is based on the Y-axis, input Y to get the X-direction value, but according to the direction of the curve change, input a Y-axis change between the last two points, the X-value of the predicted point should be close to infinity, out of the range of the curve.
Here can be processed according to the amount of change in the angle of the curve point, look at the above chart between the point and the point angle is increased in clockwise order, so the predicted point should also continue to increase the angle clockwise. So you can output the point according to the last two points Point (n), Point (n-1) between the vector angle value, or then increase the last three points between the angle of the difference angleChange.
I used increasing the amount of angle change as shown in the output below:
Finally, let's validate the real writing prediction as per the scheme above. To output the writing traces, let's change the color, blue for real points and red for predicted points.
Predicting 1 point, the red color overlaps with the real writing curve:
Predicts 2 points, red is the first predicted point and pink is the second predicted point. The pink point is offset more:
The predicted effect here is verified on my development touchscreen device (Dell P2418HT, 1080p).
It should be noted that the writing prediction is strongly correlated with the screen reporting rate (frame rate). In general, the longer the interval between the two output points, the larger the distance between the two points, the larger the error of the predicted point, but the corresponding prediction distance becomes farther, i.e., the writing delay will be reduced greatly.
When I touch move this 10Dell touch screen, the average interval between inputs is 33.3ms, one input contains 2-3 points, and the average interval between points is 16.6ms. the average interval between inputs is 33.3ms, which means that it is the touch frame 60fps reporting point. In addition, the detailed touch frame reporting rate data and open WM_POINTER message to enhance the application layer of the touch input frame rate can be seen below:Whiteboard writing delay - touch screen reporting rate - Tang, Song, Yuan, Ming, Qing 2188 - Blogs ()
According to the above touch point data, the above writing prediction program predicts 1 point, and the writing delay on this Dell touch screen can be reduced by about 16ms.