Preface
I remember one night five years ago, the order refund interface of an e-commerce platform suddenly occurred. Because the banking system network jittered, the refund request failed continuously.
Originally, the technical team just wanted to "try again a few times with kindness", but the retry code written by the developer frantically called the bank's refund interface.82 times！
In the end, it will result in repeated refunds from the user account and the platform will lose more than one million yuan.
The boss asked at the review meeting: "Why can such a basic thing be so basic to try on the interface? "
Everyone was speechless because everyone thought they only had to add oneforLoop, sleep for a few more seconds and it's all done...
This article will talk to you about 7 common solutions to try again, and I hope it will be helpful to you.
1. Violent Reincarnation Method
Problem scenario
A user registered SMS sending interface written by an intern.
In a while loop, the third party's text message interface is repeatedly called to send text messages to the user.
The code is as follows:
public void sendSms(String phone) {
int retry = 0;
while (retry < 5) { // Brainless cycle
try {
(phone);
break;
} catch (Exception e) {
retry++;
(1000); // Fixed 1 second sleep
}
}
}Accident scene
One time, the SMS server had an overload problem, causing all requests to be delayed by 3 seconds.
The code for this brute-force loop isTens of thousands of retrys were initiated at the same time within 0.5 seconds, directly exploded the SMS platform, triggeringFuse ban, even normal requests were rejected.
lesson
- 💥 No delay interval adjustment: Fixed interval causes centralized outbreak of retry requests
- 💥 Ignore exception types: Non-temporary error (such as parameter error) also try again
- 🔑 Repair plan: Add a random retry interval and filter the non-retry exceptions
2 Spring Retry
Application scenarios
Spring Retry is suitable for small and medium-sized projects, and can quickly realize basic retry and circuit breaking through annotations (such as order status query interface).
The function of interface retry is implemented by declaring the @Retryable annotation.
Configuration example
@Retryable(
value = {}, // Retry timeout exception only
maxAttempts = 3,
backoff = @Backoff(delay = 1000, multiplier = 2) // 1 second → 2 seconds → 4 seconds
)
public boolean queryOrderStatus(String orderId) {
return ("/order/" + orderId);
}
@Recover // Bottom-back method
public boolean fallback() {
return false;
}Advantages
- Declarative annotation: Concise code, decoupled from business logic
- Exponential backoff: Automatically extend the retry interval
-
Fuse integration: Combined
@CircuitBreakerCan quickly block abnormal traffic
3 Resilience4j
Advanced scenes
For some large and medium-sized systems (such as payment core interfaces) that require custom backoff algorithms, circuit breakers and multi-layer protection, we can use Resilience4j.
The core code is as follows:
// 1. Retry the configuration: Exponential backoff + random jitter
RetryConfig retryConfig = ()
.maxAttempts(3)
.intervalFunction((
1000L, // Initial interval of 1 second
2.0, // Exponential multiple
0.3 // Random jitter coefficient
))
.retryOnException(e -> e instanceof TimeoutException)
.build();
// 2. Fuse configuration: Fuse when the error rate exceeds 50%.
CircuitBreakerConfig cbConfig = ()
.slidingWindow(10, 10, .COUNT_BASED)
.failureRateThreshold(50)
.build();
// Combination use
Retry retry = ("payment", retryConfig);
CircuitBreaker cb = ("payment", cbConfig);
// Execute business logic
Supplier<Boolean> supplier = () -> ();
Supplier<Boolean> decorated = (supplier)
.withRetry(retry)
.withCircuitBreaker(cb)
.decorate();Effect
After a major e-commerce company launches this plan, the payment interfaceTimeout rate drops by 60%, and the fuse trigger frequency decreases nearly90%
We have truly achieved "not fighting back when we fight, and not scolding."
4 MQ queue
Applicable scenarios
Asynchronous scenarios with high concurrency and delay allowable asynchronous scenarios (such as logistics state synchronization).
Implementation principle
- After the first request failed, deliver the message toDelay queue
- The queue retrys consumption based on the preset delay time (such as 5 seconds, 30 seconds, 1 minute).
- If the maximum number of retry is reached, transfer toDead letter queue(Manual processing)
The RocketMQ code snippet is as follows:
// Producer sends delay message
Message<String> message = new Message();
("Order Data");
(3); // RocketMQ preset 10-second delay level
(message);
// Consumers try again
@RocketMQMessageListener(topic = "DELAY_TOPIC")
public class DelayConsumer {
@Override
public void handleMessage(Message message) {
try {
syncLogistics(message);
} catch (Exception e) {
// Retry + 1 and resend to higher latency level
resendWithDelay(message, retryCount + 1);
}
}
}How to fail RocketMQ consumer consumption, retry will be automatically initiated.
5 Timed tasks
Applicable scenarios
For business scenarios where tasks that do not require real-time feedback and allow batch processing (such as file import), we can use timing tasks.
Take Quartz as an example here.
The specific code is as follows:
@Scheduled(cron = "0 0/5 * * * ?") // Execute every 5 minutes
public void retryFailedTasks() {
List<FailedTask> list = (5); // Check failed tasks
(task -> {
try {
retryTask(task);
();
} catch (Exception e) {
();
}
(task);
});
}6 Two-stage submission
Applicable scenarios
For scenarios that strictly ensure data consistency (such as fund transfer), we can use a two-stage submission mechanism.
Key implementation
- Phase 1: Record operation flow to the database (status is "In progress")
- Phase 2: Call the remote interface and update the flow state according to the results
- Timed compensation: Resubmit the "in progress" flow flow with timeout
The general code is as follows:
@Transactional
public void transfer(TransferRequest req) {
// 1. Record flow
(req, PENDING);
// 2. Call the bank interface
boolean success = (req);
// 3. Update flow status
((), success ? SUCCESS : FAILED);
// 4. Failed to asynchronous retry
if (!success) {
("TRANSFER_RETRY_QUEUE", req);
}
}7 Distributed locks
Application scenarios
For some business scenarios that are prone to repeated submissions (such as flash sale) in multi-service instances and multi-threaded environments, we can use distributed locks.
Here we take the distributed lock of Redis + Lua as an example.
The code is as follows:
public boolean retryWithLock(String key, int maxRetry) {
String lockKey = "api_retry_lock:" + key;
for (int i = 0; i < maxRetry; i++) {
// Try to acquire a distributed lock
if ((lockKey, "1", 30, )) {
try {
return callApi();
} finally {
(lockKey);
}
}
(1000 * (i + 1)); // Wait for the lock to be released
}
return false;
}Summarize
Trying again is like a fire extinguisher in the computer room - you never want to use it, but you must ensure that you can save your life at critical moments.
Which solution do we choose in our work?
Don’t just look at the technical trends, but the spear and shield of the business, and what kind of cooperation is needed.
Finally, I would like to give you a sentence:The secret to system stability is to always be respectful of retrying.
Finally, I would like to say (please pay attention, don't mess with me for free)
If this article is helpful or inspiring to you, please help me follow my official account of the same name: Su San Talks about Technology. Your support is my greatest motivation for persisting in writing.
Please ask for three consecutive one click: like, forward, and watch.
Follow the official account: [Su San Talks about Technology], reply in the official account: When you enter a large factory, you can get the 100,000-word interview book I have compiled for free. Many friends have obtained offers from many large factories through this book.