JPA optimistic lock to change the pessimistic lock encountered some problems and thinking

contexts

Took on an outsourced project that used JPA as the ORM.

There are multiple entities in the project with the @version field

OptimisticLocingFailureException is often reported when concurrency is high.

Principle knowledge

JPA's @version implements optimistic locking by tinkering with SQL statements

UPDATE table_name SET updated_column = new_value, version = new_version WHERE id = entity_id AND version = old_version

This "Compare And Set" operation must be placed in the database layer, which guarantees the atomicity of the "Compare And Set" (the atomicity of the update statement).

If this "Compare And Set" operation is placed in the application layer, there is no guarantee of atomicity, i.e., the version may be compared successfully, but by the time the update is actually performed, the version of the database has already been changed.

This is when an error modification occurs

demand (economics)

Troubleshooting this type of error to allow transactions to complete properly

Processing - Retry

Since it's an optimistic lock reporting an error, it's a modification conflict, so just retry it automatically

Case Code

before modification

@Service
public class ProductService {
       
     @Autowired
     private ProductRepository productRepository;

     @Transactional
     public void updateProductPrice(Long productId, Double newPrice) {
          Product product = (productId).orElseThrow(()->new RuntimeException("Product not found")
          (newPrice);
          (product);
     }   
}

modified

Add a method withRetry, which can be called for places where there is a need to guarantee the success of a modification (e.g. user actions on a UI page).

@Service
public class ProductService {
       
     @Autowired
     private ProductRepository productRepository;

     public void updateProductPriceWithRetry(Long productId, Double newPrice) {
         boolean updated = false;
          //Keep retrying until it works.
          while(!updated) {
               try {
                   updateProductPrice(productId, newPrice);
                   updated = true;
               } catch (OpitimisticLockingFailureException e) {
　("updateProductPrice lock error, retrying...")
               }
          } 
 }

     @Transactional
     public void updateProductPrice(Long productId, Double newPrice) {
          Product product = (productId).orElseThrow(()->new RuntimeException("Product not found")
          (newPrice);
          (product);
     }   
}

Problems with Dependent Optimistic Locks - High Concurrency Brings High Conflict

The above retry can solve the optimistic lock error and allow the business operation to complete normally. However, it increases the burden on the database.

Also optimistic locks have their own problems:

The business layer commits transactional changes directly to the database, allowing the optimistic locking mechanism to guarantee data consistency.

At this point, the higher the concurrency, the more conflicts for modifications, the more invalid commits, the more pressure on the database

Coping with High Conflict - Introducing Pessimistic Locks

The way to resolve high conflicts is to introduce pessimistic locks at the business layer.

Acquire locks before business operations.

On the one hand, it reduces the number of concurrent transactions committed to the database, and on the other hand, it also reduces the CPU overhead of the business tier (executing business code only after obtaining a lock)

@Service
public class ProductService {
       
     @Autowired
     private ProductRepository productRepository;

     
     public void someComplicateOperationWithLock(Object params) {
          
          //The business involves several object modifications that require acquiring a lock on that object
//key=class prefix+object id
          List<String> keys = (....);
          
          //RedisLockUtil for distributed locks, can be self-packaged (can be based on redisson implementation)
//The task code is executed only after the lock is acquired, and then the lock is released at the end of the task execution.
          (keys, retryTime, retryLockTimeout, ()->someComplicateOperation(params)}):
    
     }
  

     @Transactional
     public void someComplicateOperation(Object params) {
         .....
     }   
}

Pitfalls encountered

Normally after acquiring a lock, you need to reload the latest data so that modifications don't conflict. (The previous lock acquirer may have modified the data)

However, JPA has a persistence context with a layer of caching. If the object is fished out before the lock is acquired, re-fishing after the lock is acquired will still get the data in the cache, not the latest database data.

In this case, even with pessimistic locking, there will still be a conflict when the transaction commits.

Case in point:

@Service
public class ProductService {
       
     @Autowired
     private ProductRepository productRepository;

     
     public void someComplicateOperationWithLock(Object params) {
          //Query once before getting the lock, this time the query data will be cached in the persistence context.
          String productId= xxxx;
          Product product = (productId).orElseThrow(()->throw new RuntimeException("Product not found"));
          
          //The business involves several object modifications that require acquiring a lock on that object
//key=class prefix+object id
          List<String> keys = (....);
          
          //RedisLockUtil for distributed locks, can be self-packaged
//The task code is executed only after the lock is acquired, and then the lock is released at the end of the task execution.
          (keys, retryTime, retryLockTimeout, ()->someComplicateOperation(params)}):
    
     }
  

     @Transactional
     public void someComplicateOperation(Object params) {
         .....
         //Fetch the old data in the cache
         Product product = (productId).orElseThrow(()->throw new RuntimeException("Product not found"));
         ....
     }   
}

Response - REFRESH

Within pessimistic locking, the first time the ENTITY data is loaded, the REFRESH method is used to force the latest data to be fished from the DB.

@Service
public class ProductService {
       
     @Autowired
     private ProductRepository productRepository;

     
     public void someComplicateOperationWithLock(Object params) {
          //The lock was queried once before it was acquired, and this time the query data will be cached in the persistence context
          String productId = xxxx;
          Product product = (productId).orElseThrow(()->throw new RuntimeException("Product not found"));
          
          //The business involves several object modifications that require acquiring a lock on that object
//key=class prefix+object id
          List<String> keys = (....);
          
          //RedisLockUtil for distributed locks, can be self-packaged
//The task code is executed only after the lock is acquired, and then the lock is released at the end of the task execution.
          (keys, retryTime, retryLockTimeout, ()->someComplicateOperation(params)}):
    
     }
  

     @Transactional
     public void someComplicateOperation(Object params) {
         .....
         //Fetch the old data in the cache
         Product product = (productId).orElseThrow(()->throw new RuntimeException("Product not found"));
        //Use the refresh method to force the latest data to be fetched from the database and updated into the persistence context
        EntityManager entityManager = (EntityManager.class)
        product = (product);
         ....
     }   
}

summarize

This project uses a mixed approach of optimistic locking + pessimistic locking, using pessimistic locking to limit concurrent modifications and optimistic locking for the most basic consistency protection.

About Consistency Protection

For simple applications with low write concurrency, transactions + optimistic locks are sufficient

Add a @version field to the entity.
Business methods plus @Transactional

This way the code is simplest.

The introduction of a pessimistic locking mechanism should only be considered when write concurrency is high, or when it is inferred from the business that highly concurrent write operations are likely to occur.

(The more complex the code, the more likely it is to cause problems and the harder it is to maintain)