Wednesday, October 21, 2009

Optimistic Locking With @Version

The datastore does optimistic locking, so if you update an entity inside a transaction and another request commits an update to either that same entity or some other entity in the same entity group before you commit your transaction, you will get an exception.  This keeps your data from getting stomped on when there are concurrent requests.  However, in the web-app world where clients can and typically do maintain state across requests, there is a far more likely scenario in which your data can get stomped on.  Consider the following scenario involving a bug tracking system:

Time 1: User A brings up the details page for Bug 838.
Time 2: User B brings up the details page for Bug 838.
Time 3: User A assigns Bug 838 to Jim.
Time 4: User B assigns Bug 838 to Sally.

Even though User A and User B updated Bug 838 at different times, User A's update was effectively stomped on by User B.  Sure, the fact that Bug 838 was assigned to Jim at some point may be in the history for the Bug, we can't change the fact that User B made a decision to assign Bug 838 based on out-of-date information.  In a bug tracking system this might not be such a big deal, but if you're doing something like compensation planning you'd much rather have your users receive an exception when they make an update based on out-of-date information.  Fortunately this is easy to implement using JPA/JDO on top of App Engine.  Let's use a Person object with a 'salary' property as an example.

JPA:
@Entity
public class Person {
    @Id
    @GeneratedValue(strategy=GenerationType.IDENTITY)
    private Long id;

    private int salary;

    @Version
    private long version;

    // ...getters and setters
}

public void updateSalary(EntityManager em, Person p, int newSalary) {
    em.getTransaction().begin();
    try {
        p.setSalary(newSalary);
        p = em.merge(p);
        em.getTransaction().commit();
    } catch (RollbackException e) {
        if (e.getCause() instanceof OptimisticLockException) {
            handleVersionConflict(e.getCause(), p);
        } else {
            throw e;
        }
    } finally {
        if (em.getTransaction().isActive()) {
            em.getTransaction().rollback();
        }
    }
}

JDO:

@PersistenceCapable(identityType=IdentityType.APPLICATION)
@Version(strategy=VersionStrategy.VERSION_NUMBER)
public class Person {
    @PrimaryKey
    @Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY)
    private Long id;

    private int salary;


    // ...getters and setters
}

public void updateSalary(PersistenceManager pm, Person p, int newSalary) {
    pm.currentTransaction().begin();
    try {
        p.setSalary(newSalary);
        pm.makePersistent(p);
        pm.currentTransaction().commit();
    } catch (JDOOptimisticVerificationException e) {
        handleVersionConflict(e, p);
    } finally {        
        if (pm.currentTransaction().isActive()) {
            pm.currentTransaction().rollback();
        }
    }
}

If you declare a Version field on your model object, JDO/JPA will compare the value of that field on the instance you are updating with the value of that field in the datastore.  If the version numbers are equal the version number will be incremented and your model object will be persisted.   If the version numbers are not equal the appropriate exception will be thrown.  In the above examples I've caught this exception in the update code itself, but this is really just to illustrate what's going on.  In practice I would most likely let this exception propagate out of the update method and handle it at a higher level, perhaps even as part of a generic exception handler in my presentation code.

Note that there is a performance cost to using this feature.  The datastore does not support updates with predicates the way relational databases do ('update person where version = 3') so in order to perform the version comparison it needs to do an additional get() for the entity that corresponds to your model object.  You'll have to decide for yourself whether or not the cost of the additional fetch is worthwhile.

Now, if you've looked at both the JPA and JDO examples you may have noticed that JPA requires the declaration of an explicit field annotated with @Version while JDO accepts an @Version annotation on the class without an explicit field.  If you are using JDO and would like the version stored in an explicit field you can use a DataNucleus-specific extension to tell JDO which field to use.  You can see an example of this here.

Using JPA and JDO's built-in versioning checks is an easy way to help users of your app make decisions based on up-to-date information.  Try it out!

18 comments:

  1. Great tips! I still scratching my head on how to solve this kind of problem using memcache, reader writer lock...

    ReplyDelete
  2. this blog is a great initiative, thanks for the tips!

    ReplyDelete
  3. when Time
    3: User A assigns Bug 838 to Jim.

    Time 4: User B assigns Bug 838 to Sally.

    when user B do merge() there is no need to do explicit checking of version number already changed by userA before calling merge() ?

    ReplyDelete
  4. Sorry, I don't understand your question.

    ReplyDelete
  5. @Max, may i know how to test for optismic lock?

    I loaded same entity record on 2 separate browser window then press submit (hibernate template.merge), version number incremented for both browser window, but never caught any problem with optimistic lock..

    ReplyDelete
  6. You'll want to make sure that you're submitting the version number of the object when it was read along with the data you plan to update and then ensure that the object you're updating has the version number that was submitted.

    You ca also see a unit-test of this functionality here:
    http://code.google.com/p/datanucleus-appengine/source/browse/trunk/tests/org/datanucleus/store/appengine/JPAUpdateTest.java#131

    The key is that the version number in the datastore doesn't match the version number in the object being updated.

    ReplyDelete
  7. Thanks for your post, and many thanks for your work !

    But doing a second query to check the last version number is not suffisant to guarantee Optimistic Locking (OL). The time between the second query and the moment where the entity is really stored can be arbitrary long, so another write can happens during this time. I think the test is not working for testing OL with concurrent updates.

    So does GAE datastore has built in OL ?
    As far as I explored the API, it seems that it isn't. So if such feature does not come, we will unfortunately never be able to do real OL.

    ReplyDelete
  8. Ok GAE datastore has build in OL. It throws ConcurrentModificationException when an object is modified concurrently by another transaction.
    So the actual implementation of the @Version should work. Nice job !

    ReplyDelete
  9. That's right, the GAE datastore has built-in optimistic locking for txns. Just remember that the locking is done per entity-group, not per record.

    ReplyDelete
  10. Thanks for your post.

    I tried implementing @version to my app which uses GWT for a client. The problem is I can not seem to make the locking work.

    SCENARIO:

    From list, the user selects a object for update. Update event passes the updated object to a service method (which updates data).

    The method doesn't throw an exception and it allows update.

    As a workaround, I am explicitly comparing the version currently at hand and the one currently stored.

    What is the best way to implement locking (when using GWT). Thanks a lot?

    ReplyDelete
  11. If the version of the object you're updating doesn't match the version in the datastore you should get an exception. Can you boil the problem down to a simple unit test with your model object? You fetch your object, then update the version directly in the datastore using the low-level api, then try to update your object.

    ReplyDelete
  12. This comment has been removed by a blog administrator.

    ReplyDelete
  13. This comment has been removed by a blog administrator.

    ReplyDelete
  14. This comment has been removed by a blog administrator.

    ReplyDelete
  15. Hi Max

    Your blog is a real treasure when starting to program with the gae datastore.

    I could not set up a unit test where I get an exception when testing the @version using JDO and GAE SDK 1.3.2

    I have the following scenario:
    I persisted person it with salary 10
    read it into result1
    read it into result2
    re-persisted result1 with 20
    re-persisted result2 with 30
    I expected the 2nd re-persist to fail but it did not and the final result when rereading it was 30.

    1) Where do I make wrong assumptions?
    2) Does anyone have a simple JDO unit test for the @Version?
    3) I thought I could implement uniqueness through this version handling, I would like to make sure that a person can only be created once by using a explicit primary key (not IdGeneratorStrategy.IDENTITY) and versions. Is that possible? I could not test that yet since I was not able to set up any unit test for the @version mechanism. :'(

    Thanks a lot,
    Claus

    ReplyDelete
  16. I just made a test using "explicit" versions...

    @PersistenceCapable(identityType = IdentityType.APPLICATION)
    @Version(strategy = VersionStrategy.VERSION_NUMBER, extensions = {@Extension(vendorName = "datanucleus", key = "field-name", value = "versionField")})

    the versionField (Long) is set to 1 when persiting a new object but never changes in my scenario (she post before).

    Even setting this field explicitly to 5 does not throw an exception.

    What do I miss?

    ReplyDelete
  17. @Version is not a good mechanism for enforcing uniqueness, it only detects changes in long-running transactions. If you want to make sure a person can only be created once then using a named Key, where you provide the name, is the correct approach. You'll need to perform a transactional read before you attempt to create the person, but this is pretty straightforward:

    1) beginTxn();
    2) Person p = lookupPersonByName("x");
    3) if (p != null) {
    4) throw new PersonAlreadyExists("x");
    5) }
    6) createNewPerson(p);
    7) commitTxn();

    if a person with uniquely identified by "x" gets created by a different thread in between lines 2 and 7 you'll receive an exception when you attempt to commit due to a concurrent modification of the entity group.

    For @Version, here's a test that you may find useful:
    http://code.google.com/p/datanucleus-appengine/source/browse/trunk/tests/org/datanucleus/store/appengine/JDOUpdateTest.java#111

    Hope this helps,
    Max

    ReplyDelete
  18. Hell Max,

    Yes, this will help ... I just could not imagine that there is such a thing as read-locks in the datastore that is why I did not investigate further on the transactions.

    Thanks for the test ... I will give it a try but I was told a well that I can NOT relay on the functionality of the local version.

    Thanks a lot,
    Claus

    ReplyDelete