com.coditory.sherlock:sherlock-reactor

Sherlock Distributed Lock

Single purpose and small distributed locking library for JVM. Provides multiple implementations (over single abstraction) for distributed locking:

• mongo-synchronous - Uses mongodb (tested on v3.4) and its synchronous connector to manage locks
• mongo-reactive - Uses mongodb (tested on v3.4) and its reactive connector to manage locks
• in-memory-synchronous - Stores locks in memory. Designed for testing purposes.
• in-memory-reactive - Stores locks in memory and exposes reactive api. Designed for testing purposes.
• ...postgres implementation comes next

Before using the library read about main problems of distributed locking.

Basic usage

Synchronous usage

Add dependency to build.gradle:

dependencies {
  compile "com.coditory.sherlock:sherlock-mongo-sync:0.2.2"
}

Create synchronous lock:

// Get mongo locks collection
String database = "sherlock";
MongoClient mongoClient = MongoClients.create("mongodb://localhost:27017/" + database);
MongoCollection<Document> collection = mongoClient
    .getDatabase("sherlock")
    .getCollection("locks");
// Create sherlock
Sherlock sherlock = MongoSherlock.builder()
    .withLocksCollection(collection)
    .build();
// Create a lock
DistributedLock lock = sherlock.createLock("sample-lock");

Acquire a lock:

// Acquire a lock
if (lock.acquire()) {
  try {
    System.out.println("Lock granted!");
  } finally {
    lock.release();
  }
}

...or acquire a lock in a less verbose way:

lock.acquireAndExecute(() -> {
  System.out.println("Lock granted!");
});

Reactive usage

Add dependency to build.gradle:

dependencies {
  compile "com.coditory.sherlock:sherlock-mongo-reactive:0.2.2"
  compile "com.coditory.sherlock:sherlock-api-reactor:0.2.2"
  // for RxJava API use:
  // compile "com.coditory.sherlock:sherlock-api-rxjava:0.2.2"
}

Create synchronous lock:

// Get mongo locks collection
String database = "sherlock";
MongoClient mongoClient = MongoClients.create("mongodb://localhost:27017/" + database);
MongoCollection<Document> collection = mongoClient
    .getDatabase(database)
    .getCollection("locks");
// Create sherlock
ReactorSherlock sherlock = ReactiveMongoSherlock.builder()
    .withLocksCollection(collection)
    .build(ReactorSherlock::wrapReactiveSherlock);
// Create a lock with reactor api
ReactorDistributedLock lock = sherlock.createLock("sample-lock");

Acquire a lock:

// Acquire a lock
lock.acquire()
  .filter(LockResult::isLocked)
  .flatMap(result -> {
    System.out.println("Lock granted!");
    return lock.release();
  })
  .block();

...or shorter

lock.acquireAndExecute(() -> Mono.just("Lock granted!"))
  .block();

Lock duration

There are 3 methods for acquiring a lock:

• lock.acquire() - acquires lock for a default duration (5 minutes) after which lock is automatically released
• lock.acquire(Duration.ofMinutes(3)) - acquires a lock for a specific duration
• lock.acquireForever() - acquires a lock forever. Use it wisely.

Lock types

There are 3 types of locks:

• SingleEntrantDistributedLock - lock owner cannot acquire the same lock for the second time
• ReentrantDistributedLock - lock owner can acquire the same lock multiple times
• OverridingDistributedLock - lock that acquires and releases the lock with sudo rights

SingleEntrantDistributedLock and ReentrantDistributedLock handles the problem of releasing a lock by using an expiration mechanism.

OverridingDistributedLock was created for purely administrative tasks.

Acquiring a lock by different lock types

SingleEntrantDistributedLock

DistributedLock lock = sherlock.createLock("single-entrant");
Instance A                     | Instance B
assert lock.acquire() == true; |
                               | assert lock.lock() == false
assert lock.acquire() == false;|

ReentrantDistributedLock

DistributedLock lock = sherlock.createReentrantLock("reentrant");
Instance A                     | Instance B
assert lock.acquire() == true; |
                               | assert lock.lock() == false
assert lock.acquire() == true; |

OverridingDistributedLock

DistributedLock lock = sherlock.createOverridingLock("overriding");
Instance A                     | Instance B
assert lock.acquire() == true; |
                               | assert lock.lock() == true
assert lock.acquire() == true; |
assert lock.acquire() == true; |

Releasing a lock by different lock types

SingleEntrantDistributedLock and ReentrantDistributedLock

Instance A                     | Instance B
assert lock.acquire() == true; |
                               | assert lock.lock() == false
                               | assert lock.release() == false
assert lock.release() == true; |
assert lock.release() == false;|

OverridingDistributedLock

Instance A                     | Instance B
assert lock.acquire() == true; |
                               | assert lock.lock() == true
                               | assert lock.release() == true
assert lock.release() == false;|
assert lock.release() == false;|

Lock state in storage

In mongo distributed lock implementation, lock is stored as a document:

{
  "_id": "lock-id",
  "acquiredBy": "service-instance-id",
  "acquiredAt": { "$date": 1562502838189 },
  "expiresAt": { "$date": 1562503458189 }
}

Testability

For easy stubbing and mocking all exposed api uses interfaces (see: Sherlock, ReactorSherlock, DistributedLock, ReactorDistributedLock).

Exposed interfaces have already stubs or mocks prepared. See: SherlockStub, ReactorSherlockStub, DistributedLockMock and ReactorDistributedLockMock.

Sample usage in spock tests:

def "should release a lock after operation"() {
  given: "there is a released lock"
    DistributedLockMock lock = DistributedLockMock.alwaysReleasedLock()
  when: "single instance action is executed"
    boolean taskPerformed = singleInstanceAction(lock)
  then: "the task was performed"
    taskPerformed == true
  and: "lock was acquired and released"
    lock.wasAcquiredAndReleased == true
}

def "should not perform single instance action when lock is locked"() {
  given: "there is a lock acquired by other instance"
    DistributedLockMock lock = DistributedLockMock.alwaysAcquiredLock()
  when: "single instance action is executed"
    boolean taskPerformed = singleInstanceAction(lock)
  then: "action did not perform the task"
    taskPerformed == false
  and: "action failed acquiring the lock"
    lock.wasAcquireRejected == true
  and: "action did not release the lock"
    lock.wasReleaseInvoked == false
}

Lock based migration process

Distributed locks may be used for multiple purposes one of them is a one way database migration process:

// prepare the migration
SherlockMigrator migrator = new SherlockMigrator("db-migration", sherlock)
  .addChangeSet("add db index", () -> /* ... */)
  .addChangeSet("remove stale collection", () -> /* ... */)

// run the migrtion
migrator.migrate();

Migration rules:

• migrations must not be run in parallel (neither by one nor by multiple machines)
• migration change sets are applied in order
• migration change set must be run only once per all migrations
• migration process stops when first change set fails

Migration mechanism is in experimental phase. It is available only in via sherlock-sync API.

Problems of distributed locking

Distributed locking is not a trivial concept. Before using it know its limits.

How to ensure that a that was acquired lock will be release?

It is possible the an instance that acquired a lock may go down before releasing it. Example:

if (lock.acquire()) {
  try {
    System.out.println("Lock granted!");
    System.exit(); // ...or OOM or any other cause
  } finally {
    // this part is never reached
    lock.release();
  }
}

This problem is fixed by automatically releasing a lock after expiration time.

How to ensure that an operation did not exceed a lock duration?**

Because of stop-the-world (...or multiple other causes) an operation that required a lock may take longer to finish than the lock duration. Example:

if (lock.acquire()) {
  try {
    System.out.println("Lock granted!");
    System.gc(); // ...very long break that exceeded lock duration
    criticalAction(); // invoked by two instances
  } finally {
    lock.release();
  }
}

This problem is well described by Martin Kleppmann.

This library is not designed to solve it. Simply make the lock duration as long as possible and don't use it in a per request manner.

License

sherlock-distributed-lock is published under Apache License 2.0.