1

I have a DynamoDB table with entries that can have a status (waiting, error, running...). Only up to 25 entries can have running status.

My objective is to have an AWS Lambda function that checks if there are fewer than 25 entries with "running" status, and if so, retrieve the next N entries that have value "waiting" and set them to "running".

My problem is that my Lambda function can run simultaneously with other AWS lambdas, and I guess I need some kind of locking while reading/writing to the table. I've found this library to lock the table while working on it https://python-dynamodb-lock.readthedocs.io/en/latest/index.html, but:

  1. I don't know if it locks on reading operations too
  2. It seems a dead library (last update was last year)

Do you know if this library does locking on reading too? or alternative methods? (I'd be ok by moving from DynamoDB to another kind of system)

4
  • 2
    "I've found this library to lock the table" You haven't linked to whatever library this is you are referring to, but you go on to ask questions about that library like we would know what you are talking about. You should at least edit the question to add a link to the thing you are referring to. Commented Oct 27 at 17:19
  • 1
    If your concern is multiple concurrent executions of the same Lambda function, then note that you can set concurrency limits on individual Lambda functions. This feature allows you to throttle a given function if it reaches a maximum number of concurrent executions allowed. Commented Oct 27 at 17:32
  • Sorry Mark, I thought I gave it! Commented Oct 27 at 17:56
  • 1
    For a general discussion of distributed counting semaphores with DynamoDB, see here. Commented Oct 27 at 18:10

1 Answer 1

1

So at the end I wrote my own locker functions. The idea is to have a second dynamodb table which only contains one entry. Using the conditionExpression it will be only set a new entry if it doesn't exist any, or was already expired the same one (the id of the table is lock_name, which you should always send the same one from every lambda, not lock_id! lock_id controls within the same lambda to free the lock)

Then, you free the item by removing it by its id. You can try acquiring the lock with some while sleep for several times until it fails.

def acquire_lock(lock_name: str, lock_id: str) -> bool:
    """
    Try to acquire a distributed lock.

    Args:
        lock_name (str): Name of the lock (e.g., 'step-function-scheduler').
        lock_id (str): Unique identifier for this lock acquisition attempt.

    Returns:
        bool: True if lock was successfully acquired, False otherwise.
    """
    expires_at = datetime.utcnow() + timedelta(seconds=LOCK_TIMEOUT_SECONDS)
    
    try:
        lock_table.put_item(
            Item={
                'lock_name': lock_name,
                'lock_id': lock_id,
                'expires_at': int(expires_at.timestamp()),  # Unix timestamp for TTL
                'created_at': datetime.utcnow().isoformat()
            },
            ConditionExpression='attribute_not_exists(lock_name) OR expires_at < :now',
            ExpressionAttributeValues={
                ':now': int(datetime.utcnow().timestamp())
            }
        )
        logger.info(f"Successfully acquired lock: {lock_name} with id: {lock_id}")
        return True
    except dynamodb.meta.client.exceptions.ConditionalCheckFailedException:
        logger.info(f"Failed to acquire lock: {lock_name} (already held by another instance)")
        return False
    except Exception as e:
        logger.error(f"Error acquiring lock: {e}")
        return False


def release_lock(lock_name: str, lock_id: str) -> bool:
    """
    Release a distributed lock.

    Args:
        lock_name (str): Name of the lock to release.
        lock_id (str): The lock_id that was used to acquire the lock.

    Returns:
        bool: True if lock was successfully released, False otherwise.
    """
    try:
        lock_table.delete_item(
            Key={'lock_name': lock_name},
            ConditionExpression='lock_id = :lock_id',
            ExpressionAttributeValues={':lock_id': lock_id}
        )
        logger.info(f"Successfully released lock: {lock_name}")
        return True
    except dynamodb.meta.client.exceptions.ConditionalCheckFailedException:
        logger.warning(f"Lock {lock_name} was not held by this instance (lock_id: {lock_id})")
        return False
    except Exception as e:
        logger.error(f"Error releasing lock: {e}")
        return False
Sign up to request clarification or add additional context in comments.

1 Comment

Would be good to create a context manager to encapsulate the lock, with automatic release.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.