view roundup/backends/portalocker.py @ 6565:2c2dbfc332ba

Try to handle multiple connections better. The session database is a hot spot. When multiple requests (e.g. 20) come in at the same time session database contention can get great. The original code didn't retry session database access when the open failed. This resulted in errors at the client. The second pass delayed 0.01 seconds and retried. It was better but we still had multiple second stalls. I think the first request got in, everybody else backed up and then retried at the same time. Again they stepped on each other. With logging I would see many counters go all the way to low single digits or to -1 indicating falure. This pass uses randomint to generate delays from 0-.125 seconds in 5ms increments. This performs better in testing. I rarely saw a counter less than 13 (2 failed retries). Current logging starts after 6 failures and counts down until success or failure.
author John Rouillard <rouilj@ieee.org>
date Thu, 16 Dec 2021 20:02:00 -0500
parents a50712b6ad56
children 9c3ec0a5c7fc
line wrap: on
line source

#!/usr/bin/env python
#
# portalocker.py - Cross-platform (posix/nt) API for flock-style file locking.
#
# http://code.activestate.com/recipes/65203-portalocker-cross-platform-posixnt-api-for-flock-s/
#
"""
Cross-platform (posix/nt) API for flock-style file locking.

Synopsis::

   import portalocker
   file = open("somefile", "r+")
   portalocker.lock(file, portalocker.LOCK_EX)
   file.seek(12)
   file.write("foo")
   file.close()

If you know what you're doing, you may choose to::

   portalocker.unlock(file)

before closing the file, but why?

Methods::

   lock( file, flags )
   unlock( file )

Constants::

   LOCK_EX
   LOCK_SH
   LOCK_NB

I learned the win32 technique for locking files from sample code
provided by John Nielsen <nielsenjf@my-deja.com> in the documentation
that accompanies the win32 modules.

:Author: Jonathan Feinberg <jdf@pobox.com>

Roundup Changes
---------------
2012-11-28 (anatoly techtonik)
   - Ported to ctypes
   - Dropped support for Win95, Win98 and WinME
   - Added return result
"""

from __future__ import print_function
__docformat__ = 'restructuredtext'

import os

if os.name == 'nt':
    import msvcrt
    import ctypes
    from ctypes import windll
    from ctypes.wintypes import BOOL, DWORD, HANDLE

    LOCK_SH = 0    # the default
    LOCK_NB = 0x1  # LOCKFILE_FAIL_IMMEDIATELY
    LOCK_EX = 0x2  # LOCKFILE_EXCLUSIVE_LOCK

    # --- the code is taken from pyserial project ---
    #
    # detect size of ULONG_PTR
    def is_64bit():
        return ctypes.sizeof(ctypes.c_ulong) != ctypes.sizeof(ctypes.c_void_p)
    if is_64bit():
        ULONG_PTR = ctypes.c_int64
    else:
        ULONG_PTR = ctypes.c_ulong
    PVOID = ctypes.c_void_p

    # --- Union inside Structure by stackoverflow:3480240 ---
    class _OFFSET(ctypes.Structure):
        _fields_ = [
            ('Offset', DWORD),
            ('OffsetHigh', DWORD)]

    class _OFFSET_UNION(ctypes.Union):
        _anonymous_ = ['_offset']
        _fields_ = [
            ('_offset', _OFFSET),
            ('Pointer', PVOID)]

    class OVERLAPPED(ctypes.Structure):
        _anonymous_ = ['_offset_union']
        _fields_ = [
            ('Internal', ULONG_PTR),
            ('InternalHigh', ULONG_PTR),
            ('_offset_union', _OFFSET_UNION),
            ('hEvent', HANDLE)]

    LPOVERLAPPED = ctypes.POINTER(OVERLAPPED)

    # --- Define function prototypes for extra safety ---
    LockFileEx = windll.kernel32.LockFileEx
    LockFileEx.restype = BOOL
    LockFileEx.argtypes = [HANDLE, DWORD, DWORD, DWORD, DWORD, LPOVERLAPPED]
    UnlockFileEx = windll.kernel32.UnlockFileEx
    UnlockFileEx.restype = BOOL
    UnlockFileEx.argtypes = [HANDLE, DWORD, DWORD, DWORD, LPOVERLAPPED]

elif os.name == 'posix':
    import fcntl
    LOCK_SH = fcntl.LOCK_SH  # shared lock
    LOCK_NB = fcntl.LOCK_NB  # non-blocking
    LOCK_EX = fcntl.LOCK_EX
else:
    raise RuntimeError("PortaLocker only defined for nt and posix platforms")

if os.name == 'nt':
    def lock(file, flags):
        """ Return True on success, False otherwise """
        hfile = msvcrt.get_osfhandle(file.fileno())
        overlapped = OVERLAPPED()
        if LockFileEx(hfile, flags, 0, 0, 0xFFFF0000,
                      ctypes.byref(overlapped)):
            return True
        else:
            return False

    def unlock(file):
        hfile = msvcrt.get_osfhandle(file.fileno())
        overlapped = OVERLAPPED()
        if UnlockFileEx(hfile, 0, 0, 0xFFFF0000, ctypes.byref(overlapped)):
            return True
        else:
            return False

elif os.name == 'posix':
    def lock(file, flags):
        if fcntl.flock(file.fileno(), flags) == 0:
            return True
        else:
            return False

    def unlock(file):
        if fcntl.flock(file.fileno(), fcntl.LOCK_UN) == 0:
            return True
        else:
            return False

if __name__ == '__main__':
    from time import time, strftime, localtime
    import sys

    log = open('log.txt', "a+")
    lock(log, LOCK_EX)

    timestamp = strftime("%m/%d/%Y %H:%M:%S\n", localtime(time()))
    log.write(timestamp)

    print("Wrote lines. Hit enter to release lock.")
    dummy = sys.stdin.readline()

    log.close()

Roundup Issue Tracker: http://roundup-tracker.org/