view roundup/support.py @ 6431:ada1edcc9132

issue2551142 - Import ... unique constraint failure. Full title: Import of retired node with username after active node fails with unique constraint failure. Fix this in two ways: 1) sort export on keyname, retired status so that retired nodes for a given keyname are before the acive node in the export file. This stops generating a broken export. 2) handle importing a broken export by deactivating/fixing up/clearing the active record's unique index entry temporarily. Redo the import of the retired node and resetting the active record to active. The fixup changes the unique index (keyvalue, __retired__) from (keyvalue, 0) to (keyvalue, -1). Then it retries the failed import of a retired record with keyvalue. I use -1 in case something goes wrong, It makes the record stand out in the database allowing hand recovery if needed. Rather than using -1 I could just use the id of the record like a normal retirement does. If the retry of the import fails (raises exception), reset the active record from -1 back to 0 and raise the exception. If it succeeds, reset the active record from -1 back to 0 and continue the import process. Reset __retired__ from -1 to 0 on every import. I don't think the performance loss from resetting on every exception matters as there should be very few exceptions. Also this makes the code more understandable. There is no reason to leave the -1 value in place and do a bulk rest of -1 to 0 after the class csv file is loaded. Also if a fixup is needed it is logged at level info with the rest of the database logging. Also success of the fixup is logged. Fixup failure generates a propagated exception.
author John Rouillard <rouilj@ieee.org>
date Mon, 07 Jun 2021 09:58:39 -0400
parents 01643d37785f
children 33eb82ad26ba
line wrap: on
line source

"""Implements various support classes and functions used in a number of
places in Roundup code.
"""

from __future__ import print_function
__docformat__ = 'restructuredtext'

import os, time, sys


class TruthDict:
    '''Returns True for valid keys, False for others.
    '''
    def __init__(self, keys):
        if keys:
            self.keys = {}
            for col in keys:
                self.keys[col] = 1

    def __getitem__(self, name):
        if hasattr(self, 'keys'):
            return name in self.keys
        else:
            return True


def ensureParentsExist(dest):
    if not os.path.exists(os.path.dirname(dest)):
        os.makedirs(os.path.dirname(dest))


class PrioList:
    '''Manages a sorted list.

    Currently only implements method 'append' and iteration from a
    full list interface.
    Implementation: We manage a "sorted" status and sort on demand.
    Appending to the list will require re-sorting before use.
    >>> p = PrioList()
    >>> for i in 5,7,1,-1:
    ...  p.append(i)
    ...
    >>> for k in p:
    ...  print k
    ...
    -1
    1
    5
    7

    '''
    def __init__(self):
        self.list = []
        self.sorted = True

    def append(self, item):
        self.list.append(item)
        self.sorted = False

    def __iter__(self):
        if not self.sorted:
            self.list.sort()
            self.sorted = True
        return iter(self.list)


class Progress:
    '''Progress display for console applications.

    See __main__ block at end of file for sample usage.
    '''
    def __init__(self, info, sequence):
        self.info = info
        self.sequence = iter(sequence)
        self.total = len(sequence)
        self.start = self.now = time.time()
        self.num = 0
        self.stepsize = self.total // 100 or 1
        self.steptimes = []
        self.display()

    def __iter__(self): return self

    def __next__(self):
        self.num += 1

        if self.num > self.total:
            print(self.info, 'done', ' '*(75-len(self.info)-6))
            sys.stdout.flush()
            return next(self.sequence)

        if self.num % self.stepsize:
            return next(self.sequence)

        self.display()
        return next(self.sequence)
    # Python 2 compatibility:
    next = __next__

    def display(self):
        # figure how long we've spent - guess how long to go
        now = time.time()
        steptime = now - self.now
        self.steptimes.insert(0, steptime)
        if len(self.steptimes) > 5:
            self.steptimes.pop()
        steptime = sum(self.steptimes) / len(self.steptimes)
        self.now = now
        eta = steptime * ((self.total - self.num)/self.stepsize)

        # tell it like it is (or might be)
        if now - self.start > 3:
            M = eta / 60
            H = M / 60
            M = M % 60
            S = eta % 60
            if self.total:
                s = '%s %2d%% (ETA %02d:%02d:%02d)' % (self.info,
                    self.num * 100. / self.total, H, M, S)
            else:
                s = '%s 0%% (ETA %02d:%02d:%02d)' % (self.info, H, M, S)
        elif self.total:
            s = '%s %2d%%' % (self.info, self.num * 100. / self.total)
        else:
            s = '%s %d done' % (self.info, self.num)
        sys.stdout.write(s + ' '*(75-len(s)) + '\r')
        sys.stdout.flush()

# vim: set et sts=4 sw=4 :

Roundup Issue Tracker: http://roundup-tracker.org/