Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Trying out a Retry decorator in Python

Date: 2009-11-17  |   Modified: 2010-04-27  |   Tags: python   |   2 Comments

The Python wiki has a Retry decorator example which retries calling a failure-prone function using an exponential backoff algorithm. I modified it slightly to check for exceptions instead of a False return value to indicate failure. Each time the decorated function throws an exception, the decorator will wait a period of time and retry calling the function until the maximum number of tries is used up. If the decorated function fails on the last try, the exception will occur unhandled. 

import time

def retry(ExceptionToCheck, tries=4, delay=3, backoff=2):
    """Retry decorator
    original from http://wiki.python.org/moin/PythonDecoratorLibrary#Retry
    """
    def deco_retry(f):
        def f_retry(*args, **kwargs):
            mtries, mdelay = tries, delay
            try_one_last_time = True
            while mtries > 1:
                try:
                    return f(*args, **kwargs)
                    try_one_last_time = False
                    break
                except ExceptionToCheck, e:
                    print "%s, Retrying in %d seconds..." % (str(e), mdelay)
                    time.sleep(mdelay)
                    mtries -= 1
                    mdelay *= backoff
            if try_one_last_time:
                return f(*args, **kwargs)
            return
        return f_retry # true decorator
    return deco_retry

Try an "always fail" case

@retry(Exception, tries=4)
def test_fail(text):
    raise Exception("Fail")

test_fail("it works!")

Results:

Fail, Retrying in 3 seconds...
Fail, Retrying in 6 seconds...
Fail, Retrying in 12 seconds...
Traceback (most recent call last):
  File "retry_decorator.py", line 47, in 
    test_fail("it works!")
  File "retry_decorator.py", line 26, in f_retry
    f(*args, **kwargs)
  File "retry_decorator.py", line 33, in test_fail
    raise Exception("Fail")
Exception: Fail

Try a "success" case

@retry(Exception, tries=4)
def test_success(text):
    print "Success: ", text

test_success("it works!")

Results:

Success:  it works!

Try a "random fail" case

import random

@retry(Exception, tries=4)
def test_random(text):
    x = random.random()
    if x < 0.5:
        raise Exception("Fail")
    else:
        print "Success: ", text

test_random("it works!")

Results:

Fail, Retrying in 3 seconds...
Success:  it works!

Try handling multiple exceptions

Added 2010-04-27

import random

@retry((NameError, IOError), tries=20, delay=1, backoff=1)
def test_multiple_exceptions():
    x = random.random()
    if x < 0.40:
        raise NameError("NameError")
    elif x < 0.80:
        raise IOError("IOError")
    else:
        raise KeyError("KeyError")

test_multiple_exceptions()

Results:

IOError, Retrying in 1 seconds...
NameError, Retrying in 1 seconds...
IOError, Retrying in 1 seconds...
IOError, Retrying in 1 seconds...
NameError, Retrying in 1 seconds...
IOError, Retrying in 1 seconds...
NameError, Retrying in 1 seconds...
NameError, Retrying in 1 seconds...
NameError, Retrying in 1 seconds...
IOError, Retrying in 1 seconds...
Traceback (most recent call last):
  File "retry_decorator.py", line 61, in 
    test_multiple_exceptions("hello")
  File "retry_decorator.py", line 14, in f_retry
    f(*args, **kwargs)
  File "retry_decorator.py", line 56, in test_multiple_exceptions
    raise KeyError("KeyError")
KeyError: 'KeyError'

2 Comments — feed icon Comments feed for this post


#1 Tiago Almeida commented on 2010-04-14:

This is great! Thanks for pointing this out. However i would modify it to return what the decorated function returns and throw (raise) the last exception given by the decorated function. Like this:

def retry(ExceptionToCheck, tries=4, delay=3, backoff=2):
    """Retry decorator
    original from http://wiki.python.org/moin/PythonDecoratorLibrary#Retry
    """
    def deco_retry(f):
        def f_retry(*args, **kwargs):
            mtries, mdelay = tries, delay
            while mtries > 0:
                try:
                    return f(*args, **kwargs)
                except ExceptionToCheck, e:
                    print "%s, Retrying in %d seconds..." % (str(e), mdelay)
                    time.sleep(mdelay)
                    mtries -= 1
                    mdelay *= backoff
                    lastException = e
            raise lastException
        return f_retry # true decorator
    return deco_retry

#2 Eliot commented on 2010-04-14:

Tiago: Thanks for the comment! I tried out your modified decorator just now.

Raising the exception as you do is more elegant than my method, but I don't like the new behavior. Before raising the exception the last time, it will print "Retrying in xx seconds..." and then sleep for xx seconds (without actually retrying the function).

Making it return the value that the decorated function returns is a good idea. I updated my post above.

Post a comment
Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Slicehost