Retrying failing calls • garnish

library(garnish)

Some failures are temporary: a network blip, a rate limit, a lock held by someone else. retry() calls the function again until it works or you tell it to give up. It is a port of Python’s tenacity, and is built from three questions:

When do we stop? (stop)
How long do we wait between attempts? (wait)
Which outcomes count as a failure? (retry)

Each has a default, so retry() on its own retries any error, immediately, forever. That is rarely what you want, so you will usually set stop.

A function that fails at first

The examples use a function that fails twice and then succeeds:

make_flaky <- function(fail_times) {
  calls <- 0
  function() {
    calls <<- calls + 1
    if (calls <= fail_times) stop("not yet (call ", calls, ")")
    paste("worked on call", calls)
  }
}

Apply retry() with %@%, or by calling it on the function:

flaky <- retry(stop = stop_after_attempt(5)) %@% make_flaky(2)
flaky()

## [1] "worked on call 3"

Deciding when to stop

stop_after_attempt(n) makes at most n attempts. When it fires, a retry_error condition is signalled:

never_works <- retry(stop = stop_after_attempt(3)) %@% make_flaky(Inf)
never_works()

## Error:
## ! Retry failed after 3 attempt(s): not yet (call 3)

The condition records what happened, so you can catch it and look:

fresh <- retry(stop = stop_after_attempt(3)) %@% make_flaky(Inf)
err <- tryCatch(fresh(), retry_error = identity)
err$attempt

## [1] 3

conditionMessage(err$error)

## [1] "not yet (call 3)"

If you would rather see the original error, set reraise = TRUE:

loud <- retry(stop = stop_after_attempt(2), reraise = TRUE) %@% make_flaky(Inf)
tryCatch(loud(), retry_error = function(e) "retry_error", error = conditionMessage)

## [1] "not yet (call 2)"

Or return a fallback instead of throwing. retry_error_callback receives the final state, a list with attempt, wait, elapsed, error and value:

safe <- retry(
  stop = stop_after_attempt(2),
  retry_error_callback = function(state) "fallback"
) %@% make_flaky(Inf)
safe()

## [1] "fallback"

Stop condition	Gives up when
`stop_never()`	never (the default)
`stop_after_attempt(n)`	`n` attempts have been made
`stop_after_delay(s)`	`s` seconds have passed since the first attempt
`stop_before_delay(s)`	the next wait would pass `s` seconds in total
`stop_any(a, b)`	either `a` or `b` fires
`stop_all(a, b)`	both fire

Combine them to say “try up to 10 times, but never for more than 30 seconds”:

stop_any(stop_after_attempt(10), stop_after_delay(30))

## <garnish::stop_any>
##  @ stops:List of 2
##  .. $ : <garnish::stop_after_attempt>
##  ..  ..@ max_attempt_number: num 10
##  .. $ : <garnish::stop_after_delay>
##  ..  ..@ max_delay: num 30

Deciding how long to wait

By default retry() does not wait. For anything talking to a server you want a pause, and usually one that grows. wait_exp() doubles the wait each time:

patient <- retry(
  stop = stop_after_attempt(4),
  wait = wait_exp(multiplier = 0.01)
) %@% make_flaky(3)
system.time(res <- patient())[["elapsed"]] > 0.05

## [1] TRUE

res

## [1] "worked on call 4"

Waiting the same length of time means many clients that failed together retry together. Adding randomness spreads them out, and + combines strategies:

wait_exp(multiplier = 0.1) + wait_rand(0, 0.1)

## <garnish::wait_combine>
##  @ waits:List of 2
##  .. $ : <garnish::wait_exp>
##  ..  ..@ multiplier: num 0.1
##  ..  ..@ min       : num 0
##  ..  ..@ max       : num 4.61e+18
##  ..  ..@ exp_base  : num 2
##  .. $ : <garnish::wait_rand>
##  ..  ..@ min: num 0
##  ..  ..@ max: num 0.1

Wait strategy	Waits
`wait_none()`	no time (the default)
`wait_fixed(s)`	`s` seconds
`wait_rand(min, max)`	a random time between `min` and `max`
`wait_exp()`	`multiplier * exp_base^(attempt - 1)`, between `min` and `max`
`wait_exp_jitter()`	exponential, with `"full"`, `"equal"` or `"decorrelated"` jitter
`wait_chain(a, b, ...)`	the n-th strategy for the n-th attempt
`wait_combine(a, b)` or `a + b`	the sum of the strategies

Deciding what to retry

By default every error is retried. A bad password will not fix itself on the fifth attempt, so retry only what is temporary.

retry_if_error() matches condition classes:

calls <- 0
fetch <- retry(
  stop = stop_after_attempt(3),
  retry = retry_if_error("http_error")
) %@% function() {
  calls <<- calls + 1
  rlang::abort("bad request", class = "client_error")
}
try(fetch(), silent = TRUE)
calls

## [1] 1

It made one attempt, because client_error is not an http_error. retry_if_error_message() matches the text instead, either exactly or by regular expression:

retry_if_error_message(match = "timed? ?out")

## <garnish::retry_if_error_message>
##  @ message: NULL
##  @ match  : chr "timed? ?out"

A function can also fail without erroring, by returning something unusable. retry_if_result() retries when a predicate on the value is TRUE:

polls <- 0
poll <- retry(
  stop = stop_after_attempt(5),
  retry = retry_if_result(is.null)
) %@% function() {
  polls <<- polls + 1
  if (polls < 3) NULL else "ready"
}
poll()

## [1] "ready"

retry_never() turns retrying off, which is useful for switching it off from a setting without removing the decorator.

Seeing what is happening

Silent retries are hard to debug. The hooks run around each sleep:

logged <- retry(
  stop = stop_after_attempt(4),
  wait = wait_fixed(0.01),
  before_sleep = before_sleep_log(function(msg) cat(msg, "\n")),
  after_sleep = after_sleep_log(function(msg) cat(msg, "\n"))
) %@% make_flaky(2)
logged()

## Attempt 1 failed: not yet (call 1). Retrying in 0.01 seconds. 
## Slept 0.01 seconds after attempt 1. Retrying. 
## Attempt 2 failed: not yet (call 2). Retrying in 0.01 seconds. 
## Slept 0.01 seconds after attempt 2. Retrying.

## [1] "worked on call 3"

The defaults are before_sleep_none() and after_sleep_none(). The loggers take any function that accepts one string, so you can send the message to whatever logging package you use.

Writing your own strategies

Everything above is built from five kinds of strategy, and you can add your own to any of them. They are S7 classes, so if you have not used S7, the idea is small: make a class that inherits from a base class, then say what it does by registering a method for a generic. The built-in strategies work the same way, and yours combine with them.

`retry()` argument	Inherit from	Method of	Return
`stop`	`stop_base`	`stop_fn(class, state)`	`TRUE` to give up
`wait`	`wait_base`	`wait_fn(class, attempt, prev_wait)`	seconds to sleep
`retry`	`retry_base`	`retry_fn(class, outcome)`	`TRUE` to try again
`before_sleep`	`before_sleep_base`	`before_sleep_fn(class, state)`	nothing
`after_sleep`	`after_sleep_base`	`after_sleep_fn(class, state)`	nothing

A wait that grows by a fixed step each attempt:

library(S7)

wait_linear <- new_class(
  "wait_linear",
  parent = wait_base,
  properties = list(step = class_numeric),
  constructor = function(step = 1) new_object(wait_base(), step = step)
)

method(wait_fn, wait_linear) <- function(class, attempt, prev_wait = NULL, ...) {
  class@step * attempt
}

wait_fn(wait_linear(2), attempt = 3)

## [1] 6

It composes like the built-in waits, with + and in wait_chain():

wait_fn(wait_linear(1) + wait_fixed(2), attempt = 3)

## [1] 5

A stop condition gets the state list described in ?extending_retry, which has attempt, wait, elapsed, error and value. This one gives up once an attempt fails with a message matching a pattern:

stop_on_message <- new_class(
  "stop_on_message",
  parent = stop_base,
  properties = list(pattern = class_character),
  constructor = function(pattern) new_object(stop_base(), pattern = pattern)
)

method(stop_fn, stop_on_message) <- function(class, state, ...) {
  !is.null(state$error) && grepl(class@pattern, conditionMessage(state$error))
}

attempts <- 0
fatal <- retry(
  stop = stop_any(stop_after_attempt(5), stop_on_message("permission denied"))
) %@% function() {
  attempts <<- attempts + 1
  stop("permission denied")
}
try(fatal(), silent = TRUE)
attempts

## [1] 1

It stopped after one attempt, though stop_after_attempt(5) would have allowed five, because stop_any() fires when either condition does.

A retry condition gets outcome, a list with value and error, one of which is NULL:

retry_if_negative <- new_class("retry_if_negative", parent = retry_base)

method(retry_fn, retry_if_negative) <- function(class, outcome, ...) {
  is.null(outcome$error) && outcome$value < 0
}

vals <- c(-1, -1, 5)
i <- 0
next_value <- retry(retry = retry_if_negative()) %@% function() {
  i <<- i + 1
  vals[[i]]
}
next_value()

## [1] 5

## [1] 3

If you write strategies in a package, call S7::methods_register() in its .onLoad() so the methods are registered when it loads.

Putting it together

fetch := retry(
  stop = stop_any(stop_after_attempt(5), stop_after_delay(30)),
  wait = wait_exp_jitter(base = 0.5, jitter = "full"),
  retry = retry_if_error_message(match = "timeout|503"),
  before_sleep = before_sleep_log(),
  reraise = TRUE
) %@% function(url) {
  # ...
}

Notes

Only retry things that are safe to repeat. Retrying a call that charges a card or sends an email can do it twice. Put the retry around the part that is safe.
Retrying hides failures. Use a stop condition, and a hook that logs, so a service that is down for an hour does not look like a slow one.
Decorators stack. In a %@% b %@% f, a is outermost. Put retry() outside a timing decorator to time all attempts, or inside it to time each one.
Retrying a service that is down is wasteful. For a failure that lasts minutes rather than seconds, stack circuit_breaker() outside retry() so callers fail fast once the service is clearly down.