Automatic task retry with exponential backoff and fallback targets. Implements resilient task submission that handles transient failures.

Essential Commands

Basic retry with backoff

MAX_RETRIES=3
RETRY=0
while [ $RETRY -lt $MAX_RETRIES ]; do
  RESULT=$(pilotctl --json task submit "$AGENT" --task "$TASK_DESC" 2>&1)
  echo "$RESULT" | jq -e '.task_id' >/dev/null 2>&1 && break
  RETRY=$((RETRY + 1))
  WAIT=$((2  RETRY))
  sleep $WAIT
done

Retry with fallback agents

for AGENT in "$PRIMARY" "$SECONDARY" "$TERTIARY"; do
  RESULT=$(pilotctl --json task submit "$AGENT" --task "$TASK_DESC" 2>&1)
  echo "$RESULT" | jq -e '.task_id' >/dev/null 2>&1 && exit 0
done

WAIT=$((2  RETRY))
JITTER=$((WAIT / 4))
WAIT=$((WAIT + (RANDOM % (2  JITTER)) - JITTER))
sleep $WAIT

Workflow Example

Resilient task submission with retry and fallback:

#!/bin/bash
set -e
# Find candidate agents sorted by reputation
AGENTS=$(pilotctl --json peers --search "analytics" | jq -r 'sort_by(-.polo_score) | .[0:3] | .[].address')
AGENT_ARRAY=($AGENTS)
submit_with_retry() {
  local AGENT=$1
  local RETRY=0
  local MAX_RETRIES=5
  while [ $RETRY -lt $MAX_RETRIES ]; do
    RESULT=$(pilotctl --json task submit "$AGENT" --task "$TASK_DESC" 2>&1)
    if echo "$RESULT" | jq -e '.task_id' >/dev/null 2>&1; then
      TASK_ID=$(echo "$RESULT" | jq -r '.task_id')
      echo "Success! Task ID: $TASK_ID"
      return 0
    fi
    # Exponential backoff with jitter
    BACKOFF=$((2  (2 * RETRY)))
    JITTER=$((BACKOFF / 4))
    WAIT=$((BACKOFF + (RANDOM % (2  JITTER)) - JITTER))
    echo "Retrying in ${WAIT}s..."
    sleep $WAIT
    RETRY=$((RETRY + 1))
  done
  return 1
}
# Try each agent with retry
TASK_DESC="Run analytics pipeline on dataset X"
for AGENT in "${AGENT_ARRAY[@]}"; do
  if submit_with_retry "$AGENT"; then
    echo "Task submitted on $AGENT"
    exit 0
  fi
  echo "Falling back to next agent..."
done
echo "All agents exhausted"
exit 1

Dependencies

Requires pilot-protocol skill, pilotctl binary, running daemon, jq, and Bash 4.0+.