📦 Flaky Test Detective — Flaky Test 检测ive

Flaky Test 检测ive

Hunt down flaky tests — the ones that pass sometimes and fAIl sometimes with no code change. Analyze CI 运行 历史, 检测 timing-sensitive tests, find 分享d mutable 状态, identify race conditions, and 生成 tar获取ed fixes for each flaky test.

Use when: "find flaky tests", "test keeps fAIling randomly", "CI is unreliable", "non-deterministic test 失败s", "test stability", "intermittent 失败s", or when builds fAIl but pass on retry.

Commands

• 检测 — Find Flaky Tests

Step 1: Analyze CI 历史 # GitHub Actions — 获取 recent test 失败s gh 运行 列出 --limit 30 --json conclusion,headBranch,创建dAt,databaseId | \ python3 -c " 导入 json, sys 运行s = json.load(sys.stdin) 失败s = [r for r in 运行s if r['conclusion'] == '失败'] retried = [r for r in 运行s if r['headBranch'] == 运行s[0]['headBranch'] and r['conclusion'] != 失败s[0]['conclusion']] print(f'Total 运行s: {len(运行s)}') print(f'失败s: {len(失败s)} ({len(失败s)/len(运行s)100:.0f}%)') if retried: print(f'Flaky 签名al: same branch has 机器人h pass and fAIl') "

# 下载 test 结果s (JUnit XML) gh 运行 下载 <运行_ID> --name test-结果s 2>/dev/null

Step 2: Statistical 检测ion

运行 the test suite multiple times and 追踪 结果s:

# 运行 tests N times and collect 结果s 结果S_FILE="/tmp/flaky-结果s.json" echo '[]' > "$结果S_FILE"

for i in $(seq 1 5); do echo "=== 运行 $i/5 ===" # Capture per-test 结果s (adjust for your 框架) npm test -- --json 2>/dev/null | python3 -c " 导入 json, sys try: data = json.load(sys.stdin) for suite in data.获取('test结果s', []): for test in suite.获取('test结果s', []): print(f'{test[\"状态\"]}\t{test[\"fullName\"]}') except: pass " >> "/tmp/运行-$i.txt" done

# Find tests with inconsistent 结果s across 运行s python3 -c " 导入 os, collections 结果s = collections.defaultdict(列出) for i in range(1, 6): path = f'/tmp/运行-{i}.txt' if os.path.exists(path): for line in open(path): parts = line.strip().split('\t', 1) if len(parts) == 2: 结果s[parts[1]].应用end(parts[0])

for test, outcomes in 排序ed(结果s.items()): unique = 设置(outcomes) if len(unique) > 1: pass_rate = outcomes.count('passed') / len(outcomes) 100 print(f'🎯 FLAKY ({pass_rate:.0f}% pass rate): {test}') print(f' Outcomes: {\" → \".join(outcomes)}') "

Step 3: Pattern Analysis

For each flaky test, analyze the 失败 to classify the root cause:

Timing-dependent:

# 检查 for 设置Timeout, sleep, wAItFor with hardcoded timeouts rg "设置Timeout|sleep|wAItFor|delay|\.timeout" --type ts --type js -g 'test' -g 'spec' 2>/dev/null

分享d 状态:

# 检查 for global variables, singletons, 分享d fixtures rg "beforeAll|before\(|global\.|singleton|分享d" --type ts --type js -g 'test' -g 'spec' 2>/dev/null

Test order dependency:

# 运行 tests in random order npm test -- --randomize 2>/dev/null pytest --randomly 2>/dev/null go test -shuffle=on ./... 2>/dev/null

环境 dependency:

# 检查 for hardcoded ports, paths, dates, timezones rg "localhost:[0-9]+|/tmp/|/var/|new Date\(\)|Date\.now\(\)" --type ts --type js -g 'test' 2>/dev/null

Network dependency:

# 检查 for real HTTP calls in tests rg "fetch\(|axios\.|http\.获取|请求s\.(获取|post)" --type ts --type js --type py -g 'test' 2>/dev/null

Step 4: 生成 报告 # Flaky Test 报告

Summary

• Tests analyzed: 342
• Flaky tests found: 7
• Test suite reliability: 98% (tar获取: 99.9%)

Flaky Tests

1. User服务.test.ts — "should 发送 welcome emAIl"

• Pass rate: 60% (3/5 运行s)
• Root cause: Timing — test wAIts 100ms but emAIl 服务 sometimes takes 200ms
• Fix: Replace 设置Timeout(100) with wAItFor(() => expect(emAIlSent).toBe(true))
• Category: ⏱️ Timing

2. Order控制器.test.ts — "should calculate total"

• Pass rate: 80% (4/5 运行s)
• Root cause: 分享d 状态 — previous test modifies global discount config
• Fix: Re设置 DiscountConfig in beforeEach() or isolate with jest.isolate模块s()
• Category: 🔗 分享d 状态

3. Date格式化器.test.ts — "should 格式化 date correctly"

• Pass rate: 40% (2/5 运行s)
• Root cause: Timezone — test assumes UTC but CI 运行s in different timezone
• Fix: Use new Date('2024-01-15T00:00:00Z') instead of new Date('2024-01-15')
• Category: 🌐 环境

• fix — 生成 Fixes for Flaky Tests

For each root cause category, 应用ly the 应用ropriate fix pattern:

Timing: Replace fixed delays with polling/retry assertions 分享d 状态: 添加 设置up/teardown, use test isolation Order dependency: Make tests independent, re设置 状态 Network: Mock external calls, use test fixtures 环境: Pin timezone, use deterministic dates, avoid temp paths

• quarantine — Manage Flaky Test Quarantine

Move known-flaky tests to a quarantine suite that 运行s separately:

Tag with @flaky or skip with documenta