Add autotrack — autonomous MOT tracker optimization loop [rebase&merge]

[Borda]

Summary

MOT tracker quality depends on two largely independent axes: algorithm design and hyperparameter tuning. Most published improvements conflate them — a well-tuned weaker algorithm routinely beats a poorly-tuned stronger one, making it hard to isolate what actually matters. This PR separates the axes by adding autotrack/, an autonomous optimization loop for SORT, ByteTrack, and OC-SORT on MOT17.

The goal is both practical (better trackers, reproducible tuning) and scientific (the experiment log — including every reverted change — is itself a research artifact).

Approach

Three progressive layers build on each other:

Layer 1 — SOTA trackers with solid defaults. The existing trackers/core/ implementations of SORT, ByteTrack, and OC-SORT are already competitive out of the box. This layer is the foundation; autotrack/ does not replace it.

Layer 2 — Optuna extracts the best from the existing parameter surface. optimize_tracking.py runs an Optuna study over the tracker's exposed hyperparameters (Kalman noise scales, confidence thresholds, buffer sizes). No code changes — pure tuning. FRCNN results gain 1–2.5 HOTA points; SDP gains 2–4 points. This layer alone is useful as a standalone tuning tool and can be adopted without running the agent loop.

Layer 3 — autotrack goes beyond tuning by making algorithmic improvements. This is the novel contribution. An autonomous agent iterates over structural code changes (state representation, association strategy, camera motion compensation, Kalman mechanics), measures HOTA at fixed default parameters after each change, keeps improvements, and reverts regressions. Optuna acts as a second-pass validator after each kept change to confirm the improvement is real and not a tuning artifact. The iteration log is JSONL and captures every attempt, kept or reverted.

Human defines:  research question · metric · hard boundaries
Agent decides:  what to change · what to try next

Two tools govern the loop:

Tool	Role
optimize_tracking.py --n-trials 1	Campaign metric — default params, clean code-change signal
optimize_tracking.py --n-trials N	Optuna study — warm-starts from best_config.json, validates tuned ceiling

The agent is explicitly permitted to update optimize_tracking.py as the tracker architecture evolves — adding parameters that newly exist, removing ones absorbed into the implementation, tightening search ranges as knowledge accumulates.

Benchmarks

MOT17-val, full 7-sequence eval. Defaults = fixed params from default_config.json, no tuning. +Optuna = n=500 trials. +autotrack + Optuna = in progress.

FRCNN public detections (bundled, no GPU)

Config	ByteTrack	OC-SORT	SORT
Defaults (HOTA)	50.36	49.69	49.95
+ Optuna (HOTA)	51.76	52.22	51.49
+ autotrack + Optuna (HOTA)	(pending)	(pending)	(pending)

SDP public detections (bundled, no GPU)

Config	ByteTrack	OC-SORT	SORT
Defaults (HOTA)	53.94	53.35	53.22
+ Optuna (HOTA)	56.12	57.75	56.08
+ autotrack + Optuna (HOTA)	(pending)	(pending)	(pending)

Estimated ceiling with code improvements + Optuna on FRCNN: ~61.9 HOTA (vs ~56.0 for tuning alone), derived from the DetA/AssA decomposition — DetA is bounded by the detector (~0.57–0.62 for FRCNN), but AssA has substantial headroom from ~0.55 to ~0.65 via better association logic.

Hard guarantees

Three invariants are enforced by program.md and cannot be relaxed by the agent:

• No GT leakage. The tracker sees only det/det.txt. gt/gt.txt is never accessed at inference time.
• Reproducible detections. FRCNN and SDP detections are bundled with the MOT17 benchmark. Generated detections (RF-DETR, YOLO World X) are written to content-addressed sibling directories before any agent run — they are frozen inputs, not live inference.
• Metrics via trackers.eval only. trackers/eval/ is out of scope for agent edits. The metric computation is identical across all iterations; the agent cannot move the goalposts.

Quick start

# 1. Install the optimize dependency group
uv sync --group optimize

# 2. Download MOT17-val (bundled detections, no GPU needed)
trackers download mot17 --split val --asset annotations,detections

# 3. Baseline: measure defaults
cd autotrack
uv run python optimize_tracking.py bytetrack frcnn --n-trials 1   # ~50 HOTA

# 4. Tune: Optuna over the parameter surface
uv run python optimize_tracking.py bytetrack frcnn --n-trials 500  # ~52 HOTA

To run the autonomous agent loop, point any coding agent at program.md:

claude
> Read program.md and start the experiment loop.

References

• Bewley et al., SORT, ICIP 2016
• Zhang et al., ByteTrack, ECCV 2022
• Cao et al., OC-SORT, CVPR 2023
• Luiten et al., HOTA, IJCV 2021
• Akiba et al., Optuna, KDD 2019

[Copilot]

Pull request overview

This PR introduces the new autotrack/ workflow for autonomous + Optuna-based optimization of MOT17 trackers, and updates core tracker internals to support additional post-processing and association/Kalman behaviors that the optimization loop can tune and validate.

Changes:

• Added autotrack/ tooling: Optuna runner (optimize_tracking.py), detection generation (generate_detections.py), visualization utilities, and configuration/artifact files (default_config.json, search_space.json, best_config.json, program.md).
• Extended ByteTrack and SORT utilities with new association / Kalman mechanics and MOT-gap interpolation.
• Added an optimize dependency group and adjusted repo formatting/ignore configs to support the new workflow.

Reviewed changes

Copilot reviewed 17 out of 19 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
trackers/core/sort/utils.py	Adds MOT-format short-gap interpolation helper used by autotrack evaluation output.
trackers/core/bytetrack/tracker.py	Adds stage-2 IoU threshold and IoU age discount for stage-1 ranking; updates association gating logic.
trackers/core/bytetrack/kalman.py	Adds velocity decay, miss-noise inflation, P-reset, and ORU mechanics to ByteTrack Kalman tracker.
README.md	Badge formatting change (single-line).
pyproject.toml	Adds optimize dependency group and uv git source for onnx-simplifier.
docs/trackers/ocsort.md	Reflowed paragraph formatting.
docs/trackers/comparison.md	Reflowed admonition formatting.
CODE_OF_CONDUCT.md	Reflowed paragraph formatting.
autotrack/visualize_detections.py	New utility to render MOT detections on frames.
autotrack/search_space.json	New Optuna parameter search space definitions per tracker.
autotrack/README.md	New documentation for the autotrack workflow and benchmarks.
autotrack/program.md	New campaign contract/spec for the autonomous optimization loop.
autotrack/optimize_tracking.py	New Optuna study runner + evaluation harness using trackers.eval.
autotrack/generate_detections.py	New script to generate MOT17 detections via RF-DETR / YOLO-World backends.
autotrack/default_config.json	New baseline/default parameter set for --n-trials 1 runs.
autotrack/best_config.json	New committed “best known” tuned configs used for warm-starting/guarding.
.pre-commit-config.yaml	mdformat configured with --wrap=no (drives markdown reflow behavior).
.gitignore	Adjusts ignores (including .python-version) and adds autotrack output/cache patterns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.