Hotpath fixes

[LZL0]

Summary by cubic

Speeds up the streaming hot path and reduces memory churn by buffering tokens, lazy concat via a descriptor, delta-only scans, and trimmed guardrail/observability overhead. Benchmarks now show 596K tokens/s for core and 114.9K tokens/s for the full stack, with lower latency under load.

• Refactors

  • State: add _content_buffer; lazy concat via _ContentDescriptor; append_token buffers tokens.
  • Runtime: skip per-token callbacks and guardrail observability when handlers are missing.
  • Guardrails: json_rule caches JSON detection and resets on empty content or shrink; markdown_rule skips analysis until completed; pattern_rule precompiles a combined regex, scans only deltas with small overlap, and runs a full scan at completion.
  • Drift: history uses deques sized by entropy_window; store only the last analysis window; reset honors window size; get_history returns lists.

• Bug Fixes

  • Clear _content_buffer on invalid checkpoints and fresh retries to avoid stale content.
  • pattern_rule/drift/json_rule handle content replacement/shrinks correctly (no stale caches, no unbounded growth, no full re-scan during streaming).

^{Written for commit 57e0c93. Summary will update on new commits.}

[cubic-dev-ai]

6 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/l0/guardrails.py">

<violation number="1" location="src/l0/guardrails.py:1563">
P2: The cached `is_json_content` flag is never safely invalidated for reused/new streams, so `json_rule()` can keep the previous stream's JSON/non-JSON classification and skip or mis-run validation.</violation>

<violation number="2" location="src/l0/guardrails.py:1889">
P1: Guard against an empty `patterns` list before compiling the combined regex; otherwise this rule matches the empty string everywhere and floods the stream with false violations.</violation>

<violation number="3" location="src/l0/guardrails.py:1904">
P2: This delta-only scan offset can miss earlier bad patterns during the later full-content check, because the same rule is first run against `context.delta` and then reused for the whole response.</violation>
</file>

<file name="src/l0/drift.py">

<violation number="1" location="src/l0/drift.py:409">
P2: `get_history()["last_content"]` now returns only the sliding window, which silently truncates long outputs behind the existing API name.</violation>
</file>

<file name="src/l0/state.py">

<violation number="1" location="src/l0/state.py:35">
P1: Buffering tokens here leaves `state.content` stale on abort paths, so post-abort reads and abort telemetry can drop the most recent tokens.</violation>
</file>

<file name="src/l0/runtime.py">

<violation number="1" location="src/l0/runtime.py:733">
P2: Flushing the full content string inside periodic checks makes long streams O(n²) again.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

[cubic-dev-ai]

P1: Buffering tokens here leaves state.content stale on abort paths, so post-abort reads and abort telemetry can drop the most recent tokens.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/l0/state.py, line 35:

<comment>Buffering tokens here leaves `state.content` stale on abort paths, so post-abort reads and abort telemetry can drop the most recent tokens.</comment>

<file context>
@@ -12,23 +12,33 @@ def create_state() -> State:
         state.first_token_at = now
     state.last_token_at = now
-    state.content += token
+    state._content_buffer.append(token)
     state.token_count += 1
 
</file context>

[cubic-dev-ai]

P2: get_history()["last_content"] now returns only the sliding window, which silently truncates long outputs behind the existing API name.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/l0/drift.py, line 409:

<comment>`get_history()["last_content"]` now returns only the sliding window, which silently truncates long outputs behind the existing API name.</comment>

<file context>
@@ -396,14 +396,17 @@ def _detect_excessive_hedging(self, content: str) -> bool:
-            "last_content": self._history.last_content,
+            "entropy": list(self._history.entropy),
+            "tokens": list(self._history.tokens),
+            "last_content": self._history.last_window,
         }
 
</file context>

[cubic-dev-ai]

P2: Flushing the full content string inside periodic checks makes long streams O(n²) again.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/l0/runtime.py, line 733:

<comment>Flushing the full content string inside periodic checks makes long streams O(n²) again.</comment>

<file context>
@@ -718,82 +719,94 @@ async def emit_buffered_tool_calls() -> AsyncIterator[Event]:
-                                    phase="post",
-                                    ruleCount=len(guardrails),
-                                )
+                                flush_content(state)
+                                _has_obs = event_bus._handler is not None
 
</file context>