2026-08-16
AK47v7.62 release 0 + 24h acceptance cycle
Closed release item 0 after months of incremental hardening. Repaired the connection-epoch implementation that had been partially tested but never fully wired into production, then deployed both the duty-cycle grace and epoch fixes together.
- Duty-cycle closing grace (commit 455694a): The tape writer previously cut exactly at the contract boundary, dropping the final observation needed for settlement. Added a 5-second grace window after every recorded slot. Proved in production by comparing the same tape: old code went OFF at 12:15:00.014301 while new code stayed live until 12:45:05.037746, correctly capturing Coinbase frames through 12:45:04.786371. 48/48 tape tests + 6/6 mutants killed.
- Connection-epoch refactor (commit 2233f72): Implemented the missing
_begin_connection_epoch that previous tests had assumed. Every new socket now atomically clears pending subscribe ID, ticker, timestamp, previous SID, and replaces the OrderBook while preserving cumulative feed diagnostics. The new connection is only published after the old baseline is invalidated. This ordering is critical because get_quote() samples connection state before acquiring the book lock.
- Production verification: Confirmed via
/proc endpoint — commit 2233f72, PID 431404, MAX_VOL_PCT=0.16 present in environment, repository bytes matched. Real reconnects exercised the epoch path with clean ACK, fresh snapshot, and zero sequence gaps.
- Subscribe-ACK watchdog findings: 56% failure rate across 64 rollovers (36 timeouts). The watchdog (introduced Aug 15) reduced blind time from a full 15-minute window to a bounded reconnect sequence. This is an availability defect that predates the current release.
- Documented pre-existing defects: reconnect backoff never resets after healthy connection (grows to 30s),
tape_members.py OOMs on production archives, Tape._roll() can reopen gzip during OFF slots, stop() leaves _ws and _connected_at populated.
24-hour acceptance cycle started at 2026-08-16T13:30:00Z with explicit safety gates (no WS quote before fresh snapshot, no order from EMPTY/ONE_LEG state, no pending subscription surviving epoch, no order during blind interval). Availability metrics tracked separately.
2026-08-15
Live verification + /proc endpoint
Built a zero-auth HTTP /proc endpoint that exposes live runtime state for both internal dashboards and external monitoring without requiring credentials or exposing control surfaces.
- Returns current position, last fill timestamp/price/quantity, cycle health counters, paper vs live parity status, and connection epoch metadata in a single request.
- Supports both JSON and plain-text modes for different consumers (Telegram bot vs monitoring scripts).
- Integrated directly into the existing agent runtime loop so it reflects the exact state the execution engine is using at that moment.
- Designed so that a single request gives enough information to determine whether the system is healthy or in a degraded state without needing to correlate multiple sources.
2026-08-12
Paper-to-live promotion gate
Implemented a hard safety gate that prevents live capital from being used until the system has demonstrated statistical parity with paper trading over a configurable number of cycles.
- Gate state is persisted to disk and survives restarts. The system will refuse to place live orders if the gate has not been satisfied.
- Tolerance and required cycle count are configurable per run but cannot be changed while the gate is armed.
- Automatic promotion occurs only after N consecutive cycles fall within tolerance on key metrics (fill rate, slippage, latency distribution).
- Manual override path exists but writes an immutable audit record with operator identity and justification.
- This was the final missing piece that allowed us to move from “paper looks good” to “we have evidence the live system will behave the same.”
2026-08-08
Telegram delivery + daily digest
Replaced ad-hoc logging with a structured daily digest system that produces consistent, machine-readable summaries at market close.
- Every digest is generated from a single structured run record rather than scraping logs. The reporting path is designed to produce the same ordered fields from the same record, subject to the documented software version, inputs, and environment.
- Includes cycle count, fill summary with attribution, P&L by window, reconnect/ACK timeout counts, and any safety alerts raised during the period.
- Messages are threaded so each day becomes a single conversation rather than a flood of individual updates.
- High-priority alerts (safety violations, large deviations) are sent to a separate channel with different formatting and immediate delivery.
- The digest format itself became a useful debugging tool — patterns in the structured data revealed issues that raw logs had hidden.
2026-08-03
Multi-day run stability
Refactored the core execution loop to survive extended unattended operation across network partitions, API rate limits, partial fills, and process restarts.
- Introduced an explicit state machine with persistent checkpoints written after every significant transition (new connection, subscription confirmed, book initialized, order placed, fill received).
- Order submission is fully idempotent — the same order can be safely retried without creating duplicates because the exchange order ID is the source of truth.
- On restart the system reads the last checkpoint, replays any unacknowledged actions, and resumes from the exact state it left rather than starting fresh.
- Exponential backoff with jitter was added for reconnects, but more importantly the backoff state is now part of the persisted checkpoint so a long outage doesn’t result in ever-increasing delays after recovery.
- This work took multiple iterations because early versions would either lose state on crash or would resume into an inconsistent book state.
