Chat failure emails

Official dev and production email the configured SERVER_OWNER_EMAIL (or ADMIN_NOTIFY_EMAIL) when the backend detects a terminal technical chat failure. Self-hosted servers exit before touching the deduplication store or mail transport. The existing trusted server-edition resolver controls this distinction; setting an environment label alone cannot enable the feature.

The AI worker classifies its terminal result, including typed processing failures and generic error text inside a successful task envelope, and catches exceptions and technical timeouts. Queued-payload validation, task admission, WebSocket dispatch/handler errors, and failed response delivery also notify. Every layer uses the same chat_id:message_id identity before hashing, so detecting one terminal failure at multiple boundaries does not send multiple emails. Expected credit, policy, and cancellation outcomes are excluded. Content is inspected locally and never included in the queue payload or email.

The separate email task receives only a one-way request fingerprint and allowlisted stage/category. A Redis Lua transaction marks a fingerprint seen for seven days and increments the day’s failure count. Every distinct eligible failure proceeds to a mail delivery attempt; there is no daily suppression threshold. Keys are isolated by the actual server edition and UTC calendar day. Workers share the counter and deduplication state, so restarting a worker does not reset them. Counter history expires after two days. Cache loss is a limitation; this is not durable disaster-recovery accounting.

The fingerprint is marked before mail delivery. Missing configuration, rejected mail, and ambiguous transport outcomes retain explicit states and are not retried as a second email attempt for the same failed request. This prevents duplicate mail when provider acceptance is ambiguous. Worker logs distinguish queued, accepted, rejected, missing configuration, unavailable deduplication, and unknown delivery. Accepted means accepted by the configured email service, not inbox delivery.

No Discord or OpenObserve dependency exists in this path. The existing email broker/worker remains necessary. The producer gives broker submission at most three retries with bounded intervals and waits at most one second from the chat caller’s perspective. Both asynchronous handlers and synchronous Celery boundaries receive a definite enqueue acknowledgement only when the broker call returns successfully. A queue failure logs a content-free error, and worker deduplication protects against duplicate jobs after ambiguous submission.

Durable recovery cleanup also discovers terminal technical failures such as expired dispatch claims and worker timeouts. It locks the existing preflight record before checking for a sealed response and excludes cancellation, deletion, successfully recovered turns, and preparation that was never dispatched. New technical failure transitions record a pending alert in the same transaction. They remain eligible until a definite broker acknowledgement is recorded; a later cleanup sweep retries after an unavailable queue, ambiguous acknowledgement, or cleanup-process crash. Acknowledgement is idempotent for that exact record. Previously failed records are not automatically backfilled into an email backlog. Internal candidates contain only existing correlation metadata; the email payload still contains only its hash and allowlisted stage/category. Independent infrastructure alerts remain necessary when the notification system itself is unavailable.

The recovery preflight schema adds nullable pending and acknowledgement timestamps. Apply those additive fields before loading the updated recovery transaction extension and cleanup worker. This introduces no public notification endpoint or encrypted-content access. Email contains environment, normalized stage/category, BUILD_COMMIT_SHA when valid, and daily counters. No chat/user IDs, raw exception details, prompts, response text, or ciphertext is emailed.

Verification uses focused notification units, transaction tests, and adjacent AI regression suites. The historical 2026-09-10 E2E waiver does not apply to the current work. Current unit tests use fake mail transport; they do not prove inbox delivery. Dev source deployment does not imply a production rollout.