github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
github ([personal profile] github) wrote in [site community profile] changelog2026-04-11 02:10 pm
  • Add Memory
  • Share This Entry

[dreamwidth/dreamwidth] 4d50f4: Add ESN drop/skip observability at silent-failure ...

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 4d50f4a1d495408fb95df32a63239b8bf7ea60b7 https://github.com/dreamwidth/dreamwidth/commit/4d50f4a1d495408fb95df32a63239b8bf7ea60b7 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Task/ESN/FilterSubs.pm M cgi-bin/DW/Task/ESN/ProcessSub.pm M cgi-bin/LJ/NotificationInbox.pm M cgi-bin/LJ/Subscription.pm

Log Message:

Add ESN drop/skip observability at silent-failure points

The ESN pipeline had several silent-drop paths where notifications could disappear without any log trace, making "notification went missing" bug reports impossible to investigate. This adds structured log lines and DW::Stats counters at four of those paths:

  • FilterSubs: the cluster-filter grep was a silent drop for both legitimate user-moved-cluster cases and (suspected) DB-pressure partial-load cases from LJ::load_userids. Now logs each drop at INFO with user=name(uid) sub=N cluster=expected->got reason=... and increments dw.esn.filtersubs.dropped tagged by reason.

  • ProcessSub: the three existing skip paths (subscription_not_found, user_idle, user_not_visible) kept their DEBUG log level but gained structured fields and dw.esn.processsub.skipped counters. Added a dw.esn.processsub.processed counter on the success path so baseline delivery rate is visible in Datadog.

  • Subscription::process: the configured_for_user and officialpost no-op paths now log at DEBUG with structured fields and increment dw.esn.process.skipped.

  • NotificationInbox::enqueue: inbox overflow eviction now logs at INFO with the user, oldest-kept qid, rows deleted, and max size, and increments dw.esn.inbox.evicted by the number of rows evicted.

Log levels: rare exceptions (filtersubs drop, inbox eviction) at INFO; routine skip paths (user idle, not configured) at DEBUG. Stats counters fire regardless of log level -- Datadog metrics are the primary signal, logs are forensic backup.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications