changelog | Recent Entries

You're viewing

changelog
Create a Dreamwidth Account Learn More

Reload page in style: light

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: d0f4a97384fd81eef7ce77dd92157eef6e473d34 https://github.com/dreamwidth/dreamwidth/commit/d0f4a97384fd81eef7ce77dd92157eef6e473d34 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: M .devcontainer/Dockerfile M .devcontainer/config/etc/dw-etc/config-local.pl M .dockerignore M etc/config-local.pl.example

Log Message:

devcontainer: install Manticore and tighten .dockerignore

Manticore Search is installed but not auto-started, so developers who don't touch search pay no runtime cost. Enable with a commented lj::MANTICORE line in config-local.pl plus searchd --config /etc/manticoresearch/manticore.conf.

.dockerignore now mirrors .gitignore so docker build . is hermetic on dev hosts with local state (extlib, build, ext/local, node_modules, etc.); CI checkouts were already clean.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Commit: 8faae76d08f5ff7e152dfb5084d28257f996fd08 https://github.com/dreamwidth/dreamwidth/commit/8faae76d08f5ff7e152dfb5084d28257f996fd08 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: M bin/worker/sphinx-copier M cgi-bin/DW/Controller/Search/Journal.pm M cgi-bin/DW/Controller/Support/Search.pm A cgi-bin/DW/Search.pm M cgi-bin/DW/Task/SphinxCopier.pm R t/search-manticore.t A t/search.t

Log Message:

Add DW::Search: unify Manticore and Sphinx behind one module

Controllers now call DW::Search::{search_journal,search_support} directly; the Gearman/Storable dispatch lives inside the module. lj::MANTICORE picks the synchronous SphinxQL path, lj::SPHINX_SEARCHD falls through to the legacy Gearman worker, and templates see the same result shape either way.

Both copier paths fan out to SearchCopier when lj::MANTICORE is set — DW::Task::SphinxCopier and the legacy bin/worker/sphinx-copier — so writes hit both backends during migration without having to redeploy the webservers.

t/search.t replaces t/search-manticore.t and targets SphinxQL, since Manticore 25.0 dropped the legacy Sphinx binary API Sphinx::Search used.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/ba7e09ed3116...8faae76d08f5

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: ba7e09ed3116998f6a745752a3a9bc840415912d https://github.com/dreamwidth/dreamwidth/commit/ba7e09ed3116998f6a745752a3a9bc840415912d Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: A t/search-manticore.t

Log Message:

Add t/search-manticore.t: end-to-end test for Sphinx::Search against Manticore

Validates the read path of sphinx-search-gm will work when retargeted at lj::MANTICORE — specifically that Sphinx::Search's binary protocol is accepted on Manticore port 3312, that the filter semantics match what we expect, and that BuildExcerpts works.

The test creates a throwaway RT table (dw1_selftest) in Manticore via SphinxQL CREATE TABLE, populates it with a curated set of docs covering each test scenario, runs queries, and drops the table at END. Skips entirely if lj::MANTICORE isn't configured, so CI without a Manticore instance won't fail.

Covers 15 test cases: 1. Baseline keyword match via Sphinx::Search 2. Phrase match (SPH_MATCH_PHRASE with quoted query) 3. Public entry visible in global search; other entries excluded 4. Private entry hidden by security_bits filter 5. Private entry visible with ignore_security path 6. Friends-only entry: matching allowmask bit returns doc 7. Friends-only entry: non-matching allowmask bit excludes doc 8. is_deleted=1 filtered out 9. allow_global_search=0 excluded from global 10. Journal-scoped search (journalid filter) 11. Comments excluded when SetFilterRange('jtalkid', 0, 0) 12. Comments included when that filter isn't applied 13. Sort by date_posted, ASC and DESC 14. BuildExcerpts returns highlighted keyword snippets 15. Defensive SetFilter('security_bits', [0], 1) exclusion (protects against malformed docs with a 0 in their MVA)

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: b8e8ded3b31d3871f41a68cf0dc160db5ce18d94 https://github.com/dreamwidth/dreamwidth/commit/b8e8ded3b31d3871f41a68cf0dc160db5ce18d94 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: M bin/search-tool M cgi-bin/DW/Task/SearchCopier.pm

Log Message:

SearchCopier: rewrite as direct port of SphinxCopier patterns

The prior SearchCopier took its own shape — bulk selectall_arrayref, ad-hoc chunking, per-doc log lines, wholesale DELETE-then-rebuild per journal — and missed practices SphinxCopier has been using in prod for years. Rewrite it as a near-mechanical port of SphinxCopier, with Manticore-specific deviations only where Manticore's semantics require them.

What's now matched with SphinxCopier:

work() dispatch structure, arg shape (jitemid, jtalkid, jitemids, jtalkids, full recopy), and log messages at INFO
sphinx_db()/manticore_db() opens the connection with a SET NAMES 'utf8' and errstr check
logcroak() after every query against the cluster DB and the search DB, so failures fail the task loudly and the queue retries
Full-recopy entry pass diffs dw1 vs log2 and batch-deletes missing jitemids; does NOT wipe the whole journal up front. Search stays available for the journal during the recopy.
Full-recopy comment pass has the "short path" for

( Error: Irreparable invalid markup in entry. Raw contents behind the cut. )

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 39a9497745cdab9f36c8d8cd669f7457a3595fd6 https://github.com/dreamwidth/dreamwidth/commit/39a9497745cdab9f36c8d8cd669f7457a3595fd6 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: M cgi-bin/DW/Task/SearchCopier.pm

Log Message:

SearchCopier: stream from cluster DBs, match SphinxCopier logging

importfull was doing selectall_arrayref on both log2+logtext2 and talk2+talktext2 for the journal, which loads every row into perl memory before doing anything. Workers were OOMing on real-world accounts.

Switched both loops to prepare + execute + fetchrow_hashref with mysql_use_result=1 so DBD::mysql actually streams rather than buffering the full result set client-side. Also merged the old "fetch metadata, then fetch text in batches of 1000" comment path into a single talk2 + talktext2 join, since we're streaming now. Working memory is bounded at one row at a time plus the %entry_bits map (jitemid -> bits arrayref) kept around for comment security inheritance.

Also upgraded work()'s logging to match SphinxCopier's verbosity so it's actually possible to tell who a job is for from the logs: "Search copier started for [Unknown site tag](), source ." INFO "Requested copy of only entry ." INFO "Requested copy of only comment ." INFO "Requested complete recopy of user." INFO "Copied less than a day ago. Skipping." INFO Start/branch lines emit at INFO; end-of-run summary still DEBUG on clean success and WARN only when there were errors.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 776ac7cd8c2185b53beb87c4c460205d19f00be3 https://github.com/dreamwidth/dreamwidth/commit/776ac7cd8c2185b53beb87c4c460205d19f00be3 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: A .github/workflows/tasks/worker-dw-search-copier-service.json M .github/workflows/worker22-deploy.yml R bin/schedule-copier-jobs A bin/search-tool A bin/worker/dw-search-copier A cgi-bin/DW/Task/SearchCopier.pm M cgi-bin/LJ/DB.pm M config/workers.json M etc/workers.conf

Log Message:

Add DW::Task::SearchCopier path and bin/search-tool

Stand up the new manticore-rt write path side-by-side with the legacy sphinx-copier. Nothing in production dispatches to it yet — only bin/search-tool's import-* subcommands. The two paths can run in parallel through cutover.

cgi-bin/DW/Task/SearchCopier.pm: new task class. Auto-routes to its own SQS queue (dw-prod-dw-task-searchcopier) via class-name derivation. Mirrors SphinxCopier's argument shape (full recopy, single jitemid, single jtalkid) and its security_bits / state / text-decode handling, so the search worker's filter contract stays intact when we eventually flip readers over. Tracks per-run stats (entries/comments/deletes ok/err); summary log is debug on clean success, warn when there are errors. Independent 24h memcache throttle on full recopies (separate key from sphinx-copier's).
bin/worker/dw-search-copier: 36-line runner cloned from dw-sphinx-copier; pulls from the new queue, calls work().
etc/workers.conf: add dw-search-copier: 1 so worker-manager spawns it.
bin/search-tool: CLI helper for the migration. Subcommands import-user, import-all, import-support, search, show, delete, count, flush. import-user delegates to SearchCopier->work() so the CLI and the worker share one code path. import-all replaces the retired bin/schedule-copier-jobs (deleted in this commit).
config/workers.json: register dw-search-copier as an ECS Fargate worker (256 cpu / 512 mb, spot, target 30% cpu, scale 1-10). The .github/workflows/ files are the auto-generated CI artifacts from running update-workflows.py.

LJ/DB.pm: incidental tidyall whitespace-only fixup.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: d1fab43e780fd9a699983326b4f82f034763d16d https://github.com/dreamwidth/dreamwidth/commit/d1fab43e780fd9a699983326b4f82f034763d16d Author: Mark Smith mark@dreamwidth.org Date: 2026-04-21 (Tue, 21 Apr 2026)

Changed paths: M .github/workflows/tasks/worker-birthday-notify-service.json M .github/workflows/tasks/worker-change-poster-id-service.json M .github/workflows/tasks/worker-codebuild-notifier-service.json M .github/workflows/tasks/worker-content-importer-lite-service.json M .github/workflows/tasks/worker-content-importer-service.json M .github/workflows/tasks/worker-content-importer-verify-service.json M .github/workflows/tasks/worker-directory-meta-service.json M .github/workflows/tasks/worker-distribute-invites-service.json M .github/workflows/tasks/worker-dw-change-poster-id-service.json M .github/workflows/tasks/worker-dw-distribute-invites-service.json M .github/workflows/tasks/worker-dw-embeds-service.json M .github/workflows/tasks/worker-dw-esn-cluster-subs-service.json M .github/workflows/tasks/worker-dw-esn-filter-subs-service.json M .github/workflows/tasks/worker-dw-esn-fired-event-service.json M .github/workflows/tasks/worker-dw-esn-process-sub-service.json M .github/workflows/tasks/worker-dw-import-eraser-service.json M .github/workflows/tasks/worker-dw-incoming-email-service.json M .github/workflows/tasks/worker-dw-latest-feed-service.json M .github/workflows/tasks/worker-dw-lazy-cleanup-service.json M .github/workflows/tasks/worker-dw-mass-privacy-service.json M .github/workflows/tasks/worker-dw-send-email-service.json M .github/workflows/tasks/worker-dw-sphinx-copier-service.json M .github/workflows/tasks/worker-dw-support-notify-service.json M .github/workflows/tasks/worker-dw-synsuck-service.json M .github/workflows/tasks/worker-dw-xpost-service.json M .github/workflows/tasks/worker-embeds-service.json M .github/workflows/tasks/worker-expunge-users-service.json M .github/workflows/tasks/worker-import-eraser-service.json M .github/workflows/tasks/worker-import-scheduler-service.json M .github/workflows/tasks/worker-incoming-email-service.json M .github/workflows/tasks/worker-latest-feed-service.json M .github/workflows/tasks/worker-lazy-cleanup-service.json M .github/workflows/tasks/worker-paidstatus-service.json M .github/workflows/tasks/worker-process-privacy-service.json M .github/workflows/tasks/worker-resolve-extacct-service.json M .github/workflows/tasks/worker-schedule-synsuck-service.json M .github/workflows/tasks/worker-ses-incoming-email-service.json M .github/workflows/tasks/worker-shop-creditcard-charge-service.json M .github/workflows/tasks/worker-spellcheck-gm-service.json M .github/workflows/tasks/worker-sphinx-copier-service.json M .github/workflows/tasks/worker-sphinx-search-gm-service.json M .github/workflows/tasks/worker-support-notify-service.json M config/update-workflows.py

Log Message:

Revert self-hosted log shipping, stay on Grafana Cloud

With the planned source-side log volume reduction, Grafana Cloud's per-GB pricing becomes negligible and the operational overhead of self-hosting (bandwidth, maintenance, availability) isn't worth it. Points Fluent Bit back at logs-prod-042.grafana.net with the Loki creds from SSM.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Commit: 28416189e56cdbf33dffa19ff486b97ffdad2848 https://github.com/dreamwidth/dreamwidth/commit/28416189e56cdbf33dffa19ff486b97ffdad2848 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-21 (Tue, 21 Apr 2026)

Changed paths: M cgi-bin/LJ/ESN.pm M etc/log4perl.conf

Log Message:

Reduce log volume: trim log4perl layout and drop duplicate ESN rejection log

Two small changes that together cut log bytes substantially:

etc/log4perl.conf: drop %F:%L %M from the conversion pattern. File, line, and function name added ~80 bytes of framing to every line system-wide; they're rarely useful since the trace prefix and message already identify the code path, and esn-trace searches by trace ID rather than file/line.
cgi-bin/LJ/ESN.pm: remove the filter_reject debug log in unique_matching_subs. The callee-side log in JournalNewComment::matches_filter already logs every rejection with strictly more information (reason, wanted_jtalkid), so the caller- side log was pure duplication. The dw.esn.filter metric stays.

Together these eliminate roughly half of every ESN filter-rejection pair plus ~40% bytes on every log line.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/005e4d64c9d3...28416189e56c

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: c5763c345a2ef2f8e69074808482cb6361f66c7e https://github.com/dreamwidth/dreamwidth/commit/c5763c345a2ef2f8e69074808482cb6361f66c7e Author: Mark Smith mark@dreamwidth.org Date: 2026-04-20 (Mon, 20 Apr 2026)

Changed paths: M cgi-bin/DW/External/Userinfo.pm

Log Message:

Fix stringified site object in extacct stats tag

DW::External::Site->get_site_by_id returns a blessed hashref, and concatenating with "site:" triggered Perl's default stringification, producing values like 'site:DW::External::Site::DeadJournal=HASH(0x...)'. The hash address was effectively a new label value per process, so every worker restart created a fresh set of unique series that lingered in the active-series count until they aged out. Use ->{hostname} to emit stable values like 'site:livejournal.com'.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Commit: 005e4d64c9d3d9359de4cff0c3232d2887b06632 https://github.com/dreamwidth/dreamwidth/commit/005e4d64c9d3d9359de4cff0c3232d2887b06632 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-21 (Tue, 21 Apr 2026)

Changed paths: M .github/workflows/tasks/worker-birthday-notify-service.json M .github/workflows/tasks/worker-change-poster-id-service.json M .github/workflows/tasks/worker-codebuild-notifier-service.json M .github/workflows/tasks/worker-content-importer-lite-service.json M .github/workflows/tasks/worker-content-importer-service.json M .github/workflows/tasks/worker-content-importer-verify-service.json M .github/workflows/tasks/worker-directory-meta-service.json M .github/workflows/tasks/worker-distribute-invites-service.json M .github/workflows/tasks/worker-dw-change-poster-id-service.json M .github/workflows/tasks/worker-dw-distribute-invites-service.json M .github/workflows/tasks/worker-dw-embeds-service.json M .github/workflows/tasks/worker-dw-esn-cluster-subs-service.json M .github/workflows/tasks/worker-dw-esn-filter-subs-service.json M .github/workflows/tasks/worker-dw-esn-fired-event-service.json M .github/workflows/tasks/worker-dw-esn-process-sub-service.json M .github/workflows/tasks/worker-dw-import-eraser-service.json M .github/workflows/tasks/worker-dw-incoming-email-service.json M .github/workflows/tasks/worker-dw-latest-feed-service.json M .github/workflows/tasks/worker-dw-lazy-cleanup-service.json M .github/workflows/tasks/worker-dw-mass-privacy-service.json M .github/workflows/tasks/worker-dw-send-email-service.json M .github/workflows/tasks/worker-dw-sphinx-copier-service.json M .github/workflows/tasks/worker-dw-support-notify-service.json M .github/workflows/tasks/worker-dw-synsuck-service.json M .github/workflows/tasks/worker-dw-xpost-service.json M .github/workflows/tasks/worker-embeds-service.json M .github/workflows/tasks/worker-expunge-users-service.json M .github/workflows/tasks/worker-import-eraser-service.json M .github/workflows/tasks/worker-import-scheduler-service.json M .github/workflows/tasks/worker-incoming-email-service.json M .github/workflows/tasks/worker-latest-feed-service.json M .github/workflows/tasks/worker-lazy-cleanup-service.json M .github/workflows/tasks/worker-paidstatus-service.json M .github/workflows/tasks/worker-process-privacy-service.json M .github/workflows/tasks/worker-resolve-extacct-service.json M .github/workflows/tasks/worker-schedule-synsuck-service.json M .github/workflows/tasks/worker-ses-incoming-email-service.json M .github/workflows/tasks/worker-shop-creditcard-charge-service.json M .github/workflows/tasks/worker-spellcheck-gm-service.json M .github/workflows/tasks/worker-sphinx-copier-service.json M .github/workflows/tasks/worker-sphinx-search-gm-service.json M .github/workflows/tasks/worker-support-notify-service.json M config/update-workflows.py

Log Message:

Switch to self-hosted log service

I don't really want to pay hundreds/month for storing logs and I have a server hanging out that has fast disks/RAM, might as well just self-host it. It's not production so if it dies, I can always revert/fix.

Compare: https://github.com/dreamwidth/dreamwidth/compare/ad345841bf9b...005e4d64c9d3

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 6dc7b32ef3eed8378f40270d35c3a0b7a45dd21b https://github.com/dreamwidth/dreamwidth/commit/6dc7b32ef3eed8378f40270d35c3a0b7a45dd21b Author: Mark Smith mark@dreamwidth.org Date: 2026-04-20 (Mon, 20 Apr 2026)

Changed paths: M cgi-bin/DW/External/Userinfo.pm

Log Message:

Drop high-cardinality username tag from extacct stats

The username:$user tag on dw.worker.extacct.{success,failure} tracked each remote external-site user individually, making it the top metric by active series count in Grafana Cloud billing. Site alone is bounded to the DW::External::Site enum and gives the actionable dimension.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Commit: ad345841bf9b424ceb6bf65be2136b88fb612301 https://github.com/dreamwidth/dreamwidth/commit/ad345841bf9b424ceb6bf65be2136b88fb612301 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-20 (Mon, 20 Apr 2026)

Changed paths: M cgi-bin/DW/Controller/Importer.pm M cgi-bin/DW/Logic/Importer.pm

Log Message:

Validate importer hostname against source whitelist

The /tools/importer UI offered a dropdown of three allowed sources (livejournal.com, insanejournal.com, dreamwidth.org), but set_import_data_for_user accepted whatever hostname the POST carried and INSERTed it straight into import_data. A crafted POST could inject arbitrary hostnames, which then flowed into the new hostname: tag on dw.worker.importer.job_completed as a cardinality-injection vector.

Extracts the allowed-source list into DW::Logic::Importer->allowed_sources so the controller's dropdown rendering and the logic layer's validation share one definition, and rejects any hostname not in the list.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/00b8f85a98e0...ad345841bf9b

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 00b8f85a98e039eaaf2046ccb2bbc7e1ff07d60e https://github.com/dreamwidth/dreamwidth/commit/00b8f85a98e039eaaf2046ccb2bbc7e1ff07d60e Author: Mark Smith mark@dreamwidth.org Date: 2026-04-19 (Sun, 19 Apr 2026)

Changed paths: M app.psgi M bin/starman A cgi-bin/Plack/Middleware/DW/AccessLog.pm

Log Message:

Add JSON access log middleware for Grafana Loki

Replace Starman's default Apache Combined access log with a DW::AccessLog middleware that emits one JSON object per line to psgi.errors. Fields include method, path, status, bytes, duration_ms, host, remote_ip, and user_agent — all natively parseable by Loki's | json pipeline for dashboards.

Use --no-default-middleware to suppress Plack's built-in AccessLog, and enable ContentLength explicitly in app.psgi to preserve that behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: a19cc8c787fca5419d8469a5977323ed68c450c2 https://github.com/dreamwidth/dreamwidth/commit/a19cc8c787fca5419d8469a5977323ed68c450c2 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M src/dwtool/main.go

Log Message:

Improve esn-trace URL resolution with userid lookup and precise Loki queries

Resolve the journal's userid from their profile page so we can build exact Loki regex queries matching [esn ETYPEID:JOURNALID:JTALKID:ARG2] instead of broad substring matches on ":JTALKID:" which produced false positives.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: b1e684b607034fa480f654db748922660b063d9d https://github.com/dreamwidth/dreamwidth/commit/b1e684b607034fa480f654db748922660b063d9d Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M cgi-bin/DW/Task/ESN/FiredEvent.pm M cgi-bin/LJ/DB.pm

Log Message:

Fix ESN dropping notifications due to replication lag on comment load

FiredEvent reads comment data to determine which users' Reply subscriptions are relevant. LJ::Comment::preload_rows loads via get_cluster_reader, which hits the replica — but the comment was just written to the master and may not have replicated yet. When this race is lost, the comment fails to load, relevantuserids returns empty, Reply subscriptions are excluded, and the notification is silently dropped.

Fix: wrap the subscription lookup in LJ::DB::require_master so all reads go to the cluster master. Also extend get_cluster_reader to respect the existing PRAGMAFORCE_MASTER flag, which previously only affected global (non-cluster) readers.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/eae7aaa21ba6...b1e684b60703

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 439d836de1e77290f029a08afebd37d4b177f74f https://github.com/dreamwidth/dreamwidth/commit/439d836de1e77290f029a08afebd37d4b177f74f Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: A src/dwtool/internal/config/loki.go A src/dwtool/internal/loki/client.go M src/dwtool/main.go

Log Message:

Switch dwtool log-scan and esn-trace from CloudWatch to Loki

Logs now go directly to Grafana Cloud Loki via Fluent Bit, so dwtool needs to query Loki instead of CloudWatch. Adds a Loki HTTP client that queries /loki/api/v1/query_range with LogQL, and rewrites both log-scan and esn-trace to use it.

Credentials are loaded from ~/.config/dwtool/config.json (with DWTOOL_LOKI_HOST/USER/PASSWORD env var overrides). The Loki client strips Fluent Bit key_value wrapper and log4perl timestamp/level prefix from messages so output is clean.

Other fixes: - Renamed logscan → log-scan for consistency with esn-trace - Fixed flag parsing in esn-trace: Go's flag package stops at the first positional arg, so esn-trace URL -since 24h silently ignored -since. Args are now reordered so flags come first. - log-scan uses -service instead of -groups for filtering by Loki service label instead of CloudWatch log group glob - Service column is hidden when -service is specified (redundant)

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: eae7aaa21ba6b5d07a1ed3378505a9a0b156b995 https://github.com/dreamwidth/dreamwidth/commit/eae7aaa21ba6b5d07a1ed3378505a9a0b156b995 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M .github/workflows/tasks/web-canary-service.json M .github/workflows/tasks/web-shop-service.json M .github/workflows/tasks/web-stable-service.json M .github/workflows/tasks/web-unauthenticated-service.json

Log Message:

remove cloudwatch agent sidecar from web task definitions

The CW agent was configured to collect Apache access logs but was pointing at a -DISABLED log path, so it was burning resources for nothing. Removes the sidecar container, log-share volume, and associated mount from all four web service task definitions.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/b046642d6f54...eae7aaa21ba6

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: b046642d6f54f301e4f4a128472cc9edfa8fd80c https://github.com/dreamwidth/dreamwidth/commit/b046642d6f54f301e4f4a128472cc9edfa8fd80c Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M .github/workflows/tasks/worker-birthday-notify-service.json M .github/workflows/tasks/worker-change-poster-id-service.json M .github/workflows/tasks/worker-codebuild-notifier-service.json M .github/workflows/tasks/worker-content-importer-lite-service.json M .github/workflows/tasks/worker-content-importer-service.json M .github/workflows/tasks/worker-content-importer-verify-service.json M .github/workflows/tasks/worker-directory-meta-service.json M .github/workflows/tasks/worker-distribute-invites-service.json M .github/workflows/tasks/worker-dw-change-poster-id-service.json M .github/workflows/tasks/worker-dw-distribute-invites-service.json M .github/workflows/tasks/worker-dw-embeds-service.json M .github/workflows/tasks/worker-dw-esn-cluster-subs-service.json M .github/workflows/tasks/worker-dw-esn-filter-subs-service.json M .github/workflows/tasks/worker-dw-esn-fired-event-service.json M .github/workflows/tasks/worker-dw-esn-process-sub-service.json M .github/workflows/tasks/worker-dw-import-eraser-service.json M .github/workflows/tasks/worker-dw-incoming-email-service.json M .github/workflows/tasks/worker-dw-latest-feed-service.json M .github/workflows/tasks/worker-dw-lazy-cleanup-service.json M .github/workflows/tasks/worker-dw-mass-privacy-service.json M .github/workflows/tasks/worker-dw-send-email-service.json M .github/workflows/tasks/worker-dw-sphinx-copier-service.json M .github/workflows/tasks/worker-dw-support-notify-service.json M .github/workflows/tasks/worker-dw-synsuck-service.json M .github/workflows/tasks/worker-dw-xpost-service.json M .github/workflows/tasks/worker-embeds-service.json M .github/workflows/tasks/worker-expunge-users-service.json M .github/workflows/tasks/worker-import-eraser-service.json M .github/workflows/tasks/worker-import-scheduler-service.json M .github/workflows/tasks/worker-incoming-email-service.json M .github/workflows/tasks/worker-latest-feed-service.json M .github/workflows/tasks/worker-lazy-cleanup-service.json M .github/workflows/tasks/worker-paidstatus-service.json M .github/workflows/tasks/worker-process-privacy-service.json M .github/workflows/tasks/worker-resolve-extacct-service.json M .github/workflows/tasks/worker-schedule-synsuck-service.json M .github/workflows/tasks/worker-ses-incoming-email-service.json M .github/workflows/tasks/worker-shop-creditcard-charge-service.json M .github/workflows/tasks/worker-spellcheck-gm-service.json M .github/workflows/tasks/worker-sphinx-copier-service.json M .github/workflows/tasks/worker-sphinx-search-gm-service.json M .github/workflows/tasks/worker-support-notify-service.json M config/update-workflows.py

Log Message:

Ship worker logs directly to Grafana Cloud Loki via Fluent Bit

Regenerated all 42 worker task definitions to use awsfirelens with a Fluent Bit log_router sidecar instead of the awslogs driver. Worker logs now go directly to Grafana Cloud Loki over HTTPS, bypassing CloudWatch Logs entirely.

Each task definition gains a log_router container (~15MB RSS) running aws-for-fluent-bit:3 (Fluent Bit v4.2.2, the 3.x line based on AL2023 — the 2.x line ships Fluent Bit 1.9 which has no Loki plugin). Loki credentials are injected via SSM secrets at task startup. The worker container's logDriver changes from awslogs to awsfirelens with the native loki output plugin.

Fluent Bit's own logs still go to CloudWatch (/dreamwidth/fluent-bit) so the log router itself can be debugged if needed.

Generated by config/update-workflows.py from config/workers.json.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: ece0b7970913514a534e88d20b8383a4555ebe5a https://github.com/dreamwidth/dreamwidth/commit/ece0b7970913514a534e88d20b8383a4555ebe5a Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M .github/workflows/tasks/worker-birthday-notify-service.json M .github/workflows/tasks/worker-change-poster-id-service.json M .github/workflows/tasks/worker-codebuild-notifier-service.json M .github/workflows/tasks/worker-content-importer-lite-service.json M .github/workflows/tasks/worker-content-importer-service.json M .github/workflows/tasks/worker-content-importer-verify-service.json M .github/workflows/tasks/worker-directory-meta-service.json M .github/workflows/tasks/worker-distribute-invites-service.json M .github/workflows/tasks/worker-dw-change-poster-id-service.json M .github/workflows/tasks/worker-dw-distribute-invites-service.json M .github/workflows/tasks/worker-dw-embeds-service.json M .github/workflows/tasks/worker-dw-esn-cluster-subs-service.json M .github/workflows/tasks/worker-dw-esn-filter-subs-service.json M .github/workflows/tasks/worker-dw-esn-fired-event-service.json M .github/workflows/tasks/worker-dw-esn-process-sub-service.json M .github/workflows/tasks/worker-dw-import-eraser-service.json M .github/workflows/tasks/worker-dw-incoming-email-service.json M .github/workflows/tasks/worker-dw-latest-feed-service.json M .github/workflows/tasks/worker-dw-lazy-cleanup-service.json M .github/workflows/tasks/worker-dw-mass-privacy-service.json M .github/workflows/tasks/worker-dw-send-email-service.json M .github/workflows/tasks/worker-dw-sphinx-copier-service.json M .github/workflows/tasks/worker-dw-support-notify-service.json M .github/workflows/tasks/worker-dw-synsuck-service.json M .github/workflows/tasks/worker-dw-xpost-service.json M .github/workflows/tasks/worker-embeds-service.json M .github/workflows/tasks/worker-expunge-users-service.json M .github/workflows/tasks/worker-import-eraser-service.json M .github/workflows/tasks/worker-import-scheduler-service.json M .github/workflows/tasks/worker-incoming-email-service.json M .github/workflows/tasks/worker-latest-feed-service.json M .github/workflows/tasks/worker-lazy-cleanup-service.json M .github/workflows/tasks/worker-paidstatus-service.json M .github/workflows/tasks/worker-process-privacy-service.json M .github/workflows/tasks/worker-resolve-extacct-service.json M .github/workflows/tasks/worker-schedule-synsuck-service.json M .github/workflows/tasks/worker-ses-incoming-email-service.json M .github/workflows/tasks/worker-shop-creditcard-charge-service.json M .github/workflows/tasks/worker-spellcheck-gm-service.json M .github/workflows/tasks/worker-sphinx-copier-service.json M .github/workflows/tasks/worker-sphinx-search-gm-service.json M .github/workflows/tasks/worker-support-notify-service.json M config/update-workflows.py

Log Message:

Ship worker logs directly to Grafana Cloud Loki via Fluent Bit

Regenerated all 42 worker task definitions to use awsfirelens with a Fluent Bit log_router sidecar instead of the awslogs driver. Worker logs now go directly to Grafana Cloud Loki over HTTPS, bypassing CloudWatch Logs entirely.

Each task definition gains a log_router container (~15MB RSS) running the AWS-maintained Fluent Bit image from private ECR. Loki credentials are injected via SSM secrets at task startup. The worker container's logDriver changes from awslogs to awsfirelens with the grafana-loki output plugin.

Fluent Bit's own logs still go to CloudWatch (/dreamwidth/fluent-bit) so the log router itself can be debugged if needed.

Generated by config/update-workflows.py from config/workers.json.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: 0d49a7588ae0544ef51801c977e8df41c651e3b5 https://github.com/dreamwidth/dreamwidth/commit/0d49a7588ae0544ef51801c977e8df41c651e3b5 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M cgi-bin/DW/Worker/ContentImporter.pm M cgi-bin/DW/Worker/ContentImporter/LiveJournal.pm

Log Message:

tidy

Compare: https://github.com/dreamwidth/dreamwidth/compare/d7d5b8fbd0bb...0d49a7588ae0

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: d7d5b8fbd0bb1063688f06cc820d3e65bae09a45 https://github.com/dreamwidth/dreamwidth/commit/d7d5b8fbd0bb1063688f06cc820d3e65bae09a45 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-12 (Sun, 12 Apr 2026)

Changed paths: M cgi-bin/DW/Task/ESN/ProcessSub.pm M cgi-bin/LJ/ESN.pm M etc/config-local.pl.example

Log Message:

Auto-deactivate ESN subscriptions for idle users

When processing notifications, deactivate subscriptions that belong to users who have been idle for over a year ($LJ::ESN_INACTIVE_DAYS, default 365). This happens in two places:

LJ::ESN::unique_matching_subs: if a sub requires a paid capability the user no longer has (e.g. thread tracking after paid expired) AND the user is idle, deactivate the sub before it reaches matches_filter.
DW::Task::ESN::ProcessSub: if a user is idle, deactivate the sub after the first skip so it never generates another SQS job.

Deactivated subs set the INACTIVE flag, NOT deleted. The user can still see them on /manage/tracking and reactivate them when they return. LJ::Subscription::create also automatically reactivates matching inactive subs when a user re-subscribes.

The idle threshold is configurable via $LJ::ESN_INACTIVE_DAYS in config-local.pl (documented in the example file). The same variable is used for the existing ProcessSub idle-skip check, replacing a previously hardcoded 365.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 51d098321baa231958385df0c938af13be280ccb https://github.com/dreamwidth/dreamwidth/commit/51d098321baa231958385df0c938af13be280ccb Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M .devcontainer/devcontainer.json M .devcontainer/setup.sh M CLAUDE.md

Log Message:

Support parallel worktree devcontainers for multi-Claude workflows

Three fixes that allow multiple devcontainers to run simultaneously from git worktrees without conflicts:

Add workspaceMount to always bind-mount at /workspaces/dreamwidth regardless of host folder name (worktrees have different names)
Use dynamic host ports (-p 0:8080) instead of hardcoded 8080:8080 so multiple containers don't fight over the same port
Give each workspace its own MySQL volume to prevent DB corruption
Symlink extlib from the image when it doesn't exist on disk (worktrees don't have extlib/ since it's gitignored)

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: 3fcdc3a52918fb76bc84761b4428b66cce064066 https://github.com/dreamwidth/dreamwidth/commit/3fcdc3a52918fb76bc84761b4428b66cce064066 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/LJ/Comment.pm

Log Message:

Add per-check diagnostic logging to Comment::visible_to

When called during ESN processing (CURRENT_TRACE is set), visible_to now logs which specific check caused the rejection: entry_not_loaded, entry_not_visible (with security level), screened, or poster_suspended. Previously all four cases produced a generic not_visible_to_watcher in the matches_filter trace with no way to distinguish them after the fact.

Triggered by a real case (trace 3:2296509:6184:0) where all 40 subscribers were rejected with not_visible_to_watcher on a public unscreened comment in a personal journal — couldn't determine root cause without per-check visibility.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/1c54be1b97b9...3fcdc3a52918

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 1c54be1b97b9d64d8833b984f942a9b66d326643 https://github.com/dreamwidth/dreamwidth/commit/1c54be1b97b9d64d8833b984f942a9b66d326643 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Task/ESN/FilterSubs.pm M cgi-bin/DW/Task/ESN/FindSubsByCluster.pm M src/dwtool/main.go

Log Message:

Fix false send failures and add CLI improvements

FilterSubs and FindSubsByCluster were passing the result of tasks_of_unique_matching_subs() directly to DW::TaskQueue->send(), which returns undef on an empty task list. When all subscriptions were legitimately filtered out by matches_filter, this was counted as a "failed send" (~47 false failures/min). Split the calls so empty task lists are a normal completion, not an error.

Also adds top-level help to dwtool listing available subcommands, and teaches esn-trace to accept a comment URL (parses ?thread= param, converts dtalkid to jtalkid, searches fired-event logs for the full trace ID automatically).

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: c3394d8c9d9ca1cbd8e169d3a1e85849e9affbaa https://github.com/dreamwidth/dreamwidth/commit/c3394d8c9d9ca1cbd8e169d3a1e85849e9affbaa Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Worker/ContentImporter.pm

Log Message:

Add DW::Stats metrics to content importer job outcomes

Instrument all four outcome methods (ok, fail, temp_fail, decline) in DW::Worker::ContentImporter with a single metric dw.worker.importer.job_completed, using labels for result, item, and hostname to support Prometheus-style querying.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 034f12e28b75c1c3b3ce7b6ab173616484e33ee5 https://github.com/dreamwidth/dreamwidth/commit/034f12e28b75c1c3b3ce7b6ab173616484e33ee5 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Task/ESN/FilterSubs.pm M cgi-bin/DW/Task/ESN/FindSubsByCluster.pm M cgi-bin/DW/Task/ESN/FiredEvent.pm M cgi-bin/DW/Task/ESN/ProcessSub.pm M cgi-bin/LJ/ESN.pm M cgi-bin/LJ/Event/JournalNewComment.pm M cgi-bin/LJ/NotificationInbox.pm M cgi-bin/LJ/Subscription.pm M src/dwtool/main.go

Log Message:

Add deterministic ESN trace IDs and full pipeline observability

Every ESN log line now carries a deterministic trace prefix [esn ETYPEID:JOURNALID:ARG1:ARG2] derived from the event's raw_params, making end-to-end tracing possible by grepping a single string across all four ESN worker log groups. The trace ID is computable from a comment URL without needing to capture it at event-fire time.

Pipeline changes: - All four ESN task workers (FiredEvent, FindSubsByCluster, FilterSubs, ProcessSub) prefix every log line with the trace string - LJ::ESN::unique_matching_subs replaces the silent grep with a loop that logs each matches_filter rejection at DEBUG with user/sub/etypeid - JournalNewComment::matches_filter logs specific rejection reasons at each of its 9 return-0 paths via a $reject->() closure - Subscription::process logs notification construction failures and not_configured_for_user skips with trace prefix - NotificationInbox::enqueue logs successful delivery confirmation - FindSubsByCluster now checks TaskQueue->send() return values and logs users with >5000 subs that get silently skipped - FilterSubs now checks TaskQueue->send() return value - ProcessSub dead code fixed: subscription-not-found is correctly handled as a skip (COMPLETED) not routed through $failed (FAILED)

Metrics consolidation (Prometheus best practice): - All DW::Stats calls use one metric name per stage with a result tag instead of encoding the result in the metric name - dw.esn.firedevent result:started|completed|failed - dw.esn.findsubsbycluster result:started|completed|failed - dw.esn.filtersubs result:completed|dropped|failed - dw.esn.filter result:rejected - dw.esn.matches_filter result:rejected (with reason tag) - dw.esn.processsub result:processed|skipped|failed - dw.esn.process result:failed|skipped - dw.esn.inbox result:delivered|evicted

CLI tooling: - dwtool esn-trace searches all four ESN log groups in CloudWatch and pretty-prints a cross-stage timeline

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 62c0b8e4ee1d12728c8f737eac0061161522fdaa https://github.com/dreamwidth/dreamwidth/commit/62c0b8e4ee1d12728c8f737eac0061161522fdaa Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M src/dwtool/internal/aws/ecs.go

Log Message:

Remove legacy web image from web-canary and web-unauthenticated deploy targets

Both services are now fully on web22. This removes the old "web" option from the deploy flow so only web22 is offered, matching web-shop.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: 58ffd06e3da3ae4ea7b3122dac7a46685a4c35e4 https://github.com/dreamwidth/dreamwidth/commit/58ffd06e3da3ae4ea7b3122dac7a46685a4c35e4 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M .gitignore

Log Message:

.gitignore updates

Commit: 7cbc4931b3974ac99de9e3657f389ed8125a5566 https://github.com/dreamwidth/dreamwidth/commit/7cbc4931b3974ac99de9e3657f389ed8125a5566 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Worker/ContentImporter/LiveJournal.pm M doc/dependencies-cpanm

Log Message:

Fix content-importer-verify TLS connection failures

The importer's XMLRPC calls to remote LJ-based sites were failing with "SSL wants a read first" due to a TLS 1.3 handshake issue with IO::Socket::SSL 2.098 and OpenSSL 3.0 (see openssl/openssl#7967). Force TLS 1.2 in call_xmlrpc as a workaround.

Also add missing Regexp::IPv6 dependency, which IO::Socket::IP needs for address parsing — its absence caused a secondary "Can't locate Regexp/IPv6.pm" failure.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Commit: 9eff707e837547024e178f5b0b8cbc30c6653350 https://github.com/dreamwidth/dreamwidth/commit/9eff707e837547024e178f5b0b8cbc30c6653350 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: A bin/dev/importer-debug M bin/ecs-shell

Log Message:

Add importer-debug diagnostic script and fix ecs-shell on macOS

Add bin/dev/importer-debug for diagnosing content-importer connectivity issues on ECS containers. Tests DNS, TCP, TLS, HTTP, and XMLRPC layers independently with timing and detailed error output.

Fix ecs-shell exec command failing on macOS due to base64 -w0 flag not being supported (macOS uses base64 -i instead).

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Compare: https://github.com/dreamwidth/dreamwidth/compare/388d47d72bfa...9eff707e8375

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 388d47d72bfa348ef6378e9f5e638e9fa85da491 https://github.com/dreamwidth/dreamwidth/commit/388d47d72bfa348ef6378e9f5e638e9fa85da491 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-11 (Sat, 11 Apr 2026)

Changed paths: M cgi-bin/DW/Task.pm M cgi-bin/DW/TaskQueue/LocalDisk.pm M cgi-bin/DW/TaskQueue/SQS.pm M etc/config-local.pl.example A t/taskqueue-serialize.t A t/taskqueue.t

Log Message:

Add JSON wire format for task queue with gradual rollout support

Adds serialize/deserialize methods to DW::Task that support a new v2 JSON wire format alongside the legacy Storable format. This enables non-Perl consumers (e.g. Go workers) to read task queue messages.

$LJ::TASK_QUEUE_JSON controls the rollout as a float 0-1 (e.g. 0.01 = 1% of messages serialized as JSON). Default is 0 (all Storable). All consumers can read both formats regardless of this setting. If JSON encoding fails for a task (e.g. blessed objects in args), it falls back to Storable with a warning and stat tag identifying the task class.

Also adds task queue config documentation to config-local.pl.example and comprehensive test coverage for LocalDisk lifecycle, serialization round-trips, and queue_attributes.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications

Profile

Dreamwidth Changelog

April 2026

S	M	T	W	T	F	S
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30

Syndicate

Most Popular Tags

Page Summary

github - [dreamwidth/dreamwidth] d0f4a9: devcontainer: install Manticore and tighten .docke...
github - [dreamwidth/dreamwidth] ba7e09: Add t/search-manticore.t: end-to-end test for Sphi...
github - [dreamwidth/dreamwidth] b8e8de: SearchCopier: rewrite as direct port of SphinxCopi...
github - [dreamwidth/dreamwidth] 39a949: SearchCopier: stream from cluster DBs, match Sphin...
github - [dreamwidth/dreamwidth] 776ac7: Add DW::Task::SearchCopier path and bin/search-tool
github - [dreamwidth/dreamwidth] d1fab4: Revert self-hosted log shipping, stay on Grafana C...
github - [dreamwidth/dreamwidth] c5763c: Fix stringified site object in extacct stats tag
github - [dreamwidth/dreamwidth] 6dc7b3: Drop high-cardinality username tag from extacct stats
github - [dreamwidth/dreamwidth] 00b8f8: Add JSON access log middleware for Grafana Loki
github - [dreamwidth/dreamwidth] a19cc8: Improve esn-trace URL resolution with userid looku...
github - [dreamwidth/dreamwidth] 439d83: Switch dwtool log-scan and esn-trace from CloudWat...
github - [dreamwidth/dreamwidth] b04664: Ship worker logs directly to Grafana Cloud Loki vi...
github - [dreamwidth/dreamwidth] ece0b7: Ship worker logs directly to Grafana Cloud Loki vi...
github - [dreamwidth/dreamwidth] d7d5b8: Auto-deactivate ESN subscriptions for idle users
github - [dreamwidth/dreamwidth] 51d098: Support parallel worktree devcontainers for multi-...
github - [dreamwidth/dreamwidth] 1c54be: Fix false send failures and add CLI improvements
github - [dreamwidth/dreamwidth] c3394d: Add DW::Stats metrics to content importer job outc...
github - [dreamwidth/dreamwidth] 034f12: Add deterministic ESN trace IDs and full pipeline ...
github - [dreamwidth/dreamwidth] 62c0b8: Remove legacy web image from web-canary and web-un...
github - [dreamwidth/dreamwidth] 388d47: Add JSON wire format for task queue with gradual r...

Active Entries

Style Credit

Style: Dark Purple for Funky Circles by ninetydegrees
Resources: Smoke Curl

Expand Cut Tags

No cut tags

Page generated Apr. 25th, 2026 09:03 am

Powered by Dreamwidth Studios