![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png){kind=small}
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
githubDump the FAQs and their edit history for archiving
FAQs live only in the database and are edited in place: Admin/FAQ.pm does UPDATE faq SET ... lastmodtime=NOW(), keeping only who last touched a FAQ and when, never the prior text.
The revision history does exist, though, in the translation tables. Every
edit also calls LJ::Lang::set_text for
Dump the faq and faqcat tables plus that history, joined against ml_items / ml_langs with an is_current flag. Output is JSON Lines because answers are HTML containing newlines, and JSON is given utf8(0) against a :raw handle because DW never sets mysql_enable_utf8 -- DBI returns UTF-8 bytes, and encoding them again would double-encode every non-ASCII character.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Commit: 9bcb3dea Author: Mark Smith
Dump the stats tables for archiving
Aug. 5th, 2026 05:57 amgithubDump the stats tables for archiving
/stats reads the stats table and /stats/site reads site_stats, whose
category/key columns are opaque ids resolved through the statkeylist
typemap. Only the rendered files from genstats were being archived, so the
data behind /stats/site was not preserved at all and site_stats alone would
have been unreadable.
Dump both to TSV, joining site_stats against statkeylist so the output stands on its own. mysql_use_result keeps memory flat at the ljlib baseline (~231 MB) instead of buffering the ~425k row result client-side (~273 MB), which matters in a 512 MB cron task.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Commit: 10a614c3 Author: Mark Smith
Point archive-to-s3.pl at extlib
Aug. 5th, 2026 05:37 amgithubPoint archive-to-s3.pl at extlib
The cron execs the script directly rather than going through ljlib.pl, so
inc has no extlib and Paws is not found. Same 'use lib' as checkconfig.pl
and ljumover.pl, the other scripts that run standalone.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Commit: fb60df09 Author: Mark Smith
Make archive-to-s3.pl executable
Aug. 5th, 2026 05:29 amgithubMake archive-to-s3.pl executable
startup-cron.sh execs the script directly, so without the mode bit the cron task exits 126 (cannot execute) before the script ever runs.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Commit: 02da982b Author: Mark Smith
Archive cron job output to S3
Aug. 5th, 2026 05:11 amgithubArchive cron job output to S3
Cron tasks run as one-shot Fargate tasks on an ephemeral filesystem, so anything a job writes to disk is discarded when the task exits. genstats writes htdocs/stats/stats.txt and genstatspics writes newbyday.png; both have been going nowhere since the crons moved off the old cron host, which is why /stats/stats.txt has served a file frozen at 2026-07-04.
Add bin/archive-to-s3.pl, which copies files or directories to the
dreamwidth-archive bucket under
Producing a file and shipping it have to happen in the same task to share the same disk, so startup-cron.sh now accepts "--" between chained commands. Single-command crons are unaffected; a failing step aborts the chain and its exit code becomes the task's.
The matching cron command and ARCHIVE_BUCKET/ARCHIVE_REGION environment come from dw-terraform, and must be applied only after this ships in worker22:latest.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Commit: e7c5d7f6 Author: Mark Smith
githubClean up the control strip HTML/CSS some (#3657)
Clean up the control strip HTML/CSS some
Clean up my editing attempts, too
githubDecommission the orphaned spellcheck-gm worker
3563 inlined the synchronous spellcheck helper and deleted
bin/worker/spellcheck-gm, but left the worker-spellcheck-gm ECS service, its config/workers.json entry, and its generated task def behind. The service can only run pre-#3563 images and crash-loops (exec: no such file) on any newer worker22 build, so it's the one worker that never converges on a fleet deploy.
Remove it from workers.json (the single source of truth), regenerate
worker22-deploy.yml, and drop the stale task def. dw-terraform derives its
worker set from this file over HTTP, so terraform apply will destroy the
service and its log group on the next run.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Commit: 04a64078 Author: Mark Smith
githubBatch unbounded DELETEs in clean_caches.pl (#3651)
Initial plan
Batch unbounded DELETEs and talkleft_xfp migration in clean_caches.pl
Each large DELETE against the global master (authactions, faquses, duplock, blobcache) and the talkleft_xfp migration loop now run in bounded LIMIT 1000 batches with a 1-second sleep between passes, mirroring the pattern already used for random_user_set cleanup. This prevents any single statement from holding the master long enough to stall the web tier during the daily maintenance window.
Also consolidates the previously-redundant my $count re-declarations
into a single declaration at the top of the authactions block.
Fixes dreamwidth/dreamwidth#3650
- Revert talkleft_xfp migration to original single-pass behavior
Keep the bounded LIMIT+sleep DELETE loops for authactions, faquses, duplock, and blobcache (the real fix for master saturation), but restore the talkleft_xfp section to the original single-pass approach: one SELECT LIMIT 1000, move those rows, print "rows remaining", and stop.
This avoids the infinite-loop risk introduced by the while(1) version, where a down cluster or insert/delete error would cause the same stuck rows to be re-selected indefinitely.
githubStandardize in-process caching behind DW::Cache (request + process scopes) (#3652)
- Standardize request-scoped caching behind DW::RequestCache
Request caches lived in a sprawl of package globals (%LJ::REQ_CACHE*, %LJ::REQUEST_CACHE, %LJ::S2::REQ_CACHE_*, etc.) wiped by a hand-maintained clear-list in LJ::start_request. That list drifted -- %LJ::REQ_CACHE (UniqCookie) and %LJ::REQ_CACHE_POLL already escaped it and leaked across requests on persistent workers -- and DW::CacheStats kept a second, separately drifting copy for size sampling. The SQS task workers never called start_request at all, so every request cache lived for the whole worker process.
Introduce DW::RequestCache: one module whose single clear() empties everything routed through it, so a cache added here cannot leak. It offers a namespaced KV memoization API (get/set/memoize/remove/clear_ns) plus a registration API (register_var/register_reset) for state that keeps direct package-var access. LJ::start_request's clear-block collapses to one DW::RequestCache->clear call, and DW::CacheStats now samples the same registry it clears (via ->registered), so the two sets can no longer drift.
Migrate all existing request caches onto the interface (users, rel, usertags, trustmask, S2 style/layer/layer-info, poll, uniqcookie, langdatfile, OAuth consumer/access) and register the remaining scratch/accumulator state and singleton resets. Wire DW::TaskQueue::start_work to wrap each job in start_request/end_request, matching the legacy TheSchwartz/Gearman workers; a scope guard guarantees end_request runs on the die and timeout exit paths.
Tests reaching into the old package vars are updated to the new API. Adds t/request-cache.t covering the KV API, registration, the clear() guarantee, and per-request/per-job isolation via LJ::start_request.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Address review: guard ScopeGuard DESTROY and clear OAuth consumer by id
DW::TaskQueue::ScopeGuard::DESTROY now runs its cleanup under local $@ + eval, so a die in LJ::end_request can't mask a job's own exception during unwinding.
- DW::OAuth::Consumer::deletecache clears the request-cache entry keyed by id as well as by token (the consumer is cached under both), matching the memcache invalidation right above it.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Rework into DW::Cache with request/process scopes; retire DW::CacheStats
Replace DW::RequestCache with DW::Cache, one facade over two scope singletons sharing a single store implementation: DW::Cache->request (wiped per web request / background job by LJ::start_request) and DW::Cache->process (global reference data -- props, moods, codes, styles, translations -- wiped on config reload by LJ::handle_caches, whose hand-maintained clear-list collapses to one process->clear the same way start_request's did). Each scope has the same KV memoization + register_var/register_reset API, and each scope's clear() and sizing draw from the same registry, so cleared and measured sets cannot drift.
DW::CacheStats is deleted. Cache byte sizes are now emitted by
DW::Cache->report_sizes (dw.cache.bytes tagged cache:
t/request-cache.t becomes t/cache.t and also covers scope independence and the handle_caches/process-scope integration.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Fix stale cache-terminology comments flagged in review
"process cache" comments on request-scope lookups in LJ/User/Account.pm now say "request cache" (an actual process scope exists, so the old wording misleads), and the trusted-anon rationale in LJ/Session.pm + t/session-trust.t no longer cites the retired never-cleared %LJ::REQ_CACHE; t/vgift-trans.t explains the manual rel-cache invalidation instead of referencing REQ_CACHE_REL.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
githubBatch the profile page's viewer-relationship lookups (#3648) (#3649)
- Batch the profile page's viewer-relationship lookups (#3648)
The profile page was slow in two independent ways, both CPU/memcache-bound rather than SQL-bound.
format_userlink in views/profile/blocks.tt ran once per listed user and called remote.watches / remote.trusts (or memberof) for each, every one an uncached trustmask / checkrel memcache round-trip -- ~3,000 gets for a ~1,500-relationship profile (~6s logged in). Unlike #3646 these keys are all distinct, so memoizing trustmask wouldn't help; instead the controller now loads the viewer's circle once via watcheduserids / trusted_userids / member_of_userids into {id => 1} hashes and the template does an O(1) membership check.
Separately, sort_by_username recomputed display_name on every O(n log n) comparison (~31k calls, ~1s even logged out); it now uses a Schwartzian transform so each name is computed once.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Trim change-narration and duplicate issue refs from the profile comments
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
githubHoist per-viewer trustmask lookup out of the tag loop (#3646) (#3647)
- Hoist per-viewer trustmask lookup out of the tag loop (#3646)
Rendering a tag list called LJ::S2::TagDetail once per tag, and for a logged-in non-owner that recomputed the viewer's trust relationship (trusts_or_has_member + trustmask) for every tag. Both resolve to _trustmask($u, $remote), an unmemoized memcache round-trip, so a journal with thousands of tags fired thousands of redundant gets and took ~20s.
The relationship is identical for every tag, so compute it once via the new LJ::S2::tag_viewer_context and pass it into TagDetail. As defense-in-depth, memoize trustmask per request in a dedicated %LJ::REQCACHE_TRUSTMASK (cleared in start_request, invalidated on edge changes) rather than the bare %LJ::REQ_CACHE, which is never cleared between requests and would leak stale masks across viewers.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Trim comments in the trustmask fix
Cut a redundant per-line comment, condense two headers, and drop duplicate issue references left over from the first pass.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Drop the redundant %LJ::REQ_CACHE warning from the memoization comment
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
githubLoad LJ::Location where it's used so entry locations render again (#3645)
- Load LJ::Location where it's used so entry locations render again
LJ::currents renders an entry's current location by calling LJ::Location->new inside an eval, but nothing on the render path ever loaded LJ::Location -- it was only ever called, never used/required. Once Apache/mod_perl (which broadly preloaded modules at startup) was retired in favor of Starman-only, the module stopped being resident, so the eval died silently, $loc came back undef, and the location was dropped from every entry. Mood and music don't go through LJ::Location, which is why they kept working.
Add use LJ::Location to the three files that call it: LJ::Entry (the
user-visible display path), LJ::Protocol (current_coords validation), and
LJ::Hooks::Setters (the icbm/location setter). Add t/currents.t, which
exercises LJ::currents without loading LJ::Location itself and asserts the
location renders -- it fails against the pre-fix tree and passes now.
CODE TOUR: If you set a "current location" on a post, it recently stopped showing up on the entry even though mood and music still did. The value was being saved correctly -- the site just wasn't displaying it. This restores the current location on entries.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- t/currents.t: use the Dreamwidth-only license header
It's a new file, not forked from LiveJournal; drop the fork boilerplate.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
githubDon't memoize the trusted-anon verdict in %LJ::REQ_CACHE (#3644)
%LJ::REQ_CACHE is never cleared between requests, so the cached verdict leaked across requests -- and therefore across visitors -- on a persistent worker. Drop the caching entirely: validation is one HMAC plus an already-request-cached load_userid, and only runs where a captcha would otherwise be shown. Adds a regression test simulating back-to-back requests from different browsers on one worker.
Co-authored-by: Claude Fable 5 noreply@anthropic.com
Commit: ca89c2f6 Author: Mark Smith
githubSkip captchas for logged-out browsers that recently held a good session (#3643)
- Fix DW::Captcha->site_enabled when called as a class method on the base
3594 changed the abstract base's implementationenabled to return 0, which
made every class-method DW::Captcha->site_enabled call return false -- so require_captcha_test bailed early (disabling all comment captchas) and the per-journal comment captcha setting hid itself. Have the base report whether any implementation is enabled instead; subclass behavior is unchanged, and a fallback base instance still cleanly no-ops when nothing is configured.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- Skip captchas for logged-out browsers that recently held a good session
Adds an HMAC-signed "ljtrust" cookie (LJ::Session::update_trust_cookie, set alongside the master cookie at login/renewal and re-signed on uniq rotation; never cleared on logout). It's signed over the userid, the browser's ljuniq ident, and an issue time, using the rotating LJ::get_secret pool.
LJ::Session->trusted_anon_user validates it on logged-out requests -- sig, current-uniq binding, 60-day age, cookie generation -- then re-checks the account's standing live (visible, validated email, individual), so suspension revokes the bypass immediately.
A trusted anon then gets the logged-in treatment from both captcha gates: the interstitial view gate (DW::Captcha::should_captcha_view, checked lazily at each would-show point so trusted browsers never feed the fraud tempban counter) and the comment checks (LJ::Talk::Post::require_captcha_test: journal setting R and comment_html_anon are skipped; F/A, maxcomments, rate limits, and sysbans -- everything a logged-in user would still face -- are not). Each skip increments dw.captcha.bypassed with tags mirroring dw.captcha.shown.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- Address review: derive trust window from session_length, tag bypass metric with impl type
TRUST_COOKIE_MAX_AGE now delegates to LJ::Session->session_length('long')
instead of duplicating the value, and dw.captcha.bypassed carries the same
type:
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
githubAuto-deploy web-shop to follow web-stable (#3642)
Add a deploy-shop-follows-stable job to web22-deploy.yml that promotes the same image digest to the web-shop tier after a web-stable deploy succeeds. Guarded on inputs.service == 'web-stable' so shop/canary/unauth deploys don't fan out, and needs: deploy so a failed stable deploy leaves shop on its current version.
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
Commit: d140c907 Author: Mark Smith
githubStop eagerly populating talk2row in get_talk_data (#3640)
- Stop eagerly populating talk2row in get_talk_data
get_talk_data rebuilt the packed talk2 blob under its lock, and inside that loop it also wrote a per-comment talk2row cache entry for every comment via one LJ::MemCache::add each -- thousands of sequential memcache round-trips held the lock for 10s+ on large threads, which is what made comments intermittently vanish (readers timed out waiting, per the diagnostics in #3639).
Drop the eager per-row write. The talk2row cache is already populated lazily and in batch by get_talk2_row_multi (get_multi read, single IN() DB fallback, populate-on-miss), so the eager loop was redundant pre-warming -- and it warmed all N rows when a page shows ~25. Both paths write identical talk2row schema fields, so the cached value is unchanged; only the timing (lazy vs eager) moves.
Adds t/comment-talk2-rowcache.t verifying comments load correctly and consistently from a cold cache through both get_talk_data and get_talk2_row_multi (run with and without memcache via memcache_stress).
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01J3Ak8rgLkb6FVSRKaMb8ea
- Test the comment caches across a full read/write lifecycle
Rework t/comment-talk2-rowcache.t into a cache-lifecycle test that asserts the database is read only on a cache miss and memcache serves warm reads, through reads, writes, and invalidations: cold read hits DB once and caches the blob (without eagerly caching per-comment rows); warm read serves from memcache; get_talk2_row_multi hits DB once then serves warm; posting a comment clears the blob but leaves unrelated rows; deleting a comment clears the blob and that comment's row; and reads after each write regenerate correctly.
Adds two test-only hooks ($LJ::TGET_TALK_DATA_DB, $LJ::TGET_TALK2_ROW_DB, guarded like the existing TGET_TALK_DATA_MEMCACHE hook) so the test can count DB reads on each comment-loading path.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01J3Ak8rgLkb6FVSRKaMb8ea
- Test: guard loader return values before dereferencing
Assert get_talk_data returned data and get_talk2_row_multi returned both rows, and make the is_deeply key-derefs undef-safe, so an unexpected undef in the test env fails cleanly instead of dying in keys %$undef.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01J3Ak8rgLkb6FVSRKaMb8ea
