Use account terminology consistently in interest searches (#3770)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: 9101f444 Author: Alex Southwell
Use account terminology consistently in interest searches (#3770)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: 9101f444 Author: Alex Southwell
Accept slug URLs for community posting guidelines (#3737)
Accept slug URLs for community posting guidelines
Resolve dated numeric entry slugs before entry IDs
Match journal routing for posting guidelines and shared URL lookups. Keep wrong-date rejection and verify distinct collision/absent-ID fixtures.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The dated slug form already checked that the entry belongs to the
community; the /
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Support four profile services in user tags (#3754)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: 6fbe76fd Author: Alex Southwell
Remove obsolete vid.me and blip.tv embed providers (#3763)
Test rejection of retired embed providers
Remove retired vid.me and blip.tv embed providers
Format retired embed provider regressions
Make email validation messages translatable (#3733)
Make email validation messages translatable
Test email rejection messages against the real catalog
The previous subtests stubbed LJ::Lang::ml and asserted translation keys and parameter plumbing, which restated the implementation. Check instead that each rejection keeps its error code and gives the user one message naming the problem, including the normalized address and suggested domain, and that spelling confirmation still works.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
A valid address must produce no message, and confirming a suspected misspelling must not bypass the other checks.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Add Bellwether, a workflow that merges triaged PRs on maintainer commands (#3778)
Bellwether's triage reviews and tests PRs and keeps a status comment on each one, with a hidden line recording the commit it checked and whether it passed. This workflow reads that comment and acts on it: when a maintainer replies /merge (or /approve, or approves the PR), or when a ready low-risk PR's 24-hour grace period passes with no /hold, it re-checks CI, the head commit and main, then squash-merges pinned to the reviewed commit.
It runs on PR comments, when CI finishes and hourly. It trusts only the Bellwether app's own comments and never checks out or runs PR code.
Co-authored-by: Claude Opus 5.5 noreply@anthropic.com
Commit: ea02c410 Author: Mark Smith
Skip invalid recipients without aborting email jobs (#3726)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: d83443d2 Author: Alex Southwell
Expand AGENTS.md into a contributor guide for people and their agents (#3777)
Add what reviewers expect and how PRs are reviewed: tests that exercise real behavior, how to add and retire translation strings, keeping behavior consistent across sibling code, the escaping Template Toolkit doesn't do, when a change needs a maintainer behind it, and how risk, review notes and CI approval for fork PRs work.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Add an admin page for debugging imports, and keep import logs and cache in blob storage (#3776)
Import job logs were written to local disk on whichever worker ran the job, and the LiveJournal comment export cache likewise, so neither could be seen or shared across hosts.
Logs now go through DW::Importer::LogStore: an import_logs row per job log
(item, times, chunk and byte counts, host, pid) indexing chunks stored in
DW::BlobStore namespace importlogs. DW::Importer::Logger buffers lines and
writes a chunk at ~256KB, every ~30s, and on close or destruction. A failing
store or database only warns; the unflushed buffer is retried and capped at
4MB (oldest lines dropped, with a marker). The chunk count is raised only
after the blob is stored, so an unretrievable chunk means it expired.
The comment export cache moves to blob namespace importcache, keyed by user
and the same per-import md5 as before; dry runs still read it but never write
it. Comments::on_exception's .failure file is removed: the job runner already
logs the error to the (now viewable) job log.
bin/importadm --list shows each import's logs and --log LOGID prints one.
Retention: nothing is purged in the app. Production should set S3 lifecycle
expiration rules (~30 days) on the <prefix>/importlogs/ and
<prefix>/importcache/ key prefixes. Expired chunks read back as missing.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Control cancels an import (unfinished items become aborted, idle TheSchwartz jobs are removed) and reruns failed, aborted or named items, resetting items that depended on them and releasing the ones whose dependencies have succeeded. Every status change is an UPDATE conditional on the status that was read, and a retry is refused unless it is the user's latest import, nothing is ready or queued, and no job is left in the queue. Both are recorded in the status history.
The job runner honours a cancel: a job whose item is aborted does not run, and one cancelled while it ran writes its result with "status <> 'aborted'", and on a miss completes the job without notifying the user, advancing dependents or scheduling a retry.
DW::Importer::Queue can list an import's jobs (state, failures, last errors) and remove the ones no worker has grabbed. DW::Importer::Items gains release, which readies reset items using the same dependency rule as advance.
/admin/importer/history now shows each import's items, queue state and logs, with POST actions (form auth, siteadmin:importhistory) to cancel, retry or rerun an item, and /admin/importer/log shows one log for the user it belongs to, or as a text download.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Cancel now aborts each item with a single conditional update that accepts any cancellable status, so an item the scheduler moves from ready to queued in between is still aborted. Retry waits until the import has been still for $LJ::IMPORTER_RETRY_SETTLE_SECONDS (default 60): a job the scheduler claimed an item for just before a cancel can otherwise reach the queue after retry has looked, and collide with the rerun's job.
Import logs are UTF-8 bytes from the logger to the page, as most of the site's text is, so the viewer no longer decodes twice.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The comments job now looks up the journal's FOAF page, which the fake site in t/importer-lj-jobs.t couldn't answer, so the job died.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Split LiveJournal imports into stages and add a dry-run mode (#3775)
Pin the behaviour of the pieces the importer refactor is about to move:
Local::Entries::post_event (idempotence by key, provenance, backdating,
logtime), Local::Comments::insert_comment and get_comment_map (key format,
threading, per-state effects), trust group and tag merging, groupmask
remapping and
user rewriting.
t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Fix content importer bugs found in the audit
set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Remove dead content importer code
DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.
DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.
DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.
The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.
The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Job is now the base of every item worker (it sits between TheSchwartz::Worker and DW::Worker::ContentImporter, so the worker classes and their TheSchwartz function names are unchanged). A worker implements run($ctx) and returns, or dies with, a DW::Importer::Outcome (ok, fail or temp_fail). The base owns what the eight workers each copied: loading the import and user, declining when the source is disabled, the eval and outcome handling, the item status, the notification (to every maintainer of a community), stats, the TheSchwartz job state, escalation of a temporary failure after max_retries, the registry-driven chaining, the process title and the log.
DW::Importer::Context carries the per-job state: user, import data with thawed options, registry item, logger and a cache hash. It replaces the package-global %MAPS (the remote-user and group maps are now cached in the context, so nothing has to be cleared between jobs) and remap_groupmask and get_remapped_userids take the context. DW::Importer::Logger is the one implementation of the "[delta total]" log written to the per-import file and the job's debug output, with an in-memory variant for tests.
work() no longer calls exit 0. The entries and comments workers set exit_after_job and the content-importer scripts exit from TheSchwartz's after-work hook once the job has completed, so production still reclaims memory but work() can be called in a test.
Verify, Bio, Tags, FriendGroups, Friends and Userpics are ported fully. Entries and Comments only move their skeleton and failure handling onto the base; their fetch, transform and post code is as it was, except that a failure inside a closure is raised as an outcome instead of a return value that every caller had to check. A log file that can't be created no longer fails the job. bin/importadm runs a job through work() now.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.
job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
First step of splitting the LiveJournal entries and comments jobs into fetch, transform and write stages. Nothing is wired in yet.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Source::LiveJournal owns every request to the remote site and returns the normalized records: tags, trust groups, friends, bio (FOAF), userpics, entries (with their keys and URLs), the newest entry id, identity lookups (FOAF/OPML, rename history) and the comment export. It writes nothing locally and dies with DW::Importer::Outcome objects, so the jobs only sequence work. Its transport (XML-RPC, plain HTTP, the comment export with its per-import disk cache and login session) is the first thing a job's context hands it; Source::LiveJournal::Transport is the real implementation, moved from LiveJournal.pm and Comments.pm, and tests substitute their own. DW::External::XPostProtocol::LJXMLRPC has the same XML-RPC shape but a different auth flow and error format, so it is left alone.
The comment export's window and nextid logic is carried over as it was. What changed is where state lives: a window is read, then handed over complete, so a comment waiting for its parent no longer sits in the half-read export state (on an encoding retry it used to lose its text). Threading decides order and reports what could not be placed.
All eight workers now run adapter -> transform -> write using the pure transforms from the previous commit; the writes are still the Local::* calls (the sink is next). Entries now maps security through Transform::Security, so a mask whose groups could not be mapped is stored as private instead of as usemask with an empty mask, and also when only unmapped groups were listed. The comments job tells the user about replies whose parent was never imported, besides comments on entries that were not.
Tests feed canned XML-RPC and export windows through the workers and the adapter, including nextid resume, servers with no nextid, overlapping windows and the Latin-1 fallback.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Every local write an import makes now goes through DW::Importer::Sink: posting entries, updating entry icons, inserting and updating comments, tags, trust groups and the group map, friends, interests and bio, icons, identity accounts and the remote-user map, and the search copier dispatch. The known-key queries (entry map, comment map, crossposted entries, group map, user map) are on it too.
Sink::Live wraps the existing Local::* code and owns the entry map the job reads and extends. Local::Comments no longer keeps its caches in package variables: the sink holds them and passes them in.
Sink::DryRun records each write as an action in a DW::Importer::Report and changes nothing. Reads go to a live sink, so entries and tags that already exist are reported as skipped. It makes up ids that cannot be real ones, and group ids that are valid access mask bits, so that comments can still be threaded and later jobs of one dry run (given the same world) see what the earlier ones would have made. Identity lookups use the read-only load_existing_identity_user.
All eight workers write only through the sink; remote users are resolved through it as well, and the remote-user map row is now only written when the user was not already in it (rewriting an identical row did nothing).
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Job->dry_run runs one item in-process with a Sink::DryRun and dry_run_chain runs the import's items in dependency order, each seeing what the earlier ones would have written (an item whose dependency failed is not run). The runner leaves out everything that is not the job's own work: the item's status and dependency advancement, ImportStatus notifications (the report keeps what the user would have been told), stats, TheSchwartz job state, per-import log files, the process title and the exit request. All memcache writes are suppressed, so a dry run can read through the cache but leaves nothing in it. The Verify job does not run the ban_unset console command on the remote site, which is a remote write; the report says so.
DW::Importer::Report collects the actions with counts per type, what would be skipped as already imported, warnings, comments that could not be placed and the jobs' outcomes, and renders them as text. The dry-run sink reports tags, groups and the like that already exist as skipped rather than as writes.
bin/importadm --dry-run [--type ITEM] [--import-id N] [--verbose] prints the report; --run keeps its meaning. --list now prints the directory the import logs are actually written to.
t/importer-dryrun.t runs a fixture import (public, private, friends-only, group-locked and unmappable-group entries, a crosspost, threaded, screened, deleted, anonymous, orphaned and unmapped comments) as a dry run, which changes no row of anything an import writes and puts nothing in memcache; then for real, which posts what the report listed with the security and threading it gave; then again, which adds nothing, and a dry run after that, which finds it all already there.
Two things the fixture showed. The site rejects an access mask that has both the trusted bit and groups, so Transform::Security now makes such an entry private with a warning instead of letting the post fail. And re-importing a comment rewrote its icon property to the value it already had (or to 0 on a comment with none); update_comment now writes it only when it changes.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Signature checks a detached Ed25519 signature over the exact bytes of an uploaded export, for sources that sign their files. Trusted public keys come from %LJ::IMPORTER_SIGNING_KEYS, a list per source so keys can be rotated; a key is 32 raw bytes as base64 or a PEM public key, and private keys are refused. The signature may be standard or URL-safe base64, padded or not, with any whitespace. verify() returns true or false, never dies on malformed input, logs why, and with no usable key for the source it fails closed. etc/config-local.pl.example documents the setting.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
The comment export transport wrote its XML cache on every miss, and the comments job wrote its .failure note on every exception, in a dry run too. A cache file made by a dry run is then trusted by the real import that follows.
The transport is now told when it is in a dry run (from the job context, or dry_run()): it still reads an existing cache but never writes one, and says so in the log. The failure note is not written in a dry run; the reason goes to the run's logger instead. Both places now use lexical filehandles and check the open and close, so a cache that cannot be written is logged instead of printed to a closed handle.
The rest of the dry-run path has no file writes: per-import log files are only made for real runs (a dry run logs in memory or to STDERR), the please_die file is only written when a real job fails, and the icon download is in the live sink.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Run import items through a shared job runner and add export signature checks (#3774)
Pin the behaviour of the pieces the importer refactor is about to move:
Local::Entries::post_event (idempotence by key, provenance, backdating,
logtime), Local::Comments::insert_comment and get_comment_map (key format,
threading, per-state effects), trust group and tag merging, groupmask
remapping and
user rewriting.
t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Fix content importer bugs found in the audit
set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Remove dead content importer code
DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.
DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.
DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.
The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.
The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Job is now the base of every item worker (it sits between TheSchwartz::Worker and DW::Worker::ContentImporter, so the worker classes and their TheSchwartz function names are unchanged). A worker implements run($ctx) and returns, or dies with, a DW::Importer::Outcome (ok, fail or temp_fail). The base owns what the eight workers each copied: loading the import and user, declining when the source is disabled, the eval and outcome handling, the item status, the notification (to every maintainer of a community), stats, the TheSchwartz job state, escalation of a temporary failure after max_retries, the registry-driven chaining, the process title and the log.
DW::Importer::Context carries the per-job state: user, import data with thawed options, registry item, logger and a cache hash. It replaces the package-global %MAPS (the remote-user and group maps are now cached in the context, so nothing has to be cleared between jobs) and remap_groupmask and get_remapped_userids take the context. DW::Importer::Logger is the one implementation of the "[delta total]" log written to the per-import file and the job's debug output, with an in-memory variant for tests.
work() no longer calls exit 0. The entries and comments workers set exit_after_job and the content-importer scripts exit from TheSchwartz's after-work hook once the job has completed, so production still reclaims memory but work() can be called in a test.
Verify, Bio, Tags, FriendGroups, Friends and Userpics are ported fully. Entries and Comments only move their skeleton and failure handling onto the base; their fetch, transform and post code is as it was, except that a failure inside a closure is raised as an outcome instead of a return value that every caller had to check. A log file that can't be created no longer fails the job. bin/importadm runs a job through work() now.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.
job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Signature checks a detached Ed25519 signature over the exact bytes of an uploaded export, for sources that sign their files. Trusted public keys come from %LJ::IMPORTER_SIGNING_KEYS, a list per source so keys can be rotated; a key is 32 raw bytes as base64 or a PEM public key, and private keys are refused. The signature may be standard or URL-safe base64, padded or not, with any whitespace. verify() returns true or false, never dies on malformed input, logs why, and with no usable key for the source it fails closed. etc/config-local.pl.example documents the setting.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Add an importer registry and fix scheduler races (#3773)
Pin the behaviour of the pieces the importer refactor is about to move:
Local::Entries::post_event (idempotence by key, provenance, backdating,
logtime), Local::Comments::insert_comment and get_comment_map (key format,
threading, per-state effects), trust group and tag merging, groupmask
remapping and
user rewriting.
t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Fix content importer bugs found in the audit
set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Remove dead content importer code
DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.
DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.
DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.
The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.
The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.
job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Add content importer tests and fix importer bugs (#3772)
Pin the behaviour of the pieces the importer refactor is about to move:
Local::Entries::post_event (idempotence by key, provenance, backdating,
logtime), Local::Comments::insert_comment and get_comment_map (key format,
threading, per-state effects), trust group and tag merging, groupmask
remapping and
user rewriting.
t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Fix content importer bugs found in the audit
set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Remove dead content importer code
DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.
DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Make browser tests self-contained and document them (#3767)
t/browser/editicons-rename.js needed a hand-built account and an exported cookie file, so nobody could run it. Add a fixture script that creates and purges its own account and icons, a shared helper that launches Chrome and logs in, and a README with the run command and the rules for new tests. These remain manual checks; CI doesn't run them.
Co-authored-by: Claude Opus 5.5 noreply@anthropic.com
Commit: d43b82b4 Author: Mark Smith
Keep modal close controls visible without icon fonts (#3765)
The account-links menu and the shared modal wrapper drew their close control with the Foundation icon font. When that font fails to load cross-domain (#3432), the button shows the raw "F217" codepoint. Use the same plain multiplication sign and aria-label as the login popup (#3756).
Co-authored-by: Claude Opus 5.5 noreply@anthropic.com
Commit: 247ea4f0 Author: Mark Smith
Style the canary ribbon on entry previews (#3762)
Style the canary ribbon on entry previews
Limit legacy preview CSS to the canary ribbon
Clarify header image options in Heads Up (#3757)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: 1506b0b7 Author: Alex Southwell
Keep the login popup close control visible without icon fonts (#3756)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: d1221efb Author: Alex Southwell
Show full Discord handles on profile pages (#3755)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: 2d6c316e Author: Alex Southwell
Fix header font preferences in A Brighter View (#3753)
Co-authored-by: SAPHID saphid@users.noreply.github.com
Commit: edcdfd3b Author: Alex Southwell