github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Make email validation messages translatable (#3733)

  • Make email validation messages translatable

  • Test email rejection messages against the real catalog

The previous subtests stubbed LJ::Lang::ml and asserted translation keys and parameter plumbing, which restated the implementation. Check instead that each rejection keeps its error code and gives the user one message naming the problem, including the normalized address and suggested domain, and that spelling confirmation still works.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Cover valid addresses and forced spelling against other rejections

A valid address must produce no message, and confirming a suspected misspelling must not bypass the other checks.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Add Bellwether, a workflow that merges triaged PRs on maintainer commands (#3778)

Bellwether's triage reviews and tests PRs and keeps a status comment on each one, with a hidden line recording the commit it checked and whether it passed. This workflow reads that comment and acts on it: when a maintainer replies /merge (or /approve, or approves the PR), or when a ready low-risk PR's 24-hour grace period passes with no /hold, it re-checks CI, the head commit and main, then squash-merges pinned to the reviewed commit.

It runs on PR comments, when CI finishes and hourly. It trusts only the Bellwether app's own comments and never checks out or runs PR code.

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

Commit: ea02c410 Author: Mark Smith

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Skip invalid recipients without aborting email jobs (#3726)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: d83443d2 Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Expand AGENTS.md into a contributor guide for people and their agents (#3777)

  • Expand AGENTS.md into a contributor guide for people and their agents

Add what reviewers expect and how PRs are reviewed: tests that exercise real behavior, how to add and retire translation strings, keeping behavior consistent across sibling code, the escaping Template Toolkit doesn't do, when a change needs a maintainer behind it, and how risk, review notes and CI approval for fork PRs work.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • AGENTS.md: don't require regression tests, or say review is always human

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Add an admin page for debugging imports, and keep import logs and cache in blob storage (#3776)

  • Keep import logs and the comment export cache in blob storage

Import job logs were written to local disk on whichever worker ran the job, and the LiveJournal comment export cache likewise, so neither could be seen or shared across hosts.

Logs now go through DW::Importer::LogStore: an import_logs row per job log (item, times, chunk and byte counts, host, pid) indexing chunks stored in DW::BlobStore namespace importlogs. DW::Importer::Logger buffers lines and writes a chunk at ~256KB, every ~30s, and on close or destruction. A failing store or database only warns; the unflushed buffer is retried and capped at 4MB (oldest lines dropped, with a marker). The chunk count is raised only after the blob is stored, so an unretrievable chunk means it expired.

The comment export cache moves to blob namespace importcache, keyed by user and the same per-import md5 as before; dry runs still read it but never write it. Comments::on_exception's .failure file is removed: the job runner already logs the error to the (now viewable) job log.

bin/importadm --list shows each import's logs and --log LOGID prints one.

Retention: nothing is purged in the app. Production should set S3 lifecycle expiration rules (~30 days) on the <prefix>/importlogs/ and <prefix>/importcache/ key prefixes. Expired chunks read back as missing.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add importer cancel and retry controls and an admin debugging page

DW::Importer::Control cancels an import (unfinished items become aborted, idle TheSchwartz jobs are removed) and reruns failed, aborted or named items, resetting items that depended on them and releasing the ones whose dependencies have succeeded. Every status change is an UPDATE conditional on the status that was read, and a retry is refused unless it is the user's latest import, nothing is ready or queued, and no job is left in the queue. Both are recorded in the status history.

The job runner honours a cancel: a job whose item is aborted does not run, and one cancelled while it ran writes its result with "status <> 'aborted'", and on a miss completes the job without notifying the user, advancing dependents or scheduling a retry.

DW::Importer::Queue can list an import's jobs (state, failures, last errors) and remove the ones no worker has grabbed. DW::Importer::Items gains release, which readies reset items using the same dependency rule as advance.

/admin/importer/history now shows each import's items, queue state and logs, with POST actions (form auth, siteadmin:importhistory) to cancel, retry or rerun an item, and /admin/importer/log shows one log for the user it belongs to, or as a text download.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Test the importer log viewer against the real log store

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Close importer cancel/retry races and keep import logs as bytes

Cancel now aborts each item with a single conditional update that accepts any cancellable status, so an item the scheduler moves from ready to queued in between is still aborted. Retry waits until the import has been still for $LJ::IMPORTER_RETRY_SETTLE_SECONDS (default 60): a job the scheduler claimed an item for just before a cancel can otherwise reach the queue after retry has looked, and collide with the rerun's job.

Import logs are UTF-8 bytes from the logger to the page, as most of the site's text is, so the viewer no longer decodes twice.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Print import logs from importadm as the bytes they are

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Fix the LiveJournal job test's fake site and tidy the importer test helper

The comments job now looks up the journal's FOAF page, which the fake site in t/importer-lj-jobs.t couldn't answer, so the job died.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Wrap long lines in the import log viewer

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Split LiveJournal imports into stages and add a dry-run mode (#3775)

  • Add characterization tests for the content importer

Pin the behaviour of the pieces the importer refactor is about to move: Local::Entries::post_event (idempotence by key, provenance, backdating, logtime), Local::Comments::insert_comment and get_comment_map (key format, threading, per-state effects), trust group and tag merging, groupmask remapping and [profile] user rewriting.

t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Fix content importer bugs found in the audit

  • set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.

  • Local::Comments insert/update passed a reference to a copy of the error reference, so failures never reached the caller. update_comment also compared the body numerically, which overwrote any non-numeric body when FIX_COMMENT_IMPORT was on.
  • Comments whose entry was never imported were inserted against entry 0. They are now skipped, logged, and reported to the user in a status note.
  • The bio job asked for /data/foaf twice and could not succeed.
  • remap_username_friend returned an LJ::User instead of an id after a rename.
  • get_duplicates_map built keys that could never match its lookup and its result was unused; removed rather than turned into new behaviour.
  • The comment export cache is now keyed on import_data_id: retries reuse the download, a new import fetches fresh data.
  • Admin importer controller declared the user controller's package and the user controller relied on that; each now has its own package and imports. get_queue returns an empty hash instead of a string when the queue is unreachable, render_status resets its duplicate flag per item, confirm.tt reads the right import row for community imports, and the raw DBI connects to TheSchwartz go through the shared LJ::theschwartz client.
  • fail/temp_fail no longer treat the message as a sprintf format when called without arguments, since callers pass error strings from remote sites.
  • Parenthesize the trust bit test in remap_groupmask.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Remove dead content importer code

  • DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.

  • DW::Worker::ContentImporter::UserPictures is never enqueued and calls methods that do not exist; drop it and its declarations in the content-importer workers. Nothing reads the import_upi blob or the userpics_later flag that the userpics job set for it, so those go too.
  • remap_username_feed and get_feed_account_from_url were unreachable (their only caller was commented out), and the Bio job never used its user agent.

DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add an importer registry and generic dependency chaining

DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.

DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.

The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.

The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Only cascade a scheduler clash when this scheduler aborted the item

When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Claim import items before inserting their scheduler job

The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Recover import items stranded by claim-first scheduling

The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.

  • An exception from the job insert is handled like a failed insert: the claimed row is aborted, its dependents are cascaded, and the error is logged.
  • Each scheduler pass (at most once a minute) now looks for rows that have been 'queued' for longer than $LJ::IMPORTER_STALE_CLAIM_SECONDS (default 3600) and puts those with no TheSchwartz job for their import back to 'ready', using an UPDATE guarded on the status and the last_touch it observed. Items whose job is still in the queue are never touched: running jobs (entries and comments hold grab_for for hours) and temp-failed jobs awaiting a retry keep their job row. If the queue can't be read nothing is reset. A job that completed but whose worker died before recording 'succeeded' will be run again; imports are idempotent, so that only repeats work.
  • DW::Importer::Queue is the one reader of the TheSchwartz tables: it replaces the controller's private helpers, and the status page's duplicate detection and the scheduler's recovery both ask it whether a job exists for an item of a user (optionally for one import_data_id). It goes through LJ::theschwartz, so it works with the fake queue in tests.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Run importer items through a shared job runner

DW::Importer::Job is now the base of every item worker (it sits between TheSchwartz::Worker and DW::Worker::ContentImporter, so the worker classes and their TheSchwartz function names are unchanged). A worker implements run($ctx) and returns, or dies with, a DW::Importer::Outcome (ok, fail or temp_fail). The base owns what the eight workers each copied: loading the import and user, declining when the source is disabled, the eval and outcome handling, the item status, the notification (to every maintainer of a community), stats, the TheSchwartz job state, escalation of a temporary failure after max_retries, the registry-driven chaining, the process title and the log.

DW::Importer::Context carries the per-job state: user, import data with thawed options, registry item, logger and a cache hash. It replaces the package-global %MAPS (the remote-user and group maps are now cached in the context, so nothing has to be cleared between jobs) and remap_groupmask and get_remapped_userids take the context. DW::Importer::Logger is the one implementation of the "[delta total]" log written to the per-import file and the job's debug output, with an in-memory variant for tests.

work() no longer calls exit 0. The entries and comments workers set exit_after_job and the content-importer scripts exit from TheSchwartz's after-work hook once the job has completed, so production still reclaims memory but work() can be called in a test.

Verify, Bio, Tags, FriendGroups, Friends and Userpics are ported fully. Entries and Comments only move their skeleton and failure handling onto the base; their fetch, transform and post code is as it was, except that a failure inside a closure is raised as an outcome instead of a return value that every caller had to check. A log file that can't be created no longer fails the job. bin/importadm runs a job through work() now.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Treat an unreadable queue database as "don't know", not "no job"

Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.

job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add normalized import records and pure transforms

First step of splitting the LiveJournal entries and comments jobs into fetch, transform and write stages. Nothing is wired in yet.

  • DW::Importer::Records documents the record shapes sources produce and sinks consume: entry, comment, tag, trust group, friend, userpic, bio and the action a sink reports.
  • Transform::LJMarkup holds the user-tag, poll, embed and template rewriting that the entries and comments jobs each carried inline, keeping the differences between the two, and returns warnings instead of logging.
  • Transform::Security maps remote security, access mask and group map to a local security and mask. It fails closed: an unrecognised security or mask is private with a warning, unmapped groups are dropped and never widened, and a mask with nothing mappable left is private with the "Could not determine groups to post to." warning the entries job used to add inline.
  • Threading orders comments parent first, keeps replies whose parent has not arrived yet for the next window, and reports comments whose entry is not imported and replies whose parent never was.
  • Source::LiveJournal::Keys builds and reads entry and comment keys. The strings are pinned against what the importer's own writers store.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Move LiveJournal remote I/O into a source adapter with an injected transport

DW::Importer::Source::LiveJournal owns every request to the remote site and returns the normalized records: tags, trust groups, friends, bio (FOAF), userpics, entries (with their keys and URLs), the newest entry id, identity lookups (FOAF/OPML, rename history) and the comment export. It writes nothing locally and dies with DW::Importer::Outcome objects, so the jobs only sequence work. Its transport (XML-RPC, plain HTTP, the comment export with its per-import disk cache and login session) is the first thing a job's context hands it; Source::LiveJournal::Transport is the real implementation, moved from LiveJournal.pm and Comments.pm, and tests substitute their own. DW::External::XPostProtocol::LJXMLRPC has the same XML-RPC shape but a different auth flow and error format, so it is left alone.

The comment export's window and nextid logic is carried over as it was. What changed is where state lives: a window is read, then handed over complete, so a comment waiting for its parent no longer sits in the half-read export state (on an encoding retry it used to lose its text). Threading decides order and reports what could not be placed.

All eight workers now run adapter -> transform -> write using the pure transforms from the previous commit; the writes are still the Local::* calls (the sink is next). Entries now maps security through Transform::Security, so a mask whose groups could not be mapped is stored as private instead of as usemask with an empty mask, and also when only unmapped groups were listed. The comments job tells the user about replies whose parent was never imported, besides comments on entries that were not.

Tests feed canned XML-RPC and export windows through the workers and the adapter, including nextid resume, servers with no nextid, overlapping windows and the Latin-1 fallback.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Write imports through a sink, with a live and a dry-run implementation

Every local write an import makes now goes through DW::Importer::Sink: posting entries, updating entry icons, inserting and updating comments, tags, trust groups and the group map, friends, interests and bio, icons, identity accounts and the remote-user map, and the search copier dispatch. The known-key queries (entry map, comment map, crossposted entries, group map, user map) are on it too.

Sink::Live wraps the existing Local::* code and owns the entry map the job reads and extends. Local::Comments no longer keeps its caches in package variables: the sink holds them and passes them in.

Sink::DryRun records each write as an action in a DW::Importer::Report and changes nothing. Reads go to a live sink, so entries and tags that already exist are reported as skipped. It makes up ids that cannot be real ones, and group ids that are valid access mask bits, so that comments can still be threaded and later jobs of one dry run (given the same world) see what the earlier ones would have made. Identity lookups use the read-only load_existing_identity_user.

All eight workers write only through the sink; remote users are resolved through it as well, and the remote-user map row is now only written when the user was not already in it (rewriting an identical row did nothing).

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add dry-run mode for imports

DW::Importer::Job->dry_run runs one item in-process with a Sink::DryRun and dry_run_chain runs the import's items in dependency order, each seeing what the earlier ones would have written (an item whose dependency failed is not run). The runner leaves out everything that is not the job's own work: the item's status and dependency advancement, ImportStatus notifications (the report keeps what the user would have been told), stats, TheSchwartz job state, per-import log files, the process title and the exit request. All memcache writes are suppressed, so a dry run can read through the cache but leaves nothing in it. The Verify job does not run the ban_unset console command on the remote site, which is a remote write; the report says so.

DW::Importer::Report collects the actions with counts per type, what would be skipped as already imported, warnings, comments that could not be placed and the jobs' outcomes, and renders them as text. The dry-run sink reports tags, groups and the like that already exist as skipped rather than as writes.

bin/importadm --dry-run [--type ITEM] [--import-id N] [--verbose] prints the report; --run keeps its meaning. --list now prints the directory the import logs are actually written to.

t/importer-dryrun.t runs a fixture import (public, private, friends-only, group-locked and unmappable-group entries, a crosspost, threaded, screened, deleted, anonymous, orphaned and unmapped comments) as a dry run, which changes no row of anything an import writes and puts nothing in memcache; then for real, which posts what the report listed with the security and threading it gave; then again, which adds nothing, and a dry run after that, which finds it all already there.

Two things the fixture showed. The site rejects an access mask that has both the trusted bit and groups, so Transform::Security now makes such an entry private with a warning instead of letting the post fail. And re-importing a comment rewrote its icon property to the value it already had (or to 0 on a comment with none); update_comment now writes it only when it changes.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add Ed25519 verification for signed export uploads

DW::Importer::Signature checks a detached Ed25519 signature over the exact bytes of an uploaded export, for sources that sign their files. Trusted public keys come from %LJ::IMPORTER_SIGNING_KEYS, a list per source so keys can be rotated; a key is 32 raw bytes as base64 or a PEM public key, and private keys are refused. The signature may be standard or URL-safe base64, padded or not, with any whitespace. verify() returns true or false, never dies on malformed input, logs why, and with no usable key for the source it fails closed. etc/config-local.pl.example documents the setting.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Make dry runs write no files

The comment export transport wrote its XML cache on every miss, and the comments job wrote its .failure note on every exception, in a dry run too. A cache file made by a dry run is then trusted by the real import that follows.

The transport is now told when it is in a dry run (from the job context, or dry_run()): it still reads an existing cache but never writes one, and says so in the log. The failure note is not written in a dry run; the reason goes to the run's logger instead. Both places now use lexical filehandles and check the open and close, so a cache that cannot be written is logged instead of printed to a closed handle.

The rest of the dry-run path has no file writes: per-import log files are only made for real runs (a dry run logs in memory or to STDERR), the please_die file is only written when a real job fails, and the icon download is in the live sink.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Run import items through a shared job runner and add export signature checks (#3774)

  • Add characterization tests for the content importer

Pin the behaviour of the pieces the importer refactor is about to move: Local::Entries::post_event (idempotence by key, provenance, backdating, logtime), Local::Comments::insert_comment and get_comment_map (key format, threading, per-state effects), trust group and tag merging, groupmask remapping and [profile] user rewriting.

t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Fix content importer bugs found in the audit

  • set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.

  • Local::Comments insert/update passed a reference to a copy of the error reference, so failures never reached the caller. update_comment also compared the body numerically, which overwrote any non-numeric body when FIX_COMMENT_IMPORT was on.
  • Comments whose entry was never imported were inserted against entry 0. They are now skipped, logged, and reported to the user in a status note.
  • The bio job asked for /data/foaf twice and could not succeed.
  • remap_username_friend returned an LJ::User instead of an id after a rename.
  • get_duplicates_map built keys that could never match its lookup and its result was unused; removed rather than turned into new behaviour.
  • The comment export cache is now keyed on import_data_id: retries reuse the download, a new import fetches fresh data.
  • Admin importer controller declared the user controller's package and the user controller relied on that; each now has its own package and imports. get_queue returns an empty hash instead of a string when the queue is unreachable, render_status resets its duplicate flag per item, confirm.tt reads the right import row for community imports, and the raw DBI connects to TheSchwartz go through the shared LJ::theschwartz client.
  • fail/temp_fail no longer treat the message as a sprintf format when called without arguments, since callers pass error strings from remote sites.
  • Parenthesize the trust bit test in remap_groupmask.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Remove dead content importer code

  • DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.

  • DW::Worker::ContentImporter::UserPictures is never enqueued and calls methods that do not exist; drop it and its declarations in the content-importer workers. Nothing reads the import_upi blob or the userpics_later flag that the userpics job set for it, so those go too.
  • remap_username_feed and get_feed_account_from_url were unreachable (their only caller was commented out), and the Bio job never used its user agent.

DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add an importer registry and generic dependency chaining

DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.

DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.

The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.

The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Only cascade a scheduler clash when this scheduler aborted the item

When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Claim import items before inserting their scheduler job

The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Recover import items stranded by claim-first scheduling

The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.

  • An exception from the job insert is handled like a failed insert: the claimed row is aborted, its dependents are cascaded, and the error is logged.
  • Each scheduler pass (at most once a minute) now looks for rows that have been 'queued' for longer than $LJ::IMPORTER_STALE_CLAIM_SECONDS (default 3600) and puts those with no TheSchwartz job for their import back to 'ready', using an UPDATE guarded on the status and the last_touch it observed. Items whose job is still in the queue are never touched: running jobs (entries and comments hold grab_for for hours) and temp-failed jobs awaiting a retry keep their job row. If the queue can't be read nothing is reset. A job that completed but whose worker died before recording 'succeeded' will be run again; imports are idempotent, so that only repeats work.
  • DW::Importer::Queue is the one reader of the TheSchwartz tables: it replaces the controller's private helpers, and the status page's duplicate detection and the scheduler's recovery both ask it whether a job exists for an item of a user (optionally for one import_data_id). It goes through LJ::theschwartz, so it works with the fake queue in tests.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Run importer items through a shared job runner

DW::Importer::Job is now the base of every item worker (it sits between TheSchwartz::Worker and DW::Worker::ContentImporter, so the worker classes and their TheSchwartz function names are unchanged). A worker implements run($ctx) and returns, or dies with, a DW::Importer::Outcome (ok, fail or temp_fail). The base owns what the eight workers each copied: loading the import and user, declining when the source is disabled, the eval and outcome handling, the item status, the notification (to every maintainer of a community), stats, the TheSchwartz job state, escalation of a temporary failure after max_retries, the registry-driven chaining, the process title and the log.

DW::Importer::Context carries the per-job state: user, import data with thawed options, registry item, logger and a cache hash. It replaces the package-global %MAPS (the remote-user and group maps are now cached in the context, so nothing has to be cleared between jobs) and remap_groupmask and get_remapped_userids take the context. DW::Importer::Logger is the one implementation of the "[delta total]" log written to the per-import file and the job's debug output, with an in-memory variant for tests.

work() no longer calls exit 0. The entries and comments workers set exit_after_job and the content-importer scripts exit from TheSchwartz's after-work hook once the job has completed, so production still reclaims memory but work() can be called in a test.

Verify, Bio, Tags, FriendGroups, Friends and Userpics are ported fully. Entries and Comments only move their skeleton and failure handling onto the base; their fetch, transform and post code is as it was, except that a failure inside a closure is raised as an outcome instead of a return value that every caller had to check. A log file that can't be created no longer fails the job. bin/importadm runs a job through work() now.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Treat an unreadable queue database as "don't know", not "no job"

Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.

job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add Ed25519 verification for signed export uploads

DW::Importer::Signature checks a detached Ed25519 signature over the exact bytes of an uploaded export, for sources that sign their files. Trusted public keys come from %LJ::IMPORTER_SIGNING_KEYS, a list per source so keys can be rotated; a key is 32 raw bytes as base64 or a PEM public key, and private keys are refused. The signature may be standard or URL-safe base64, padded or not, with any whitespace. verify() returns true or false, never dies on malformed input, logs why, and with no usable key for the source it fails closed. etc/config-local.pl.example documents the setting.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Add an importer registry and fix scheduler races (#3773)

  • Add characterization tests for the content importer

Pin the behaviour of the pieces the importer refactor is about to move: Local::Entries::post_event (idempotence by key, provenance, backdating, logtime), Local::Comments::insert_comment and get_comment_map (key format, threading, per-state effects), trust group and tag merging, groupmask remapping and [profile] user rewriting.

t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Fix content importer bugs found in the audit

  • set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.

  • Local::Comments insert/update passed a reference to a copy of the error reference, so failures never reached the caller. update_comment also compared the body numerically, which overwrote any non-numeric body when FIX_COMMENT_IMPORT was on.
  • Comments whose entry was never imported were inserted against entry 0. They are now skipped, logged, and reported to the user in a status note.
  • The bio job asked for /data/foaf twice and could not succeed.
  • remap_username_friend returned an LJ::User instead of an id after a rename.
  • get_duplicates_map built keys that could never match its lookup and its result was unused; removed rather than turned into new behaviour.
  • The comment export cache is now keyed on import_data_id: retries reuse the download, a new import fetches fresh data.
  • Admin importer controller declared the user controller's package and the user controller relied on that; each now has its own package and imports. get_queue returns an empty hash instead of a string when the queue is unreachable, render_status resets its duplicate flag per item, confirm.tt reads the right import row for community imports, and the raw DBI connects to TheSchwartz go through the shared LJ::theschwartz client.
  • fail/temp_fail no longer treat the message as a sprintf format when called without arguments, since callers pass error strings from remote sites.
  • Parenthesize the trust bit test in remap_groupmask.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Remove dead content importer code

  • DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.

  • DW::Worker::ContentImporter::UserPictures is never enqueued and calls methods that do not exist; drop it and its declarations in the content-importer workers. Nothing reads the import_upi blob or the userpics_later flag that the userpics job set for it, so those go too.
  • remap_username_feed and get_feed_account_from_url were unreachable (their only caller was commented out), and the Bio job never used its user agent.

DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Add an importer registry and generic dependency chaining

DW::Importer::Registry is now the one place that says what can be imported: the sources and their sites, and for each item its worker class, status event typeid, dependencies (per journal type where they differ), initial status, community support, options, worker process and whether success notifies the user. The item to class map used to be copied into the scheduler, both controllers and importadm; the typeids lived in LJ::Event::ImportStatus; dependency rules were in the controller, the JavaScript and the SQL of five workers. All of those now read the registry, as do the worker scripts' schwartz_decl lists, get_queue and DW::Logic::Importer::allowed_sources. The choose-data page hands its dependency map to jquery.importer.js as data. Item names, worker classes and typeids are unchanged, since queued jobs and stored rows refer to them.

DW::Importer::Items::advance replaces the per-worker chaining SQL and is called from the shared ok/fail: finishing an item readies the waiting items whose dependencies have all succeeded, and a permanent failure aborts the waiting items that depended on it. The readiness graph is the one the workers encoded (verify -> bio/userpics/friendgroups/tags, friendgroups -> friends/entries, tags -> entries for communities only, entries -> comments). The abort cascade is new: previously a failure left everything downstream in 'init' forever. It also subsumes verify's "Invalid password" special case.

The scheduler's body moves to DW::Importer::Scheduler so it can be run once under test; bin/worker/import-scheduler is a thin loop around it. When the scheduler has to abort an item because an older job holds its uniqkey, the items waiting for it are now aborted too.

The choose-data step now rejects an empty selection (the lj_ prefix match counted lj_form_auth, so it never did), expands the selection from the registry again at confirm so a hand-built request can't create items that would wait forever, and lists items in registry order. set_import_items_for_user now sets last_touch, which strict-mode MySQL requires.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Only cascade a scheduler clash when this scheduler aborted the item

When two schedulers pick up the same ready item, the one whose job insert clashes found nothing to abort (the other had already queued it) but still aborted everything waiting on the item, so a live import never got past it. Cascade only when the abort actually changed the row.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Claim import items before inserting their scheduler job

The scheduler inserted the TheSchwartz job first and marked the row queued afterwards. In that window a second scheduler could still see the row as ready, clash on the insert and abort the live item's dependents, and a job that ran and finished at once had its 'succeeded' overwritten with 'queued'. Move the row from ready to queued with a guarded update first, skip it if another scheduler got there, and abort only the row this scheduler claimed when the insert fails.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  • Recover import items stranded by claim-first scheduling

The scheduler now claims a row (ready -> queued) before it inserts the job, and only selects 'ready' rows. If it died between the two, or the insert threw, the row stayed 'queued' with no job and its dependents waited forever.

  • An exception from the job insert is handled like a failed insert: the claimed row is aborted, its dependents are cascaded, and the error is logged.
  • Each scheduler pass (at most once a minute) now looks for rows that have been 'queued' for longer than $LJ::IMPORTER_STALE_CLAIM_SECONDS (default 3600) and puts those with no TheSchwartz job for their import back to 'ready', using an UPDATE guarded on the status and the last_touch it observed. Items whose job is still in the queue are never touched: running jobs (entries and comments hold grab_for for hours) and temp-failed jobs awaiting a retry keep their job row. If the queue can't be read nothing is reset. A job that completed but whose worker died before recording 'succeeded' will be run again; imports are idempotent, so that only repeats work.
  • DW::Importer::Queue is the one reader of the TheSchwartz tables: it replaces the controller's private helpers, and the status page's duplicate detection and the scheduler's recovery both ask it whether a job exists for an item of a user (optionally for one import_data_id). It goes through LJ::theschwartz, so it works with the fake queue in tests.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Treat an unreadable queue database as "don't know", not "no job"

Queue->job_count returned a definite 0 whenever any TheSchwartz database could be reached, and turned a failed job query into an empty result. Stale-claim recovery reads 0 as "the job is gone", so a long-running entries job whose queue database was briefly offline, or whose query failed, would have its item reset to ready and run a second time.

job_count now returns undef unless every configured database was connected to and every query on it succeeded (a function name missing from a readable funcmap still means no jobs there). A job that was seen is still reported even if something else could not be read. all_dbhs gives callers the reached and configured counts; each_dbh keeps its behaviour. The status page's duplicate check shares the path and benefits too.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Add content importer tests and fix importer bugs (#3772)

  • Add characterization tests for the content importer

Pin the behaviour of the pieces the importer refactor is about to move: Local::Entries::post_event (idempotence by key, provenance, backdating, logtime), Local::Comments::insert_comment and get_comment_map (key format, threading, per-state effects), trust group and tag merging, groupmask remapping and [profile] user rewriting.

t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML data. It now serves canned responses through a stubbed user agent, so it tests our identity mapping offline and also covers the OPML and local-user branches.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Fix content importer bugs found in the audit

  • set_import_data_options_for_user updated import_data without userid, so every user's import with the same per-user id received the options.

  • Local::Comments insert/update passed a reference to a copy of the error reference, so failures never reached the caller. update_comment also compared the body numerically, which overwrote any non-numeric body when FIX_COMMENT_IMPORT was on.
  • Comments whose entry was never imported were inserted against entry 0. They are now skipped, logged, and reported to the user in a status note.
  • The bio job asked for /data/foaf twice and could not succeed.
  • remap_username_friend returned an LJ::User instead of an id after a rename.
  • get_duplicates_map built keys that could never match its lookup and its result was unused; removed rather than turned into new behaviour.
  • The comment export cache is now keyed on import_data_id: retries reuse the download, a new import fetches fresh data.
  • Admin importer controller declared the user controller's package and the user controller relied on that; each now has its own package and imports. get_queue returns an empty hash instead of a string when the queue is unreachable, render_status resets its duplicate flag per item, confirm.tt reads the right import row for community imports, and the raw DBI connects to TheSchwartz go through the shared LJ::theschwartz client.
  • fail/temp_fail no longer treat the message as a sprintf format when called without arguments, since callers pass error strings from remote sites.
  • Parenthesize the trust bit test in remap_groupmask.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

  • Remove dead content importer code

  • DW::ContentImporter (queue_import/current_job) has no callers and its import_job format conflicts with the scheduler's.

  • DW::Worker::ContentImporter::UserPictures is never enqueued and calls methods that do not exist; drop it and its declarations in the content-importer workers. Nothing reads the import_upi blob or the userpics_later flag that the userpics job set for it, so those go too.
  • remap_username_feed and get_feed_account_from_url were unreachable (their only caller was commented out), and the Bio job never used its user agent.

DW::Worker::ImportEraser is left alone: it is still declared in the deployed worker configuration alongside DW::Task::ImportEraser.

Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Make browser tests self-contained and document them (#3767)

t/browser/editicons-rename.js needed a hand-built account and an exported cookie file, so nobody could run it. Add a fixture script that creates and purges its own account and icons, a shared helper that launches Chrome and logs in, and a README with the run command and the rules for new tests. These remain manual checks; CI doesn't run them.

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

Commit: d43b82b4 Author: Mark Smith

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Keep modal close controls visible without icon fonts (#3765)

The account-links menu and the shared modal wrapper drew their close control with the Foundation icon font. When that font fails to load cross-domain (#3432), the button shows the raw "F217" codepoint. Use the same plain multiplication sign and aria-label as the login popup (#3756).

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

Commit: 247ea4f0 Author: Mark Smith

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Style the canary ribbon on entry previews (#3762)

  • Style the canary ribbon on entry previews

  • Limit legacy preview CSS to the canary ribbon

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Clarify header image options in Heads Up (#3757)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: 1506b0b7 Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Keep the login popup close control visible without icon fonts (#3756)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: d1221efb Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Show full Discord handles on profile pages (#3755)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: 2d6c316e Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Fix header font preferences in A Brighter View (#3753)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: edcdfd3b Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Retain the community when leaving the layer editor (#3752)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: 911f5493 Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Show a pointer cursor for disclosure controls (#3744)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: 26dd35cc Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Remove language settings from the Display description (#3743)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: d61e311f Author: Alex Southwell

github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
[personal profile] github

Refresh calendar counts after entry date edits (#3742)

Co-authored-by: SAPHID saphid@users.noreply.github.com

Commit: 42da0e36 Author: Alex Southwell

Profile

Dreamwidth Changelog

November 2026

S M T W T F S
12 34567
8 9101112 1314
15 161718192021
22232425262728
2930     

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Nov. 16th, 2026 11:53 pm
Powered by Dreamwidth Studios