Split LiveJournal imports into stages and add a dry-run mode (#3775)
- Add characterization tests for the content importer
Pin the behaviour of the pieces the importer refactor is about to move:
Local::Entries::post_event (idempotence by key, provenance, backdating,
logtime), Local::Comments::insert_comment and get_comment_map (key format,
threading, per-state effects), trust group and tag merging, groupmask
remapping and
user rewriting.
t/importer-remap-username.t used to ask livejournal.com for FOAF and OPML
data. It now serves canned responses through a stubbed user agent, so it
tests our identity mapping offline and also covers the OPML and local-user
branches.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
Fix content importer bugs found in the audit
set_import_data_options_for_user updated import_data without userid, so
every user's import with the same per-user id received the options.
- Local::Comments insert/update passed a reference to a copy of the error
reference, so failures never reached the caller. update_comment also
compared the body numerically, which overwrote any non-numeric body when
FIX_COMMENT_IMPORT was on.
- Comments whose entry was never imported were inserted against entry 0.
They are now skipped, logged, and reported to the user in a status note.
- The bio job asked for /data/foaf twice and could not succeed.
- remap_username_friend returned an LJ::User instead of an id after a rename.
- get_duplicates_map built keys that could never match its lookup and its
result was unused; removed rather than turned into new behaviour.
- The comment export cache is now keyed on import_data_id: retries reuse the
download, a new import fetches fresh data.
- Admin importer controller declared the user controller's package and the
user controller relied on that; each now has its own package and imports.
get_queue returns an empty hash instead of a string when the queue is
unreachable, render_status resets its duplicate flag per item, confirm.tt
reads the right import row for community imports, and the raw DBI connects
to TheSchwartz go through the shared LJ::theschwartz client.
- fail/temp_fail no longer treat the message as a sprintf format when called
without arguments, since callers pass error strings from remote sites.
- Parenthesize the trust bit test in remap_groupmask.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
DW::Worker::ImportEraser is left alone: it is still declared in the deployed
worker configuration alongside DW::Task::ImportEraser.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Add an importer registry and generic dependency chaining
DW::Importer::Registry is now the one place that says what can be imported:
the sources and their sites, and for each item its worker class, status
event typeid, dependencies (per journal type where they differ), initial
status, community support, options, worker process and whether success
notifies the user. The item to class map used to be copied into the
scheduler, both controllers and importadm; the typeids lived in
LJ::Event::ImportStatus; dependency rules were in the controller, the
JavaScript and the SQL of five workers. All of those now read the registry,
as do the worker scripts' schwartz_decl lists, get_queue and
DW::Logic::Importer::allowed_sources. The choose-data page hands its
dependency map to jquery.importer.js as data. Item names, worker classes and
typeids are unchanged, since queued jobs and stored rows refer to them.
DW::Importer::Items::advance replaces the per-worker chaining SQL and is
called from the shared ok/fail: finishing an item readies the waiting items
whose dependencies have all succeeded, and a permanent failure aborts the
waiting items that depended on it. The readiness graph is the one the workers
encoded (verify -> bio/userpics/friendgroups/tags, friendgroups ->
friends/entries, tags -> entries for communities only, entries -> comments).
The abort cascade is new: previously a failure left everything downstream in
'init' forever. It also subsumes verify's "Invalid password" special case.
The scheduler's body moves to DW::Importer::Scheduler so it can be run once
under test; bin/worker/import-scheduler is a thin loop around it. When the
scheduler has to abort an item because an older job holds its uniqkey, the
items waiting for it are now aborted too.
The choose-data step now rejects an empty selection (the lj_ prefix match
counted lj_form_auth, so it never did), expands the selection from the
registry again at confirm so a hand-built request can't create items that
would wait forever, and lists items in registry order. set_import_items_for_user
now sets last_touch, which strict-mode MySQL requires.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Only cascade a scheduler clash when this scheduler aborted the item
When two schedulers pick up the same ready item, the one whose job insert
clashes found nothing to abort (the other had already queued it) but still
aborted everything waiting on the item, so a live import never got past it.
Cascade only when the abort actually changed the row.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
- Claim import items before inserting their scheduler job
The scheduler inserted the TheSchwartz job first and marked the row queued
afterwards. In that window a second scheduler could still see the row as
ready, clash on the insert and abort the live item's dependents, and a job
that ran and finished at once had its 'succeeded' overwritten with
'queued'. Move the row from ready to queued with a guarded update first,
skip it if another scheduler got there, and abort only the row this
scheduler claimed when the insert fails.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
- Recover import items stranded by claim-first scheduling
The scheduler now claims a row (ready -> queued) before it inserts the job,
and only selects 'ready' rows. If it died between the two, or the insert
threw, the row stayed 'queued' with no job and its dependents waited forever.
- An exception from the job insert is handled like a failed insert: the
claimed row is aborted, its dependents are cascaded, and the error is logged.
- Each scheduler pass (at most once a minute) now looks for rows that have been
'queued' for longer than $LJ::IMPORTER_STALE_CLAIM_SECONDS (default 3600) and
puts those with no TheSchwartz job for their import back to 'ready', using an
UPDATE guarded on the status and the last_touch it observed. Items whose job
is still in the queue are never touched: running jobs (entries and comments
hold grab_for for hours) and temp-failed jobs awaiting a retry keep their job
row. If the queue can't be read nothing is reset. A job that completed but
whose worker died before recording 'succeeded' will be run again; imports are
idempotent, so that only repeats work.
- DW::Importer::Queue is the one reader of the TheSchwartz tables: it replaces
the controller's private helpers, and the status page's duplicate detection
and the scheduler's recovery both ask it whether a job exists for an item of
a user (optionally for one import_data_id). It goes through LJ::theschwartz,
so it works with the fake queue in tests.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Run importer items through a shared job runner
DW::Importer::Job is now the base of every item worker (it sits between
TheSchwartz::Worker and DW::Worker::ContentImporter, so the worker classes
and their TheSchwartz function names are unchanged). A worker implements
run($ctx) and returns, or dies with, a DW::Importer::Outcome (ok, fail or
temp_fail). The base owns what the eight workers each copied: loading the
import and user, declining when the source is disabled, the eval and
outcome handling, the item status, the notification (to every maintainer of
a community), stats, the TheSchwartz job state, escalation of a temporary
failure after max_retries, the registry-driven chaining, the process title
and the log.
DW::Importer::Context carries the per-job state: user, import data with
thawed options, registry item, logger and a cache hash. It replaces the
package-global %MAPS (the remote-user and group maps are now cached in the
context, so nothing has to be cleared between jobs) and remap_groupmask and
get_remapped_userids take the context. DW::Importer::Logger is the one
implementation of the "[delta total]" log written to the per-import file and
the job's debug output, with an in-memory variant for tests.
work() no longer calls exit 0. The entries and comments workers set
exit_after_job and the content-importer scripts exit from TheSchwartz's
after-work hook once the job has completed, so production still reclaims
memory but work() can be called in a test.
Verify, Bio, Tags, FriendGroups, Friends and Userpics are ported fully.
Entries and Comments only move their skeleton and failure handling onto the
base; their fetch, transform and post code is as it was, except that a
failure inside a closure is raised as an outcome instead of a return value
that every caller had to check. A log file that can't be created no longer
fails the job. bin/importadm runs a job through work() now.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Treat an unreadable queue database as "don't know", not "no job"
Queue->job_count returned a definite 0 whenever any TheSchwartz database could
be reached, and turned a failed job query into an empty result. Stale-claim
recovery reads 0 as "the job is gone", so a long-running entries job whose
queue database was briefly offline, or whose query failed, would have its item
reset to ready and run a second time.
job_count now returns undef unless every configured database was connected to
and every query on it succeeded (a function name missing from a readable
funcmap still means no jobs there). A job that was seen is still reported
even if something else could not be read. all_dbhs gives callers the reached
and configured counts; each_dbh keeps its behaviour. The status page's
duplicate check shares the path and benefits too.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Add normalized import records and pure transforms
First step of splitting the LiveJournal entries and comments jobs into fetch,
transform and write stages. Nothing is wired in yet.
- DW::Importer::Records documents the record shapes sources produce and
sinks consume: entry, comment, tag, trust group, friend, userpic, bio and
the action a sink reports.
- Transform::LJMarkup holds the user-tag, poll, embed and template rewriting
that the entries and comments jobs each carried inline, keeping the
differences between the two, and returns warnings instead of logging.
- Transform::Security maps remote security, access mask and group map to a
local security and mask. It fails closed: an unrecognised security or mask
is private with a warning, unmapped groups are dropped and never widened,
and a mask with nothing mappable left is private with the "Could not
determine groups to post to." warning the entries job used to add inline.
- Threading orders comments parent first, keeps replies whose parent has not
arrived yet for the next window, and reports comments whose entry is not
imported and replies whose parent never was.
- Source::LiveJournal::Keys builds and reads entry and comment keys. The
strings are pinned against what the importer's own writers store.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Move LiveJournal remote I/O into a source adapter with an injected transport
DW::Importer::Source::LiveJournal owns every request to the remote site and
returns the normalized records: tags, trust groups, friends, bio (FOAF),
userpics, entries (with their keys and URLs), the newest entry id, identity
lookups (FOAF/OPML, rename history) and the comment export. It writes nothing
locally and dies with DW::Importer::Outcome objects, so the jobs only
sequence work. Its transport (XML-RPC, plain HTTP, the comment export with
its per-import disk cache and login session) is the first thing a job's
context hands it; Source::LiveJournal::Transport is the real implementation,
moved from LiveJournal.pm and Comments.pm, and tests substitute their own.
DW::External::XPostProtocol::LJXMLRPC has the same XML-RPC shape but a
different auth flow and error format, so it is left alone.
The comment export's window and nextid logic is carried over as it was. What
changed is where state lives: a window is read, then handed over complete, so
a comment waiting for its parent no longer sits in the half-read export state
(on an encoding retry it used to lose its text). Threading decides order and
reports what could not be placed.
All eight workers now run adapter -> transform -> write using the pure
transforms from the previous commit; the writes are still the Local::* calls
(the sink is next). Entries now maps security through Transform::Security, so
a mask whose groups could not be mapped is stored as private instead of as
usemask with an empty mask, and also when only unmapped groups were listed.
The comments job tells the user about replies whose parent was never
imported, besides comments on entries that were not.
Tests feed canned XML-RPC and export windows through the workers and the
adapter, including nextid resume, servers with no nextid, overlapping windows
and the Latin-1 fallback.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Write imports through a sink, with a live and a dry-run implementation
Every local write an import makes now goes through DW::Importer::Sink:
posting entries, updating entry icons, inserting and updating comments,
tags, trust groups and the group map, friends, interests and bio, icons,
identity accounts and the remote-user map, and the search copier dispatch.
The known-key queries (entry map, comment map, crossposted entries, group
map, user map) are on it too.
Sink::Live wraps the existing Local::* code and owns the entry map the job
reads and extends. Local::Comments no longer keeps its caches in package
variables: the sink holds them and passes them in.
Sink::DryRun records each write as an action in a DW::Importer::Report and
changes nothing. Reads go to a live sink, so entries and tags that already
exist are reported as skipped. It makes up ids that cannot be real ones, and
group ids that are valid access mask bits, so that comments can still be
threaded and later jobs of one dry run (given the same world) see what the
earlier ones would have made. Identity lookups use the read-only
load_existing_identity_user.
All eight workers write only through the sink; remote users are resolved
through it as well, and the remote-user map row is now only written when
the user was not already in it (rewriting an identical row did nothing).
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Add dry-run mode for imports
DW::Importer::Job->dry_run runs one item in-process with a Sink::DryRun and
dry_run_chain runs the import's items in dependency order, each seeing what
the earlier ones would have written (an item whose dependency failed is not
run). The runner leaves out everything that is not the job's own work: the
item's status and dependency advancement, ImportStatus notifications (the
report keeps what the user would have been told), stats, TheSchwartz job
state, per-import log files, the process title and the exit request. All
memcache writes are suppressed, so a dry run can read through the cache but
leaves nothing in it. The Verify job does not run the ban_unset console
command on the remote site, which is a remote write; the report says so.
DW::Importer::Report collects the actions with counts per type, what would
be skipped as already imported, warnings, comments that could not be placed
and the jobs' outcomes, and renders them as text. The dry-run sink reports
tags, groups and the like that already exist as skipped rather than as
writes.
bin/importadm --dry-run [--type ITEM] [--import-id N] [--verbose] prints the
report; --run keeps its meaning. --list now prints the directory the import
logs are actually written to.
t/importer-dryrun.t runs a fixture import (public, private, friends-only,
group-locked and unmappable-group entries, a crosspost, threaded, screened,
deleted, anonymous, orphaned and unmapped comments) as a dry run, which
changes no row of anything an import writes and puts nothing in memcache;
then for real, which posts what the report listed with the security and
threading it gave; then again, which adds nothing, and a dry run after that,
which finds it all already there.
Two things the fixture showed. The site rejects an access mask that has both
the trusted bit and groups, so Transform::Security now makes such an entry
private with a warning instead of letting the post fail. And re-importing a
comment rewrote its icon property to the value it already had (or to 0 on a
comment with none); update_comment now writes it only when it changes.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Add Ed25519 verification for signed export uploads
DW::Importer::Signature checks a detached Ed25519 signature over the exact
bytes of an uploaded export, for sources that sign their files. Trusted
public keys come from %LJ::IMPORTER_SIGNING_KEYS, a list per source so keys
can be rotated; a key is 32 raw bytes as base64 or a PEM public key, and
private keys are refused. The signature may be standard or URL-safe base64,
padded or not, with any whitespace. verify() returns true or false, never
dies on malformed input, logs why, and with no usable key for the source it
fails closed. etc/config-local.pl.example documents the setting.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com
- Make dry runs write no files
The comment export transport wrote its XML cache on every miss, and the
comments job wrote its .failure note on every exception, in a dry run too. A
cache file made by a dry run is then trusted by the real import that follows.
The transport is now told when it is in a dry run (from the job context, or
dry_run()): it still reads an existing cache but never writes one, and says so
in the log. The failure note is not written in a dry run; the reason goes to
the run's logger instead. Both places now use lexical filehandles and check
the open and close, so a cache that cannot be written is logged instead of
printed to a closed handle.
The rest of the dry-run path has no file writes: per-import log files are only
made for real runs (a dry run logs in memory or to STDERR), the
please_die file is only written when a real job fails, and the icon download
is in the live sink.
Co-Authored-By: Claude Sonnet 5.5 noreply@anthropic.com