github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
github ([personal profile] github) wrote in [site community profile] changelog2026-04-22 07:10 pm
  • Add Memory
  • Share This Entry

[dreamwidth/dreamwidth] 776ac7: Add DW::Task::SearchCopier path and bin/search-tool

Branch: refs/heads/main Home: https://github.com/dreamwidth/dreamwidth Commit: 776ac7cd8c2185b53beb87c4c460205d19f00be3 https://github.com/dreamwidth/dreamwidth/commit/776ac7cd8c2185b53beb87c4c460205d19f00be3 Author: Mark Smith mark@dreamwidth.org Date: 2026-04-23 (Thu, 23 Apr 2026)

Changed paths: A .github/workflows/tasks/worker-dw-search-copier-service.json M .github/workflows/worker22-deploy.yml R bin/schedule-copier-jobs A bin/search-tool A bin/worker/dw-search-copier A cgi-bin/DW/Task/SearchCopier.pm M cgi-bin/LJ/DB.pm M config/workers.json M etc/workers.conf

Log Message:

Add DW::Task::SearchCopier path and bin/search-tool

Stand up the new manticore-rt write path side-by-side with the legacy sphinx-copier. Nothing in production dispatches to it yet — only bin/search-tool's import-* subcommands. The two paths can run in parallel through cutover.

  • cgi-bin/DW/Task/SearchCopier.pm: new task class. Auto-routes to its own SQS queue (dw-prod-dw-task-searchcopier) via class-name derivation. Mirrors SphinxCopier's argument shape (full recopy, single jitemid, single jtalkid) and its security_bits / state / text-decode handling, so the search worker's filter contract stays intact when we eventually flip readers over. Tracks per-run stats (entries/comments/deletes ok/err); summary log is debug on clean success, warn when there are errors. Independent 24h memcache throttle on full recopies (separate key from sphinx-copier's).

  • bin/worker/dw-search-copier: 36-line runner cloned from dw-sphinx-copier; pulls from the new queue, calls work().

  • etc/workers.conf: add dw-search-copier: 1 so worker-manager spawns it.

  • bin/search-tool: CLI helper for the migration. Subcommands import-user, import-all, import-support, search, show, delete, count, flush. import-user delegates to SearchCopier->work() so the CLI and the worker share one code path. import-all replaces the retired bin/schedule-copier-jobs (deleted in this commit).

  • config/workers.json: register dw-search-copier as an ECS Fargate worker (256 cpu / 512 mb, spot, target 30% cpu, scale 1-10). The .github/workflows/ files are the auto-generated CI artifacts from running update-workflows.py.

LJ/DB.pm: incidental tidyall whitespace-only fixup.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

To unsubscribe from these emails, change your notification settings at https://github.com/dreamwidth/dreamwidth/settings/notifications