github: shadowy octopus with the head of a robot, emblazoned with the Dreamwidth swirl (Default)
github ([personal profile] github) wrote in [site community profile] changelog2026-08-05 05:11 am

Archive cron job output to S3

Archive cron job output to S3

Cron tasks run as one-shot Fargate tasks on an ephemeral filesystem, so anything a job writes to disk is discarded when the task exits. genstats writes htdocs/stats/stats.txt and genstatspics writes newbyday.png; both have been going nowhere since the crons moved off the old cron host, which is why /stats/stats.txt has served a file frozen at 2026-07-04.

Add bin/archive-to-s3.pl, which copies files or directories to the dreamwidth-archive bucket under //. It uses Paws (already in doc/dependencies-cpanm) rather than the AWS CLI, which is not in the image, and dies on any failure so the cron failure alarm fires instead of an archive being silently skipped.

Producing a file and shipping it have to happen in the same task to share the same disk, so startup-cron.sh now accepts "--" between chained commands. Single-command crons are unaffected; a failing step aborts the chain and its exit code becomes the task's.

The matching cron command and ARCHIVE_BUCKET/ARCHIVE_REGION environment come from dw-terraform, and must be applied only after this ships in worker22:latest.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Commit: e7c5d7f6 Author: Mark Smith