[commit:
http://hg.dwscoalition.org/dw-free/rev/ea5fc2c10527]
http://bugs.dwscoalition.org/show_bug.cgi?id=4112This is basically a revamp of the comment import code so that it runs in
constant(ish) memory. The idea is to publish comments as they are body
fetched.
Since we still have to do the metadata import first (well, maybe not, but we
do it that way for now) I have also changed from using hashes (memory
intensive when there's millions) to using arrays.
I've also added preloading for users and props which saves a little bit of
time, too.
This still needs more testing, particularly with really large import
targets.
Patch by
mark.
Files modified:
- bin/worker/content-importer
- bin/worker/import-scheduler
- cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm
- cgi-bin/DW/Worker/ContentImporter/LiveJournal/Entries.pm
- cgi-bin/DW/Worker/ContentImporter/Local/Comments.pm
- cgi-bin/LJ/Entry.pm
- cgi-bin/LJ/Talk.pm
- cgi-bin/LJ/User.pm
- cgi-bin/LJ/Web.pm
( Diff: 9 files changed. )