Dreamwidth Changelog

[commit: http://hg.dwscoalition.org/dw-free/rev/46d43779211f]

LJ::Comment singletons considered bad for your health

After much digging, it turned out that the slowness encountered on re-
importing was caused by this very simple thing: iterating over, and over,
and over through the %singletons hash to look for which singletons had not
yet been loaded.

Instead of doing that, let's just create some arrays wherein we keep track
of the poor little babies that haven't yet been ripped from the comforting
embrace of the database. Er, well, I mean... let's just keep an array of
which ones aren't loaded yet. Then it's O(N) to load them and everything
goes much, much faster.

Before:

[0.5872s 80666.4s] Attempting to import remote id 300548

After:

[0.0030s 1351.3s] Attempting to import remote id 289204

In other words, it had taken almost a whole day to get to 300,000 on the re-
import. After this change, it has now gotten to nearly the same point in 23
minutes. That's a little bit faster.

Patch by mark.

Files modified:

• cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm
• cgi-bin/LJ/Comment.pm

( Diff: 2 files changed. )

[commit: http://hg.dwscoalition.org/dw-free/rev/8b78cdfc4bc2]

Fix issue with truncated picture keywords

The summary here is that sometimes, when people use characters that are not
standard Latin (i.e., when the byte value is higher than 0x7f), sometimes we
get in a situation where the LJ::text_trim function cuts off the right edge
because it assumes UTF-8 input and the input isn't.

The solution is to encode it as UTF-8, which is what this patch does. This
patch also re-enables the comment updating since that's the only way to fix
the situation where someone has run into this bug. Simply re-importing will
fix any broken comments.

This was a pain to track down.

Patch by mark.

Files modified:

• cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm

( Diff: 1 file changed. )