[dw-free] Fix issue with truncated picture keywords

[mark]

[commit: http://hg.dwscoalition.org/dw-free/rev/8b78cdfc4bc2]

Fix issue with truncated picture keywords

The summary here is that sometimes, when people use characters that are not
standard Latin (i.e., when the byte value is higher than 0x7f), sometimes we
get in a situation where the LJ::text_trim function cuts off the right edge
because it assumes UTF-8 input and the input isn't.

The solution is to encode it as UTF-8, which is what this patch does. This
patch also re-enables the comment updating since that's the only way to fix
the situation where someone has run into this bug. Simply re-importing will
fix any broken comments.

This was a pain to track down.

Patch by mark.

Files modified:

• cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm

--------------------------------------------------------------------------------
diff -r 139ce6219023 -r 8b78cdfc4bc2 cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm
--- a/cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm	Tue Jan 31 09:52:41 2012 -0600
+++ b/cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm	Wed Feb 01 00:45:26 2012 +0000
@@ -320,6 +320,9 @@
             # character encoding
             $comment->[C_body] = encode_utf8( $comment->[C_body] || '' );
             $comment->[C_subject] = encode_utf8( $comment->[C_subject] || '' );
+            foreach my $prop ( keys %{$comment->[C_props]} ) {
+                $comment->[C_props]->{$prop} = encode_utf8( $comment->[C_props]->{$prop} );
+            }
 
             # this body is done
             $comment->[C_body_fixed] = 1;                
@@ -358,9 +361,7 @@
                 $comment->[C_local_parentid] = $jtalkid_map->{$comment->[C_remote_parentid]}+0;
                 $comment->[C_id] = $jtalkid;
 
-# Temporarily disabled, this slows down reimports a LOT and we aren't
-# really doing anything with them.
-#DW::Worker::ContentImporter::Local::Comments->update_comment( $u, hashify( $comment ), \$err );
+                DW::Worker::ContentImporter::Local::Comments->update_comment( $u, hashify( $comment ), \$err );
                 $log->( 'ERROR: %s', $err ) if $err;
 
                 $comment->[C_done] = 1;
--------------------------------------------------------------------------------

Comments

[kareila]

We love you Mark!

[mark]

Awww, thank you. Mutual. <3

[denise]

Wow. So small a patch for all the hassle it caused. Heh.

[mark]

The intermediate steps took something like six hours and I put in at least 100 lines of debugging to create lots of temporary files. They were all named /tmp/dammit through /tmp/dammit5 or something.

[denise]

Only 'dammit'? You're a stronger man than I. :)