![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
[commit: http://hg.dwscoalition.org/dw-free/rev/46d43779211f]
LJ::Comment singletons considered bad for your health
After much digging, it turned out that the slowness encountered on re-
importing was caused by this very simple thing: iterating over, and over,
and over through the %singletons hash to look for which singletons had not
yet been loaded.
Instead of doing that, let's just create some arrays wherein we keep track
of the poor little babies that haven't yet been ripped from the comforting
embrace of the database. Er, well, I mean... let's just keep an array of
which ones aren't loaded yet. Then it's O(N) to load them and everything
goes much, much faster.
Before:
[0.5872s 80666.4s] Attempting to import remote id 300548
After:
[0.0030s 1351.3s] Attempting to import remote id 289204
In other words, it had taken almost a whole day to get to 300,000 on the re-
import. After this change, it has now gotten to nearly the same point in 23
minutes. That's a little bit faster.
Patch by
mark.
Files modified:
LJ::Comment singletons considered bad for your health
After much digging, it turned out that the slowness encountered on re-
importing was caused by this very simple thing: iterating over, and over,
and over through the %singletons hash to look for which singletons had not
yet been loaded.
Instead of doing that, let's just create some arrays wherein we keep track
of the poor little babies that haven't yet been ripped from the comforting
embrace of the database. Er, well, I mean... let's just keep an array of
which ones aren't loaded yet. Then it's O(N) to load them and everything
goes much, much faster.
Before:
[0.5872s 80666.4s] Attempting to import remote id 300548
After:
[0.0030s 1351.3s] Attempting to import remote id 289204
In other words, it had taken almost a whole day to get to 300,000 on the re-
import. After this change, it has now gotten to nearly the same point in 23
minutes. That's a little bit faster.
Patch by
![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
Files modified:
- cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm
- cgi-bin/LJ/Comment.pm