Skip to:
Content
Pages
Categories
Search
Top
Bottom

Large Forum Import Hangs ( Vbulletin 3 )


  • themefurnace
    Participant

    @themefurnace

    Hello,

    I wondered if anyone had any advice for importing larger forums,
    I run the importer and get 140k topics imported, replies are hanging around the 12k mark ( there are 800K posts on the original forum).

    Ive changed php.ini max execution times etc so I no longer get errors there.
    Ive tried this on my server and also locally with no joy.

Viewing 2 replies - 1 through 2 (of 2 total)

  • Stephen Edgar
    Moderator

    @netweb

    I am expecting this will be a reply in the source database has some weird content.

    To quote @JohnJamesJacoby here:

    These types of faults are typically due to some encoding issue in that post. Sometimes this is from an encoding conversion to UTF8 that was done at some point. Whatever it is, it’s nearly impossible to detect without setting a time limit on queries and bisecting the difference until we can narrow down the offending entry, and skip it. Because queries could naturally take a long time, and because it results in missed data, I’m not comfortable with this approach.

    The workaround is to find and delete the offending reply which is time consuming and tricky but the only way at this stage.

    Here’s how I do debug this and find the offending topics/replies:

    • Once the import hangs, click stop, take a note of where it was up to, e.g. Converting 11,299 – 11,300
    • Click start again and the import will continue from where it left off
    • If it hangs again, take note of where it was up to again
    • Once finished make a fresh copy of the source database
    • Now drop all the rows in the database that are not in the range of rows that failed during import
    • Perform a fresh import using this database with the “Rows Limit” setting set to 1
    • The import should hang on the offending topic/reply
    • Again, take note of the topic/reply it was up to
    • Click stop, click start again to continue on
    • Once you have your list of offenders and the import has finished delete these rows from this database and test again to make sure it completes without hanging
    • Now back in your full database (presumably a copy of the original) delete these same rows from the database and run the import again

    themefurnace
    Participant

    @themefurnace

    Hi Stephen,

    Thanks for your advice – I have done some more testing and its not actually stopping completely – Once its past 12k replies it starts going very slowly.
    Im up to 13,500 replies now but its incredibly slow, I wonder if there are any tips / solutions for this problem ?

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.
Skip to toolbar