How to reestablish connection to Sharepoint after a failed migration

i) What authentication mode are you using?

ii) 

jonathan manafi said:

If the task is not resumed quickly enough

You can auto-resume after a non-critical error, if that helps.

iii)

> Because if the source is large and >5000 records have already been written, I cannot start another migration to the list without first recreating it and starting from scratch.

Why is that? If it is the 5000 item view limit in SharePoint - you should establish a workaround for that anyway (e.g. folders or an indexed filter for the default view).

i) It looks like O365 authentication?

ii) It's definitely worth looking at, but these have all been classified as critical. Where should I look to see if that's already set and/or to set it? But it does continue processing after reporting warnings; I don't know if that's something that would otherwise need to be resumed manually.

iii) Yes, I'm encountering the item view threshold. I'm migrating them over as list items but Sharepoint is set to display as folders; should that make a difference? Also, I previously tried creating a view to show only items where the title started with certain letters (A - Am, for example) then set that view as the default for the list; but, it did not change the outcome. Maybe because the filter wasn't indexed? I can't remember if it was.

Are you able to give me steps to properly configure a list to accept migrations after 5000 records?

Thank you.

i) You can find the authentication type in the Options - SharePoint - SharePoint environment. Yes, it looks a lot like O365.

 i-a) You can adjust authentication cookie settings under "settings".

i-b) If you can use SingleSignOn - OAuth, this usually makes things a lot easier (and will also reduce throttling).

ii) If you can continue processing, it is a non-critical error. The option is here:

iii-a) 
>  I'm migrating them over as list items but Sharepoint is set to display as folders
Are you migrating the items into folders (with max. 5000 items per folder)? That is usually the easiest way around the 5000-item-view limit.
You can then build additional views that ignore the folders, just leave the default view to honor them.

iii-b)
> Maybe because the filter wasn't indexed?
Yes, the field for the filter needs to be indexed for this to work. There are also a few more restrictions for this to run  - there are a couple of articles on the web that deal with this.

https://docs.microsoft.com/en-us/microsoft-365/community/large-lists-large-libraries-in-sharepoint

https://www.2plus2.com/Blog/August-2017/SharePoint-Document-Library-Challenges-Item-Thresh.aspx

https://sharepointmaven.com/how-to-overcome-sharepoint-5000-item-limit-threshold/

https://www.abelsolutions.com/resolve-5000-item-limit-list-document-library/

i) You can find the authentication type in the Options - SharePoint - SharePoint environment. Yes, it looks a lot like O365.

 i-a) You can adjust authentication cookie settings under "settings".

i-b) If you can use SingleSignOn - OAuth, this usually makes things a lot easier (and will also reduce throttling).

ii) If you can continue processing, it is a non-critical error. The option is here:

iii-a) 
>  I'm migrating them over as list items but Sharepoint is set to display as folders
Are you migrating the items into folders (with max. 5000 items per folder)? That is usually the easiest way around the 5000-item-view limit.
You can then build additional views that ignore the folders, just leave the default view to honor them.

iii-b)
> Maybe because the filter wasn't indexed?
Yes, the field for the filter needs to be indexed for this to work. There are also a few more restrictions for this to run  - there are a couple of articles on the web that deal with this.

https://docs.microsoft.com/en-us/microsoft-365/community/large-lists-large-libraries-in-sharepoint

https://www.2plus2.com/Blog/August-2017/SharePoint-Document-Library-Challenges-Item-Thresh.aspx

https://sharepointmaven.com/how-to-overcome-sharepoint-5000-item-limit-threshold/

https://www.abelsolutions.com/resolve-5000-item-limit-list-document-library/

i) Yes, those are the settings that are selected for authentication, and the account being used is a Sharepoint/AD account.

i-a) I'm not sure what settings are ideal, but these are active/checked

  Retry 5 times before reporting throttle error

  Refresh cookies 30 mins before they are due to expire

  Cache authentication cookies in configuration file

i-b) I don't see an option for SSO; am I just blind?

ii) I'm trying another migration with this setting unchecked, so I'll see how it fares.

iii-a) I'm honestly not sure if that's the case. There aren't any folders specifically created in the list(s) ahead of time, so I'm not setting any limits that I know of. There's a field called FolderPath from the source database that's mapped to Folder in Sharepoint, so that's what is being used to create the folder structure in the list.

The source db has a structure of 

Vendor

    -- 2016

        - Invoice1

        - Invoice2

    -- 2017

        - Invoice1

        - Invoice2

    -- 2018

        - Invoice1

        - Invoice2

and that's how the folders are being created during the migration, using any / in the FolderPath value to designate a sub-directory

iii-b) To coincide with the new migration attempt, I setup indexes on a handful of columns in the list ahead of time, so I'll see if that helps as well

i-b)
What version of MNSP are you using? This was added in 7.0.0 (but please use at least 7.0.1, preferably 7.1.0):

i-a) I am honestly not up-to-date what the best settings are currently - I know we haven't changed the defaults for years, and since Microsoft has changed SharePoint Online behavior often, these are likely not ideal.

iiia) If your default view in SharePoint starts showing folders (and no items within folders), you should be good.
You don't need to pre-create folders - mapping a Notes fields to the "Folders" field in SharePoint is exactly the recommended way.

Hi Peter,

i-b) We have version 6.15.0.2045. This software was installed previously by a Quest consultant - do we need to have a proper license to upgrade to newer versions? Is there any reason I'd have trouble upgrading the software myself, or is it a pretty standard install/upgrade procedure?

When I got to work this morning, the migration was still running. Unchecking the "prompt when recoverable errors" option allowed it to run all night and continue past the 2 errors that were encountered during that time, so it was a success beyond what I'd been experiencing.

However, right before the migration hit the 24 hour mark, every record it tried to write gave either a 401 unauthorized or 403 forbidden error (I started at 11:58 AM 8/24, this started happening at 11:47 8/25). It ended with 6258 errors by the end.

I guess because the "prompt for errors" option isn't checked there's no option at the end of migration to migrate failed items? So, I tried to load failed items from the log but the limit is 1500, so after doing some surgery on the log, I tried migrating 460 of the failed items. At that point, I get the "exceeded list view threshold" error.

The list now contains 30,000+ records, but because of the folders, the default view only has 528 items. Any ideas?

7.1 will run with a version 6 license - but you need to be under a valid support contract to obtain the new binaries/installer.

The installer upgrades from at least two versions back, so 6.15.0 -> 7.0.1 should work. I would expect 6.15 -> 7.1 will also work, but I am not sure.

I am ooo today - will try to do some testing with >5000 items into folders on Monday.

I am testing right now with 8600+ items stored in folders in a SharePoint Online list, and I can start new migration jobs without problems (all using O365 authentication, MNSP 7).

Microsoft names another limit at 20,000 items, but I can only find that it is relevant for index creation: "Automatic index creation when sorting in the modern experience is limited to lists and libraries with less than 20,000 items."