The BIG Perforance Test lots of records

Feb 5, 2013 at 7:57 PM
OK,
I am now digging into a large list/document library.

There are way over the recommended record count. I think I remember seeing 200,000+ records. So SLAM may save the clients butt.

I started running a Sync on the document library and I am not seeing any records for the document library in the EventLog table yet.

I was looking through the code and I am trying to see how you are querying the records. Are you returning all of the columns or just the ones requested in the slam.config file? I did not notice any ViewFields calls to a SPQuery.

I am just trying to see if this will work or it it will blow up the server. I am seeing the app pol recycle and the server has 16 GB of RAM on it. So I know it is grabbing a lot of data.
Developer
Feb 5, 2013 at 8:44 PM
SLAM sync is still iterating over the collection of list items. It does only save field data for mapped fields, but it still needs to go through every item.

Do you see it doing anything? In the message log is there an error?
Feb 5, 2013 at 9:19 PM
I just checked the database. It did pull the first 55 records but it also threw a error, looks like it just died, kind of one of those generic errors.

System.NullReferenceException: Object reference not set to an instance of an object.
at Slam.Data.SqlLogDataRepository.LogEvent(String tableSchema, String eventName, String typeName, String message)
at Slam.Logging.SqlLogger.LogEvent(String eventName, String typeName, String message)
at Slam.Events.DefaultEvents.Update(SPListItem listItem, IDataServices dataServices, IConfigurationManager configuration, ILogger logger, String eventPrefix)
at Slam.Events.ListItemUpdater.Update(SPListItem listItem)
at Slam.SharePoint.SPListItemExtender.SlamUpdate(SPListItem listItem, Boolean runWithElevatedPriveleges, Boolean saveItem, Boolean saveAsSystem, String updateUserName, Boolean isNew, Boolean updateFolderChildren)
at Slam.SharePoint.SPListItemExtender.SlamUpdate(SPListItem listItem, Boolean runWithElevatedPriveleges, Boolean saveItem, Boolean saveAsSystem)
at Slam.Activation.SqlDataSynchronizeTypeProcessor.SynchronizeListItem(SPListItem listItem)
at Slam.Activation.SqlDataSynchronizeTypeProcessor.<>c__DisplayClass4.<Process>b__0()
Feb 5, 2013 at 10:36 PM
I never dug into why, but when I attempted to use SLAM to sync with an even smaller list (<90k items), it would never get past the 40k mark before it failed/stopped. I ended up not using SLAM to do the initial sync.

Instead I allowed SLAM to create the database structure then populated the data using an SSIS package. The SharePoint List Source and Destination components published my Microsoft helped to do this.

After that, SLAM only needed to be activated to start listening to modifications to the list and keep them up to date.
Feb 6, 2013 at 12:51 PM
UGH,
If that is the case, I am just use SSIS for the complete process. I can always run that on a schedule.

But before I give up, I am going to download the content database and try and run the locally.
Feb 8, 2013 at 8:23 PM
OK,
I have been doing some digging and I found where the error is happening.
\2010\2.0\AWS.SLAM\LAYOUTS\SLAM\Synchronization.aspx.cs
Line 321
return Cache["syncstatus" + hdnSessionKey.Value.ToString()].ToString();
Error: Object reference not set to an instance of an object

I am not sure exactly what this does yet, I am still digging, if this looks like something you may know about, let me know.
Feb 13, 2013 at 3:21 PM
OK,
I have been doing some digging in the code. It looks like it is really an out of memory issue.

When I looked through the code I am not seeing where you are calling the view fields. I know if you limit the view fields to just the ones you need, it will increased performance.

Can you tell me where you are calling the lists, I may try and build a function that pulls only the fields needed in your XML.
Developer
Feb 13, 2013 at 3:44 PM
For sync all of the loops through each list's items is in SqlDataSynchronizeTypeProcessor. Each loop looks like:
foreach (SPListItem listItem in list.Items)
{
    SynchronizeListItem(listItem);
}
The challenge for updating the sync process to not use SPListItems is that it won't be able to use SlamUpdate.
Feb 13, 2013 at 4:06 PM
What if there were a separate method called if it was doing a manual sync?
Developer
Feb 13, 2013 at 4:28 PM
I am not sure what I was thinking earlier, but actually it wouldn't be a problem. A loop over a GetItems result would obviously be exchangeable for the loop over Items.
Feb 13, 2013 at 8:09 PM
I guess the issue here is a few things.
1) Can the fields needed from the list be determined easily?
2) I am not an expert of you code but I do a tremendous amount of SharePoint coding. Is this something you could implement?

If you have a collection of fields I can write a method that would convert it to XML, if needed for a view fields.

I just hope I can slim the process down so it does not the out of memory error.
Developer
Feb 14, 2013 at 4:01 PM
With 200000 items in a list it may still be that limiting the fields returned will solve the memory issue. It may be better to try paging at first with a RowLimit of, say, 1000 items while still returning all fields.

Otherwise, to get the fields for a type in the SqlDataSynchronizeTypePProcessor you would do currentDataMapping.GetTypeFields(inputType.Id, inputType.Name).
Feb 14, 2013 at 4:29 PM
Do you have paging turned on in your code? I have not looked yet.
Feb 18, 2013 at 1:07 PM
Just to let you know, I am doing this on a webpart but it is taking 15 seconds to load the records. But I am doing it with view fields and SPQuery. I am just worried if I start digging into this that it might end up being a rewrite and then it would be better for me to start from scratch.
Mar 5, 2013 at 8:45 PM
OK,
I am trying to update my code, what folder do I use? 2.0 or 3.0?
Developer
Mar 5, 2013 at 8:52 PM
Use Main.
Mar 5, 2013 at 8:55 PM
Are you using VS 2010 or 2012 for this?
Developer
Mar 5, 2013 at 8:58 PM
VS 2012
Mar 5, 2013 at 9:23 PM
OK,
It is crunching on the live server! I am not seeing any memory leaks anymore. This might work. I will let you know tomorrow.
Mar 7, 2013 at 7:29 PM
OK,
Please let em know if you see any issues and maybe when you could have this in the final version?
Rick

Here is my code:

This is the method I added under Core\Activation\SqlDataSynchronizeTypeProcessor.cs
    /// <summary>
    /// Helper method pages the list items.
    /// </summary>
    /// <param name="currList">This is the current List object</param>      
    private void ProcessList(SPList currList)
    {
        SPQuery spQuery = new SPQuery();
        spQuery.RowLimit = 1000;
        spQuery.ViewAttributes = "Scope=\"Recursive\"";

        do
        {
            SPListItemCollection collListItems = currList.GetItems(spQuery);

            foreach (SPListItem listItem in collListItems)
            {
                SynchronizeListItem(listItem);
            }
            spQuery.ListItemCollectionPosition = collListItems.ListItemCollectionPosition;
        } while (spQuery.ListItemCollectionPosition != null);

    }
I replaced:
    //Call update for every item in the list
    //foreach (SPListItem listItem in list.Items)
    //{
    //    SynchronizeListItem(listItem);
    //}


    //foreach (SPListItem listItem in list.Folders)
    //{
    //    SynchronizeListItem(listItem);
    //}
With:
    ProcessList(list);