Performance enhancement for Searchlight indexing
Summary
=======
Improve the time it takes to index (initial or re-index) all plugins.
Problem Statement
=================
When indexing (first time or re-indexing) we will index all resource group types sequentially. We loop through all plugins, indexing each one in turn. The result is that the time it takes to re-index is equal to the sum of the time for all plugins. This may take longer than it should. In some cases a lot longer.
The time it takes to complete the full index is:
n
O( ∑ T(i) )
i=0
When n is the number of plugins and T(i) is the time it takes for plugin i to index.
Description
===========
Two enhancements are proposed.
Enhancement #1:
We should change the algorithm to index in parallel, rather than in serial. As we are looping through each plugin to re-index, we should spin each indexing task into it's own thread. This way the time it takes to index is the time it takes the longest plugin to re-index.
With this enhancement, the time it takes to complete the index is:
n
O( MAX( T(i) ) )
When n is the number of plugins and T(i) is the time it takes for plugin i to index.
Enhancement #2:
Some plugins can take a very looooong time to re-index. An example is a fully loaded Swift. For these cases, we will want to spin off the re-index into an asynchronous task and allow it to finish. Even though the task will finish in the same amount of time, the "searchlight-manage sync index" command will return a lot sooner.
Blueprint information
- Status:
- Complete
- Approver:
- Travis Tripp
- Priority:
- High
- Drafter:
- Rick Aulino
- Direction:
- Approved
- Assignee:
- Rick Aulino
- Definition:
- Approved
- Series goal:
- Accepted for newton
- Implementation:
-
Implemented
- Milestone target:
-
newton-3
- Started by
- Travis Tripp
- Completed by
- Travis Tripp
Related branches
Related bugs
Sprints
Whiteboard
Rick,
I think it would be good to put this into a spec. Should be lightweight. With copy paste of this. It just will serve as some good spec docs.
My one question is how will indexing be communicated to be stopped across threads when one type fails?
Thanks,
Travis
Gerrit topic: https:/
Addressed by: https:/
Spec for performance enhancements for Searchlight indexing.
Addressed by: https:/
To speed up Searchlight indexing and re-indexing we should use threads instead of inexing resource types sequentially.
