Implications of on-the-fly incremental indexing in a CMS

[jookyg]

Hello,

My company uses a custom built ASP.NET CMS, and we've licensed Zoom Search as our search tool. So far we love it, especially after some of the other 3rd party tools we've tried.

Now we have a situation where a client wants updates to appear in search results immediately, instead of waiting for the nightly reindex. We want to set up our CMS to automatically re-index the updated page on the fly whenever the admin clicks Save, using System.Diagnostics.Process.Start to call the indexer or a .bat file with the appropriate arguments.

It's a 3500+ page site, which may see dozens of updates each day. Multiple admins may be updating different pages simultaneously. Security is a big concern, so we will need to lock down the incremental indexing calls pretty tightly.

I have some questions about this setup:

• What will happen if several incremental indexing requests are submitted to the application simultaneously, or in quick succession before the previous request completes? This situation was problematic in the last search tool we tried to use--it basically discarded all other requests until the first one completed. Do I need to implement some sort of queue?
• What are the security implications of this approach? I realize this is not a Zoom Search question specifically, but any insights are appreciated.

Thank you!

Greg Clapp,
Tenrec, Inc.

[Ray]

Hi Greg,

1) Zoom does not support handling simultaneous requests to write to the same files. So yes, you will need to implement something to prevent that from happening. If your script simply creates a file upon launching the task, and removing it when it's done -- and it refuses to launch the indexer when this file is present -- could do the job. Or detect the index .tmp files being written to (though this leaves room for error as it can take some time to initialize or complete indexing).

2) I don't think there are any major concerns if you can control who has access to the scripted task that launches the indexer. And if you restrict access to the .zcfg file, etc.

[jookyg]

Thanks for the response, Ray!

We ended up building an indexing queue using the Hangfire utility in .NET. This way there is only ever one index request processing. It works great. We have bombarded it with requests in testing (up to 70 within a 30 second period), and every request was processed and every update successfully added to the index. My company is loving Zoom Search, and the client is happy.