Parallel generation of index

[qintian-qe]

Due to the large amount of business code, each index update requires a large amount of time. Is there any way to shorten the time of index generation, such as generating index files in parallel, but opengrok does not seem to support parallel generation of indexes and generating related configuration files

[vladak]

The indexer is already heavily parallelized. I don't understand your claim here:

opengrok does not seem to support parallel generation of indexes and generating related configuration files
so you will have to expand on that.

[qintian-qe]

Does the term "highly parallel indexing" refer to indexing multiple projects simultaneously, or does it refer to generating indexes in parallel on a single project. Currently, I manage more than 10TB of code, and I update the code every day and incrementally generate index files. I need to find ways to shorten the time of index generation

[qintian-qe]

Can I generate index files for different projects on multiple servers in parallel? "Then merge them onto a single server. I previously tried to merge the generated index file and configuration, but failed."

[vladak]

Both. When the indexer is run to index multiple projects, these are indexed in parallel. Then the processing of the files from these projects uses shared set of thread pools. If you run the indexer and observe the system resource utilization, then it mostly fully utilizes the CPU and the I/O load is also high. This has been valid for the recent versions.

For incremental reindex, it depends on the type of repositories. For Git, I made that really efficient because the indexer extract the set of files from the changesets added to the repository since the last time so it does not have to traverse the directory tree for that repository.

You might want to take a look at https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases

Also, as with any performance tuning, you might as well collect the data about the system utilization while indexer is running. If you want to dig deeper, https://github.com/oracle/opengrok/wiki/Monitoring#indexer will help you.

[vladak]

Can I generate index files for different projects on multiple servers in parallel? "Then merge them onto a single server. I previously tried to merge the generated index file and configuration, but failed."

That should be possible however will require some tricky manipulations with both input and output data.

[qintian-qe]

Okay, thank you very much. I will try further in the direction you provided

Both. When the indexer is run to index multiple projects, these are indexed in parallel. Then the processing of the files from these projects uses shared set of thread pools. If you run the indexer and observe the system resource utilization, then it mostly fully utilizes the CPU and the I/O load is also high. This has been valid for the recent versions.

For incremental reindex, it depends on the type of repositories. For Git, I made that really efficient because the indexer extract the set of files from the changesets added to the repository since the last time so it does not have to traverse the directory tree for that repository.

You might want to take a look at https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases

Also, as with any performance tuning, you might as well collect the data about the system utilization while indexer is running. If you want to dig deeper, https://github.com/oracle/opengrok/wiki/Monitoring#indexer will help you.