"indexed" field and indexing multiple projects in parallel

[aerofeev2k]

Hi,

It used to be possible to reindex multiple projects in parallel by running a targeted opengrok indexer like:

... -jar opengrok.jar -R /etc/opengrok/configuration.xml -W /etc/opengrok/configuration.xml ... /PROJECT PROJECT

I think this was even suggested in some doc, I can't find this in Wiki anymore. Is this targeted incremental reindexing mode still supported?

The reason I'm asking is because I keep seeing this new per-repository "Indexed" field sporadically turned off in /etc/opengrok/configuration.xml for projects that have already been indexed. This confuses webapp causing the projects to disappear from the web page.

Thanks!

[vladak]

Using -W to write configuration for a per project reindex is undesirable. In fact, recently I added a warning to the code

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/Indexer.java

Lines 335 to 339 in 7626bae

	if (!subFiles.isEmpty() && configFilename != null) {
	LOGGER.log(Level.WARNING, "The collection of entries to process is non empty ({0}), seems like " +
	"the intention is to perform per project reindex, however the -W option is used. " +
	"This will likely not work.", subFiles);
	}

This is because that might store (depends what are the other indexer options) configuration specific for the indexed project, wiping out configuration of the other projects. Also, if you run multiple per project indexers in parallel, it might corrupt the configuration.

The per project workflow is described on https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow

[aerofeev2k]

So the workflow depends on the webapp to accept/apply incremental project status changes in a thread-safe manner?

If the webapp is down, there seems to be no way to index in parallel and handle /etc/opengrok/configuration.xml updates w/o overwriting other project's "indexed" field? Unless I tell OpenGrok to save to a unique temporary file and then implement my own merging that would grab only a specific project's section from the temporary file, lock /etc/opengrok/configuration.xml and apply the section to it?

[vladak]

Basically yes. This is not so about thread safe rather than about having global configuration in one place. There needs to be a way to unify the information about all the projects/repositories so that when the webapp is redeployed or the application server restarted, the web app has the information about all projects/repositories. The webapp being used as place to merge the pieces of information is convenient and in line with overwhelming majority of use cases.

Do you have a use case for running per project reindex offline ?

[aerofeev2k]

I don't have a use case, but things happen...

[vladak]

Well, one could try to merge the XML config files however this is very error prone (#2329).

[aerofeev2k]

There may still be a problem.

Is -U pushing the entire global config into webapp at the end of the project indexing or just the part that's related to the project? If it is pushing everything, then even if I get current config from webapp before indexing the project, and even if I just rely on -U and never do -W, there can still be issues with other project config overwrites, no?

[vladak]

In per project reindex mode, -U is used just to set the indexed property to true of the project being indexed and update the metadata about its repositories (if any) so they can be shown on the index page.

[vladak]

And to trigger the rebuild of suggester data.