Configure Remote Search

This section shows how to configure Remote Search for your HCL Digital Experience 9.5 environments. This guidance is for containerized deployments running on Kubernetes. If you are looking for guidance on configuration of Remote Search for non-containerized deployments, please see Remote search service

Introduction

For using the HCL Digital Experience Search with multiple Pods of Core running, it is required to have Remote Search configured. Remote Search itself is provided as a singleton Pod that will provide all required functionality.

Both Core and Remote Search need to be configured in a certain way to allow a functional Remote Search setup.
From release CF211 onwards, you can either use the automated configuration controlled via the Helm Chart deployment values or use the classic setup method using manual steps.

The configuration is a one-off process. Once you have Remote Search configured, the settings that were changed are persisted in the corresponding Pods Persistent Volumes and will persist after future updates to newer CF releases of HCL Digital Experience.

Warning

If you have configured search collections already and want them and their index backed up, see the topic Exporting and importing search collections for further information.

The process of configuring Remote Search will include deleting the default Search Service and its corresponding Search collections.

Warning

If you change your administrative user password, you have to adjust the security configuration of your Search Collection Sources, otherwise they cannot be crawled. Refer to Changing the HCL Digital Experience administrator password for more information.

Warning

If you see that the Remote Search Service in the DX Search Administration is unavailable after a DX Remote Search Pod restart, it may be required to restart DX Core as well. You can use the following command to perform a ripple restart of all DX Core Pods:

# Perform the restart of DX Core
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl rollout restart statefulset dx-core -n dxns

Tip

This guide uses kubectl for all commands that are related to Kubernetes. If you are Running on OpenShift, you can replace the kubectl command with oc.

Automated configuration

Note

This feature is only available for HCL Digital Experience 9.5 CF211 and newer versions. If you are running on a previous version and want to configure Remote Search, please see the Manual configuration steps.

Pre-requisites and limitations

Before you can leverage the automated configuration, you need to be aware on certain requirements and limitations that the automated configuration has:

This configuration will only work in containerized deployments running on Kubernetes.
The configuration of DX Core will not continue automatically if you have configured JCR Text Search or the suppression of automated default search collection creation.
Both DX Core and DX Remote Search have their credentials configured correctly inside your custom-values.yaml used for the Helm deployment.
There is currently no ConfigEngine task or similar being performed on either DX Core or DX Remote Search Pods

Configuration properties

To leverage the automated configuration, you will need to adjust the custom-values.yaml that you are using for your Helm deployment. (See Custom value files).

Inside your custom-values.yaml you can configure the following section to leverage Remote Search automated configuration:

# Configuration section of hcl-dx-deployment
configuration:
  # Remote Search specific configuration parameters
  remoteSearch:
    # If set to true, Remote Search will be configured automatically
    autoConfigure: true
    # Define Maximum time (in seconds) that DX Core will wait for the Remote Search Pod to run its configuration tasks
    coreWaitTime: 5400
    # Define Maximum time (in seconds) that DX Remote Search will wait for the Core Pod to run its configuration tasks
    remoteSearchWaitTime: 5400
    # Define the maximum amount of retries that DX Remote Search will attempt to run its configuration task
    remoteSearchConfigTries: 10

The following parameters are available for adjustments:

Parameter	Possible values	Explanation
`autoConfigure`	`true` or `false`(default)	This will enable or disable the automated configuration. After this value is set and the new `custom-values.yaml` is applied, the Pod restart of Core and Remote Search will handle the automated configuration.
`coreWaitTime`	Number, seconds (`5400` is default)	Since Core and Remote Search are depending on each other for this configuration process, Core will have to wait for Remote Search to finish its configuration. Based on the performance of the environment that HCL Digital Experience is deployed on, this configuration process time will vary.
`remoteSearchWaitTime`	Number, seconds (`5400` is default)	Since Core and Remote Search are depending on each other for this configuration process, Remote Search will have to wait for Core to finish its configuration and start its JVM in the background (See Configuration flow).
`remoteSearchConfigTries`	Number (`10` is default)	Remote Search will attempt a retry of its configuration tasks if the Core Pod is not reachable during the configuration process. This parameter limits the amount of retries until Remote Search will continue with a regular startup without a successful configuration.

It is recommended to have generous wait time values, as not waiting long enough by either of the applications may cause the configuration to fail.

The given values should suffice for general deployments, but if you are applying those changes directly during an update from one CF to a newer one, it is good to increase those values based on already experienced startup times in your deployment.

If the configuration process fails due to too low wait time values check the Troubleshooting section that contains steps to re-trigger the configuration process.

Apply configuration

After you have adjusted your custom-values.yaml with the settings mentioned in the previous sections, you can use helm upgrade with the reference to your custom-values.yaml if you have an existing deployment you want to configure. If you are deploying fresh, you can use helm install for the initial setup of DX.

This will apply the new configuration and the (re-)start of both the Remote Search and Core Pods will perform the steps necessary to their respective configuration.

Configuration flow

This section explains the flow that occurs when the automated configuration is performed. This section does not contain any steps that are required to be executed by you and just serves as information material.

Phase 1: Startup

After the automated configuration has been enabled via the Helm values, both DX Core and DX Remote Search Statefulsets will be updated / created.

In case of an existing deployment that is re-configured, both the DX Core and DX Remote Search Pods will get restarted.
After the Pod start, both will run their usual startup routine and then enter a specific Remote Search configuration logic. All log messages relevant to those actions will be prefixed with [RS Autoconfig].

Phase 2: Core Pod Stand-By, Remote Search waiting for Core Pod

To perform the automated configuration of Remote Search, the DX Core Pod will go into a stand-by mode.
In this mode, it will start up the DX Core JVM, but will not set the Pod to ready. This ensures that no active traffic is routed to this Core Pod during the configuration steps (incl. JVM restarts). The DX Core Pod will remain in that mode until the DX Remote Search Pod becomes ready or the wait time expires (defined by coreWaitTime in the Configuration properties section).

The DX Remote Search Pod will wait for a DX Core Pod with a running background JVM and detect a suitable candidate.

Once the DX Remote Search Pod found a viable DX Core Pod, it will continue with the configuration tasks that are required. This includes:

Remotely retrieving the LTPA Token from the DX Core Pod for SSO
Remotely retrieving the DX Core Pods SSL Certificate for trusted communication
Setting up the serverindex file
Importing the DX Core LTPA Token into the DX Remote Search WAS

If during that configuration step any issue occurs, this is likely due to DX Core being restarted or the designated DX Core Pod not being available anymore.
In that case, DX Remote Search will retry the configuration, starting with the search for a viable DX Core Pod again.

The retry count is limited by the remoteSearchConfigTries property described in the Configuration properties section.

After finishing all configuration tasks, DX Remote Search will re-enter its regular startup routine and become ready.

Phase 3: Remote Search finished, Core Pod configuration

Once the DX Remote Search Pod becomes ready, DX Core will notice that and continue with its Remote Search configuration steps. If DX Remote Search does not get ready in time (defined by coreWaitTime in the Configuration properties section), the regular startup will continue.

Before running the configuration, the configuration logic will check if:

The JCR Text Search configuration
The suppression of automated default search collection creation

are enabled. This could be a sign of Remote Search being already configured.

If those settings are set, the configuration will not continue. To override this manually, see the Troubleshooting section.

The configuration steps in DX Core include:

Remotely retrieving the DX Remote Search SSL Certificate for trusted communication
Enabling suppression of automated default search collection creation
Configuration for JCR Text Search
Deletion of the Default Search Service
Creation of a Remote Search Service
Creation of Search Collections (Portal and JCRCollection1) incl. their providers

After these configuration steps are performed, the DX Core Pod will continue with its regular startup routine.

Phase 4: Configuration completed

After all configuration steps have been performed, the new Search Service with the new Search Collections are configured. You can see them in the Search Administration.

Next Pod restarts will automatically recognize that the automated configuration has already been performed and will not run any additional steps.

Troubleshooting

All steps performed via the automated configuration are safe to be run multiple times. This means re-running the configuration after a failed attempt is generally considered safe.

Logs

If you want to see what went wrong during the configuration, look in the logs of both the DX Core Pod and the DX Remote Search Pod for messages prefixed with [RS Autoconfig]. The startup routines perform verbose logging that clearly states which steps have been performed. There is no additional log setting required to enable the log messages from the automated configuration.

Adjusting timeouts

It may be that in your environment, the startup of DX Core or DX Remote Search take longer than 90 minutes. If that is the case, adjust the timeout wait times in the Configuration properties section.

After adjusting these properties, you can re-run the automated configuration.

Overriding pre-flight checks

If case you have at some point configured either the suppression of automated default search collection creation or the JCR Text Search, but not configured Remote Search yet, you can override the pre-flight checks that would cause the configuration to not continue in that case.

To do so, connect into the DX Core Pod and create a file. See the following example:

# Connect into your DX Core Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-core-0 -c core -- /bin/bash
# Create the preflight check override file
# This file is persisted in the DX Core Pods Persistent Volume
touch /opt/HCL/wp_profile/rsAutoConfigOverridePreflight

If your automated configuration has failed due to those checks, you can re-run it using the commands from the next section.

Re-run automated configuration steps

In case your automated configuration failed, you can easily re-run it using the following procedure:

Remove the configuration semaphore file in the DX Core Pod:

# Connect into your DX Core Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-core-0 -c core -- /bin/bash
# Remove the config status semaphore file
rm -f /opt/HCL/wp_profile/rsAutoConfigStatus

Remove the configuration semaphore file in the DX Remote Search Pod:

# Connect into your DX Remote Search Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-remote-search-0 -c remote-search -- /bin/bash
# Remove the config status semaphore file
rm -f /opt/HCL/AppServer/profiles/prs_profile/rsAutoConfigStatus

Perform a restart rollout of DX Core and DX Remote Search:

# Perform the restart of DX Remote Search
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl rollout restart statefulset dx-remote-search -n dxns
# Perform the restart of DX Core
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl rollout restart statefulset dx-core -n dxns

Manual configuration

For manual configuration, you can rely on existing ConfigEngine tasks that will assist you in configuration of both the DX Core Pod and the DX Remote Search Pod.

Pre-requisites and limitations

This configuration is intended for containerized deployments running on Kubernetes.
Both DX Core and DX Remote Search have their credentials configured correctly inside your custom-values.yaml used for the Helm deployment.
There is currently no ConfigEngine task or similar being performed on either DX Core or DX Remote Search Pods
Both DX Core and DX Remote Search are deployed in your environment and the Pods are up and ready.

Configure the Core Pod

Execute the ConfigEngine task:

# Connect into your DX Core Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-core-0 -c core -- /bin/bash
# Inside the container execute the ConfigEngine task
# make sure that your deployment prefix (dx in this sample, e.g. dx-remote-search) is adjusted to your environment
# Ensure to use the right credentials for DWasPassword
/opt/HCL/wp_profile/ConfigEngine/./ConfigEngine.sh configure-portal-for-remote-search -DWasPassword=[WAS ADMIN PASSWORD] -Dremote.search.host.name="dx-remote-search" -Dremote.search.host.port="9043" -Dremote.search.cert.alias="remotesearchalias" -Dremote.search.iiop.url="iiop://dx-remote-search:2809" -Dremote.search.index.directory="/opt/HCL/AppServer/profiles/prs_profile/SearchCollections"

After this ConfigEngine task has been executed, a file containing the LTPA Key export will be created inside the Container.
Copy that file out of the Container into your local host:

# Copy out the LTPA key export
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl cp dx-core-0:/home/dx_user/LTPAKeyExported ./LTPAKeyExported -c core -n dxns

Configure the Remote Search Pod

Note

Before configuring the remote search pod, you must set up single sign-on (SSO) between the WebSphere Application Server running in the core pod and the WebSphere Application Server running in the remote-search pod. See the topics Single sign-on for authentication using LTPA cookies and Creating a single-sign on domain between HCL Portal and the remote search service for pre-requisites and instructions for setting up SSO.

Copy the LTPA Key exported from DX Core into the Remote Search Pod:

# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl cp ./LTPAKeyExported dx-remote-search-0:/home/dx_user/ -c remote-search -n dxns

Execute the ConfigEngine task:

# Connect into your DX Remote Search Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-remote-search-0 -c remote-search -- /bin/bash
# Inside the container execute the ConfigEngine task
# make sure that your deployment prefix (dx in this sample, e.g., dx-core) is adjusted to your environment
# Ensure to use the right credentials for DWasPassword
/opt/HCL/AppServer/profiles/prs_profile/ConfigEngine/./ConfigEngine.sh configure-remote-search-server-for-remote-search -DWasPassword=[WAS ADMIN PASSWORD] -Dportal.host.name="dx-core" -Dportal.port.number="10042" -Dportal.cert.alias="portaldockeralias" -Dremote.search.host.name="localhost"

Restart both Core and Remote Search

To restart Remote Search:

# Connect into your DX Remote Search Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-remote-search-0 -c remote-search -- /bin/bash
# Create a semaphore file which will ensure, the Pod is not recycled during your restart
touch /home/dx_user/configInProgress
# Perform restart, fill in the proper credentials
/opt/HCL/AppServer/profiles/prs_profile/bin/./stopServer.sh server1 -user [WAS ADMINISTRATOR NAME] -password [WAS ADMINISTRATOR PASSWORD]
/opt/HCL/AppServer/profiles/prs_profile/bin/./startServer.sh server1
# Remove semaphore file
rm -f /home/dx_user/configInProgress

To restart DX Core:

# Connect into your DX Core Pod
# adjust the deployment prefix "dx-" and the namespace "dxns" to match your environment
kubectl exec -it -n dxns dx-core-0 -c core -- /bin/bash
# Create a semaphore file which will ensure, the Pod is not recycled during your restart
touch /opt/app/configInProgress
# Perform restart, fill in the proper credentials
/opt/HCL/wp_profile/bin/./stopServer.sh WebSphere_Portal -user [WAS ADMINISTRATOR NAME] -password [WAS ADMINISTRATOR PASSWORD]
/opt/HCL/wp_profile/bin/./startServer.sh WebSphere_Portal
# Remove semaphore file
rm -f /opt/app/configInProgress

Search Services and Collections

New Search Service

Create a new search service and use the following values for a Remote Search services configuration to a Kubernetes container deployment. See the section on Creating a new search service for more information.

For Search Services configuration, the following values are used:

Item	Value
IIOP_URL	`iiop://dx-remote-search:2809`
PSE TYPE	Select ejb from the pull down.
EJB	`ejb/com/ibm/hrl/portlets/WsPse/WebScannerLiteEJBHome`
DefaultCollectionsDirectory	Leave empty.
Search service implementation	Select Portal Search Service Type from the pull down.
CONFIG_FOLDER_PATH	Did not set (differs from non-container instructions).

Tip

Take care when defining the IIOP_URL parameter, as the target URL is the Service of Remote Search inside your Kubernetes deployment. The prefix dx-deployment may be different in your deployment scenario and is based on the deployment name chosen during the helm install command.

Note

Once completed and saved, the HCL Digital Experience 9.5 container deployment has a new search service called Remote PSE service EJB, with a green check mark confirming that the service was correctly set up and can communicate with the Remote Search container.

New Search Collections

Based on the previously created Remote Search service, create a Portal Search Collection and a JCR Search Collection using the following parameters.

Portal Search Collection

Use the following parameters to create a Portal search collection.

Parameter	Value
Search collection name	`Portal Search Collection`
Search collection location	`/opt/HCL/AppServer/profiles/prs_profile/SearchCollections/PortalSearchCollection`

Note

The Search collection location is relative to the remote search container. Furthermore, one places the collection in the profile of the Remote Search server because the profile of the remote search server is persisted. One obviously wants the search indexes persisted across restarts.

Create the two following Content Sources:

Portal Content Source

Parameter	Value
Content Source Type	Portal Site
Content Source Name	Portal Content Source
Collect documents linked from this URL	`https://dx-core-service:10042/wps/seedlist/myserver?Source=com.ibm.lotus.search.plugins.seedlist.retriever.portal.PortalRetrieverFactory&Action=GetDocuments&Range=100`

In the Security panel, use the DX Core Service name (e.g., dx-core) as the host name, along with the username wpsadmin and the associated password for wpsadmin. You can also specify Realm as CrawlerUsersRealm.

Tip

Note: The host dx-core and port 10042 are the Kubernetes service host and the port for DX Core. In this case, 10042 is the HttpQueueInboundDefaultSecure port on the HCL DX 9.5 server. Adjust this according to your deployment configuration.

WCM Content Source

Parameter	Value
Content Source Type	Seedlist provider
Content Source Name	Portal Content Source
Collect documents linked from this URL	`https://dx-core-service:10042//wps/seedlist/myserver?SeedlistId=&Source=com.ibm.workplace.wcm.plugins.seedlist.retriever.WCMRetrieverFactory&Action=GetDocuments`

In the Security panel, use the DX Core Service name (e.g., dx-core) as the host name, along with the username wpsadmin and the associated password for wpsadmin. You can also specify Realm as CrawlerUsersRealm.

Tip

Note: The host dx-core and port 10042 are the Kubernetes service host and the port for DX Core. In this case, 10042 is the HttpQueueInboundDefaultSecure port on the HCL DX 9.5 server. Adjust this according to your deployment configuration.

JCR Search Collection

Use the following parameters to create a Content Source JCR search collection.

Parameter	Value
Search collection name	`JCRCollection1`
Search collection location	`/opt/HCL/AppServer/profiles/prs_profile/SearchCollections/JCRCollection1`

Note

The Search collection location is relative to the remote search container. Furthermore, one places the collection in the profile of the Remote Search server because the profile of the remote search server is persisted. One obviously wants the search indexes persisted across restarts.

Create the following Content Source:

JCR Content Source

Parameter	Value
Content Source Type	Seedlist provider
Content Source Name	Portal Content Source
Collect documents linked from this URL	`https://dx-core:10042/wps/seedlist/myserver?Action=GetDocuments&Format=ATOM&Locale=en_US&Range=100&Source=com.ibm.lotus.search.plugins.seedlist.retriever.jcr.JCRRetrieverFactory&Start=0&SeedlistId=1@OOTB_CRAWLER1`

In the Security panel, use the DX Core Service name (e.g., dx-core) as the host name, along with the username wpsadmin and the associated password for wpsadmin. You can also specify Realm as CrawlerUsersRealm.

Notes

Click the create button to create the security realm definition inside the frame where you first entered the data. This action ensures that you have the security realm defintion before saving the content source.
The host dx-core and port 10042 are the Kubernetes service host and the port for DX Core. In this case, 10042 is the HttpQueueInboundDefaultSecure port on the HCL DX 9.5 server. Adjust this according to your deployment configuration.
The parsing of the SeedlistId positional parameter in this URL uses an index of the virtual portal being crawled. In this case, 1 (in 2 places) represents the base virtual portal.