2025-09-10

Resolving Certificate Read Permission Issues in OpenSearch

In this article, we will discuss certificate read permission issues and how to resolve them.

Reading time: 5 minutes

By Daniel Gleim

Technical Content Creator

Resolving Certificate Read Permission Issues in OpenSearch

OpenSearch clusters rely heavily on TLS certificates for secure communication between nodes and with clients. When these certificates become inaccessible due to permission issues, your entire cluster can grind to a halt. If you’ve ever encountered cryptic SSL errors or found your OpenSearch nodes failing to start with certificate-related messages, you know how frustrating this can be.

Picture this scenario: you’re running an OpenSearch cluster and suddenly encounter the following error when trying to start your nodes:

copy

java.nio.file.NoSuchFileException: /usr/share/opensearch/config/admin.pem

Or perhaps you see one of these instead:

copy

Unable to read the file ca.crt. Please make sure this file exists and is readable regarding to permissions

copy

Unable to read the file root-ca.pem. Please make sure this file exists and is readable regarding to permissions

All of these messages point to the same underlying issue — OpenSearch cannot locate or read the certificate files it needs for secure communication. In this particular case, the admin.pem certificate file appears to be missing or inaccessible, preventing the OpenSearch service from starting properly.

Root Cause Analysis

While the error may look like the file is missing, in most cases the problem is related to file permission or ownership issues. Here’s what’s happening under the hood:

When OpenSearch starts, it runs under a specific user account (usually opensearch or elasticsearch). The service needs to read various certificate files from the configuration directory to establish secure connections. If these certificate files have restrictive permissions, are owned by a different user, or have been mounted in a way that strips permissions (common in containerized deployments), the OpenSearch process cannot access them.

The most common causes include:

Certificate files owned by root instead of the OpenSearch service user
Restrictive file permissions that don’t allow the service user to read the certificates
Incorrect directory permissions on the config folder itself
Certificate files copied or moved without preserving proper ownership
In containerized or Kubernetes deployments, certificates mounted from Secrets with incorrect permissions
Files in the wrong format (e.g., private keys not converted to PKCS#8), which can trigger read errors even if permissions are correct
SELinux security contexts preventing access even when standard permissions appear correct

Solution

The fix involves ensuring that your certificate files have the correct ownership, permissions, and format. Here’s how to resolve this step by step:

Step 1: Identify the OpenSearch Service User

First, determine which user account runs your OpenSearch service:

copy

ps aux | grep opensearch

This will typically show opensearch as the user running the process. Also check what group the user belongs to, as this may vary by distribution:

copy

id opensearch

Step 2: Check Current Certificate Permissions

Navigate to your OpenSearch config directory and examine the current permissions:

copy

cd /usr/share/opensearch/config
ls -la *.pem

You’ll likely see output showing that the certificate files are owned by root or have permissions that don’t allow the OpenSearch user to read them.

Step 3: Fix Ownership and Permissions

Change the ownership of all certificate files to the OpenSearch user. Note that the group name may vary by distribution:

copy

# Most common
sudo chown opensearch:opensearch *.pem

# On some distributions, you might need
sudo chown opensearch:elasticsearch *.pem

# Or check the actual group first
sudo chown opensearch:$(id -gn opensearch) *.pem

Set appropriate permissions for private keys and public certificates. Since private keys don’t always follow the same naming pattern, it’s best to use a robust approach:

Option 1 – Secure by default, loosen for public certs

copy

# First lock down all PEM files
sudo chmod 600 *.pem

# Then loosen permissions for public certs
sudo chmod 644 admin.pem node.pem

Option 2 – Explicitly list files

copy

sudo chmod 600 admin_private.pem node_private.pem
sudo chmod 644 admin.pem node.pem

Option 3 – Use find to set permissions recursively

copy

sudo find /usr/share/opensearch/config -type f -name "*.pem" -exec chmod 600 {} \;
# Loosen public certs afterward as needed

Shortcut for specific naming conventions If you consistently name private keys with -key.pem, you can run:

copy

sudo chmod 644 *.pem
sudo chmod 600 *-key.pem

This works well if your keys follow the *-key.pem naming convention, but will miss keys with other names.

Step 4: Verify Directory Permissions

Make sure the config directory itself is accessible:

copy

sudo chown -R opensearch:opensearch /usr/share/opensearch/config
sudo chmod 755 /usr/share/opensearch/config

Step 5: Handle SELinux Contexts (If Applicable)

On RHEL, CentOS, or other SELinux-enabled systems, check and fix SELinux contexts:

copy

# Check current SELinux context
ls -Z /usr/share/opensearch/config/*.pem

# Restore proper SELinux contexts
sudo restorecon -R /usr/share/opensearch/config/

If you’re still having issues, you may need to set a specific context:

copy

sudo semanage fcontext -a -t bin_t "/usr/share/opensearch/config/.*\.pem"
sudo restorecon -R /usr/share/opensearch/config/

Step 6: Verify File Format (Optional but Recommended)

If permissions look correct but OpenSearch still cannot read the file, check the format of your certificates and keys. Private keys should be in PKCS#8 format for OpenSearch to read them without error:

copy

# Convert to PKCS#8 format (unencrypted)
openssl pkcs8 -topk8 -inform PEM -outform PEM -in old-key.pem -out key.pem -nocrypt

# If your existing key is encrypted, you'll need to provide the passphrase
openssl pkcs8 -topk8 -inform PEM -outform PEM -in old-key.pem -out key.pem -nocrypt -passin pass:your_passphrase

Step 7: Restart OpenSearch and Monitor Logs

With the correct permissions and formats in place, restart your OpenSearch service:

copy

sudo systemctl restart opensearch

Check the service status to confirm it starts successfully:

copy

sudo systemctl status opensearch

For real-time troubleshooting, monitor the logs as the service starts:

copy

journalctl -u opensearch -f

This will help you catch any remaining certificate issues as they occur.

Configuration Best Practices

To prevent similar issues in the future, consider these configuration recommendations:

Configuration Aspect	Recommendation	Explanation
Certificate ownership	`opensearch:opensearch`	Ensures the service user can read certificate files
Public certificate permissions	`644`	Allows read access while maintaining security
Private key permissions	`600`	Restricts access to owner only for maximum security
Config directory permissions	`755`	Allows directory traversal while maintaining security
File format	PKCS#8 for private keys	Ensures compatibility with OpenSearch
SELinux context	Proper security contexts	Prevents access issues on SELinux-enabled systems

For containerized deployments, ensure that any mounted secrets or volumes preserve these ownership and permission settings. Some Kubernetes Secret mounts, for example, may default to root ownership and 0600 permissions, which can block the opensearch user.

Takeaways

Certificate permission issues are among the most common stumbling blocks when setting up or maintaining OpenSearch clusters. The key lessons here are:

Always check file ownership first – When troubleshooting certificate-related errors, your first step should be examining who owns the certificate files and whether the service user can access them.

Don’t forget the format – Even with correct permissions, improperly formatted keys or certificates can prevent OpenSearch from starting.

Consider SELinux and security contexts – On enterprise Linux distributions, standard permissions might not be enough. Always check SELinux contexts when troubleshooting persistent access issues.

Use live logging for debugging – The journalctl -f command is invaluable for watching certificate loading in real-time and catching issues as they happen.

Follow the principle of least privilege – Give certificate files the minimum permissions needed. Public certificates need read access (644), while private keys should be readable only by the owner (600).

Automate permission and format checks – Consider adding both permission-setting and format-validation steps to your deployment scripts to avoid manual oversights.

Test after certificate changes – Whenever you update, copy, or move certificate files, always verify that permissions and formats are correct before restarting services.

For more detailed information about OpenSearch security configuration, check out the official OpenSearch security documentation. Proper certificate management is crucial not just for getting your cluster running, but for maintaining the security and integrity of your entire search infrastructure.

Resolving Certificate Read Permission Issues in OpenSearch

In this article, we will discuss certificate read permission issues and how to resolve them.

Resolving Certificate Read Permission Issues in OpenSearch

Root Cause Analysis

Solution

Step 1: Identify the OpenSearch Service User

Step 2: Check Current Certificate Permissions

Step 3: Fix Ownership and Permissions

Step 4: Verify Directory Permissions

Step 5: Handle SELinux Contexts (If Applicable)

Step 6: Verify File Format (Optional but Recommended)

Step 7: Restart OpenSearch and Monitor Logs

Configuration Best Practices

Takeaways

You may also like