Skip to main content

Secure credentials for connector service

Connector Service integrates with Secure Store to keep database passwords, API keys, and other secrets out of Connector configuration. Credentials stored in Secure Store are resolved at connection time and never appear in the Connector spec, in API responses, or in worker pod environments.

How it works​

When a Connector uses Secure Store credentials:

  1. A Workspace Administrator uploads the secret value to Secure Store once, under a name like postgres-password.
  2. The Connector's configuration references the secret name — not the secret value — using a credentials block.
  3. When a user opens a connection, the Connector Service workload manager reads the secret value from Secure Store and holds it in memory for that connection.
  4. The worker requests the credential from the workload manager over an internal, authenticated call when it opens a data stream. The value is never written to an API response or to a pod environment variable.

This means end users can open connections against a secured Connector without ever seeing or handling the underlying credentials.

Supported credential types​

ProviderCredential keyDescription
PostgreSQLPGPASSWORDDatabase password
SnowflakeSNOWFLAKE_PASSWORDSnowflake user password
SnowflakeSNOWFLAKE_PRIVATE_KEYPKCS#8 private key for key-pair authentication
Amazon S3 / MinIOAWS_ACCESS_KEY_IDAccess key ID
Amazon S3 / MinIOAWS_SECRET_ACCESS_KEYSecret access key
Azure Blob StorageAZURE_STORAGE_KEYStorage account key

Admin setup: storing a secret​

A Workspace Administrator uploads the secret value to Secure Store in the workspace where the Connector will live. The following example uses the h2o-secure-store Python library:

import os
import h2o_secure_store as hss
from h2o_secure_store.clients.secret_version.secret_version import SecretVersion

clients = hss.login(
    environment=os.environ["H2O_CLOUD_ENVIRONMENT"],
    platform_token=os.environ["H2O_CLOUD_CLIENT_PLATFORM_TOKEN"],
)

workspace = f"workspaces/{os.environ['WORKSPACE_ID']}"

# Create the secret once. It holds metadata only, not the value.
clients.secret_client.create_secret(parent=workspace, secret_id="postgres-password")

# Store the value as a new version of the secret.
clients.secret_version_client.create_secret_version(
    parent=f"{workspace}/secrets/postgres-password",
    secret_version=SecretVersion(value=b"<actual-password>"),
)
Tip

To rotate the password, create a new version of the same secret. By default, the connection uses the latest version of the secret. To pin a specific version, see Pinning a secret version.

Admin setup: referencing the secret in a Connector​

When creating the Connector, reference the Secure Store secret name in the credentials field instead of embedding the password in data_source_config:

import os
from h2o_connector_service import Client

client = Client(
    h2o_cloud_url=os.environ["H2O_CLOUD_ENVIRONMENT"],
    refresh_token=os.environ["H2O_CLOUD_CLIENT_PLATFORM_TOKEN"],
)

# Connector config does NOT contain the password
config = {
    "PGHOST": "<your-database-host>",
    "PGPORT": "5432",
    "PGDATABASE": "<your-database-name>",
    "PGUSER": "<your-database-user>",
}

# Credentials reference the Secure Store secret name, not the value
credentials = {
    "source": "CREDENTIAL_SOURCE_SECURE_STORE",
    "config": {
        "PGPASSWORD": "postgres-password",  # Key = credential name; value = Secure Store secret name
    },
}

connector = client.connectors.create(
    os.environ["WORKSPACE_ID"],
    metadata={"name": "<connector-id>"},
    data_source_type="postgresql",
    data_source_config=config,
    credentials=credentials,
)
print(f"Created connector: {connector.metadata.name}")
Note

Use the bare secret name, for example postgres-password. Do not use the full resource name workspaces/<workspace>/secrets/postgres-password. The platform reads the secret from the same workspace as the connection.

Blob storage example​

For S3 or MinIO connectors that use two credential values:

credentials = {
    "source": "CREDENTIAL_SOURCE_SECURE_STORE",
    "config": {
        "AWS_ACCESS_KEY_ID": "aws-access-key-id",        # Secure Store secret name
        "AWS_SECRET_ACCESS_KEY": "aws-secret-access-key", # Secure Store secret name
    },
}

Pinning a secret version​

By default, the platform reads the latest version of each secret. To use a specific version, add secret_versions. The key is the credential name. The value is the version ID: the last segment of the version name that create_secret_version returns (workspaces/<workspace>/secrets/<secret>/versions/<version-id>). Version IDs are random strings, not numbers.

credentials = {
    "source": "CREDENTIAL_SOURCE_SECURE_STORE",
    "config": {
        "PGPASSWORD": "postgres-password",
    },
    "secret_versions": {
        "PGPASSWORD": "a4nxgehq2s06",
    },
}

End-user experience​

Data Scientists and Data Engineers open connections against a Secure Store-backed Connector in exactly the same way as any other connector. No additional configuration is required:

with client.open_session(
    workspace_id=os.environ["WORKSPACE_ID"],
    connector="connectors/<connector-id>",
    worker="workers/<worker-id>",
    extraction={"query": "SELECT id, name FROM my_table"},
) as session:
    session.stream_to_csv("output.csv")

The credential resolution is transparent to the end user.

Admin and user responsibilities​

TaskRole
Upload secret value to Secure StoreWorkspace Administrator
Create Connector with credentials referencing the secretWorkspace Administrator
Open connections and stream dataData Scientist / Data Engineer

To open a session, end users need the connections/CREATE, connections/GET, and connections/DELETE permissions in the workspace. They do not need access to Secure Store or to the Connector's credential configuration.

Troubleshooting​

SymptomCauseResolution
Connection fails with a credential resolution errorThe Secure Store secret name in the Connector's credentials.config does not match an existing secret in the workspaceAsk your Workspace Administrator to verify the secret name and workspace ID
Connection fails with a credential resolution error after the secret name was checkedThe value is a full resource name such as workspaces/<workspace>/secrets/<name>Use the bare secret name
Connection fails with an authorization errorThe platform does not have permission to read from Secure StoreAsk your Platform Administrator to verify the authorization policy for the Connector Service workload manager
Connector creation is accepted but connections always failThe secret was created in a different workspace than the ConnectorEnsure the secret and Connector are in the same workspace
Feedback