# Self-hosted Visibility feature setup

A [Visibility](/temporal-service/visibility) store is set up as a part of your
[Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions
that exist on your Temporal Service.

A Visibility store is required in a Temporal Service setup because it is used by [Temporal Web UI](/web-ui) and
[Temporal CLI](/cli) to pull [Workflow Execution](/workflow-execution) data and enables features like batch operations
on a group of Workflow Executions.

With the Visibility store, you can use [List Filters](/list-filter) with [Search Attributes](/search-attribute) to list
and filter Workflow Executions that you want to review or act upon.

Supported Visibility stores include:

- Elasticsearch v7 with Temporal Server v1.7 and later
- Elasticsearch v8 with Temporal Server v1.18 and later
- OpenSearch 2+ with Temporal Server v1.30.1 and later
- MySQL v8.0.17 and later with Temporal Server v1.20 and later
- PostgreSQL v12 and later with Temporal Server v1.20 and later
- SQLite v3.31.0 and later with Temporal Server v1.20 and later

## Current and legacy Visibility support 

[Advanced Visibility](/visibility#advanced-visibility) is the current generation of Temporal Visibility. It supports the
modern query model, including [custom Search Attributes](/search-attribute#custom-search-attribute).

This page also includes guidance for the legacy (deprecated in Temporal Server v1.21 and removed in v1.24)
[standard Visibility](#legacy-standard-visibility) model for older deployments and migration work. In this context,
"advanced" and "standard (legacy)" refer to the current and legacy generations of Temporal Visibility, respectively.

The following compatibility matrix summarizes which generation of Visibility each store supports and the Temporal Server
versions required:

| Store                                                                                                               | Advanced Visibility (current)                                                                                                                                    | Standard Visibility (legacy)                                          |
| :------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------- |
| [Elasticsearch](#elasticsearch) <br/><br/>Recommended for any setup that spawns more than a few Workflow Executions | v7 on Temporal Server v1.7+, v8 on Temporal Server v1.18+                                                                                                        | Not supported                                                         |
| [OpenSearch](#elasticsearch)                                                                                        | 2+ on Temporal Server v1.30.1+                                                                                                                                   | Not supported                                                         |
| [MySQL](#mysql)                                                                                                     | v8.0.17+ on Temporal Server v1.20+                                                                                                                               | v5.7+ on older deployments before Temporal Server v1.24               |
| [PostgreSQL](#postgresql)                                                                                           | v12+ on Temporal Server v1.20+                                                                                                                                   | v9.6+ on older deployments before Temporal Server v1.24               |
| [SQLite](#sqlite)                                                                                                   | v3.31.0+ on Temporal Server v1.20+                                                                                                                               | Not supported                                                         |
| [Cassandra](#cassandra)                                                                                             | Not supported.<br/><br/>To migrate from Cassandra to a supported advanced Visibility store, see [Migrating Visibility database](#migrating-visibility-database). | Deprecated in Temporal Server v1.21, removed in Temporal Server v1.24 |

You can use any combination of the supported databases for your Persistence and Visibility stores. For updates, check
[Server release notes](https://github.com/temporalio/temporal/releases).

Temporal Server v1.21 introduced support for a secondary Visibility store in your Temporal Service to enable
[Dual Visibility](/dual-visibility). This is useful for migrating your Visibility store database.

## How to set up MySQL Visibility store 

> **💡 Tip:**
> Support, stability, and dependency info
>
> - MySQL v5.7 and later.
> - Advanced Visibility is available on MySQL v8.0.17 and later with Temporal Server v1.20 and later.
> - MySQL v5.7 support applied to older standard Visibility deployments before Temporal Server v1.24.
>

You can set MySQL as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

If using MySQL v8.0.17 or later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

### Persistence configuration

Set your MySQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `mysql-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: mysql-visibility
  #...
  datastores:
    default:
      #...
    mysql-visibility:
      sql:
        pluginName: 'mysql8' # For MySQL v8.0.17 and later. For earlier versions, use "mysql" plugin.
        databaseName: 'temporal_visibility'
        connectAddr: ' ' # Remote address of this database; for example, 127.0.0.0:3306
        connectProtocol: ' ' # Protocol example: tcp
        user: 'username_for_auth'
        password: 'password_for_auth'
        maxConns: 2
        maxIdleConns: 2
        maxConnLifetime: '1h'
#...
```

For details on the configuration parameters and values, see
[Temporal Service configuration](/references/configuration#sql).

To enable advanced Visibility features on your MySQL Visibility store, upgrade to MySQL v8.0.17 or later with Temporal
Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) on how to upgrade your
Temporal Server and database schemas.

For example configuration templates, see
[MySQL Visibility store configuration](https://github.com/temporalio/temporal/blob/main/config/development-mysql8.yaml).

### Database schema and setup

Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for
[MySQL v8.0.17 and later](https://github.com/temporalio/temporal/tree/main/schema/mysql/v8/visibility).

The following example shows how to set up your MySQL as both your persistence and Visibility store using
`temporal-sql-tool`. Refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples
with different databases.

[compose/scripts/setup-mysql.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql.sh)
```sh
set -eu

# Validate required environment variables
: "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}"
: "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"

echo 'Starting MySQL schema setup...'
echo 'Waiting for MySQL port to be available...'
nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}
echo 'MySQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned

# Create and setup visibility database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility update-schema -d /etc/temporal/schema/mysql/v8/visibility/versioned

echo 'MySQL schema setup complete'
```

Note that the script uses
[temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go)
to run the setup.

## How to set up PostgreSQL Visibility store 

> **💡 Tip:**
> Support, stability, and dependency info
>
> - PostgreSQL v9.6 and later.
> - Advanced Visibility is available on PostgreSQL v12 and later with Temporal Server v1.20 and later.
> - PostgreSQL v9.6 through v11 support applied to older standard Visibility deployments before Temporal Server v1.24. We
>   recommend upgrading to PostgreSQL 12 or later.
>

You can set PostgreSQL as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

If using PostgreSQL v12 or later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

### Persistence configuration

Set your PostgreSQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `postgres-visibility` and define the datastore configuration
in your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: postgres-visibility
  #...
  datastores:
    default:
    #...
    postgres-visibility:
      sql:
        pluginName: 'postgres12' # For PostgreSQL v12 and later. For earlier versions, use "postgres" plugin.
        databaseName: 'temporal_visibility'
        connectAddr: ' ' # remote address of this database; for example, 127.0.0.0:5432
        connectProtocol: ' ' # protocol example: tcp
        user: 'username_for_auth'
        password: 'password_for_auth'
        maxConns: 2
        maxIdleConns: 2
        maxConnLifetime: '1h'
#...
```

To enable advanced Visibility features on your PostgreSQL Visibility store, upgrade to PostgreSQL v12 or later with
Temporal Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) for details on
how to upgrade your Temporal Server and database schemas.

### Database schema and setup

Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for
[PostgreSQL v12 and later](https://github.com/temporalio/temporal/tree/main/schema/postgresql/v12/visibility)

The following example shows how to set up your PostgreSQL as both persistence and Visibility store using
`temporal-sql-tool`. Refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples
with different databases.

[compose/scripts/setup-postgres.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-postgres.sh)
```sh
set -eu

# Validate required environment variables
: "${POSTGRES_SEEDS:?ERROR: POSTGRES_SEEDS environment variable is required}"
: "${POSTGRES_USER:?ERROR: POSTGRES_USER environment variable is required}"

echo 'Starting PostgreSQL schema setup...'
echo 'Waiting for PostgreSQL port to be available...'
nc -z -w 10 ${POSTGRES_SEEDS} ${DB_PORT:-5432}
echo 'PostgreSQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal create
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned

# Create and setup visibility database
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility create
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility setup-schema -v 0.0
temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versioned

echo 'PostgreSQL schema setup complete'
```

Note that the script uses
[temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go)
to run the setup.

## How to set up SQLite Visibility store 

> **💡 Tip:**
> Support, stability, and dependency info
>
> - SQLite v3.31.0 and later.
>

You can set SQLite as your [Visibility store](/temporal-service/visibility). Verify
[supported versions](/self-hosted-guide/visibility) before you proceed.

Temporal supports only an in-memory database with SQLite; this means that the database is automatically created when
Temporal Server starts and is destroyed when Temporal Server stops.

You can change the configuration to use a file-based database so that it is preserved when Temporal Server stops.
However, if you use a file-based SQLite database, upgrading your database schema to enable advanced Visibility features
is not supported; in this case, you must delete the database and create it again to upgrade.

If using SQLite v3.31.0 and later as your Visibility store with Temporal Server v1.20 and later, any
[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a
Namespace in that Temporal Service.

### Persistence configuration

Set your SQLite Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `sqlite-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
persistence:
  # ...
  visibilityStore: sqlite-visibility
  # ...
  datastores:
    # ...
    sqlite-visibility:
      sql:
        user: 'username_for_auth'
        password: 'password_for_auth'
        pluginName: 'sqlite'
        databaseName: 'default'
        connectAddr: 'localhost'
        connectProtocol: 'tcp'
        connectAttributes:
          mode: 'memory'
          cache: 'private'
        maxConns: 1
        maxIdleConns: 1
        maxConnLifetime: '1h'
        tls:
          enabled: false
          caFile: ''
          certFile: ''
          keyFile: ''
          enableHostVerification: false
          serverName: ''
```

SQLite (v3.31.0 and later) has advanced Visibility enabled by default.

### Database schema and setup

Visibility data is stored in a database table called `executions_visibility` that must be set up according to the
schemas defined (by supported versions) in
https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql.

For an example of setting up the SQLite schema, see
[Temporalite](https://github.com/temporalio/temporalite/blob/main/server.go) setup.

## Legacy standard Visibility configuration 

The following section applies to older self-hosted deployments that still use standard Visibility. For new deployments,
use one of the advanced Visibility backends described earlier on this page.

### How to set up Cassandra Visibility store 

> **💡 Tip:**
> Support, stability, and dependency info
>
> - Cassandra supported only standard Visibility. Standard Visibility was deprecated in Temporal Server v1.21 and removed
>   in v1.24. For updates, check the [Temporal Server release notes](https://github.com/temporalio/temporal/releases).
> - We recommend migrating from Cassandra to any of the other supported databases for Visibility.
>

Advanced Visibility is not supported with Cassandra.

To enable current Visibility features, use MySQL, PostgreSQL, SQLite, Elasticsearch, or OpenSearch as your Visibility
store. We recommend Elasticsearch or OpenSearch for any Temporal Service setup that handles more than a few Workflow
Executions because these backends support the Visibility request load and help optimize performance.

To migrate from Cassandra to a supported SQL database, see
[Migrating Visibility database](#migrating-visibility-database).

### Persistence configuration

Set your Cassandra Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then
define the Visibility store configuration under `datastores`.

The following example shows how to set a Visibility store `cass-visibility` and define the datastore configuration in
your Temporal Service configuration YAML.

```yaml
#...
persistence:
  #...
  visibilityStore: cass-visibility
  #...
  datastores:
    default:
    #...
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_visibility'
#...
```

### Database schema and setup

Visibility data is stored in a database table called `executions_visibility` that must be set up according to the
schemas defined (by supported versions) in https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility.

The following example shows how to set up your Cassandra Visibility store using `temporal-cassandra-tool`. For more
examples with different databases, refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts).

```bash
#...
# set your Cassandra environment variables
: "${KEYSPACE:=temporal}"
: "${VISIBILITY_KEYSPACE:=temporal_visibility}"

: "${CASSANDRA_SEEDS:=}"
: "${CASSANDRA_PORT:=9042}"
: "${CASSANDRA_USER:=}"
: "${CASSANDRA_PASSWORD:=}"
: "${CASSANDRA_TLS_ENABLED:=}"
: "${CASSANDRA_CERT:=}"
: "${CASSANDRA_CERT_KEY:=}"
: "${CASSANDRA_CA:=}"
: "${CASSANDRA_REPLICATION_FACTOR:=1}"
#...
# set connection details
#...
# set up Cassandra schema
setup_cassandra_schema() {
  #...
  # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}"
    fi
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
  #...
}
```

## How to integrate Elasticsearch or OpenSearch into a Temporal Service 

You can integrate Elasticsearch or OpenSearch with your Temporal Service as your Visibility store. We recommend using
one of these backends for large-scale operations on the Temporal Service.

To integrate Elasticsearch or OpenSearch with your Temporal Service, edit the `persistence` section of your
`development.yaml` configuration file to add the search backend as the `visibilityStore`, and run the index schema setup
commands.

Use the following version guidance:

- Elasticsearch v7 is supported with Temporal Server v1.7 and later.
- Elasticsearch v8 is supported with Temporal Server v1.18 and later.
- OpenSearch 2+ is supported with Temporal Server v1.30.1 and later.

The examples in this section use Elasticsearch. For OpenSearch, use the same datastore configuration shape and
operational flow unless a release note for your target Temporal Server version says otherwise.

### Persistence configuration

Set your Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the
search backend configuration under `datastores`.

The following example shows how to set a Visibility store named `es-visibility` and define the Elasticsearch datastore
configuration in your Temporal Service configuration YAML.

```yaml
persistence:
  ...
  visibilityStore: es-visibility
  datastores:
    ...
    es-visibility: # Define the Elasticsearch datastore connection information under the `es-visibility` key
      elasticsearch:
        version: "v7"
        url:
          scheme: "http"
          host: "127.0.0.1:9200"
        indices:
          visibility: temporal_visibility_v1_dev
```

### Index schema and index

To set up Elasticsearch as your Visibility store, use the `temporal-elasticsearch-tool` available in the
`temporalio/admin-tools` image.

The following example shows how to set up an Elasticsearch Visibility store with a MySQL persistence store using
`temporal-elasticsearch-tool`. For more examples with different databases, refer to the
[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts).

[compose/scripts/setup-mysql-es.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql-es.sh)
```sh
set -eu

# Validate required environment variables
: "${ES_SCHEME:?ERROR: ES_SCHEME environment variable is required}"
: "${ES_HOST:?ERROR: ES_HOST environment variable is required}"
: "${ES_PORT:?ERROR: ES_PORT environment variable is required}"
: "${ES_VISIBILITY_INDEX:?ERROR: ES_VISIBILITY_INDEX environment variable is required}"
: "${ES_VERSION:?ERROR: ES_VERSION environment variable is required}"

: "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}"
: "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"

echo 'Starting MySQL and Elasticsearch schema setup...'
echo 'Waiting for MySQL port to be available...'
nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}
echo 'MySQL port is available'

# Create and setup temporal database
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal create
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0
temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned

# Setup Elasticsearch index
# temporal-elasticsearch-tool is available in v1.30+ server releases
if [ -x /usr/local/bin/temporal-elasticsearch-tool ]; then
  echo 'Using temporal-elasticsearch-tool for Elasticsearch setup'
  temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" setup-schema
  temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" create-index --index $ES_VISIBILITY_INDEX
else
  echo 'Using curl for Elasticsearch setup'
  echo 'WARNING: curl will be removed from admin-tools in v1.30.'
  echo 'Waiting for Elasticsearch to be ready...'
  max_attempts=30
  attempt=0
  until curl -s -f "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s"; do
    attempt=$((attempt + 1))
    if [ $attempt -ge $max_attempts ]; then
      echo "ERROR: Elasticsearch did not become ready after $max_attempts attempts"
      echo "Last error from curl:"
      curl "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s" 2>&1 || true
      exit 1
    fi
    echo "Elasticsearch not ready yet, waiting... (attempt $attempt/$max_attempts)"
    sleep 2
  done
  echo ''
  echo 'Elasticsearch is ready'
  echo 'Creating index template...'
  curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/_template/temporal_visibility_v1_template" -H 'Content-Type: application/json' --data-binary "@/etc/temporal/schema/elasticsearch/visibility/index_template_$ES_VERSION.json"
  echo ''
  echo 'Creating index...'
  curl --head --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" 2>/dev/null || curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX"
  echo ''
fi

echo 'MySQL and Elasticsearch setup complete'
```

### Elasticsearch privileges

Ensure that the following privileges are granted for the Elasticsearch Temporal index:

- **Read**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `create`, `index`, `delete`, `read`
- **Write**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `write`
- **Custom Search Attributes**
  - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices):
    `manage`
  - [cluster privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster):
    `monitor` or `manage`.

## How to set up Dual Visibility 

To enable [Dual Visibility](/dual-visibility), set up a secondary Visibility store with your primary Visibility store,
and configure your Temporal Service to enable read and/or write operations on the secondary Visibility store.

With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to
write to primary only, secondary only, or to both primary and secondary stores.

#### Set up secondary Visibility store

Set the secondary store with the `secondaryVisibilityStore` configuration key in your Persistence configuration, and
then define the secondary Visibility store configuration under `datastores`.

You can configure any of the [supported databases](/self-hosted-guide/visibility) as your secondary store.

Examples:

To configure MySQL as a secondary store with Cassandra as your primary store, do the following.

```yaml
persistence:
  visibilityStore: cass-visibility # This is your primary Visibility store
  secondaryVisibilityStore: mysql-visibility # This is your secondary Visibility store
  datastores:
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_primary_visibility'
    mysql-visibility:
      sql:
        pluginName: 'mysql8' # Verify supported versions. Use a version of SQL that supports advanced Visibility.
        databaseName: 'temporal_secondary_visibility'
        connectAddr: '127.0.0.1:3306'
        connectProtocol: 'tcp'
        user: 'temporal'
        password: 'temporal'
```

To configure Elasticsearch as both your primary and secondary store, use the configuration key
`elasticsearch.indices.secondary_visibility`, as shown in the following example.

```yaml
persistence:
  visibilityStore: es-visibility
  datastores:
    es-visibility:
      elasticsearch:
        version: 'v7'
        logLevel: 'error'
        url:
          scheme: 'http'
          host: '127.0.0.1:9200'
        indices:
          visibility: temporal_visibility_v1
          secondary_visibility: temporal_visibility_v1_new
        closeIdleConnectionsInterval: 15s
```

#### Database schema and setup

The database schema and setup for a secondary store depends on the database you plan to use.

- [MySQL](#mysql)
- [PostgreSQL](#postgresql)
- [SQLite](#sqlite)
- [Elasticsearch](#elasticsearch)

For the Cassandra and MySQL configuration in the previous example, an example setup script would be as follows.

```bash
#...
# set your Cassandra environment variables
: "${KEYSPACE:=temporal}"
: "${VISIBILITY_KEYSPACE:=temporal_primary_visibility}"

: "${CASSANDRA_SEEDS:=}"
: "${CASSANDRA_PORT:=9042}"
: "${CASSANDRA_USER:=}"
: "${CASSANDRA_PASSWORD:=}"
: "${CASSANDRA_TLS_ENABLED:=}"
: "${CASSANDRA_CERT:=}"
: "${CASSANDRA_CERT_KEY:=}"
: "${CASSANDRA_CA:=}"
: "${CASSANDRA_REPLICATION_FACTOR:=1}"
#...
# set connection details
#...
# set up Cassandra schema
setup_cassandra_schema() {
  #...
  # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}"
    fi
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0
    temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
  #...
}
#...
# set your MySQL environment variables
: "${DBNAME:=temporal}"
: "${VISIBILITY_DBNAME:=temporal_secondary_visibility}"
: "${DB_PORT:=}"
: "${MYSQL_SEEDS:=}"
: "${MYSQL_USER:=}"
: "${MYSQL_PWD:=}"
: "${MYSQL_TX_ISOLATION_COMPAT:=false}"

#...
# set connection details
#...
# set up MySQL schema
setup_mysql_schema() {
    #...
    # use valid schema for the version of the database you want to set up for Visibility
    VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/mysql/${MYSQL_VERSION_DIR}/visibility/versioned
    if [[ ${SKIP_DB_CREATE} != true ]]; then
        temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" create
    fi
    temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" setup-schema -v 0.0
    temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
#...
}
```

For Elasticsearch as both primary and secondary Visibility store configuration in the previous example, an example setup
script would be as follows.

```bash
#...
# Elasticsearch
: "${ENABLE_ES:=false}"
: "${ES_SCHEME:=http}"
: "${ES_SEEDS:=}"
: "${ES_PORT:=9200}"
: "${ES_USER:=}"
: "${ES_PWD:=}"
: "${ES_VERSION:=v7}"
: "${ES_VIS_INDEX:=temporal_visibility_v1_dev}"
: "${ES_SEC_VIS_INDEX:=temporal_visibility_v1_new}"
: "${ES_SCHEMA_SETUP_TIMEOUT_IN_SECONDS:=0}"

#...

# Validate your ES environment
#...
# Wait for ES to start
#...
# Set up Elasticsearch index
setup_es_index() {
    ES_SERVER="${ES_SCHEME}://${ES_SEEDS%%,*}:${ES_PORT}"
    # ES_SERVER is the URL of Elasticsearch server i.e. "http://localhost:9200".
    SETTINGS_URL="${ES_SERVER}/_cluster/settings"
    SETTINGS_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json
    TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template"
    SCHEMA_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json
    INDEX_URL="${ES_SERVER}/${ES_VIS_INDEX}"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n"
    curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n"

    # Checks for and sets up Elasticsearch as a secondary Visibility store
    if [[ ! -z "${ES_SEC_VIS_INDEX}" ]]; then
      SEC_INDEX_URL="${ES_SERVER}/${ES_SEC_VIS_INDEX}"
      curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${SEC_INDEX_URL}" --write-out "\n"
    fi
}
```

#### Update Temporal Service configuration

With the primary and secondary stores set, update the `system.secondaryVisibilityWritingMode` and
`system.enableReadFromSecondaryVisibility` configuration keys in your self-hosted Temporal Service's dynamic
configuration YAML file to enable read and/or write operations to the secondary Visibility store.

For example, to enable write operations to both primary and secondary stores, but disable reading from the secondary
store, use the following.

```yaml
system.secondaryVisibilityWritingMode:
  - value: 'dual'
    constraints: {}
system.enableReadFromSecondaryVisibility:
  - value: false
    constraints: {}
```

For details on the configuration options, see:

- [Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings)
- [Migrating Visibility databases](#migrating-visibility-database)

## How to migrate Visibility database 

To migrate your Visibility database, [set up a secondary Visibility store](#dual-visibility) to enable
[Dual Visibility](/dual-visibility), and update the dynamic configuration in your Temporal Service to update the read
and write operations for the Visibility store.

Dual Visibility setup is optional but useful in gradually migrating your Visibility data to another database.

Before you begin, verify [supported databases and versions](/self-hosted-guide/visibility) for a Visibility store.

The following steps describe how to migrate your Visibility database.

After you make any changes to your [Temporal Service configuration](/temporal-service/configuration), ensure that you
restart your services.

#### Set up secondary Visibility store

1. In your Temporal Service configuration,
   [add a secondary Visibility store](/references/configuration#secondaryvisibilitystore) to your Visibility setup under
   the Persistence configuration.

   Example: To migrate from Cassandra to Elasticsearch, add Elasticsearch as your secondary database and set it up. For
   details, see [secondary Visibility database schema and setup](#dual-visibility).

   ```yaml
   persistence:
   visibilityStore: cass-visibility
   secondaryVisibilityStore: es-visibility
   datastores:
     cass-visibility:
     cassandra:
       hosts: '127.0.0.1'
       keyspace: 'temporal_visibility'
     es-visibility:
     elasticsearch:
       version: 'v7'
       logLevel: 'error'
       url:
       scheme: 'http'
       host: '127.0.0.1:9200'
       indices:
       visibility: temporal_visibility_v1_dev
       closeIdleConnectionsInterval: 15s
   ```

1. Update the [dynamic configuration](/temporal-service/configuration#dynamic-configuration) keys on your self-hosted
   Temporal Service to enable write operations to the secondary store and disable read operations. Example:

   ```yaml
   system.secondaryVisibilityWritingMode:
   - value: "dual"
   constraints: {}
   system.enableReadFromSecondaryVisibility:
   - value: false
   constraints: {}
   ```

At this point, Visibility data is read from the primary store, and all Visibility data is written to both the primary
and secondary store. This setting applies only to new Visibility data generated after Dual Visibility is enabled. It
does not migrate any existing data in the primary store to the secondary store.

For details on write options to the secondary store, see
[Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings).

#### Run in dual mode

When you enable a secondary store, only new Visibility data is written to both primary and secondary stores. The primary
store still holds the Workflow Execution data from before the secondary store was set up.

Running in dual mode lets you plan for closed and open Workflow Executions data from before the secondary store was set
up in your self-hosted Temporal Service.

Example:

- To manage closed Workflow Executions data, run in dual mode until the Namespace
  [Retention Period](/temporal-service/temporal-server#retention-period) is reached. After the Retention Period,
  Workflow Execution data is removed from the Persistence and Visibility stores. If you want to keep the closed Workflow
  Executions data after the set Retention Period, you must set up [Archival](/self-hosted-guide/archival).
- To manage data for all open Workflow Executions, run in dual mode until all the Workflow Executions started before
  enabling Dual Visibility mode are closed. After the Workflow Executions are closed, verify the Retention Period and
  set up Archival if you need to keep the data beyond the Retention Period.

You can run your Visibility setup in dual mode for an indefinite period, or until you are ready to deprecate the primary
store and move completely to the secondary store without losing data.

#### Deprecate primary Visibility store

When you are ready to deprecate your primary store, follow these steps.

1. Update the dynamic configuration YAML to enable read operations from the secondary store. Example:

   ```yaml
   system.secondaryVisibilityWritingMode:
   - value: "dual"
   constraints: {}
   system.enableReadFromSecondaryVisibility:
   - value: true
   constraints: {}
   ```

   At this point, Visibility data is read from the secondary store only. Verify whether data on the secondary store is
   correct.

1. When the secondary store is vetted and ready to replace your current primary store, change your Temporal Service
   configuration to set the secondary store as your primary, and remove the dynamic configuration set in the previous
   steps. Example:

   ```yaml
   persistence:
   visibilityStore: es-visibility
   datastores:
     es-visibility:
     elasticsearch:
       version: 'v7'
       logLevel: 'error'
       url:
       scheme: 'http'
       host: '127.0.0.1:9200'
       indices:
       visibility: temporal_visibility_v1_dev
       closeIdleConnectionsInterval: 15s
   ```

## Managing custom Search Attributes 

To manage custom Search Attributes on Temporal Cloud, use the [`tcld`](/cloud/tcld/namespace#search-attributes) CLI tool.
With Temporal Cloud, you can create and rename custom Search Attributes. If you need to delete a custom Search Attribute, contact Support at [support.temporal.io](https://support.temporal.io).
To manage custom Search Attributes on a self-hosted Temporal Service, use the [Temporal CLI](/cli/command-reference/operator#search-attribute).
With a self-hosted Temporal Service, you can create and remove custom Search Attributes.

If you're self-hosting, verify whether your [Visibility database](/self-hosted-guide/visibility#supported-databases) version supports custom Search Attributes before proceeding.

> **⚠️ Caution:**
> Do not use sensitive data or PII in Search Attributes
>
> Do not include sensitive data, secrets, or personally identifiable information (PII) in Search Attribute **names or values**.
> Search Attribute values are stored unencrypted in the Visibility store and are not processed by a custom [Payload Codec](/payload-codec#payload-codec).
> The Temporal Server must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality.
>
> Attribute names are also visible in Namespace configuration, query expressions, and Temporal UI.
> Using sensitive data in either names or values risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2.
>

### How to create custom Search Attributes 

Creating a custom Search Attribute in your Visibility store makes it available to use in your Workflow metadata and
[List Filters](/list-filter).

**On Temporal Cloud**

To create custom Search Attributes on Temporal Cloud, use
[`tcld namespace search-attributes add`](/cloud/tcld/namespace/#search-attributes). For example, to add a custom Search
Attributes "CustomSA" to your Temporal Cloud Namespace "YourNamespace", run the following command.
`tcld namespace search-attributes add --namespace YourNamespace --search-attribute "CustomSA"`

**On self-hosted Temporal Service**

To create custom Search Attributes in your self-hosted Temporal Service Visibility store, use
`temporal operator search-attribute create` with `--name` and `--type` command options.

For example, to create a Search Attribute called `CustomSA` of type `Keyword`, run the following command:

```
temporal operator search-attribute create --name="CustomSA" --type="Keyword"
```

Note that if you use a SQL database with advanced Visibility capabilities, you are required to specify a Namespace when
creating a custom Search Attribute. For example:

```
temporal operator search-attribute create --name="CustomSA" --type="Keyword" --namespace="yournamespace"
```

You can also create multiple custom Search Attributes when you set up your Visibility store.

The following example shows how custom Search Attributes can be created during Visibility store setup for SQL databases.
For setup examples, refer to the [samples-server repository](https://github.com/temporalio/samples-server)

```bash
add_custom_search_attributes() {
    until temporal operator search-attribute list --namespace "${DEFAULT_NAMESPACE}"; do
      echo "Waiting for namespace cache to refresh..."
      sleep 1
    done
    echo "Namespace cache refreshed."

    echo "Adding Custom*Field search attributes."

    temporal operator search-attribute create --namespace "${DEFAULT_NAMESPACE}" --yes \
        --name="CustomKeywordField" --type="Keyword" \
        --name="CustomStringField" --type="Text" \
        --name="CustomTextField" --type="Text" \
        --name="CustomIntField" --type="Int" \
        --name="CustomDatetimeField" --type="Datetime" \
        --name="CustomDoubleField" --type="Double" \
        --name="CustomBoolField" --type="Bool"
}
```

For Temporal Server v1.19 and earlier, or if using Elasticsearch for advanced Visibility, you can create custom Search
Attributes without a Namespace association, as shown in the following example.

```bash
add_custom_search_attributes() {
       echo "Adding Custom*Field search attributes."
       temporal operator search-attribute create \
        --name="CustomKeywordField" --type="Keyword" \
        --name="CustomStringField" --type="Text" \
        --name="CustomTextField" --type="Text" \
        --name="CustomIntField" --type="Int" \
        --name="CustomDatetimeField" --type="Datetime" \
        --name="CustomDoubleField" --type="Double" \
        --name="CustomBoolField" --type="Bool"
}
```

When your Visibility store is set up and running, these custom Search Attributes are available to use in your Workflow
code.

### How to remove custom Search Attributes 

To remove a Search Attribute key from your self-hosted Temporal Service Visibility store, use the command
`temporal operator search-attribute remove`. Removing Search Attributes is not supported on Temporal Cloud.

For example, if using Elasticsearch for advanced Visibility, to remove a custom Search Attribute called `CustomSA` of
type Keyword use the following command:

```
temporal operator search-attribute remove \
    --name="your_custom_attribute"
```

If you use a SQL database for advanced Visibility on Temporal Server v1.20 and later, you need to specify the Namespace
in your command, as shown in the following command:

```
temporal operator search-attribute remove \
    --name="your_custom_attribute" \
    --namespace="your_namespace"
```

To check whether the Search Attribute was removed, run

```
temporal operator search-attribute list
```

and check the list.

If you're on Temporal Server v1.20 and later, specify the Namespace from which you removed the Search Attribute. For
example,

```
temporal search-attribute list --namespace="yournamespace"
```

Note that if you use [SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, a new custom
Search Attribute is mapped to a database field name in the Visibility store `custom_search_attributes` table. Removing
this custom Search Attribute removes the mapping with the database field name but does not remove the data. If you
remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database
field of the one that was recently removed. This might cause unexpected results when you use the List API to retrieve
results using the new custom Search Attribute. These constraints do not apply if you use Elasticsearch.