Creating Clusters¶

Note

Only global administrators and Solution Manager administrators can create clusters. More information is available in the Authorization section.

Note

It is not possible to create servers in automated cloud mode environments.

Solution Manager administrators have several ways to create clusters:

Click on the environment, (AWS) (Azure), and then click New cluster on the emerging menu.

Create new cluster from environment¶

When you create an environment, the Solution Manager Administration Tool will suggest you to create a new cluster. Accept the suggestion.

Suggestion for creating a cluster¶

Note

Only one automated cluster can be created in each environment.

New AWS Cluster¶

Note

The following menus are disabled when accessing Solution Manager from Agora:

TLS Configuration
Advanced Options
Configuration of the My Applications Panel
Cluster instances (VDP, Data Catalog and Scheduler):
- Load Balancing & Auto Scaling:
  - Load Balancer Name
  - Load Balancer DNS
  - Auto Scaling Group Name
- Advanced Options:
  - AMI
  - Security Groups
  - Security Group name
  - Instance Name Prefix

Global Configuration¶

Whatever path you follow, the following dialog will open to create a new cluster:

Dialog to create an automated AWS cluster¶

A cluster can be composed of three different types of servers: Virtual DataPort, Data Catalog and Scheduler. Each of them will be configured in its own tab, but common properties are configured in the global configuration tab. This dialog considers the following fields:

Name: Descriptive name for the cluster.
Description: Extensive description about the cluster. This field is optional.
Key Pair: The key pair used to connect to the EC2 instances that are part of the cluster, you must have access to the private key to connect to the instances. Please check AWS key pairs for further details.

Note

Denodo Solution Manager does not create nor store key pairs, you have to manually create a key pair using AWS console and store it in a safe location in case that you needed it to manually connect to an instance.
Subnet: Subnet to use for this cluster. You can select an existing subnet or create a new one providing the CIDR block of the subnet. This is the subnet that will be used for the resources. The instances will be launched with a private or a public IP address according to the subnet auto-assign public IP setting. Check AWS VPC subnets for further details.

TLS Configuration¶

In this section you can define the cluster SSL/TLS configuration.

Dialog to configure TLS¶

You can enable TLS by providing the needed certificates in one of the following formats:

Java KeyStore: You have to provide the JKS format file and the public certificate in CER format. You can optionally provide a chain of certificates to trust in CER format.
PEM: You have to provide both the private key file and the public certificate in PEM format. You can optionally provide a chain of certificates to trust in PEM format.
PKCS #12: You have to provide the PKCS #12 bundle containing the private and public keys and its password.

Note

A KeyStore will be created to store the certificates if you have not chosen to provide it directly.

You always have to provide:

KeyStore password: This password will be used in the KeyStore storing the private certificates.
TrustStore password: This password will be used in the TrustStore storing the public certificates.

Note

By default the TrustStore password used in the JVM is “changeit”.

When configuring TLS, new AMIs containing the certificates will be created from the AMIs you have chosen, replacing them. Neither the certificates nor the passwords can be retrieved editing the cluster, if you ever replace the AMI manually, you will have to configure TLS again.

Trusted TLS Certificates¶

Note

Trusted TLS Certificates configuration is only available in Agora.

In this section you can manage (add, remove and download) TLS Certificates that you want all the servers in the cluster to trust. You can provide certificates in DER or PEM format. This configuration is independent of the TLS Configuration for the cluster.

Dialog to configure trusted TLS certificates¶

After the cluster creation or a cluster restart, the added certificates will be imported in the truststores of all the servers of the cluster.

When editing the cluster, you can remove certificates from the list and save the cluster, in order to remove them from the server’s truststores. Also, there is a button to download saved certificates.

Advanced Options¶

In this section you can configure the following elements:

Dialog to configure AWS Advanced options¶

IAM role: the instance profile that will be assigned to the instances launched. Please check Instance Profiles for further details. This value overrides the global one defined in Automated Cloud Mode.
S3 logging enabled: Use S3 to store logs from all cluster instances.
S3 base location URL: this is the URL of the bucket and folder where the logs of the cluster will be saved. For instance s3://my-bucket/denodo/. This value overrides the global one defined in Automated Cloud Mode. When input, an IAM role must be defined.

Note

See an example of minimum permissions for S3 access.

When S3 logging is enabled, it causes a new Appender to be included in the log4j2 configuration file for each of the servers, called CloudAppender (check How to store Denodo logs in Amazon S3), with the following features:

The path where the logs are generated is:
- cluster_name: name of the cluster, defined at its creation.
- server_type: the possible values are vdp, sch and dc.
- instance_id: identifier of the instance in AWS. E.g. i-08bb3bd651739de1a.
For example:
```
s3://my-bucket/denodo/logs/cluster/vdp/i-08bb3bd651739de1a
```
By default, the new Appender creates a new log file every 60 minutes. The parameter stagingBufferAge of the Appenders controls this.
Name of the files created by the Appender:

<hostname>.log.<date>

For example:
ip-10-200-215-79.eu-west-1.compute.internal.log.20230327094358

Each time the instance is booted the Solution Manager is responsible for configuring the Appender, be aware that the following properties of the Appender are overwritten at each boot based on the user data of the instance, that is, the values configured in the Solution Manager Administration Tool:

<s3Bucket>bucket-name</s3Bucket>

<s3Path>path</s3Path>

<s3Region>region</s3Region>

Note

log4j2.xml file location per server:

Virtual DataPort: <DENODO_HOME>/conf/vdp/.

Scheduler: <DENODO_HOME>/conf/scheduler/.

Data Catalog: <DENODO_HOME>/conf/data-catalog/.

Configuration of the My Applications Panel¶

In this section you can fill additional information regarding how to access the different tools in the main panel (for instance if it is running behind a load balancer):

Dialog to configure My Applications Panel¶

Design Studio URL: The URL of the Design Studio that will appear under “My Applications” for the cluster.
VDP Server Load Balancer URL: The URL of the Virtual DataPort servers load balancer that will appear under “My Applications” for the cluster.
Scheduler Administration Tool URL: The URL of the Scheduler Administration Tool load balancer that will appear under “My Applications” for the cluster.
Scheduler Server Load Balancer URL: The URL of the Scheduler servers load balancer that will appear under “My Applications” for the cluster.
Data Catalog Load Balancer URL: The URL of the Data Catalog load balancer that will appear under “My Applications” for the cluster.

Servers Configuration¶

Each group of servers has the following specific configuration:

Dialog to create an AWS cluster (Instances)¶

Enabled: Whether the cluster will contain instances of this type of server or not. This is not shown for Virtual DataPort as a cluster must always contains at least 1 instance of such server type.
Number of instances: Number of instances of the server type that the cluster will have.
Instance Type: Types of the EC2 instance that will be created, each type fits different use cases. Please check AWS instance types for further details. Not all instances types are supported, only supported types are listed.

Load Balancing & Auto Scaling¶

Each group of servers will have a network load balancer that will serve as the single point of contact for the instances. Please check AWS network load balancers for further details.

Dialog to create an AWS cluster (Load Balancing & Auto Scaling)¶

Load Balancer Name: The name of the load balancer that will be created. An autogenerated one appears but you can set your own name.
Load Balancer DNS: Once correctly created, the DNS name assigned by Amazon to the load balancer. You will use this as the entry point for the group of servers.
Internet Facing Load Balancer: Whether the balancer will be reachable from the internet or not.

When this option is enabled, Solution Manager will configure a default internet gateway for the VPC if needed.
Launch instances in Auto Scaling Group: If enabled the instances will be launched in an auto scaling group, this ensures you have the correct number of instance available at all moment. Please check AWS auto scaling for further details.

This is not supported for Scheduler servers.

Important

If you change the value of the Launch instances in Auto Scaling Group option once the cluster is already created, make sure you recreate the cluster without selecting the Minimizing downtime option.
Auto Scaling Group Name: The name of the auto scaling group that will be created. An autogenerated one appears but you can set your own name.

EBS Storage¶

This section lets you configure the servers storage.

Dialog to create an AWS cluster (EBS Storage)¶

Volume Size (GiB): Size of the EBS volume for each instance in gibibytes.
EBS Volume Type: AWS provides different volume types that differ in performance characteristics so you can choose the most appropriate for your use case. Please check AWS EBS types for further details.

When the chosen volume supports it an IOPS field will let you specify the desired value.
EBS Optimized: Enable EC2 instances to fully use the IOPS provisioned on an EBS volume. Please check AWS EBS optimized for further details.

To encrypt the content of all the Amazon Elastic Block Storage volumes (Amazon EBS) that Solution Manager will create, do this:
1. Configure the AWS account that the Solution Manager uses, to encrypt the new EBS volumes. The documentation of AWS explains how to do this (Turn on automatic encryption of new Amazon EBS volumes). After this, all the new EBS volumes created with this account will be encrypted.
2. Read Using key policies in AWS KMS for more details. Note that the user has to be allowed to both use the key and attach persistent resources to successfully use it in EC2.

Advanced Options¶

This section contains advanced options for the servers.

Dialog to create an AWS cluster (Advanced Options)¶

AMI: Amazon Machine Instance (AMI) that will be used as “template” to launch this AWS EC2 instance.

You can enter either select an AMI provided by Denodo or enter the ID of your own AMI (Custom).

When selecting Custom, you cannot use any AMI; you have to create one following the instructions of the appendix Creating A Custom AWS AMI for the Denodo Platform.

To obtain the list of AMIs provided by Denodo, the Solution Manager connects to the Denodo Support Site using the Denodo Support Access Keys you entered on the page Configuration > Automated mode.

If you leave this empty, this instance will be launched with the AMI you entered in the box Default AMI of the page Configuration > Automated mode.
JVM Options: JVM options used in the Denodo Platform of each server.
Security Groups: A security group acts as a virtual firewall that controls the traffic for the instances. You can choose to create a new security group or choose from the existing ones. Please check AWS security groups for further details.

If you choose to create a new security group you can specify a custom name instead of the autogenerated one.
Instance Name Prefix: All the instances in this cluster will be assigned names beginning with this prefix plus a random part.