Creating Clusters

Note

Only global administrators and Solution Manager administrators can create clusters. More information is available in the Authorization section.

Note

It is not possible to create servers in automated cloud mode environments.

Solution Manager administrators have several ways to create clusters:

  1. Click on the environment, aws-env-node (AWS) azure-env-node (Azure), and then click New cluster on the emerging menu.

Create new cluster from environment

Create new cluster from environment

  1. When you create an environment, the Solution Manager Administration Tool will suggest you to create a new cluster. Accept the suggestion.

Suggestion for creating a cluster

Suggestion for creating a cluster

Note

Only one automated cluster can be created in each environment.


New AWS Cluster

Global Configuration

Whatever path you follow, the following dialog will open to create a new cluster:

Dialog to create an automated AWS cluster

Dialog to create an automated AWS cluster

A cluster can be composed of three different types of servers: Virtual DataPort, Data Catalog and Scheduler. Each of them will be configured in its own tab, but common properties are configured in the global configuration tab. This dialog considers the following fields:

  • Name: Descriptive name for the cluster.

  • Description: Extensive description about the cluster. This field is optional.

  • Key Pair: The key pair used to connect to the EC2 instances that are part of the cluster, you must have access to the private key to connect to the instances. Please check AWS key pairs for further details.

    Note

    Denodo Solution Manager does not create nor store key pairs, you have to manually create a key pair using AWS console and store it in a safe location in case that you needed it to manually connect to an instance.

  • Subnet: Subnet to use for this cluster. You can select an existing subnet or create a new one providing the CIDR block of the subnet. This is the subnet that will be used for the resources. The instances will be launched with a private or a public IP address according to the subnet auto-assign public IP setting. Check AWS VPC subnets for further details.

TLS Configuration

In this section you can define the cluster SSL/TLS configuration.

Dialog to configure TLS

Dialog to configure TLS

You can enable TLS by providing the needed certificates in one of the following formats:

  • Java KeyStore: You have to provide the JKS format file and the public certificate in CER format. You can optionally provide a chain of certificates to trust in CER format.

  • PEM: You have to provide both the private key file and the public certificate in PEM format. You can optionally provide a chain of certificates to trust in PEM format.

  • PKCS #12: You have to provide the PKCS #12 bundle containing the private and public keys and its password.

Note

A KeyStore will be created to store the certificates if you have not chosen to provide it directly.

You always have to provide:

  • KeyStore password: This password will be used in the KeyStore storing the private certificates.

  • TrustStore password: This password will be used in the TrustStore storing the public certificates.

    Note

    By default the TrustStore password used in the JVM is “changeit”.

When configuring TLS, new AMIs containing the certificates will be created from the AMIs you have chosen, replacing them. Neither the certificates nor the passwords can be retrieved editing the cluster, if you ever replace the AMI manually, you will have to configure TLS again.

Advanced options

In this section you can configure the following elements:

Dialog to configure AWS Advanced options

Dialog to configure AWS Advanced options

  • IAM role: the instance profile that will be assigned to the instances launched. Please check Instance Profiles for further details. This value overrides the global one defined in Automated Cloud Mode.

  • S3 logging enabled: Use S3 to store logs from all cluster instances.

  • S3 base location URL: this is the URL of the bucket and folder where the logs of the cluster will be saved. For instance s3://my-bucket/denodo/. This value overrides the global one defined in Automated Cloud Mode. When input, an IAM role must be defined.

Note

See an example of minimum permissions for S3 access.

Configuration of the My Applications Panel

In this section you can fill additional information regarding how to access the different tools in the main panel (for instance if it is running behind a load balancer):

Dialog to configure My Applications Panel

Dialog to configure My Applications Panel

  • Design Studio URL: The URL of the Design Studio that will appear under “My Applications” for the cluster.

  • VDP Server Load Balancer URL: The URL of the Virtual DataPort servers load balancer that will appear under “My Applications” for the cluster.

  • Scheduler Administration Tool URL: The URL of the Scheduler Administration Tool load balancer that will appear under “My Applications” for the cluster.

  • Scheduler Server Load Balancer URL: The URL of the Scheduler servers load balancer that will appear under “My Applications” for the cluster.

  • Data Catalog Load Balancer URL: The URL of the Data Catalog load balancer that will appear under “My Applications” for the cluster.

Servers Configuration

Each group of servers has the following specific configuration:

Dialog to create an AWS cluster (Instances)

Dialog to create an AWS cluster (Instances)

  • Enabled: Whether the cluster will contain instances of this type of server or not. This is not shown for Virtual DataPort as a cluster must always contains at least 1 instance of such server type.

  • Number of instances: Number of instances of the server type that the cluster will have.

  • Instance Type: Types of the EC2 instance that will be created, each type fits different use cases. Please check AWS instance types for further details. Not all instances types are supported, only supported types are listed.

Load Balancing & Auto Scaling

Each group of servers will have a network load balancer that will serve as the single point of contact for the instances. Please check AWS network load balancers for further details.

Dialog to create an AWS cluster (Load Balancing & Auto Scaling)

Dialog to create an AWS cluster (Load Balancing & Auto Scaling)

  • Load Balancer Name: The name of the load balancer that will be created. An autogenerated one appears but you can set your own name.

  • Load Balancer DNS: Once correctly created, the DNS name assigned by Amazon to the load balancer. You will use this as the entry point for the group of servers.

  • Internet Facing Load Balancer: Whether the balancer will be reachable from the internet or not.

    When this option is enabled, Solution Manager will configure a default internet gateway for the VPC if needed.

  • Launch instances in Auto Scaling Group: If enabled the instances will be launched in an auto scaling group, this ensures you have the correct number of instance available at all moment. Please check AWS auto scaling for further details.

    This is not supported for Scheduler servers.

    Important

    If you change the value of the Launch instances in Auto Scaling Group option once the cluster is already created, make sure you recreate the cluster without selecting the Minimizing downtime option.

  • Auto Scaling Group Name: The name of the auto scaling group that will be created. An autogenerated one appears but you can set your own name.

EBS Storage

This section lets you configure the servers storage.

Dialog to create an AWS cluster (EBS Storage)

Dialog to create an AWS cluster (EBS Storage)

  • Volume Size (GiB): Size of the EBS volume for each instance in gibibytes.

  • EBS Volume Type: AWS provides different volume types that differ in performance characteristics so you can choose the most appropriate for your use case. Please check AWS EBS types for further details.

    When the chosen volume supports it an IOPS field will let you specify the desired value.

  • EBS Optimized: Enable EC2 instances to fully use the IOPS provisioned on an EBS volume. Please check AWS EBS optimized for further details.

    To encrypt the content of all the Amazon Elastic Block Storage volumes (Amazon EBS) that Solution Manager will create, do this:

    1. Configure the AWS account that the Solution Manager uses, to encrypt the new EBS volumes. The documentation of AWS explains how to do this (Turn on automatic encryption of new Amazon EBS volumes). After this, all the new EBS volumes created with this account will be encrypted.

    2. Read Using key policies in AWS KMS for more details. Note that the user has to be allowed to both use the key and attach persistent resources to successfully use it in EC2.

Advanced Options

This section contains advanced options for the servers.

Dialog to create an AWS cluster (Advanced Options)

Dialog to create an AWS cluster (Advanced Options)

  • AMI: Amazon Machine Instance (AMI) that will be used as “template” to launch this AWS EC2 instance.

    You can enter either select an AMI provided by Denodo or enter the ID of your own AMI (Custom).

    When selecting Custom, you cannot use any AMI; you have to create one following the instructions of the appendix Creating A Custom AWS AMI for the Denodo Platform.

    To obtain the list of AMIs provided by Denodo, the Solution Manager connects to the Denodo Support Site using the Denodo Support Access Keys you entered on the page Configuration > Automated mode.

    If you leave this empty, this instance will be launched with the AMI you entered in the box Default AMI of the page Configuration > Automated mode.

  • JVM Options: JVM options used in the Denodo Platform of each server.

  • Security Groups: A security group acts as a virtual firewall that controls the traffic for the instances. You can choose to create a new security group or choose from the existing ones. Please check AWS security groups for further details.

    If you choose to create a new security group you can specify a custom name instead of the autogenerated one.

  • Instance Name Prefix: All the instances in this cluster will be assigned names beginning with this prefix plus a random part.


New Azure Cluster

Global Configuration

Whatever path you follow, the following dialog will open to create a new cluster:

Dialog to create an automated Azure cluster

Dialog to create an automated Azure cluster

A cluster can be composed of three different types of servers: Virtual DataPort, Data Catalog and Scheduler. Each of them will be configured in its own tab, but common properties are configured in the global configuration tab. This dialog considers the following fields:

  • Name: Descriptive name for the cluster.

  • Description: Extensive description about the cluster. This field is optional.

  • Administrator Account: The fields Administrator Username and Administrator Password define the needed account when creating Azure Virtual Machines.

    Note

    The username can be a maximum of 20 characters in length, cannot be a disallowed value, end with a period (“.”), or contain the following characters: / ” [ ] : | < > + = ; , ? * @

    The password must be at least 12 characters long and meet the defined complexity requirements.

  • Subnet: Subnet to use for this cluster. You can select an existing subnet or create a new one providing the CIDR block of the subnet. This is the subnet that will be used for the resources. Please check Azure Virtual Network documentation for further details.

TLS Configuration

In this section you can define the cluster SSL/TLS configuration.

Dialog to configure TLS

Dialog to configure TLS

You can enable TLS by providing the needed certificates in one of the following formats:

  • Java KeyStore: You have to provide the JKS format file and the public certificate in CER format. You can optionally provide a chain of certificates to trust in CER format.

  • PEM: You have to provide both the private key file and the public certificate in PEM format. You can optionally provide a chain of certificates to trust in PEM format.

  • PKCS #12: You have to provide the PKCS #12 bundle containing the private and public keys and its password.

Note

A KeyStore will be created to store the certificates if you have not chosen to provide it directly.

You always have to provide:

  • KeyStore password: This password will be used in the KeyStore storing the private certificates.

  • TrustStore password: This password will be used in the TrustStore storing the public certificates.

    Note

    By default the TrustStore password used in the JVM is “changeit”.

When configuring TLS, new images containing the certificates will be created from the images you have chosen, replacing them. Neither the certificates nor the passwords can be retrieved editing the cluster, if you ever replace the image manually, you will have to configure TLS again.

Advanced Options

In this section you can configure the following elements:

Dialog to configure Azure Advanced options

Dialog to configure Azure Advanced options

  • Azure Blob Storage logging enabled: Use a Blob Storage Account to store logs from all cluster instances. Please check Introduction to Azure Blob storage for further details.

  • Azure Blob storage base location URL: URL of the blob storage with container and prefix where the logs will be saved. This value overrides the global one defined in Automated Cloud Mode.

  • Storage account connection string: The connection string that can be used to authorize access to data in your storage account via Shared Key authorization. This value overrides the global one defined in Automated Cloud Mode.

  • Role to assign: the Azure custom role that will be assigned to the virtual machines launched. It must previously exist in the Azure portal, where it is configured with the desired permissions and assignable scopes. Please check Azure custom roles for further details. This value overrides the global one defined in Automated Cloud Mode.

  • Role assignment Scope: an Azure custom role has assignable scopes, here you can configure the scope to use when assigning the role to the virtual machines. It is possible to configure a subscription or resource group scope. Please check the Azure documentation for further details. This value overrides the global one defined in Automated Cloud Mode.

Note

The next permissions are required to manage role asignments: Microsoft.Authorization/roleAssignments/write and Microsoft.Authorization/roleAssignments/delete

Configuration of the My Applications Panel

In this section you can fill additional information regarding how to access the different tools in the main panel (for instance if it is running behind a load balancer):

Dialog to configure My Applications Panel

Dialog to configure My Applications Panel

  • Design Studio URL: The URL of the Design Studio that will appear under “My Applications” for the cluster.

  • VDP Server Load Balancer URL: The URL of the Virtual DataPort servers load balancer that will appear under “My Applications” for the cluster.

  • Scheduler Administration Tool URL: The URL of the Scheduler Administration Tool load balancer that will appear under “My Applications” for the cluster.

  • Scheduler Server Load Balancer URL: The URL of the Scheduler servers load balancer that will appear under “My Applications” for the cluster.

  • Data Catalog Load Balancer URL: The URL of the Data Catalog load balancer that will appear under “My Applications” for the cluster.

Servers Configuration

Each group of servers has the following specific configuration:

Dialog to create an Azure cluster (Instances)

Dialog to create an Azure cluster (Instances)

  • Enabled: Whether the cluster will contain instances of this type of server or not. This is not shown for Virtual DataPort as a cluster must always contains at least 1 instance of such server type.

  • Number of instances: Number of instances of the server type that the cluster will have.

  • Virtual Machine Size: The desired VM size to support the instance that you want to create. Azure offers a wide variety of sizes to support many types of uses. Please check Sizes for virtual machines in Azure for further details.

Load Balancing & Auto Scaling

Each group of servers will have a load balancer that will serve as the single point of contact for the instances. Please check Load Balancer documentation for further details.

Dialog to create an Azure cluster (Load Balancing & Auto Scaling)

Dialog to create an Azure cluster (Load Balancing & Auto Scaling)

  • Load Balancer Name: The name of the load balancer that will be created. An autogenerated one appears but you can set your own name.

  • Load Balancer DNS: Once correctly created, the DNS name assigned by Azure to the load balancer. You will use this as the entry point for the group of servers.

  • Internet Facing Load Balancer: Whether the balancer will be reachable from the internet or not.

  • Launch instances in a Scale Set: If enabled the instances will be launched in a scale set, this ensures you have the correct number of instance available at all moment. Please check Virtual Machine Scale Sets documentation for further details.

    This is not supported for Scheduler servers.

    Important

    If you change the value of the Launch instances in a Scale Set option once the cluster is already created, make sure you recreate the cluster without selecting the Minimizing downtime option.

  • Scale Set Name: The name of the scale set that will be created. An autogenerated one appears but you can set your own name.

Disk

This section lets you configure the disk attached to the Virtual Machine. Please check Introduction to Azure managed disks for further details.

Dialog to create an Azure cluster (Disk)

Dialog to create an Azure cluster (Disk)

Advanced Options

This section contains advanced options for the servers.

Dialog to create an Azure cluster (Advanced Options)

Dialog to create an Azure cluster (Advanced Options)

  • Image: Selected image that will be used as the base operating system or application for the Virtual Machine.

    You can enter either select an image provided by Denodo or enter the ID of your own (Custom).

    When selecting Custom, you cannot use any image; you have to create one following the instructions of the appendix Creating A Custom Azure Image for the Denodo Platform.

    To obtain the list of images provided by Denodo, the Solution Manager connects to the Denodo Support Site using the Denodo Support Access Keys you entered on the page Configuration > Automated mode.

    If you leave this empty, this instance will be launched with the image you entered in Default Region & Images on the page Configuration > Automated mode.

  • JVM Options: JVM options used in the Denodo Platform of each server.

  • Network Security Groups: A security group acts as a virtual firewall that controls the traffic for the instances. You can choose to create a new security group or choose from the existing ones. Please check Network Security Groups documentation for further details.

    If you choose to create a new network security group you can specify a custom name instead of the autogenerated one.

  • Instance Name Prefix: All the instances in this cluster will be assigned names beginning with this prefix plus a random part.

  • Use public IP on instances: The instances will be launched with a public IP address according to this setting.