Server Ports¶

The Scheduler server uses three port numbers for client communications: the server execution port, the server stoppage port, and the auxiliary port.

The port changes will take effect the next time the Scheduler server restarts.

Threads Pool¶

The Scheduler server allows you to execute various extraction jobs simultaneously. Additionally, the VDP, VDPCache and VDPIndexer jobs allow the same job to run different queries concurrently on the same data source, varying the parameters.

You can specify the maximum number of jobs that the server will execute concurrently with the parameter Maximum number of concurrent jobs (by default 20). A change to the number of concurrent jobs will take effect the next time the Scheduler server is restarted.

With regards to VDP, VDPCache and VDPIndexer jobs, the Scheduler server uses a pool of reusable threads for managing the execution of multiple queries which the same job can generate. The parameters that can be configured are as follows:

• Default number of threads (corePoolSize). This represents the number of threads in the pool from which the inactive threads are reused (20 by default). Whilst there are fewer threads than these in the pool, new threads will continue to be created. When a thread is requested and the number of threads in the pool is the same as or more than this value, inactive threads are returned, if they exist; otherwise, new threads will continue to be created until obtaining the value established by the following parameter. Intuitively, this parameter indicates the number of threads that the system should have active simultaneously in normal load conditions.

• Maximum number of threads (maximumPoolSize). Represents the maximum number of pool threads (by default 60).

• Keep alive timeout (milliseconds). Specifies the maximum time in milliseconds that an inactive thread stays in the pool, if the number of threads exceeds the total indicated in the Default number of threads (by default 0). If the value is 0, the threads created above this value end, once the execution of their job has been completed. Otherwise, the ones that exceed the time specified by this parameter will end.

This thread pool works this way:

• The executor service has an internal queue, used for holding tasks before they are executed.

• The Default number of threads parameter is the number of core threads that will be instantiated and kept in the pool.

• When a new task is submitted,

  • if fewer than the Default number of threads threads are running, the Executor always prefers adding a new thread rather than queuing.

  • if the Default number of threads or more threads are running, the Executor always prefers queuing a request rather than adding a new thread. So:

    • if the internal queue is not full, the task is queued.

      If this task is in the queue more than the allowed time, it is dequeued and its execution is discarded.

    • If internal queue is full, a new thread is created unless this would exceed the Maximum number of threads, in which case, the task will be rejected.

      There is a rejection policy to reattempt the enqueue of the task a certain number of times, at different intervals.

    • The tasks will be waiting in the queue for a thread for a maximum amount of time.

Mail Settings¶

This section allows you to modify the properties of the outgoing mail server (Handler Section) used to send reports on the execution of jobs.

You can specify the following information:

• SMTP server. Machine name in which the e-mail server runs.

• Port. Port number in which the server runs. By default, 25.

• Protocol. Protocol used to send the e-mails: SMTP or SMTPS.

• From. The e-mail address used by Scheduler to send the mail.

• Subject. The subject of the mail. The subject may contain variables, which will be replaced by their corresponding values at execution time (they take their value from the job data). The allowed variables are: projectName, projectID, jobName, jobID, jobStartTime and jobResult, with the format @{<var_name>}. It is possible to format jobStartTime variable by specifying output date pattern. The syntax is @{jobStartTime,outputDateFormat:yyyy-MM-dd}. As an example:

  Denodo Scheduler Notification - @{jobResult} [@{projectName}
  - @{jobName}_@{jobID}] at @{jobStartTime,outputDateFormat:yyyy-MM-dd HH:mm:ss}
  

• Trust servers. If set to “*”, all hosts are trusted. If set to a whitespace separated list of hosts, those hosts are trusted. Otherwise, it depends on the certificate the server presents.

• Timeout. The maximum time to wait while sending mail to the server, once the connection is established. Set “0” to wait indefinitely.

• Connection Timeout. The maximum time to wait to establish a connection to the server. Set “0” to wait indefinitely.

• Enable TLS. If true, enables the use of the STARTTLS command (if supported by the server) to switch the connection to a TLS-protected connection before issuing any login commands. Note that an appropriate trust store must be configured so that the client will trust the server’s certificate. Defaults to false.

• Enable debug. If true, it allows to trace the mail connection. In addition, it is necessary to add the following logger to the <DENODO_HOME>/conf/scheduler/log4j2.xml file:

  <Logger name="com.denodo.scheduler.core.commons.utils.MailUtil" level="DEBUG" />
  

• Send as HTML. If true, the mailed reports are transformed to HTML format with the XSLT template <DENODO_HOME>/work/scheduler/mail/XMLtoHTML.xsl. Otherwise the report is sent as a plain string.

Additionally, if the outgoing mail server requires authentication, user name (Username) and its password (Password) must be specified.

Once configured the mail properties, the link Test mail configuration allows you to test if it is properly configured. You have to specify a list of email addresses (separated by a “,”) a test mail will be sent to and click on Send email. If the configuration is right, those recipients will receive an email with a body informing that it is a test email.

Virtual DataPort Settings¶

Denodo Scheduler servers can delegate the authentication of the users to a Denodo Virtual DataPort server (see section Authentication). Moreover, when Scheduler connects to Virtual DataPort to authenticate a user, it also retrieves its roles (which determine the user access rights in Scheduler).

In this way, Scheduler needs to know information about the Virtual DataPort server used for authentication and roles retrieval. This section allows specifying the following information:

• Host. Machine name in which the Virtual DataPort server runs.

• Port. Port number in which the Virtual DataPort server runs.

• Database (optional). Name of the Virtual DataPort server database, which should be used for the authentication (useful when the database uses LDAP authentication).

Changes in any of these parameters require the user to logout and login again to take effect.

Kerberos Settings¶

1. Select Use Kerberos.

2. In the box Server Principal enter the “Service Principal Name” (SPN) used to create the keytab file. That is, the SPN with the Fully Qualified Domain Name (FQDN) of the server where the Scheduler Server is running. For example, “HTTP/denodo-prod.subnet1.contoso.com@CONTOSO.COM”.

   Note

   It is not necessary for the SPN of the Scheduler server to start with HTTP. This is only mandatory when web browsers request a Kerberos ticket, as they do it for the service “HTTP/<host name of the URL you are accessing>”. Here, the Kerberos ticket is requested by the Scheduler administration tool, not by the browser. However, in the case of the Scheduler administration tool, it is mandatory that the SPN starts with HTTP, since in this case the browser requests the Kerberos ticket. Therefore, on the Scheduler server you could use a SPN such as “SCHED/denodo-prod.subnet1.contoso.com@CONTOSO.COM”, if you wish.

3. In the box Keytab file enter the path to the keytab file.

4. Leave the Configuration file box empty unless the host where this Scheduler server runs does not belong to a Kerberos realm (e.g., a Windows Active Directory domain). If this host does not belong to a Kerberos realm, do one of the following:

   1. Enter the path to the krb5.conf or krb5.ini file with the Kerberos settings.

   2. Or follow the steps described in the appendix Enabling Kerberos Authentication Without Joining a Kerberos Realm of the Installation Guide.

   Note

   This is the only setting you have to configure to use Kerberos authentication in the Virtual DataPort data sources

5. We recommend selecting the check box Activate Kerberos debug mode the first time you set up Kerberos in case you run into any issues. Once Kerberos has been set up, disable this.

   When this option is enabled, the Java Runtime Environment logs messages related to Kerberos in the standard output but not in the log files. To see these messages, launch the Scheduler server from the command line with the script <DENODO_HOME>/bin/scheduler_startup.

   Important

   In Windows environments, you must redirect the server’s output to a file. Otherwise, you will not be able to check the log messages related to Kerberos.

   To redirect these debug messages to a file, start the Server like this:

   For Windows:

   cd <DENODO_HOME>
   cd bin
   scheduler_startup.bat > scheduler_kerberos_logging.log 2>&1
   

   For Linux:

   cd <DENODO_HOME>
   cd bin
   ./scheduler_startup.sh > scheduler_kerberos_logging.log 2>&1
   

Changes in this parameter requires the server to be restarted to take effect.

Database Metadata Settings¶

Denodo Scheduler comes with an out-of-the-box embedded database (Apache Derby) where stores all of its metadata (projects, jobs, etc.). Nevertheless, you can specify another database to store the metadata.

As a first step, we suggest creating a schema in the metadata database exclusively dedicated to store all the data related to the Scheduler 9. While not mandatory, it is strongly recommended to keep things better organized and avoid naming conflicts.

As for the recommended size for the external database, it is difficult to give an exact number, since it is very variable depending on the specific metadata and the number of job executions. Each job execution generates a report, so the more executions, the more reports and the more size. The size of the reports depends on the number of errors during the job execution, the number of loading and indexing processes, the number of parameters (if the job is parameterized), etc. As a recommendation, reserve 5GB of space for the database (in most cases, it should be enough with less space, but we recommend a high value to avoid data corruption in case of lack of space).

Cluster Settings¶

Clustering works by having each node of the cluster share the same database. So, when enabling this feature you also have to configure the Scheduler server to use the common database (see Database Metadata Settings). As can be seen in the warning of the previous section, you should never configure a non-clustered server to use the same database as any other server, so you must not configure a common database without enabling the clustering features (note that the changes will be effective after restarting the server).

You can configure the following parameters:

• Cluster. Enable it in order to turn on clustering features. As stated in the previous subsection, this property must be set to “true” if you are having multiple instances of Scheduler using the same set of database tables.

• Node identifier. Can be any string, but must be unique for each Scheduler server in the cluster. You may use the value “AUTO” as the node identifier if you wish the identifier to be generated for you.

  Note

  When the Node Identifier is “AUTO”, the auto-generated name can be greater than the allowed number of characters (45) to store in database, so a Data Truncation exception would be thrown in that case. If so, set a fixed name (unique for each server in the cluster) with the appropriate size.

• Check interval. Sets the frequency (in milliseconds) at which this instance checks-in with the other instances of the cluster. Affects the quickness of detecting failed instances. Recommended values are, for instance, 2000 or 3000 milliseconds.

These changes will take effect the next time the Scheduler server restarts.

Configuring a cluster of Scheduler servers is the key for a High Availability (HA) scenario and provides the following features:

• Load-balancing: occurs automatically, with each node of the cluster firing jobs as quickly as it can. When a trigger’s firing time occurs, the first node to acquire it (by placing a lock on it) is the node that will fire it. Only one node will fire the job for each firing. It will not necessarily be the same node each time.

• Fail-over: occurs when one of the nodes fails while in the midst of executing one or more jobs. When a node fails, the other nodes detect the condition and identify the jobs in the database that were in progress within the failed node and they are re-executed by the remaining nodes. Notice that the complete job would be re-executed, so if it has some exporters that generate files, they might be duplicated (unless they are configured to be overwritten). Besides, the recommendation is to export these files to a shared drive to be accessible to any server in the cluster.

Server configuration settings (the ones explained in this section) can be synchronized among servers by using the Import/Export feature (see Import/Export Backups and Use of the Import/Export Scripts for Backup) and including the configuration.