USER MANUALS

Audit Trail in Scheduler

Starting with the update 8.0u20240306, it is possible to maintain an audit trail of changes and actions performed on the server metadata of Scheduler.

This way, we want to give an answer to this question: when and who did what on what element and which was the result.

These changes can be stored in a JSON file and/or in a database, depending on your configuration. In both cases, we are levering Log4J support, so enabling this feature involves logging configuration (as explained below).

Audit Information

The audit information contains the following fields:

  • StartTime: start time of the requested action.

  • EndTime: finish time of the requested action.

  • Elapsed Milliseconds: duration (in milliseconds) of the action.

  • SessionId: internal identifier for the user’s session.

  • ClientIP: the ip of the client.

  • UserName: login name of the user executing the action.

  • UserAuthenticationMethod: which authentication method used the user to log in (LOCAL / REMOTE / SSO / KERBEROS).

  • UserBatchMode: whether the user executing the action was in batched mode or not.

  • UserPermissions: the permissions of the user executing the action.

  • RequestType: type of operation performed by the method (START / STOP / ENABLE / DISABLE / CREATE / READ / UPDATE / DELETE, etc).

  • ElementTypes: element types (Job / DataSource / Project / ServerInfo, etc) over the request was done.

  • AffectedElements: information about the elements (identifier and name) that were affected by the execution of the audited method.

  • Endpoint: the action performed. It contains the method signature with its name and parameters.

  • Arguments: input arguments for the executed method.

  • State: Status of the action (OK / ERROR / WARNING).

  • Warnings: only for bulk methods, the information about the elements that could not be executed will be shown. It only applies when State=WARNING.

  • Thread Name: Name of the thread that generated the logging event.

  • ThreadID: identifier of the thread that generated the logging event.

  • LogLevel: log level the event was generated at.

Masking Sensitive Information

By default, when auditing events, sensitive or environment-dependent information will be masked.

It is possible to control which information should be masked with the following properties:

  • com.denodo.util.audit.config.AuditMaskingConfig.obfuscate.ip: set to false when you do not want to obfuscate the IP address of the users.

  • com.denodo.util.audit.config.AuditMaskingConfig.obfuscate.user: set to false when you do not want to obfuscate the user names.

  • com.denodo.util.audit.config.AuditMaskingConfig.obfuscate.auditMask: set to false when you do not want to obfuscate sensitive properties (mainly, environment dependent properties, such as server uris, host names, ports, etc).

  • com.denodo.util.audit.config.AuditMaskingConfig.obfuscate:

    • true will enable all obfuscation (no matter the values from the other properties).

    • false will disable all obfuscation (no matter the values from the other properties).

    • If not set, the previous properties will control the obfuscation.

Note

Passwords and keytabs will always be masked, regardless of these properties.

How to Configure Audit in Scheduler

To enable auditing in Scheduler, you have to follow these steps:

  1. Enable loggers in <DENODO_HOME>/conf/scheduler/log4j2.xml

    1. Replace <Configuration> with <Configuration packages="com.denodo.util.audit.log">

    2. Configure appenders: you can define appenders (inside <Appenders> section) to a JSON file and/or to a database (you must configure at least one of them):

      • Appender to JSON:

        <RollingFile name="REQUESTOUT-JSON" fileName="<DENODO_HOME>/logs/scheduler/scheduler-requests.json"
              filePattern="<DENODO_HOME>/logs/scheduler/scheduler-requests.json.%i">
              <JsonTemplateLayout eventTemplateUri="classpath:DenodoJsonLayout.json"
                 stackTraceEnabled="false" charset="UTF-8" />
              <SizeBasedTriggeringPolicy size="10 MB" />
              <DefaultRolloverStrategy max="7" />
        </RollingFile>
        

        Replace <DENODO_HOME> properly.

      • Appender to database:

        <JDBC name="REQUESTOUT-DB" tableName="sched_audit_logs" ignoreExceptions="false">
              <ConnectionFactory
                 class="com.denodo.scheduler.core.server.quartz.connection.SchedulerPoolingConnectionProvider"
                 method="getAuditDBConnection" />
              <AuditDBLayout />
        
              <ColumnMapping name="startTime" type="java.time.ZonedDateTime" />
              <ColumnMapping name="endTime" type="java.time.ZonedDateTime" />
              <ColumnMapping name="elapsedTime" type="java.lang.Long" />
              <ColumnMapping name="sessionID" />
              <ColumnMapping name="clientIP" />
              <ColumnMapping name="userName" />
              <ColumnMapping name="userAuthMethod" />
              <ColumnMapping name="userBatchedMode" type="java.lang.Boolean" />
              <ColumnMapping name="userPermissions" />
              <ColumnMapping name="requestType" />
              <ColumnMapping name="elementTypes" />
              <ColumnMapping name="affectedElements" />
              <ColumnMapping name="endpoint" />
              <ColumnMapping name="arguments" type="java.sql.Clob" />
              <ColumnMapping name="state" />
              <ColumnMapping name="warnings" type="java.sql.Clob" />
              <ColumnMapping name="threadName" />
              <ColumnMapping name="threadId" type="java.lang.Long" />
              <ColumnMapping name="logLevel" />
        </JDBC>
        
    3. Add logger (with appenders to file and/or database, depending on the previous step):

      <Logger name="EventLogger" level="INFO" additivity="false">
            <AppenderRef ref="REQUESTOUT-JSON" />
            <AppenderRef ref="REQUESTOUT-DB" />
      </Logger>
      
      Depending on the log level:
      • INFO: default level. It will mainly record actions that make changes to the metadata, such as creations, updates and deletions (for instance: create a job).

      • DEBUG: in addition to the information registered with INFO, it will include getters (for instance: get jobs).

      • TRACE: in addition to the information registered with DEBUG, it will include internal metadata calls and minor getters (for instance: get meta configuration for VDP jobs).

        • Besides auditing more methods from the API, it also includes more information for each one, as explained in Audit Information.

          • UserAuthenticationMethod.

          • UserBatchMode.

          • UserPermissions.

  2. Only when the appender to database is configured, you have to do these additional steps:

    1. Configure the auditing database:

      • You can configure the target database where to store the audit information. Note that by default, the audit will use the same database as the metadata.

      • If you want to use a different database than the one used to store the Scheduler metadata (this is highly recommended), then you have to manually configure its settings in <DENODO_HOME>/conf/scheduler/ConfigurationParameters.properties

        Example of audit database settings for PostgreSQL
        # Audit DataSource
        AuditDataSource/driverClassName=org.postgresql.Driver
        AuditDataSource/url=jdbc:postgresql://auditdb.denodo.com:5432/audit
        AuditDataSource/user=audit_user
        AuditDataSource/password=audit_password
        AuditDataSource/databaseName=PostgreSQL
        AuditDataSource/useKerberos=false
        AuditDataSource/classpath=postgresql-12
        
    2. Create the table (sched_audit_logs) to store the audit information in the audit database set up in the previous step:

      • Execute sql script tables_audit_trail_<db>.sql from <DENODO_HOME>/scripts/scheduler/sql/<db>.

Configure Masking

To configure how the audit information should be masked, the properties from Masking Sensitive Information have to be configured in <DENODO_HOME>/conf/scheduler/ConfigurationParameters.properties.

Add feedback