Creating Kafka Listeners¶

Follow these steps to create a Kafka listener:

1. Click Kafka Listener on the File > New > Listeners menu or right-click the Server Explorer and click Kafka Listener on the New > Listeners menu.

2. Set listener name. This name cannot be repeated within the same database. Required field

3. In the Kafka server you specify the IP and port of Kafka server. Required field.

4. Set group ID. This field allows multiple listeners to perform load balancing if they belong to the same consumer. Required field.

5. Click On to enable the listener. If you click Off, the listener will be created but it will be disabled.

6. Select the VDP user name that Virtual DataPort will use to execute the queries received by the listener. The user selected has to be a local user. It has to be a user created in Virtual DataPort.

   For example, if we have two users:

   1. admin: is an “administrator” or a user promoted to “local administrator” of the database so it can access any view of the database.

   2. user1: is a “normal user” that only has EXECUTE privileges over this database.

      If you select the user “admin”, the listener will be able to execute any query. However, if you select “user1”, the CREATE, DROP, INSERT, UPDATE and DELETE queries will fail because “user1” only has EXECUTE privileges.

7. If you want to use a parameterized query, select the Kafka messages contain the value of… check box and enter the query in the box below.

   As we have explained in the previous section, when using this option, the string @LISTENEREXPRESSION is replaced at runtime by the value received from the Kafka server.

8. Select Input Plain or Binary. Plain accepts JSON, XML or Plain values. Binary accepts byte[] from input topic and the value is encoded to BASE64 for it’s use in VDP. You can use a custom function in order to decode and transform to blob type to compare against blob fields. Required field.

9. Set Input topic from where Kafka listener will receive messages. Required field.

10. Select Ignore reply to in case Listener Kafka will not send response of query executed against VDP to another Topic.

11. Select Output XML, JSON or Binary to send responses back to Kafka server in that specific format. Binary has limitation to 1 row and 1 column response and field must be blob type. If you use Binary output, a byte array is sent to the Reply topic. Required if ignore reply to is not selected.

12. Set Reply topic in order to specify the topic to send response of VDP query. Required if ignore reply to is not selected.

13. Select Split the response in several messages in order to send N rows per message when response of VDP has multiple rows.

14. Set Max. number of rows in each message to specify the number of rows to include in each message. Obligatory if Split the response in several messages is selected.

15. In the Authentication drop-down, specify the authentication method to connect to the Kafka server:

    1. NONE VDP will connect to Kafka Server using PLAINTEXT protocol.

    2. SASL_PLAINTEXT VDP will connect to Kafka Server using SASL_PLAINTEXT protocol. If SSL is not enabled the security protocol used is SASL_PLAINTEXT. This requires to specify a username and password.

    3. SASL_GSSAPI VDP will connect to Kafka Server using Kerberos protocol. You have to provide the principal, the path to the keytab file in the computer that Virtual DataPort runs and the service name. If the check box SSL is cleared, the security protocol used is SASL_PLAINTEXT.

16. Select SSL enabled to connect to Kafka with SSL. When SSL is enabled with SASL_PLAINTEXT or SASL_GSSAPI authentication method, the security protocol used is SASL_SSL, otherwise, if NONE authentication method is set, the security protocol used is SSL.

    When selected, provide this:

    1. Truststore file location: location to the truststore file.

    2. Truststore password: password of the truststore.

17. Select Send error messages to Dead Letter Queue if you want to store messages that are not processed successfully and not send it to the Reply topic.

    A Dead Letter Queue in Kafka is one or more Kafka topics that receive and store messages that could not be processed in another streaming pipeline because of an error. If you select this option you can indicate the topic to send the error messages to.

18. In the Metadata tab, you can set the folder where the Kafka listener will be stored and provide a description for it.

Creating a Kafka listener¶

After creating the listener, you can open the Kafka Listener Status dialog (Tools > Listeners > Kafka Listeners menu) to see the list of existing Kafka listeners.

List of existing Kafka listeners¶

To enable or disable a Kafka listener, select the check box next to its name and click Enable selected or Disable selected.

To delete a listener, right-click on it in the Server Explorer and click Drop.

Using Design Studio the process is similar.

After the listener is created you can see the list of Kafka listeners with the Kafka Listener Status dialog.