You can translate the question and the replies:

Configure Bulk data load API for databricks source in denodo 8.0

Would like to know how to configure bulk data load API for Databricks source in denodo 8.0. I have gone through the Bulk data load API docs for Databricks, spark and impala. But I didnt understand how to configure it. Can you please let know the step by step process. Getting the below stack trace Cannot run program "/opt/denodo/Denodo8.0/databricks": error=13, Permission deniedcom.denodo.util.exceptions.InternalErrorException: Cannot run program "/opt/denodo/Denodo8.0/databricks": error=13, Permission denied at com.denodo.vdb.util.datamovement.TestDataMovementUtil.a3(Unknown Source) at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.a3(Unknown Source) at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.b3(Unknown Source) at com.denodo.vdb.util.tablemanagement.sql.HadoopTableManager.testDataMovement(Unknown Source) at com.denodo.vdb.util.introspectionservice.actions.TestHdfsDataMovementAction.execute(Unknown Source) at com.denodo.vdb.interpreter.execution.TestHdfsDataMovementAction.exec(Unknown Source) at com.denodo.vdb.interpreter.execution.Action.run(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.f3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.d3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source) at com.denodo.vdb.interpreter.execution.Action.start(Unknown Source) at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source) at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source) at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source) at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.denodo.internal.o.a.r.server.ObjectRef.invoke(ObjectRef.java:110) at com.denodo.internal.o.a.r.netty.RMIServerHandler.dispatch(RMIServerHandler.java:182) at com.denodo.internal.o.a.r.netty.RMIServerHandler.channelRead(RMIServerHandler.java:80) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at com.denodo.internal.i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) ***Information about encapsulated exception*** java.io.IOException: Cannot run program "/opt/denodo/Denodo8.0/databricks": error=13, Permission denied at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) at com.denodo.vdb.util.hdfs.HdfsUtil.a3(Unknown Source) at com.denodo.vdb.util.datamovement.TestDataMovementUtil.a3(Unknown Source) at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.a3(Unknown Source) at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.b3(Unknown Source) at com.denodo.vdb.util.tablemanagement.sql.HadoopTableManager.testDataMovement(Unknown Source) at com.denodo.vdb.util.introspectionservice.actions.TestHdfsDataMovementAction.execute(Unknown Source) at com.denodo.vdb.interpreter.execution.TestHdfsDataMovementAction.exec(Unknown Source) at com.denodo.vdb.interpreter.execution.Action.run(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.f3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.d3(Unknown Source) at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source) at com.denodo.vdb.interpreter.execution.Action.start(Unknown Source) at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source) at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source) at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source) at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.denodo.internal.o.a.r.server.ObjectRef.invoke(ObjectRef.java:110) at com.denodo.internal.o.a.r.netty.RMIServerHandler.dispatch(RMIServerHandler.java:182) at com.denodo.internal.o.a.r.netty.RMIServerHandler.channelRead(RMIServerHandler.java:80) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at com.denodo.internal.i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.io.IOException: error=13, Permission denied at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340) at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271) at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ... 32 more # # Check the following: # - The specified URI exists. # - The environment variable HADOOP_USER_NAME has been defined in the system where the Virtual DataPort server runs. # - The value of HADOOP_USER_NAME is an existing account in the database and it has privileges to list the contents of the HDFS URI. ################################################################################ Data movement failed
user
12-02-2021 10:06:48 -0500
code

1 Answer

Hi, In order to configure Databricks for bulk data load, I would perform the following steps: * Install Python3 and the Databricks Client after which it is possible to set the authentication. * Once the client application is downloaded. In the Virtual DataPort Administration tool, create a new JDBC data source using Databricks. * Navigate to the Read & Write tab of the data source, check the option "Use Bulk Data Load APIs" and provide the needed configuration details. Along with these settings, I would also make sure if the credentials used to connect to the data source has permission to write. If I was to use a connection setting similar to [Impala](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/bulk_data_load/impala#impala), I would make sure to set the environment variables **HADOOP_HOME ** and **HADOOP_USER_NAME** also making sure the user-provided has read and write privileges. You can take a look at the [Databricks](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/bulk_data_load/databricks) section of the Virtual DataPort Administration Guide. You can also take a look at this Knowledge Base[ How to connect to Azure Databricks from Denodo](https://community.denodo.com/kb/view/document/How%20to%20connect%20to%20Azure%20Databricks%20from%20Denodo?category=Data+Sources) for additional information. Hope this helps!
Denodo Team
15-02-2021 05:49:28 -0500
code
You must sign in to add an answer. If you do not have an account, you can register here