Hi,
I am trying to setup Azure Databricks + ADLS as cache.
On the Linux server, logged into the docker container runnig Denodo instance, and I installed databricks-cli and configured with token. I can see the directories when executing dbfs ls, which means I was able to connect.
On the Connection tab, I am able to successfully connect to databricks.
On the Read & Write tab, after I provided "dbfs" as Databricks executable location, At first I was getting: "**Unable to test bulk load: Internal error: java.lang.Exception: Hadoop executable file does not exists.**"
When I changed it to: "/usr/local/bin/dbfs" I am now getting:
"Listing HDFS URI contents ... FAILED
Check the following:
- The specified URI exists.
- The environment variable HADOOP_USER_NAME has been defined in the system where the Virtual DataPort server runs.
- The value of HADOOP_USER_NAME is an existing account in the database and it has privileges to list the contents of the HDFS URI.
Data movement failed
Click "Show Details" for more information."
Details:
"Product name: SparkSQL
Product version: 3.1.2
################################################################################
# Listing HDFS URI contents
#
# /usr/local/bin/dbfs ls dbfs:/denodo_mppcache
#
# Error executing hadoop commandcom.denodo.util.exceptions.InternalErrorException: Error executing hadoop command
at com.denodo.vdb.util.datamovement.TestDataMovementUtil.a3(Unknown Source)
at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.a3(Unknown Source)
at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.b3(Unknown Source)
at com.denodo.vdb.util.tablemanagement.sql.HadoopTableManager.testDataMovement(Unknown Source)
at com.denodo.vdb.util.introspectionservice.actions.TestHdfsDataMovementAction.execute(Unknown Source)
at com.denodo.vdb.interpreter.execution.TestHdfsDataMovementAction.exec(Unknown Source)
at com.denodo.vdb.interpreter.execution.Action.run(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.f3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.d3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source)
at com.denodo.vdb.interpreter.execution.Action.start(Unknown Source)
at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source)
at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source)
at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source)
at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor714.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.denodo.internal.o.a.r.server.ObjectRef.invoke(ObjectRef.java:110)
at com.denodo.internal.o.a.r.netty.RMIServerHandler.dispatch(RMIServerHandler.java:183)
at com.denodo.internal.o.a.r.netty.RMIServerHandler.channelRead(RMIServerHandler.java:81)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at com.denodo.internal.i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
***Information about encapsulated exception***
java.lang.Exception: Error executing hadoop command
at com.denodo.vdb.util.datamovement.TestDataMovementUtil.a3(Unknown Source)
at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.a3(Unknown Source)
at com.denodo.vdb.util.hdfs.HdfsOperationsExternalExecutableImpl.b3(Unknown Source)
at com.denodo.vdb.util.tablemanagement.sql.HadoopTableManager.testDataMovement(Unknown Source)
at com.denodo.vdb.util.introspectionservice.actions.TestHdfsDataMovementAction.execute(Unknown Source)
at com.denodo.vdb.interpreter.execution.TestHdfsDataMovementAction.exec(Unknown Source)
at com.denodo.vdb.interpreter.execution.Action.run(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.f3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.d3(Unknown Source)
at com.denodo.vdb.interpreter.execution.processor.VDBActionProcessor.a3(Unknown Source)
at com.denodo.vdb.interpreter.execution.Action.start(Unknown Source)
at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source)
at com.denodo.vdb.interpreter.execution.ExecutionEngine.execute(Unknown Source)
at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source)
at com.denodo.vdb.vdbinterface.server.QueryExecutorImpl.executeStatement(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor714.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.denodo.internal.o.a.r.server.ObjectRef.invoke(ObjectRef.java:110)
at com.denodo.internal.o.a.r.netty.RMIServerHandler.dispatch(RMIServerHandler.java:183)
at com.denodo.internal.o.a.r.netty.RMIServerHandler.channelRead(RMIServerHandler.java:81)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at com.denodo.internal.i.n.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at com.denodo.internal.i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
#
# Check the following:
# - The specified URI exists.
# - The environment variable HADOOP_USER_NAME has been defined in the system where the Virtual DataPort server runs.
# - The value of HADOOP_USER_NAME is an existing account in the database and it has privileges to list the contents of the HDFS URI.
################################################################################
STDERR > Traceback (most recent call last):
STDERR > File "/usr/local/bin/dbfs", line 11, in <module>
STDERR > sys.exit(dbfs_group())
STDERR > File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1128, in __call__
STDERR > return self.main(*args, **kwargs)
STDERR > File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1034, in main
STDERR > _verify_python_env()
STDERR > File "/usr/local/lib/python3.6/dist-packages/click/_unicodefun.py", line 100, in _verify_python_env
STDERR > raise RuntimeError("\n\n".join(extra))
STDERR > RuntimeError: Click will abort further execution because Python was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/unicode-support/ for mitigation steps.
STDERR >
STDERR > This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:
STDERR >
STDERR > export LC_ALL=C.UTF-8
STDERR > export LANG=C.UTF-8
Data movement failed"
I am sure the specified URI exists. What am I missing? I am running Denodo 8.0. Following the below:
https://community.denodo.com/docs/html/browse/latest/en/vdp/administration/bulk_data_load/databricks
Edit: I did ran the export Lc.. but it makes no difference.
