Server operation
In this section we will talk about how to put in production the wrappers that we have developed in the previous sections. While in development the wrappers were tested
from within the WGT, by clicking on the Test wrapper button. In production we will deploy these wrappers in the Denodo server, which will make them appear as base views
that we can query in the same way as any other virtual view.
Environment preparation
The first step to deploy the wrappers into the Denodo server is to start both the Denodo server and the Browser pool server. We have already discussed the Denodo server in the Data Virtualization Basics tutorial so check it out for additional information.
The Browser Pool server is part of ITPilot: its responsibility is to create,
manage and destroy instances of browser windows that will be used by the wrappers to perform the required actions on web pages. So before continuing this tutorial start
the server by clicking the "play" button of either the Denodo server in the VDP tab of the Denodo Platform Control Center, or the Wrapper server in the ITP tab of the
Denodo Platform Control Center.
Both servers are actually the same server: the Wrapper server in ITP is just an alias for the Denodo server in the VDP tab. If you start the Denodo server, after a while you will see the ITP Wrapper server becomes online also (as they are actually the same server).
Start the browser pool by clicking the “play” button of the Browser Pool server in the ITP tab of the Denodo Platform Control Center.
Once both servers appear as “Started”, continue to the next section.
Deploying the wrapper in the server
The next step is to deploy the wrapper in the server. In the Wrapper Generation Tool, open any of your finished wrappers (for example, the Yahoo search wrapper),
then select the menu option Deploy > Server deploy…. This will open the Deploy dialog.
In this dialog we can specify what and how to deploy it. We can choose to deploy the currently open wrapper, to deploy a full project of wrappers or all the wrappers present in the WGT.
Select the “Create base view (with Virtual DataPort only)” option. That will enable the control for specifying what base view name we want to give the wrapper within the Denodo server. Leave it as yahoosearch.
Finally, fill out the Server URI, login and password fields. Remember that by default ITPilot is going to deploy the wrapper in the “itpilot” database (as specified in
the last part of the default Server URI: //localhost:9999/itpilot ). You should change this to the name of the virtual database where you want to put the wrapper
(for example, //localhost:9999/myvirtualdatabase).
Once all the parameters are set, click “Ok” and the wrapper will be deployed in the server.
Running the wrapper from VDP
Now that the wrapper is deployed in the server and a base view has been created, we can run it in the same way as we did for other types of datasources in the Data Virtualization Basics tutorial:
Open the VDP Administration tool and connect to the server and database where you deployed your Yahoo search wrapper.
Double-click the “yahoosearch” base view.
Click “Execute”.
The wrapper has one mandatory parameter (the search term) so we have to include a condition over that field. Specify the value of the query parameter (for example, ‘Technology'). Remember to type the value within single quotes, as this field is of type string.
Click “Execute”.
You will see that a new browser window appears and does the navigations, and at the end you will see your results appear within the Administration tool.
The browser window will not close itself when the wrapper ends its execution. In operation mode the browsers are managed by the Browser Pool server, and one of its tasks is to pool unused windows and reuse them for subsequent wrappers, so the cost of destroying and creating new windows is removed.
Integrating the web data with other sources
Finally, let's think about a simple integration of data from the web with other sources. Take the US State healthcare data wrapper that we developed here. This wrapper receives a mandatory parameter representing a state name.
We could see this wrapper becoming useful in the following scenario: we operate a web site that allows users to search healthcare providers across the US. Through the web analytics software used in the site we have a daily list of the 5 most searched US states, in CSV format.
We can now do the following:
Create a new Delimited File data source in VDP to get the most popular states data.
Create a new join between the popular states base view and the healthcare base view. Join by the state name coming from the DF source and the state name parameter in the wrapper. This will automatically create a nested join between the two sources, and during execution it will read every row from the file, then execute the wrapper once per row passing as input the US state specified in the DF.
This is one of the multiple ways of integrating the data that is coming from web sources with data that is living in other formats. The main idea is that once the wrapper is deployed in the Denodo server, it behaves as a relational table and is a first-class citizen of the virtualization server. Another example of what we could do with this kind of integration is use the Toodledo wrapper to insert in our TODO list all the items that need to be checked manually after running a Data Virtualization process (for example, cases too complex to be processed automatically and that need some human to decide what the next steps are).
Closing words
Congratulations! This lesson concludes the Denodo Platform Web Automation tutorial. We have seen how to create basic wrappers, from the first navigation recording, data extraction, pagination, variables, etc. If you want to learn more you can check the Documents section for a full reference of all the capabilities of ITPilot, and do not forget to check out our Denodo Platform Training offering for full online or face-to-face training courses that explore in-depth all the capabilities of the Denodo Platform.