Scenarios And Recommended Uses¶
This section describes the two recommended models when using the integration with a version control system (VCS):
Distributed model. The characteristics of this model are:
- Each developer has a local installation of the Virtual DataPort server and its administration tool. This installation is under the control of the developer.
- Each developer has a local database for each project.
- This is the recommended model because managing global elements (i.e. jars and i18n maps) is easier than with the centralized model.
The section Developers Use Their Own Virtual DataPort Server describes this approach.
Centralized model. The characteristics of this model are:
- There is a single Virtual DataPort server and each developer uses its Administration Tool to connect to this Server.
- For each project, there is one central database that is managed by the project manager or the administrator.
- The administrator creates a separate database for each developer and for each project. When working on a project, each developer connects to her own database so her changes do not override the changes of the other developers unless she checks the changes into the VCS.
The section Developers Share the Same Virtual DataPort Server describes this approach.
We are going to describe these two models with an example. In this example, there is a team of users formed by:
- Mary: the administrator of all the Denodo servers.
- John: a Denodo developer who is working from Denver.
- Scott and Steve: two Denodo developers working from Madrid.
The developers will be working on two projects: project1 and project2.
There are two Virtual DataPort servers, one per location.
The policies of this company define three environments:
- Development: where John, Scott and Steve will import data sources, create views, publish Web services, etc.
- Testing: where the work developed in the development environment is tested.
- Production: where there are two Virtual DataPort servers that are accessed by third-party applications from other departments of the company.
Regardless of the model chosen for the VCS integration, the first time the team uses this feature, Mary the administrator has to perform the following steps on her Virtual DataPort server:
Enable VCS in the Server if not enabled yet. The section Virtual DataPort Server Configuration explains how to do this.
Create two VCS environments: “devel_denver” and “devel_madrid”. In this example, the environment “devel_denver” and “devel_madrid” will both be used in the development process.
We create one environment for each location where the members of the development team are working. We do this, because in each location, there is a replica of the data sources they use. I.e. there is one Oracle database in Denver and another Oracle database in Madrid.
If all the team members were accessing the same data sources, we would create only one environment.
Do not create an environment for testing or production because as we will explain in the section Promoting a Database from Development to Testing and Production, we do not recommend using VCS to promote databases between environments.
In the Server located in Denver, set the default VCS environment to “devel_denver” and for the Server located in “Madrid”, “devel_madrid”.
When a new project starts, Mary has to perform the following steps:
Create one central database for each new project. These databases will be used by the administrator to integrate the changes made by all the developers.
In this example, she creates two databases: “project1” and “project2”.
Enable the VCS integration on these two databases. To do this, open the “VCS Management” dialog and click Databases. Then, click configure for project1 and select the check box Use version control. Do the same for “project2”.
Check in the new databases on the VCS server so the developers can begin using them. To do this, on the elements tree, right-click the database “project1” and click VCS > Check in.
What the developers do next depends on the model selected regarding the use of VCS:
Developers Use Their Own Virtual DataPort Server¶
In this model, each member of the development team (John, Scott and Steve) uses his own Virtual DataPort server to work.
The diagram below describes this scenario.
Each developer must follow these steps to begin working on a project:
- Enable VCS in their own Server if not enabled yet. The section Virtual DataPort Server Configuration explains how to do this.
- Select the default environment. For John, that will be “devel_denver” and for Scott and Steve, “devel_madrid”.
- Check out the databases project1 and project2 that Mary already has created and checked in to the VCS.
After this, the developers can begin working and checking in their changes to share them with their colleagues.
Let us illustrate with an example the workflow a team of developers should follow.
John connects to the database “project1”, creates a JDBC data source against a local MySQL instance and creates a base view. Then he checks the new elements in. Note that this can be achieved in two ways:
- Checking in the entire database “project1”.
- Checking the base view in. In this case, the Server detects that the base view depends on the data source and checks it in too.
This action generates two properties files for the JDBC data source: one associated to the “devel_denver” environment (which John has selected) and a default one that will be used when a user who has selected other environment checks the JDBC data source out.
Working with Global Elements
Now, John adds a jar with custom functions to his server and creates a custom i18n map. Then, he creates a new base view “view_2” (and configures it to use the custom i18n map) and then a selection view (“p_view_2”) over the base view. The selection view uses one of the custom functions included in the jar John added before.
After this work is done, John checks “p_view_2” in. The base view “view_2”, the jar and the i18n map are also checked in, as “p_view_2” depends on them.
Scott starts working on “project1” too, creates several elements and checks them in. Then he checks the whole database out to look at John’s work. As Scott is working with the environment “devel_madrid” and John with “devel_denver”, the data source John had created is checked out into Scott’s server using the default properties.
Scott modifies the connection details of the JDBC data source so it will access a different MySQL instance and checks the data source in. Now there is a properties file for the JDBC data source associated to the environment “devel_madrid” (in addition to the one associated to “devel_denver”).
Dealing with conflicts
John makes some changes to the selection view he had created before (“p_view_2”) and checks it in.
Scott is looking at his local version of “p_view_2” and discovers a typo in the view’s description. He corrects it and tries to check the changes in, but finds out that there is a conflict due to John’s latest changes. Scott discards his changes, checks the view out (accepting to lose his local changes), corrects the description again and checks the view in.
A new developer arrives
A new developer, Steve, joins the team in Madrid. He configures his server and checks “project1” and “project2” out, obtaining all the work already done by John and Scott, so he can start working on new elements for the projects.
This time, a .properties file for the “devel_madrid” environment already exists for the JDBC data source that John had created in the first place. Because of this, when Steve checks out the database, the data source is configured with the connection details that work for him.
Promoting a Database from Development to Testing and Production¶
After the development team has finished his work, it is time for testing.
The recommended steps to promote a database from one environment to the other is the same regardless of the model selected to use the VCS integration.
To promote a database from development to testing, the administrator must follow these steps:
Connect to the central database of the project (in our example, project1 or project2) and check out the changes from the VCS.
Export to a file the entire database with the option “Export environment specific properties separately”.
Edit the generated properties file to adapt its values to the testing environment. E.g. Modify the URL of the JDBC data sources to point to the testing databases along with new credentials, etc.
On subsequent promotions of a database, the administrator will be able to use the properties file created on previous occasions.
If the administrator imports a VQL file and the properties file does not have all the properties, the import process will fail without changing anything on the Server. If this happens, the import process will return a list of the properties that are missing so the administrator can add them to the properties file.
Import the VQL file with the properties file into the testing server.
After this, the users can begin testing the views they have created.
Regarding this process, we recommend the following:
- The process to promote a database from testing to production should be the same.
- To promote changes from one environment to another (e.g. from
development to testing), we recommend:
- Promote the entire database and not just a set of views. This avoids that accidentally, some changes are not promoted.
- In the target Server, drop the existing database before importing the VQL file of the database you want to promote. This ensures that there are no elements that existed on a previous version, but that were deleted during the development of the project.
- To export and import a database from one environment to the other, we recommend developing a script that invokes the import and export scripts included in the Denodo Platform.
- To promote a database from testing to production use a VQL file and not VCS because the process of importing a database from a VCS repository takes longer than from a VQL file.
- Because of the recommendation above, we also recommend promoting databases from development to testing using a VQL file. That way, when you migrate a database from development to testing, you test that the script that migrates databases between environments works.