Delphix First Month

My company is trying out a new product called Delphix.

We have had Delphix installed for about a month and I wanted this post to contain my first impressions for others who may be considering it.  Essentially Delphix provides fast copies of databases so that a terabyte database can be refreshed in minutes.  My most recent example took three minutes for a development db that is just over one TB.

Delphix is an appliance running Open Solaris that runs in a VMWare ESX virtual machine.  At least, that is how we have it installed.  You license Delphix by the number of CPUs in the virtual machine so you can use it as much as you want as long as the CPUs can handle it.  Your database file systems are connected to the Delphix appliance using NFS.  So, the critical component of Delphix is the network connection between your target database servers and the appliance.  In our case we put new 10 gigabit ethernet hardware in to connect our HP-UX Itanium virtual machines (target database servers) to the Delphix vm.  We made sure they were all on the same IP subnet with nothing but a nice high speed switch between them all.  After some initial testing Delphix’s performance and support teams made some network configuration changes that helped with an initial performance/hang issue.  Looks like it may have something to do with the packet flow control within TCP/IP but it works fine now.

The Delphix appliance uses RMAN to pull data from a source database.  The source database has to be in archivelog mode and for best performance you need to enable block change tracking.  Delphix does an initial full level 0 backup and then regular incremental level 1 backups.  To create a new clone copy of the original database you pick one of these incremental backups – called snapshots – and point to the target machine and Delphix automatically mounts the appropriate NFS filesystems with the datafiles, tempfile, redo logs, etc. and brings up a clone of the source system with the new name you designate.

Also, Delphix takes snapshots of the clone databases – called VDBs – on a regular basis so these can be used as backups of the clones themselves and you can clone the clones – sounds like Star Wars Clone Wars doesn’t it?

One interesting challenge is space management.  Each new clone takes up very little space – until you start updating it.  I did some tests where I copied a 30 gig table to a new table and the vdb which previously took less than 100 megabytes of space now took gigabytes.  Everything is compressed so I think it was less than half of the 30 gig but the point is that the more the copies get updated the more disk space you need for the copies.  If you have an application that needs copies quickly but the copies aren’t heavily updated you can make many copies with very little disk space.  I think this kind of capability opens up all kinds of possibilities we haven’t considered before since each copy without Delphix would take up an equal amount of space as the original if you just use normal disk storage.  I.e. Before Delphix we had to minimize the number of production copies we used for development and testing.  Imagine how our processes might change if we can spin off a quick production copy, use it for a short while, and then get rid of it all with minimal additional disk space and with the clone occurring in minutes.

Right now I’m working with a Delphix consultant on a script that the developers can use to refresh their own database from the most recent snapshot of its source.  Delphix has a couple of ways it can be controlled from a Unix shell script.  One way is with ssh and their command line interface (CLI).  You can setup ssh with a public and private key so you can ssh into the VM as the “delphix admin” user and then run a set of commands in Delphix’s proprietary language.  Delphix also has a GUI but it is nice that you can run a script and do things like kick off a refresh or clone.  Also, there is a web service and they gave me a sample Python script to communicate with the VM through the web service but I haven’t delved into it.  For one thing, we don’t have Python installed on our HP-UX servers.  You can access Delphix’s documentation online here including the CLI interface commands.

Well, I’m not trying to sell Delphix to anyone but I thought it would be good to put down some of my experiences.  The database cloning and refreshing is remarkably fast.  Time will tell how the network NFS performance holds out during heavy development and testing but assuming we manage that performance and the disk space usage I’m pleased with the remarkable clone times that are supported by the Delphix VM.

– Bobby

About Bobby

I live in Chandler, Arizona with my wife and three daughters. I work for US Foods, the second largest food distribution company in the United States. I've been working as an Oracle database administrator and PeopleSoft administrator since 1994. I'm very interested in Oracle performance tuning.
This entry was posted in Uncategorized. Bookmark the permalink.

17 Responses to Delphix First Month

  1. Pete says:

    Thanks for the review, Bobby. What are the requirements for the target system? Is it an ESX host or other? Also, how does Delphix handle the network stack configuration when spinning up clones? Is this a manual procedure or do you just feed Delphix a pool of available IPs to dole out?

    Cheers

    ~Pete

    • Bobby says:

      Pete,

      In our case the target servers are HP-UX virtual machines running on a physical HP-UX blade. I guess HP has its own form of virtualization that runs under HP-UX. So, the target servers in our case are not running under ESX even though the Delphix appliance is. We setup five VM’s on the one blade and our Unix team configured their IP addresses and related settings.

      All Delphix does when it creates a new clone is mount several NFS filesystems and run the necessary sqlplus and rman commands to bring up a clone of the source database. You have to setup ssh keys for the oracle user on the target machine and set aside a small directory for the Delphix utility scripts and the Delphix appliance ssh’s into your target db server host and runs the needed commands. You have to edit the sudoers file so oracle can mount the filesystems.

      – Bobby

  2. Pingback: Virtual DB | Oracle Scratchpad

  3. Ramesh Shrinivasan says:

    Hi Bobby,
    Thanks for the posting. We are in the preliminary stage of evaluating Delphix and I was wondering if I could ping you off-line about some questions we had.

    Thanks,

  4. Peter Jobi says:

    Bobby, this is a great article!
    We have implemented Delphix,, but now looking how to automate the processes using CLI. And there is not enough information when it comes to creating scripts for provisioning, Pre and post scripts for developers to run as they provision or rollback, or refreshes — We have several VDBs we are using IBM AIX – Have you gotten any scripting for developer refreshes, creating clones/provision? Thanks Peter

    • Bobby says:

      Did you get any consulting from Delphix? Raju from Delphix created our CLI script. I got his OK to upload it to the blog. Here is a zip of the script. I changed our server name to YOURSERVERNAME so you would have to replace that with the host name for your Delphix VM. I believe that the script takes the ORACLE_SID and finds the most recent snapshot of the dSource for that vdb and does the refresh based on that snapshot.

      – Bobby

      • Peter Jobi says:

        We did get consulting – training, installation, but have not gotten any CLI training or not that am aware of. Thanks I will look in this attachment

  5. Peter Jobi says:

    Thanks, this looks similar to something got together. We are looking at getting help with scripting to auto provision and create snapshots. This is so the Developers, or any user with privileges can use to provision and take snapshots from the server and not through the Delphix GUI — I hope we something soon.. thanks again for the upload

  6. Peter Jobi says:

    How does Delphix handle wallets from encrypted Oracle database (where TDE and masking is used) to multiple VDBs
    Do we have to create multiple Oracle homes for each VDB instance? Or can we do file mapping?

    • Bobby says:

      Peter,

      I asked Travis at Delphix about this and he may come back with a better answer from within Delphix. I found this in their documentation:

      http://docs.delphix.com/display/DOCS31/Provisioning+a+VDB+from+an+Encrypted+Oracle+Database

      This sentence suggests that you have to have each VDB that is a copy of the original encrypted database on a different host. I.e. one to one relationship between vdb and wallet/host. “When provisioning a VDB from an encrypted dSource, you must provision to a target environment other than the source environment. This is because the two instances of the database, the dSource and the VDB, will attempt to access the same wallet file.” Delphix uses the term “environment” to mean the host or virtual machine where your virtual database runs. I’ve never used TDE but I know the vdb is just a clone of the source db so I would think you would have the same issue if you used RMAN to clone an encrypted database multiple times.

  7. Peter Jobi says:

    Thanks very much Bobby! I had already looked at that document, but was very detailed. We are already have a single VDB1 created from a clone and want to make several more in the same environment where the VDB1 is currently running. But am stuck on encrypted databases.
    Thanks again. Have a nice weekend!

  8. Bobby says:

    Peter,

    Raju from Delphix is telling me that you can have a different wallet for each database so you can have multiple encrypted vdbs on the same host and same Oracle home. I haven’t verified this.

    – Bobby

  9. Kyle Hailey says:

    Hi Bobby, nice write up and fun to see it’s in your top 10 posts
    Best
    Kyle
    http://kylehailey.com

  10. HI Bobby, Its was really nice to read the article. Our company is trying out this product, Can you help us in providing the details such as (Hardware requirements , integration with source and destination server) etc…. It will be great help.

    • Bobby says:

      Sudheer,

      Thanks for your question. I think that Delphix can answer the sort of questions you have. In our case Delphix provided us with scripts to run on our source and current development and test databases to figure out the number of CPUs needed for the Delphix VM and the network throughput needed. These scripts looked at the I/O activity on these systems and Delphix plugged this information into some internal spreadsheet to figure out the hardware requirements.

      Also, Delphix supplies you with a simple installation guide for connecting to your source and destination servers. It was pretty easy for me to follow.

      Nothing is more important than the network connection between your Delphix VM and the hosts for your Delphix virtual databases. Most of our issues came in this area. We setup the hosts for our vdbs on HP-UX virtual machines and this was a mistake because HP’s implementation of network virtualization was inefficient. I advise you to work closely with Delphix support as they verify your network performance and double check it yourself. You should see sub millisecond latencies and the expected throughput in both directions.

      Next, be sure the disk subsystem that supports the Delphix VM is comparable in performance to your production disk subsystem or you won’t get a fair comparison in performance tests on a Delphix vdb when compared to your production source database. I.e. you could see a query run slower on your vdb than in production if the disk system is slower on your Delphix VM than that on your production source database.

      – Bobby

Leave a Reply