How to refresh Test, Dev Systems with Prod Data?

When I attended PASS Summit last October, there were many people whom I met during breakfast and lunch and sometimes in the lobby and we discussed a variety of topics on  SQL Server. During one of those conversation, I was asked to give my suggestion on how to refresh their Development and Test environments with the Production data. So I thought it would be nice to write the same, so that it might help some of those who are looking for the same info. 

Backup & Restore

One of the most common mechanism of refreshing environments is using database backups on the production server and restore on the DEV, QA server to refresh the existing one. The advantage of using backup / restore is it is easy to do it. You don’t have to perform any special steps in this procedure other than restoring, coz your most recent full backup of production database should be good. However, the downside of this is that your DEV / QA database would be the same size of the production and your DEV / QA machine might not be anywhere near the size of your PROD box. The disk space too might be a factor to consider when using this procedure.

SQL Server Integration Services (SSIS)

If you are aware and comfortable using SSIS packages, you can utilize this platform to refresh and  make other customization after your DEV / QA database is refreshed. With SSIS, you can automate the database refresh and setup a SQL Agent job and when you have to refresh you can run the job manually (or on a pre-determined schedule).. Some of the benefits using SSIS is the manual tasks that might be needed after you refresh the data, such as scrubbing off critical data (such as SSNs or Credit Card data etc), adding additional Developer / Test users etc.. You might also not want all of the data from your production databases, only the most recent year’s worth of data is good to test, in such cases backup restore will not solve the purpose, the data that is not necessary has to be dropped after restoring the database backup…

Import Export Wizard

Import Export Wizard would be another handy tool that can be used for data refresh, using this tool does not need any other program (such BIDS or SSDT as in the case of SSIS). You can pick only those table(s) that needed in the DEV / QA environment.. The disadvantage of this tool is that you cannot customize beyond a certain level and scheduling is not an option unless you save the SSIS package at the end of this wizard.. Also, you cannot push data simultaneously to 2 different databases such as DEV and QA at the same time (where as in SSIS you can do so using Multi-Cast Data Transformation).. 

Other options

Finally there are other options such as Scripting the database objects and transferring the data using stored procedures at either the source or destination (which involves a linked server between the source and destination servers) and / or command line utility called Bcp. 

Overall, I would prefer SSIS as my choice to refresh QA / Dev databases..

Do you like this site? Like our FB page @ Facebook.com\LearnSQLWithBru so that, you know when there is a new blog post.

–Bru Medishetty

SSIS – Creating a new folder and deploying package in that folder

In this blog we shall learn how to create a folder in MSDB using SSMS and deploying a package in that folder. This would be beneficial when there are a large number of SSIS packages deployed on your SSIS Server. Having separate folders enables you to keep all SSIS packages related to a specific group / department as one group and makes it easier for managing them.

Creating a folder

First connect to the SSIS instance using SSMS. Expand the node Stored Packages and then expand MSDB. The below screen shot displays such an example. Note that Accounting folder is expanded to display the list of folders currently available. We would be creating a new folder in Accounting.

On the folder under which you would like to create a new folder (in this example Accounting folder) right-click and choose New Folder from the pop-up menu.

Create New Folder dialog box is displayed, when you can enter the name of the folder that we would like to create. Click OK to create the folder.

Once the folder is created, you can check in SSMS if you see the folder name under the parent folder.

Deploying the package in a folder

Once you are done with creating the folder, then you can start deploying your SSIS packages to the new folder. For step by step instructions on how to deploy SSIS packages, please read one of my earlier blogs here. In this blog we shall only look at the step that matters.

When deploying the package, in the step where you have to choose the SQL Server Target where the SSIS package is to be deployed, type the SQL Server name and then under Package Path, click the ellipses button to display the folder structure of that SQL Server and navigate to the destination folder.

Select that folder and click OK..  

Click Next and continue deployment of the SSIS package.

Do you like this site? Like our FB page @ Facebook.com\LearnSQLWithBru so that, you know when there is a new blog post

— Bru Medishetty

List of SSIS Error Messages

Today, when I was working on SSIS packages, I got an error which as often in SSIS is not clear and descriptive. I searched online for the description and explanation and found this link in MSDN. http://msdn.microsoft.com/en-us/library/ms345164.aspx

The page contions all messages in SSIS (be it Error or Warning or Informational etc). It contains the error cumber and what does it mean.. Hope it helps you too in finding the explanation/ description for your error/ warning code.

Do you like this site? Like our FB page @ Facebook.com\LearnSQLWithBru so that, you know when there is a new blog post.

– Bru Medishetty