Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2008 Forums
 SSIS and Import/Export (2008)
 Batch process in ssis package

Author  Topic 

avlakshmi77
Starting Member

15 Posts

Posted - 2012-01-24 : 04:28:58
Hi

I created one ssis package. checked for 1000 records. it is working fine. here i getting whole data at a time.
But in live system having lakhs of records. Retriving the data from different server. Some times it is giving 'sql time out exception'.

Some one suggested me to the use batch process.
can u suggest how to use batch process.

thanks in advance.

yosiasz
Master Smack Fu Yak Hacker

1635 Posts

Posted - 2012-01-30 : 19:18:34
do you have a key column on your source data?

1. create temp db

create TABLE [dbo].migration_temp(KeyField int not null)
GO

2. Create a processing table that holds all rows that you want to move based on a key field

create table dbo.Process(keyFieldId int NOT NULL, isProcessed bit NOT NULL default((0)))


3. Dump into Process all unique keyField values

insert into dbo.Process(keyFieldID)
SELECT keyFieldID FROm SourceTableView


4.dump batch rows into temp table

WHILE (SELECT COUNT(*) FROM dbo.Process WHERE isProcessed = 0) > 0

INSERT INTO dbo.migration_temp(KeyField)
SELECT TOP 100 KeyFieldID
FROM dbo.Process with (nolock)
WHERE isProcessed = 0



6. Batch process your code

7. Update process table

UPDATE tgt
SET tgt.isProcessed = 1
FROM dbo.Process tgt with (nolock)
inner join dbo.migration_temp src
on src.KeyFieldID= tgt.KeyFieldID



If you don't have the passion to help people, you have no passion
Go to Top of Page

visakh16
Very Important crosS Applying yaK Herder

52326 Posts

Posted - 2012-01-30 : 19:41:16
quote:
Originally posted by avlakshmi77

Hi

I created one ssis package. checked for 1000 records. it is working fine. here i getting whole data at a time.
But in live system having lakhs of records. Retriving the data from different server. Some times it is giving 'sql time out exception'.

Some one suggested me to the use batch process.
can u suggest how to use batch process.

thanks in advance.



can you explain what package is doing first?
is it moving data between servers?
are you using sql task or data flow for data transfer?

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page

avlakshmi77
Starting Member

15 Posts

Posted - 2012-02-07 : 23:35:58
Hi vishak

can you explain what package is doing first?
is it moving data between servers?
Moving data between different servers. source is from 4 servers and destination is 1 server.

are you using sql task or data flow for data transfer?
In this we are using DataFlowTask for sending data between source to destination.
For each table i am using one data flow task
If the number of records are more it is giving the error.
Go to Top of Page
   

- Advertisement -