Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
shilpash
Posting Yak Master
103 Posts |
 Posted - 2013-06-19 : 09:50:09
|
| Hi,I am facing this problem.I have a source db from ado.net which has million of rows.It takes lot of time for initial load as well and now I want to perform incremental load but it does not have date or time stamp column and also no columns which increments in a sequential order.I don't want to do lookup for million of rows.Is there any other suggestion?Thanks in advance |
|
|
James K
Master Smack Fu Yak Hacker
3873 Posts |
Posted - 2013-06-19 : 10:26:22
|
| Does the source table have a primary key? If there is, you can use that, and how efficient it is will depend on whether the PK is ever-increasing (which you said it is not), how random the inserts are etc. Regardless, that would be your best option.However, if the table is a heap, I don't see any easy way to solve the problem. It is akin to a having a basket filled with white balls, and each day a few more white balls are added and you are asked top pick out the new balls that were added.If the table is a heap and you are able to modify the source table and add an identity column to it, that will solve the problem. But you may not have access to it, and even if you do, when you alter the source table, that may cause all types of other issues that rely on that table having a fixed schema. |
 |
|
|
shilpash
Posting Yak Master
103 Posts |
Posted - 2013-06-19 : 10:41:47
|
| Yes.Table has a pk and inserts are done frequently but when inserts are performed,the pk value could be anything,its not in a sequential manner like pk last value was 100 and its not going to be 101 for new rows. |
|
|
|
James K
Master Smack Fu Yak Hacker
3873 Posts |
Posted - 2013-06-19 : 10:57:12
|
| Even if the PK is not sequential, given the scenario, that (or another unique index) is the only thing you can rely on. You could try to optimize its use - for example, by creating a bookkeeping table that has only the primary key column that would hold all the already exported primary keys and joining with that table to determine what has already been exported. |
|
|
|
|
|
|
|
|
Incremental load for million of rows