Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
whitmoj
Yak Posting Veteran
68 Posts |
 Posted - 2012-10-18 : 05:52:01
|
| Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any helpWhitmojIf I have inspired one person today then my job is done. |
|
|
theboyholty
Posting Yak Master
226 Posts |
Posted - 2012-12-19 : 06:16:56
|
| One approach to this could be to load each table into SQL and put it a stage table (truncated before each file is imported), then run a separate routine to import the data from the stage table to the main table. At this stage you could do a check to see if the file has been imported previously, and if it has then do nothing.Its probably not the most elegant solution but it should do the job.---------------------------------------------------------------------------------http://www.mannyroadend.co.uk A Bury FC supporters website and forum |
 |
|
|
nigelrivett
Master Smack Fu Yak Hacker
3385 Posts |
Posted - 2012-12-19 : 06:52:19
|
| Two approaches - I use bothHave a control table which contains the file name and status.One task looks at the folder and populates with the file names.Another task then gets filenames from that table to load and updates the status.That means that to reload a file you just need to update the status in the table - especially useful for testing.Separating the tasks means that you can poll more frequently with the first task and keep a record of when the files arrive.Another option is to move/rename the file when it is processed.As I said I always do both. Process all files in a folder using the control table and move to an archive folder when complete.The control table and status can also be used to split the processing into multiple small steps which is especially handy for troubleshooting and restarting processes that have failed.==========================================Cursors are useful if you don't know sql.SSIS can be used in a similar way.Beer is not cold and it isn't fizzy. |
|
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2012-12-19 : 07:46:25
|
quote:
Originally posted by whitmoj
Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any helpWhitmojIf I have inspired one person today then my job is done.
Have a procedure which takes filename as parameter and returns a boolean result to indicate file exists or not.Populate a boolean variable created in SSIS with boolean result. Use precedence constraint with expression and constraint option and add condition @FileExists == false and connect it to process and subsequent steps. this will make sure it ignores existing filessee similar logic herehttp://visakhm.blogspot.in/2012/05/package-to-implement-daily-processing.htmlonly difference is to check for existence of file instead of pattern match in above example------------------------------------------------------------------------------------------------------SQL Server MVPhttp://visakhm.blogspot.com/ |
|
|
|
|
|
|
SSIS