Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
sschrupp
Starting Member
7 Posts |
Posted - 2010-11-29 : 17:07:44
|
Hi everyone!I'm pretty new to SQL Server. Company is making me learn as I go. Whee!Here's my scenario. User provides a spreadsheet with columns "Team", "Timing", "L1", "L2", "L1_Exclude", "Field1", "Value1", "Field1_Exclude", "Value1_Exclude"... plus more of course. I then have to take that info and build a query depending on data in the different fields or lack thereof.L1 and L2 are normal criteria if they contain data. L1_Exclude would be a NOT criteria. Field1 and Value1 are dynamic criteria if data has been given and then the related excludes are NOT criteria if data is given.In Field1 will be the name of a column that's in one of our tables, and then Value1 will be a value that would be found in that column. So if Field1 on the spreadsheet = "Program" and Value1 = "Widget" then the query would need to have as part of the WHERE statement WHERE [Program] = 'Widget' AND...That Field1 could be one of maybe 20 different column names and likewise for the Field1_Exclude.So being new the only thing I could think of is to use a Cursor and dynamically build a query based on the supplied criteria, run the query, then go on to the next record of criteria.This works. But slow of course. After 40 minutes we got through around 200 records of my criteria records yielding around 6 million records inserted in another table. I've been told I should expect around 600 records each month to go through like this.Is there some way to deal with this that would be quicker?Here is what I have now for code:DECLARE NonReportable CURSOR FORSELECT PSME.Import_NonReportable.*FROM PSME.Import_NonReportableOPEN NonReportableFETCH NEXT FROM NonReportable INTO @reporting_date, @team, @process, @count, @timing, @percentage, @L1, @L2, @L3, @L4, @L1_exclude, @L2_exclude, @L3_exclude, @L4_exclude, @field1, @value1, @field2, @value2, @field3, @value3, @field1_exclude, @value1_exclude, @process_typeWHILE @@FETCH_STATUS = 0 BEGIN --build the criteria part of query SET @where_string = '(L1 <> ''Region1'' AND L1 <> ''Region2'') AND (L2 <> ''Widget1'' AND L2 <> ''Widget2'') AND (L3 <> ''Widget3'')' IF @L1 <> 'All' AND @L1 <> '' AND NOT(@L1 IS NULL) SET @where_string = @where_string + ' AND L1 = ''' + @L1 + '''' IF @L2 <> '' AND NOT(@L2 IS NULL) SET @where_string = @where_string + ' AND L2 = ''' + @L2 + '''' IF @L3 <> '' AND NOT(@L3 IS NULL) SET @where_string = @where_string + ' AND L3 = ''' + @L3 + '''' IF @L4 <> '' AND NOT(@L4 IS NULL) SET @where_string = @where_string + ' AND L4 = ''' + @L4 + '''' IF @L1_exclude <> '' AND NOT(@L1_exclude IS NULL) SET @where_string = @where_string + ' AND L1 <> ''' + @L1_exclude + '''' IF @L2_exclude <> '' AND NOT(@L2_exclude IS NULL) SET @where_string = @where_string + ' AND L2 <> ''' + @L2_exclude + '''' IF @L3_exclude <> '' AND NOT(@L3_exclude IS NULL) SET @where_string = @where_string + ' AND L3 <> ''' + @L3_exclude + '''' IF @L4_exclude <> '' AND NOT(@L4_exclude IS NULL) SET @where_string = @where_string + ' AND L4 <> ''' + @L4_exclude + '''' IF @field1 <> '' AND NOT(@field1 IS NULL) SET @where_string = @where_string + ' AND [' + @field1 + '] = ''' + @value1 + '''' IF @field2 <> '' AND NOT(@field2 IS NULL) SET @where_string = @where_string + ' AND [' + @field2 + '] = ''' + @value2 + '''' IF @field1_exclude <> '' AND NOT(@field1_exclude IS NULL) SET @where_string = @where_string + ' AND [' + @field1_exclude + '] <> ''' + @value1_exclude + '''' --insert results into NonReportable history SELECT @query_string = 'INSERT INTO PSME.History_NonReportable ' + 'SELECT ''' + @reporting_date + ''', ''' + @process + ''', ''0'', ''0'', NULL, NULL, NULL,''' + @team + ''', Idms_Region, PMS_System, DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse, NULL, Operation, Program, Program_Segment, Program_Subsegment, Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category, Syn_Sec, NULL, NULL, Risk_Segment, Count(AcctSchd), ''' + @process_type + ''' FROM PSME.Reporting_AcctWalk_CurrentMonth WHERE ' + @where_string + ' GROUP BY Idms_Region, PMS_System, DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse, Operation, Program, Program_Segment, Program_Subsegment, Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category, Syn_Sec, Risk_Segment' EXEC (@query_string) --get total count of accounts for timing SELECT @totalcount = COUNT(CountOfAcctSchd) FROM PSME.History_NonReportable WHERE CountTiming = 0 --update count and timing UPDATE PSME.History_NonReportable SET CountTiming = @count * @percentage * CountOfAcctSchd / @totalCount, Time_Per = @count * @timing * @percentage * CountOfAcctSchd / @totalCount FROM PSME.History_NonReportable WHERE CountTiming = 0 --fetch next from nonreportable FETCH NEXT FROM NonReportable INTO @reporting_date, @team, @process, @count, @timing, @percentage, @L1, @L2, @L3, @L4, @L1_exclude, @L2_exclude, @L3_exclude, @L4_exclude, @field1, @value1, @field2, @value2, @field3, @value3, @field1_exclude, @value1_exclude, @process_type ENDCLOSE NonReportableDEALLOCATE NonReportable Be gentle, I'm still trying to learn.. hehe.Thanks in advance! |
|
|
TimSman
Posting Yak Master
127 Posts |
Posted - 2010-11-29 : 17:39:02
|
| So, they give you a spreadsheet, where each row represents a specific query, with each column holding the fields/values they want to use?Could you dump the spreadsheet into a table, and then build a query by stringing stuff together into a dynamic sql statement? |
 |
|
|
dataguru1971
Master Smack Fu Yak Hacker
1464 Posts |
Posted - 2010-11-29 : 18:01:38
|
You could probably do something like this, if you are stuck with dynamic SQL..essentially build the query based on parameters passed...not pretty, but way better than a cursor.Create proc pNastyAssDynamicQuery ( @reporting_date, @team varchar(20), @process varchar(20), @count varchar(20), @timing varchar(20), @percentage varchar(20), @L1 varchar(20), @L2 varchar(20), @L3 varchar(20), @L4 varchar(20), @L1_exclude varchar(20), @L2_exclude varchar(20), @L3_exclude varchar(20), @L4_exclude varchar(20), @field1 varchar(20), @value1 varchar(20), @field2 varchar(20), @value2 varchar(20), @field3 varchar(20), @value3 varchar(20), @field1_exclude varchar(20), @value1_exclude varchar(20), @process_type varchar(20))ASDeclare @Where_Clause varchar(8000)Select @Where_Clause = ' 1=1' + 'L1 not in (''Region1'',''Region2'') + AND L2 not in(''Widget1'',''Widget2'') + AND L3 != ''Widget3'' 'IF NOT isNull(@L1,'') and @L1 != 'All' Select @Where_Clause = 'AND ' + @Where_Clause + char(39) + @L1 + char(39)IF NOT isNull(@L2,'') Select @Where_Clause = 'AND ' + @Where_Clause + char(39) + @L2 + char(39)---etcSELECT @query_string = 'INSERT INTO PSME.History_NonReportable ' + 'SELECT ''' + @reporting_date + ''', ''' + @process + ''', ''0'', ''0'', NULL, NULL, NULL,''' + @team + ''', Idms_Region, PMS_System, DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse, NULL, Operation, Program, Program_Segment, Program_Subsegment, Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category, Syn_Sec, NULL, NULL, Risk_Segment, Count(AcctSchd), ''' + @process_type + ''' FROM PSME.Reporting_AcctWalk_CurrentMonth WHERE ' + @where_string + ' GROUP BY Idms_Region, PMS_System, DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse, Operation, Program, Program_Segment, Program_Subsegment, Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category, Syn_Sec, Risk_Segment' Poor planning on your part does not constitute an emergency on my part. |
 |
|
|
sschrupp
Starting Member
7 Posts |
Posted - 2010-11-30 : 08:34:44
|
Yeah, I have the spreadsheet in a table (Import_NonReportable). I just couldn't think of a way to use a query against the table considering column names are a mystery.dataguru1971, I see that you're doing what I have in the cursor except having it as a stored procedure. So I'd do:EXEC pNastyAssDynamicQuery @blah, @blah, @blah, ... But how would I go about executing the proc for each record in my criteria riddled table (Import_NonReportable)? That's why I went with the cursor because it's the only way my limited knowledge allows me to run through a table's records.I mean the proc runs a query based on the criteria records in Import_NonReportable, but how do I run a query on Import_NonReportable that sends the data to the proc? Also, what is the 1 = 1 bit about?Thanks! |
 |
|
|
dataguru1971
Master Smack Fu Yak Hacker
1464 Posts |
Posted - 2010-11-30 : 12:49:00
|
In some cases, you may not need any additional where criteria, and the 1=1 would guarantee at least 1 condition was always present. It looks like you may not need that one.Yes, it is the same as cursor, but you could use openrowset or openquery to iterate through the rows in excel from...the procedure could be called from the VBA where you can pass the paramaters row by row. Poor planning on your part does not constitute an emergency on my part. |
 |
|
|
sschrupp
Starting Member
7 Posts |
Posted - 2010-11-30 : 13:45:38
|
| Unfortunately we don't have access to openrowset or openquery. I can call a package that imports data into SQL Server and then work within SQL Server just fine, but that's the only interaction we're allowed with any outside sources.Hang on.. you can run stored procedures from VBA? I know how to run queries but after tons of searching I could never figure out if it was possible to run stored procedures. I've resorted to calling batch files that call packages that call procedures. If you can tell me how to call a procedure and pass in parameters I'd build a shrine in your name with lots of candles and feathers! And is it possible to call packages as well rather than me calling a batch file that does it?You might have just made my year if you're suggesting what I think you are...And now that you have me thinking in that direction and I stare at my code to call queries is it by chance "cmd.CommandText = whateverYouWantToDoInsteadOfOnlySQL"?? Huge face to palm action if it's that simple. |
 |
|
|
dataguru1971
Master Smack Fu Yak Hacker
1464 Posts |
Posted - 2010-11-30 : 17:13:53
|
quote: And now that you have me thinking in that direction and I stare at my code to call queries is it by chance "cmd.CommandText = whateverYouWantToDoInsteadOfOnlySQL"?? Huge face to palm action if it's that simple.
Provided your user id has permissions.. Pretty much like that. The VBA part is not hard. As far as executing DTSX...that is do-able as well. I did it from a few C# applications I wrote, but not from VBA. I can't image it is that much different, but you could execute T-SQL using [url=http://msdn.microsoft.com/en-us/library/ms175046.aspx]sp_cmdshell[/url] Poor planning on your part does not constitute an emergency on my part. |
 |
|
|
sschrupp
Starting Member
7 Posts |
Posted - 2010-12-02 : 09:16:49
|
| Thank you so much for the help! I'll have to play around with these ideas a bit today. |
 |
|
|
sschrupp
Starting Member
7 Posts |
Posted - 2010-12-03 : 09:42:21
|
Thanks again for the help. I've got it working fine now. Although it's still fairly slow, but I guess that's expected since these are dynamic queries being run.In case anyone else has a similar scenario they need help with this is how I call a stored procedure with parameters using Excel.Sub CallProcExample() 'create empty parameters With cmd .Parameters.Append .CreateParameter("@reporting_date", adDBDate, adParamInput, , "1/1/2001") .Parameters.Append .CreateParameter("@team", adWChar, adParamInput, 50, "") .Parameters.Append .CreateParameter("@process", adWChar, adParamInput, 150, "") .Parameters.Append .CreateParameter("@process_type", adWChar, adParamInput, 50, "") .Parameters.Append .CreateParameter("@count", adDouble, adParamInput, , 0) 'etc. etc. End With 'set values for parameters Do Until Cells(row, 1).value = "" reportDate = Cells(row, dateCol).value Set param = cmd.Parameters.Item("@reporting_date") param.value = reportDate team = Cells(row, teamCol).value Set param = cmd.Parameters.Item("@team") param.value = team process = Cells(row, processCol).value Set param = cmd.Parameters.Item("@process") param.value = process processType = Cells(row, processTypeCol).value Set param = cmd.Parameters.Item("@process_type") param.value = processType count = Cells(row, countCol).value Set param = cmd.Parameters.Item("@count") param.value = count 'etc. etc. procString = "dbo.Update_NonReportable_Test" Call ExecuteProcExample(procString) row = row + 1 LoopEnd SubSub ExecuteProcExample(procedureName As String) Set cmd.ActiveConnection = cn cmd.CommandText = procedureName cmd.CommandType = adCmdStoredProc cmd.ExecuteEnd SubAnd then the procedure similar to what Dataguru1971 has above.Getting those parameters to work took a lot of experimentation and research. Definitely learned a lot of new stuff with this approach!In the back of my mind I'm thinking there's probably a way to streamline the setting of parameter values using some sort of loop and function but I'll probably have to come back to that when I have more time. |
 |
|
|
|
|
|
|
|