Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 General SQL Server Forums
 New to SQL Server Programming
 Variable column names put in query. CURSOR best?

Author  Topic 

sschrupp
Starting Member

7 Posts

Posted - 2010-11-29 : 17:07:44
Hi everyone!

I'm pretty new to SQL Server. Company is making me learn as I go. Whee!

Here's my scenario. User provides a spreadsheet with columns "Team", "Timing", "L1", "L2", "L1_Exclude", "Field1", "Value1", "Field1_Exclude", "Value1_Exclude"... plus more of course. I then have to take that info and build a query depending on data in the different fields or lack thereof.

L1 and L2 are normal criteria if they contain data. L1_Exclude would be a NOT criteria. Field1 and Value1 are dynamic criteria if data has been given and then the related excludes are NOT criteria if data is given.

In Field1 will be the name of a column that's in one of our tables, and then Value1 will be a value that would be found in that column. So if Field1 on the spreadsheet = "Program" and Value1 = "Widget" then the query would need to have as part of the WHERE statement WHERE [Program] = 'Widget' AND...

That Field1 could be one of maybe 20 different column names and likewise for the Field1_Exclude.

So being new the only thing I could think of is to use a Cursor and dynamically build a query based on the supplied criteria, run the query, then go on to the next record of criteria.

This works. But slow of course. After 40 minutes we got through around 200 records of my criteria records yielding around 6 million records inserted in another table. I've been told I should expect around 600 records each month to go through like this.

Is there some way to deal with this that would be quicker?

Here is what I have now for code:

DECLARE NonReportable CURSOR FOR

SELECT PSME.Import_NonReportable.*
FROM PSME.Import_NonReportable

OPEN NonReportable

FETCH NEXT FROM NonReportable INTO @reporting_date, @team, @process, @count, @timing, @percentage,
@L1, @L2, @L3, @L4, @L1_exclude, @L2_exclude, @L3_exclude,
@L4_exclude, @field1, @value1, @field2, @value2, @field3, @value3,
@field1_exclude, @value1_exclude, @process_type

WHILE @@FETCH_STATUS = 0
BEGIN
--build the criteria part of query
SET @where_string = '(L1 <> ''Region1'' AND L1 <> ''Region2'')
AND (L2 <> ''Widget1'' AND L2 <> ''Widget2'') AND (L3 <> ''Widget3'')'

IF @L1 <> 'All' AND @L1 <> '' AND NOT(@L1 IS NULL)
SET @where_string = @where_string + ' AND L1 = ''' + @L1 + ''''

IF @L2 <> '' AND NOT(@L2 IS NULL)
SET @where_string = @where_string + ' AND L2 = ''' + @L2 + ''''

IF @L3 <> '' AND NOT(@L3 IS NULL)
SET @where_string = @where_string + ' AND L3 = ''' + @L3 + ''''

IF @L4 <> '' AND NOT(@L4 IS NULL)
SET @where_string = @where_string + ' AND L4 = ''' + @L4 + ''''

IF @L1_exclude <> '' AND NOT(@L1_exclude IS NULL)
SET @where_string = @where_string + ' AND L1 <> ''' + @L1_exclude + ''''

IF @L2_exclude <> '' AND NOT(@L2_exclude IS NULL)
SET @where_string = @where_string + ' AND L2 <> ''' + @L2_exclude + ''''

IF @L3_exclude <> '' AND NOT(@L3_exclude IS NULL)
SET @where_string = @where_string + ' AND L3 <> ''' + @L3_exclude + ''''

IF @L4_exclude <> '' AND NOT(@L4_exclude IS NULL)
SET @where_string = @where_string + ' AND L4 <> ''' + @L4_exclude + ''''

IF @field1 <> '' AND NOT(@field1 IS NULL)
SET @where_string = @where_string + ' AND [' + @field1 + '] = ''' + @value1 + ''''

IF @field2 <> '' AND NOT(@field2 IS NULL)
SET @where_string = @where_string + ' AND [' + @field2 + '] = ''' + @value2 + ''''

IF @field1_exclude <> '' AND NOT(@field1_exclude IS NULL)
SET @where_string = @where_string + ' AND [' + @field1_exclude + '] <> ''' + @value1_exclude + ''''

--insert results into NonReportable history
SELECT @query_string = 'INSERT INTO PSME.History_NonReportable ' +
'SELECT ''' + @reporting_date + ''', ''' + @process + ''', ''0'',
''0'', NULL, NULL, NULL,''' +
@team + ''', Idms_Region, PMS_System,
DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse,
NULL, Operation, Program, Program_Segment, Program_Subsegment,
Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category,
Syn_Sec, NULL, NULL, Risk_Segment, Count(AcctSchd), ''' +
@process_type + '''
FROM PSME.Reporting_AcctWalk_CurrentMonth
WHERE ' + @where_string +
' GROUP BY Idms_Region, PMS_System,
DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse,
Operation, Program, Program_Segment, Program_Subsegment,
Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category,
Syn_Sec, Risk_Segment'

EXEC (@query_string)

--get total count of accounts for timing
SELECT @totalcount = COUNT(CountOfAcctSchd)
FROM PSME.History_NonReportable
WHERE CountTiming = 0

--update count and timing
UPDATE PSME.History_NonReportable
SET CountTiming = @count * @percentage * CountOfAcctSchd / @totalCount,
Time_Per = @count * @timing * @percentage * CountOfAcctSchd / @totalCount
FROM PSME.History_NonReportable
WHERE CountTiming = 0

--fetch next from nonreportable
FETCH NEXT FROM NonReportable INTO @reporting_date, @team, @process, @count, @timing, @percentage,
@L1, @L2, @L3, @L4, @L1_exclude, @L2_exclude, @L3_exclude,
@L4_exclude, @field1, @value1, @field2, @value2, @field3, @value3,
@field1_exclude, @value1_exclude, @process_type
END

CLOSE NonReportable
DEALLOCATE NonReportable


Be gentle, I'm still trying to learn.. hehe.

Thanks in advance!

TimSman
Posting Yak Master

127 Posts

Posted - 2010-11-29 : 17:39:02
So, they give you a spreadsheet, where each row represents a specific query, with each column holding the fields/values they want to use?

Could you dump the spreadsheet into a table, and then build a query by stringing stuff together into a dynamic sql statement?
Go to Top of Page

dataguru1971
Master Smack Fu Yak Hacker

1464 Posts

Posted - 2010-11-29 : 18:01:38
You could probably do something like this, if you are stuck with dynamic SQL..essentially build the query based on parameters passed...not pretty, but way better than a cursor.


Create proc pNastyAssDynamicQuery (
@reporting_date
, @team varchar(20)
, @process varchar(20)
, @count varchar(20)
, @timing varchar(20)
, @percentage varchar(20)
, @L1 varchar(20)
, @L2 varchar(20)
, @L3 varchar(20)
, @L4 varchar(20)
, @L1_exclude varchar(20)
, @L2_exclude varchar(20)
, @L3_exclude varchar(20)
, @L4_exclude varchar(20)
, @field1 varchar(20)
, @value1 varchar(20)
, @field2 varchar(20)
, @value2 varchar(20)
, @field3 varchar(20)
, @value3 varchar(20)
, @field1_exclude varchar(20)
, @value1_exclude varchar(20)
, @process_type varchar(20)
)
AS

Declare @Where_Clause varchar(8000)

Select @Where_Clause = ' 1=1' +
'L1 not in (''Region1'',''Region2'') +
AND L2 not in(''Widget1'',''Widget2'') +
AND L3 != ''Widget3'' '

IF NOT isNull(@L1,'') and @L1 != 'All'
Select @Where_Clause = 'AND ' + @Where_Clause + char(39) + @L1 + char(39)

IF NOT isNull(@L2,'')
Select @Where_Clause = 'AND ' + @Where_Clause + char(39) + @L2 + char(39)

---etc


SELECT @query_string = 'INSERT INTO PSME.History_NonReportable ' +
'SELECT ''' + @reporting_date + ''', ''' + @process + ''', ''0'',
''0'', NULL, NULL, NULL,''' +
@team + ''', Idms_Region, PMS_System,
DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse,
NULL, Operation, Program, Program_Segment, Program_Subsegment,
Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category,
Syn_Sec, NULL, NULL, Risk_Segment, Count(AcctSchd), ''' +
@process_type + '''
FROM PSME.Reporting_AcctWalk_CurrentMonth
WHERE ' + @where_string +
' GROUP BY Idms_Region, PMS_System,
DIS_Ind, Bus_Code, L1, L2, L3, L4, Business_Segment, Inhouse,
Operation, Program, Program_Segment, Program_Subsegment,
Territory, Location, Region, Orig_Prod_Type, Curr_Prod_Type, Curr_Prod_Category,
Syn_Sec, Risk_Segment'




Poor planning on your part does not constitute an emergency on my part.

Go to Top of Page

sschrupp
Starting Member

7 Posts

Posted - 2010-11-30 : 08:34:44
Yeah, I have the spreadsheet in a table (Import_NonReportable). I just couldn't think of a way to use a query against the table considering column names are a mystery.

dataguru1971, I see that you're doing what I have in the cursor except having it as a stored procedure. So I'd do:

EXEC pNastyAssDynamicQuery @blah, @blah, @blah, ...


But how would I go about executing the proc for each record in my criteria riddled table (Import_NonReportable)? That's why I went with the cursor because it's the only way my limited knowledge allows me to run through a table's records.

I mean the proc runs a query based on the criteria records in Import_NonReportable, but how do I run a query on Import_NonReportable that sends the data to the proc?

Also, what is the 1 = 1 bit about?

Thanks!
Go to Top of Page

dataguru1971
Master Smack Fu Yak Hacker

1464 Posts

Posted - 2010-11-30 : 12:49:00
In some cases, you may not need any additional where criteria, and the 1=1 would guarantee at least 1 condition was always present. It looks like you may not need that one.

Yes, it is the same as cursor, but you could use openrowset or openquery to iterate through the rows in excel from...the procedure could be called from the VBA where you can pass the paramaters row by row.



Poor planning on your part does not constitute an emergency on my part.

Go to Top of Page

sschrupp
Starting Member

7 Posts

Posted - 2010-11-30 : 13:45:38
Unfortunately we don't have access to openrowset or openquery. I can call a package that imports data into SQL Server and then work within SQL Server just fine, but that's the only interaction we're allowed with any outside sources.

Hang on.. you can run stored procedures from VBA? I know how to run queries but after tons of searching I could never figure out if it was possible to run stored procedures. I've resorted to calling batch files that call packages that call procedures. If you can tell me how to call a procedure and pass in parameters I'd build a shrine in your name with lots of candles and feathers! And is it possible to call packages as well rather than me calling a batch file that does it?

You might have just made my year if you're suggesting what I think you are...

And now that you have me thinking in that direction and I stare at my code to call queries is it by chance "cmd.CommandText = whateverYouWantToDoInsteadOfOnlySQL"?? Huge face to palm action if it's that simple.
Go to Top of Page

dataguru1971
Master Smack Fu Yak Hacker

1464 Posts

Posted - 2010-11-30 : 17:13:53
quote:
And now that you have me thinking in that direction and I stare at my code to call queries is it by chance "cmd.CommandText = whateverYouWantToDoInsteadOfOnlySQL"?? Huge face to palm action if it's that simple.



Provided your user id has permissions.. Pretty much like that. The VBA part is not hard.


As far as executing DTSX...that is do-able as well. I did it from a few C# applications I wrote, but not from VBA. I can't image it is that much different, but you could execute T-SQL using [url=http://msdn.microsoft.com/en-us/library/ms175046.aspx]sp_cmdshell[/url]




Poor planning on your part does not constitute an emergency on my part.

Go to Top of Page

sschrupp
Starting Member

7 Posts

Posted - 2010-12-02 : 09:16:49
Thank you so much for the help! I'll have to play around with these ideas a bit today.
Go to Top of Page

sschrupp
Starting Member

7 Posts

Posted - 2010-12-03 : 09:42:21
Thanks again for the help. I've got it working fine now. Although it's still fairly slow, but I guess that's expected since these are dynamic queries being run.

In case anyone else has a similar scenario they need help with this is how I call a stored procedure with parameters using Excel.


Sub CallProcExample()
'create empty parameters
With cmd
.Parameters.Append .CreateParameter("@reporting_date", adDBDate, adParamInput, , "1/1/2001")
.Parameters.Append .CreateParameter("@team", adWChar, adParamInput, 50, "")
.Parameters.Append .CreateParameter("@process", adWChar, adParamInput, 150, "")
.Parameters.Append .CreateParameter("@process_type", adWChar, adParamInput, 50, "")
.Parameters.Append .CreateParameter("@count", adDouble, adParamInput, , 0)
'etc. etc.
End With

'set values for parameters
Do Until Cells(row, 1).value = ""
reportDate = Cells(row, dateCol).value
Set param = cmd.Parameters.Item("@reporting_date")
param.value = reportDate

team = Cells(row, teamCol).value
Set param = cmd.Parameters.Item("@team")
param.value = team

process = Cells(row, processCol).value
Set param = cmd.Parameters.Item("@process")
param.value = process

processType = Cells(row, processTypeCol).value
Set param = cmd.Parameters.Item("@process_type")
param.value = processType

count = Cells(row, countCol).value
Set param = cmd.Parameters.Item("@count")
param.value = count

'etc. etc.

procString = "dbo.Update_NonReportable_Test"

Call ExecuteProcExample(procString)

row = row + 1
Loop

End Sub
Sub ExecuteProcExample(procedureName As String)
Set cmd.ActiveConnection = cn
cmd.CommandText = procedureName
cmd.CommandType = adCmdStoredProc
cmd.Execute
End Sub


And then the procedure similar to what Dataguru1971 has above.

Getting those parameters to work took a lot of experimentation and research. Definitely learned a lot of new stuff with this approach!

In the back of my mind I'm thinking there's probably a way to streamline the setting of parameter values using some sort of loop and function but I'll probably have to come back to that when I have more time.
Go to Top of Page
   

- Advertisement -