| Author |
Topic |
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-13 : 15:30:46
|
| Hi,I have asked this question on the MSDN forumswith no response as yet. Maybe you guys canshed some light on this:[url]http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2826915&SiteID=1[/url]We have an apparent File Handle leak occurring in our SQL server.Last weekend the server ran out of non-paged pool space:Event ID 2019 -The server was unable to allocate from the systemnonpaged pool because the pool was empty.In the process monitor the sqlservr.exe process had Mem Usageof 503,528KB and a handle count of 3,306,263 (!). Using poolmonthe highest usage of the non-paged pool was for the Muta tag, with211,610,768 Bytes. Restarting the sql service fixes thistemporarily, but already we are back to 427,000 handles forsqlservr.exe and climbing steadily at about 200,000 handlesper day. Is there a fix for this? Cheers,GeoffPSWe have:Microsoft SQL Server 2005 - 9.00.3054.00 (Intel X86)Mar 23 2007 16:28:52 Copyright (c) 1988-2005 Microsoft Corporation Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2) (Meaning Windows Server 2003 R2 SP2 :-) ) |
|
|
recontrasalo
Starting Member
12 Posts |
Posted - 2008-02-14 : 13:48:31
|
| How many CPUs & memory are there on this server? |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-14 : 13:56:51
|
| Hi Recontrasalo,It's a Dell PE 2900 with 2 quad core Intel E5310 CPUs.The system has 2GB of RAMCheers,Geoff |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-14 : 20:06:33
|
do you have any CLR procs or extended stored procs on this instance? Perhaps there's an extended proc that's not releasing resources. similar thing could possibly happen with a CLR proc/trigger/udf/etc but I haven't seen that. elsasoft.org |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-15 : 10:05:01
|
Hi Jezemine,Thanks for the suggestion!Yes, there is one extended store procedure used forGoldmine 7 (gm7sql.dll). How can I find out if it isthis xp that is at fault?Not sure what a CLR proc is. Can you point me to adescription of that?Cheers,GeoffPS Now at 700,000 file handles. That about 1/4the way to a Non-paged pool crunch. |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-15 : 12:13:10
|
quote: Originally posted by glt101 Hi Jezemine,Thanks for the suggestion!Yes, there is one extended store procedure used forGoldmine 7 (gm7sql.dll). How can I find out if it isthis xp that is at fault?
stop calling it and see if the handle count stops growing?quote: Originally posted by glt101 Not sure what a CLR proc is. Can you point me to adescription of that?
CLR procs are procs implemented in managed code, such as C#. They are only available in 2005. See BOL for details.My money is on the xp though. You have to test xps very thoroughly because they run in sqlserver's process space. If one of them AV's, it will bring your server down. Or if one of them leaks, it will bring your server down slowly. You might think about contacting the vendor of the this xp and see if this is a known issue, pehaps they have an updated DLL. elsasoft.org |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-15 : 12:26:38
|
| Hi Jezemine,Hmmm. Not really possible to stop the xp running:production server, yada, yada, yada...Isn't there some way in SQL Server to look at theresources used by the xp?My only other approach is to set up a continuoustest on our test server and watch what happens. Butthat will take more time to arrange.Cheers,Geoff |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-15 : 16:03:37
|
are you saying that your test team let this system pass through to production without catching this? I think they need a sound whipping.  elsasoft.org |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-15 : 17:28:50
|
"Test team"? Sorry. Not familiar with that concept either.Is that it BOL too? Cheers,Geoff |
 |
|
|
rmiao
Master Smack Fu Yak Hacker
7266 Posts |
Posted - 2008-02-15 : 23:48:52
|
| How did you set sql memory? What's db size? Any resource error in sql log? |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-16 : 19:47:00
|
| Hi rmiao,SQL Memory is set to the default (this machine is an SQLserver only and has only one instance), so in this casethe Microsoft SQL Server Management Studio shows2,147,483,647 for "Maximum Server memory (in MB)". Ipresume that should say "(in Bytes)" since the serverhas 2 GB of RAM :D !The DB size in the Database properties window shows:9427.13MG (General) and in the Files dialog shows:Customers, Data, PRIMARY, 6907Customers, log, N/A, 2521Since 2/11/08 there is only 1 error in the SQL Log andthat relates to a logon failure. There are no referencesto the xp dll other than one to note it is used to executethe xp.Thanks for the help!Cheers,GeoffPS Now at 800,000 handles... GULP |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-16 : 19:56:57
|
one thing you could do is profile sql server with Profiler and PerfMon at the same time. with profiler, look for calls to the xp. In perfmon, monitor the handle count. Then you can interleave the traces (I believe the profiler in 2005 supports this) and see if there appears to be a connection between when the xp is called and when the handle count goes up.I am 99% sure this leak is not the fault of sql server's relational engine. the engine goes through very extensive testing at ms, and in the real world as well through heavy usage in installations all over the world. Such a gross leak would surely have been caught and fixed by now if it was in the engine.did you contact the vendor of your xp's dll and ask them if they know about a leak? maybe they have a fix. elsasoft.org |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-16 : 21:25:22
|
| Hi jezemine,Thank for that idea. I'll try it on Monday.The xp is made by Frontrange and is part ofan extension to the Goldmine system that theyno longer offically support. However, thereare many users like ourselves who were left"high and dry" by that decision and so witheach new schema for their DB they release anew version of the dll. That way they keepusers "happy", but don't have the supportload. Nice business model eh?!Anyway. Better get off my soapbox.If I can correlate the handle count increasewith the calls in SQL Profiler, as you suggest,I might be able to cajole them into fixing thexp... ...if that's really the issue.Thanks for all the help. You guys beat MSDNforums and sqlcentral into a cocked hat asthey say.I'll post back on Monday.Cheers,Geoff |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-02-17 : 04:21:15
|
I thought .Net should handle this situation with the garbage collector? E 12°55'05.25"N 56°04'39.16" |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-17 : 10:05:57
|
GC only takes care of memory allocation on the managed heap. it does not keep track of other OS resources such as file handles, regkey handles, GDI handles, threads, etc. many .net developers are under the (incorrect) assumption that they don't have to worry about resource management in their code. they do. what they don't have to worry about as much is memory management. it is certainly possible to write an app in managed code that will bring your server down after continuous running for a few days or weeks.give this a read for more info on how to properly do resource management in managed code:http://www.bluebytesoftware.com/blog/PermaLink.aspx?guid=88e62cdf-5919-4ac7-bc33-20c06ae539ae elsasoft.org |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-02-17 : 16:21:22
|
I think I will keep my VB6.0 and internal Windows API calls for a while  E 12°55'05.25"N 56°04'39.16" |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-17 : 21:21:40
|
coming from the C++ world, I often find coding in C# frustrating because you don't have deterministic cleanup the way you do in C++. Also the fact that you have to remember to call Dispose() on all IDisposables was disgusting to me when I first learned it years ago. the "using" keyword makes this a little easier, but it would have been much better if the compiler would just add a call to Dispose() when an IDisposable goes out of scope, similar to the way a C++ compiler adds a call to the destructor when a stack-allocated object goes out of scope. bleh. rant over. elsasoft.org |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-18 : 15:00:29
|
| Hi Jezemine,Congrats. You were correct. Each call to the xp incrementsthe handle count for sqlservr.exe by 2. The call to the xpis inside at least two cursors and is used extensively bythe business logic in the Goldmine extension. Indeed eachindividual business operation increases the count by 202handles.So, now to tackle Frontrange. Thanks for the help!Cheers,Geoff |
 |
|
|
jezemine
Master Smack Fu Yak Hacker
2886 Posts |
Posted - 2008-02-18 : 17:09:59
|
ouch. Q: what's worse than using a dreaded cursor in sql server?A: using a cursor that calls a leaky xp. not only will it be slow, it will bring down your server!  elsasoft.org |
 |
|
|
eyechart
Master Smack Fu Yak Hacker
3575 Posts |
Posted - 2008-02-19 : 04:38:12
|
quote: Originally posted by jezemine ouch. Q: what's worse than using a dreaded cursor in sql server?A: using a cursor that calls a leaky xp. not only will it be slow, it will bring down your server!  elsasoft.org
maybe it would be worse if you created the cursor that execs the leaky xp using dynamic sql? but just barely :)-ec |
 |
|
|
glt101
Starting Member
28 Posts |
Posted - 2008-02-19 : 10:01:56
|
Hi folks,Just so I am not unfairly tarred by this:the cursors are in code supplied in theapplication extension.OTOH, how would one replace the cursorfunctionality anyway, if one were to needit? Here's an example of the cursor. Idon't want to get into the detailsof what all this does. Just how one wouldreplace the cursor behavior. Is there astandard idiom to use instead?OPEN fieldcursorFETCH NEXT FROM fieldcursor INTO @fieldname, @valueWHILE @@fetch_status = 0 BEGIN SELECT @tablename = convert(sysname,o.name), @datatype = convert (sysname,case when t.xusertype > 255 then t.name else d.NAME end) FROM sysobjects o, master.sys.types d, systypes t, syscolumns c LEFT OUTER JOIN syscomments m on c.cdefault = m.id AND m.colid = 1 WHERE o.name in ('CONTACT1','CONTACT2') AND o.id = c.id AND t.xtype = system_type_id AND o.type <> 'P' AND c.xusertype = t.xusertype AND c.name like @fieldname IF @fieldname = 'MERGECODES' SELECT @tablename = CASE WHEN @gmversion = '4' AND @tablename IS NOT NULL THEN 'CONTACT2' WHEN @gmversion <> '4' OR @gmversion IS NULL THEN 'CONTACT1' ELSE NULL END IF UPPER(@tablename) = 'CONTACT1' BEGIN IF UPPER(@fieldname) = 'OWNER' SELECT @value = UPPER(@value) SELECT @setcmd1 = ' ' + @fieldname + ' = ' + CASE WHEN UPPER(@datatype) NOT IN ('FLOAT', 'INT', 'REAL','NUMERIC','SMALLINT','TINYINT','DECIMAL') AND @value IS NOT NULL THEN '"' + @value + '"' WHEN @value IS NULL THEN 'NULL' ELSE @value END + CASE WHEN (UPPER(@fieldname) IN ('COMPANY','CONTACT','LASTNAME', 'CITY','KEY1','KEY2','KEY3', 'KEY4','KEY5') OR (UPPER(@fieldname) IN ('STATE', 'COUNTRY') AND @gmversion <> 4)) AND @value IS NOT NULL THEN ', u_' + @fieldname + ' = "' + UPPER(@value) + '"' WHEN (UPPER(@fieldname) IN ('COMPANY','CONTACT','LASTNAME', 'CITY','KEY1','KEY2','KEY3', 'KEY4','KEY5') OR (UPPER(@fieldname) IN ('STATE', 'COUNTRY') AND @gmversion <> 4)) AND @value IS NULL THEN ', u_' + @fieldname + ' = ""' ELSE "" END + ' WHERE recid = ' + CASE WHEN @recid LIKE '%''%' THEN '"' + @recid + '"' WHEN @recid LIKE '%"%' THEN '''' + @recid + '''' ELSE '''' + @recid + '''' END EXEC('UPDATE contact1 SET ' + @setcmd1) EXEC GMW_UpdateSyncLog 'contact1', @recid, @user, 'U', @fieldname END IF UPPER(@tablename) = 'CONTACT2' BEGIN /*this block of accountno and recid steps should only occur once the most*/ IF @accountno IS NULL SELECT @accountno = accountno FROM contact1 WHERE recid = @recid IF @c2recid IS NULL SELECT @C2recid = recid FROM contact2 WHERE accountno = @accountno IF @c2recid IS /*STILL*/ NULL --there is no contact2 record for this contact BEGIN /*get a recid*/ EXEC master..xp_GMNewRecid @user, @c2recid OUTPUT INSERT contact2 (accountno, recid) VALUES (@accountno, @c2recid) EXEC GMW_UpdateSyncLog 'contact2', @c2recid, @user, 'N' END SELECT @setcmd1 = ' ' + @fieldname + ' = ' + CASE WHEN UPPER(@datatype) NOT IN ('FLOAT', 'INT', 'REAL','NUMERIC','SMALLINT','TINYINT','DECIMAL') AND @value IS NOT NULL THEN '"' + @value + '"' WHEN @value IS NULL THEN 'NULL' ELSE @value END + ' WHERE recid = ' + CASE WHEN @c2recid LIKE '%''%' THEN '"' + @c2recid + '"' WHEN @c2recid LIKE '%"%' THEN '''' + @c2recid + '''' ELSE '''' + @c2recid + '''' END EXEC('UPDATE contact2 SET ' + @setcmd1) EXEC GMW_UpdateSyncLog 'contact2', @c2recid, @user, 'U', @fieldname END SELECT @tablename = NULL FETCH NEXT FROM fieldcursor INTO @fieldname, @value END CLOSE fieldcursorDEALLOCATE fieldcursorCheers,Geoff |
 |
|
|
Next Page
|