Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2005 Forums
 SQL Server Administration (2005)
 sqlservr.exe handle count of 3,306,263

Author  Topic 

glt101
Starting Member

28 Posts

Posted - 2008-02-13 : 15:30:46
Hi,

I have asked this question on the MSDN forums
with no response as yet. Maybe you guys can
shed some light on this:
[url]http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2826915&SiteID=1[/url]

We have an apparent File Handle leak occurring in our SQL server.
Last weekend the server ran out of non-paged pool space:

Event ID 2019 -The server was unable to allocate from the system
nonpaged pool because the pool was empty.

In the process monitor the sqlservr.exe process had Mem Usage
of 503,528KB and a handle count of 3,306,263 (!). Using poolmon
the highest usage of the non-paged pool was for the Muta tag, with
211,610,768 Bytes. Restarting the sql service fixes this
temporarily, but already we are back to 427,000 handles for
sqlservr.exe and climbing steadily at about 200,000 handles
per day.

Is there a fix for this?

Cheers,
Geoff

PS
We have:
Microsoft SQL Server 2005 - 9.00.3054.00 (Intel X86)
Mar 23 2007 16:28:52
Copyright (c) 1988-2005 Microsoft Corporation Enterprise Edition
on Windows NT 5.2 (Build 3790: Service Pack 2)
(Meaning Windows Server 2003 R2 SP2 :-) )

recontrasalo
Starting Member

12 Posts

Posted - 2008-02-14 : 13:48:31
How many CPUs & memory are there on this server?
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-14 : 13:56:51
Hi Recontrasalo,

It's a Dell PE 2900 with 2 quad core Intel E5310 CPUs.
The system has 2GB of RAM

Cheers,
Geoff
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-14 : 20:06:33
do you have any CLR procs or extended stored procs on this instance? Perhaps there's an extended proc that's not releasing resources. similar thing could possibly happen with a CLR proc/trigger/udf/etc but I haven't seen that.


elsasoft.org
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-15 : 10:05:01
Hi Jezemine,
Thanks for the suggestion!

Yes, there is one extended store procedure used for
Goldmine 7 (gm7sql.dll). How can I find out if it is
this xp that is at fault?

Not sure what a CLR proc is. Can you point me to a
description of that?

Cheers,
Geoff

PS Now at 700,000 file handles. That about 1/4
the way to a Non-paged pool crunch.
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-15 : 12:13:10
quote:
Originally posted by glt101

Hi Jezemine,
Thanks for the suggestion!

Yes, there is one extended store procedure used for
Goldmine 7 (gm7sql.dll). How can I find out if it is
this xp that is at fault?



stop calling it and see if the handle count stops growing?

quote:
Originally posted by glt101


Not sure what a CLR proc is. Can you point me to a
description of that?



CLR procs are procs implemented in managed code, such as C#. They are only available in 2005. See BOL for details.

My money is on the xp though. You have to test xps very thoroughly because they run in sqlserver's process space. If one of them AV's, it will bring your server down. Or if one of them leaks, it will bring your server down slowly.

You might think about contacting the vendor of the this xp and see if this is a known issue, pehaps they have an updated DLL.


elsasoft.org
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-15 : 12:26:38
Hi Jezemine,

Hmmm. Not really possible to stop the xp running:
production server, yada, yada, yada...
Isn't there some way in SQL Server to look at the
resources used by the xp?

My only other approach is to set up a continuous
test on our test server and watch what happens. But
that will take more time to arrange.

Cheers,
Geoff
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-15 : 16:03:37
are you saying that your test team let this system pass through to production without catching this? I think they need a sound whipping.


elsasoft.org
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-15 : 17:28:50
"Test team"? Sorry. Not familiar with that concept either.
Is that it BOL too?

Cheers,
Geoff
Go to Top of Page

rmiao
Master Smack Fu Yak Hacker

7266 Posts

Posted - 2008-02-15 : 23:48:52
How did you set sql memory? What's db size? Any resource error in sql log?
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-16 : 19:47:00
Hi rmiao,

SQL Memory is set to the default (this machine is an SQL
server only and has only one instance), so in this case
the Microsoft SQL Server Management Studio shows
2,147,483,647 for "Maximum Server memory (in MB)". I
presume that should say "(in Bytes)" since the server
has 2 GB of RAM :D !

The DB size in the Database properties window shows:
9427.13MG (General) and in the Files dialog shows:
Customers, Data, PRIMARY, 6907
Customers, log, N/A, 2521

Since 2/11/08 there is only 1 error in the SQL Log and
that relates to a logon failure. There are no references
to the xp dll other than one to note it is used to execute
the xp.

Thanks for the help!

Cheers,
Geoff
PS Now at 800,000 handles... GULP
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-16 : 19:56:57
one thing you could do is profile sql server with Profiler and PerfMon at the same time. with profiler, look for calls to the xp. In perfmon, monitor the handle count. Then you can interleave the traces (I believe the profiler in 2005 supports this) and see if there appears to be a connection between when the xp is called and when the handle count goes up.

I am 99% sure this leak is not the fault of sql server's relational engine. the engine goes through very extensive testing at ms, and in the real world as well through heavy usage in installations all over the world. Such a gross leak would surely have been caught and fixed by now if it was in the engine.

did you contact the vendor of your xp's dll and ask them if they know about a leak? maybe they have a fix.


elsasoft.org
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-16 : 21:25:22
Hi jezemine,

Thank for that idea. I'll try it on Monday.
The xp is made by Frontrange and is part of
an extension to the Goldmine system that they
no longer offically support. However, there
are many users like ourselves who were left
"high and dry" by that decision and so with
each new schema for their DB they release a
new version of the dll. That way they keep
users "happy", but don't have the support
load. Nice business model eh?!

Anyway. Better get off my soapbox.

If I can correlate the handle count increase
with the calls in SQL Profiler, as you suggest,
I might be able to cajole them into fixing the
xp... ...if that's really the issue.


Thanks for all the help. You guys beat MSDN
forums and sqlcentral into a cocked hat as
they say.

I'll post back on Monday.

Cheers,
Geoff
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2008-02-17 : 04:21:15
I thought .Net should handle this situation with the garbage collector?



E 12°55'05.25"
N 56°04'39.16"
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-17 : 10:05:57
GC only takes care of memory allocation on the managed heap. it does not keep track of other OS resources such as file handles, regkey handles, GDI handles, threads, etc. many .net developers are under the (incorrect) assumption that they don't have to worry about resource management in their code. they do. what they don't have to worry about as much is memory management. it is certainly possible to write an app in managed code that will bring your server down after continuous running for a few days or weeks.

give this a read for more info on how to properly do resource management in managed code:
http://www.bluebytesoftware.com/blog/PermaLink.aspx?guid=88e62cdf-5919-4ac7-bc33-20c06ae539ae




elsasoft.org
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2008-02-17 : 16:21:22
I think I will keep my VB6.0 and internal Windows API calls for a while



E 12°55'05.25"
N 56°04'39.16"
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-17 : 21:21:40
coming from the C++ world, I often find coding in C# frustrating because you don't have deterministic cleanup the way you do in C++. Also the fact that you have to remember to call Dispose() on all IDisposables was disgusting to me when I first learned it years ago. the "using" keyword makes this a little easier, but it would have been much better if the compiler would just add a call to Dispose() when an IDisposable goes out of scope, similar to the way a C++ compiler adds a call to the destructor when a stack-allocated object goes out of scope. bleh. rant over.


elsasoft.org
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-18 : 15:00:29
Hi Jezemine,

Congrats. You were correct. Each call to the xp increments
the handle count for sqlservr.exe by 2. The call to the xp
is inside at least two cursors and is used extensively by
the business logic in the Goldmine extension. Indeed each
individual business operation increases the count by 202
handles.

So, now to tackle Frontrange. Thanks for the help!

Cheers,
Geoff
Go to Top of Page

jezemine
Master Smack Fu Yak Hacker

2886 Posts

Posted - 2008-02-18 : 17:09:59
ouch.

Q: what's worse than using a dreaded cursor in sql server?

A: using a cursor that calls a leaky xp. not only will it be slow, it will bring down your server!




elsasoft.org
Go to Top of Page

eyechart
Master Smack Fu Yak Hacker

3575 Posts

Posted - 2008-02-19 : 04:38:12
quote:
Originally posted by jezemine

ouch.

Q: what's worse than using a dreaded cursor in sql server?

A: using a cursor that calls a leaky xp. not only will it be slow, it will bring down your server!




elsasoft.org



maybe it would be worse if you created the cursor that execs the leaky xp using dynamic sql? but just barely :)



-ec
Go to Top of Page

glt101
Starting Member

28 Posts

Posted - 2008-02-19 : 10:01:56
Hi folks,

Just so I am not unfairly tarred by this:
the cursors are in code supplied in the
application extension.

OTOH, how would one replace the cursor
functionality anyway, if one were to need
it? Here's an example of the cursor. I
don't want to get into the details
of what all this does. Just how one would
replace the cursor behavior. Is there a
standard idiom to use instead?



OPEN fieldcursor

FETCH NEXT FROM fieldcursor INTO @fieldname, @value

WHILE @@fetch_status = 0
BEGIN
SELECT
@tablename = convert(sysname,o.name),
@datatype = convert (sysname,case
when t.xusertype > 255 then t.name
else d.NAME
end)
FROM
sysobjects o,
master.sys.types d,
systypes t,
syscolumns c
LEFT OUTER JOIN syscomments m on c.cdefault = m.id
AND m.colid = 1
WHERE
o.name in ('CONTACT1','CONTACT2')
AND o.id = c.id
AND t.xtype = system_type_id
AND o.type <> 'P'
AND c.xusertype = t.xusertype
AND c.name like @fieldname

IF @fieldname = 'MERGECODES'
SELECT @tablename = CASE WHEN @gmversion = '4' AND @tablename IS NOT NULL
THEN 'CONTACT2'
WHEN @gmversion <> '4' OR @gmversion IS NULL
THEN 'CONTACT1'
ELSE NULL
END

IF UPPER(@tablename) = 'CONTACT1'
BEGIN
IF UPPER(@fieldname) = 'OWNER'
SELECT @value = UPPER(@value)

SELECT @setcmd1 = ' ' + @fieldname + ' = ' + CASE WHEN UPPER(@datatype) NOT IN ('FLOAT', 'INT',
'REAL','NUMERIC','SMALLINT','TINYINT','DECIMAL')
AND @value IS NOT NULL
THEN '"' + @value + '"'
WHEN @value IS NULL THEN 'NULL'
ELSE @value END +
CASE WHEN (UPPER(@fieldname) IN ('COMPANY','CONTACT','LASTNAME',
'CITY','KEY1','KEY2','KEY3',
'KEY4','KEY5') OR
(UPPER(@fieldname) IN ('STATE', 'COUNTRY')
AND @gmversion <> 4)) AND @value IS NOT NULL
THEN ', u_' + @fieldname + ' = "' + UPPER(@value) + '"'
WHEN (UPPER(@fieldname) IN ('COMPANY','CONTACT','LASTNAME',
'CITY','KEY1','KEY2','KEY3',
'KEY4','KEY5') OR
(UPPER(@fieldname) IN ('STATE', 'COUNTRY')
AND @gmversion <> 4)) AND @value IS NULL
THEN ', u_' + @fieldname + ' = ""' ELSE "" END +
' WHERE recid = ' + CASE
WHEN @recid LIKE '%''%' THEN '"' + @recid + '"'
WHEN @recid LIKE '%"%' THEN '''' + @recid + ''''
ELSE '''' + @recid + ''''
END
EXEC('UPDATE contact1 SET ' + @setcmd1)
EXEC GMW_UpdateSyncLog 'contact1', @recid, @user, 'U', @fieldname
END

IF UPPER(@tablename) = 'CONTACT2'
BEGIN
/*this block of accountno and recid steps should only occur once the most*/
IF @accountno IS NULL
SELECT @accountno = accountno FROM contact1 WHERE recid = @recid

IF @c2recid IS NULL
SELECT @C2recid = recid FROM contact2 WHERE accountno = @accountno

IF @c2recid IS /*STILL*/ NULL --there is no contact2 record for this contact
BEGIN
/*get a recid*/
EXEC master..xp_GMNewRecid @user, @c2recid OUTPUT
INSERT contact2 (accountno, recid) VALUES (@accountno, @c2recid)
EXEC GMW_UpdateSyncLog 'contact2', @c2recid, @user, 'N'
END

SELECT @setcmd1 = ' ' + @fieldname + ' = ' + CASE WHEN UPPER(@datatype) NOT IN ('FLOAT', 'INT',
'REAL','NUMERIC','SMALLINT','TINYINT','DECIMAL')
AND @value IS NOT NULL
THEN '"' + @value + '"'
WHEN @value IS NULL THEN 'NULL'
ELSE @value END +
' WHERE recid = ' + CASE
WHEN @c2recid LIKE '%''%' THEN '"' + @c2recid + '"'
WHEN @c2recid LIKE '%"%' THEN '''' + @c2recid + ''''
ELSE '''' + @c2recid + ''''
END
EXEC('UPDATE contact2 SET ' + @setcmd1)
EXEC GMW_UpdateSyncLog 'contact2', @c2recid, @user, 'U', @fieldname
END

SELECT @tablename = NULL
FETCH NEXT FROM fieldcursor INTO @fieldname, @value

END

CLOSE fieldcursor
DEALLOCATE fieldcursor



Cheers,
Geoff
Go to Top of Page
    Next Page

- Advertisement -