Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
Author |
Topic |
dteviot
Starting Member
11 Posts |
Posted - 2010-01-20 : 21:50:24
|
Hi, I’m hoping someone can help me with a problem I’m having.To summarise: randomly, ADO calls return the same result set that was returned to a previous call.Has anyone seen this problem before? Or have any idea what might be causing this or how to fix it? Example.I have the following code:using (var cmd = Connection.CreateCommand()){ Connection.Open(); cmd.CommandType = CommandType.Text; cmd.CommandText = "INSERT INTO AssemblyBinaries (AssemblyData) VALUES (@AssemblyData); SELECT @@IDENTITY AS AssemblyId"; cmd.Parameters.Add(new SqlParameter("@AssemblyData", SqlDbType.Image, 2147483647, ParameterDirection.Input, true, 0, 0, "AssemblyData", DataRowVersion.Current, assemblyBinary.AssemblyData)); using (IDataReader reader = cmd.ExecuteReader()) { reader.Read(); etc... However, when I run this code, instead of reader having a result set with a single column named “AssemblyId”, sometimes it would have an entirely different result set.Further investigation showed the result set that was returned looked like the results of the previous SQL query on the connection.e.g. The result set contained four columns “VersionMajor”, “VersionMinor”, “BuildNumber” and “DatabaseType”. Looking at the log from the SQL Server profiler, I looked for the “Insert into AssemblyBinaries...” query. Checking the connection, I see that the prior call on that connection was “SELECT VersionMajor, VersionMinor, BuildNumber, DatabaseType FROM DatabaseVersion”.A timing of the events is also interesting.In the Profiler log, the “Select VersionMajor...” query started at 15:13:51.010 and finished at 15:13:51.057The “Insert into AssemblyBinaries...” query started at 15:14:35.143 and finished at 15:14:49.047However, the C# client log shows reader.Read() returned at 15:14:35. That is, immediately after the SQL server started the query, but long before the SQL server finished processing the query. (I therefore deduce the problem isn’t in the SQL server. Which only leaves the ADO stack on the client as the culprit.)Additional notes.[*]I’m using SQL server 2005.[*]ADO is using connection pooling. When it’s turned off, the problem stops.[*]Multiple Active Result Sets isn’t a factor. Problem occurs even when it’s explicitly turned off, although most MS docs say it’s off by default. (There are other docs that say it’s on by default.)[*]The problem occurs more frequently on multicore PCs than single core.This problem occurs randomly, sometimes the calls work, and sometimes they don’t.[*]I’ve tried turning on ADO logging (via ETW) but it didn’t show anything useful.[*]The application is running on a VM. |
|
Kristen
Test
22859 Posts |
Posted - 2010-01-21 : 03:02:27
|
"ADO is using connection pooling. When it’s turned off, the problem stops."Before I got to this point I was thinking that Connection Pooling would most likely be the culprit.I've read about mis-behaviour of connection pooling, but not for some time.Update all drivers etc. / check on latest version? |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-24 : 22:02:52
|
quote: Originally posted by KristenUpdate all drivers etc. / check on latest version?
Thanks for your suggestion. Unfortunately, I see the problem with both SP2 and SP3. (I'm not aware of any other Service Packs for SQL 2005.)Also, I've used Reflector on the ADO.NET DLL, (System.Data.DLL) and it appears that its SQL Server classes don't rely on anything else to talk to the SQL server. That is, the .Net Framework contains everything needed to talk to SQL server, and the DLL does not appear to have changed since 2.0. That is, there is no patched version of ADO.NET 2.0. (.Net 3.5 SP1, still contains the same ADO.NET DLLs as 2.0) |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-01-25 : 02:20:28
|
Sounds like you have all the latest drivers then. Sorry, I haven't got other ideas beyond that.I think if it was me I would include a "marker" in every query so that the client, and any Server Sprocs, could be sure that it had the right resultset, and then fail the moment the resultset "marker" was wrong. That might help you point the finger at the problem by having the application fail the moment the problem arose.But that strikes me as quite a lot of work just to get a better test-bed |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2010-01-25 : 10:46:47
|
What version Windows and IIS are you running at web server?What MDAC version is installed on IIS server?Are you properly closing and destroying your objects?Are you executing any SET ROWCOUNT statements?By the way, most likely the problem is at the web server, not the database server. |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-25 : 15:54:22
|
quote: Originally posted by russell What version Windows and IIS are you running at web server?/
I'm running Windows 2003. (And I've also tried Windows 2003 R2 and yesterday, 2008 R2.) 2003 R2 has the problem, 2008 didn't show the problem, but the number of test runs was too small to be sure that the problem doesn't occur.I doubt IIS is the problem, as what I have is a service written in C# that talks to the database directly, using ADO.NET 2.0.quote: What MDAC version is installed on IIS server?
I don't know, whatever came with the version of SQL server that was installed. Also, it's probably not relevant, as ADO.NET 2.0 does not use MDAC.quote: Are you properly closing and destroying your objects?
Well, as you can see from the example code, in that failure case, I'm creating a new command object from scratch. (Although the connection object is probably being recycled.)quote: Are you executing any SET ROWCOUNT statements?
Yes, in some stored procedures. How could that be responsible for the problem?quote: By the way, most likely the problem is at the web server, not the database server.
I don't see how that's possible. While I do use IIS, it's not between the Service having the problem and the database. |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2010-01-25 : 16:26:03
|
there was a known issue with set rowcount in one batch "sticking" in subsequent batches when connection pooling is enabled. not sure if it was ever resolved with a patch. i'll see if i can google it out for you.why do u say iis or the web server can't be responsible? your C# code is executing on the web server (or application server), not the database server, right?should look in the registry of the web server and see what the latest version of ADO/X is.i don't see where you're explicitly closing any connections |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-25 : 22:01:39
|
quote: Originally posted by russell there was a known issue with set rowcount in one batch "sticking" in subsequent batches when connection pooling is enabled. not sure if it was ever resolved with a patch. i'll see if i can google it out for you.
I think it's this: http://support.microsoft.com/kb/310617. I don't think it's relevant, as it applies to SQL 2000, and applies to the Rowcount setting itself.quote: why do u say iis or the web server can't be responsible? your C# code is executing on the web server (or application server), not the database server, right?
No. The C# code is a stand alone, self hosted service. It provides services to IIS, but doesn't use IIS itself.quote: i don't see where you're explicitly closing any connections
I removed that code from the sample I showed, but it is there. (Also, the fact that I had to open the connection before I could use it indicates that it was previously closed.) |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-01-26 : 02:52:01
|
"Also, the fact that I had to open the connection before I could use it indicates that it was previously closed"I think Russell was meaning that if you don't explicitly close a DB connection there is the possibility that the polling considers it as still being open and "hands it out" to the next Customer, whereas explicitly closing it tells the Pool that the connection is now dead. |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2010-01-26 : 08:55:57
|
dteviot, it seems you don't understand a lot of things here. only trying to help, but almost everything you said in the last post is wrong.just because you called an open() method doesn't mean you called a close() method previously. closing and destroying objects is just good programming practice. not doing so is rubbish. and if you program in a language that doesn't do automatic discarding of objects and garbage collection, then you'll quickly run out of memory. there's a term for this: memory leak.now, as for the C# service...it uses the objects installed on the server it lives on, not the SQL Server. so the ADO version being used depends on the box hosting the service. tell me, does it live on the web server, the database server or somewhere else?are you running iis and sql on the same box?i would try explicitly executing a SET ROWCOUNT 0 to see if that clears it up. may not be the problem but worth a shot. it is completely possible that this doesn't help, but you haven't given a lot of information yet for us to be a lot of help. |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
|
Kristen
Test
22859 Posts |
Posted - 2010-01-26 : 09:45:55
|
"i would try explicitly executing a SET ROWCOUNT 0 to see if that clears it up"Our in-house policy is to use SET ROWCOUNT 0 immediately after any explicit SET ROWCOUNT n - too great a risk of some other code being added that uses the current ROWCOUNT and only performance half-a-job. Lots of such things can be out-of-sight and out-of-mind too - a Trigger pushing associated data to another table; a sub-process using a working table ... all not expecting to be running with a restriction on the ROWCOUNT |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-26 : 16:49:35
|
Kristen, Russell,Firstly, Thank you very much for your time in responding to my question, it is appreciated.I don't think the rowcount bug is the problem here.- I've checked our code, Set rowcount is only used in stored procs, (RecordSet.MaxRecords is never used) and the pattern is as Kristne describes, set rowcount X; <SQL operation>; Set rowcount 0
- The rowcount bug (rowcount not cleared on a pooled connection) doesn't match the problem I'm seeing (previous queries result set is returned)
- The rowcount bug is in SQL 2000, I'm running SQL 2005.
quote: now, as for the C# service...it uses the objects installed on the server it lives on, not the SQL Server.
Agreed.quote: so the ADO version being used depends on the box hosting the service.
Um, yes and no. I'm using the SQL Native classes in ADO.NET 2.0. (e.g. SQLConnection http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnection.aspx, SQLCommand http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlcommand.aspx, etc.)While I can't say I've definitely proven it, I'm pretty confident that ADO.NET 2.0 does NOT use ADO, MDAC or SQL Native Client components if you're using the SNI classes to talk to SQL Server.(Justification: the ADO.NET SQL Server classes reside in System.Data.dll. Inspecting this DLL with Reflector clearly shows the implementation of the TDS layer of the the protocol. Proving that it also contains the transport layer is more difficult (as those functions are native code rather than IL) but their names strongly suggests that the SQL Native Interface is completely contained within the DLL, with no use of the ADO or SQL Native Client DLLs. That is, if using ADO.NET 2.0, and the SQL Server Native functions, the components used by the client are entirely due to the version of the .Net Framework installed.Well, with the exception of the Windows DLLs that are being used to provide the TCP or Named pipes networking layer.quote: are you running iis and sql on the same box?
Yes, I'm running IIS, SQL Server and the C# service on the same box.quote: just because you called an open() method doesn't mean you called a close() method previously.
Um, the Open() call means the connection is in a closed state at the point the Open() was called.If you try running code like this: using (SqlConnection connection = new SqlConnection(connectionString)) { connection.Open(); connection.Open(); The second Open() will throw an InvalidOperationException, "The connection was not closed."That said, it's possible the connection was in a closed state because it was created, but hadn't used (or opened) prior to my using it.But I can see in the ETW trace that the connection was closed prior to my opening it. |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-01-26 : 16:57:46
|
Re: your final point. I think it is important that you explicitly close the connection. That has a higher chance of telling the pool to "clean up" than relying on automatic connection close (which might fail, or get closed down because it isn't responding quickly enough, or all-sorts when the APP closes - and even if that all works OK the chances are the CLOSE won't be implictly generated when the APP does a nose-dive-crash - although the chance of you closing everything when that happens is pretty remote too! |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2010-01-26 : 17:42:15
|
Actually the rowcount bug was ADO 2.6 I'll stand by the explicitly closing connections -- this should go without saying. but I am not wholly convinced that's your problem at this point. still it is such an obvious mistake that it ought to be fixed and see if it doesn't help. that said, the .net runtime should clean that up for you.Also, correct me if I'm wrong, but pooling is a client technology, not a SQL Server one, so again, the problem points to ADO.net, IIS, or the application code itself. |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-28 : 14:46:24
|
quote: Originally posted by russellI'll stand by the explicitly closing connections -- this should go without saying. but I am not wholly convinced that's your problem at this point. still it is such an obvious mistake that it ought to be fixed and see if it doesn't help. that said, the .net runtime should clean that up for you.
I've had another look at the ETW logs. The connection is definitely being closed. The really interesting thing is that the Open() doesn't appear to go correctly. To explain, the ETW log shows the difference between an open where the connection is just fetched from the pool, and where the connection actually connects to the SQL server. Likewise, it also shows when Close() just results in the connection being returned to the pool, and when it's actually disconnected from the server. What I can see is that, for the failure case, the close after the preceeding SQL call, the connection was fullly closed. But, the following Open() call pulled the connection from the pool. That is, the connection was fully closed, then used without being properly opened. (Assuming I'm reading the ETW log correctly.)quote: Also, correct me if I'm wrong, but pooling is a client technology, not a SQL Server one, so again, the problem points to ADO.net, IIS, or the application code itself.
Yes, pooling is a client technology. At the current time, I'm suspecting the problem is a threading problem. That is, the client code is calling the ADO.NET across threads in such a way that the ADO.NET internal datasstructures are getting corrupted. |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-01-28 : 15:14:24
|
Any chance that, when the problem arises, the previous user of the pooled connection had pulled all the data? i..e was closing the connection "early"? Perhaps there is another recordset, or some warning messages that have not been retrieved ("WARNING: NULLs in aggregate" or somesuch) - I'm thinking that this may cause the Pool to think the Connection still has valid data.But its just a guess, I like your Threading hypothesis too. |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-01-28 : 20:07:57
|
quote: Originally posted by Kristen Any chance that, when the problem arises, the previous user of the pooled connection had pulled all the data? i..e was closing the connection "early"? Perhaps there is another recordset, or some warning messages that have not been retrieved ("WARNING: NULLs in aggregate" or somesuch) - I'm thinking that this may cause the Pool to think the Connection still has valid data.
Well, there can't have been a second recordset. We did have S.P.s that returned multiple record sets, but handing them was such a pain that they were removed years ago. All our calls to the database return either 0 or 1 recordset. Also, in every failure case I've examined, the prior call's result set should have contained no more than a single row.It's possible there was an error message on the connection, but I've no idea how to prove it was missed. (I was under the impression that ADO.NET looked for them in the datastream, and raises an exception if they're found.) |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-01-29 : 02:46:37
|
"It's possible there was an error message on the connection, but I've no idea how to prove it was missed."Indeed Re-running the SQL that was used, manually, would show the resultset plus any warning error."(I was under the impression that ADO.NET looked for them in the datastream, and raises an exception if they're found.)"I don't know how ADO.NET handles them. If its a fatal error I would assume you get that, but for warnings they tend to be in an "Errors Collection" that you can examine, and next-through.And even if that IS the case the Connection Pooling system ought to handle it.I think this is long odds, so don't start any hares running! I only mentioned it because it popped into my mind. |
|
|
dteviot
Starting Member
11 Posts |
Posted - 2010-02-01 : 15:24:54
|
quote: Originally posted by KristenRe-running the SQL that was used, manually, would show the resultset plus any warning error.
Um, not always. For example, consider the call SELECT VersionMajor, VersionMinor, BuildNumber, DatabaseType FROM DatabaseVersion Every time I've run it manually, it has always returned a result set with a single row of four integers. With no errors or warnings. However, its still possible for something to go wrong (e.g. Deadlock, page fault, out of memory, or any of a number of other transitory problems.) The manual run has no way of showing that this happened in a prior run. |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2010-02-01 : 15:49:50
|
A few things...Here is something you can test:Change the SQL statement to a stored proc:Create Proc test @AssemblyData varbinary(max)ASSET NOCOUNT ONINSERT INTO AssemblyBinaries (AssemblyData)VALUES (@AssemblyData)SELECT @@IDENTITY AS AssemblyIdGO Is the text in row option on or off on the AssemblyBinaries table? Also, you''re using deprecated image data type.Did you ever go back and explicitly close your connections?What does your connection string look like (when you haven't disabled pooling)?If that SP doesn't help, can try SET ROWCOUNT 0 just below the SET NOCOUNT ON |
|
|
Next Page
|
|
|
|
|