Float gives single precision? - SQL Server Forums

Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

All Forums

SQL Server 2005 Forums

Other SQL Server Topics (2005)

Float gives single precision?

Author

Topic

John2011
Starting Member

2 Posts

Posted - 2011-08-17 : 13:59:07

I'm writing a stored procedure in Transact-SQL. My code looks like:

Declare @tmpResult float(53)
Set @tmpResult = Cast(1.123456789 as float(53))
Print @tmpResult

The print gives 1.12346 (with or without the cast or (53))

Am I on dope or is this single precision? Any ideas why it's not double?

Thanks

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2011-08-17 : 14:20:31

It's a result of the PRINT operation, not the value itself.

Float is neither single nor double precision. Float has loss built-in, according to IEEE 854.

N 56°04'39.26"
E 12°55'05.63"

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2011-08-17 : 14:23:23

Declare @tmpResult float(53)
Set @tmpResult = Cast(1.123456789 as float(53))
select [@tmpResult] = @tmpResult

Results:

@tmpResult
----------------------------------------------------- 
                                          1.123456789

CODO ERGO SUM

John2011
Starting Member

2 Posts

Posted - 2011-08-17 : 15:45:09

Thanks for your help.

Should have realized it was converting to nvarchar...

Subscribe to SQLTeam.com

SQLTeam.com Articles via RSS

SQLTeam.com Weblog via RSS

- Advertisement -

Resources