Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2008 Forums
 SSIS and Import/Export (2008)
 import large amount of data

Author  Topic 

pille
Starting Member

15 Posts

Posted - 2011-02-25 : 22:28:03
hello all
whats the story with ms sql ?
i tried the express version (just to write the scripts)
i downloaded the enterprise version of 2008
but it seems i cant get more as 1000 rows in a table in one go.
i tried the sqlcmd, import wizard, sql statements - all failed.
i searched in google and the limitation seems to have many people. i did not find any solution yet.
do i need to write another script which count the entries up to 1000 and import it ?
finally we want to import 1 TB of data!!!!!
all in 1000 steps ????
any advice guys ???
thank you

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2011-02-25 : 23:25:05
Your problem is not clear. You haven't shown us the code, posted any errors, or really given enough information for us to help. Please be more detailed so that we may help.

Even Express edition can handle millions of rows.

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

pille
Starting Member

15 Posts

Posted - 2011-02-26 : 02:38:21
hello tara
thank you for your answer
i was thinking its a known issues !!!
code looks like
INSERT INTO [SDC].[dbo].[Timestamp]
([Timestamp])
VALUES
('00:00:36'),
('00:00:37'),
('00:00:38'),
('00:00:39'),
('00:00:40'),
('00:00:41'),
......
84400 lines
the error
Msg 10738, Level 15, State 1, Line 1004
The number of row value expressions in the INSERT statement exceeds the maximum allowed number of 1000 row values.
the same messages i get if i try to import. import as txt file and only the time (instead of ('00:00:40'), so 00:00:40).
later, if the code is finish i need to read apache logs in. every line data time ip domain link .....
how i could do it if sql cant manage over 1000 lines ?
hope its more clear now
Go to Top of Page

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2011-02-26 : 19:00:35
Where are these values coming from? A text file? If so, then use BULK INSERT.

Otherwise, you can use individual INSERT statements or perhaps even INSERT/SELECT UNION ALL.

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

pille
Starting Member

15 Posts

Posted - 2011-02-28 : 06:38:38
Hello Tara
the records are comming from testfiles (round about 5000 - each with 5000-150000 lines)
what could you advice ?
i am normalizise the database right now
to get rid of redudance datasets

thank you
Go to Top of Page

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2011-02-28 : 13:24:59
You should be using BULK INSERT to quickly bulk load data from files into SQL Server. Do not use INSERT commands.

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page
   

- Advertisement -