Showing posts with label files. Show all posts
Showing posts with label files. Show all posts

Friday, March 30, 2012

RAID5 for all files?

In the last few months I've run across two places that had all their
files, including both data and logs, on big, fat RAID5 partitions.
In fact, in one place even the OS and pagefile were on RAID5!
Is this, like, a good idea all of a sudden, and nobody told me?
I hautily informed them that putting in separate physical drives for a
RAID1 set for logs, might provide a load/scalability/performance
factor of 2x all by itself. Is it at all likely that this is actually
the case? Just wondering.
Thanks.
Josh
No...Raid 5 still sucks. The baarf web site is still in
operation - http://www.baarf.com/
Logs being separated out on a Raid 1 or Raid 10 is still
recommended - see the Storage Best Practices:
http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/storage-top-10.mspx
-Sue
On Sun, 05 Aug 2007 20:17:18 -0700, JXStern
<JXSternChangeX2R@.gte.net> wrote:

>In the last few months I've run across two places that had all their
>files, including both data and logs, on big, fat RAID5 partitions.
>In fact, in one place even the OS and pagefile were on RAID5!
>Is this, like, a good idea all of a sudden, and nobody told me?
>I hautily informed them that putting in separate physical drives for a
>RAID1 set for logs, might provide a load/scalability/performance
>factor of 2x all by itself. Is it at all likely that this is actually
>the case? Just wondering.
>Thanks.
>Josh
|||As Sue mentions it is still not the best practice to use Raid5 for a busy
OLTP system.
Andrew J. Kelly SQL MVP
"JXStern" <JXSternChangeX2R@.gte.net> wrote in message
news:kd4db35ek3ge7ldau600i4tk7igv6irns6@.4ax.com...
> In the last few months I've run across two places that had all their
> files, including both data and logs, on big, fat RAID5 partitions.
> In fact, in one place even the OS and pagefile were on RAID5!
> Is this, like, a good idea all of a sudden, and nobody told me?
> I hautily informed them that putting in separate physical drives for a
> RAID1 set for logs, might provide a load/scalability/performance
> factor of 2x all by itself. Is it at all likely that this is actually
> the case? Just wondering.
> Thanks.
> Josh
>
|||On Mon, 6 Aug 2007 08:50:20 -0400, "Andrew J. Kelly"
<sqlmvpnooospam@.shadhawk.com> wrote:

>As Sue mentions it is still not the best practice to use Raid5 for a busy
>OLTP system.
And even less good for a busy ETL system building gigabyte tables and
output files?
J.
sql

RAID5 for all files?

In the last few months I've run across two places that had all their
files, including both data and logs, on big, fat RAID5 partitions.
In fact, in one place even the OS and pagefile were on RAID5!
Is this, like, a good idea all of a sudden, and nobody told me?
I hautily informed them that putting in separate physical drives for a
RAID1 set for logs, might provide a load/scalability/performance
factor of 2x all by itself. Is it at all likely that this is actually
the case? Just wondering.
Thanks.
JoshNo...Raid 5 still sucks. The baarf web site is still in
operation - http://www.baarf.com/
Logs being separated out on a Raid 1 or Raid 10 is still
recommended - see the Storage Best Practices:
[url]http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/storage-top-10.mspx[
/url]
-Sue
On Sun, 05 Aug 2007 20:17:18 -0700, JXStern
<JXSternChangeX2R@.gte.net> wrote:

>In the last few months I've run across two places that had all their
>files, including both data and logs, on big, fat RAID5 partitions.
>In fact, in one place even the OS and pagefile were on RAID5!
>Is this, like, a good idea all of a sudden, and nobody told me?
>I hautily informed them that putting in separate physical drives for a
>RAID1 set for logs, might provide a load/scalability/performance
>factor of 2x all by itself. Is it at all likely that this is actually
>the case? Just wondering.
>Thanks.
>Josh|||As Sue mentions it is still not the best practice to use Raid5 for a busy
OLTP system.
Andrew J. Kelly SQL MVP
"JXStern" <JXSternChangeX2R@.gte.net> wrote in message
news:kd4db35ek3ge7ldau600i4tk7igv6irns6@.
4ax.com...
> In the last few months I've run across two places that had all their
> files, including both data and logs, on big, fat RAID5 partitions.
> In fact, in one place even the OS and pagefile were on RAID5!
> Is this, like, a good idea all of a sudden, and nobody told me?
> I hautily informed them that putting in separate physical drives for a
> RAID1 set for logs, might provide a load/scalability/performance
> factor of 2x all by itself. Is it at all likely that this is actually
> the case? Just wondering.
> Thanks.
> Josh
>|||On Mon, 6 Aug 2007 08:50:20 -0400, "Andrew J. Kelly"
<sqlmvpnooospam@.shadhawk.com> wrote:

>As Sue mentions it is still not the best practice to use Raid5 for a busy
>OLTP system.
And even less good for a busy ETL system building gigabyte tables and
output files?
J.

RAID5 for all files?

In the last few months I've run across two places that had all their
files, including both data and logs, on big, fat RAID5 partitions.
In fact, in one place even the OS and pagefile were on RAID5!
Is this, like, a good idea all of a sudden, and nobody told me?
I hautily informed them that putting in separate physical drives for a
RAID1 set for logs, might provide a load/scalability/performance
factor of 2x all by itself. Is it at all likely that this is actually
the case? Just wondering. :)
Thanks.
JoshNo...Raid 5 still sucks. The baarf web site is still in
operation - http://www.baarf.com/
Logs being separated out on a Raid 1 or Raid 10 is still
recommended - see the Storage Best Practices:
http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/storage-top-10.mspx
-Sue
On Sun, 05 Aug 2007 20:17:18 -0700, JXStern
<JXSternChangeX2R@.gte.net> wrote:
>In the last few months I've run across two places that had all their
>files, including both data and logs, on big, fat RAID5 partitions.
>In fact, in one place even the OS and pagefile were on RAID5!
>Is this, like, a good idea all of a sudden, and nobody told me?
>I hautily informed them that putting in separate physical drives for a
>RAID1 set for logs, might provide a load/scalability/performance
>factor of 2x all by itself. Is it at all likely that this is actually
>the case? Just wondering. :)
>Thanks.
>Josh|||As Sue mentions it is still not the best practice to use Raid5 for a busy
OLTP system.
--
Andrew J. Kelly SQL MVP
"JXStern" <JXSternChangeX2R@.gte.net> wrote in message
news:kd4db35ek3ge7ldau600i4tk7igv6irns6@.4ax.com...
> In the last few months I've run across two places that had all their
> files, including both data and logs, on big, fat RAID5 partitions.
> In fact, in one place even the OS and pagefile were on RAID5!
> Is this, like, a good idea all of a sudden, and nobody told me?
> I hautily informed them that putting in separate physical drives for a
> RAID1 set for logs, might provide a load/scalability/performance
> factor of 2x all by itself. Is it at all likely that this is actually
> the case? Just wondering. :)
> Thanks.
> Josh
>|||On Mon, 6 Aug 2007 08:50:20 -0400, "Andrew J. Kelly"
<sqlmvpnooospam@.shadhawk.com> wrote:
>As Sue mentions it is still not the best practice to use Raid5 for a busy
>OLTP system.
And even less good for a busy ETL system building gigabyte tables and
output files?
J.

Wednesday, March 28, 2012

RAID 6

Any comments on using RAID 6 for Data Files for SQL Server?
Hi Greg
"Greg Larsen" wrote:

> Any comments on using RAID 6 for Data Files for SQL Server?
In what way?
I would expect this to be similar to Raid 5 performance depending on how the
parity is calculated, but you get extra fault tollerance.
John
|||I would expect it to be at least a millisecond or two slower on average for
writes due to having to wait (sometimes) for the second parity write to the
extra disk. This could be a performance drain on write-intensive (i.e.
OLTP) databases.
TheSQLGuru
President
Indicium Resources, Inc.
"John Bell" <jbellnewsposts@.hotmail.com> wrote in message
news:2E2819DC-966C-48E3-934B-DAB3194D2F74@.microsoft.com...
> Hi Greg
> "Greg Larsen" wrote:
> In what way?
> I would expect this to be similar to Raid 5 performance depending on how
> the
> parity is calculated, but you get extra fault tollerance.
> John

RAID 6

Any comments on using RAID 6 for Data Files for SQL Server?Hi Greg
"Greg Larsen" wrote:

> Any comments on using RAID 6 for Data Files for SQL Server?
In what way?
I would expect this to be similar to Raid 5 performance depending on how the
parity is calculated, but you get extra fault tollerance.
John|||I would expect it to be at least a millisecond or two slower on average for
writes due to having to wait (sometimes) for the second parity write to the
extra disk. This could be a performance drain on write-intensive (i.e.
OLTP) databases.
TheSQLGuru
President
Indicium Resources, Inc.
"John Bell" <jbellnewsposts@.hotmail.com> wrote in message
news:2E2819DC-966C-48E3-934B-DAB3194D2F74@.microsoft.com...
> Hi Greg
> "Greg Larsen" wrote:
>
> In what way?
> I would expect this to be similar to Raid 5 performance depending on how
> the
> parity is calculated, but you get extra fault tollerance.
> John

RAID 6

Any comments on using RAID 6 for Data Files for SQL Server?Hi Greg
"Greg Larsen" wrote:
> Any comments on using RAID 6 for Data Files for SQL Server?
In what way?
I would expect this to be similar to Raid 5 performance depending on how the
parity is calculated, but you get extra fault tollerance.
John|||I would expect it to be at least a millisecond or two slower on average for
writes due to having to wait (sometimes) for the second parity write to the
extra disk. This could be a performance drain on write-intensive (i.e.
OLTP) databases.
--
TheSQLGuru
President
Indicium Resources, Inc.
"John Bell" <jbellnewsposts@.hotmail.com> wrote in message
news:2E2819DC-966C-48E3-934B-DAB3194D2F74@.microsoft.com...
> Hi Greg
> "Greg Larsen" wrote:
>> Any comments on using RAID 6 for Data Files for SQL Server?
> In what way?
> I would expect this to be similar to Raid 5 performance depending on how
> the
> parity is calculated, but you get extra fault tollerance.
> John

Monday, March 26, 2012

RAID 1 and write caching raid controllers

Currently I have a database server that has the log files on a raid array
supported by a controller that does NOT support write caching, only 100%
read. I have another raid controller on the server that does support write
caching.
Would there be any performace benefits from having the raid 1 array with the
log files supported by the raid controller with write caching over one that
does not?
Thanks!!!
yes, as unless there is a rollback or some recovery operation the log is
mostly a write-to file therefore you could reduce some queue time if you put
the log on the write cache controller...but what are you going to put ont he
read caching controller ?
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
> Currently I have a database server that has the log files on a raid array
> supported by a controller that does NOT support write caching, only 100%
> read. I have another raid controller on the server that does support write
> caching.
> Would there be any performace benefits from having the raid 1 array with
> the
> log files supported by the raid controller with write caching over one
> that
> does not?
> Thanks!!!
|||Great, thanks for the reply.
Dunno. I suppose nothing.
"David J. Cartwright" wrote:

> yes, as unless there is a rollback or some recovery operation the log is
> mostly a write-to file therefore you could reduce some queue time if you put
> the log on the write cache controller...but what are you going to put ont he
> read caching controller ?
> "gracie" <gracie@.discussions.microsoft.com> wrote in message
> news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
>
>
|||Is the cache battery backed up and is the machine on a UPS?
SQL does read the log too.
Paul
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:15119B01-2F8D-464D-A287-75D3CFFDBBD3@.microsoft.com...[vbcol=seagreen]
> Great, thanks for the reply.
> Dunno. I suppose nothing.
> "David J. Cartwright" wrote:
sql

RAID 1 and write caching raid controllers

Currently I have a database server that has the log files on a raid array
supported by a controller that does NOT support write caching, only 100%
read. I have another raid controller on the server that does support write
caching.
Would there be any performace benefits from having the raid 1 array with the
log files supported by the raid controller with write caching over one that
does not?
Thanks!!!yes, as unless there is a rollback or some recovery operation the log is
mostly a write-to file therefore you could reduce some queue time if you put
the log on the write cache controller...but what are you going to put ont he
read caching controller ?
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
> Currently I have a database server that has the log files on a raid array
> supported by a controller that does NOT support write caching, only 100%
> read. I have another raid controller on the server that does support write
> caching.
> Would there be any performace benefits from having the raid 1 array with
> the
> log files supported by the raid controller with write caching over one
> that
> does not?
> Thanks!!!|||Great, thanks for the reply.
Dunno. I suppose nothing.
"David J. Cartwright" wrote:
> yes, as unless there is a rollback or some recovery operation the log is
> mostly a write-to file therefore you could reduce some queue time if you put
> the log on the write cache controller...but what are you going to put ont he
> read caching controller ?
> "gracie" <gracie@.discussions.microsoft.com> wrote in message
> news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
> > Currently I have a database server that has the log files on a raid array
> > supported by a controller that does NOT support write caching, only 100%
> > read. I have another raid controller on the server that does support write
> > caching.
> >
> > Would there be any performace benefits from having the raid 1 array with
> > the
> > log files supported by the raid controller with write caching over one
> > that
> > does not?
> >
> > Thanks!!!
>
>|||Is the cache battery backed up and is the machine on a UPS?
SQL does read the log too.
Paul
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:15119B01-2F8D-464D-A287-75D3CFFDBBD3@.microsoft.com...
> Great, thanks for the reply.
> Dunno. I suppose nothing.
> "David J. Cartwright" wrote:
>> yes, as unless there is a rollback or some recovery operation the log is
>> mostly a write-to file therefore you could reduce some queue time if you
>> put
>> the log on the write cache controller...but what are you going to put ont
>> he
>> read caching controller ?
>> "gracie" <gracie@.discussions.microsoft.com> wrote in message
>> news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
>> > Currently I have a database server that has the log files on a raid
>> > array
>> > supported by a controller that does NOT support write caching, only
>> > 100%
>> > read. I have another raid controller on the server that does support
>> > write
>> > caching.
>> >
>> > Would there be any performace benefits from having the raid 1 array
>> > with
>> > the
>> > log files supported by the raid controller with write caching over one
>> > that
>> > does not?
>> >
>> > Thanks!!!
>>

RAID 1 and write caching raid controllers

Currently I have a database server that has the log files on a raid array
supported by a controller that does NOT support write caching, only 100%
read. I have another raid controller on the server that does support write
caching.
Would there be any performace benefits from having the raid 1 array with the
log files supported by the raid controller with write caching over one that
does not?
Thanks!!!yes, as unless there is a rollback or some recovery operation the log is
mostly a write-to file therefore you could reduce some queue time if you put
the log on the write cache controller...but what are you going to put ont he
read caching controller ?
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
> Currently I have a database server that has the log files on a raid array
> supported by a controller that does NOT support write caching, only 100%
> read. I have another raid controller on the server that does support write
> caching.
> Would there be any performace benefits from having the raid 1 array with
> the
> log files supported by the raid controller with write caching over one
> that
> does not?
> Thanks!!!|||Great, thanks for the reply.
Dunno. I suppose nothing.
"David J. Cartwright" wrote:

> yes, as unless there is a rollback or some recovery operation the log is
> mostly a write-to file therefore you could reduce some queue time if you p
ut
> the log on the write cache controller...but what are you going to put ont
he
> read caching controller ?
> "gracie" <gracie@.discussions.microsoft.com> wrote in message
> news:A7EEFFFA-A36A-4928-A64C-0A44435CB74A@.microsoft.com...
>
>|||Is the cache battery backed up and is the machine on a UPS?
SQL does read the log too.
Paul
"gracie" <gracie@.discussions.microsoft.com> wrote in message
news:15119B01-2F8D-464D-A287-75D3CFFDBBD3@.microsoft.com...[vbcol=seagreen]
> Great, thanks for the reply.
> Dunno. I suppose nothing.
> "David J. Cartwright" wrote:
>

Tuesday, March 20, 2012

quote in csv

I'm using the reporting services web service to export reports to csv files.
When I export to a csv, a quote mark is appended at the beginning and end of
each line. I'm using the default device info settings. This is not the
Qualifier, because I tried changing that from ["] to other characters in the
devinfo settings, and the ["] is still inserted.
How can I prevent the ["] from being inserted? It causes problems when I
open the file in Excel, because Excel thinks it means the whole row is one
field, and it gets truncated after 256 characters.
Thanks!
BillTwo different solutions depending on RS 2000 or RS 2005. Note, either you
need to be concerned with a merged cells problem when exporting CSV.
Solution 1:
Depending on how you design your reports you can do the following to export
to Excel. Or, what I do sometimes is make a copy of the report and clean it
up for data export and then hide it in list view. If you export from Report
Manager it puts CSV data in unicode which Excel puts all in one column. If
you export in ASCII then Excel does just as you want. To prevent a problem
with cells (Excel will object to sorting the data) you need to remove any
textboxes you have (for instance with a title, showing the parameters run
etc) and instead add additional header rows, merge the cells and put your
text in there instead. I add a link at the top of the report that says
Export Data. With RS 2005 you will be able to configure it to use ASCII
instead of Unicode.
Here is an example of a Jump to URL link I use. This causes Excel to come up
with the data in a separate window:
="javascript:void(window.open('" & Globals!ReportServerUrl &
"?/SomeFolder/SomeReport&ParamName=" & Parameters!ParamName.Value &
"&rs:Format=CSV&rc:Encoding=ASCII','_blank'))"
If you don't want to have it appear in a new window then do this in jump to
URL:
=Globals!ReportServerUrl & "?/SomeFolder/SomeReport&ParamName=" &
Parameters!ParamName.Value & "&rs:Format=CSV&rc:Encoding=ASCII"
Solution 2:
Do the above design change to avoid the merged cell problem with sorting in
Excel. Then modify rsreportserver.config. Reboot after the change. The below
shows commenting out the existing entry and putting in the needed change to
have CSV export as ASCII
. <!--
<Extension Name="CSV"
Type="Microsoft.ReportingServices.Rendering.CsvRenderer.CsvReport,Microsoft.ReportingServices.CsvRendering"/>
-->
<Extension Name="CSV"
Type="Microsoft.ReportingServices.Rendering.CsvRenderer.CsvReport,Microsoft.ReportingServices.CsvRendering">
<Configuration>
<DeviceInfo>
<Encoding>ASCII</Encoding>
</DeviceInfo>
</Configuration>
</Extension>
Very nice and very fast.
Bruce Loehle-Conger
MVP SQL Server Reporting Services
"bill" <belgie@.datamti.com> wrote in message
news:%235tVqK7ZGHA.3684@.TK2MSFTNGP05.phx.gbl...
> I'm using the reporting services web service to export reports to csv
> files.
> When I export to a csv, a quote mark is appended at the beginning and end
> of each line. I'm using the default device info settings. This is not
> the Qualifier, because I tried changing that from ["] to other characters
> in the devinfo settings, and the ["] is still inserted.
> How can I prevent the ["] from being inserted? It causes problems when I
> open the file in Excel, because Excel thinks it means the whole row is one
> field, and it gets truncated after 256 characters.
> Thanks!
> Bill
>
>
>|||I just realized that the quotes are being added when I save the csv, and
aren't there when it is generated by the reporting service.
"bill" <belgie@.datamti.com> wrote in message
news:%235tVqK7ZGHA.3684@.TK2MSFTNGP05.phx.gbl...
> I'm using the reporting services web service to export reports to csv
> files.
> When I export to a csv, a quote mark is appended at the beginning and end
> of each line. I'm using the default device info settings. This is not
> the Qualifier, because I tried changing that from ["] to other characters
> in the devinfo settings, and the ["] is still inserted.
> How can I prevent the ["] from being inserted? It causes problems when I
> open the file in Excel, because Excel thinks it means the whole row is one
> field, and it gets truncated after 256 characters.
> Thanks!
> Bill
>
>
>

Friday, March 9, 2012

Quick Question Importing

Two Quick Question's,

Are you better off importing data through Excel or through Text files in terms of ease of use \ Speed \ Efficency etc or does it make a differance ?

Also if I am loading data into a SQL database should I always use the "SQL Server Destination" rather than the "OLE DB Destination"

Hope Q's aren't too basic, both seem to work for me, but I just want to make sure im using the right one.

Thank you

You may find text files a little easier than excel files, because in case of excel files, you may need to do a transformation from unicode to non-unicode characters.

For your second question, SQL Server destination can be used only if you are loading the data to the local server. It can't be used for a remote server. When using a local server, SQL Server Destination, will have a performance benefit over OLEDB Destination. But if you are working with a remote server, OLEDB destination is the only one you can use.

|||

Thank you very much Ranjeeta that was exactly what I was looking for :)

Just as an afterthought in the case of Excel "you may need to do a transformation from unicode to non-unicode characters. "

Is this tricky to do ? or would it be a case of using "Data Conversion" to change the datatypes from unicode to non unicode charactors

Again thanks

|||Not tricky. Just the matter of using a "Data conversion", so just an additional step.|||I would recommend not using Excel to much... You might get into trouble because of Excel interpreting fields sometimes as text sometimes as numbers... That's not always a problem (depends on the structure of you tables) and it's nothing making Excel as a source impossible but it can take you some time to get it working...|||

Thanks for the help, I have opted to not use excel and have resorted instead to text files or csv files.

However I still seem to be running into the same type of problem. All the text being imported weither text, date's or numbers all come in as Data Type "DT_STR" Code 1252 (not sure if the code bit is relevant).

I have then inserted a Data Conversion task to try and change the datatypes but none of it seems to work, I keep getting the same type of error

"Error at Data Flow [SQL Server Destination[9]]: The Column "Copy of Column 0" can't be inserted because the conversion between types DT_R8 and DT_STR is not supported."

Error at Data Flow task [DTS Pipeline]: "Componant SQL Server Destination" (9) failed validation and returned status "VS_ISBROKEN"

I have also tried DT_R4, DT_u18, DT_Date etc all with the same errror.

Ive worked on this for the last few days now but it always seem to be the same error so Im second guessing everything now trying to find a solution.

Surely I should be using "Data Conversion" to change the data types inbetween my source and destination column. ?

Im sure its user error somehow but I cant understand whats going wrong

|||Did you set the field's properties correctly in the data source adapter? If you did so we have to sort out where the pipeline changes from having correct metadata to only having strings there... Check the output's settings from the data source and the pipeline's metadata just after the data source... If there are only strings (and you have set the field's properties) perhaps recreate the data source adapter for the text file...|||

Sorry for the Delay in getting back to you Thomas I have been trying to figure out where I have been running into errors so I deleted all and am in the process of building them back up again.

When I rebuilt the flat file source connection I used Suggest Data Types for the following Sample Data
10012 C961910 1 380.92 12/05/1997 IE HELENA 1
10013 C961711 3 3174.35 27/07/1999 FR MARYM 7
10013 C961711 3 2459.23 25/05/1999 FR MAURA 6

and these are the data types it suggested I should use

Column 0 eight-byte signed integer [DT_18]

Column 1 string [DT_STR]

Column 2 eight-byte signed integer [DT_18]

Column 3 double-precision float [DT_R8]

Column 4 string [DT_STR]

Column 5 string [DT_STR]

Column 6 eight-byte signed integer [DT_18]

Im a little cautous of "string [DT_STR] so I thought maybe the following would be more accurate ? although I didnt use them in this instance but when I do I get the same error

eight-byte unsigned integer [DT_U18]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]
currency [DT_CY]
date [DT_DATE]
text stream [DT_TEXT]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]

It then goes straight to the "SQL Destination" and create a table

CREATE TABLE [SQL Server Destination] (
[Column 0] BIGINT,
[Column 1] VARCHAR(7),
[Column 2] BIGINT,
[Column 3] DOUBLE PRECISION,
[Column 4] DATETIME,
[Column 5] VARCHAR(2),
[Column 6] VARCHAR(7),
[Column 7] BIGINT
)

Then when I run the package it gives me the following errors

"Error at data flow task SQL Server Destination 75: The Column "Column 4" can't be inserted because the conversion between types DT_DATE and DT_DBTIMESTAMP is not supported.
"Error at Data flow task [DTS PIPELINE] :Compnant "SQL Server Destination" 75 Faileds validation and returned validation status "VS_ISBROKEN"

Any idea's ?

And thanks for all your help on this :)

|||Did you try using DT_DBTIMESTAMP instead of DT_DATE in your source's output?|||

Yes it seems no matter what datatype I give that column

database date [DT_DBDATE]
database time [DT_DBTIME]
database timestamp [DT_DBTIMESTAMP]
date [DT_DATE]

They all come back with the same error

|||

Is it possible to use something like the following to edit the date format ?

http://sqlservercode.blogspot.com/2005/09/date-formatting-in-sql-server_21.html

I tried to use it but I couldn't get it working at all

|||

Something what should work is:

- import the date as string

- use an expression in a "Derived Column" transform to cut the date into pieces, rearange it to a format wórking with SQL Server and convert to a "working" datatype

I wrote already about that: http://forums.microsoft.com/MSDN/showpost.aspx?postid=199322&siteid=1

|||

Perfect Thomas thanks so much for your help

In the end the derived column between the source and destination fixed 90% of my woes :) which is excellent news for me

Again thank you, I appreciate it.

|||

Also the excel worksheet has a limit of 65536 recrods where as flat file has a limit of one million.

Quick Question Importing

Two Quick Question's,

Are you better off importing data through Excel or through Text files in terms of ease of use \ Speed \ Efficency etc or does it make a differance ?

Also if I am loading data into a SQL database should I always use the "SQL Server Destination" rather than the "OLE DB Destination"

Hope Q's aren't too basic, both seem to work for me, but I just want to make sure im using the right one.

Thank you

You may find text files a little easier than excel files, because in case of excel files, you may need to do a transformation from unicode to non-unicode characters.

For your second question, SQL Server destination can be used only if you are loading the data to the local server. It can't be used for a remote server. When using a local server, SQL Server Destination, will have a performance benefit over OLEDB Destination. But if you are working with a remote server, OLEDB destination is the only one you can use.

|||

Thank you very much Ranjeeta that was exactly what I was looking for :)

Just as an afterthought in the case of Excel "you may need to do a transformation from unicode to non-unicode characters. "

Is this tricky to do ? or would it be a case of using "Data Conversion" to change the datatypes from unicode to non unicode charactors

Again thanks

|||Not tricky. Just the matter of using a "Data conversion", so just an additional step.|||I would recommend not using Excel to much... You might get into trouble because of Excel interpreting fields sometimes as text sometimes as numbers... That's not always a problem (depends on the structure of you tables) and it's nothing making Excel as a source impossible but it can take you some time to get it working...|||

Thanks for the help, I have opted to not use excel and have resorted instead to text files or csv files.

However I still seem to be running into the same type of problem. All the text being imported weither text, date's or numbers all come in as Data Type "DT_STR" Code 1252 (not sure if the code bit is relevant).

I have then inserted a Data Conversion task to try and change the datatypes but none of it seems to work, I keep getting the same type of error

"Error at Data Flow [SQL Server Destination[9]]: The Column "Copy of Column 0" can't be inserted because the conversion between types DT_R8 and DT_STR is not supported."

Error at Data Flow task [DTS Pipeline]: "Componant SQL Server Destination" (9) failed validation and returned status "VS_ISBROKEN"

I have also tried DT_R4, DT_u18, DT_Date etc all with the same errror.

Ive worked on this for the last few days now but it always seem to be the same error so Im second guessing everything now trying to find a solution.

Surely I should be using "Data Conversion" to change the data types inbetween my source and destination column. ?

Im sure its user error somehow but I cant understand whats going wrong

|||Did you set the field's properties correctly in the data source adapter? If you did so we have to sort out where the pipeline changes from having correct metadata to only having strings there... Check the output's settings from the data source and the pipeline's metadata just after the data source... If there are only strings (and you have set the field's properties) perhaps recreate the data source adapter for the text file...|||

Sorry for the Delay in getting back to you Thomas I have been trying to figure out where I have been running into errors so I deleted all and am in the process of building them back up again.

When I rebuilt the flat file source connection I used Suggest Data Types for the following Sample Data
10012 C961910 1 380.92 12/05/1997 IE HELENA 1
10013 C961711 3 3174.35 27/07/1999 FR MARYM 7
10013 C961711 3 2459.23 25/05/1999 FR MAURA 6

and these are the data types it suggested I should use

Column 0 eight-byte signed integer [DT_18]

Column 1 string [DT_STR]

Column 2 eight-byte signed integer [DT_18]

Column 3 double-precision float [DT_R8]

Column 4 string [DT_STR]

Column 5 string [DT_STR]

Column 6 eight-byte signed integer [DT_18]

Im a little cautous of "string [DT_STR] so I thought maybe the following would be more accurate ? although I didnt use them in this instance but when I do I get the same error

eight-byte unsigned integer [DT_U18]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]
currency [DT_CY]
date [DT_DATE]
text stream [DT_TEXT]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]

It then goes straight to the "SQL Destination" and create a table

CREATE TABLE [SQL Server Destination] (
[Column 0] BIGINT,
[Column 1] VARCHAR(7),
[Column 2] BIGINT,
[Column 3] DOUBLE PRECISION,
[Column 4] DATETIME,
[Column 5] VARCHAR(2),
[Column 6] VARCHAR(7),
[Column 7] BIGINT
)

Then when I run the package it gives me the following errors

"Error at data flow task SQL Server Destination 75: The Column "Column 4" can't be inserted because the conversion between types DT_DATE and DT_DBTIMESTAMP is not supported.
"Error at Data flow task [DTS PIPELINE] :Compnant "SQL Server Destination" 75 Faileds validation and returned validation status "VS_ISBROKEN"

Any idea's ?

And thanks for all your help on this :)

|||Did you try using DT_DBTIMESTAMP instead of DT_DATE in your source's output?|||

Yes it seems no matter what datatype I give that column

database date [DT_DBDATE]
database time [DT_DBTIME]
database timestamp [DT_DBTIMESTAMP]
date [DT_DATE]

They all come back with the same error

|||

Is it possible to use something like the following to edit the date format ?

http://sqlservercode.blogspot.com/2005/09/date-formatting-in-sql-server_21.html

I tried to use it but I couldn't get it working at all

|||

Something what should work is:

- import the date as string

- use an expression in a "Derived Column" transform to cut the date into pieces, rearange it to a format wórking with SQL Server and convert to a "working" datatype

I wrote already about that: http://forums.microsoft.com/MSDN/showpost.aspx?postid=199322&siteid=1

|||

Perfect Thomas thanks so much for your help

In the end the derived column between the source and destination fixed 90% of my woes :) which is excellent news for me

Again thank you, I appreciate it.

|||

Also the excel worksheet has a limit of 65536 recrods where as flat file has a limit of one million.

Quick Question Importing

Two Quick Question's,

Are you better off importing data through Excel or through Text files in terms of ease of use \ Speed \ Efficency etc or does it make a differance ?

Also if I am loading data into a SQL database should I always use the "SQL Server Destination" rather than the "OLE DB Destination"

Hope Q's aren't too basic, both seem to work for me, but I just want to make sure im using the right one.

Thank you

You may find text files a little easier than excel files, because in case of excel files, you may need to do a transformation from unicode to non-unicode characters.

For your second question, SQL Server destination can be used only if you are loading the data to the local server. It can't be used for a remote server. When using a local server, SQL Server Destination, will have a performance benefit over OLEDB Destination. But if you are working with a remote server, OLEDB destination is the only one you can use.

|||

Thank you very much Ranjeeta that was exactly what I was looking for :)

Just as an afterthought in the case of Excel "you may need to do a transformation from unicode to non-unicode characters. "

Is this tricky to do ? or would it be a case of using "Data Conversion" to change the datatypes from unicode to non unicode charactors

Again thanks

|||Not tricky. Just the matter of using a "Data conversion", so just an additional step.|||I would recommend not using Excel to much... You might get into trouble because of Excel interpreting fields sometimes as text sometimes as numbers... That's not always a problem (depends on the structure of you tables) and it's nothing making Excel as a source impossible but it can take you some time to get it working...|||

Thanks for the help, I have opted to not use excel and have resorted instead to text files or csv files.

However I still seem to be running into the same type of problem. All the text being imported weither text, date's or numbers all come in as Data Type "DT_STR" Code 1252 (not sure if the code bit is relevant).

I have then inserted a Data Conversion task to try and change the datatypes but none of it seems to work, I keep getting the same type of error

"Error at Data Flow [SQL Server Destination[9]]: The Column "Copy of Column 0" can't be inserted because the conversion between types DT_R8 and DT_STR is not supported."

Error at Data Flow task [DTS Pipeline]: "Componant SQL Server Destination" (9) failed validation and returned status "VS_ISBROKEN"

I have also tried DT_R4, DT_u18, DT_Date etc all with the same errror.

Ive worked on this for the last few days now but it always seem to be the same error so Im second guessing everything now trying to find a solution.

Surely I should be using "Data Conversion" to change the data types inbetween my source and destination column. ?

Im sure its user error somehow but I cant understand whats going wrong

|||Did you set the field's properties correctly in the data source adapter? If you did so we have to sort out where the pipeline changes from having correct metadata to only having strings there... Check the output's settings from the data source and the pipeline's metadata just after the data source... If there are only strings (and you have set the field's properties) perhaps recreate the data source adapter for the text file...|||

Sorry for the Delay in getting back to you Thomas I have been trying to figure out where I have been running into errors so I deleted all and am in the process of building them back up again.

When I rebuilt the flat file source connection I used Suggest Data Types for the following Sample Data
10012 C961910 1 380.92 12/05/1997 IE HELENA 1
10013 C961711 3 3174.35 27/07/1999 FR MARYM 7
10013 C961711 3 2459.23 25/05/1999 FR MAURA 6

and these are the data types it suggested I should use

Column 0 eight-byte signed integer [DT_18]

Column 1 string [DT_STR]

Column 2 eight-byte signed integer [DT_18]

Column 3 double-precision float [DT_R8]

Column 4 string [DT_STR]

Column 5 string [DT_STR]

Column 6 eight-byte signed integer [DT_18]

Im a little cautous of "string [DT_STR] so I thought maybe the following would be more accurate ? although I didnt use them in this instance but when I do I get the same error

eight-byte unsigned integer [DT_U18]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]
currency [DT_CY]
date [DT_DATE]
text stream [DT_TEXT]
text stream [DT_TEXT]
eight-byte unsigned integer [DT_U18]

It then goes straight to the "SQL Destination" and create a table

CREATE TABLE [SQL Server Destination] (
[Column 0] BIGINT,
[Column 1] VARCHAR(7),
[Column 2] BIGINT,
[Column 3] DOUBLE PRECISION,
[Column 4] DATETIME,
[Column 5] VARCHAR(2),
[Column 6] VARCHAR(7),
[Column 7] BIGINT
)

Then when I run the package it gives me the following errors

"Error at data flow task SQL Server Destination 75: The Column "Column 4" can't be inserted because the conversion between types DT_DATE and DT_DBTIMESTAMP is not supported.
"Error at Data flow task [DTS PIPELINE] :Compnant "SQL Server Destination" 75 Faileds validation and returned validation status "VS_ISBROKEN"

Any idea's ?

And thanks for all your help on this :)

|||Did you try using DT_DBTIMESTAMP instead of DT_DATE in your source's output?|||

Yes it seems no matter what datatype I give that column

database date [DT_DBDATE]
database time [DT_DBTIME]
database timestamp [DT_DBTIMESTAMP]
date [DT_DATE]

They all come back with the same error

|||

Is it possible to use something like the following to edit the date format ?

http://sqlservercode.blogspot.com/2005/09/date-formatting-in-sql-server_21.html

I tried to use it but I couldn't get it working at all

|||

Something what should work is:

- import the date as string

- use an expression in a "Derived Column" transform to cut the date into pieces, rearange it to a format wórking with SQL Server and convert to a "working" datatype

I wrote already about that: http://forums.microsoft.com/MSDN/showpost.aspx?postid=199322&siteid=1

|||

Perfect Thomas thanks so much for your help

In the end the derived column between the source and destination fixed 90% of my woes :) which is excellent news for me

Again thank you, I appreciate it.

|||

Also the excel worksheet has a limit of 65536 recrods where as flat file has a limit of one million.

Wednesday, March 7, 2012

Quick PDF Generation

Are the PDF files that rs generate protected? Can someone with Adobe Acrobat
modify them?no and yes. It´s kind of Adove 4.0 Format.
HTH, Jens Suessmeyer.
"Alex" <Alex@.discussions.microsoft.com> schrieb im Newsbeitrag
news:32F38409-7BCD-44B3-B47B-165E93462EBF@.microsoft.com...
> Are the PDF files that rs generate protected? Can someone with Adobe
> Acrobat
> modify them?

Saturday, February 25, 2012

Queuing Files

I have a requirement of instantiating a job which will copy and move the
newly created file once a file has been created in a folder. How can I
perform this in SQL Server. Can you please give your thoughts about this.
What I was planning was to write a visual basic code which will pool the
files getting created into the folder and accordingly copy and paste into
another folder. I feel this is not a better method.
I really wanted a queuing system to be implemented.Hi, there's a script I am using to copy files:
USE [msdb]
GO
/****** Object: Job [CopyToCluster] Script Date: 03/20/2006 15:24:31 ******/
BEGIN TRANSACTION
DECLARE @.ReturnCode INT
SELECT @.ReturnCode = 0
/****** Object: JobCategory [Database Maintenance] Script Date: 03/20/2006
15:24:31 ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'Database
Maintenance' AND category_class=1)
BEGIN
EXEC @.ReturnCode = msdb.dbo.sp_add_category @.class=N'JOB', @.type=N'LOCAL',
@.name=N'Database Maintenance'
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
END
DECLARE @.jobId BINARY(16)
EXEC @.ReturnCode = msdb.dbo.sp_add_job @.job_name=N'CopyToCluster',
@.enabled=1,
@.notify_level_eventlog=2,
@.notify_level_email=0,
@.notify_level_netsend=0,
@.notify_level_page=0,
@.delete_level=0,
@.description=N'Copies all files rom the local backup directory to
\\datacluster\\Backup',
@.category_name=N'Database Maintenance',
@.owner_login_name=N'YOUR_USER', @.job_id = @.jobId OUTPUT
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Copy files] Script Date: 03/20/2006 15:24:31 ******/
EXEC @.ReturnCode = msdb.dbo.sp_add_jobstep @.job_id=@.jobId, @.step_name=N'Copy
files',
@.step_id=1,
@.cmdexec_success_code=0,
@.on_success_action=1,
@.on_success_step_id=0,
@.on_fail_action=2,
@.on_fail_step_id=0,
@.retry_attempts=2,
@.retry_interval=0,
@.os_run_priority=0, @.subsystem=N'CmdExec',
@.command=N'c:\copybackups.cmd $(DATE)$(TIME)',
@.flags=4
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
EXEC @.ReturnCode = msdb.dbo.sp_update_job @.job_id = @.jobId, @.start_step_id =
1
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
EXEC @.ReturnCode = msdb.dbo.sp_add_jobschedule @.job_id=@.jobId,
@.name=N'CopyBackupFilesSched',
@.enabled=1,
@.freq_type=4,
@.freq_interval=1,
@.freq_subday_type=1,
@.freq_subday_interval=0,
@.freq_relative_interval=0,
@.freq_recurrence_factor=0,
@.active_start_date=20060106,
@.active_end_date=99991231,
@.active_start_time=50000,
@.active_end_time=235959
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
EXEC @.ReturnCode = msdb.dbo.sp_add_jobserver @.job_id = @.jobId, @.server_name
= N'(local)'
IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
COMMIT TRANSACTION
GOTO EndSave
QuitWithRollback:
IF (@.@.TRANCOUNT > 0) ROLLBACK TRANSACTION
EndSave:
and this is copypackups.cmd:
rem PR 2006-01-09
rem This batch file copies daily backups of databases to network location
rem
@.echo . > c:\copybackups.log
@.echo %1 Copying files from c:\SQLBackup to \\datacluster\Backup... >>
c:\copybackups.log
xcopy c:\SQLBackup\*.* \\datacluster\Backup\*.* /L /R /H /D /V /Y /F /C >>
c:\copybackups.log
xcopy c:\SQLBackup\*.* \\datacluster\Backup\*.* /R /H /D /V /Y /F /C >>
c:\copybackups.log
@.echo done >> c:\copybackups.log
@.echo . >> c:\copybackups.log
Note that this file does not use date and time passed by the job. I have
another job that removes older files from local directory.
HTH
Peter|||Thank you very much for your response.
What I need is watch for newly added files to a folder. The moment a new
file is added, I need to copy and paste into another location. This is
something like a service which keeps on monitoring a folder for new files
coming in.
Finally once the day is fininshed I will have a copy of the master folder in
another location also. I need to do this on receipt of each file, not finall
y
at the end of a day.
Thanks & Regards,
VB Babunath
"Rogas69" wrote:

> Hi, there's a script I am using to copy files:
> USE [msdb]
> GO
> /****** Object: Job [CopyToCluster] Script Date: 03/20/2006 15:24:31 ******/
> BEGIN TRANSACTION
> DECLARE @.ReturnCode INT
> SELECT @.ReturnCode = 0
> /****** Object: JobCategory [Database Maintenance] Script Date: 03/20/2006
> 15:24:31 ******/
> IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'Databa
se
> Maintenance' AND category_class=1)
> BEGIN
> EXEC @.ReturnCode = msdb.dbo.sp_add_category @.class=N'JOB', @.type=N'LOCAL',
> @.name=N'Database Maintenance'
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> END
> DECLARE @.jobId BINARY(16)
> EXEC @.ReturnCode = msdb.dbo.sp_add_job @.job_name=N'CopyToCluster',
> @.enabled=1,
> @.notify_level_eventlog=2,
> @.notify_level_email=0,
> @.notify_level_netsend=0,
> @.notify_level_page=0,
> @.delete_level=0,
> @.description=N'Copies all files rom the local backup directory to
> \\datacluster\\Backup',
> @.category_name=N'Database Maintenance',
> @.owner_login_name=N'YOUR_USER', @.job_id = @.jobId OUTPUT
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> /****** Object: Step [Copy files] Script Date: 03/20/2006 15:24:31 ******/
> EXEC @.ReturnCode = msdb.dbo.sp_add_jobstep @.job_id=@.jobId, @.step_name=N'Co
py
> files',
> @.step_id=1,
> @.cmdexec_success_code=0,
> @.on_success_action=1,
> @.on_success_step_id=0,
> @.on_fail_action=2,
> @.on_fail_step_id=0,
> @.retry_attempts=2,
> @.retry_interval=0,
> @.os_run_priority=0, @.subsystem=N'CmdExec',
> @.command=N'c:\copybackups.cmd $(DATE)$(TIME)',
> @.flags=4
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> EXEC @.ReturnCode = msdb.dbo.sp_update_job @.job_id = @.jobId, @.start_step_id
=
> 1
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> EXEC @.ReturnCode = msdb.dbo.sp_add_jobschedule @.job_id=@.jobId,
> @.name=N'CopyBackupFilesSched',
> @.enabled=1,
> @.freq_type=4,
> @.freq_interval=1,
> @.freq_subday_type=1,
> @.freq_subday_interval=0,
> @.freq_relative_interval=0,
> @.freq_recurrence_factor=0,
> @.active_start_date=20060106,
> @.active_end_date=99991231,
> @.active_start_time=50000,
> @.active_end_time=235959
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> EXEC @.ReturnCode = msdb.dbo.sp_add_jobserver @.job_id = @.jobId, @.server_nam
e
> = N'(local)'
> IF (@.@.ERROR <> 0 OR @.ReturnCode <> 0) GOTO QuitWithRollback
> COMMIT TRANSACTION
> GOTO EndSave
> QuitWithRollback:
> IF (@.@.TRANCOUNT > 0) ROLLBACK TRANSACTION
> EndSave:
>
> and this is copypackups.cmd:
> rem PR 2006-01-09
> rem This batch file copies daily backups of databases to network location
> rem
> @.echo . > c:\copybackups.log
> @.echo %1 Copying files from c:\SQLBackup to \\datacluster\Backup... >>
> c:\copybackups.log
> xcopy c:\SQLBackup\*.* \\datacluster\Backup\*.* /L /R /H /D /V /Y /F /C >>
> c:\copybackups.log
> xcopy c:\SQLBackup\*.* \\datacluster\Backup\*.* /R /H /D /V /Y /F /C >>
> c:\copybackups.log
> @.echo done >> c:\copybackups.log
> @.echo . >> c:\copybackups.log
>
> Note that this file does not use date and time passed by the job. I have
> another job that removes older files from local directory.
> HTH
> Peter
>
>