In order to keep a streaming tape streaming it must be fed data at a
very high rate of speed. When that data is coming from a remote system
that's generally impossible to do unless a program like Lee McLoughlin's
"buffer" is used. The thing is, buffer only allows a maximum buffer of
around 16Mb. I've found that with an SDLT320 that only ends up
streaming the drive for a few seconds, for instance like this (linux
system with tape drive, remote linux, same sort of thing if remote
is Solaris):

ssh root@$REMOTE "dump -0 -q -B 2000000000 -f - -u $TPARTITION | gzip
-3 -c" <dev/null 2>>$BLOG | \
buffer -m 16384k -p 75 -B -o /dev/nst0 2>>$BLOG

What I'm looking for is a program that can buffer to two disk files of
say 1G each, filling one while writing the other to tape.
Probably these should be on two different disks. Anyway, the idea
is that when the buffer file is filled it can easily stream the full
1Gb to tape. The tape spends much more time streaming, and much, much
less time rewinding and waiting - which wears both the drive and the
tape. If a 16M buffer results in 5 seconds of streaming then a 1Gb
buffer file should be around 5 minutes of streaming.

The question is - is there such a program around already? I've googled
but keywards like "file" and "buffer" are so common that it's hard to
find anything.

Thanks,

David Mathog

David Mathog wrote:
> In order to keep a streaming tape streaming it must be fed data at a
> very high rate of speed. When that data is coming from a remote system
> that's generally impossible to do unless a program like Lee McLoughlin's
> "buffer" is used. The thing is, buffer only allows a maximum buffer of
> around 16Mb. I've found that with an SDLT320 that only ends up
> streaming the drive for a few seconds, for instance like this (linux
> system with tape drive, remote linux, same sort of thing if remote
> is Solaris):
>
> ssh root@$REMOTE "dump -0 -q -B 2000000000 -f - -u $TPARTITION | gzip
> -3 -c" <dev/null 2>>$BLOG | \
> buffer -m 16384k -p 75 -B -o /dev/nst0 2>>$BLOG
>
> What I'm looking for is a program that can buffer to two disk files of
> say 1G each, filling one while writing the other to tape.
> Probably these should be on two different disks. Anyway, the idea
> is that when the buffer file is filled it can easily stream the full
> 1Gb to tape. The tape spends much more time streaming, and much, much
> less time rewinding and waiting - which wears both the drive and the
> tape. If a 16M buffer results in 5 seconds of streaming then a 1Gb
> buffer file should be around 5 minutes of streaming.
>
> The question is - is there such a program around already? I've googled
> but keywards like "file" and "buffer" are so common that it's hard to
> find anything.



You either need to convert you network to gigabit

or

if the remote system is on the other side of a WAN I think
you should look aroud for a WAN link file replication solution
that will replicate you remote files compleately to local storage
Symantec/Veritas does this type of software , IBM Tivoli also
, and ofcouse all major storage vendors have software that
does this with their hardware.

This type of software usully work by copying changed diskblocs
from the remote systems to local storage, therby preventing
the transfer of complete files when a record in a file or database
is updated.


//Lars

David Mathog wrote:

> In order to keep a streaming tape streaming it must be fed data at a
> very high rate of speed. When that data is coming from a remote system
> that's generally impossible to do unless a program like Lee McLoughlin's
> "buffer" is used. The thing is, buffer only allows a maximum buffer of
> around 16Mb. I've found that with an SDLT320 that only ends up
> streaming the drive for a few seconds, for instance like this (linux
> system with tape drive, remote linux, same sort of thing if remote
> is Solaris):
>
> ssh root@$REMOTE "dump -0 -q -B 2000000000 -f - -u $TPARTITION | gzip
> -3 -c" <dev/null 2>>$BLOG | \
> buffer -m 16384k -p 75 -B -o /dev/nst0 2>>$BLOG
>
> What I'm looking for is a program that can buffer to two disk files of
> say 1G each, filling one while writing the other to tape.
> Probably these should be on two different disks. Anyway, the idea
> is that when the buffer file is filled it can easily stream the full
> 1Gb to tape. The tape spends much more time streaming, and much, much
> less time rewinding and waiting - which wears both the drive and the
> tape. If a 16M buffer results in 5 seconds of streaming then a 1Gb
> buffer file should be around 5 minutes of streaming.
>
> The question is - is there such a program around already? I've googled
> but keywards like "file" and "buffer" are so common that it's hard to
> find anything.
>
> Thanks,
>
> David Mathog

How did you wind up with a tape drive that needs a higher data rate than
your system can supply? Given what tape drives, especially high
performance tape drives, cost it seems like a waste of money. If you
manage to provide a gigabyte buffer, your tape drive will stream for
five minutes, sit idle for X minutes, stream for five minutes. . . .

Or am I missing something?

In article <ebqjhc$l0t$1@naig.caltech.edu>,
David Mathog <mathog@caltech.edu> wrote:
>In order to keep a streaming tape streaming it must be fed data at a
>very high rate of speed. When that data is coming from a remote system
>that's generally impossible to do unless a program like Lee McLoughlin's
>"buffer" is used. The thing is, buffer only allows a maximum buffer of
>around 16Mb.

Thomas Maier-Komor's mbuffer program expands upon buffer. It supports
large memory buffers and also appears to support disk based buffers.
http://www.maier-komor.de/mbuffer.html

Another approach is to go to a disk to disk to tape backup. Setup a
staging disk as large as your largest partition needing to be backed up;
striping several disks together for performance and capacity might be a
good idea. Your dump just creates a single large file in the staging area
and when it is done you use buffer/mbuffer to transfer it to tape.
--
Jim Prescott - Computing and Networking Group jgp@seas.rochester.edu
School of Engineering and Applied Sciences, University of Rochester, NY

Richard B. Gilbert wrote:

> How did you wind up with a tape drive that needs a higher data rate than
> your system can supply?

The data source is coming over the network, local disk backups don't
need any of this hocus pocus and the drive streams nicely along.

> Given what tape drives, especially high
> performance tape drives, cost it seems like a waste of money.

Nope, this is saving money by using one tape drive to
back up multiple systems.

> If you
> manage to provide a gigabyte buffer, your tape drive will stream for
> five minutes, sit idle for X minutes, stream for five minutes. . . .

Exactly what I'm looking for. It's now spending about half its time
rewinding at short intervals. It would be much better if it wrote
continuously for 5 minutes and then sat idle for 5 minutes if the total
throughput was about the same since it would cause less wear on the
tapes and tape drive.

The tape drive streams at 16Mb/sec but the maximum data rate over
the 100baseT connection is 100/8 = 12.5Mb/sec, in the real world it's
probably around 12Mb. So the tape drive outruns the network by about
4Mb/sec. There's nothing I can do about that since changing the
network isn't an option. The original buffer only provides a
16Mb buffer which guarantees a tape drive stall ever 4 seconds (or
so, depending on the percent settings.)

Regards,

David Mathog

Jim Prescott wrote:

> Thomas Maier-Komor's mbuffer program expands upon buffer. It supports
> large memory buffers and also appears to support disk based buffers.
> http://www.maier-komor.de/mbuffer.html

That seems to be pretty much what I was looking for. I'll give it a try.

Thanks!

David Mathog

David Mathog wrote:

> Richard B. Gilbert wrote:
>
>> How did you wind up with a tape drive that needs a higher data rate
>> than your system can supply?
>
>
> The data source is coming over the network, local disk backups don't
> need any of this hocus pocus and the drive streams nicely along.
>
>> Given what tape drives, especially high performance tape drives, cost
>> it seems like a waste of money.
>
>
> Nope, this is saving money by using one tape drive to
> back up multiple systems.
>
>> If you manage to provide a gigabyte buffer, your tape drive will
>> stream for five minutes, sit idle for X minutes, stream for five
>> minutes. . . .
>
>
> Exactly what I'm looking for. It's now spending about half its time
> rewinding at short intervals. It would be much better if it wrote
> continuously for 5 minutes and then sat idle for 5 minutes if the total
> throughput was about the same since it would cause less wear on the
> tapes and tape drive.
>
> The tape drive streams at 16Mb/sec but the maximum data rate over
> the 100baseT connection is 100/8 = 12.5Mb/sec, in the real world it's
> probably around 12Mb. So the tape drive outruns the network by about
> 4Mb/sec. There's nothing I can do about that since changing the
> network isn't an option. The original buffer only provides a
> 16Mb buffer which guarantees a tape drive stall ever 4 seconds (or
> so, depending on the percent settings.)
>
> Regards,
>
> David Mathog

Can you "stage" your backup to a local disk and then copy it to tape?
It could require a honkin big disk that might cost more than a couple of
gigabit ethernet NICs. . . . BTW you don't have to upgrade the whole
network; one GB NIC per system and a crossover cable could make a BIG
difference.

Richard B. Gilbert wrote:
> Can you "stage" your backup to a local disk and then copy it to tape?

Not in general. One of the partitions is over 90Gb, which is bigger
than all of the disks on the backup system. mbuffer (see another post
in this thread) allows for 5Gb or so buffer file, and that should be
enough to eliminate 99% of the rewinds.

Thanks,

David Mathog