On Tue, 23 May 2006 15:15:46 +0000 (UTC), Sylvain Robitaille <syl@alcor.concordia.ca> wrote:

[suggested]
> tar cf - -C /some/path . |ssh somehost "(cd /new/path && tar xpvf -)"

Alright, done it the safe'n'secure way

After the usual cp -al kernel source, config, compile, then:

grant@sempro:~/linux/hal/linux-2.6.16.18a$ time target-install

target-install: Compile target: 'hal', Kernel version: '2.6.16.18a'
Remove old module files in /home/hal/lib/modules/2.6.16.18a, done.
run 'make install'
....
* Writing new kernel files to /home/hal/boot

target-install: transfer kernel and modules to hal

root@hal's password:
boot/bzImage-2.6.16.18a
boot/System.map-2.6.16.18a
boot/config-2.6.16.18a
lib/modules/2.6.16.18a/
lib/modules/2.6.16.18a/modules.usbmap
....
target-install: Done!

real 0m11.351s
user 0m1.120s
sys 0m0.260s

includes time taken to type in target root's password.

root@hal:~# ls -lrt /boot/
....
-rw-r--r-- 1 root root 21843 2006-05-24 19:28 config-2.6.16.18a
-rw-r--r-- 1 root root 1134140 2006-05-24 19:28 bzImage-2.6.16.18a
-rw-r--r-- 1 root root 538846 2006-05-24 19:28 System.map-2.6.16.18a
root@hal:~# ls -lrt /lib/modules/
total 4
drwxr-xr-x 4 root root 1024 2005-06-09 07:40 2.4.31/
drwxr-xr-x 4 root root 1024 2006-05-23 17:04 2.4.33-pre3/
drwxr-xr-x 4 root root 1024 2006-05-23 17:06 2.4.32-hf32.4/
drwxr-xr-x 3 root root 1024 2006-05-24 19:28 2.6.16.18a/

The target-install bash script performs sanity checking, cleans
out staging area module files, then:

# perform 'make install' to staging area
echo " run 'make install'"
INSTALL_PATH=$TARGET/$HOST/boot INSTALL_MOD_PATH=$TARGET/$HOST make install

# perform kernel transfer to target
echo -e "\n$ME: transfer kernel and modules to $HOST\n"

tar cf - -C $TARGET/$HOST $KERNEL $SYSTEM $CONFIG $MODULES | \
ssh root@$HOST "( \
cd / && \
rm -rf /lib/modules/$VERSION && \
tar xvf - && \
chown root:root /boot/*-$VERSION && \
chown -R root:root /lib/modules/$VERSION \
)"

echo -e "\n$ME: Done!\n"
# end

Works as advertised! Not yet tested for 2.4 series yet as
installkernel script needs some work. The chown to root at
the target is because I've decided to force host compile and
install as non-root to avoid overwriting the compile host's
kernel. (yes, I've done that by accident

The compile target exports nothing to NFS, the compile host
exports /home/<target> work area so I can copy the target's
/boot/config*, this NFS export is mounted and written to from
the target as required. Better now? 8-)

Curious? See the script at:
<http://bugsplatter.mine.nu/bash/kernel/target-install.gz>

Grant.
--
WinXP: Access Start->Turn Off Computer, then while holding Ctrl-Alt-Shift,
left click on Cancel. This terminates Windows Explorer...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In alt.os.linux.slackware, Grant dared to utter,
> Never used netcat, I'd use rsync for casual large transfers, or
> the usual (to me) cp -a ... over NFS.

rsync is good, but cp over NFS is crap! You can easily spend hours
upon hours moving thousands of files which have only a couple of bytes
in them when tar | nc would handle this in just a few minutes time.

I used to use NFS for copying large amounts of data. I started one
such process some time ago, and went home. When I came back the next
day, it was still running. I ran a tar to nc and the entire 120GB was
moved in something like half an hour.

- --
It is better to hear the rebuke of the wise,
Than for a man to hear the song of fools.
Ecclesiastes 7:5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.7 (GNU/Linux)

iD8DBQFEdGF9z8zcalmVmBkRApIAAJ9u4JloNIIzhtUp3N232B BSi/D+ZACgvkbm
FyixalOxJP7LpDg10YXxZQc=
=iYc0
-----END PGP SIGNATURE-----

On Wed, 24 May 2006 09:35:41 -0500, +Alan Hicks+ <alan@lizella.netWORK> wrote:

>rsync is good, but cp over NFS is crap! You can easily spend hours
>upon hours moving thousands of files which have only a couple of bytes
>in them when tar | nc would handle this in just a few minutes time.

Crap, just shows you dunno how to setup NFS

~$ cat /etc/fstab
# /etc/fstab for slackware on sempro -- 2006-04-02
....
peetoo:/home/install /home/install nfs noauto,user,hard,intr
peetoo:/home/public /home/public nfs noauto,user,hard,intr
....
# end
grant@sempro:~$ mount /home/install/
grant@sempro:~$ ls -l /home/install/test578.iso
-rw-r--r-- 1 grant wheel 578408448 2006-04-26 19:08 /home/install/test578.iso
grant@sempro:~$ time cat /home/install/test578.iso >/dev/null

real 0m54.260s <<== 10.7 MBytes/second via 100Mbps localnet over NFS
user 0m0.010s
sys 0m1.260s
grant@sempro:~$

Grant.
--
WinXP: Access Start->Turn Off Computer, then while holding Ctrl-Alt-Shift,
left click on Cancel. This terminates Windows Explorer...

Grant wrote:

> Alright, done it the safe'n'secure way

:-)

> Works as advertised!

I knew it would!

> Not yet tested for 2.4 series yet as installkernel script needs some
> work.

The part about copying from the staging area to the target system won't
change for that, though.

> The chown to root at the target is because I've decided to force host
> compile and install as non-root ....

Probably a good idea anyway.

> The compile target exports nothing to NFS, the compile host
> exports /home/<target> work area so I can copy the target's
> /boot/config*, this NFS export is mounted and written to from
> the target as required. Better now? 8-)

You can't scp the config file? Or perhaps keep a local copy on the
compile host (since that's the system where you're using that file
anyway)?

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Grant wrote:

> ... just shows you dunno how to setup NFS

I've said it before: setting up NFS is easy, but doing it _right_ is
rather non-trivial.

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In alt.os.linux.slackware, Grant dared to utter,
>>rsync is good, but cp over NFS is crap! You can easily spend hours
>>upon hours moving thousands of files which have only a couple of bytes
>>in them when tar | nc would handle this in just a few minutes time.
>
> Crap, just shows you dunno how to setup NFS

Perhaps. :-)

> grant@sempro:~$ time cat /home/install/test578.iso >/dev/null
>
> real 0m54.260s <<== 10.7 MBytes/second via 100Mbps localnet over NFS

This isn't really the case I was referring to. NFS definitely does
good for single large files (though not as good as a quick netcat,
IMHO) but handles large amounts of very small files very poorly.
Consider the case of some one needing a full backup of a Windows
machine (including all temporary internet files!) or perhaps backing up
a mailserver for several hundred IMAP accounts in maildir format, each
with thousands of e-mails. In this situation, NFS IME has performed
very poorly in these cases. I'd like to see a comparison with your NFS
setup though, if you don't mind.

- --
It is better to hear the rebuke of the wise,
Than for a man to hear the song of fools.
Ecclesiastes 7:5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.7 (GNU/Linux)

iD8DBQFEdLBVz8zcalmVmBkRApPQAJwOwRXZ6YVoxZXh+h1yp+ q9sFXIAACdHlM9
y4W2MTs77CpPwIGR6jZkzxc=
=N6Gc
-----END PGP SIGNATURE-----

On Wed, 24 May 2006 17:15:44 +0000 (UTC), Sylvain Robitaille <syl@alcor.concordia.ca> wrote:

>Grant wrote:
....
>> Not yet tested for 2.4 series yet as installkernel script needs some
>> work.
>
>The part about copying from the staging area to the target system won't
>change for that, though.

That's right, though for a long time I've meant to do the 2.4 compile
step: "[make dep;] [make clean;] time make bzImage modules" from a
script so the next "make install modules_install" step may succeed.

Script I posted last night is now morphing into a generalised
kernel-build driver, plus updating installkernel script to suit
local and this new remote install method.

>> The compile target exports nothing to NFS, the compile host
>> exports /home/<target> work area so I can copy the target's
>> /boot/config*, this NFS export is mounted and written to from
>> the target as required. Better now? 8-)
>
>You can't scp the config file?

I could, but the need is occasional, for priming the compile host,
and old habits die hard What happened is I reversed the NFS
setup first, looking to avoid the tar, ssh --> then I switch as
merits of the tar, ssh approach became clear

> Or perhaps keep a local copy on the
>compile host ...

Yep, compile host keeps most recent staging area files, so target
..configs available for later kernel updates.

The other excuse for the target root rw export was for reading
/etc/lilo.conf and perhaps rewriting it, that's where scp will
be handy.

Grant.
--
WinXP: Access Start->Turn Off Computer, then while holding Ctrl-Alt-Shift,
left click on Cancel. This terminates Windows Explorer...

On Wed, 24 May 2006 15:12:05 -0500, +Alan Hicks+ <alan@lizella.netWORK> wrote:

>In alt.os.linux.slackware, Grant dared to utter,
>>>rsync is good, but cp over NFS is crap! You can easily spend hours
>>>upon hours moving thousands of files which have only a couple of bytes
>>>in them when tar | nc would handle this in just a few minutes time.
>>
>> Crap, just shows you dunno how to setup NFS
>
>Perhaps. :-)
>
>> grant@sempro:~$ time cat /home/install/test578.iso >/dev/null
>>
>> real 0m54.260s <<== 10.7 MBytes/second via 100Mbps localnet over NFS
>
>This isn't really the case I was referring to. NFS definitely does
>good for single large files (though not as good as a quick netcat,
>IMHO)

By what, 10% maybe? Not worth remembering yet another tool

>IMHO) but handles large amounts of very small files very poorly.

_Any_ network transfer handles many small file transfers poorly,
to various degrees, but I don't do large numbers of files over
NFS, silly. Instead, I'll create a tarball, for instance to
backup box deltree's config:

root@sempro:~# time tar cf /home/public/sempro/2006-05-25-config.tar /boot/config-2.* /etc/ /usr/local/bin/ /usr/local/etc/
tar: Removing leading `/' from member names

real 0m1.029s
user 0m0.010s
sys 0m0.050s
root@sempro:~# ls -lrt /home/public/sempro/
-rw-r--r-- 1 root root 7280640 2006-05-25 09:42 2006-05-25-config.tar

~7.3MB/s, the destination is pIII 450MHz CPU with IDE HDDs on
40-pin (udma33), not the fastest of NFS targets, but the other fast
box is this here winxp thing I'm typing at.

>Consider the case of some one needing a full backup of a Windows
>machine (including all temporary internet files!)

Backup 4GB winxp partition in dual SATA box:
root@sempro:~# time $(dd if=/dev/sda1 bs=64k of=/dev/sdb10; sync)
64259+1 records in
64259+1 records out

real 1m19.360s <<== 53MB/s
user 0m0.060s
sys 0m44.900s

root@sempro:~# mount /dev/sdb10 /mnt/hd/
root@sempro:~# ls /mnt/hd/
AUTOEXEC.BAT Intel10.0/ RECYCLER/ ntldr
CONFIG.SYS MSDOS.SYS System\ Volume\ Information/ pagefile.sys
Documents\ and\ Settings/ NTDETECT.COM WINDOWS/
IO.SYS Program\ Files/ boot.ini

Alright, over NFS then:
root@sempro:~# umount /dev/sda1
root@sempro:~# time dd if=/dev/sda1 bs=32k of=/home/public/sempro/2006-05-25-dev-sda1
128519+1 records in
128519+1 records out

real 8m38.251s <<== 8.1MB/s
user 0m0.090s
sys 0m29.900s
root@sempro:~# ls -lrt /home/public/sempro/
....
-rw-r--r-- 1 root root 4211311104 2006-05-25 09:35 2006-05-25-dev-sda1

root@sempro:~# mount /dev/sda1 /mnt/hd/
root@sempro:~# find /mnt/hd/ * | wc -l
12228

Grant.
--
WinXP: Access Start->Turn Off Computer, then while holding Ctrl-Alt-Shift,
left click on Cancel. This terminates Windows Explorer...