HOWTO sync your files

In other languages: English | Afrikaans | Albanian | Arabic | Brazilian | Bulgarian | Catalan | Chinese | Croatian | Czech | Danish | Dutch | Esperanto | Estonian | Filipino | Finnish | Flemish | French | German | Greek | Hebrew | Hindi | Hungarian | Indonesian | Italian | Japanese | Latvian | Lithuanian | Macedonian | Malay | Malayalam | Norwegian (Bokmål) | Norwegian (Nynorsk) | Persian | Polish | Portuguese | Romanian | Russian | Serbian | Slovak | Slovenian | Spanish | Swedish | Turkish | Ukrainian | Urdu

HOWTO synchronize your files

In this article we will tell you how to synchronize various files from your computer and backup data to multiple shell accounts. Solution covered in this article will not require installing and configuring some special software on remote server.

Getting started

First, you need to create "repository" and place there all the data you want to synchronize.

 mkdir ~/.home
 cp -fr .tcshrc ~/.home
 cp -fr .irssi ~/.home

Also, you should create Makefile in your repository. The idea is to run it after your data will be transfered to remote machine.

 all:
 	chmod 711	$(HOME)/.ssh \
 			$(HOME)/.irssi
 	chmod 600	$(HOME)/.tcshrc \
 			$(HOME)/.bash_profile \
 			$(HOME)/.logout \
 			$(HOME)/private.tar.gz.bf64 \
 			$(HOME)/CHECKSUM

We also recommend you to read this: HOWTO SSH without a password.

ClusterIt!

We will use clusterit to transfer data from your repository to remote machine. Clusterit is a collection of clustering tools to turn your ordinary everyday pile of UNIX workstations into a speedy parallel beast. This facilitates managing a set of program executing in parallel with a very flexible authentication scheme. If you don't want to use it, you will have to edit synchronization script. First, you need to install it. Under FreeBSD you may do:

make -C /usr/ports/net/clusterit/ WITHOUT_X11=yes install clean

Config file for clusterit is a file pointed to by CLUSTER enviroment variable. It's just a list of cluster nodes.

GROUP:shells
shellium.org
another.free.shell.isp
GROUP:work
work.machine1
work.machine2
LUMP:all
shells
work

Scripting

The following script will compress ~/private directory, encrypt it and copy all files to your cluster. After this "make" command will be executed on all cluster nodes.

 #!/bin/sh
 #
 # $Header: $
 #
 
 passwd=""
 prx=""
 prx_type=""
 repo="${HOME}/.home"
 group=""
 node=""
 objs="private"
 dir="${HOME}"
 
 while getopts p:s:t:r:g:w:o:d:c:h opt; do
 	case ${opt} in
 		p) passwd=${OPTARG}	;;
 		s) prx=${OPTARG}	;;
  		t) prx_type=${OPTARG}	;;
  		r) repo=${OPTARG}	;;
  		g) group=${OPTARG}	;;
 		w) node=${OPTARG}	;;
 		o) objs=${OPTARG}	;;
 		d) dir=${OPTARG}	;;
 		c) CLUSTER=${OPTARG}	;;
 		h) echo backup data and sync configs with clusterit tool:
 		   echo http://www.garbled.net/clusterit.html
 		   exit 0		;;
 		*)
 	esac
 done
 
 CLUSTER=${CLUSTER:-"${HOME}/.cluster"}
 
 if [ "${passwd}" = "" ]; then
 	echo -n 'Password: '
 	read passwd
 fi
 
 if [ "${prx}" = "" ]; then
 	RCMD_CMD=${RCMD_CMD:-"ssh"}
 	RCP_CMD=${RCP_CMD:-"scp"}
 else
 	RCMD_CMD="ssh -o ProxyCommand='nc -X ${prx_type} -x ${prx} %h %p'"
 	RCP_CMD="scp -o ProxyCommand='nc -X ${prx_type} -x ${prx} %h %p'"
 fi
 
 cd ${dir}
 
 echo compressing...
 for obj in ${objs}; do
 	tar czf ${repo}/${obj}.tar.gz ${obj}
 done
 
 cd ${repo}
 
 echo encrypting...
 : > CHECKSUM
 for obj in ${objs}; do
 	openssl bf -a -k ${passwd} -in ${obj}.tar.gz -out ${obj}.tar.gz.bf64
 	rm -rf ${obj}.tar.gz
 	openssl md5  -c ${obj}.tar.gz.bf64 >> CHECKSUM
 	openssl sha1 -c ${obj}.tar.gz.bf64 >> CHECKSUM
 	chmod 600 ${obj}.tar.gz.bf64
 done
 chmod 600 CHECKSUM
 
 echo Total size: `du -sh ${repo} | cut -d '	' -f 1`
 
 if [ "${group}" != "" ]; then
 	group="-g ${group}"
 fi
 
 if [ "${node}" != "" ]; then
 	node="-w ${node}"
 fi
 
 echo ClusterIt!
 pcp -c -r ${group} ${node} `ls -a ${repo} | awk '! /^(.|..)$/ { print "'${repo}'/" $1 }'` .
 dsh ${group} ${node} make
 
 # # if you don't have ClusterIt...
 #
 # for host in `...`; do
 #  scp -r ${repo}/ ${host}:. &
 # done
 #
 # wait
 #
 # for host in `...`; do
 #  ssh ${host} make &
 # done