I recently decided that I needed to come up with a system for managing custom and commonly used python modules. In doing so I decided upon a directory structure similar to this:
volatile/
dev/
prod/
Where volatile would be an unsafe set of packages, not yet tested in a staging environment (constantly updated), dev would be the packages currently in the staging environment and prod would be the packages currently installed, or ready to deploy to the production environment. This has been done before, nothing new. Along the way I thought to myself, how can I conserve disk space, avoiding up to three copies of the same file? An easy and logical way to avoid this would be hard links. So, rather than having three copies of a file that gets approved through to production, I would hardlink back to the original. I will demand that new files into volatile be uniquely named, so avoid overwriting. As files become obsolete, they will be pruned down the line (a system to handle this would need to be developed).
SO! Now that we got passed the background, too the yummy tips.
I have this awesome directory structure of existing files, right? How do I duplicate it creating links instead of files, and creating the directories as needed(can’t safely hardlink directories). I knew I could use rsync with --link-dest. It would look something like this:
rsync -avz --link-dest /volatile /dev/ \
rsync -avz --link-dest /volatile /prod/
-a says to preserve things, -v is verbose, -z says compress and --link-dest says to make links of the destination files.
I also thought of some find/exec magic creating directories and linking files, it would look something like this:
cd /volatile; find . -type d -exec mkdir -p /dev/{} \; \
find . type f -exec ln {} /dev/{} \;
Repeating the same thing for the /prod/ directory.
Both the above solutions feel hackish, but would work. I thought, there has to be a simpler way, this is a fairly routine operation. After checking out google, I was pointed to the cp(1) man page, which uncovered this gem:
-l, --link
link files instead of copying
Woah, I knew it. So much shorter and uses the right tool for the job!
cp -r -l volatile/* dev/; cp -r -l volatile/* prod/
It’s amazing what you’ll uncover with a tad bit of googling and man page searching!
