Today I was deploying some software updates in our Solaris 10 servers and, since such updates are distributed in files archived with the GNU
tar utility (not shipped by default in Solaris 10), I had to convert them before moving them to our servers. I'm quite accustomed to such a procedure: that's why (amongst other more important reasons) I always have an OpenSolaris virtual machine running in my other desktops (thanks to VirtualBox).
Today, notwithstanding the fact the I had a gut feeling that I was not doing the correct thing, I decided to open a Mac OS X terminal and do the job from there. GNU tar and pax are there, aren't they? There should not be any problem, thought I.
Well, I should have paid attention to that gut feeling of mine: to spare the required OpenSolaris boot, I lost much more than what I had planned. I learned something new but... what was it worth? A bunch of non portable behaviours of Mac OS X? Just the confirmation I needed, once again (I realize I'm a stubborn fellow): if you can, do not rely on a Mac to do your UNIX-ish tasks. Chances are you're going to waste more than you believed.
This is what happened.
Making a pax File for a Solaris Host
This was meant to be a quick task. I copied the archive from a Solaris NFS mount where, some days ago, I had stored the file. I ran
gunzip to decompress the file, ran
tar to unpack the files, ran
pax to store the resulting directory in a
pax file and used
rsync to quickly transfer the resulting archive back to the target Solaris machine:
$ gunzip [file.tar.gz]
$ tar xf [file.tar]
$ pax -w -f ./[file.pax] [dir]
$ rsync -vz --inline [file.pax] [myserver]
I then connected to the target server and used pax to explode the transferred archive. Since one of the things I feared was Mac OS X messing around with file permissions, I quickly checked the results on the Solaris side and everything looked fine. Well, sort of...
The next thing I usually do with such bundles is checking the differences between the running version and the soon-to-be-deployed one to be sure I won't miss any customization I made in the instance to be patched. It usually is another easy job since very few XML files are usually involved. Great was my surprise when diff began to write out hundreds of differences between one version and another.
Doh! It seemed as though somewhere in between somebody had decided to duplicate each file following a naming pattern; if the expected file was named x, I was seeing two files in Solaris: x and ._x. Where did these file came out from?
Back to the Mac
The original directory had been deleted just after producing the pax file so that the first I did was using pax to unpack the archive in the Mac. As I told you, everything looked fine. Except for an @ character that should not be there:
drwxr-xr-x@ [...snip...]
Maybe you don't know about it but that means that such a file has extended attributes set. Extended attributes are basically metadata: the file system does not interpret them and the ls command uses the @ symbol after the file permissions to inform you that they are there.
Strange Things Are Happening
There's nothing strange about extended attributes: it's very important to be aware of their existence to correctly handle them when, for example, you need to do a backup or move files from one file system to another (unless you don't mind losing them.) Solaris' pax and tar, for example, both support archival of extended attributes even if, by default, they ignore them. Since I know that my original archive had no extended attribute set in any of its file, I just ran a plain pax to build my archive. Maybe OS X pax defaults to a different behaviour so that I checked pax man page.
Doh! Oddly, OS X pax man page makes no mention at all of extended attributes, notwithstanding the fact that, undoubtedly, it was in fact managing some. Which? In OS X, just run xattr -l to find out.
Quarantine
A xattr told me that the com.apple.quarantine extended attribute was set in the culprit and a quick script confirmed that every file in that directory had that attribute set. According to Apple's documentation, that attribute is set by recent versions of OS X to mark the file as a potential threat. Do you remember when finder kindly asks you if you're really sure you want to open a downloaded file? The quarantine extended attribute is the mechanism OS X programs use to store that information.
How did it make it there?
But wait! I downloaded a tar.gz file and stored in an NFS 4 mounted file system for later processing. The question is:
- How did that attribute was stored in the NFS 4 file system where I stored the file?
- How did that attribute make it into every file stored into the pax archive?
As far as it concerns the first question, NFS 4 does support extended attributes. But again, should it be there, I would have known because I checked it directly in the Solaris host and, moreover, the box that acts as a NFS server runs some cron scripts that checks (and fixes to my taste) ACLs and extended attributes in some of the exported NFS file systems. It was very easy to find out that OS X, when storing such a file in a remote NFS share, instead of using NFS 4 extended attributes it stores them in a separate file named with the pattern mentioned above. I then discovered that the shared NFS file system in question is plagued by ._* files. Everywhere.
As far as it concerns the second question, well... I copied once again the file using the finder and performed the same operations I did before. The resulting files all had the quarantine extended attribute set. Who's the culprit, I do not know. I don't even care. The fact is that OS X is performing this operation with tools such as pax and tar without informing the user and, what's even worse, without even mentioning it in such commands' man pages.
Conclusion
The reasoning above explains why, even if Solaris pax (such as NFS) aren't managing any extended attribute, I did end up with duplicated files which were inserted into the pax file produced with Mac OS X. Probably they aren't going to hurt my deployment but surely they should not be there.
For the sake of curiosity, I checked that removing the ._* files in the NFS share made everything go back to normal.
Nevertheless, since I feel there are many OS X dark corners left for me to discover, I won't be making such an error again. Instead of waiting a couple of minutes for a virtual machine to bootstrap, I ended up wasting an entire hour for a (yet another...) bad documented and non-portable idiosyncrasy of an operating system that brings no added value with it.
