Some Reasons Why Solaris Is a Great Java Development Platform

Some days ago I posted "The Death of OpenSolaris: Choosing an OS for a Java Developer" in which I stated that Solaris is a great platform for a Java developer. The point of that post was simply wondering about which Solaris version I'd use since the demise of OpenSolaris. What the post did fail in clarifying, as Neil's comment made me realize, were the reasons why you should choose Solaris as your development platforms. I decided to write this follow up to that post to quickly summarize my favorite ones introducing some use cases where such technologies come in handy.

Software Availability

Although Solaris continues to be a niche OS (such as many other platforms are, anyway) in the last few years Sun and the community made an excellent job at promoting it as a desktop alternative for developers. There existed even a specific distribution for developers: Solaris Express Developer Edition. It was discontinued and there really is no need for it nowadays, anyway. Late Solaris distributions (such as SXCE, OpenSolaris, OpenIndiana), include (either bundled or in the official package repository):

• Data bases (MySQL, PostgreSQL).
• Web Servers (Apache, Java Enterprise System Web Server, etc.).
  
• Application servers (Glassfish).
• The SAMP stack (Solaris + Apache + MySQL + PHP).
• IDEs (NetBeans, Eclipse).
• Support for other popular languages (Ruby, Groovy, etc.).
• Identity management (LDAP, Java Enterprise System Identity Server).
  

Solaris also is a platform of choice in the enterprise hence common enterprise software packages are supported and you, as a Java developer or Java architect, won't miss the pieces you need to build your development environment. The very basic software packages I often need as a Java developers are:

• Oracle RDBMS.
• Oracle WebLogic Application Server.
• IBM WebSphere Application Server.
  
• JBoss Application Server.

Solaris' Technologies

Solaris has got some unique technologies that other UNIX (and UNIX-like) systems that might be used as development platforms are lacking (or ported from Solaris.) What's important here is not "technologies on their own" or technologies that are helpful only in big enterprise environments, but the fact that:

• They're pretty well integrated in Solaris and are built to take advantage of each other.
• There are common use cases in which these technologies are really helpful to a developer.

Each one of them would deserve several posts on their own, however, I'll try to make some concise examples.

Solaris Service Management Facility

Although this technology is probably most useful to a system administrator, as a developer I often took advantage of it. SMF is a framework that provides a unified model for services and services management. The basic recipe only needs an XML descriptor for a service. SMF lets you:

• Define a service: startup scripts location, parameters and semantics.
• Establish dependencies between services:
• Services and service instances may depend on other service instances.
• Service startup is preformed in parallel respecting service dependencies.
  
• Enhanced security and fine-grained role based access control:
• A service can be assigned only the minimum required set of privileges it needs to run.
• Service management can be delegated to non-root users using Solaris RBAC (Role-Based Access Control).
• Service health control:
• Service auto-restarts.
• Service health is enhanced by cooperation with Solaris Fault Manager which prevents service degradation when hardware failures occur.
• Automatic inetd Services Wrapper: SMF automatically wraps inetd services.
  

A Typical Use Case

Every software package I use has its own SMF descriptor (either provided with the package or defined by me) and it dramatically reduces the time I need to set up a development machine. In the case of WebSphere Application Server, for example, I have separate service instances for:

• WebSphere IHS.
• WebSphere Application Server.
• WebSphere Application Server DMGR.
• WebSphere Application Server cluster nodes.

Dependencies are defined between them and I can startup the required WebSphere services with just a line of code:

svcadm enable [websphere-service-name]

and SMF will take care of everything.

The usage pattern for SMF can be enhanced further. Let's suppose you're working in one or more projects and each one of them requires distinct set of running services. What usually happens is one of the following:

• You install them all and let them run.
• You install them all and start and stop them manually when you switch working project.

Resources are always few for a developers and some are paranoid about sparing them. With SMF you can:

• Define a SMF service for each of your projects.
• For every projects, define dependencies with the services you need.

This way, at a minimum, you can start and shutdown, with a single command, every service you need for a specific project. No more:

• Custom shell scripts for every service.
• Custom configuration entries for inetd services (such as Subversion, Apache, etc.)
  
• Specific OS customization.
• Running services when you don't need them and waste resources you could use otherwise.
  

Example of SMF service manifest customization can be found in the following posts:

• Registering a service with SMF.
• Credentials and Projects for a Solaris 10 SMF-Managed Service.
• Setting Up Subversion as an inetd Service.

ZFS

The ZFS filesystem is unique as far as it concerns flexibility and ease of use. With an incredibly lean set of commands, you can:

• Create file systems on the fly.
• Snapshot file systems on the fly.
• Clone file system on the fly with almost null space usage overhead.

There's a huge literature about ZFS and I'll limit to describe my favorite use cases.

Use case: Multiplexing Your Development Environment.

Software installations are just the beginning of your user experience. Often, we spend time:

• Configuring our environments.
• Fine-tuning them.
• Defining the set of additional libraries we need.
• Defining the set of server resources (JDBC, JMS, etc.) our applications use.

And so on. The list is endless.

Sometimes it's necessary to prepare different environments for different projects or different development stages of the same application. Instead of losing time and resource to build different environments I'll usually proceed as follows:

• Install and configure my environment.
• Make a ZFS snapshot of it.
  
• Make a ZFS clone of it for every additional setup I need.

Oracle JDeveloper is a good example of an application I often clone. JDeveloper is fundamentally a single user environment, despite adopting the common approach of using a per-user configuration directory in the user's home directory. Instead of fiddling with scripts to set per-user configuration parameters, I just install it once, snapshot it's installation directory and make a ZFS clone, one per environment. I use several clones of the JDeveloper environment myself, in my user home directory.

The power of ZFS clones can be used by the Zones infrastructure, as we'll see in the following section, thus enhancing further its power. Cloning a ZFS filesystem is also advantageous while dealing with big installations such as disk images of your favorite virtualization technologies.

Additional posts I wrote about ZFS that could clarify some of its use cases are:

• Sparing Disk Space With ZFS Clones While Setting Up a Developer Environment.
• Sun xVM: Cloning Your domU.
  

Containers and Other Virtualization Technologies

I consider Solaris a superior desktop virtualization platform. Once again, with a couple of commands. you can easily create a paravirtualized Solaris instance (a Zone). The zones infrastructure is ZFS-aware and can take advantage of it.

Zones can be configured with a command line interface to its XML configuration file. Creating a zone is straightforward and, since they're a lightweight technology, you can create as much zones as you need. If you're using ZFS, the process of cloning a zone is incredibly simple and fast.

Use Case: Clustering an Application Server

During the development of your Java EE application you will tipically need an instance of one (or more) of the following:

• An application server.
• A web server.
• A data base;
• An user registry.

It's also desirable to have them running on isolated environments so that you can mimic the expected production configuration. With zones it's easy: just create as many zones as you need and each one of them will behave as a separate Solaris instance: every zone will have, for example:

• Its own network card(s) and IP configuration.
• Its own users, groups, roles and security policies.
• Its own services.

Instead of installing and configuring an environment multiple times, you will prepare "master" zones with the services you need. I've got a "master" zone for every one of the following:

• WebSphere Application Server.
• WebLogic Application Server.
• Oracle DB.
• MySQL DB.
• LDAP directory.

and so forth. With one simple command (zoneadm clone [-m copy] [-s zfs_snapshot] source_zone) you'll end up with a brand new working environment in a question of minutes.

Use Case: VirtualBox and ZFS

Sometimes you'll rather work on a virtualized instance of some other OS, such as GNU/Linux, FreeBSD and Windows. Solaris is a great VirtualBox host and the power of ZFS will let you:

• Create "master" images for every OS or every "OS role" you need.
• Clone them on the fly to create a brand new virtual OS image.

In my case, I've got:

• A master Windows 7 client with Visual Studio for .NET development.
• A master Windows Server 2008.
• A master Windows Server 2008 (a clone of the previous one) with SQL Server 2008.
• A master Debian GNU/Linux.

Every time I need a new instance I just have to clone the disk image. In a matter of seconds I've got the environment I need. Not only I'm sparing precious time, I'm also sparing a vast amount of disk space. Should I store all of the images (and zones) I use without the ZFS technology and I'd need at least 4 times as much disks as I've got.

Use Case: A Virtualized Network Stack

Solaris provides you pretty powerful network virtualization capabilities. You can, for example, create as many virtual NICs as you need and use them independently either in Solaris Zones or as network cards for other virtualization technologies (such as VirtualBox.) Network cards can be interconnected with virtual switch (etherstubs) and enable you to create "networks in a box." Not only you can use virtualized instances to mimic your production environment: you'll be able to create a virtualized network to emulate the complex network policies your environment could need.

If you need to test an environment whose configuration would be impossible to replicate without additional physical machines, that's where virtualization technologies (such as Zones or VirtualBox) and the virtualized network stack come in handy. My developer environment for a project I'm working for is made up of:

• Two zones with two load balanced IBM IHS instances.
• A zone with an LDAP directory.
• Two zones with two clustered instances of IBM WebSphere Application Server.
• A Zone with an instance of IBM WebSphere DMGR.

With Solaris, I can replicate the production environment in my box and respect each and every network configuration we use. Without these technologies, it would have been much harder to accomplish this goal or I would end up with custom configurations (for example, to avoid port clashes). In all cases, I'd lose much more time on the administration and configuration of such environments if zones weren't so easy to use.

DTrace

DTrace power is extremely easy to explain to a developer. At the same time, it's difficult to grasp its usefulness without trying it yourself. DTrace on Solaris provides tens of thousand of probes out of the box and others can be created on the fly. This "probes" provide you an extremely powerful mean of troubleshoot problems in either your applications and the underlying operating systems. To use the probes you've got to use scripts written in the D language. Fortunately, this language is pretty easy by design and you can write powerful D scripts in a few lines of code.

DTrace is unobtrusive and let you troubleshoot problems immediately, without modifying your application, even in a production environment. Some IDEs, such as NetBeans, have powerful plugins that let you write D scripts and see the data collected by the probes in beautiful graphics.

As a developer, I valued DTrace usefulness more than once. Instead of troubleshooting problems having to dig into the source code and introduce additional code (even in the cases in which aspects come in handy), I could use a D script to observe the application from the outside and quickly collect data that could help me determine where the problem could be.

In some cases, moreover, you could find yourself dealing with situations in which there's no code available. I could quickly troubleshoot a problem I was having with WebSphere Application Server with a D script instead of relying on WebSphere tracing facilities and the task of interpreting log files.

Conclusion

So much for an introductory post. The possibility of building a development environment as close as possible as your target environment is a "must" for any development platform. Additionally, I consider that working on a environment as close as possible as the production environment not only gives you additional value and insights during an application development stage, but should also considered a mandatory requirement for every project we're involved into. Solaris provides all of the tools a developer need to accomplish this goal.

Solaris is a complex enterprise operating system with many features you won't probably ever use. Nevertheless, there's a use case for many others of them, as I tried to point out in this post. Since some of these technologies were developed with an open source license, they are also available on other operating systems: ZFS is available on FreeBSD and there exist a community effort to port it to OS X; DTrace is available on OS X, Linux and FreeBSD.

The "Solaris advantage" is that all of these technologies are highly integrated and take advantage of each other. The result is worth more than the sum of them. These technologies have got a very polished and easy to use administrative interfaces: when time is important, "How you do it" is fundamental.

I hope that these insights might help you understand if and when the Solaris operating system might be useful to you. Even if you consider that it's not, I suggest you give it a try anyway: it's always good to add new technologies to your tool box.

The Death of OpenSolaris: Choosing an OS for a Java Developer

A Bit of History: The Struggles of OpenSolaris

This is no news: you probably know all about it.

As a long time Solaris user, the recent years have been full of good news for me.

I remember starting with GNU/Linux at home to have "something similar" to the Solaris workstations I used at work. It was the time when software would most likely compile on Solaris rather than on Linux.

Years later I bought my first Sun Workstation: it was the time when trying to compile on Solaris packages that would supposedly compile on a POSIX system was a pain. Still, I continued to regard Solaris as a stable platform to keep on using it for my work duties, such as Java programming.

Then came Solaris 10 and all of its wonderful technologies such as ZFS, Zones and DTrace, just to cite a few. With it, there came the Solaris Express distributions which, at last, filled a long standing gap between Solaris and other operating systems, providing us a pretty modern desktop environment.

In late 2008 came the first OpenSolaris distribution. I installed it, played with it, but kept on using SXCE for my workstations. The main reason was compatibility with many Sun packages, such as the Java Enterprise System or the Sun Ray Server Software, that had more than one glitch on OpenSolaris.

When SXCE was discontinued, I waited for the 2010.xx OpenSolaris release to upgrade my systems. Unfortunately, that release will never see the light.

The Oracle Leaked Memo (the dignifying uppercase is a must given Oracle prolonged silence over the subject) shed a light over Oracle plans for Solaris proper and OpenSolaris. Part of the "good news" is that the Solaris Express program has been resurrected and the first binary distribution is expected later this year.

The bad news is that the code, at least the parts of it that will be released with an open source license, will be released after the releases of the full Solaris Operating Systems. Basically, our privileged observation point over the development of the operating system has been shut down.

Lots of ink has been been spilled since the Leaked Memo and plenty of information, discussions and wars are going on in the blogosphere. I'm not an authoritative source to speak about the subject and it's not even clear to me what I'm going to do, now.

Benefits of Solaris for a Java Developer

Solaris has been my operating system of choice since before I started working in the IT industry. As a student, I grew up with Solaris at the data center of my University and the Slackware I used at home seemed like a kid toy, compared to it. After graduating, I started working as a design engineer for a leading microprocessors producer. Needless to say, Solaris was the platform we ran our design software upon. Then, I moved to a consulting firm and started my career as a Java architect.

Solaris was and is the platform of choice for most of the clients I've been working for. Even today, the application servers, the cluster software, the database, and most of the infrastructure used by my clients run on Solaris. It always seemed a sound choice to me, then, developing software on the same platform that will run it in production.

IDEs, Tools and Runtimes

As a Java developer, I can run all of my tools I need on a supported platform. My favorite IDEs (NetBeans and JDeveloper), the application servers my clients use (WebLogic and WebSphere, mostly), the databases used by my applications (MySQL, Oracle RDBMS, PostgreSQL): all of them run and are supported on Solaris. Some of them are even bundled with it or readily available by Sun sponsored package repositories. The Eclipse platform, to cite a widely use IDE for Java, is available in the OpenSolaris IPS repository, too.

Solaris Technologies

Solaris 10 also integrates DTrace, a powerful, unobtrusive framework that allows you to observe and troubleshoot application and OS problem in real time, even in production systems with an almost null overhead. DTrace has allowed us to diagnose strange production quirks with no downtime: once you've tried DTrace and the D language, there's no going back to "just" a debugger, even in the development stages of your project.

Other kinds of problems does not show up in your debugger or are extremely hard to catch. It might be the case of network or file systems problems. That's where DTrace comes in handy: it lets you observe with an incredibly high detail what's going on in your application and in the kernel of the operating systems, if it's necessary to dig so deep.

Solaris Virtualization Technologies

Solaris is also an ideal host virtualization platform. Solaris can "virtualize itself" with Containers, Zones and Logical Domains: you can start a Zone in no time (and almost no space overhead), assign a resource cap to it and also build a virtualized network in a box to simulate a complex network environment.

One of the problems that I encountered during the development of big enterprise system is that the development environment, and sometimes even the integration environment, is very different than the production one. It surely is a methodology problem: nevertheless, developers have few weapons to counteract. For example, applications that appear to run fine on a single node may not run correctly on a server cluster, or scale badly.

The more you wait to catch a bug, the more impact will have a patch for it. That's why in my development team, for example, we use Solaris Zones to simulate a network cluster of IBM WebSphere Application Servers and a DB cluster. All of them run in completely isolated Zones in one physical machine and communicate on a virtual network with virtual etherstubs (sort of a network switch), VLANs and routing rules. This environment lets us simulate exactly how the software will behave in the production system. Without a flexible and lightweight virtualization technology it would have been much more difficult and expensive to prepare a similar environment.

And if you (still) need to run other platforms, you can use Xen or VirtualBox to run, for example, your favorite Linux distro, Windows, or *BSD.

Summarizing

Enumerating the advantages of Solaris is difficult in such a reduced space, however I'll try:

• It's a commercially supported operating system: that's an option, since Solaris is free for development purpose. Nonetheless, it's an important point to take into account.
• Is (very) well documented: there's plenty of official and unofficial documentation.
• It's easy to administer: Solaris is pretty easy to administer, even if you're not a seasoned system administrator.
• It's an UNIX system: with all of its bells and whistles.
• It is a great virtualization platform.
• It has some unique technologies that add much value to its offering, such as ZFS and DTrace.

If you're a Java developer and haven't given Solaris I try, I strongly suggest you do it. Maybe you'll start to benefit from other Solaris 10 technologies such as Zones and ZFS, even for running your home file or media server.

Complaints

I often hear complaints about Solaris coming from different sources and with the most imaginative arguments: proprietary, closed, old, difficult to use. I usually answer inviting users to try it and see for themselves before judging it (if that's the case). Most of the times I'm not surprised to discover that the complaining guy had minimal or null exposure to Solaris.

Also, I'd like to point out that no OS I tried is a swiss army knife. Solaris is a server-oriented OS with a good desktop but it's not comparable with other operating systems for such an use. So: no comparison with Linux, please. It would be so unjust as comparing Linux and Mac OS X for the average home user. ;)

Alternatives

Since Java "runs anywhere", there's plenty of choice for a Java developer.

Since I own a laptop with Mac OS X, I've built a small development environment with all of the tools I need. Mac OS X is a great operating systems that comes with many fancy features out of the box and, although it has some idiosyncrasy with Java (read: you have to use the JVM shipped by Apple), it's a good OS for a Java developer. Since the Mac OS X hype has begun, there's plenty of packages for it and a big ecosystem which is still growing. Still, many software packages run in the enterprise aren't supported on Mac OS X. Since I prefer to have an environment as close as possible as the production one, I think that OS X is not the best choice for the average Java EE architect.

I've also been an hardcore Slackware and Debian user for a long time. An enterprise Java developer would miss nothing in a modern GNU/Linux distribution, nowadays, and most of the software packages you'll find in the enterprise will run on your GNU/Linux distribution.

No need to talk about Windows, either.

So, why Solaris? Every OS has its own advantages and disadvantages. The point is to just recognize them. Mac OS X, in my opinion, is the best OS for a home user. I would change it for no Windows and no Linux. But as far as it concerns my developers' duties, every other OS just lacks the features and the stability that make Solaris great. ZFS, DTrace and Zones, for my use cases, are killer features.

What's Next?

You've decided to give Solaris a try, so: which is Your distribution? I don't know.

Solaris Express/Oracle Solaris

I strongly suspect that my wait will be prolonged and I will finally upgrade my machines as soon as Solaris Express has been released. Upgrading to Solaris 10 09/10 is not possible since I'm using some ZFS pools whose version is not yet supported by Solaris proper but it is a sound choice for a starter.

The advantage I see in using one of these versions is the availability of optional support and the good level of integration with the most commonly used software packages that Oracle is likely to guarantee.

OpenIndiana

You should also know that OpenSolaris sources have been (sort-of) forked and two new projects are born: Illumos and OpenIndiana. The project were started by Nexenta employees and volunteers of the OpenSolaris community. The first projects aims at maintaining the OpenSolaris code and the parts of the code that are closed or code that upstream might choose not to maintain. The OpenIndiana project aims at producing binary distribution of the operating system built upon the Illumos source code. OpenIndiana will provide a really open source, binary compatible alternative to Solaris and Solaris Express.

Sounds good and I'll willingly support it. In the meantime I've installed OpenIndiana in a couple of virtual machines and the first impressions are very good. I suppose it hasn't passed enough time yet for diverging changes to have emerged.

If you prefer a more modern desktop with a recent Gnome interface, drop Solaris 10 and go for OpenIndiana, if you don't feel like waiting for Solaris Express. In any case, switching between the two shouldn't pose any problems. What's clear to me is that I won't consider using both operating systems: I'll have to make a choice.

Support Availability

As an enterprise user and a Java developer, I've always been more concerned about OS support and support for the packages I use, rather than about eye candy. Even at the cost of running a proprietary platform.

In conclusion: I'll wait for Solaris Express to be released and only then will decide which one I'll use for my purposes between Oracle Solaris Express and  OpenIndiana. My heart is betting for OpenIndiana. My brain is betting for Oracle Solaris Express and Solaris proper. Only time will tell which one is right (for me.)

Follow-Up

A follow-up of this blog post is avaible at this address. In this post I'll try to summarize some use cases in which the technology we introduced in this post are effective and add real value to your development duties.

I hope you enjoy it.

Accessing ZFS Snapshots From Windows With "Previous Versions"

ZFS Snapshots

Solaris' ZFS snapshots are a great tool that allows us to instantly create a block level snapshot of a ZFS file system. ZFS uses copy-on-write semantics: newly written data is stored on new blocks while blocks containing the older data are retained (when referenced by, for example, a snapshot.) Since, at snapshot creation, data has already been allocated, both snapshot creation time and snapshot required storage (additional to referenced blocks) are almost negligible. Snapshots can be sent (to a file, over the network, etc.) and received on a destination host (as a file or as a ZFS file system.)

Using ZFS snapshots on Solaris and OpenSolaris is dead easy and incredibly flexible: not only you can snapshot a ZFS file systems for Solaris' own use. Since ZFS file systems are easily shared with many protocols such as NFS, CIFS or iSCSI, you can for example take a snapshot of a ZFS volume used as a Mac OS X Time Machine disk (what I call a two-dimensions time machine), or of a file systems shared with CIFS with Windows clients.

Where Are The Snapshots?

Solaris users probably know that snapshot can be accessed using the special .zfs directory on a ZFS file system. But what if you're accessing this filesystems remotely, for example via CIFS? A first approach might be making .zfs directories visible by setting their snapdir property accordingly:

# zfs set snapdir=visible your/zfs/file/system

Although this method perfectly works and may even seems natural to Solaris users, it might seem to lack the required amount of "user friendliness" required by the typical Windows user. Fortunately, thanks to CIFS, there's another solution which is just as easy and that perfectly integrates with the Windows user experience.

Shadow Copy and the Previous Versions Shell Extension

Microsoft introduced a technology called Shadow Copy (a.k.a. Volume Snapshot Service) back in Windows XP SP 2 and Windows Server 2003. Shadow copy is similar to ZFS snapshots in that it takes block-level snapshots of a running file system (although much more limited.) Microsoft also introduced a shell extension, called Previous Versions, that let the user browse through the previous versions of a file that's been shadow copied. With this extension, a new tab in your Windows Explorer's File Properties window, you can browse and restore a previous version of a modified or deleted file. It seems natural, then, to use this extensions to browse through the ZFS snapshots too.

CIFS

That's exactly what the CIFS guys thought while developing this wonderful Solaris service. CIFS is the natural choice for sharing a ZFS file systems with Windows clients. CIFS implements the SMB protocol and it's, moreover, a wonderfully easy service to configure and maintain. Since CIFS exists, I'm not longing for Samba any more. If you mount a CIFS share in a Previous Versions-enabled copy of Windows, you'll automatically get access to ZFS snapshots without the burden of manually accessing .zfs directories.

Conclusion

As you can see in the following screenshot, Solaris ZFS snapshot are visible in the Previous Versions tab:

Conclusions

ZFS, CIFS and Windows Previous Versions are a great team when sharing ZFS file systems to your Windows clients. Windows has got the most usable interface when you need access to your ZFS snapshots. At least once Windows is superior to Mac OS X which has a fancy, usable but pretty basic interface.

Backing up JIRA and Confluence taking advantage of ZFS snapshots

If you're running an instance of JIRA or Confluence (or many other software packages as well) you probably want to make sure that your data is properly and regularly backed. If you've got some experience with JIRA or Confluence you surely have noticed the bundled XML backup facility: a scheduled backup service which takes advantage of it it's even running by default in your instances.

The effectiveness of such a backup facility depends on the size of your installation but the rule of thumb is that it's a mechanism that does not scale well as the amount of data stored in your instances grows up. In fact, XML backup was thought for small-scale installations and is not a recommended backup strategy for larger scale deployments.

In the case of JIRA I still continue to run automated XML backups since they do not store attachments in them but as far as it concerns Confluence, I always disable the automated XML backup and rely on native database backup and attachment storage backup. The database backup must be performed with the native database tools such as pg_dump for PostgreSQL. The backup of your instance's attachments will depend on the type of storage in use. If you're storing your attachment in the database, your attachments will be backed up automatically during your database backup. If you store your attachments in a file systems, as it's the case for both JIRA and Confluence default installations, there's plenty of tools out there to get the job done such as tar, pax, cpio and rsync (to name just a few). Each one of these have advantages and drawbacks and I won't enter in a detailed discussion: it suffices to say that none can beat a Solaris ZFS-based JIRA or Confluence installation. 

Since ZFS inception I've been taking advantage of its characteristics more and more often and snapshots are a ZFS killer feature that will considerably ease your administration duties. Whenever I install a new instance on a Solaris Zone, I set up a ZFS file system for hosting both the database files and JIRA or Confluence home directories:

# zfs create my-pool/my/db/files

# zfs create my-pool/jira/or/confluence/home

Taking a snapshot of a ZFS file system is a one-liner:

# zfs snapshot file-system-name@snapshot-name

In an instant your snapshot will be done and you will be able to send it to another device for permanent storage. ZFS snapshots, combined (or not...) with another tool such as rsync, will incredibly simplify backing up your files and, also, maintaining a cheap history (in terms of storage overhead) of changes in case you need to roll back your file systems (and hence the data stored in your application) in case you needed it.

Take into account that, to recover a single file from a snapshot in case your original pool crashes, you will need to ZFS receive the snapshot in another pool for files to be accessible. That's why I still rely on a scheduled rsync backup together with ZFS snapshots just in case, although with a much lower frequency than in the pre-ZFS epoch.

Mac OS X as an iSCSI initiator: Time Machine on ZFS

As I described in previous posts, I setup an iSCSI target with Solaris COMSTAR backed by a ZFS volume. I want to use this volume as a disk for Mac OS X Time Machine. This way, I'll get the best of the two technologies: a pretty looking and easy to manage Time Machine for backing up my MacBook backed by an enterprise-level, redundant and scalable ZFS volume published as an iSCSI target over my private LAN.

No more consumer disks on a table, no more poor hardware-implemented file system sharing protocol. No more worries to lose a disk. Just Solaris, ZFS, COMSTAR and a LAN.

Mac OS X as an iSCSI initiator

Although it's a subject most spoken of, Apple hasn't released yet the necessary components for Mac OS X to be an iSCSI initiator. Fortunately there exists a solid and free solution by Studio Network Solutions: globalSAN iSCSI Initiator for OS X. Just download it, install it, restart your OS X and a new panel will appear in your System Settings:

Connecting to a target

Connecting to a target is really easy: just use the globalSAN iSCSI GUI to add the target:

The target name is obviously retrieved from your target configuration.

Using the disk

If you read the previous post, you'll know that this target is backed by a ZFS volume which must be formatted before being used. With the Disk Utility you can format the new disk:

Using the disk with the Time Machine

To use the new disk with the Time Machine you just follow the usual procedure:

Conclusion

That's it. Using a ZFS volume as a disk for Mac OS X Time Machine is just a few clicks away. Next time you plan to buy a new external hard disk, just wait and take into account that a robust enterprise-level solution is available with not much more than the necessary budget to purchase a couple of consumer disks.

Setting up Solaris COMSTAR and an iSCSI target for a ZFS volume

COMSTAR stands for Common Multiprotocol SCSI Target: it basically is a framework which can turn a Solaris host into a SCSI target. Before COMSTAR made its appearance, there was a very simple way to share a ZFS file system via iSCSI: just setting the shareiscsi property on the file system was sufficient, such as you do to share it via NFS or CIFS with the sharenfs and sharesmb properties.

COMSTAR brings a more flexible and better solution: it's not as easy as using those ZFS properties, but it is not that hard, either. Should you need more complex setup and features, COMSTAR includes a wide set of advanced features such as:

• Scalability.
• Compatibility with generic host adapters.
• Multipathing.
• LUN masking and mapping functions.

The official COMSTAR documentation is very detailed and it's the only source of information about COMSTAR I use. If you want to read more about it, please check it out.

Enabling the COMSTAR service

COMSTAR runs as a SMF-managed service and enabling is no different than usual. First of all, check if the service is running:

# svcs \*stmf\*

STATE          STIME    FMRI

disabled       11:12:50 svc:/system/stmf:default

If the service is disable, enable it:

# svcadm enable svc:/system/stmf:default

After that, check that the service is up and running:

# svcs \*stmf\*

STATE          STIME    FMRI

online         11:12:50 svc:/system/stmf:default

# stmfadm list-state

Operational Status: online

Config Status     : initialized

ALUA Status       : disabled

ALUA Node         : 0

Creating SCSI Logical Units

You're not required to master the SCSI protocols to setup COMSTAR but knowing the basics will help you understand the next steps you'll go through. Oversimplifying, a SCSI target is the endpoint which is waiting client (initiator) connections. For example, a data storage device is a target and your laptop may be an initiator. Each target can provide multiple logical units: each logical unit is the entity that performs "classical" storage operations, such as reading and writing from and to disk.

Each logical unit, then, is backed by some sort of storage device; Solaris and COMSTAR will let you create logical units backed by one of the following storage technologies:

• A file.
• A thin-provisioned file.
• A disk partition.
• A ZFS volume.

In this case, we'll choose the ZFS volume as our favorite backing storage technology.

Why ZFS volumes?

One of the wanders of ZFS is that it isn't just another filesystem: ZFS combines the volume manager and the file system providing you best of breed services from both world. With ZFS you can create a pool out of your drives and enjoy services such as mirroring and redundancy. In my case, I'll be using a RAID-Z pool made up of three eSATA drives for this test:

enrico@solaris:~$ zpool status tank-esata

  pool: tank-esata

 state: ONLINE

 scrub: scrub completed after 1h15m with 0 errors on Sun Feb 14 06:15:16 2010

config:

        NAME        STATE     READ WRITE CKSUM

        tank-esata  ONLINE       0     0     0

          raidz1-0  ONLINE       0     0     0

            c7t0d0  ONLINE       0     0     0

            c8t0d0  ONLINE       0     0     0

            c8t1d0  ONLINE       0     0     0

errors: No known data errors

Inside pools, you can create file systems or volumes, the latter being the equivalent of a raw drive connected to your machine. File systems and volumes use the storage of the pool without any need for further partitioning or slicing. You can create your file systems almost instantly. No more repartition hell or space estimation errors: file systems and volumes will use the space in the pool, according to the optional policies you might have established (such as quotas, space allocation, etc.)

ZFS, moreover, will let you snapshot (and clone) your file systems on the fly almost instantly: being a Copy-On-Write file system, ZFS will just write modification on the disk, without any overhead and when the blocks are no more referenced, they'll be automatically freed. ZFS snapshot are Solaris a much optimized version of Apple's time machine.

Creating a ZFS volume

Creating a volume, provided you've already have a ZFS pool, it's as easy as:

# zfs create -V 250G tank-esata/macbook0-tm

The previous command creates a 250GB volume called macbook0-tm on pool tank-esata. As expected you will find the raw device corresponding to this new volume:

# ls /dev/zvol/rdsk/tank-esata/

[...snip...]  macbook0-tm  [...snip...]

Creating a logical unit

To create a logical unit for our ZFS volume, we can use the following command:

# sbdadm create-lu /dev/zvol/rdsk/tank-esata/macbook0-tm

Created the following LU:

              GUID                    DATA SIZE           SOURCE

--------------------------------  -------------------  ----------------

600144f00800271b51c04b7a6dc70001  268435456000         /dev/zvol/rdsk/tank-esata/macbook0-tm

Logical units are identified by a unique ID, which is the GUID shown in sbdadm output. To verify and get a list of the available logical units we can use the following command:

# sbdadm list-lu

Found 1 LU(s)

              GUID                    DATA SIZE           SOURCE

--------------------------------  -------------------  ----------------

600144f00800271b51c04b7a6dc70001  268435456000         /dev/zvol/rdsk/tank-esata/macbook0-tm

Indeed, it finds the only logical unit we created so far.

Mapping the logical unit

The logical unit we created in the previous section is not available to any initiator yet. To make your logical unit available, you must choose how to map them. Basically, you've got two choices:

• Mapping it for all initiators on every port.
• Mapping it selectively.

In this test, taking into account that it's a home setup on a private LAN, I'll go for simple mapping. Please, choose carefully your mapping strategy according to your needs. If you need more information on selective mapping, check the official COMSTAR documentation.

To get the GUID of the logical unit you can use the sbdadm or the stmfadm commands:

# stmfadm list-lu -v

LU Name: 600144F00800271B51C04B7A6DC70001

    Operational Status: Offline

    Provider Name     : sbd

    Alias             : /dev/zvol/rdsk/tank-esata/macbook0-tm

    View Entry Count  : 0

    Data File         : /dev/zvol/rdsk/tank-esata/macbook0-tm

    Meta File         : not set

    Size              : 268435456000

    Block Size        : 512

    Management URL    : not set

    Vendor ID         : SUN

    Product ID        : COMSTAR

    Serial Num        : not set

    Write Protect     : Disabled

    Writeback Cache   : Enabled

    Access State      : Active

To create the simple mapping for this logical unit, we run the following command:

# stmfadm add-view 600144f00800271b51c04b7a6dc70001

Configuring iSCSI target ports

As outlined in the introduction, with COMSTAR a new iSCSI transport implementation has been introduced that replaces the old implementation. Since the two implementation are incompatible and only one can run at a time, please check which one you're using. Nevertheless, consider switching to the new implementation as soon as you can.

The old implementation is registered as the SMF service svc:/system/iscsitgt:default and the new implementation is registered as svc:/network/iscsi/target.

enrico@solaris:~$ svcs \*scsi\*

STATE          STIME    FMRI

disabled       Feb_03   svc:/system/iscsitgt:default

online         Feb_03   svc:/network/iscsi/initiator:default

online         Feb_16   svc:/network/iscsi/target:default

If you're running the new COMSTAR iSCSI transport implementation, you can now create a target with the following command:

# itadm create-target

Target iqn.1986-03.com.sun:02:7674e54f-6738-4c55-d57d-87a165eda163 successfully created

If you want to check and list the targets you can use the following command:

# itadm list-target

TARGET NAME                                                  STATE    SESSIONS

iqn.1986-03.com.sun:02:7674e54f-6738-4c55-d57d-87a165eda163  online   0

Configuring the iSCSI target for discovery

The last thing left to do to have your iSCSI target configured for discovery. Discovery is the process which an initiator use to get a list of available targets. You can opt for one of the three iSCSI discovery methods:

• Static discovery: a static target address is configured.
• Dynamic discovery: targets are discovered by initiators using an intermediary iSNS servers.
• SendTargets discovery: configuring the SendTargets option on the initiator.

I will opt for static discovery because I've got a very small number of targets and I want to control which initiators connect to my target. To configure static discovery just run the following command:

# devfsadm -i iscsi

Next steps

Configuring a target is a matter of few commands. It took me much more time to write down this blog post than having my COMSTAR target running.

The next steps wil be having an initiator connect to your target. I detailed how to configure a Mac OS/X instance as an iSCSI initiator on another post.

Using ZFS with Apple's Time Machine

The many of us who got accustomed to the ZFS wonders won't willingly trade ZFS for another file system, ever. But even though many ZFS users are running Solaris on their machines, including laptops, as I am, there are cases in which running another OS is desirable: that's when I seek the best option to integrate the other systems I'm running with Solaris and ZFS.

In the simplest case using some file system protocol such as NSF or CIFS is sufficient (and desirable): that's how I share the ZFS file systems where I archive my photos, my video, my music and so on. Sharing such file systems with another UNIX, Windows or Mac OS/X (just to cite some), it's just some commands away.

In other occasions accessing a file system is not sufficient: that's the case with Apple's Time Machine, which is expecting a whole disk for its own sake connected locally.

Fortunately, integrating ZFS and Time Machine is pretty easy if you're running a COMSTAR-enabled Solaris. Although setting up COMSTAR is a very well documented topic by the Solaris and OpenSolaris documentation, I'll give you a walk through the necessary steps to get the job done and having your time machine making its backup on a ZFS volume. You'll end up with the benefits of both world: a multidimensional time machine which will take advantage of ZFS snapshotting and cloning capabilities.

The steps I'll detail in the following posts are:

• Setting up Solaris COMSTAR and configuring an iSCSI target port for a ZFS volume.
• Setting up Mac OS/X as an iSCSI initiator and configuring Time Machine. 

With such a solution, you will need no USB/FireWire/anything-else drive hanging around. You won't need to rely on consumer drives which implement some kind of file system sharing protocol which, as explained earlier, won't fit into the time machine use case.

Just a network connection and a box to install Solaris, ZFS and COMSTAR, and you'll provide a scalable, enterprise-level, easy to maintain solution for your storage needs.

Apple Discontinues Mac OS/X ZFS Project

One of the reasons I really love Solaris 10 (and OpenSolaris) is ZFS. I'm also running Solaris at home because of ZFS. I'd also thought about buying an Apple machine some day or the other but was waiting about ZFS being integrated into Mac OS/X. Well, it seems that the game is over:

Apple has posted a note on its ZFS project web page informing the users that the project is being discontinued. Apparently, there's a ZFS MacOS refugee camp (in their own words) on Google Groups.

I'm really sorry about that. ZFS is a great technology and it would have fitted very well on Mac OS/X, although it's (mainly) a consumer oriented Operating System. I was wondering about a ZFS-enabled Time Machine (sort of OpenSolaris Time Slider). Real cool, but it'll never be: rumors suggest that it was a political, rather than technical choice.

Nonetheless I think Mac lovers have lost the opportunity of running ZFS as their file system. As far as it concerns myself, a real argument about switching (at least the laptop) has just faded away and I think I'll keep on sticking with Sun Desktops and with some compatible laptop here at home.

Why I manually manage my iPhone's music

The problem

I own a pretty large music library which is hosted on a dedicated ZFS file system. Access to UNIX clients is provided via NFS and access to Windows clients is provided via CIFS. So far, so good. The problem is that the library is huge, very huge: not only the number of files is high, but some files themselves are huge. Whenever I buy a new CD, I rip it and encode it with a lossless codec in order to store the sufficient information in the case I need to burn another copy of the CD. On the Solaris Operating System, I'm using FLAC to encode such files. Alongside these lossless-encoded files, I use to encode another copy of the files in an easier to handle format, more suitable to use with portable devices. In this case, I use to re-encode FLAC files with an MP3 encoder.

Keeping organized such a library isn't difficult and the only problem I had so far is expanding storage according to my needs and backing it up: since I'm using ZFS, I'm an happier (and a wiser) man.

The problem with iTunes and such kind of programs is that they don't fit very well in the big library and networked storage scenario. Starting iTunes would take long to complete and, moreover, network would be the bottleneck. I never liked such a waste of resources and this is one of the reasons I never, ever, used a program to keep "organized" my music library.

But, what's the matter with the iPhone? Well, the iPhone is a glorified iPod and we all know that Apple is so kind to not allow us to read or write files on our phone but using iTunes.

iTunes synchronization

The iTunes way is very simple and idiot-proof: the iPhone is kept in sync with your iTunes-managed libraries: music, videos but also contacts, application and so forth. Kept I all of my music in just one laptop, that would (probably) be great but as I told you that's not (fortunately) the case.

The first times I synchronized my iPhone I used to:

• Adding files to my library.
• Synchronizing the iPhone.

Unfortunately, if my laptop cannot access the CIFS shares where the music is, iTunes just sees missing files and your iPhone will be empty after the next synchronization. Not so good.

The following times I thought I'd better copy files locally, first, and then synchronize. Good! Well, no. Because, unless you leave those files there (forever!), you'd hit the same behavior I described earlier. Replicating seldom is a good idea. Replicating such a library, definitely is not.

That's the kind of idiosyncrasy I hate in an end user program. Or it isn't an idiosyncrasy at all and it's me who's a strange user. Anyway, that's why I switched to manually manage my iPhone files. No library synchronization. I just copy files from the CIFS share directly into the iPhone. Just as if it was a plain old phone. No stale files on my laptop to keep iTunes happy.

ZFS filesystems with compression enabled made my system unresponsive

At home, I'm running some services onto a Solaris Express Community Edition. The last time I live-upgraded it was to build 113. Before running SXCE, I was running Solaris 10 on it. This workstation also manages a RAIDZ-1 pool composed of four USB drives. I was perfectly aware about the performance penalty I was paying for running USB instead of eSATA (or even internal SATA), but there was no choice, then. By the way, that pool is just used to offline some snapshot backup scheduled at night, so I wasn't usually hit by the slowness of this solution.

The most annoying bug affecting this setup was this:

6586537: async zio taskqs can block out userland commands

The effect was the unresponsiveness of the userland commands I was using, including Xorg itself, when doing some big I/O on a compressed file system of that pool. Originally, the compressed file systems were lzjb and moved to gzip-based compression as soon as it was released, but before knowing I was affected by that bug. The bug really made the system unusable: sometimes even the keyboard echo could be delayed more than 5 seconds. I haven't any other machine running compressed ZFS file systems and I couldn't test the bug status so often. Moreover, that's the typical machine you'd better not touch, unless you've got a lot of spare time to recover from a disaster (just in case). With time, at the end, I just learned to live with it.



The good news is that I just realized that the bug was fixed and committed in SXCE build 115 and scheduled for Solaris 10 Update 9. I live upgraded my old SXCE to build 116, the same I'm happily running on my laptop, and I must recognize that the system is really, really much more responsive when ZFS compressed file systems are being stressed. Maybe there also were improvements on the USB side, who knows.


If you're running older build affected by this bug, upgrade now.

ZFS spared me a lot of time and saved my data (once more)

Yes. Since I was introduced to it, almost every day something remembers me that I'm running the Solaris OS and its ZFS file system.

While developing part of the architecture for the .NET application I'm working on I decided to take a (freaky) break. I stopped my virtualized Windows instance (run on Sun xVM VirtualBox, of course), I put the kettle on the hob, put a coffee spoon of Gunpowder green tea on the teapot, and... launched a:

$ VBoxManage modifyhd winxp.vdi --compact

I went to the kitchen, waited the canonical 2 minutes watching tea leafs opening and releasing their flavor in the water, returned to my desk and: nothing could go worse than that. I just hit VirtualBox bug number 3864. The operation had failed and the hard disk image, as far as I could see, was greatly damaged. Windows wouldn't start correctly and would force a chkdsk on the entire disk. A great number of errors were found and the hard disk was not fixed.

Fortunately enough, since I use ZFS, I'm missing no backup. Every night, at least, a script snapshots a set of filesystems and sends a replication stream to another server where the ZFS stream is received (if you're wondering, the answer is "No, I don't use the Time Slider (yet)). I just had to send back the last saved snapshot and my VDI image was as good as this very same morning. Data? Well, I'm prudent enough to periodically save all of my work in a Subversion server I set up.

Once day more I'm happy to run ZFS.

Sparing disk space with ZFS clones while setting up a developer environment for multiple users

I have got some Solaris 10 workstations which are shared by a bunch of users which daily develop their application in the native Solaris 10 OS. Every user has to be able to log in in whichever machine and use his own environment. For this reason we took the usual approach of setting up NFS remote mounting of home directories and managing user accounts in an LDAP.

The developers also use a virtualized Windows and a virtualized Debian GNU/Linux during their duties and the initial deploy of the workstations didn't take into account the viability of accessing via NFS files as big as xVM hard disks. Replicating hard disk images was an issue and saving workstations hard disk space was another: with an average 15 GB per hard disk (OS and applications fully configured), per OS, per user, the 200 GB hard disk that equips the workstations seemed insufficient.

In this case, and in all the cases where you can share at least an initial set of data, ZFS helped us solving this problem. ZFS clones are filesystems generated from a snapshot of another ZFS filesystem. Initially a clone consumes no space and, as modifications starts to be applied, it begin to diverge from the snapshot it was generated from and starts allocating space. In our case, I snapshotted the initial installation of Sun xVM VirtualBox, I created a clone per user and set the mountpoint of the ZFS filesystem into the user home directory:

# zfs snapshot virtualbox/installation/dir@initial
# zfs clone virtualbox/installation/dir@initial virtualbox/users/name-0
...
# zfs clone virtualbox/installation/dir@initial virtualbox/users/name-n
# zfs set mountpoint=/export/name-0/.VirtualBox virtualbox/users/name-0
...
# zfs set mountpoint=/export/name-n/.VirtualBox virtualbox/users/name-0

ZFS clones spared me a lot of troubles and, moreover, every user has got a dedicated filesystems for its VirtualBox with all of the advantages provided by ZFS such as snapshots, restores and backups. We used the same approach to provide a common set of low-volatility filesystems to our users such as big SVN repositories and documentation directories.

Backing up ZFS file systems

This is one of the good things ZFS has brought us. Backing up a file system is a ubiquitous problem, even in your home PC, if you're wise and care about your data. As many things in ZFS, due to the telescoping nature of this file system (using words of ZFS' father, Jeff Bonwick), backing up is tightly connected to other ZFS' concepts: in this case, snapshots and clones.

Snapshotting

ZFS lets the administrator perform inexpensive snapshots of a mounted filesystem. Snapshots are just what their name implies: a photo of a ZFS file system in a given point in time. Since that moment, the file system from which the snapshot was generated and the snapshot itself begin to branch and the space required by the snapshot will roughly be the space occupied by the differences between these two entities. If you delete a 1 GB file from a snapshotted filesystem, for example, the space accounted for that file will go in charge of the snapshot which, obviously, must keep track of it because that file existed when the snapshot was created. So far, so good (and easy). Creating snapshot is also incredibly easy: provided that you have a role with the required privileges you just issued the following command:

$ pfexec zfs snapshot zpool-name/filesystem-name@snapshot-name

Now you have a photo of the zpool-name/filesystem-name ZFS file system in a given point in time. You can check about its existence by issuing:

$ zfs list -t snapshot

which in this moment, in my machines, gives me:

$ zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
rpool/export/home/enrico@20081231 71.3M - 14.9G -
[...]

This means that the ZFS file system which hosts my home directory has been snapshotted and there's a snapshot named 20081231.

Cloning

Cloning is pretty much like snapshotting with the difference that the result of the operation is another ZFS file system, obviously mounted in another mount point, which can be used like whichever file system. Like snapshots, the clone and the originating file system will begin to diverge and differences will begin to occupy space in the clone. The official ZFS administration documentation has detailed and complete information about this topic.

Backing up

This isn't really how documentation calls it: they just refer to it with ZFS send and receive operations. As seen, we've got a mean to snapshot a file system: there's no need to unmount a file system or run the risk of getting a set of inconsistent data because a modification occurred during the operation. This alone is worth switching to ZFS, in my opinion. Now there's more: a snapshot can be dumped (serialized) to a file with a simple command:

$ pfexec zfs send zpool-name/filesystem-name@snapshot-name > dump-file-name

This file contains the entire ZFS file system: files and all the rest of metadata. Everything. The good thing is that you can receive a ZFS file system just doing:

$ pfexec zfs receive another-zpool-name/another-filesystem-name <>

This operation creates the another-filesystem-name on pool another-zpool-name (it can even be the same zpool you generated the dump from) and a snapshot called snapshot-name will also be created. In the case of full dumps, the destination file system must not exist and will be created for you. Easy. Full back up with just two lines, a bit of patience and sufficient disk space.

There are the usual variations on the theme. You don't really need store the dump in a file, you could just pipe send into receive and do it in just one line with no need of extra storage for the dump file:

# zfs send zpool-name/filesystem-name@snapshot-name | zfs receive another-zpool-name/another-filesystem-name

And if you want to send it to another machine, no problems at all:

# zfs send zpool-name/filesystem-name@snapshot-name | ssh anothermachine zfs receive another-zpool-name/another-filesystem-name

Incredibly simple. ZFS is really revolutionary.

Incremental backups

ZFS, obviously, lets you do incremental send and receive with the -i option which lets you send the differences between one snapshot and another. These differences will be loaded and applied at the receiver side: in this case, obviously, the source snapshot must already exist. You start with a full send and then you go on with increments. It's the way I'm backing up our machines and it's fast, economic and reliable. A great reason to switch to ZFS, let alone Solaris.

Deploying an USB based ZFS storage pool at home

I recently bought a Sun Ultra 20 M2 workstation which I mainly use as a Java development platform. Nevertheless, I wanted to take advantage of Solaris 10 in my home network too and, after much reading about ZFS, the first thing I wanted to implement was a personal file server.

I started digging into official documentation at OpenSolaris.org and at Sun to discover the full ZFS possibilities and determine which was the best setup for 3-users small network. Minimum system requirements were met, so the first decision I had to take was: which devices I'm populating my zpool with? The requirements I had were very simple:

• I needed enough storage to weekly backup 3 machines; a rough estimate was 30 GB for machine so a total of 100 GB would be sufficient for the moment (for the sake of simplicity, backup is going to be done with rsync);
  
• 100 GB would be required to host and share my CD collection between all the clients I have at home;
  
• all the extra storage would be welcome and used as "scratch" space;
• a replication scheme should be implemented. Priority is given to storage rather than to performance.
  

So, which kind of disks was I going to use?

This wonderful machine, unfortunately, natively supports only two internal SATA(/SAS) disks and the available I/O ports were:

• 6 USB 2.0 (2 in the front and 4 in the back)
  
• 2 FireWire 400 (IEEE 1394a)
  

There was not so much to play with. One option I considered was buying an additional SATA controller to drive some additional SATA disk. Solaris Express Developer Edition 05/07 includes the following drivers:

• marvell88sx (Marvell 88SX SATA controller)
• si3124 (SiliconImage 3124/3132 SATA controller)
• ahci (Intel ICH6 and VIA vt8251 SATA controllers)

This option was particularly interesting because of the superior performance of an SATA disk compared to an USB 2.0 high speed device. Many controller I've seen use both Marvell and SiliconImage chips and finally a server class SuperMicro controller caught my attention. My workstation, by the way, is not equipped with PCI-X slots and, even if that controller is PCI compatible, I didn't want to run an underpowered solution. This option was finally discarded. Had I known this before, I would have seriously considered buying a Sun Ultra 40, which supports up to 8 internal SATA drives.

At this point I was left to choose between FireWire and USB. FireWire ports on the Ultra 20 M2 are IEEE 1394a, whose data rate is limited to 400 MBit/s. IEEE 1394a data rate is slightly inferior to the data rate ofUSB 2.0, which is 480 MBit/s. Nevertheless, FireWire peer-to-peer network architecture and FireWire support for memory-mapped device allows a more effective and less resource consuming approach for storage resulting in less CPU consumption, so FireWire appeared as an attractive solution. Unfortunately two FireWire ports are not enough for me, because they do not leave room for any replication scheme but a two-way mirror, which is a replication scheme I would avoid trying to maximize storage availability.

ZFS, indeed (as of Solaris Express Developer Edition 05/07), provides data redundancy in two flavors: mirrored or RAID-Z. An n-way mirror is a set of n disks where n-1 copies of the data are made during writing. This solution can survive various disk failures and provides (roughly) parallel read access to the n copies of data. RAID-Z is available with single or double parity. Citing official documentation, a RAID-Z configuration with n disks of size x with p parity disks can hold approximately (n-p)*x bytes and can withstand one device failing before data integrity is compromised.

At the end, I bought 3 350 GB Lacie USB 2.0 desktop disks and created my first zpool with them. I also added as a cache device an old 200 GB Iomega disk which I couldn't use as pool device. As the size of the pool depends on the smallest disk, a (350-350-350-200) GB RAID Z1 configuration would be equivalent to a (200-200-200-200) GB which turns to be roughly 600 GB. A (350-350-350) RAID Z1 configuration grants roughly 700 GB, which is 100 GB bigger!

The zpool was easily created with just one command and a few seconds wait:

# zpool create tank raidz c2t0d0 c3t0d0 c4t0d0 cache c6t0d0
# zpool status
pool: tank
state: ONLINE
scrub: scrub completed with 0 errors on Sat Apr 19 00:58:31 2008
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
cache
c6t0d0 ONLINE 0 0 0

errors: No known data errors

Now the pool's ready for hosting filesystems. I created one filesystem for every users' home and set a custom quota for everyone of them. The home filesystems are auto mounted at user login with just a one liner (assuming home filesystems are created in the /tank/zones/ssh-zone/home subtree):

# cat /etc/auto_home
+auto_home
* -fstype=lofs :/tank/zones/ssh-zone/home/&

I also set gzip compression for every filesystem with

# zfs set compression=gzip [filesystem-name]

The whole process took me less than half an hour (most of which I spent reading zpool and zfs man pages) and now have a single-parity RAID-Z pool hosting a quota-based, gzip compressed filesystem for every user, created just in time thanks to ZFS.