Showing posts with label dtrace. Show all posts
Showing posts with label dtrace. Show all posts

Sunday, 3 October 2010

Some Reasons Why Solaris Is a Great Java Development Platform

Some days ago I posted "The Death of OpenSolaris: Choosing an OS for a Java Developer" in which I stated that Solaris is a great platform for a Java developer. The point of that post was simply wondering about which Solaris version I'd use since the demise of OpenSolaris. What the post did fail in clarifying, as Neil's comment made me realize, were the reasons why you should choose Solaris as your development platforms. I decided to write this follow up to that post to quickly summarize my favorite ones introducing some use cases where such technologies come in handy.

Software Availability

Although Solaris continues to be a niche OS (such as many other platforms are, anyway) in the last few years Sun and the community made an excellent job at promoting it as a desktop alternative for developers. There existed even a specific distribution for developers: Solaris Express Developer Edition. It was discontinued and there really is no need for it nowadays, anyway. Late Solaris distributions (such as SXCE, OpenSolaris, OpenIndiana), include (either bundled or in the official package repository):
  • Data bases (MySQL, PostgreSQL).
  • Web Servers (Apache, Java Enterprise System Web Server, etc.).
  • Application servers (Glassfish).
  • The SAMP stack (Solaris + Apache + MySQL + PHP).
  • IDEs (NetBeans, Eclipse).
  • Support for other popular languages (Ruby, Groovy, etc.).
  • Identity management (LDAP, Java Enterprise System Identity Server).

Solaris also is a platform of choice in the enterprise hence common enterprise software packages are supported and you, as a Java developer or Java architect, won't miss the pieces you need to build your development environment. The very basic software packages I often need as a Java developers are:
  • Oracle RDBMS.
  • Oracle WebLogic Application Server.
  • IBM WebSphere Application Server.
  • JBoss Application Server.

Solaris' Technologies

Solaris has got some unique technologies that other UNIX (and UNIX-like) systems that might be used as development platforms are lacking (or ported from Solaris.) What's important here is not "technologies on their own" or technologies that are helpful only in big enterprise environments, but the fact that:
  • They're pretty well integrated in Solaris and are built to take advantage of each other.
  • There are common use cases in which these technologies are really helpful to a developer.

Each one of them would deserve several posts on their own, however, I'll try to make some concise examples.

Solaris Service Management Facility

Although this technology is probably most useful to a system administrator, as a developer I often took advantage of it. SMF is a framework that provides a unified model for services and services management. The basic recipe only needs an XML descriptor for a service. SMF lets you:
  • Define a service: startup scripts location, parameters and semantics.
  • Establish dependencies between services:
    • Services and service instances may depend on other service instances.
    • Service startup is preformed in parallel respecting service dependencies.
  • Enhanced security and fine-grained role based access control:
    • A service can be assigned only the minimum required set of privileges it needs to run.
    • Service management can be delegated to non-root users using Solaris RBAC (Role-Based Access Control).
  • Service health control:
    • Service auto-restarts.
    • Service health is enhanced by cooperation with Solaris Fault Manager which prevents service degradation when hardware failures occur.
  • Automatic inetd Services Wrapper: SMF automatically wraps inetd services.

A Typical Use Case

Every software package I use has its own SMF descriptor (either provided with the package or defined by me) and it dramatically reduces the time I need to set up a development machine. In the case of WebSphere Application Server, for example, I have separate service instances for:
  • WebSphere IHS.
  • WebSphere Application Server.
  • WebSphere Application Server DMGR.
  • WebSphere Application Server cluster nodes.

Dependencies are defined between them and I can startup the required WebSphere services with just a line of code:

svcadm enable [websphere-service-name]

and SMF will take care of everything.

The usage pattern for SMF can be enhanced further. Let's suppose you're working in one or more projects and each one of them requires distinct set of running services. What usually happens is one of the following:
  • You install them all and let them run.
  • You install them all and start and stop them manually when you switch working project.

Resources are always few for a developers and some are paranoid about sparing them. With SMF you can:
  • Define a SMF service for each of your projects.
  • For every projects, define dependencies with the services you need.

This way, at a minimum, you can start and shutdown, with a single command, every service you need for a specific project. No more:
  • Custom shell scripts for every service.
  • Custom configuration entries for inetd services (such as Subversion, Apache, etc.)
  • Specific OS customization.
  • Running services when you don't need them and waste resources you could use otherwise.

Example of SMF service manifest customization can be found in the following posts:

ZFS

The ZFS filesystem is unique as far as it concerns flexibility and ease of use. With an incredibly lean set of commands, you can:
  • Create file systems on the fly.
  • Snapshot file systems on the fly.
  • Clone file system on the fly with almost null space usage overhead.

There's a huge literature about ZFS and I'll limit to describe my favorite use cases.

Use case: Multiplexing Your Development Environment.

Software installations are just the beginning of your user experience. Often, we spend time:
  • Configuring our environments.
  • Fine-tuning them.
  • Defining the set of additional libraries we need.
  • Defining the set of server resources (JDBC, JMS, etc.) our applications use.

And so on. The list is endless.

Sometimes it's necessary to prepare different environments for different projects or different development stages of the same application. Instead of losing time and resource to build different environments I'll usually proceed as follows:
  • Install and configure my environment.
  • Make a ZFS snapshot of it.
  • Make a ZFS clone of it for every additional setup I need.

Oracle JDeveloper is a good example of an application I often clone. JDeveloper is fundamentally a single user environment, despite adopting the common approach of using a per-user configuration directory in the user's home directory. Instead of fiddling with scripts to set per-user configuration parameters, I just install it once, snapshot it's installation directory and make a ZFS clone, one per environment. I use several clones of the JDeveloper environment myself, in my user home directory.

The power of ZFS clones can be used by the Zones infrastructure, as we'll see in the following section, thus enhancing further its power. Cloning a ZFS filesystem is also advantageous while dealing with big installations such as disk images of your favorite virtualization technologies.

Additional posts I wrote about ZFS that could clarify some of its use cases are:

Containers and Other Virtualization Technologies

I consider Solaris a superior desktop virtualization platform. Once again, with a couple of commands. you can easily create a paravirtualized Solaris instance (a Zone). The zones infrastructure is ZFS-aware and can take advantage of it.

Zones can be configured with a command line interface to its XML configuration file. Creating a zone is straightforward and, since they're a lightweight technology, you can create as much zones as you need. If you're using ZFS, the process of cloning a zone is incredibly simple and fast.

Use Case: Clustering an Application Server

During the development of your Java EE application you will tipically need an instance of one (or more) of the following:
  • An application server.
  • A web server.
  • A data base;
  • An user registry.

It's also desirable to have them running on isolated environments so that you can mimic the expected production configuration. With zones it's easy: just create as many zones as you need and each one of them will behave as a separate Solaris instance: every zone will have, for example:
  • Its own network card(s) and IP configuration.
  • Its own users, groups, roles and security policies.
  • Its own services.

Instead of installing and configuring an environment multiple times, you will prepare "master" zones with the services you need. I've got a "master" zone for every one of the following:
  • WebSphere Application Server.
  • WebLogic Application Server.
  • Oracle DB.
  • MySQL DB.
  • LDAP directory.

and so forth. With one simple command (zoneadm clone [-m copy] [-s zfs_snapshot] source_zone) you'll end up with a brand new working environment in a question of minutes.

Use Case: VirtualBox and ZFS

Sometimes you'll rather work on a virtualized instance of some other OS, such as GNU/Linux, FreeBSD and Windows. Solaris is a great VirtualBox host and the power of ZFS will let you:
  • Create "master" images for every OS or every "OS role" you need.
  • Clone them on the fly to create a brand new virtual OS image.

In my case, I've got:
  • A master Windows 7 client with Visual Studio for .NET development.
  • A master Windows Server 2008.
  • A master Windows Server 2008 (a clone of the previous one) with SQL Server 2008.
  • A master Debian GNU/Linux.

Every time I need a new instance I just have to clone the disk image. In a matter of seconds I've got the environment I need. Not only I'm sparing precious time, I'm also sparing a vast amount of disk space. Should I store all of the images (and zones) I use without the ZFS technology and I'd need at least 4 times as much disks as I've got.

Use Case: A Virtualized Network Stack

Solaris provides you pretty powerful network virtualization capabilities. You can, for example, create as many virtual NICs as you need and use them independently either in Solaris Zones or as network cards for other virtualization technologies (such as VirtualBox.) Network cards can be interconnected with virtual switch (etherstubs) and enable you to create "networks in a box." Not only you can use virtualized instances to mimic your production environment: you'll be able to create a virtualized network to emulate the complex network policies your environment could need.

If you need to test an environment whose configuration would be impossible to replicate without additional physical machines, that's where virtualization technologies (such as Zones or VirtualBox) and the virtualized network stack come in handy. My developer environment for a project I'm working for is made up of:
  • Two zones with two load balanced IBM IHS instances.
  • A zone with an LDAP directory.
  • Two zones with two clustered instances of IBM WebSphere Application Server.
  • A Zone with an instance of IBM WebSphere DMGR.

With Solaris, I can replicate the production environment in my box and respect each and every network configuration we use. Without these technologies, it would have been much harder to accomplish this goal or I would end up with custom configurations (for example, to avoid port clashes). In all cases, I'd lose much more time on the administration and configuration of such environments if zones weren't so easy to use.

DTrace

DTrace power is extremely easy to explain to a developer. At the same time, it's difficult to grasp its usefulness without trying it yourself. DTrace on Solaris provides tens of thousand of probes out of the box and others can be created on the fly. This "probes" provide you an extremely powerful mean of troubleshoot problems in either your applications and the underlying operating systems. To use the probes you've got to use scripts written in the D language. Fortunately, this language is pretty easy by design and you can write powerful D scripts in a few lines of code.

DTrace is unobtrusive and let you troubleshoot problems immediately, without modifying your application, even in a production environment. Some IDEs, such as NetBeans, have powerful plugins that let you write D scripts and see the data collected by the probes in beautiful graphics.

As a developer, I valued DTrace usefulness more than once. Instead of troubleshooting problems having to dig into the source code and introduce additional code (even in the cases in which aspects come in handy), I could use a D script to observe the application from the outside and quickly collect data that could help me determine where the problem could be.

In some cases, moreover, you could find yourself dealing with situations in which there's no code available. I could quickly troubleshoot a problem I was having with WebSphere Application Server with a D script instead of relying on WebSphere tracing facilities and the task of interpreting log files.

Conclusion

So much for an introductory post. The possibility of building a development environment as close as possible as your target environment is a "must" for any development platform. Additionally, I consider that working on a environment as close as possible as the production environment not only gives you additional value and insights during an application development stage, but should also considered a mandatory requirement for every project we're involved into. Solaris provides all of the tools a developer need to accomplish this goal.

Solaris is a complex enterprise operating system with many features you won't probably ever use. Nevertheless, there's a use case for many others of them, as I tried to point out in this post. Since some of these technologies were developed with an open source license, they are also available on other operating systems: ZFS is available on FreeBSD and there exist a community effort to port it to OS X; DTrace is available on OS X, Linux and FreeBSD.

The "Solaris advantage" is that all of these technologies are highly integrated and take advantage of each other. The result is worth more than the sum of them. These technologies have got a very polished and easy to use administrative interfaces: when time is important, "How you do it" is fundamental.

I hope that these insights might help you understand if and when the Solaris operating system might be useful to you. Even if you consider that it's not, I suggest you give it a try anyway: it's always good to add new technologies to your tool box.

Wednesday, 29 September 2010

The Death of OpenSolaris: Choosing an OS for a Java Developer

A Bit of History: The Struggles of OpenSolaris

This is no news: you probably know all about it.

As a long time Solaris user, the recent years have been full of good news for me.

I remember starting with GNU/Linux at home to have "something similar" to the Solaris workstations I used at work. It was the time when software would most likely compile on Solaris rather than on Linux.

Years later I bought my first Sun Workstation: it was the time when trying to compile on Solaris packages that would supposedly compile on a POSIX system was a pain. Still, I continued to regard Solaris as a stable platform to keep on using it for my work duties, such as Java programming.

Then came Solaris 10 and all of its wonderful technologies such as ZFS, Zones and DTrace, just to cite a few. With it, there came the Solaris Express distributions which, at last, filled a long standing gap between Solaris and other operating systems, providing us a pretty modern desktop environment.

In late 2008 came the first OpenSolaris distribution. I installed it, played with it, but kept on using SXCE for my workstations. The main reason was compatibility with many Sun packages, such as the Java Enterprise System or the Sun Ray Server Software, that had more than one glitch on OpenSolaris.

When SXCE was discontinued, I waited for the 2010.xx OpenSolaris release to upgrade my systems. Unfortunately, that release will never see the light.

The Oracle Leaked Memo (the dignifying uppercase is a must given Oracle prolonged silence over the subject) shed a light over Oracle plans for Solaris proper and OpenSolaris. Part of the "good news" is that the Solaris Express program has been resurrected and the first binary distribution is expected later this year.

The bad news is that the code, at least the parts of it that will be released with an open source license, will be released after the releases of the full Solaris Operating Systems. Basically, our privileged observation point over the development of the operating system has been shut down.

Lots of ink has been been spilled since the Leaked Memo and plenty of information, discussions and wars are going on in the blogosphere. I'm not an authoritative source to speak about the subject and it's not even clear to me what I'm going to do, now.

Benefits of Solaris for a Java Developer

Solaris has been my operating system of choice since before I started working in the IT industry. As a student, I grew up with Solaris at the data center of my University and the Slackware I used at home seemed like a kid toy, compared to it. After graduating, I started working as a design engineer for a leading microprocessors producer. Needless to say, Solaris was the platform we ran our design software upon. Then, I moved to a consulting firm and started my career as a Java architect.

Solaris was and is the platform of choice for most of the clients I've been working for. Even today, the application servers, the cluster software, the database, and most of the infrastructure used by my clients run on Solaris. It always seemed a sound choice to me, then, developing software on the same platform that will run it in production.

IDEs, Tools and Runtimes

As a Java developer, I can run all of my tools I need on a supported platform. My favorite IDEs (NetBeans and JDeveloper), the application servers my clients use (WebLogic and WebSphere, mostly), the databases used by my applications (MySQL, Oracle RDBMS, PostgreSQL): all of them run and are supported on Solaris. Some of them are even bundled with it or readily available by Sun sponsored package repositories. The Eclipse platform, to cite a widely use IDE for Java, is available in the OpenSolaris IPS repository, too.

Solaris Technologies

Solaris 10 also integrates DTrace, a powerful, unobtrusive framework that allows you to observe and troubleshoot application and OS problem in real time, even in production systems with an almost null overhead. DTrace has allowed us to diagnose strange production quirks with no downtime: once you've tried DTrace and the D language, there's no going back to "just" a debugger, even in the development stages of your project.

Other kinds of problems does not show up in your debugger or are extremely hard to catch. It might be the case of network or file systems problems. That's where DTrace comes in handy: it lets you observe with an incredibly high detail what's going on in your application and in the kernel of the operating systems, if it's necessary to dig so deep.

Solaris Virtualization Technologies

Solaris is also an ideal host virtualization platform. Solaris can "virtualize itself" with Containers, Zones and Logical Domains: you can start a Zone in no time (and almost no space overhead), assign a resource cap to it and also build a virtualized network in a box to simulate a complex network environment.

One of the problems that I encountered during the development of big enterprise system is that the development environment, and sometimes even the integration environment, is very different than the production one. It surely is a methodology problem: nevertheless, developers have few weapons to counteract. For example, applications that appear to run fine on a single node may not run correctly on a server cluster, or scale badly.

The more you wait to catch a bug, the more impact will have a patch for it. That's why in my development team, for example, we use Solaris Zones to simulate a network cluster of IBM WebSphere Application Servers and a DB cluster. All of them run in completely isolated Zones in one physical machine and communicate on a virtual network with virtual etherstubs (sort of a network switch), VLANs and routing rules. This environment lets us simulate exactly how the software will behave in the production system. Without a flexible and lightweight virtualization technology it would have been much more difficult and expensive to prepare a similar environment.

And if you (still) need to run other platforms, you can use Xen or VirtualBox to run, for example, your favorite Linux distro, Windows, or *BSD.

Summarizing

Enumerating the advantages of Solaris is difficult in such a reduced space, however I'll try:
  • It's a commercially supported operating system: that's an option, since Solaris is free for development purpose. Nonetheless, it's an important point to take into account.
  • Is (very) well documented: there's plenty of official and unofficial documentation.
  • It's easy to administer: Solaris is pretty easy to administer, even if you're not a seasoned system administrator.
  • It's an UNIX system: with all of its bells and whistles.
  • It is a great virtualization platform.
  • It has some unique technologies that add much value to its offering, such as ZFS and DTrace.


If you're a Java developer and haven't given Solaris I try, I strongly suggest you do it. Maybe you'll start to benefit from other Solaris 10 technologies such as Zones and ZFS, even for running your home file or media server.

Complaints

I often hear complaints about Solaris coming from different sources and with the most imaginative arguments: proprietary, closed, old, difficult to use. I usually answer inviting users to try it and see for themselves before judging it (if that's the case). Most of the times I'm not surprised to discover that the complaining guy had minimal or null exposure to Solaris.

Also, I'd like to point out that no OS I tried is a swiss army knife. Solaris is a server-oriented OS with a good desktop but it's not comparable with other operating systems for such an use. So: no comparison with Linux, please. It would be so unjust as comparing Linux and Mac OS X for the average home user. ;)

Alternatives

Since Java "runs anywhere", there's plenty of choice for a Java developer.

Since I own a laptop with Mac OS X, I've built a small development environment with all of the tools I need. Mac OS X is a great operating systems that comes with many fancy features out of the box and, although it has some idiosyncrasy with Java (read: you have to use the JVM shipped by Apple), it's a good OS for a Java developer. Since the Mac OS X hype has begun, there's plenty of packages for it and a big ecosystem which is still growing. Still, many software packages run in the enterprise aren't supported on Mac OS X. Since I prefer to have an environment as close as possible as the production one, I think that OS X is not the best choice for the average Java EE architect.

I've also been an hardcore Slackware and Debian user for a long time. An enterprise Java developer would miss nothing in a modern GNU/Linux distribution, nowadays, and most of the software packages you'll find in the enterprise will run on your GNU/Linux distribution.

No need to talk about Windows, either.

So, why Solaris? Every OS has its own advantages and disadvantages. The point is to just recognize them. Mac OS X, in my opinion, is the best OS for a home user. I would change it for no Windows and no Linux. But as far as it concerns my developers' duties, every other OS just lacks the features and the stability that make Solaris great. ZFS, DTrace and Zones, for my use cases, are killer features.

What's Next?

You've decided to give Solaris a try, so: which is Your distribution? I don't know.

Solaris Express/Oracle Solaris

I strongly suspect that my wait will be prolonged and I will finally upgrade my machines as soon as Solaris Express has been released. Upgrading to Solaris 10 09/10 is not possible since I'm using some ZFS pools whose version is not yet supported by Solaris proper but it is a sound choice for a starter.

The advantage I see in using one of these versions is the availability of optional support and the good level of integration with the most commonly used software packages that Oracle is likely to guarantee.

OpenIndiana

You should also know that OpenSolaris sources have been (sort-of) forked and two new projects are born: Illumos and OpenIndiana. The project were started by Nexenta employees and volunteers of the OpenSolaris community. The first projects aims at maintaining the OpenSolaris code and the parts of the code that are closed or code that upstream might choose not to maintain. The OpenIndiana project aims at producing binary distribution of the operating system built upon the Illumos source code. OpenIndiana will provide a really open source, binary compatible alternative to Solaris and Solaris Express.

Sounds good and I'll willingly support it. In the meantime I've installed OpenIndiana in a couple of virtual machines and the first impressions are very good. I suppose it hasn't passed enough time yet for diverging changes to have emerged.

If you prefer a more modern desktop with a recent Gnome interface, drop Solaris 10 and go for OpenIndiana, if you don't feel like waiting for Solaris Express. In any case, switching between the two shouldn't pose any problems. What's clear to me is that I won't consider using both operating systems: I'll have to make a choice.

Support Availability

As an enterprise user and a Java developer, I've always been more concerned about OS support and support for the packages I use, rather than about eye candy. Even at the cost of running a proprietary platform.

In conclusion: I'll wait for Solaris Express to be released and only then will decide which one I'll use for my purposes between Oracle Solaris Express and  OpenIndiana. My heart is betting for OpenIndiana. My brain is betting for Oracle Solaris Express and Solaris proper. Only time will tell which one is right (for me.)

Follow-Up

A follow-up of this blog post is avaible at this address. In this post I'll try to summarize some use cases in which the technology we introduced in this post are effective and add real value to your development duties.

I hope you enjoy it.