Just noticed the site update today at http://www.opensolaris.org/. The website will be decom-ed after March 4th. This is the final symbolic nail in the coffin of what myself and many others I know had worked very hard for and shown great passion. The nail was driven long back with the Oracle acquisition of SUN, soon the post-mortem report will be filed in some dusty corner of the great hall of Oracle. It was simply a fantastic OS platform but a great example of Free and OpenSource software done wrong, completely wrong by clueless management who would not listen to people who have lived and breathed OpenSource for many years.
I still remember the churning and frustration I felt inside when SUN launched OpenSolaris (http://desktoplinux.com/news/NS2543956200.html) without even a passing reference to BeleniX on which it was built. The first release of OpenSolaris was simply BeleniX outfitted with new clothes. I had toiled hard, very hard to bring all the distro and livecd technologies to the OpenSolaris platform. Given it was OpenSource and a labor of love, but when you take something in it’s entirety and completely ignore the creator/origin, it is equivalent to stealing. Forget about mentions in public, I did not even get any recognition within the company (apart from a few bones in the form of a few $$ being thrown) for all that work that made one of their flagship OS platform releases possible. The vitriol that had had burned me from inside is hard to describe. It was at that instant I had decided on leaving SUN where once I had planned to build a long-term career.
People doing hard work, good work and not getting recognized within organizations typically happens due to lack of visibility of their work outside of their silos. All of SUN knew about my work till the CEO and still nothing. I was completely and thoroughly sidelined while other teams did launches and launch parties. I never quite understood what was my fault. Or was it politics?
This is now long past and I was over that hump by the end of 2008. However it still remains as the most bitter memory of my life till date (apart from my father losing both his kidneys). Fortunately the work environment I went into after leaving SUN is quite different, in a positive way and is far from these issues.
Anyway, may you rest in peace OpenSolaris.
Deutsch: Logo von OpenSolaris als Vektorgrafik (Photo credit: Wikipedia)
I had to upgrade a build server running BeleniX 0.8 recently to OpenIndiana. This was needed to move to a more updated environment for building the next BeleniX release. Obviously BeleniX 0.8 being a custom-built distro based on SVR4 packaging had no upgrade path to an OpenIndiana release based on IPS.
What I actually needed was to have OpenIndiana installed into a new ZFS boot environment and boot from it which would allow me to go back to BeleniX 0.8 if I needed it. I of course had the same capability in BeleniX via the Network Installer that I had written earlier. Now I needed the same for OpenIndiana. So I spent a week modifying and experimenting and I now have a network installer for OpenIndiana. You can grab it from http://www.belenix.org/binfiles/install_openindiana.
Obviously I will have to update the BeleniX version once the next release is out since we will be moving to an RPM5 based packaging. This network install technique can be adapted to any distro and will allow a multi-boot setup based on ZFS boot environments. Apart from this there are other possibilities that I can think of. At present the network install script uses the package collection from the slim_install package group. A new package group for a non-GUI base environment can be leveraged via this script. This can also play a part in a minimal CD based environment to quickly install predefined setup, something not unlike the Automated Installer but without all that complexity. If gPXE can be properly made to handle booting an OpenSolaris kernel then it will be possible to deliver such install environments over HTTP for eg.
I have recently started playing with OpenIndiana as part of renewed effort in the BeleniX space to realign with the latest developments. We intend to collaborate and participate in the OpenIndiana community. We are still firming up the plans and approaches.
In the meantime I have been playing with OpenIndiana a bit and find it quite appealing once the few initial hiccups are worked around. They are minor in any case. It is fast and smooth especially the speed with which Firefox loads. This is how the OpenSolaris distro project should have been setup I feel, driven and “owned” by the community from which SUN could have derived value. However communities were formed and then messed up. For example http://lwn.net/Articles/370157/, and BeleniX efforts which were broadly leveraged to build a flagship product and then the original source summarily ignored.
However that is all past now the organization having succumbed to the many problems that riddled it. Some things stand out like diversity issues and lack of business innovation. There are interesting developments happening now in the fledgling community around Illumos and OpenIndiana. I am excited by all that and motivated to participate and contribute.
OpenIndiana in all this is really looking good and I highly recommend it for folks to try out. The technical credit no doubt goes to the erstwhile SUN engineers around distribution, OS, install, desktop and related groups who put in countless hours of effort. Credit also goes to folks like Alaisdair, Guido for getting the community to rally round all this, organizing and actually delivering something excellent which was earlier missing in action. This is just the start though and a lot of concerted effort is needed by the community to keep this going and developing. Lets hope for the best.
On the lines of the excellent bio by Peter Tribble I am posting my own bio here as required by the OGB candidacy rules.
DECLARATION OF INTERESTS
- I am Platforms Engineer cum OS and apps Developer presently employed with Goldman Sachs. Goldman uses SUN Solaris and Redhat Linux among other platforms in the firm. However my engagement with the OpenSolaris community is a hobby activity that I do outside of my work with my employer’s consent. As such all my views and actions are entirely my own done using personal resources and time and not related in any way to my employment.
- I have been a user and developer on Solaris and Linux platforms for the last 12 years (of a total industry exposure of 13.5 yrs) and have been participating in the OpenSolaris community from it’s early days. I have previously been employed by SUN in the Solaris Sustaining engineering group working on various aspects of the OS from userland to the kernel.
- I am also the creator of the BeleniX distribution of OpenSolaris: http://www.belenix.org/. It was the second distribution of OpenSolaris that came out after SchilliX borrowing some concepts from SchilliX. It was the first non-SUN OpenSolaris distro to bring a full-fledged GUI desktop based completely on the X.org OSS stack and eventually matured into a stand-alone desktop distro. It brought in several innovations to OpenSolaris and formed the foundation for the OpenSolaris distro from SUN.
- I am a core contributor in a few OpenSolaris communities like X-Windows, Distribution etc. I contribute to the Fully Open X project off and on and have recently started another project called libtaskq (http://sourceforge.net/projects/libtaskq/) based on the TaskQ kernel framework from OpenSolaris.
- I co-lead one of the oldest and very active OpenSolaris user groups, the Bangalore OpenSolaris User Group with another OpenSolaris community member Sriram Narayanan.
I live and work in Bangalore, India’s Silicon capital. However I was born in the eastern city of Calcutta which was once the capital of the British rule in India. I did my studies in Calcutta at Asutosh College under the auspices of the University of Calcutta. However I do not come from an Engineering background. My majors in Graduation were Geography, Geology and Economics while I studied Economics, Statistics and Maths in high school. I had a deep interest in Biology and Geomorphology till high-school and actually wanted to do Biotechnology as a career!
However I developed an interest in Computing as a hobby during the March of 1990 (thanks to my mom) and my first exposure was on the BBC Model B microcomputer which I hacked to death at my Mom’s office – Birla Industrial and Technological Museum. After that I completed all the typical topics of a Computer Enginering course as a hobby while studying Geography and moved from the BBC to a PC-AT and all the subsequent Intel processor models.
After hacking around with Borland Pascal, C/C++, Win32 etc. my first introduction to *nix was on Slackware Linux 0.1. By that time I have completed a PG diploma course on Software Engineering and my first job had me working first on FoxPro and then on Oracle on WinNT.
My second job provided me a big break when I joined HCL Technologies in the southern tropical city of Chennai where I started working at the dedicated Cisco offshore development center. That was the time when I came into touch with Solaris 2.5.1 logging onto large engineering servers via big-screen TektroniX X-Terminals. That experience at HCL – Cisco provided me with a wealth of resources and expertise. I later started having my first SPARC desktop and SUN Ultra 5. I worked across various Cisco groups including Test Automation group, Network simulators, Network Management group with my work touching a vast array of computing technologies starting from router chips and OS platforms and continuing till Java and webservices frameworks. I played with the guts of routers costing hundreds and thousands of dollars apart from a variety of SUN Servers.
After my 5.5 yr stint at HCL – Cisco I decided to accept an offer at SUN Microsystem’s Solaris Sustaining Engineering group and worked there for 4.5 yrs till the middle of 2008 when I jumped ship to Goldman Sachs in their Platforms Engineering group. In SUN I worked on various pieces including, commands, libraries, systems management and a few kernel components as part of my OpenSolaris dabblings.
I have been a voting member of the OpenSolaris community from some time.
Ksysguard working on OpenSolaris
Anyone who might have tried the earlier KDE4.3 packages for BeleniX may have noticed that Ksysguard (CTRL+ESC or Kmenu -> Applications -> System -> System Monitor) basically shows a blank slate. The process list is empty, CPU and Network stats are unavailable. The number of exposed sensors are too few.
I spent the last few days hacking on that component and got an initial working version that implements all the basic functionality for the OpenSolaris platform. There are still bugs to iron out and new sensors to add (using DTrace here can open up lots of possibilities). The current patch is here. The kdebase4-workspace package has been published into the BeleniX repository.
Ksysguard working on OpenSolaris
I recently received my complimentary copy of “OpenSolaris Bible” thanks to Wiley. I have been identified as one of the contributors as I helped review content for Chapter 2. There are good write-ups on the various OpenSolaris distributions and I am happy to note the nice stuff written about BeleniX.
This is one massive piece of work no doubt apparent from the bulk of the book itself. I have been leafing through the pages and found this to be an invaluable reference for my day to day work. The text is detailed and lucid with lots of good examples. I found the little “Cross-Ref” entries to be very helpful. These direct the reader to related content in other chapters – probably the closest that one can come to hyperlinks in printed content, but at the same time a fair bit more detailed than simple hyperlinks.
I’d recommend anyone using or working on OpenSolaris to grab a copy of this very definitive reference.
Just noticed this bit of news: http://www.tgdaily.com/content/view/41348/118/. Steve Wozniak one of the top wizards of the Computer Industry has joined Fusion IO. Now an obvious thought comes to mind. How much will it rock having ZFS L2ARC on a Fusion-IO device ?
In continuation of my compressed ramdisk work I took the liberty to prepare a modified MilaX 0.3.2 test ISO image that contains a lofi compressed ramdisk image dynamically decompressed on the fly at runtime. You can download the ISO from here: milax032_cramdisk.iso, milax032_cramdisk.iso.md5sum. This image is also modified to mount ‘/’ with noatime option to reduce unnecessary writes to the ramdisk.
The source code and diffs bundle is also updated with a few more changes to optimize memory usage and performance and now has binaries for B95 since MilaX 0.3.2 is based off B95.
Many will be knowing that all the distros of OpenSolaris use a ramdisk when booting off a livecd. In fact a ramdisk is used also during normal boot off harddisk. This is called a boot archive (/platform/i86pc/boot_archive for eg.) for a normal harddisk boot and miniroot or microroot in case of livecd.
These are basically compressed filesystem images that get decompressed and loaded into RAM by Grub. The kernel loads basic essential modules from this image in RAM till it is ready to access the real root filesystem on disk and continue with normal boot. In case of a livecd it keeps on using this pseudo RAM-disk as it’s root filesystem, since root filesystem must be read-write.
The ramdisk size on OpenSolaris has been growing quite large due to the kernel itself and system libraries growing in size/number. The livecd ramdisk is quite a bit larger than the boot archive. In addition the ramdisk on SPARC is particularly big as RISC binaries are larger. There have been techniques introduced to tackle this for the standard Solaris install miniroot and the boot archive. One of them is Dcfs that I blogged about earlier. The other one is individual files compressed with Gzip inside the boot archive. These are filesystem-level changes that have introduced a couple of layers in the kernel to deal with different situations. But the livecd ramdisk size is still big and using Dcfs for this purpose has issues (dcfs is read-only) as discussed in this thread: http://mail.opensolaris.org/pipermail/caiman-discuss/2008-December/008263.html
The livecd ramdisk size also affects BeleniX and it uses various techniques to reduce the nunber of files included, but it is still an issue. Seeing the above discussion thread it recently occurred to me that moving compression to the ramdisk module makes it transparent and avoids multiple different filesystem-level implementations. The same lofi compression approach used to compress livecd contents can be used but it must be tweaked to make it read-write. One easy way to allow writes is to do a per-segment copy on write since the image is broken up into fixed-size segments and each segment individually compressed. Since this is a ramdisk, new memory can be allocated the first time a segment is written to. The segment is uncompressed and stored there and all subsequent access to this segment goes to this memory block. A different index holds pointers to these alternate segments or 0 otherwise.
However changing the ramdisk module alone is not enough. The ramdisk module is not available very early in boot. The kernel runtime linker at this stage is tasked to assemble essential modules for the kernel to function normally. It thus uses a very simple ramdisk reading rig along with very basic filesystem support. See here: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/krtld/bootrd.c and here: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/fs/. Incidentally you will notice a decompress.c file that implements decompression of Gzipped files in the boot archive. These things (along with Zlib code) get complied into the kernel binary (/platform/i86pc/unix for 32-bit x86). If we are using a compressed ramdisk then these places need to be changed. Note also that Grub places the initial ramdisk in contiguous physical memory which makes the diskread function in bootrd.c very simple.
Taking advantage of a bunch of holidays I started hacking on this and I am pleased to note that I have a working implementation now. That is the kernel is able to boot off a lofi-compressed ramdisk and a writable compressed livecd ramdisk also works. Making changes to the ramdisk module was fairly easy and it took me a day to get a working read-write implementation. Changing bits of the early boot code in the guts of the kernel was much more tricky and took me about 6 days. Problems at this early stage are hard to debug since kmdb itself is a module and is not yet loaded leaving only printf debugging. But too much of printf output is also bad as they scroll off the screen! Memory corruptions result in an “Unexpected trap” message and backtraces do not appear most of the time. There was a performance issue as well mostly caused by zlib’s inflate being called too many times. I resolved this by adding a minimal LRU cache for holding uncompressed segments. Another thing I had to tackle was caused by the way pointers directly into the ramdisk area are doled out to support so-called “cached” reads. This does not work for a compressed ramdisk.
I have placed a tarball here: http://www.belenix.org/binfiles/cramdisk.tar.gz . This tarball contains all the modified source code and pre-built binaries. One can test the ramdisk module by following instructions that I emailed to caiman-discuss. Testing the modified kernel can be done by folks creating livecds. These changs are against B104. This stuff is initial code and I am still testing it. I have found that using a 64K segment size gives a good balance between compression effectiveness and performance overhead.
This approach has advantages. It results in a smaller boot archive than can be had by current approaches. Since this is filesystem-agnostic, 2 different filesystem-level implementations (Dcfs and Gzip) are not needed. Grub loading time from livecd is reduced by orders of magnitude since Grub does not have to decompress. Finally it mitigates the livecd ramdisk size issue where using Dcfs is problematic.
To make my own life easy I finally sat down and wrote a utilty that automates the entire process of patching and building OpenSolaris and XVM(XEN) from source and also generates the binary packages. I had wrote about building OpenSolaris from source on OpenSolaris itself. See my blog entry titled “Self – Host Almost! Building OpenSolaris on OpenSolaris“. That entry missed out a few details and in any case there is a lot of minutiae making for a mistake-prone manual work.
The utility takes a workspace containing the OpenSolaris build snapshot tarballs and patches to apply and at the end delivers binary packages. It is also has the ability to check for pre-requisites for running the build. For eg. presence of proper version of SUN Studio 12, development and locale headers, assembler etc. It should work on both BeleniX and OpenSolaris though I have found time to only test it on BeleniX till date.
Getting it: On BeleniX just execute spkg updatecatalog; spkg install autobuilder. On OpenSolaris execute pkgadd -d http://www.belenix.org/binfiles/autobuilder.pkg
Setting up: Create a directory like say osol_ws. Now create osol_ws/downloads and osol_ws/patches. If you have patches for ON create another directory osol_ws/patches/on_patches and copy them into it. Copy your XEN patches if any into osol_ws/patches/xvm_patches. Now run osol_builder prereq while connected to the net. This will check for prerequisites for building OSOL on your distro and fix most of the requirements except for installing the proper SUN Studio 12 which you will have to do manually since it requires a login to download from SUN’s site.
Building: Now download all the requires build snapshot tarballs into osol_ws/downloads: on-src.tar.bz2, xvm-src.tar.bz2, SUNWonbld.i386.tar.bz2 or SUNWonbld.sparc.tar.bz2 , on-closed-bins-nd.i386.tar.bz2 or on-closed-bins-nd.sparc.tar.bz2. Finally fire up the script: osol_builder build -R /path/to/osol_ws -d myosol
Running the utility without arguments prints a complete help text that will explain the subcommands and options.
I have only been tested this on i386, though it should work on SPARC as well since the architecture is detected at runtime. However the script only builds for non-debug stuff. So support needs to be added to allow building debug binaries as well. In addition one may want to only build either XVM or ON whereas today it builds both. It delivers a pre-determined set of nightly options (in workspace/patches dir) that may not be suitable to your needs. This needs to be customizable. Obviously one can think of other enhancements and obviously fixes/enhancements to the utility are welcome. Since the utiliy is a mix of little bit of shell script and Python the source code is there when you install the package. Look at: /usr/bin/osol_builder, /usr/bin/osol_builder.py.
If you do find the script useful or find bugs please drop an email to email@example.com or even put a comment here.
Update(Dec, 17, 1008):
The package file is now updated as I discovered a bug and also changed a patch into a script to avoid having to maintain a build-specific patch.