Cisco SG500-28 Swith

Wow.  Normally, I try to keep fairly positive when reviewing items, even if they have some down sides.  But this switch has me absolutely astounded with just how horrible it is.

Let's start out with the first thing that I noticed.  The documentation says that it will accept a DHCP lease, but if one is not available, it will fall back to a default IP.  In actuality, when the unit boots up, it tells the DHCP server that it wants to use a specific IP address which is NOT the one mentioned in the docs.  The DHCP server says "No", then offers a valid lease.  The switch never responds to the DHCP server, and does not respond on the documented IP address, the one it asked for, OR the one it was offered.  So much for the DHCP client.

Moving on, if you change the stacking mode on the switch, all other settings (including passwords) get reset to factory defaults.    That's pitiful.

Looking through the firmware release notes, horrible bugs have been fixed (a timeout in the web management could crash or reboot the switch!), but other bugs remain, with no workaround.  Browse the release notes, see for yourself.

I can't do five minutes of configuration on this switch without encountering yet another "What were they thinking?" moment.  Needless to say, more of these switches are NOT in my future.

SuperMicro TwinBlades, Software Setup

So, I've covered most of the hardware for the SuperBlades.  But what about a software configuration?  These machines are acting as front-end web servers, which pull code and data from a NAS. However, there's no reason to have enterprise disks in each of these blades just for the OS, so let's get some diskless booting happening!


The first TwinBlade got the same 40W CPUs as the others, but each got a pair of 600GB Velociraptors in RAID 1.  There we go, single disk failures are taken care of.  But what about an OS crash, a fat-fingered remote shutdown, or other issue?  Well, there is a nifty piece of software called DRBD, which essentially acts as RAID 1 across a network.  So, complete data sets exist on 4 disks, across two systems.  By tying in a couple of other packages, active or passive takeover can be achieved if one system fails.

NOTE:  These will NOT be exhaustive steps, just the important bits to get it all to work.
Take your primary and backup machines, and install DRBD on them.  Take your disk or partition, and set them up as a DRBD device.  If you're going to use DRBD, you need to understand the mechanics, so read their documentation for setup info.


Then, mount your DRBD device.  We'll use "/export" as a mount point for this example.

Now, first you need to make sure that each machine gets the right info, so let's do some DHCP work:

host blade-1 {
hardware ethernet 01:02:03:04:05:06;
fixed-address 10.11.12.13;
next-server 10.11.12.14;
if ((exists user-class) and (option user-class = "gPXE")) {
        option root-path "iscsi:10.0.1.28::::iqn.2012-04.com.foobar.iscsi1:blade-1";
        filename "";
  } else {
        filename "undionly.kpxe";
  }
}

This does a few things.  First, it gives the machine a fixed IP address.  It then tells it where to look for the next bit of the boot sequence, via the "next-server" statement.  Then, the if-else bracket tells it to grab the "undionly.kpxe" bootloader (courtesy of gPXE) if is acquiring a DHCP address for the first time.  But, once it has that, and is executing the gPXE environment and asks for the DHCP info again, it instead gives it the necessary info for gPXE to get the iSCSI disk going.  Crazy stuff.

So, then you need a TFTP server on 10.11,12.14, handing out the "undionly.kpxe" file - which can be compiled from the gPXE source.  Read-only, don't be a security fool.

Then, create a disk file on your iSCSI target:

dd if=/dev/zero of=./blade-1-disk.img bs=1 count=1 seek=17179869183

Why 17179869183?  Well, that's 16 gigs minus the one byte which we will write, so the file ends up at exactly 16 gigs.  Since we only wrote a single byte to the end of the file, not only is the creation fast, it also takes advantage of Linux's ability to use sparse files, which will save you some disk space - at least until you actually fill that up.

Now, let's set up IET to serve that out.  First, in your ietd.conf:

Target iqn.2012-04.com.foobar.iscsi1:blade-1
        Lun 0 Path=/export/blade-1-disk.img,Type=fileio,ScsiSN=Blade1
        InitialR2T no
        ImmediateData yes

Then, in your initiators.allow so that only the relevant machine sees the LUN:
iqn.2012-04.com.foobar.iscsi1:blade-1-disk.img 10.11.12.13

Now, you should be able to start up your CentOS installer, choose "advanced storage configuration", install to the iSCSI target, and rock and roll.

SuperMicro sbi-7226t-t2 TwinBlade

Now that we've discussed the chassis, let's talk about the blades themselves.  SuperMicro has blades optimized for storage, for GPU usage, or simply for traditional builds.  The TwinBlades are optimized for density, packing two entire dual-socket systems onto one board.  It's actually quite similar to their "1U Twin", but with both systems on a single PCB.
 

Each of these systems gets two LGA 1366 sockets, 8 DIMM sockets, 2 x 2.5" HDD hot-swap bays, and dual gigabit ethernet ports.  They also have the option of a 10GBE or Infiniband riser card.  They also have onboard IPMI, which coupled with the chassis's CMM, gives you a very powerful remote management capability.  Individual systems (systems, not blades) can be powered up, powered down, reset, have remote KVM, be monitored for temperature and electrical draw, and more.   CPUs may be used with a TDP up to 95W.

SuperMicro SBE-720E-R90 Blade Chassis

Blade systems are, by definition, built in, on, and around the idea of a chassis, and SuperMicro's offerings are no different. A blade chassis consists of much more than just some sheet metal to hold parts in place. They are complex, compact, wonderful pieces of hardware!

Starting out, SuperMicro has a few different blade chassis available. If you want to use their TwinBlades, you need to use their 720D or 720E chassis. The "E" model can handle larger power supplies, and has extra expansion slots for Infiniband switches.

Each of the chassis can handle up to ten blades, including their TwinBlades. TwinBlades have two seperate, dual-socket systems on the same blade - so, in essence, you can pack up to 20 dual-socket servers in 7U!

Density, however, is only one of the benefits. Another is the chassis management, or CMM module. The first function of the CMM is to provide local and remote KVM, with virtual media redirection, allowing you to perform cold-boot configuration and installation from remote locations. Secondly, it performs power monitoring and management on the units, allowing you to power blades up and down as needed. It also provides IPMI on a chassis level. This is a tremendously cost-effective solution, as 3rd-party remote KVM and power management systems can be VERY expensive. There is, however, one catch: While SuperMicro's marketing says that a CMM unit is included, mine showed up without one, and I was told that I have to buy one separately. That seems a bit suspicious, but it is the first time I've had ANY negative experience with SuperMicro - and I've spent a lot of money with them. They've pulled through on things like finding parts that were long out of production and obsolete years before I called, so I'll let this one thing slide.


Dense I/O is another benefit. The chassis have slots for various I/O modules, and SuperMicro has modules and switches for gigabit ethernet, 10 gigabit ethernet, and 10/40gb Infiniband. As mentioned earlier, slots for Infiniband switches appear on the 720E model, but not the 720D.  Be sure to check their networking matrix to find the correct networking options for your chassis and blades.

The last benefit that I will cover in this post is the power. This chassis uses HIGHLY efficient power supplies (94%+!). While that may not seem like a big deal, consider that compared to lesser power supplies, that alone can shave 5%-15% off of your electrical budget, which also shaves 5%-15% off of your cooling budget. For disaster contingency, that also translates to longer UPS runtime, as well as lower fuel usage and longer runtime for your generator. Additionally, the benefits of power supply consolidation kick in, and all in all, this is a very electrically-efficient system. That may not seem flashy, but anyone who has worked in high-density installations will appreciate the energy savings.

Now, some notes on power: 40 sockets, 160 DIMMS, and 40 hard drives still take lots of juice. Plan on two 220V circuits, 120V need not apply.  While more power-hungry CPUs (and GPUs!) can suck down more power, by using 40W TDP L5630 CPUs and booting diskless from a SAN, my systems pull just a tiny bit over 100W each, which includes the power used for the CMM, switches, fans, etc..

If you are not filling all of the slots with blades, dummy blades must be used to manage airflow. I was originally told that the chassis did not include any dummy blades, but when it showed up... it was, indeed, populated with just enough dummy blades to fill the slots that I wouldn't yet fill. SuperMicro tells me that these chassis are individually configured for each order, so that would explain the attention to detail. SuperMicro has always produced top-notch systems, and this is no exception. I also found that the sales department at SuperMicro was quite helpful, and willing to assist with configuration and purchase questions. Way to go, Supermicro!

SuperMicro TwinBlade Introduction

I'm going to start a series of posts about the SuperMicro TwinBlade system. There is enough material to cover that I'm going to break it into parts. These are very slick systems, and I'm quite excited to get started.

First, I'll cover the chassis. Second, the blades. Third, the software setup that I'm using. Fourth, the CMM. Stay tuned!