Cisco SG500-28 Swith

Wow.  Normally, I try to keep fairly positive when reviewing items, even if they have some down sides.  But this switch has me absolutely astounded with just how horrible it is.

Let's start out with the first thing that I noticed.  The documentation says that it will accept a DHCP lease, but if one is not available, it will fall back to a default IP.  In actuality, when the unit boots up, it tells the DHCP server that it wants to use a specific IP address which is NOT the one mentioned in the docs.  The DHCP server says "No", then offers a valid lease.  The switch never responds to the DHCP server, and does not respond on the documented IP address, the one it asked for, OR the one it was offered.  So much for the DHCP client.

Moving on, if you change the stacking mode on the switch, all other settings (including passwords) get reset to factory defaults.    That's pitiful.

Looking through the firmware release notes, horrible bugs have been fixed (a timeout in the web management could crash or reboot the switch!), but other bugs remain, with no workaround.  Browse the release notes, see for yourself.

I can't do five minutes of configuration on this switch without encountering yet another "What were they thinking?" moment.  Needless to say, more of these switches are NOT in my future.

SuperMicro TwinBlades, Software Setup

So, I've covered most of the hardware for the SuperBlades.  But what about a software configuration?  These machines are acting as front-end web servers, which pull code and data from a NAS. However, there's no reason to have enterprise disks in each of these blades just for the OS, so let's get some diskless booting happening!


The first TwinBlade got the same 40W CPUs as the others, but each got a pair of 600GB Velociraptors in RAID 1.  There we go, single disk failures are taken care of.  But what about an OS crash, a fat-fingered remote shutdown, or other issue?  Well, there is a nifty piece of software called DRBD, which essentially acts as RAID 1 across a network.  So, complete data sets exist on 4 disks, across two systems.  By tying in a couple of other packages, active or passive takeover can be achieved if one system fails.

NOTE:  These will NOT be exhaustive steps, just the important bits to get it all to work.
Take your primary and backup machines, and install DRBD on them.  Take your disk or partition, and set them up as a DRBD device.  If you're going to use DRBD, you need to understand the mechanics, so read their documentation for setup info.


Then, mount your DRBD device.  We'll use "/export" as a mount point for this example.

Now, first you need to make sure that each machine gets the right info, so let's do some DHCP work:

host blade-1 {
hardware ethernet 01:02:03:04:05:06;
fixed-address 10.11.12.13;
next-server 10.11.12.14;
if ((exists user-class) and (option user-class = "gPXE")) {
        option root-path "iscsi:10.0.1.28::::iqn.2012-04.com.foobar.iscsi1:blade-1";
        filename "";
  } else {
        filename "undionly.kpxe";
  }
}

This does a few things.  First, it gives the machine a fixed IP address.  It then tells it where to look for the next bit of the boot sequence, via the "next-server" statement.  Then, the if-else bracket tells it to grab the "undionly.kpxe" bootloader (courtesy of gPXE) if is acquiring a DHCP address for the first time.  But, once it has that, and is executing the gPXE environment and asks for the DHCP info again, it instead gives it the necessary info for gPXE to get the iSCSI disk going.  Crazy stuff.

So, then you need a TFTP server on 10.11,12.14, handing out the "undionly.kpxe" file - which can be compiled from the gPXE source.  Read-only, don't be a security fool.

Then, create a disk file on your iSCSI target:

dd if=/dev/zero of=./blade-1-disk.img bs=1 count=1 seek=17179869183

Why 17179869183?  Well, that's 16 gigs minus the one byte which we will write, so the file ends up at exactly 16 gigs.  Since we only wrote a single byte to the end of the file, not only is the creation fast, it also takes advantage of Linux's ability to use sparse files, which will save you some disk space - at least until you actually fill that up.

Now, let's set up IET to serve that out.  First, in your ietd.conf:

Target iqn.2012-04.com.foobar.iscsi1:blade-1
        Lun 0 Path=/export/blade-1-disk.img,Type=fileio,ScsiSN=Blade1
        InitialR2T no
        ImmediateData yes

Then, in your initiators.allow so that only the relevant machine sees the LUN:
iqn.2012-04.com.foobar.iscsi1:blade-1-disk.img 10.11.12.13

Now, you should be able to start up your CentOS installer, choose "advanced storage configuration", install to the iSCSI target, and rock and roll.

SuperMicro sbi-7226t-t2 TwinBlade

Now that we've discussed the chassis, let's talk about the blades themselves.  SuperMicro has blades optimized for storage, for GPU usage, or simply for traditional builds.  The TwinBlades are optimized for density, packing two entire dual-socket systems onto one board.  It's actually quite similar to their "1U Twin", but with both systems on a single PCB.
 

Each of these systems gets two LGA 1366 sockets, 8 DIMM sockets, 2 x 2.5" HDD hot-swap bays, and dual gigabit ethernet ports.  They also have the option of a 10GBE or Infiniband riser card.  They also have onboard IPMI, which coupled with the chassis's CMM, gives you a very powerful remote management capability.  Individual systems (systems, not blades) can be powered up, powered down, reset, have remote KVM, be monitored for temperature and electrical draw, and more.   CPUs may be used with a TDP up to 95W.

SuperMicro SBE-720E-R90 Blade Chassis

Blade systems are, by definition, built in, on, and around the idea of a chassis, and SuperMicro's offerings are no different. A blade chassis consists of much more than just some sheet metal to hold parts in place. They are complex, compact, wonderful pieces of hardware!

Starting out, SuperMicro has a few different blade chassis available. If you want to use their TwinBlades, you need to use their 720D or 720E chassis. The "E" model can handle larger power supplies, and has extra expansion slots for Infiniband switches.

Each of the chassis can handle up to ten blades, including their TwinBlades. TwinBlades have two seperate, dual-socket systems on the same blade - so, in essence, you can pack up to 20 dual-socket servers in 7U!

Density, however, is only one of the benefits. Another is the chassis management, or CMM module. The first function of the CMM is to provide local and remote KVM, with virtual media redirection, allowing you to perform cold-boot configuration and installation from remote locations. Secondly, it performs power monitoring and management on the units, allowing you to power blades up and down as needed. It also provides IPMI on a chassis level. This is a tremendously cost-effective solution, as 3rd-party remote KVM and power management systems can be VERY expensive. There is, however, one catch: While SuperMicro's marketing says that a CMM unit is included, mine showed up without one, and I was told that I have to buy one separately. That seems a bit suspicious, but it is the first time I've had ANY negative experience with SuperMicro - and I've spent a lot of money with them. They've pulled through on things like finding parts that were long out of production and obsolete years before I called, so I'll let this one thing slide.


Dense I/O is another benefit. The chassis have slots for various I/O modules, and SuperMicro has modules and switches for gigabit ethernet, 10 gigabit ethernet, and 10/40gb Infiniband. As mentioned earlier, slots for Infiniband switches appear on the 720E model, but not the 720D.  Be sure to check their networking matrix to find the correct networking options for your chassis and blades.

The last benefit that I will cover in this post is the power. This chassis uses HIGHLY efficient power supplies (94%+!). While that may not seem like a big deal, consider that compared to lesser power supplies, that alone can shave 5%-15% off of your electrical budget, which also shaves 5%-15% off of your cooling budget. For disaster contingency, that also translates to longer UPS runtime, as well as lower fuel usage and longer runtime for your generator. Additionally, the benefits of power supply consolidation kick in, and all in all, this is a very electrically-efficient system. That may not seem flashy, but anyone who has worked in high-density installations will appreciate the energy savings.

Now, some notes on power: 40 sockets, 160 DIMMS, and 40 hard drives still take lots of juice. Plan on two 220V circuits, 120V need not apply.  While more power-hungry CPUs (and GPUs!) can suck down more power, by using 40W TDP L5630 CPUs and booting diskless from a SAN, my systems pull just a tiny bit over 100W each, which includes the power used for the CMM, switches, fans, etc..

If you are not filling all of the slots with blades, dummy blades must be used to manage airflow. I was originally told that the chassis did not include any dummy blades, but when it showed up... it was, indeed, populated with just enough dummy blades to fill the slots that I wouldn't yet fill. SuperMicro tells me that these chassis are individually configured for each order, so that would explain the attention to detail. SuperMicro has always produced top-notch systems, and this is no exception. I also found that the sales department at SuperMicro was quite helpful, and willing to assist with configuration and purchase questions. Way to go, Supermicro!

SuperMicro TwinBlade Introduction

I'm going to start a series of posts about the SuperMicro TwinBlade system. There is enough material to cover that I'm going to break it into parts. These are very slick systems, and I'm quite excited to get started.

First, I'll cover the chassis. Second, the blades. Third, the software setup that I'm using. Fourth, the CMM. Stay tuned!

Shuttle SH67H3

I've built a number of systems out of these platforms recently, and I have to say... I love them. I really love them.

Most small systems like this suffer from being cramped and having thermal issues. The SH67H3 solves both of these in a very intelligent way, one that is long overdue: Instead of releasing all of the CPU heat inside of the chassis, it is pulled via heatpipes to a heatsink at the BACK of the chassis, and the heat is immediately spit out the back.

That means that the inside stays much cooler than if it were using a traditional heatsink setup. It also means that you don't have the large CPU heatsink right in the middle of the chassis, getting in the way. I've had no problem with 95W Sandy Bridge CPUs on these, they stay cool and quiet.

The unit also comes with room for 2 3.5" hard drives, and a 5.25" optical drive. Pulling the drive cage out is easy, and because the heatsink is not in the middle of the chassis, accessing the drive connectors is a snap as well.

It also comes with a PCI-e x16 slot in case you want discrete video, and the slot is spaced to accommodate dual-slot video cards. Very nice!

The power supply is a 300W 80+ unit. For a single hard drive and CPU with onboard graphics, that's probably plenty - but Shuttle also sells their PC63J power supply, with a 500W rating. If you are using a high-draw video card, that would be a good route to take.

Assembling one of these is surprisingly easy and fast. If you're in the market for a small machine like this, take a look at the SH67H3.

Lacie MosKeyTo Flash Drive Review

Lacie's MosKeyTo flash drives can be summed up in one word: Miniscule. These things are tiny. What you see here, which looks almost like a bare USB connector, is the [i]entire drive[/i].

Flash drives have always been handy, but carrying one hasn't. These are so tiny that I simply clipped mine to my keyring, and haven't noticed it since. It's so tiny that unless I need it... I usually forget that I even have it. When people ask about it, it usually takes a few minutes before they believe that such a small device could be a flash drive... let alone a 16GB flash drive.

Speeds are quite reasonable, in fact, faster than I expected based on the size and cost. If there is a downside to the MosKeyTo, it would be that it is SO small that on a tight USB slot, sometimes it takes a good grip to be able to pull it out when you're finished.

Now, the downside... while the lanyard is strong, carrying it on my keychain did eventually cause the cover to come loose and get lost. And, after a few months, being exposed to the static of getting carried in a pocket killed it. I've had the same thing happen with most drives that I carried for months without a cover, so that isn't necessarily a problem with the drive itself, just an inherent risk of such a small package.