Debugging the Backblaze Red Pods(1/11/09) by the Backblaze Server Team |
Explanation: Backblaze has the concept of a "storage pod", which is defined as a "good way" of organizing one CPU, one gigabit network card, and some number of hard drives to store customer data on. Click here for a full description of Backblaze 45 terabyte Red Storage Pods. On this page are a few pictures of how we debugged the problems with those pods.
Below we decided to drill holes in one of the storage pod cases to expose the power connectors of the Sata Port Multiplier boards. This allows quick rewiring using different power supply configurations to rule out power issues. Below Brian (in the gloves) drills some holes while Casey steadies the pod case.
After drilling the holes, we used a stone grinder bit on a drill to polish the holes so they wouldn't cut cables (or fingers).
Experiment #1: use higher quality molex connectors. Our theory was the non-standard (and cheap) 90 degree molex power connectors were causing issues. So the first experiment used the same standard power supply but wired up using standard, nice, expensive, molex power connectors now possible with the extra space below the holes. Below is the finished experiment (which had no improvement) in the datacenter rack reserved for this experiment.
A picture of the top of an empty pod. You can see the power wiring is normally completely hidden under the circuit boards so very hard to get to. With our modified "swiss cheese pod" we can wire it incredibly fast and in different ways.
While bringing the experimental pod online, here is a "Crash Cart" hooked up to our cabinet, with Tim's Macbook also logged in watching monitoring screens.
Experiment #2: use 4 separate power supplies to power one red pod to rule out any power issues. Below Tim prepares the 3 extra power supplies.
Here you can see "experiment #2" deployed. The original power supply is still powering the motherboard, but here we also deploy 3 more power supplies (for a total of 4 power supplies) to power all the Port Multipliers. This experiment did not yield any improvement.
Same picture as above, just the opposite side of the datacenter cabinet so you can see how it's all wired up.
All done! Please email me with any comments.