AWS FreeBSD instance won’t come up. Screams for manual fsck.

This is a short write up if after a reset or a reboot your FreeBSD (or Linux) instance doesn’t come online, stalls and
“aws ec2 get-console-output” returns something like that among its lines:

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

  • Just power-off the faulty instance either from Web interface or using cli:
    aws ec2 stop-instances --instance your_instance_id
    
  • Again, using Web interface or cli (if you know its id) detach volume the root filesystem lives on.
    aws ec2 detach-volume --volume-id faulty_volume_id --instance your_instance_id
    
  • Create a new minimal instance and attach the volume that was detached during the previous step.
  • Boot it up and simply run fsck manually as advised.
    For FreeBSD you will have to add an entry into /etc/fstab otherwise fsck would complain:

    # fsck -y /dev/xbd5a 
    fsck: Could not determine filesystem type
    

    In my case I just add a single line:

    echo "/dev/xbd5a /mnt ufs rw 1 1" >> /etc/fstab
    
  • After that, just do the reverse: detach the volume and connect it back to your main instance and power it up.

Hope everything is golden at this point.

How to power off USP-V when it doesn’t want to

In the era of VSP G1000 this post may sound dated but I still hope that it would help some poor sole in the same situation I was sometime ago. The task was laughably simple: power off USP-V, power it up just to make sure it still could boot up flawlessly and shut it down again. All went nice and dandy till the final step – the ultimate power off. So I opened DKC panel and switched simultaneously two switches: PS-ENABLE and PS-OFF (all in accordance with the maintenance manual).

IMG_1927.JPG

So far so good. During the first power off iteration it took the array (DKC and one DKU) roughly 15-20 minutes to cut off the power from its components so switches on AC boxes could be turned into OFF position. But not this time… After waiting for one hour and a half the system was still up. However, and thankfully the disks had been spun down successfully. That opened a door for a forceable shut down procedure. Which is simple and straightforward as a samurai’s sword.

  • Open DKC panel and unscrew it as shown on the picture below.
  • IMG_1928.JPG

  • Pay attention to the jumpers. We will be using JP3 which is right above JP2.
  • IMG_1929.JPG

  • If you don’t have a jumper (I didn’t have one) there is also a workaround. Go to the back of a DKU, open its door and at the bottom there is a recess with another set of jumpers.
  • IMG_1935-0.JPG

  • It is save to pick and pull out any of these jumpers (remember that the disks had to be powered off before that).
    I’ve been told that these jumpers define the physical position of a DKU rack relatively to a DKC (is it on the left or right and how far DKU-R1, DKU-L1, etc.)
  • Once you have a jumper put it into JP3 in the DKC panel and turn CHK RST switch on but pressing on its upper half.
  • A moment after that the array would be shut down.