AWS FreeBSD instance won’t come up. Screams for manual fsck.

This is a short write up if after a reset or a reboot your FreeBSD (or Linux) instance doesn’t come online, stalls and
“aws ec2 get-console-output” returns something like that among its lines:

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

  • Just power-off the faulty instance either from Web interface or using cli:
    aws ec2 stop-instances --instance your_instance_id
    
  • Again, using Web interface or cli (if you know its id) detach volume the root filesystem lives on.
    aws ec2 detach-volume --volume-id faulty_volume_id --instance your_instance_id
    
  • Create a new minimal instance and attach the volume that was detached during the previous step.
  • Boot it up and simply run fsck manually as advised.
    For FreeBSD you will have to add an entry into /etc/fstab otherwise fsck would complain:

    # fsck -y /dev/xbd5a 
    fsck: Could not determine filesystem type
    

    In my case I just add a single line:

    echo "/dev/xbd5a /mnt ufs rw 1 1" >> /etc/fstab
    
  • After that, just do the reverse: detach the volume and connect it back to your main instance and power it up.

Hope everything is golden at this point.

7 comments on “AWS FreeBSD instance won’t come up. Screams for manual fsck.

  • Hi Philip,
    No worries.
    Thanks, for letting me know it worked for you.

    Cheers.
    BR,
    Sergey

  • Awesome tip, thanks for the help on this and saving me from lots of panic or rebuilding from backups!!

  • Heya, Justin.
    Thanks for stopping by and leaving a comment.
    I’m excited to hear this smallish tip worked for you and saved you from restoring.

    Cheers.
    Sergey

  • I find it odd/annoying that after all these years, AWS still doesn’t provide a simple interactive “serial” console (even if limited to TERM=dumb). Not that a basic in webpage vt100 emulator is beyond Amazon’s abilities.

  • Can’t agree more. The lack of any console support and a blank response from “aws ec2 get-console-output” another day after OS upgrade triggered that “enough is enough” moment and I eventually kissed AWS goodbye.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>