Another FreeBSD drop into the Digital Ocean

After several happy years with FreeBSD running in AWS I finally have switched to Digital Ocean. That happened a few days ago and was driven mainly by the lack of the console support which “aws ec2 get-console-output”, in my opinion, is certainly not.
After the upgrade to FreeBSD 11 I found my instance unreachable and had absolutely no clue what was wrong with it. In that situation “aws ec2 get-console-output” was totally useless with its succinct single-worded output – “Output”. Last time when I had a similar issue after another upgrade I at least was able to glean some helpful information with get-console-output to fix the problem. Not this time though.
So without further ado and armed with Tarsnap backups, I jumped on to DO’s bandwagon with ZFS and HTML5 console which, I hope, would be able to save me should I hit the same boot problem again. As an extra bonus, DO instance is a bit cheaper and beefier than the one I had in AWS. But as always… horses for courses.

Pair “Listen queue overflow” FreeBSD errors with pcb

Just yesterday, after an upgrade to MySQL 5.7.12, saw plenty of errors were being logged in the system:

sonewconn: pcb 0xfffff8006311c870: Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff8006311c870: Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff8006311c870: Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff8006311c870: Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences)

There is a great post that explains how to find the culprit. In a nutshell, there are two quick options:

  1. Use “lsof -itcp -stcp:listen -P” and grep for pcb.
  2. Or since “the overflow happens when the queue is at about 150% capacity” (as mentioned in the original post), it’s possible to match the number from the error (151 in my case) with an output from “netstat -an -p tcp -L”.

In my case that was trivial as both Postfix and Dovecot complained about missing libmysqlclient.so.18 shared library which was replaced with libmysqlclient.so.20 after the upgrade. Rebuilding from ports and restarting both of them fixed the issue and no hassling with kern.ipc.somaxconn was needed.

Bryan Cantrill in BSD Now podcast

The lastest episode of BSD Now (103) podcast brought in a fantastic and hilarious interview with Bryan Cantrill who is well known for his wit and right on the bullseye rants. It’s been awhile since I cried laughing so unquestionably this video is a highly recommended. Not to mention that his talk was very educational both from the technical (epoll, kqueue) and historical point of views. Bookmarked and added to the favorites.

AWS FreeBSD instance won’t come up. Screams for manual fsck.

This is a short write up if after a reset or a reboot your FreeBSD (or Linux) instance doesn’t come online, stalls and
“aws ec2 get-console-output” returns something like that among its lines:

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

  • Just power-off the faulty instance either from Web interface or using cli:
    aws ec2 stop-instances --instance your_instance_id
    
  • Again, using Web interface or cli (if you know its id) detach volume the root filesystem lives on.
    aws ec2 detach-volume --volume-id faulty_volume_id --instance your_instance_id
    
  • Create a new minimal instance and attach the volume that was detached during the previous step.
  • Boot it up and simply run fsck manually as advised.
    For FreeBSD you will have to add an entry into /etc/fstab otherwise fsck would complain:

    # fsck -y /dev/xbd5a 
    fsck: Could not determine filesystem type
    

    In my case I just add a single line:

    echo "/dev/xbd5a /mnt ufs rw 1 1" >> /etc/fstab
    
  • After that, just do the reverse: detach the volume and connect it back to your main instance and power it up.

Hope everything is golden at this point.

How Cisco Ironport picks an outgoing IP address if there are many on the same subnet

If your IronPort is configured with multiple IP addresses on the same subnet and you wonder which one is used then Cisco has an answer for you:

Q: Which is the default used IP address (AUTO) if there are multiple IP addresses on the same subnet?
A: If there are multiple IP addresses configured within the same subnet as the default gateway, the IP address with the lowest number based on a c-string search will be used.

More details with the example are available at Cisco Email Security Appliance section of www.cisco.com web site.

So before adding a new one make sure that the default interface has been defined either by a content/message filter, deliveryconfig or alt-src host action. Assume that altsrchost CLI command should be good too.

P.S. Tagged this post as FreeBSD only because Cisco IronPort is based on FreeBSD

ruBSD 2014 is coming

Just like in year 2013 Yandex will be hosting ruBSD 2014 (content is in Russian) event on the 13th of December. It’s funny that I learnt about it from BSD Now podcast which, btw, I highly recommend. In the last episode, apart from the already mentioned ruBSD 2014 conference, Allan Jude and Kris Moore also mentioned that videos from recent MeetBSD California 2014 have been recently published as well as from OpenZFS Developer Summit 2014.
But I digressed. Returning to ruBSD, the agenda looks very promising:

  • VM improvements coming soon to FreeBSD. Scott Long (Netflix).
  • Four years of pkg. Baptiste Daroussin (FreeBSD).
  • Functional and high-performance SCSI target based on CTL and ZFS. Alexander Motin (iXsystems).
  • New sendfile(2). Gleb Smirnov (Nginx, Inc.).
  • Practical use of IPv6. Alexander Chernigov (Yandex).
  • Building packages though emulation. Sean Bruno (Limelight Networks).

Registration is free but the number of seats is limited.

Jumping into another cloud

After almost 4 years of being a Rackspace user I’ve moved to a new home – Amazon AWS. The main reason that gave me a nudge was the issue I was hit by after upgrading to 10.1-Release. Temporary solution did work but it was too costly to consider it as permanent so I started to look around. The decision was quick as I knew where to look at – thanks to FreeBSD Journal Issue July/Aug 2014 in which Colin Percival, a well-known FreeBSD committer, describes how to provision FreeBSD on EC2. Besides, he also publishes FreeBSD AMIs which gives hope that we won’t be left in the dark.
Can’t tell anything specific about AWS, it definitely fits my unassuming needs, but what I really like is their aws cli tool and of course waiting for my very first billing invoice.

Cheers.