Installing and configuring Graphite with Collectd on FreeBSD

Using Graphite for a smidgen blog like mine is definitely an overkill and Munin fits much better but I wasn’t able to confront my curiosity. So that’s why this post.
Before I start, I’d like to mention that even if I’ll be speaking about my endeavor with Graphite on a system running FreeBSD 10.0-RC5, the same configuration would certainly work on Linux. But since different distros install Graphite’s components into different directories (certainly different from what FreeBSD does) you should update your configuration files smartly.

  • First things first… Install the packages:
  • # pkg install py27-graphite-web
    Updating repository catalogue
    The following 13 packages will be installed:
    
    	Installing sqlite3: 3.8.2
    	Installing py27-cairo: 1.10.0_1
    	Installing py27-zope.interface: 3.8.0_1
    	Installing py27-thrift: 0.9.1,1
    	Installing py27-setuptools: 2.0.1
    	Installing py27-whisper: 0.9.12
    	Installing py27-sqlite3: 2.7.6_3
    	Installing py27-twistedCore: 13.2.0
    	Installing py27-django: 1.6.1
    	Installing py27-txamqp: 0.3_2
    	Installing py27-django-tagging: 0.3.1
    	Installing py27-carbon: 0.9.12
    	Installing py27-graphite-web: 0.9.12
    
    The installation will require 62 MB more space
    
    8 MB to be downloaded
    
    Proceed with installing packages [y/N]: y
    
  • That was easy. Out next step is to setup Carbon. You could read more about the Carbon daemons here:
  • # cd /usr/local/etc/carbon
    # cp storage-schemas.conf.example storage-schemas.conf
    # cp carbon.conf.example carbon.conf
    
  • Open your favorite editor and set directory paths:
    GRAPHITE_ROOT        = /usr/local/graphite
    GRAPHITE_CONF_DIR    = /usr/local/etc/carbon
    GRAPHITE_STORAGE_DIR = /usr/local/graphite/storage/
    STORAGE_DIR          = /usr/local/graphite/storage/
    LOCAL_DATA_DIR       = /usr/local/graphite/storage/whisper/
    CONF_DIR             = /usr/local/etc/carbon
    LOG_DIR              = /usr/local/graphite/storage/log/
    PID_DIR              = /var/run
    
  • That’s basically it. I had to change some other parameters in the file to make the carbon process listen on the localhost interface only since by default it listens on 0.0.0.0:
  • LINE_RECEIVER_INTERFACE = 127.0.0.1
    PICKLE_RECEIVER_INTERFACE = 127.0.0.1
    CACHE_QUERY_INTERFACE = 127.0.0.1
    
    [relay]
    LINE_RECEIVER_INTERFACE = 127.0.0.1
    LINE_RECEIVER_PORT = 2013
    PICKLE_RECEIVER_INTERFACE = 127.0.0.1
    PICKLE_RECEIVER_PORT = 2014
    
    [aggregator]
    LINE_RECEIVER_INTERFACE = 127.0.0.1
    PICKLE_RECEIVER_PORT = 2024
    
  • Everything is ready and carbon could be started.
  • # /usr/local/etc/rc.d/carbon start
    Starting carbon.
    Traceback (most recent call last):
      File "/usr/local/bin/carbon-cache.py", line 28, in <module>
        from carbon.util import run_twistd_plugin
      File "/usr/local/lib/python2.7/site-packages/carbon/util.py", line 21, in <module>
        from twisted.scripts._twistd_unix import daemonize
    ImportError: cannot import name daemonize
    </pre>
    <li>What's that?!. Thankfully, this could be fix in a second by installing daemonize and editing util.py</li>
    <pre>
    # easy_install pip
    # pip install daemonize
    # vi /usr/local/lib/python2.7/site-packages/carbon/util.py
    

    Original version:

    from twisted.scripts._twistd_unix import daemonize

    After editing:

    #from twisted.scripts._twistd_unix import daemonize
    import daemonize

  • Starting carbon once again. And don’t forget to add carbon_enable=”YES” into your rc.conf file.
  • # /usr/local/etc/rc.d/carbon start
    Starting carbon.
    Starting carbon-cache (instance a)
    

    Continue reading

Upgrading FreeBSD 9.2 to 10.0-RC4 on Rackpsace’s instance

In reality it turned out to be a fairly straightforward procedure:

  • This is crucial. You must save XENHVM config before the upgrade otherwise you won’t be able to re-build the kernel which is required to bring the network up.
  • # cp /usr/src/sys/amd64/conf/XENHVM /root/amd64_XENHVM
    # cp /usr/src/sys/i386/conf/XENHVM /root/i386_XENHVM
    
  • Perform the upgrade as usual and just follow the instructions, i.e. reboot when you’re asked to:
  • # freebsd-update -r 10.0-RC4 upgrade
    Looking up update.FreeBSD.org mirrors... 5 mirrors found.
    Fetching metadata signature for 9.2-RELEASE from update5.freebsd.org... done.
    Fetching metadata index... done.
    Fetching 1 metadata patches. done.
    Applying metadata patches... done.
    Fetching 1 metadata files... done.
    Inspecting system... done.
    
    WARNING: This system is running a "xenhvm" kernel, which is not a
    kernel configuration distributed as part of FreeBSD 9.2-RELEASE.
    This kernel will not be updated: you MUST update the kernel manually
    before running "/usr/sbin/freebsd-update install".
    
    ... Skipped for brevity ... 
    
    To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
    
  • Run “freebsd-update install”. When it’s done you’ll be asked to reboot the server. Do not think twice and reboot safely here.
  • When the server comes up run “freebsd-update install” for the second time.
  • In the end you’ll be asked to repeat “freebsd-update install” for the third time. Run it lightheartedly but do not reboot! Instead, you should rebuild the kernel here:
  • # cd /usr/src
    # cp /root/amd64_XENHVM /usr/src/sys/amd64/conf/XENHVM
    # cp /root/i386_XENHVM /usr/src/sys/i386/conf/XENHVM
    # make buildkernel KERNCONF=XENHVM
    # make installkernel KERNCONF=XENHVM
    # shutdown -r now
    
  • After reboot comes a moment of the glory:
  • # freebsd-version 
    10.0-RC4
    

Grow root ZFS pool online under FreeBSD 9.2 using Rackspace’s instance

If you opted for a FreeBSD instance on Rackspace and chose standard configuration then you’ll end up with 512MB of RAM and 20GB of space. That’s ok for the starter but eventually you’d need more one day and will upgrade RAM to 1GB for example. With this upgrade your disk will grow upto 40GB but OS would not notice that (Rackspace doesn’t support automatic disk partitioning on FreeBSD). But you could fix that easily by yourself with a few simple CLI commands.
So lets plunge into the command line and see what we have and what we could do about that:

  • Before the upgrade disk’s layout looks similarly to what is shown below:
  •  
    # gpart show -p
    =>      1  1048575    ada2  MBR  (512M)
            1  1048575  ada2s1  linux-data  (512M)
    
    =>      34  41942973    ada0  GPT  (20G)
            34        94  ada0p1  freebsd-boot  (47k)
           128  41942879  ada0p2  freebsd-zfs  (20G)
    
    # zpool list
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  19.9G  3.67G  16.2G    18%  1.00x  ONLINE  -
    
  • But after, things somewhat have changed:
  • # gpart show
    =>      1  2097151  ada2  MBR  (1.0G)
            1  2097151     1  linux-data  (1G)
    
    =>      34  41942973  ada0  GPT  (40G) [CORRUPT]
            34        94     1  freebsd-boot  (47k)
           128  41942879     2  freebsd-zfs  (20G)
    
    # zpool list
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  19.9G  3.67G  16.2G    18%  1.00x  ONLINE  -
    
  • Notice that gpart reports ada0 as corrupted and zpool still can’t see extra 20GB. But don’t panic. In our case this [CORRUPT] warning just identifies that the disk’s size has changed. Moreover we could confirm that by checking kernel’s messages using dmesg command:
  • GEOM: ada0: the secondary GPT header is not in the last LBA.
    
  • Let me partly quote gpart’s man page so the above message becomes more clear:
  • …The GPT primary metadata is stored at the beginning of the device. For
    redundancy, a secondary (backup) copy of the metadata is stored at the
    end of the device…

    If the size of the device has changed (e.g., volume expansion) the sec-
    ondary GPT header will no longer be located in the last sector. This is
    not a metadata corruption, but it is dangerous because any corruption of
    the primary GPT will lead to loss of the partition table. This problem
    is reported by the kernel with the message:

    GEOM: provider: the secondary GPT header is not in the last LBA.

  • Bingo. Thankfully, this is not critical and something that could be fixed in a second:
  • # gpart recover ada0
    
    # gpart show
    =>      1  2097151  ada2  MBR  (1.0G)
            1  2097151     1  linux-data  (1G)
    
    =>      34  83886013  ada0  GPT  (40G)
            34        94     1  freebsd-boot  (47k)
           128  41942879     2  freebsd-zfs  (20G)
      41943007  41943040        - free -  (20G)
    
  • Excellent. No more [CORRUPT] messages and as a bonus we could see an extra free space. So let’s grow our partition:
  • # gpart resize -i 2 ada0
    gpart: Device busy
    
  • Hm, device busy?! Well, there is a workaround for this one too:
  • # sysctl kern.geom.debugflags=16
    kern.geom.debugflags: 0 -> 16
    # gpart resize -i 2 ada0
    
    # gpart show
    =>      1  2097151  ada2  MBR  (1.0G)
            1  2097151     1  linux-data  (1G)
    
    =>      34  83886013  ada0  GPT  (40G)
            34        94     1  freebsd-boot  (47k)
           128  83885919     2  freebsd-zfs  (40G)
    
    # zpool list
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  19.9G  3.67G  16.2G    18%  1.00x  ONLINE  -
    
  • Finally, let’s deal with our zpool and help it to see the full capacity of our disk:
  • # zpool set autoexpand=on zroot
    # zpool online -e zroot ada0p2 ada0p2
    
    # zpool list
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  39.9G  3.67G  36.2G     9%  1.00x  ONLINE  -
    
  • That’s it. And thanks for reading.

Bye!

Never make plans but I’ll try

From New Year’s Resolutions for SysAdmins presented by USENIX I have picked just three for myself:

  • “Gather more metrics and become a data driven IT team.”
  • “I will not ever again work for a cheapskate business owner who cannot, or will not, pay me a fair rate.”
  • “Learn to be a better coder.”

The latter one one is the most important one.