• Infrastructure monitoring and the journey to the cloud

    At work we are in the process of moving from a datacenter centric infrastructure to AWS. This is a great journey with many interesting challenges and today we will discuss one of them, the monitoring of our infrastructure.

  • DNS over TLS forwarding with Unbound and Quad9

    In my previous post I explained how to build a DNS sinkhole with Unbound by downloading block lists from different sources. I also tried to use dnscrypt in the setup, but I had to disable it because the service provided was unreliable.

    Yesterday Cloudflare announced that they were providing a “privacy-first consumer DNS service”, whatever it means.

    Since it’s Easter and I have more free time than usual, I thought it would be cool to have a look and update my DNS sinkhole at home.

    While I was searching for information related to DNS over TLS, that is one of the main features provided by Cloudflare, I came across Quad9, that it’s offering the same service. They have been in the news a lot but I didn’t play attention because the media outlets only reported it as an alternative to Google DNS and back then I was too busy.

    In a nutshell, Quad9 is a sinkhole that blocks DNS requests to malicious domains, that is pretty much the same I am doing at home with Unbound and a shell script, but with more resources. My blackhole hast more than 30K domains blacklisted, that is not bad at all :)

    At the end, I decided to use the DNS over TLS resolvers from Quad9, but you can find the resolvers from Cloudflare commented out in the configuration file. I will keep my own list of blocked domains for the time being, but I may kill it in the future because my configuration fails every now and then when the domain names have non-acii characters.

    The minimum configuration options are:

    • ssl-upstream tells Unbound to use TLS to communicate with the upstream server.
    • ip_add@port to define the upstream server.

    Additionally I am using configuration parameters that come in handy:

    • minimal-responses: yes

      Reduces the size of the response when possible to improve the performance a bit.

    • prefetch: yes

      Fetch the about to expire cache elements.

    • qname-minimisation: yes

      Best effort to send minimum amount of info to the upstream servers but not super helpful.

    Notice that Unbound is not running daemonized because it’s being monitored by the Daemontools supervisor. That is also why the configuration and control files are not placed in the usual locations.

            access-control: allow
            access-control: allow
            do-daemonize: no
            logfile: ""
            username: unbound
            directory: /usr/local/var/service/unbound
            chroot: /usr/local/var/service/unbound
            pidfile: /usr/local/var/service/unbound/unbound.pid
            verbosity: 1
            minimal-responses: yes
            prefetch: yes
            qname-minimisation: yes
            # we are doing DNS over TLS
            ssl-upstream: yes
            root-hints: /usr/local/var/service/unbound/config/root.hints
            # my DNS zone at home
            include:    /usr/local/var/service/unbound/config/local.zone
            # autogenerated every night to block malicious domains
            include:    /usr/local/var/service/unbound/config/blackhole.zone
       name: "."
       forward-addr:         # quad9.net primary
       forward-addr: # quad9.net secondary
       #forward-addr:        # cloudflare primary
       #forward-addr:        # cloudflare secondary
           control-enable: yes
           control-interface: /usr/local/var/service/unbound/control.clt
           control-use-cert: no
  • Fixing OpenSC after updating to MacOS Sierra

    Sierra introduced restrictions to the ssh-agent (new version of OpenSSH) by limiting the PKCS#11 libraries that can be loaded to a list of whitelisted directories. As of now, this is public domain because I must be one of the last persons updating from El Capitan to Sierra. Yes, I am not an early adopter!

    So, until now I had an alias in my .bashrc that was loading my SSH key in the Yubikey to the ssh agent. The alias was just fine but the library is now outside the trusted path, that is “/usr/lib:/usr/local/lib”.

    alias load_key="ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so"
    alias unload_key="ssh-add -e /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so"

    As always, the error message is everything but useful:

    $ ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so
    Enter passphrase for PKCS#11:
    Could not add card "/Library/OpenSC/lib/pkcs11/opensc-pkcs11.so": agent refused operation

    I had to run the ssh-agent in debug mode to understand what was happening (Google is your friend) and the output said: provider not whitelisted.

    $ ssh-agent -d -a /tmp/agent.socket
    SSH_AUTH_SOCK=/tmp/agent.socket; export SSH_AUTH_SOCK;
    echo Agent pid 2918;
    debug2: fd 3 setting O_NONBLOCK
    debug3: fd 4 is O_NONBLOCK
    debug1: type 20
    refusing PKCS#11 add of "/Library/OpenSC/lib/opensc-pkcs11.so": provider not whitelisted
    debug1: XXX shrink: 3 < 4
    $ SSH_AUTH_SOCK="/tmp/agent.socket" ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so
    Enter passphrase for PKCS#11:
    Could not add card "/Library/OpenSC/lib/pkcs11/opensc-pkcs11.so": agent refused operation

    Goggling again, the first hit is the bug report that was opened last year.

    MacOS: cannot use /usr/local/lib/opensc-pkcs11.so (provider not whitelisted)

    At the end, I had to modify my aliases to the library located in the trusted path.

    alias load_key="ssh-add -s /usr/local/lib/opensc-pkcs11.so"
    alias unload_key="ssh-add -e /usr/local/lib/opensc-pkcs11.so"

    The original post where I setup SSH public key authentication with security tokens

  • Backing up my Githup repos

    I am using runwhen together with daemontools to launch and monitor the backup. The run script used by the svc service executes runwhen commands to sleep until the next run (every hour) and then launch the backup script. The service is running in a dedicated jail.

    The run script listed below uses some runwhen commands (rw-add,rw-matchand rw-sleep) to wake-up every hour and setuidgid to run the service with an unprivileged user.

    exec 2>&1
    exec setuidgid gitbackup \
    rw-add   n d1S             now1s        \
    rw-match \$now1s ,M=00     wake         \
    rw-sleep \$wake                         \

    The actual backup script that iterates over all the git repos and fetches the changes.

    exec 2>&1
    cd /usr/home/gitbackup/backup
    echo "===="
    echo "===="
    for repo in `ls -d1 *.git`; do
            cd $repo && /usr/local/bin/git fetch --all
            cd -
    echo "===="

    checking the output log

    $ cat /var/service/backups/log/main/current | tai64nlocal
    2018-02-05 18:00:00.098641500 ====
    2018-02-05 18:00:00.150083500 Mon Feb  5 18:00:00 CET 2018
    2018-02-05 18:00:00.180056500 ====
    2018-02-05 18:00:00.211689500 Fetching origin
    2018-02-05 18:00:01.073738500 From https://github.com/xgarcias/ansible-cmdb-freebsd-template
    2018-02-05 18:00:01.073743500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:01.091577500 Fetching origin
    2018-02-05 18:00:02.185366500 From https://github.com/xgarcias/ansible-daemontools
    2018-02-05 18:00:02.185371500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:02.203049500 Fetching origin
    2018-02-05 18:00:04.180310500 From https://github.com/xgarcias/ansible-macbook
    2018-02-05 18:00:04.180315500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:04.198104500 Fetching origin
    2018-02-05 18:00:06.448429500 From https://github.com/xgarcias/daemontools-dyndns
    2018-02-05 18:00:06.448434500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:06.466266500 Fetching origin
    2018-02-05 18:00:08.299785500 From https://github.com/xgarcias/daemontools-poudriere
    2018-02-05 18:00:08.299790500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:08.321755500 Fetching origin
    2018-02-05 18:00:09.749956500 From https://github.com/xgarcias/daemontools-unbound-sinkhole
    2018-02-05 18:00:09.749961500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:09.771744500 Fetching origin
    2018-02-05 18:00:11.113934500 From https://github.com/xgarcias/elasticsearch-plugin-readonlyrest
    2018-02-05 18:00:11.113939500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:11.135774500 Fetching origin
    2018-02-05 18:00:12.703191500 From https://github.com/xgarcias/freebsd_local_ports
    2018-02-05 18:00:12.703197500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:12.724967500 Fetching origin
    2018-02-05 18:00:13.583204500 From https://github.com/xgarcias/xgarcias.github.io
    2018-02-05 18:00:13.583209500  * branch            HEAD       -> FETCH_HEAD
    2018-02-05 18:00:13.601461500 ====
  • Querying ANS/IP records via non rate-limited REST API

    Querying ANS/IP records via non rate-limited unauthenticated REST API.

    More Info

    Also, you can use @DuckDuckGo to get the same results with the !Arin and !Ripe bang searches.