Infrastructure monitoring and the journey to the cloud•
At work we are in the process of moving from a datacenter centric infrastructure to AWS. This is a great journey with many interesting challenges and today we will discuss one of them, the monitoring of our infrastructure.
DNS over TLS forwarding with Unbound and Quad9•
In my previous post I explained how to build a DNS sinkhole with Unbound by downloading block lists from different sources. I also tried to use dnscrypt in the setup, but I had to disable it because the service provided was unreliable.
Yesterday Cloudflare announced that they were providing a “privacy-first consumer DNS service”, whatever it means.
Announcing 184.108.40.206: the fastest, privacy-first consumer DNS service - https://t.co/xiM3yllWHj pic.twitter.com/5keff8uuD2— Cloudflare (@Cloudflare) April 1, 2018
Since it’s Easter and I have more free time than usual, I thought it would be cool to have a look and update my DNS sinkhole at home.
While I was searching for information related to DNS over TLS, that is one of the main features provided by Cloudflare, I came across Quad9, that it’s offering the same service. They have been in the news a lot but I didn’t play attention because the media outlets only reported it as an alternative to Google DNS and back then I was too busy.
In a nutshell, Quad9 is a sinkhole that blocks DNS requests to malicious domains, that is pretty much the same I am doing at home with Unbound and a shell script, but with more resources. My blackhole hast more than 30K domains blacklisted, that is not bad at all :)
At the end, I decided to use the DNS over TLS resolvers from Quad9, but you can find the resolvers from Cloudflare commented out in the configuration file. I will keep my own list of blocked domains for the time being, but I may kill it in the future because my configuration fails every now and then when the domain names have non-acii characters.
The minimum configuration options are:
- ssl-upstream tells Unbound to use TLS to communicate with the upstream server.
- ip_add@port to define the upstream server.
Additionally I am using configuration parameters that come in handy:
Reduces the size of the response when possible to improve the performance a bit.
Fetch the about to expire cache elements.
Best effort to send minimum amount of info to the upstream servers but not super helpful.
Notice that Unbound is not running daemonized because it’s being monitored by the Daemontools supervisor. That is also why the configuration and control files are not placed in the usual locations.
server: interface: 10.10.10.10 access-control: 127.0.0.0/8 allow access-control: 10.10.10.0/24 allow do-daemonize: no logfile: "" username: unbound directory: /usr/local/var/service/unbound chroot: /usr/local/var/service/unbound pidfile: /usr/local/var/service/unbound/unbound.pid verbosity: 1 minimal-responses: yes prefetch: yes qname-minimisation: yes # we are doing DNS over TLS ssl-upstream: yes root-hints: /usr/local/var/service/unbound/config/root.hints # my DNS zone at home include: /usr/local/var/service/unbound/config/local.zone # autogenerated every night to block malicious domains include: /usr/local/var/service/unbound/config/blackhole.zone forward-zone: name: "." forward-addr: 220.127.116.11@853 # quad9.net primary forward-addr: 18.104.22.168@853 # quad9.net secondary #forward-addr: 22.214.171.124@853 # cloudflare primary #forward-addr: 126.96.36.199@853 # cloudflare secondary remote-control: control-enable: yes control-interface: /usr/local/var/service/unbound/control.clt control-use-cert: no
Fixing OpenSC after updating to MacOS Sierra•
Sierra introduced restrictions to the ssh-agent (new version of OpenSSH) by limiting the PKCS#11 libraries that can be loaded to a list of whitelisted directories. As of now, this is public domain because I must be one of the last persons updating from El Capitan to Sierra. Yes, I am not an early adopter!
So, until now I had an alias in my .bashrc that was loading my SSH key in the Yubikey to the ssh agent. The alias was just fine but the library is now outside the trusted path, that is “/usr/lib:/usr/local/lib”.
alias load_key="ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so" alias unload_key="ssh-add -e /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so"
As always, the error message is everything but useful:
$ ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so Enter passphrase for PKCS#11: Could not add card "/Library/OpenSC/lib/pkcs11/opensc-pkcs11.so": agent refused operation
I had to run the ssh-agent in debug mode to understand what was happening (Google is your friend) and the output said: provider not whitelisted.
$ ssh-agent -d -a /tmp/agent.socket SSH_AUTH_SOCK=/tmp/agent.socket; export SSH_AUTH_SOCK; echo Agent pid 2918; debug2: fd 3 setting O_NONBLOCK debug3: fd 4 is O_NONBLOCK debug1: type 20 refusing PKCS#11 add of "/Library/OpenSC/lib/opensc-pkcs11.so": provider not whitelisted debug1: XXX shrink: 3 < 4
$ SSH_AUTH_SOCK="/tmp/agent.socket" ssh-add -s /Library/OpenSC/lib/pkcs11/opensc-pkcs11.so Enter passphrase for PKCS#11: Could not add card "/Library/OpenSC/lib/pkcs11/opensc-pkcs11.so": agent refused operation
Goggling again, the first hit is the bug report that was opened last year.
MacOS: cannot use /usr/local/lib/opensc-pkcs11.so (provider not whitelisted)
At the end, I had to modify my aliases to the library located in the trusted path.
alias load_key="ssh-add -s /usr/local/lib/opensc-pkcs11.so" alias unload_key="ssh-add -e /usr/local/lib/opensc-pkcs11.so"
The original post where I setup SSH public key authentication with security tokens
Backing up my Githup repos•
I am using runwhen together with daemontools to launch and monitor the backup. The run script used by the svc service executes runwhen commands to sleep until the next run (every hour) and then launch the backup script. The service is running in a dedicated jail.
The run script listed below uses some runwhen commands (rw-add,rw-matchand rw-sleep) to wake-up every hour and setuidgid to run the service with an unprivileged user.
#!/bin/sh exec 2>&1 exec setuidgid gitbackup \ rw-add n d1S now1s \ rw-match \$now1s ,M=00 wake \ rw-sleep \$wake \ /home/gitbackup/update.sh
The actual backup script that iterates over all the git repos and fetches the changes.
#!/bin/sh exec 2>&1 cd /usr/home/gitbackup/backup echo "====" date echo "====" for repo in `ls -d1 *.git`; do cd $repo && /usr/local/bin/git fetch --all cd - done echo "===="
checking the output log
$ cat /var/service/backups/log/main/current | tai64nlocal 2018-02-05 18:00:00.098641500 ==== 2018-02-05 18:00:00.150083500 Mon Feb 5 18:00:00 CET 2018 2018-02-05 18:00:00.180056500 ==== 2018-02-05 18:00:00.211689500 Fetching origin 2018-02-05 18:00:01.073738500 From https://github.com/xgarcias/ansible-cmdb-freebsd-template 2018-02-05 18:00:01.073743500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:01.091577500 Fetching origin 2018-02-05 18:00:02.185366500 From https://github.com/xgarcias/ansible-daemontools 2018-02-05 18:00:02.185371500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:02.203049500 Fetching origin 2018-02-05 18:00:04.180310500 From https://github.com/xgarcias/ansible-macbook 2018-02-05 18:00:04.180315500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:04.198104500 Fetching origin 2018-02-05 18:00:06.448429500 From https://github.com/xgarcias/daemontools-dyndns 2018-02-05 18:00:06.448434500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:06.466266500 Fetching origin 2018-02-05 18:00:08.299785500 From https://github.com/xgarcias/daemontools-poudriere 2018-02-05 18:00:08.299790500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:08.321755500 Fetching origin 2018-02-05 18:00:09.749956500 From https://github.com/xgarcias/daemontools-unbound-sinkhole 2018-02-05 18:00:09.749961500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:09.771744500 Fetching origin 2018-02-05 18:00:11.113934500 From https://github.com/xgarcias/elasticsearch-plugin-readonlyrest 2018-02-05 18:00:11.113939500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:11.135774500 Fetching origin 2018-02-05 18:00:12.703191500 From https://github.com/xgarcias/freebsd_local_ports 2018-02-05 18:00:12.703197500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:12.724967500 Fetching origin 2018-02-05 18:00:13.583204500 From https://github.com/xgarcias/xgarcias.github.io 2018-02-05 18:00:13.583209500 * branch HEAD -> FETCH_HEAD 2018-02-05 18:00:13.601461500 ====
Querying ANS/IP records via non rate-limited REST API•
Querying ANS/IP records via non rate-limited unauthenticated REST API.
Also, you can use @DuckDuckGo to get the same results with the !Arin and !Ripe bang searches.
Periodic reminder of ARIN's public, non rate-limited, unauthenticated REST API for ASN/IP/Network lookups— Andrew Morris (@Andrew___Morris) January 27, 2018
docs: https://t.co/iMWUOFZcgr pic.twitter.com/Yc7HQ69xrI
You can also use !Arin and !Ripe bang searches on @DuckDuckGo to quickly lookup IP information— Greg Bray (@GBrayUT) January 27, 2018