Moving to new machine binti

Installation of the new machine "binti" in new server

Anoia > Anna (US) Moving to Binti > Ursula (SE)

Binti as the main character of a science fiction novella written by Nnedi Okorafor (2015). https://en.wikipedia.org/wiki/Binti_(novel)

We have setup a virtual server (KVM) on our host server for you:

 Host server:    ursula.tachanka.org
 Virtual server: binti.tachanka.org
 Storage:        120G
 Memory:         4096M
 CPU:            2

Network configuration for your virtual server: Public IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org

 ssh binti@ursula.tachanka.org
 

Steps to do:

Install Debian Stable

Set up full disk encryption in your virtual machine during the installation, and keep encrypted backups of the passphrase as well as backups of any important data, as we do not keep backups of your data by default.

Add ssh keys inside binti

Next steps:

Install and configure some security things : ssh server iptable + ufw fail2ban things like chkrootkit rkhunter etckeeper ? configure an everyday mail report sent to sysadmins ? Install and configure some softwares on the host : apache2 LXC

Reinstall AS by testing current back up system > shutting donw anoia and passing it over to tachanka

Documentation:

   Add new tech documentation
   Reframe current wiki page: https://alexandria.anarchaserver.org/index.php/Machine

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

b01 : yes, I could add the keys in/au /home/binti/.sshthorized_keys I can connect to the server with ssh binti@ursula.tachanka.org

Once connected with ssh binti@ursula.tachanka.org it is possible to reach the installation screen with :

   screen -x

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Installation of debian stable 10 buster

Network configuration for your virtual server: IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org

B01 creates a new password for root in binti > 40 characters and will share with sysadmin over GPG

A user Anarcha is created for non administrative tasks inside the server > b01 creates a new password > 40 characters and will share with sysadmin over GPG

Setting up full disk encryption in Binti during the installation - create a swap of 8 GB ? Pass phrase is created for encryption > b01 creates a new password > 40 characters and will share with sysadmin over GPG

We answer yes to be part of the debian package survey

Grub created and located in /dev/vda/

We could decrypt and log in binti with passwords created

Put the LUKS passphrase when the VM boot

Connect to the virtual screen session to see the boot process (an admin must have put your public key in the rights place)

ssh binti@ursula.tachanka.org

You should see a prompt that ask for the LUKS passphrase, put it and then to detach from the screen session :

 "C-\ d" to detach

Improve the security

ssh

basic sshd_config same as previous :

PasswordAuthentication no
X11Forwarding no
PermitRootLogin no

Cammbiamos puerto al 2212, ahora nos logueamos con

ssh -p 2212 user@binti.tachanka.org

Folder ssh keys of AS sysadmin added inside binti Process: Created users for all syadmin, add each at sudo group and added related SSH key (the one used for anoia) so you can access the server, from there you can decide to keep old key or to generate a new one for binti as described here https://alexandria.anarchaserver.org/index.php/Access_server

unattended upgrades

Activar unattended upgrades > se hace con el script que actualiza todas los containers mejor, hay que configurarlo, no?

sudo apt install unattended-upgrades
sudo apt install apt-listchanges
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "false";

Si queremos que nos llege un mail cada dia con las actualizaciones que se hicieron hay que configurar un servicio de correo (no lo hicimos)

En /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

fail2ban

Activar fail2ban:

apt-get install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

borramos las jails que no usamos

Activamos las jails: sshd , apache y apache badbots (dejamos para investigar como protejer los containers)

Firewall

ufw 
ufw default deny incoming
ufw default allow outgoing
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 2212/tcp -> 
ufw allow 8000/tcp

Security -References

Resources on security and devops/sysadmin:

• https://dev.to/shosta/security-headers-to-use-on-your-webserver-3id5
• https://content-security-policy.com/
• http://opsreportcard.com/section/11
• https://the-sysadmin-book.com/

About unattended upgrades

• https://haydenjames.io/how-to-enable-unattended-upgrades-on-ubuntu-debian/
• https://wiki.debian.org/UnattendedUpgrades

Fail2ban

• https://www.a2hosting.com/kb/security/hardening-a-server-with-fail2ban
• https://labekka.red/servidoras-feministas/2019/10/09/fanzine-parte-4.html#os

UFW

• https://help.ubuntu.com/community/UFW

Apache2 web server

Installation of apache2 server

apt install apache2

Specific configuration : add some modules

a2enmod ssl
a2enmod rewrite
a2enmod proxy_http
a2ensite > to activate the imported virtual hosts

Some changes has been made to increase the MaxWorkers numbers, see this article

Find which Multi-Processing Module (MPM) is currently in use:

apache2ctl -V | grep MPM
Server MPM:     event

Edit the appropriated file :

nano /etc/apache2/mods-enabled/mpm_event.conf
<IfModule mpm_event_module>
       StartServers            3
       MinSpareThreads         5
       MaxSpareThreads         10
       ServerLimit             250
       ThreadLimit             64
       ThreadsPerChild         25
       MaxRequestWorkers       1250
       MaxConnectionsPerChild   0
</IfModule>

Check the numbers of connexions (to check if the server is not under DDoS attack) :

netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

Install LXC containers

Installing the base of LXC following :

• https://wiki.debian.org/LXC
• https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html

Restoration of the container from the previous machine with duplicity

Config of the containers with this type of config (at the end, the most basic one), there has been changes of the config file with debian 10 buster :

nano /var/lib/lxc/livingdata/config
lxc.net.0.type = veth
lxc.net.0.hwaddr = 00:16:3e:ff:1f:df
lxc.net.0.link = virbr0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = /var/lib/lxc/livingdata/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.tty.max = 4
lxc.uts.name = livingdata
lxc.arch = amd64
lxc.pty.max = 1024
lxc.start.auto = 1

is working and giving adresse 192.168.122.*

vhost apache proxy to the containers

Restoration of the apache2 vhost to proxy the traffic to the containers, exemple of transition subdomain / container :

nano /etc/apache2/sites-enabled/transitional.conf

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        ServerName transitional.anarchaserver.org
        ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
        CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
  ProxyPreserveHost       On
  ProxyRequests           Off
  ProxyPass / http://192.168.x.x/
  ProxyPassReverse http://192.168.x.x/ /
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>
RewriteEngine on
RewriteCond %{SERVER_NAME} =transitional.anarchaserver.org
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

nano /etc/apache2/sites-enabled/transitional-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName transitional.anarchaserver.org
        ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
        CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
        ProxyPreserveHost       On
        ProxyRequests           Off
        ProxyPass /  https://transitional.anarchaserver.org/
        ProxyPassReverse / https://transitional.anarchaserver.org/
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>
       SSLEngine on
       SSLProxyEngine On
       SSLProxyVerify none
       SSLProxyCheckPeerCN off
       SSLProxyCheckPeerName off
       SSLProxyCheckPeerExpire off

RewriteEngine on
SSLCertificateFile /etc/letsencrypt/live/anarchaserver.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/anarchaserver.org/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
</IfModule>

Add the ip of the container in /etc/hosts if the ssl proxy is a dns :

nano /etc/hosts 
192.168.x.x transitional.anarchaserver.org

SSL certificates

Utilisation of certbot with gandi plug-in (our registrar) to create and update the certificates for the main domain and all subdomains (wildcard) : One certificate for all the subdomains

• Certbot wildcard tab : https://certbot.eff.org/lets-encrypt/debianbuster-apache
• Plug in Gandi for automatic renew https://github.com/obynio/certbot-plugin-gandi
• How to get the Gandi API key https://doc.livedns.gandi.net/

Install Gandi plug-in on Debian 10 :

apt remove python-cryptography
pip install cryptography
pip install certbot-plugin-gandi

Activate mod ssl for Apache :

a2enmod ssl
systemctl restart apache2

Create and Install the certificates :

certbot certonly -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini -d anarchaserver.org -d "*.anarchaserver.org" --server https://acme-v02.api.letsencrypt.org/directory

To renew them automatically, and restart apache2 add in crontab as root :

crontab -e
0 0 * * 0 /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory --post-hook "systemctl restart apache2"

Don't forget to restart apache

systemctl restart apache2

Also remove the Timer driven by systemd (installed by default with certbot threw the package mamager) to let the cron job do the renew process :

• List Existing Timers

systemctl list-timers
systemctl list-timers --all

• remove the certbot.timer and disable certbot.service

rm /etc/systemd/system/timers.target.wants/certbot.timer
systemctl disable certbot.service

• Check if the crontab works

crontab -e
cat /var/mail/root (a mail is sent each time the cronjob is done
tail -f /var/log/letsencrypt/letsencrypt.log

If certbot: error: unrecognized arguments: --certbot-plugin-gandi:dns-credentials so the crontab doesn't use the good certbot version, to check :

# certbot --version
-bash: /usr/local/bin/certbot: No such file or directory <- Former location of certbot 0.35 installed via pip
# which certbot
/usr/bin/certbot <- Location of certbot 0.28 installed with apt
# whereis certbot
certbot: /usr/bin/certbot /usr/share/man/man1/certbot.1.gz
# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

In anarchaserver case, the cron job was using the bad version of certbot (/usr/bin/certbot was throwing the error), let's remove them :

apt remove certbot python3-certbot

Indicate full path for the good certbotin the cronjob

*/5 * * * * /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory

Maybe we should add a hook to restart apache ?

Netdata monitoring and alert system

Installation of Netdata to monitor the server activity and to send alert once something is not working, now available at https://netdata.anarchaserver.org

• Installation https://learn.netdata.cloud/docs/agent/packaging/installer

 bash <(curl -Ss https://my-netdata.io/kickstart.sh) --stable-channel --disable-telemetry

• Proxy behind existing apache server https://learn.netdata.cloud/docs/agent/running-behind-apache
• Edit config https://learn.netdata.cloud/docs/agent/configuration-guide

 /etc/netdata/edit-config health_alarm_notify.conf

• Configuration of the alerts https://learn.netdata.cloud/docs/agent/health/notifications/ at the moment configured to send alert to b01 by mail

To restric access to netdata with an authetification Utilisation of auth Digest with apache :

 a2enmod auth_digest

Create a user

 htdigest -c /etc/apache2/.htpasswd netdata netdata

Add this conf to the vhost netdata-le-ssl.conf

<Proxy *>
        AuthType Digest
        AuthName "netdata"
        AuthDigestProvider file
        AuthUserFile /etc/apache2/.htpasswd
        Require valid-user
</Proxy>

force redirection to https in the other vhost

Serveur Sonification

It uses promiscuous~ from Pure Data / ext13 and create a stream > serveur icecast 2

 apt-get install puredata pd-pdogg pd-ext13 icecast2

Pour v2, ajouter

 apt install pd-ggee pd-zexy pd-iemlib

The PD patch : anarchasonification.pd and to check the audio device :

 puredata -nogui -listdev

To launch the patch :

 puredata -nogui anarchasonification.pd

The result can be listened here : http://anarchaserver.org:8000/anarchasonification.ogg.m3u

To launch the patch at the server start-up with systemctl

 nano /lib/systemd/system/anarchasonification.service

 [Unit]
 Description=Server Sonification net activity
 
 [Service]
 Type=simple
 ExecStart=/usr/bin/pd -nogui -noaudio /data/pnode/algo/pnodelocalenetsonification.pd
 StandardOutput=null
 Restart=on-failure
 
 # Configures the time to wait before service is stopped forcefully.
 TimeoutStopSec=30
 
 [Install]
 WantedBy=multi-user.target
 Alias=serversonification.service

take into account the modification int the service

 sudo systemctl daemon-reload

launch it

 sudo systemctl start omnioussonification.service

activate it at start-up

 sudo systemctl enable omnioussonification.service