Moving to new machine binti

Installation of the new machine "binti" in new server

Anoia > Anna (US) Moving to Binti > Ursula (SE)

Binti as the main character of a science fiction novella written by Nnedi Okorafor (2015). https://en.wikipedia.org/wiki/Binti_(novel)

We have setup a virtual server (KVM) on our host server for you:

 Host server:    ursula.tachanka.org
 Virtual server: binti.tachanka.org
 Storage:        120G
 Memory:         4096M
 CPU:            2

Network configuration for your virtual server: Public IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org

 ssh binti@ursula.tachanka.org
 

Steps to do:

Install Debian Stable

Set up full disk encryption in your virtual machine during the installation, and keep encrypted backups of the passphrase as well as backups of any important data, as we do not keep backups of your data by default.

Add ssh keys inside binti

Next steps:

Install and configure some security things : ssh server iptable + ufw fail2ban things like chkrootkit rkhunter etckeeper ? configure an everyday mail report sent to sysadmins ? Install and configure some softwares on the host : apache2 LXC

Reinstall AS by testing current back up system > shutting donw anoia and passing it over to tachanka

Documentation:

   Add new tech documentation
   Reframe current wiki page: https://alexandria.anarchaserver.org/index.php/Machine

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

b01 : yes, I could add the keys in/au /home/binti/.sshthorized_keys I can connect to the server with ssh binti@ursula.tachanka.org

Once connected with ssh binti@ursula.tachanka.org it is possible to reach the installation screen with :

   screen -x

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Installation of debian stable 10 buster

Network configuration for your virtual server: IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org

B01 creates a new password for root in binti > 40 characters and will share with sysadmin over GPG

A user Anarcha is created for non administrative tasks inside the server > b01 creates a new password > 40 characters and will share with sysadmin over GPG

Setting up full disk encryption in Binti during the installation - create a swap of 8 GB ? Pass phrase is created for encryption > b01 creates a new password > 40 characters and will share with sysadmin over GPG

We answer yes to be part of the debian package survey

Grub created and located in /dev/vda/

We could decrypt and log in binti with passwords created

Improve the security

ssh

basic sshd_config same as previous :

PasswordAuthentication no
X11Forwarding no
PermitRootLogin no

Cammbiamos puerto al 2212, ahora nos logueamos con

ssh -p 2212 user@binti.tachanka.org

Folder ssh keys of AS sysadmin added inside binti Process: Created users for all syadmin, add each at sudo group and added related SSH key (the one used for anoia) so you can access the server, from there you can decide to keep old key or to generate a new one for binti as described here https://alexandria.anarchaserver.org/index.php/Access_server

unattended upgrades

Activar unattended upgrades > se hace con el script que actualiza todas los containers mejor, hay que configurarlo, no?

sudo apt install unattended-upgrades
sudo apt install apt-listchanges
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "false";

Si queremos que nos llege un mail cada dia con las actualizaciones que se hicieron hay que configurar un servicio de correo (no lo hicimos)

En /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

fail2ban

Activar fail2ban:

apt-get install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

borramos las jails que no usamos

Activamos las jails: sshd , apache y apache badbots (dejamos para investigar como protejer los containers)

Firewall

ufw 
ufw default deny incoming
ufw default allow outgoing
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 2212/tcp -> 
ufw allow 8000/tcp

Security -References

Resources on security and devops/sysadmin:

• https://dev.to/shosta/security-headers-to-use-on-your-webserver-3id5
• https://content-security-policy.com/
• http://opsreportcard.com/section/11
• https://the-sysadmin-book.com/

About unattended upgrades

• https://haydenjames.io/how-to-enable-unattended-upgrades-on-ubuntu-debian/
• https://wiki.debian.org/UnattendedUpgrades

Fail2ban

• https://www.a2hosting.com/kb/security/hardening-a-server-with-fail2ban
• https://labekka.red/servidoras-feministas/2019/10/09/fanzine-parte-4.html#os

UFW

• https://help.ubuntu.com/community/UFW

Apache2 web server

Installation of apache2 server

apt install apache2

Specific configuration : add some modules

a2enmod ssl
a2enmod rewrite
a2enmod proxy_http
a2ensite > to activate the imported virtual hosts

Some changes has been made to increase the MaxWorkers numbers, see this article

Find which Multi-Processing Module (MPM) is currently in use:

apache2ctl -V | grep MPM
Server MPM:     event

Edit the appropriated file :

nano /etc/apache2/mods-enabled/mpm_event.conf
<IfModule mpm_event_module>
       StartServers            3
       MinSpareThreads         5
       MaxSpareThreads         10
       ServerLimit             250
       ThreadLimit             64
       ThreadsPerChild         25
       MaxRequestWorkers       1250
       MaxConnectionsPerChild   0
</IfModule>

Check the numbers of connexions (to check if the server is not under DDoS attack) :

netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

Install LXC containers

Installing the base of LXC following :

• https://wiki.debian.org/LXC
• https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html

Restoration of the container from the previous machine with duplicity

Config of the containers with this type of config (at the end, the most basic one), there has been changes of the config file with debian 10 buster :

nano /var/lib/lxc/livingdata/config
lxc.net.0.type = veth
lxc.net.0.hwaddr = 00:16:3e:ff:1f:df
lxc.net.0.link = virbr0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = /var/lib/lxc/livingdata/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.tty.max = 4
lxc.uts.name = livingdata
lxc.arch = amd64
lxc.pty.max = 1024
lxc.start.auto = 1

is working and giving adresse 192.168.122.*

vhost apache proxy to the containers

Restoration of the apache2 vhost to proxy the traffic to the containers, exemple of transition subdomain / container :

nano /etc/apache2/sites-enabled/transitional.conf

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        ServerName transitional.anarchaserver.org
        ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
        CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
  ProxyPreserveHost       On
  ProxyRequests           Off
  ProxyPass / http://192.168.x.x/
  ProxyPassReverse http://192.168.x.x/ /
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>
RewriteEngine on
RewriteCond %{SERVER_NAME} =transitional.anarchaserver.org
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

nano /etc/apache2/sites-enabled/transitional-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName transitional.anarchaserver.org
        ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
        CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
        ProxyPreserveHost       On
        ProxyRequests           Off
        ProxyPass /  https://transitional.anarchaserver.org/
        ProxyPassReverse / https://transitional.anarchaserver.org/
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>
       SSLEngine on
       SSLProxyEngine On
       SSLProxyVerify none
       SSLProxyCheckPeerCN off
       SSLProxyCheckPeerName off
       SSLProxyCheckPeerExpire off

RewriteEngine on
SSLCertificateFile /etc/letsencrypt/live/anarchaserver.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/anarchaserver.org/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
</IfModule>

Add the ip of the container in /etc/hosts if the ssl proxy is a dns :

nano /etc/hosts 
192.168.x.x transitional.anarchaserver.org

SSL certificates

Utilisation of certbot with gandi plug-in (our registrar) to create and update the certificates for the main domain and all subdomains (wildcard) : One certificate for all the subdomains

• Certbot wildcard tab : https://certbot.eff.org/lets-encrypt/debianbuster-apache
• Plug in Gandi for automatic renew https://github.com/obynio/certbot-plugin-gandi
• How to get the Gandi API key https://doc.livedns.gandi.net/

Install Gandi plug-in on Debian 10 :

apt remove python-cryptography
pip install cryptography
pip install certbot-plugin-gandi

Activate mod ssl for Apache :

a2enmod ssl
systemctl restart apache2

Create and Install the certificates :

certbot certonly -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini -d anarchaserver.org -d "*.anarchaserver.org" --server https://acme-v02.api.letsencrypt.org/directory

To renew them automatically, and restart apache2 add in crontab as root :

crontab -e
0 0 * * 0 /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory --post-hook "systemctl restart apache2"

Don't forget to restart apache

systemctl restart apache2

Also remove the Timer driven by systemd (installed by default with certbot threw the package mamager) to let the cron job do the renew process :

• List Existing Timers

systemctl list-timers
systemctl list-timers --all

• remove the certbot.timer and disable certbot.service

rm /etc/systemd/system/timers.target.wants/certbot.timer
systemctl disable certbot.service

• Check if the crontab works

crontab -e
cat /var/mail/root (a mail is sent each time the cronjob is done
tail -f /var/log/letsencrypt/letsencrypt.log

If certbot: error: unrecognized arguments: --certbot-plugin-gandi:dns-credentials so the crontab doesn't use the good certbot version, to check :

# certbot --version
-bash: /usr/local/bin/certbot: No such file or directory <- Former location of certbot 0.35 installed via pip
# which certbot
/usr/bin/certbot <- Location of certbot 0.28 installed with apt
# whereis certbot
certbot: /usr/bin/certbot /usr/share/man/man1/certbot.1.gz
# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

In anarchaserver case, the cron job was using the bad version of certbot (/usr/bin/certbot was throwing the error), let's remove them :

apt remove certbot python3-certbot

Indicate full path for the good certbotin the cronjob

*/5 * * * * /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory

Maybe we should add a hook to restart apache ?

Netdata monitoring and alert system

Installation of Netdata to monitor the server activity and to send alert once something is not working, now available at https://netdata.anarchaserver.org

• Installation https://learn.netdata.cloud/docs/agent/packaging/installer

 bash <(curl -Ss https://my-netdata.io/kickstart.sh) --stable-channel --disable-telemetry

• Proxy behind existing apache server https://learn.netdata.cloud/docs/agent/running-behind-apache
• Edit config https://learn.netdata.cloud/docs/agent/configuration-guide

 /etc/netdata/edit-config health_alarm_notify.conf

• Configuration of the alerts https://learn.netdata.cloud/docs/agent/health/notifications/ at the moment configured to send alert to b01 by mail

To restric access to netdata with an authetification Utilisation of auth Digest with apache :

 a2enmod auth_digest

Create a user

 htdigest -c /etc/apache2/.htpasswd netdata netdata

Add this conf to the vhost netdata-le-ssl.conf

<Proxy *>
        AuthType Digest
        AuthName "netdata"
        AuthDigestProvider file
        AuthUserFile /etc/apache2/.htpasswd
        Require valid-user
</Proxy>

force redirection to https in the other vhost

Serveur Sonification

It uses promiscuous~ from Pure Data / ext13 and create a stream > serveur icecast 2

 apt-get install puredata pd-pdogg pd-ext13 icecast2

Pour v2, ajouter

 apt install pd-ggee pd-zexy pd-iemlib

Création d'une carte son virtuelle :

 modprobe snd-dummy

 * http://superuser.com/questions/344760/how-to-create-a-dummy-sound-card-device-in-linux-server

Pour rendre la création de la carte son virtuelle permanente :

 nano /etc/modules

 # /etc/modules: kernel modules to load at boot time.
 #
 # This file contains the names of kernel modules that should be loaded
 # at boot time, one per line. Lines beginning with "#" are ignored.
 # Parameters can be specified after the module name.
 snd-dummy

Pour debian jessie :

 sudo nano /etc/modprobe.d/alsa.conf

1. ALSA portion

alias char-major-116 snd alias snd-card-0 snd-dummy

1. module options should go here

1. OSS/Free portion

alias char-major-14 soundcore alias sound-slot-0 snd-card-0

1. card #1

alias sound-service-0-0 snd-mixer-oss alias sound-service-0-1 snd-seq-oss alias sound-service-0-3 snd-pcm-oss alias sound-service-0-8 snd-seq-oss alias sound-service-0-12 snd-pcm-oss

 * http://www.alsa-project.org/main/index.php/Matrix%3aModule-dummy

The PD patch : Template:Media 02:anarchasonification.pdFile:anarchasonification.pd and to check the audio device :

 puredata -nogui -listdev
 audio input devices:
 1. HDA Intel PCH (hardware)
 2. HDA Intel PCH (plug-in)
 3. Dummy (hardware)
 4. Dummy (plug-in)
 audio output devices:
 1. HDA Intel PCH (hardware)
 2. HDA Intel PCH (plug-in)
 3. Dummy (hardware)
 4. Dummy (plug-in)
 API number 1

Not sure this step is necessary at the end ...

To launch the patch :

 puredata -nogui anarchasonification.pd

Labomedia's icecast server is configured in the patch for the moment ... it should produce a devsonification.ogg on http://labomedia.org:8000

Variante Rasberry Pi avec Pick up

 sudo apt install puredata pd-pdogg pd-ext13 pd-zexy pd-iemlib pd-ggee
 sudo pd -nogui -listdev
 sudo pd -nogui -audiodev 3 pnodelocalenetsonification-withoutpwd.pd

Le patch v2 Template:Media 11:pnodelocalenetsonification-withoutpwd.pd

To launch the patch at the server start-up with systemctl

 nano /lib/systemd/system/omnioussonification.service

y mettre :

 [Unit]
 Description=Omnious Sonification Service
 
 [Service]
 ExecStart=/home/b01/omnioussonification.sh
 StandardOutput=null
 
 ExecStop=/bin/sh -c "killall pd"
 
 [Install]
 WantedBy=multi-user.target
 Alias=omnioussonification.service

Créer un script dans par exemple /home/b01/omnioussonification.sh

 #!/bin/bash
 sudo puredata -nogui /home/b01/omnioussonification.pd
 exit 0

Une variante sans script :

 [Unit]
 Description=Server Sonification net activity
 
 [Service]
 Type=simple
 ExecStart=/usr/bin/pd -nogui -noaudio /data/pnode/algo/pnodelocalenetsonification.pd
 StandardOutput=null
 Restart=on-failure
 
 # Configures the time to wait before service is stopped forcefully.
 TimeoutStopSec=30
 
 [Install]
 WantedBy=multi-user.target
 Alias=serversonification.service

commande magique pour prise en compte des modifs a posteriori dans le .service ?

 sudo systemctl daemon-reload

lancer le service

 sudo systemctl start omnioussonification.service

activer le service au démarrage du serveur

 sudo systemctl enable omnioussonification.service