This repository was archived by the owner on Oct 8, 2022. It is now read-only.
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
#+TITLE: ACM Tech Stack
#+DESCRIPTION: Moving all of ACM's services to cloud providers is going to be a tough one.
*Update: For the foreseeable future, this repository will be archived. Will consider revival when infrastructure returns to deployment using Ansible and Terraform.*
This repository includes all the files necessary to describe and deploy the
technology stack of ACM @ UCSD. You will find files such as:
- Ansible roles and playbooks
- Terraform files
- Custom patches and images for certain services
* Introduction
ACM @ UCSD's tech stack is complex enough to warrant documentation. This repository,
as well as this document, provide a description so as to test and deploy the technology
stack.
Deployment and provisioning of instances supported by Ansible and Terraform.
* TL;DR
Update any changes for the infrastructure, configuration files or environment variables.
Start the instances:
#+BEGIN_SRC sh
terraform apply
#+END_SRC
Afterwards, add your private SSH key to each of the hosts and run this:
#+BEGIN_SRC sh
ansible-galaxy install -r requirements.yml
ansible-playbook -i production stack.yml
#+END_SRC
* Tech Stack Layout
The tech stack has the following main services:
- Membership Portal UI & API
- ACMURL
- BreadBot
- Password Manager
Services will be deployed on separate instances of reasonable size for
each one.
We will begin by provisioning instances for each service, and then we'll proceed
with configuring the domains for each of them.
* Provisioning
All of the services above most likely require a maximum of 1 GB of RAM.[fn:current-stack-requirements]
In order to maintain uptime for all components of the stack, each
service will run on a separate instance of appropriate size.
Additionally, each service will be provisioned with separate Ansible
playbooks.
We'll begin with the standard configuration for most AWS instances; a SSH keypair
and a security group allowing trivial access to HTTP, HTTPS and SSH. I've included
my personal public key (@Storm_FireFox1) for the purposes of demonstration.
#+BEGIN_SRC terraform :tangle instances.tf
provider "aws" {
region = "us-west-1"
}
resource "aws_key_pair" "stormfirefox1" {
key_name = "stormfirefox1-key"
public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCj25h71sryk/5epo3yObh8HspR6ODMyaF/TSSC1SpIB2f9mPUKeCA8k32mrNpRCtVVwAhCn9OoVLfbfs4KzPNrxVByexqH6FORXY75mRUVdqF2qv9WDk7lANiTXV12+b9sMhtbGsU7r+CXGMCN+QZvnvLFb9xadKxNhJIyy+yQFu931uKl+GcGgePs4yj8uRLQ+o1yNU1WxzZgGHbBRAv/B98RCWoJTd+8RHfT/OxQL0PKx/LdHae/ium99xEJQ5DrbtMI+9BEM4vKN0FMCuVjPQ8sC8tIxzyqWZYyCIkiZmoH7TKGvD1Yg9jI9RxcuXB5793seiORAw9pH/U5w/8JKie5eGWXgrJkHMqqrvNGrw6TEk2gfaBllm8QSfS8O97xs6/BC1ENXUZ9R4rEGKpW9+pzHx/FTyD6FMMRstRcPjPCtU6F15g6XYAAjpbvxn4RiLT1Rk3mbJtM2xTAIUjYSfukGKzAs5gRp3TSNgdN+nsmcbJRMsA8FXubogDtYUO3VbpjsALA/bBHtjKziBVhbnoQ6PrXFrFaojeSq4Sf8iOOXK29Xfv/TSc+potmfw4wQMwSXJgOBFFhv9Xa/LwGf+1KpHpsfHO4xJ2yLpTUoiryvsbDsNk0+JbkMUgfNrJ+W01Lwf4qMKXbBOtNFf8gkTnj2Af3N6OCFa8XdgRO+Q== STORM_GPG_2FB5275E"
}
resource "aws_security_group" "allow_https_ssh" {
name = "allow_https_ssh"
description = "Allow HTTPS/SSH traffic"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
#+END_SRC
We'll continue with the easiest instances to plan, which are those for
small services; BreadBot and Password Manager.
#+BEGIN_SRC terraform :tangle instances.tf
# Ubuntu 20.04 LTS instance, for good measure
# Find other AMI's at https://cloud-images.ubuntu.com/locator/ec2/
resource "aws_instance" "breadbot" {
ami = "ami-021809d9177640a20"
instance_type = "t3a.nano"
key_name = aws_key_pair.stormfirefox1.key_name
security_groups = [aws_security_group.allow_https_ssh.name]
tags = {
Name = "BreadBot"
}
}
resource "aws_instance" "pass" {
ami = "ami-021809d9177640a20"
instance_type = "t3a.nano"
key_name = aws_key_pair.stormfirefox1.key_name
security_groups = [aws_security_group.allow_https_ssh.name]
tags = {
Name = "Password Manager"
}
}
#+END_SRC
Additionally, we may in the future, move Minecraft to AWS; in which case
we'll need to provision that as well in the future. We'll put it in here
for prosperity.
#+BEGIN_SRC terraform :tangle instances.tf
# RETIRED TEMPORARILY
# resource "aws_instance" "minecraft" {
# ami = "ami-021809d9177640a20"
# instance_type = "t3a.medium"
# key_name = aws_key_pair.stormfirefox1.key_name
# security_groups = [aws_security_group.allow_https_ssh.name]
# tags {
# Name = "Minecraft Server"
# }
# }
#+END_SRC
We will also include instances for the membership portal; we will need
one for the API, and we'll use AWS' RDS in order to host the PostgreSQL
database:
#+BEGIN_SRC terraform :tangle instances.tf
# ON STANBY ON HEROKU, WIP
# resource "aws_instance" "membership-portal" {
# ami = "ami-0a63cd87767e10ed4"
# instance_type = "t3a.micro"
# tags {
# Name = "Membership Portal API"
# }
# }
#
# resource "aws_db_instance" "membership-portal-db" {
# allocated_storage = 10
# engine = "postgres"
# instance_class = "db.t3.micro"
# name = var.dbName
# username = var.dbUser
# password = var.dbPass
# }
#+END_SRC
Lastly, we'll include the S3 buckets used by the Membership Portal:
#+BEGIN_SRC terraform :tangle instances.tf
resource "aws_s3_bucket" "acmucsd" {
bucket = "acmucsd"
acl = "private"
}
resource "aws_s3_bucket" "acmucsd-membership-portal" {
bucket = "acmucsd-membership-portal"
acl = "private"
}
#+END_SRC
Note the above variables used originate using the provided =.env= file. Edit
its contents with values of your choice. The playbooks will also use them
to properly deploy =.env= files.
Using the [[https://calculator.aws][AWS Calculator]], we obtain the cost for all the instances per year:
| Instance | Instance Type | Cost Per Month |
|---------------------+---------------+----------------|
| BreadBot | t3a.nano | 5.29 |
| Pass | t3a.nano | 5.29 |
| Portal API | t3a.micro | 9.38 |
| Portal Database | db.t3.micro | 16.55 |
|---------------------+---------------+----------------|
| Total Cost / Month: | | 36.51 |
| Total Cost / Year: | | 438.12 |
|---------------------+---------------+----------------|
#+TBLFM: @6$3=vsum(@I..@II)::@7$3=vsum(@I..@II)*12
Assuming all goes well, running Terraform will deploy the stack.
#+BEGIN_SRC sh
terraform init && terraform apply
#+END_SRC
We will now begin deploying the software for each service.
We'll document the SSH commands used in tandem with the respective Ansible task
and role.
[fn:current-stack-requirements] Funnily enough, this is hard to quantify properly. Pass and BreadBot both occupied the GCP =f1.micro=, which has 0.6 GB of RAM, so maybe a bit more is useful. The API, however, is up for discussion, considering Heroku's less obvious nature with system requirements.
* Domain Pointing
We'll need to configure the domain for each single service, along with a few other
manual record pointers for domains. We will use the =acmucsd.com= domain, as that
is where most of our tech stack is linked to:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_zone" "acmucsd-com-public" {
name = "acmucsd.com"
comment = "ACM's main domain for stuff."
tags = {
}
}
resource "aws_route53_zone" "acmurl-com-public" {
name = "acmurl.com"
comment = "ACM's domain used for URL shortening."
tags = {
}
}
#+END_SRC
We'll add the standard records for domains in the mix, including the root A record,
NS records and SOA records:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "acmucsd.com"
type = "A"
records = ["104.198.14.52"]
ttl = "3600"
}
resource "aws_route53_record" "www-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "www.acmucsd.com"
type = "A"
records = ["104.198.14.52"]
ttl = "3600"
}
resource "aws_route53_record" "acmucsd-com-NS" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "acmucsd.com"
type = "NS"
records = ["ns-2024.awsdns-61.co.uk.", "ns-278.awsdns-34.com.", "ns-1200.awsdns-22.org.", "ns-591.awsdns-09.net."]
ttl = "172800"
}
resource "aws_route53_record" "acmucsd-com-SOA" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "acmucsd.com"
type = "SOA"
records = ["ns-2024.awsdns-61.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"]
ttl = "900"
}
#+END_SRC
For the root domain, we'll also add the necessary CNAME records for pointing to Sendgrid's
DKIM TXT records:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "s1-_domainkey-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "s1._domainkey.acmucsd.com"
type = "CNAME"
records = ["s1.domainkey.u17821998.wl249.sendgrid.net"]
ttl = "3600"
}
resource "aws_route53_record" "s2-_domainkey-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "s2._domainkey.acmucsd.com"
type = "CNAME"
records = ["s2.domainkey.u17821998.wl249.sendgrid.net"]
ttl = "3600"
}
#+END_SRC
The domains added here are, in reality, ACM AI's domains for their own infrastructure.
We'll leave the records here to be explained by them:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "ai-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "ai.acmucsd.com"
type = "CNAME"
records = ["acmucsd-ai.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "sendgrid-ai-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "17821998.ai.acmucsd.com"
type = "CNAME"
records = ["sendgrid.net"]
ttl = "3600"
}
resource "aws_route53_record" "api-ai-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "api.ai.acmucsd.com"
type = "A"
records = ["104.155.168.98"]
ttl = "3600"
}
resource "aws_route53_record" "open-ai-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "open.ai.acmucsd.com"
type = "CNAME"
records = ["openai-acm-ai.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "compete-ai-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "compete.ai.acmucsd.com"
type = "A"
records = ["34.120.177.157"]
ttl = "3600"
}
resource "aws_route53_record" "apitest-ai-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "apitest.ai.acmucsd.com"
type = "A"
records = ["35.208.23.117"]
ttl = "3600"
}
resource "aws_route53_record" "em4616-ai-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "em4616.ai.acmucsd.com"
type = "CNAME"
records = ["u17821998.wl249.sendgrid.net"]
ttl = "3600"
}
resource "aws_route53_record" "url4522-ai-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "url4522.ai.acmucsd.com"
type = "CNAME"
records = ["sendgrid.net"]
ttl = "3600"
}
#+END_SRC
The Membership Portal is continuing to be hosted on Heroku, so we'll need CNAMEs
pointing to Heroku's DNS services for the API:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "api-test-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "api-test.acmucsd.com"
type = "CNAME"
records = ["acmucsd-portal-testing.herokuapp.com"]
ttl = "3600"
}
resource "aws_route53_record" "api-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "api.acmucsd.com"
type = "CNAME"
records = ["shallow-koi-v9n1nho6ee48b480cn08m1hr.herokudns.com"]
ttl = "3600"
}
#+END_SRC
Up next will come an influx of CNAMEs pointing to Netlify apps, where most React
sites are hosted by ACM:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "design-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "design.acmucsd.com"
type = "CNAME"
records = ["acmdesign.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "hack-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "hack.acmucsd.com"
type = "CNAME"
records = ["acmhack-site.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "members-test-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "members-test.acmucsd.com"
type = "CNAME"
records = ["members-nightly.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "members-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "members.acmucsd.com"
type = "CNAME"
records = ["acmucsd.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "space2020-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "space2020.acmucsd.com"
type = "CNAME"
records = ["space2020.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "splash-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "splash.acmucsd.com"
type = "CNAME"
records = ["acmucsd-purple.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "static-template-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "static-template.acmucsd.com"
type = "CNAME"
records = ["acm-static.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "tree-acmucsd-com-CNAME" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "tree.acmucsd.com"
type = "CNAME"
records = ["acmucsd-tree.netlify.app"]
ttl = "3600"
}
resource "aws_route53_record" "vote-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "vote.acmucsd.com"
type = "A"
records = ["217.156.97.70"]
ttl = "3600"
}
#+END_SRC
Each instance from the ones we provisioned has their own domain, so let's assign A records for all
the instances we just deployed. We'll set two domains for ACMURL: =acmurl.com= and =url.acmucsd.com=:
#+BEGIN_SRC terraform :tangle domains.tf
resource "aws_route53_record" "bot-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "bot.acmucsd.com"
type = "A"
records = [aws_instance.breadbot.public_ip]
ttl = "3600"
}
resource "aws_route53_record" "acmurl-com-A" {
zone_id = aws_route53_zone.acmurl-com-public.zone_id
name = "acmurl.com"
type = "A"
records = [aws_instance.breadbot.public_ip]
ttl = "3600"
}
resource "aws_route53_record" "url-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "url.acmucsd.com"
type = "A"
records = [aws_instance.breadbot.public_ip]
ttl = "3600"
}
resource "aws_route53_record" "pass-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "pass.acmucsd.com"
type = "A"
records = [aws_instance.pass.public_ip]
ttl = "3600"
}
resource "aws_route53_record" "mc-acmucsd-com-A" {
zone_id = aws_route53_zone.acmucsd-com-public.zone_id
name = "mc.acmucsd.com"
type = "A"
records = ["51.81.26.152"]
ttl = "3600"
}
#+END_SRC
* Before Configuring
For debugging purposes, the =tech-stack= repo provides
a Vagrantfile to deploy test instances locally. You can boot each of them by
installing Vagrant and VirtualBox; afterwards, you run this command in the terminal:
#+BEGIN_SRC sh
vagrant up
#+END_SRC
For the purposes of making this documentation easier to follow, we'll assume
that you are provisioning instances to configure using either the provided
Terraform file or the provided Vagrantfile.
=tech-stack= provides two Ansible inventory files in order to ease its use;
=testing= and =production=; =testing= points to the local Ansible boxes, whereas
=production= points to the Terraform-provisioned instances; you may use either
one by including it in each Ansible playbook command:
#+BEGIN_SRC sh
ansible-playbook -i <inventory>
#+END_SRC
It is also recommended you set all of the environment variables in every file for
each service you wish to configure. There are =.env.example= files provided for
each service; all that is necessary is to set the variables for each instance
by copying each of the files and then adding the secrets:
#+BEGIN_SRC sh
cp .env.example .env
#+END_SRC
A self-hosted Vault instance would fix the problem of hosting secrets like these,
but that's too complex for this setup.
* Password Manager
The password manager is, in essence, a =bitwarden_rs= instance deployed on top
of a Bitwarden image. We will use the =pass= Ansible host for all the following
commands once we start building the playbook.
The easiest way, by far, to install =bitwarden_rs= is to use the Docker Compose tutorial
provided by the [[https://github.com/dani-garcia/bitwarden_rs/wiki/Using-Docker-Compose][bitwarden_rs wiki]].
First, configure the remote machine to be able to connect to it using your SSH
keys. Either [[https://www.ssh.com/ssh/keygen/][create a new one]] or provide the public key. If developing using a
Vagrant box, use this command to import the =pass= settings to your
configuration:
#+BEGIN_SRC sh
vagrant ssh-config --host pass >> $HOME/.ssh/config
#+END_SRC
You can also set these parameters manually in your SSH configuration file for
the Terraform-provisioned instances.[fn:ssh-terraform]
First, we'll want to install Docker by following the [[https://docs.docker.com/engine/install/ubuntu/][installation guide]] for Ubuntu:
#+BEGIN_SRC sh :dir /ssh:pass|sudo:pass:~/ :results drawer
sudo apt-get update
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
#+END_SRC
Similarly for Docker Compose, using this [[https://docs.docker.com/compose/install/][installation guide]]:
#+BEGIN_SRC sh :dir /ssh:pass|sudo:pass:~/
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
#+END_SRC
#+RESULTS:
| docker-compose version 1.26.2 | build eefe0d31 |
Second, we'll need to copy over the Docker Compose file for the =pass= instance, provided
in the repo, along with the environment variables for it:
#+BEGIN_SRC sh
scp ./roles/pass/files/docker-compose.yml pass:~/
scp ./roles/pass/files/.env pass:~/
scp ./roles/pass/files/Caddyfile pass:~/
#+END_SRC
Afterwards, we'll want to log into the pass instance and create the directory used
by =bitwarden_rs= in the same directory.
#+BEGIN_SRC sh :dir /ssh:pass:~/
mkdir ~/bw-data
#+END_SRC
Afterwards, all we have to do is spin up the Docker Compose service.
#+BEGIN_SRC sh :dir /ssh:pass|sudo:pass:~/ :results drawer
sudo docker-compose up -d
#+END_SRC
The Ansible role =pass= covers the installation process of the password manager
for a brand new Ubuntu instance. You will also need to deploy the =gerlingguy.docker=
role for the instance.
Run the =pass.yml= playbook to configure the password manager:
#+BEGIN_SRC sh :results drawer
ansible-playbook -i production pass.yml
#+END_SRC
Up next, we will configure BreadBot.
[fn:ssh-terraform] In reality, this is not probably necessary; AWS/Azure will likely configure
SSH keys for likely accounts in the instances, but that will require some Terraform configuration;
good task for later.
* BreadBot
BreadBot is the Discord bot for ACM, and is a Node.js bot running on a light
instance. Installation, while time-consuming, is not difficult at all.
We will use the =bot= Ansible host for all the following commands once we start
building the playbook.
Configure the remote machine to be able to connect to it using your SSH
keys. Either [[https://www.ssh.com/ssh/keygen/][create a new one]] or provide the public key. If developing using a
Vagrant box, use this command to import the =bot= instance settings to your
configuration:
#+BEGIN_SRC sh
vagrant ssh-config --host bot >> $HOME/.ssh/config
#+END_SRC
First, we will install all the required dependencies: git, npm and Node:
#+BEGIN_SRC sh :dir /ssh:bot|sudo:bot:~/ :results drawer
sudo apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs gcc g++ make
#+END_SRC
Second, we will clone the Git repo for BreadBot in the home directory:
We'll also assume the main user to run BreadBot is saved in the environment variable
=BREADBOT_USER= on the remote shell:
#+BEGIN_SRC sh :dir /ssh:bot|sudo:bot:~/ :results drawer :var BREADBOT_USER="bot"
sudo mkdir -p /opt
git clone https://github.com/acmucsd/discord-bot /opt/discord-bot
sudo chown -R $BREADBOT_USER:$BREADBOT_USER /opt/discord-bot
#+END_SRC
Afterwards, we will copy over the set-up environment variables using the provided .env file:
#+BEGIN_SRC sh
scp ./files/breadbot/.env bot:/opt/discord-bot/
#+END_SRC
Finally, we will install all the required Node dependencies:
#+BEGIN_SRC sh :dir /ssh:bot:/opt/discord-bot :results drawer
npm install
#+END_SRC
Additionally, we would like to ensure the Discord bot remains running even when our shell dies.
The easiest way to manage this (considering Ansible will also manipulate our instances) is using
a Systemd service unit. These are not that hard to configure, and we really just need a basic service
to import the environment variable and run the =npm= task for BreadBot. If you want to peruse the file,
you can find it at =files/breadbot/service.conf= but in all essence, the important part is:
#+BEGIN_SRC conf
[Service]
EnvironmentFile=/opt/discord-bot/.env
ExecStart=/usr/bin/npm run start
ExecStop=/usr/bin/pkill -f npm
WorkingDirectory=/opt/discord-bot
#+END_SRC
We'll copy this service unit file over to the server and install it under the system services. Run this
on your machine:
#+BEGIN_SRC sh
scp ./files/breadbot/breadbot.service bot:/opt/discord-bot
#+END_SRC
Afterwards, run this on the =bot= instance:
#+BEGIN_SRC sh :dir /ssh:bot|sudo:bot:~/ :results drawer
sudo mv /opt/discord-bot/breadbot.service /etc/systemd/system
sudo chown root:root /etc/systemd/system/breadbot.service
#+END_SRC
Now, we simply have to reload the Systemd daemon and start the service. From here on out, Systemd is responsible
to manage the BreadBot task:
#+BEGIN_SRC sh :dir /ssh:bot|sudo:bot:~/ :results drawer
sudo systemctl daemon-reload
sudo systemctl start breadbot.service
#+END_SRC
The Ansible role =breadbot= covers the installation process of BreadBot
for a brand new Ubuntu instance. You will also need to deploy the =gerlingguy.nodejs=
role for the instance.
Run the =bot.yml= playbook to configure the password manager:
#+BEGIN_SRC sh :results drawer
ansible-playbook -i production bot.yml
#+END_SRC
* Minecraft
Minecraft is a tricky instance to configure, primarily because of the backup
functionality, thanks to its location within GDrive. =rclone= makes
this functionality easier to configure, though.
Remember to configure the remote machine to be able to connect to it using your
SSH keys. Either [[https://www.ssh.com/ssh/keygen/][create a new one]] or provide the public key. If developing using
a Vagrant box, use this command to import the =mc= instance settings to your
configuration:
#+BEGIN_SRC sh
vagrant ssh-config --host mc >> $HOME/.ssh/config
#+END_SRC
Fortunately, the bulk of configuration for the Minecraft server is already
written by Gideon Tong over at his [[https://github.com/gideontong/ACM-Minecraft-Config][GitHub repo]]. Although not 100% complete, it's
good enough of a starter and is already imported to this repository.
Additionally to the provided GitHub repo, however, we have included:
- the Gdrive Rclone configuration to allow mounting the backup location
- the service files for running the server
- the scripts to load the Overviewer world rendering on the browser
- the world restore functionality using an additional Ansible playbook
The first, most important thing is to obtain a Gdrive service account for the
address =minecraft@acmucsd.org=, which is where the Minecraft server hosts its
backup. While there are many [[https://help.talend.com/reader/E3i03eb7IpvsigwC58fxQg/ol2OwTHmFbDiMjQl3ES5QA][tutorials]] out there on how to do that, obtaining
the service account credentials JSON is up to you. These credentials allow easy
and safe mounting of the Gdrive on the backup instance, which is where we'll do
backups.
Once you obtain the Google service account JSON file, you should add it to the
Minecraft files directory as =files/mc/credentials.json= so that the Ansible
playbook can properly extract it (*DO NOT CHECK INTO VERSION CONTROL*)
As for the Rclone configuration file that is included with the =mc= directory,
the Google console side is documented enough by following the =rclone config=
command interactive prompts on the server, so it's best to run that remotely and
get a new configuration file, should default credentials and others items
change. Otherwise, if nothing's changed, you may use the checked in version of
the configuration file.
We will first create the user =minecraft=:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
sudo useradd -m minecraft
#+END_SRC
Then, we'll create the configuration directory on the server:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
sudo mkdir -p /opt/backup
sudo chown -R minecraft:minecraft /opt/backup
#+END_SRC
Afterwards, we'll transfer the configuration files there. Run this on your local machine:
#+BEGIN_SRC sh
scp ./roles/mc/files/rclone.conf mc:~/
scp ./roles/mc/files/gdrive.service mc:~/
scp ./roles/mc/files/credentials.json mc:~/
scp ./roles/mc/files/minecraft.service mc:~/
#+END_SRC
Then run this on the server instance:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
sudo mkdir -p /opt/backup
sudo chown -R minecraft:minecraft /opt/backup
sudo mv ~/credentials.json /opt/backup
sudo mv ~/rclone.conf /opt/backup
sudo mv ~/gdrive.service /etc/systemd/system
sudo mv ~/minecraft.service /etc/systemd/system
#+END_SRC
Now, we will need to install two programs; =rclone= and =borg=, used for the
backup functionality. We'll also need to install Java for running the Minecraft
server, and We'll also install Minecraft Overviewer, necessary for rendering the
map browser viewer of the Minecraft map. This process requires adding an APT
repository to the machine as well. Lastly, we also want Caddy, a webserver to
display the Minecraft browser map we'll generate later:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
echo 'deb https://overviewer.org/debian ./' \
| sudo tee -a /etc/apt/sources.list.d/overviewer.list
echo "deb [trusted=yes] https://apt.fury.io/caddy/ /" \
| sudo tee -a /etc/apt/sources.list.d/caddy-fury.list
wget -O - https://overviewer.org/debian/overviewer.gpg.asc | sudo apt-key add - # key for Overviewer
sudo apt-get update
curl https://rclone.org/install.sh | sudo bash
sudo apt-get install -y default-jdk minecraft-overviewer caddy
#+END_SRC
The latest =borg= binary has a more involved installation process, since =borg=
has an older version in the Ubuntu repositories. We need a newer version so we
can easily extract the latest backup from the Borg backup repository whenever
we initialize a new Minecraft instance:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
curl -s https://api.github.com/repos/borgbackup/borg/releases/latest \
| grep browser_download_url \
| grep linux64 \
| cut -d '"' -f 4 | head -n 1 \
| wget -qi - -O borg-linux64
sudo cp borg-linux64 /usr/local/bin/borg
sudo chown root:root /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg
#+END_SRC
Now we will need to construct the Rclone mount for the backup functionality.
Fortunately, the service unit file required for the mount was already written by
Gideon, so we'll just have to copy the service file to the remote machine and
start and enable the service:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
sudo mkdir -p /mnt/gdrive
sudo chown -R minecraft:minecraft /mnt/gdrive
sudo systemctl daemon-reload
sudo systemctl enable gdrive
sudo systemctl start gdrive
#+END_SRC
We will now extract the latest backup from the Borg backup repository.
This will take a while, so sit back:
#+BEGIN_SRC sh :dir /ssh:mc|sudo:mc:~/ :results drawer
cd /
export BORG_PASSPHRASE='jyPr5QToT&Wca6hfrvtZA5'
export LAST_BORG_BACKUP=$(borg list --last 1 /mnt/gdrive/backup | cut -d ' ' -f1)
borg extract /mnt/gdrive/backup::$LAST_BORG_BACKUP
#+END_SRC
Now that we're done with restoring the world, as well as the other necessary
plugins and configuration files, we'll install the Paper Minecraft server JAR
directly from the [[https://papermc.io/api/v1/paper/1.15.2/latest/download][Paper API]]. We can download the file in the respective
=minecraft= directory:
#+BEGIN_SRC sh :dir /ssh:mc:~/ :results drawer
wget https://papermc.io/api/v1/paper/1.15.2/latest/download -O /opt/minecraft/paper.jar
sudo chown minecraft:minecraft /opt/minecraft/paper.jar
#+END_SRC
Next, we'll render the Minecraft map using Overviewer and also provide the proper Caddyfile
directives to Caddy, in order to serve the HTML map generated by Overviewer when done:
#+BEGIN_SRC sh :dir /ssh:mc:~/ :results drawer
overviewer.py /opt/minecraft/world /opt/minecraft/map
cat <<EOF | sudo tee -a /etc/caddy/Caddyfile
mc.acmucsd.com {
root * /opt/minecraft/map
file_server
}
EOF
sudo systemctl restart caddy
#+END_SRC
We'll add the crontab entry to run the nightly maintenance script for the Minecraft
server:
#+BEGIN_SRC sh :dir /ssh:mc:~/ :results drawer
cat <<EOF | crontab -
0 5 * * * /opt/minecraft/nightly.sh
EOF
#+END_SRC
And finally, after many commands, you may now start the Minecraft server:
#+BEGIN_SRC sh :dir /ssh:mc:~/ :results drawer
sudo systemctl start minecraft
#+END_SRC
The Ansible role =mc= covers the installation process of the Minecraft server
for a brand new Ubuntu instance. I recommend you run the playbook, though, as this
server requires additional roles.
Run the =bot.yml= playbook to configure the password manager:
#+BEGIN_SRC sh :results drawer
ansible-playbook -i production mc.yml
#+END_SRC
Up next, we will configure ACM Live.
* ACM AI API
ACM AI has a suborg-specific API used to fulfill their own needs for member outreach,
such as weekly newsletters and more. Primarily, it's a Docker image served behind a
HTTPS reverse proxy. This setup is likely to be similar to the Password Manager.
First, configure the remote machine to be able to connect to it using your SSH
keys. Either [[https://www.ssh.com/ssh/keygen/][create a new one]] or provide the public key. If developing using a
Vagrant box, use this command to import the =acm-ai-api= settings to your
configuration:
#+BEGIN_SRC sh
vagrant ssh-config --host acm-ai-api >> $HOME/.ssh/config
#+END_SRC
First, we'll want to install Docker by following the [[https://docs.docker.com/engine/install/ubuntu/][installation guide]] for Ubuntu:
#+BEGIN_SRC sh :dir /ssh:acm-ai-api|sudo:acm-ai-api:~/ :results drawer
sudo apt-get update
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
#+END_SRC
Similarly for Docker Compose, using this [[https://docs.docker.com/compose/install/][installation guide]]:
#+BEGIN_SRC sh :dir /ssh:acm-ai-api|sudo:acm-ai-api:~/
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
#+END_SRC
Fortunately, we are gracefully provided a Docker image that constantly follows the latest
version on Docker Hub, which fortunately allows for an easy run:
#+BEGIN_SRC sh :dir /ssh:acm-ai-api|sudo:acm-ai-api:~/ :results drawer
docker pull acmucsd/acm-ai-api:latest
docker run -d --name "ACM AI API" -p 9000:9000 acmucsd/acm-ai-api:latest
#+END_SRC
We'll also need a Caddy instance in front of this Docker container to provide SSL.
We'll install Caddy first:
#+BEGIN_SRC sh :dir /ssh:acm-ai-api|sudo:acm-ai-api:~/
echo "deb [trusted=yes] https://apt.fury.io/caddy/ /" \
| sudo tee -a /etc/apt/sources.list.d/caddy-fury.list
sudo apt-get update
sudo apt-get install -y caddy
#+END_SRC
We'll save the Caddyfile and then run the service:
#+BEGIN_SRC sh :dir /ssh:acm-ai-api|sudo:acm-ai-api:~/
cat <<EOF | sudo tee -a /etc/caddy/Caddyfile
api.ai.acmucsd.com {
reverse_proxy 127.0.0.1:9000
}
EOF
sudo systemctl restart caddy
#+END_SRC
Remember to configure the remote machine to be able to connect to it using your
SSH keys. Either [[https://www.ssh.com/ssh/keygen/][create a new one]] or provide the public key. If developing using
a Vagrant box, use this command to import the =live= SSH settings to your
configuration:
#+BEGIN_SRC sh
vagrant ssh-config --host live >> $HOME/.ssh/config
#+END_SRC
Most, if not all, command blocks following this line are run by =root=, so
become =root= before continuing.
We will begin by updating the Ubuntu instance entirely:
#+BEGIN_SRC sh :dir /ssh:ubuntu@live.acmucsd.com|sudo:live.acmucsd.com:~/ :results drawer
apt update && apt upgrade -y
#+END_SRC
Afterwards, we'll install BigBlueButton using their install script. We'll also remove
the demo. This will take a while:
#+BEGIN_SRC sh :dir /ssh:live|sudo:live:~/ :results drawer
wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh \
| bash -s -- -v xenial-220 -s live.acmucsd.com -e dev@acmucsd.org -g
apt-get purge bbb-demo
#+END_SRC
Now that it BBB is installed, we'll need to update the welcome message for ACM Live streams:
Add these properties to the BBB properties file:
#+BEGIN_SRC sh :dir /ssh:live|sudo:live:~/ :results drawer
cat <<EOF >> /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
defaultWelcomeMessage=Welcome to <b>%%CONFNAME%%</b>!<br><br>If you haven't already, please sign in at ACM's <a href="https://members.acmucsd.com/"><u>Membership Portal</u></a>.
defaultWelcomeMessageFooter=Thank you for using <strong>ACM Live</strong>.
EOF
#+END_SRC
We will also need to configure HTTPS redirecting to BBB in the Nginx configuration file
provided by BBB:
#+BEGIN_SRC sh :dir /ssh:live|sudo:live:~/ :results drawer
# Change to HTTPS
sudo sed -i 's/80/443 ssl/g' /etc/nginx/sites-available/bigbluebutton
# Add heredoc to beginning of file
sudo cat <<EOF /etc/nginx/sites-available/bigbluebutton > /etc/nginx/sites-available/bigbluebutton
server {
listen 80;
listen [::]:80;
server_name live.acmucsd.com;
return 301 https://$server_name$request_uri;
}
EOF
#+END_SRC
A backup functionality for ACM Live is also required to be coded, which requires
=rclone= installed again, as well as another Google service account credentials JSON.
Follow the steps from the "Minecraft" section to obtain another set of credentials.
Now BBB is fully configured, we can begin configuring its frontend, Greenlight.
We'll begin by creating an admin account. You may set the user password using
environment variables:
#+BEGIN_SRC sh :dir /ssh:live|sudo:live:~/ :results drawer
export PASSWORD= # insert password for admin account here
docker exec greenlight-v2 bundle exec rake user:create["ACM Live","live@acmucsd.org","$PASSWORD","admin"]
#+END_SRC
Once the Admin account is created, log in with it at =live.acmucsd.com= and get to the admin panel.
There you should:
- Change "regular color" to =#fc3b7d=
- Set branding image to the file located at =files/live/acmlive.png=
- Set registration policy to "Join by Invitation"
You may now also invite users at your leisure using the admin account or create
accounts using the above =docker exec= command. Administrator panel is recommended, however.