Kubernetes

Deploy FoxIDs in your Kubernetes (K8s) cluster or Docker Desktop with Kubernetes enabled.

This is a description of how to make a default deployment, log in for the first time and some considerations. It is expected that you will need to customize the yaml files to suit your needs, preferences and environment.

Pre requirements:

• You have a Kubernetes cluster or Docker Desktop with Kubernetes enabled.
• You have basic knowledge about Kubernetes.
• You have kubectl installer on your workstation.
• You have Helm installer on your workstation and your cluster.
  Install Helm on windows with this CMD command winget install Helm.Helm

This is a list of useful commands in the end of this description.

This deployment include:

• Two websites one for FoxIDs and one for the FoxIDs Control (Client and API) in two docker images foxids/foxids and foxids/foxids-control.
• The two websites is exposed on two different domains secured with automatically generated Let's Encrypt certificates.
• MongoDB is a NoSQL database and contains all data including tenants, environments and users. Deployed with the official MongoDB Docker image.
• Redis cache holds sequences (e.g., login and logout), data cache to improve performance and handle counters to secure authentication against various attacks. Deployed with the official Redis Docker image.
• Logs are written to stdout where the logs can be picked up by Kubernetes.

Optionally use PostgreSql instead of MongoDB and optionally opt out Redis and save cache data in the database (MongoDB or PostgreSql). Without a Redis cache you need to select None as data cache.

Deployment

The deployment is carried out in the described order.

Get ready

Clone the git repository or download as ZIP. The K8s yaml configuration files is in the ./Kubernetes folder.
Open a console and navigate to the ./Kubernetes folder.

Persistent volumes

You need persistent volumes for MongoDB and Redis.

If you are using Kubernetes in Docker Desktop you can create persistent volumes on the host file system - not recommended for production.

In a Kubernetes cluster use or create suitable persistent volumes and create two persistent volume claim with the names mongo-data-pvc for MongoDB and the name redis-data-pvc for Redis.

Kubernetes in Docker Desktop

Create persistent volume for MongoDB

kubectl apply -f k8s-mongo-pv-dockerdesktop.yaml

Create persistent volume claim for MongoDB

kubectl apply -f k8s-mongo-pvc-dockerdesktop.yaml

Create persistent volume for Redis

kubectl apply -f k8s-redis-pv-dockerdesktop.yaml

Create persistent volume claim for Redis

kubectl apply -f k8s-redis-pvc-dockerdesktop.yaml

Otherwise, you might be able to use dynamic storage provisioning with StorageClass.

Create persistent volume claim for Mongo

kubectl apply -f k8s-mongo-pvc-dynamic.yaml

Create persistent volume claim for Redis

kubectl apply -f k8s-redis-pvc-dynamic.yaml

Namespace

This guide generally uses the namespace foxids, consider changing the namespace to suit your kubernetes environment.

Create namespace

kubectl create namespace foxids

MongoDB

Change the username and password for MongoDB in k8s-mongo-secret.yaml. The username and password is base64 encoded.

You base64 encoded "the text" in a command prompt depending on you platform:

Windows

powershell "[convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(\"the text\"))"

Linux / Mac

echo -n "the text" | base64

Add the MongoDB secret

kubectl apply -f k8s-mongo-secret.yaml -n foxids

Create MongoDB
Optionally expose MongoDB on port 27017 by uncomment the LoadBalancer

kubectl apply -f k8s-mongo-deployment.yaml -n foxids

Add a ConfigMap for the MongoDB service

kubectl apply -f k8s-mongo-configmap.yaml -n foxids

Redis

Create Redis

kubectl apply -f k8s-redis-deployment.yaml -n foxids

Add a ConfigMap for the Redis service

kubectl apply -f k8s-redis-configmap.yaml -n foxids

FoxIDs websites

Domains
The two FoxIDs websites is configured to use two domains that you create and manage in your DNS. Configure the k8s-foxids-deployment.yaml file with your selected domains:

• The FoxIDs site domain https://id.itfoxtec.com (two places in the file) is change to your domain - id.my-domain.com
• The FoxIDs Control site domain https://control.itfoxtec.com is change to your domain - control.my-domain.com

Email provider
You can optionally configure a global email provider or later configure email providers per environment. FoxIDs supports sending emails with SendGrid and SMTP.

The global email provider is configured in the k8s-foxids-deployment.yaml file on the foxids container/pod in the env: section.
This example show how to add Outlook / Microsoft 365 with SMTP:

- name: "Settings__Smtp__FromEmail"
    value: "[email protected]"
- name: "Settings__Smtp__FromName" # Optional from name associated to the email address 
    value: "e.g, my company name"
- name: "Settings__Smtp__Host"
    value: "smtp.office365.com"
- name: "Settings__Smtp__Port"
    value: "587"
- name: "Settings__Smtp__Username"
    value: "[email protected]"
- name: "Settings__Smtp__Password"
    value: "xxxxxxx"

Deploy
Create the two FoxIDs websites

kubectl apply -f k8s-foxids-deployment.yaml -n foxids

The configuration require a Nginx controller. You can optionally change the configuration to use another controller.

Install Ingress-Nginx controller with two commands

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx --force-update
helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace

Optionally verify Ingress-Nginx installation

kubectl get pod -n ingress-nginx

If you try again in a few minutes you should get an EXTERNAL-IP

kubectl get svc -n ingress-nginx ingress-nginx-controller

DNS records to the two domains need to point to the installations IP address to enable the Let's Encrypt online validation.
The firewall needs to accept requests on port 80 and 443. Let's encrypt validates the domain ownership on port 80.

Optionally scale the Ingress-Nginx controller

kubectl scale deployment ingress-nginx-controller -n ingress-nginx --replicas=2

Install Cert-manager with two commands

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set crds.enabled=true

Optionally verify Cert-manager installation

kubectl get pods -n cert-manager

Add your email in the k8s-letsencrypt-issuer.yaml (two places) file.

Configure Let's Encrypt

kubectl apply -f k8s-letsencrypt-issuer.yaml -n foxids

The k8s-foxids-ingress-deployment.yaml file is configured with the domains:

• The FoxIDs site domain id.itfoxtec.com (two places in the file) is change to your domain - id.my-domain.com
• The FoxIDs Control site domain control.itfoxtec.com (two places in the file) is change to your domain - control.my-domain.com

Consider to start with Let's Encrypt in staging to avoid hitting the Let's Encrypt production rate limit (staging certificates is not trusted by the browser).
Optionally select to use stating or production in the k8s-foxids-ingress-deployment.yaml file, default configured for production.

Add ingress with certificate bound domains

kubectl apply -f k8s-foxids-ingress-deployment.yaml -n foxids

Optionally verify Ingress

kubectl get ingress -n foxids

Optionally verify certificate issuer

kubectl describe ClusterIssuer letsencrypt-production -n foxids
#staging 
# kubectl describe ClusterIssuer letsencrypt-staging -n foxids

Optionally check if the certificate is ready (READY should be True)

kubectl get certificate -n foxids

And optionally verify the certificate

kubectl describe certificate letsencrypt-production -n foxids
#staging 
# kubectl describe certificate letsencrypt-staging -n foxids

First login

Open your FoxIDs Control site domain in a browser. It should redirect to the FoxIDs site where you login with the default admin user [email protected] and password FirstAccess! (you are required to change the password on first login).
You are then redirected back to the FoxIDs Control site in the master tenant. You can add more tenants in the master tenant and e.g., configure admin users.

Then click on the main tenant and authenticate once again with the same default admin user email and password (the default admin user email and password is the same for both the master tenant and the main tenant, but it is two different users).
You are now logged into the main tenant and can start to configure your applications and authentication methods.

Seed data

The database is automatically seeded based on the configured domains. Therefor, you need to delete the database if the domains are changed.
To delete the data; You can either stop the database pod and delete the physical database folder or files. Or expose the database endpoint and open the database in MongoDB Compress (download MongoDB Compass Download (GUI)) and delete the database.
Thereafter, the FoxIDs Control pod needs to be restarted to initiate a new seed process.

Advanced option: The domains can also be changed by hand din the database.

Considerations

This section lists some deployment and security considerations.

Kubernetes Service Mesh
It is recommended to use a Kubernetes Service Mesh to achieve a zero-trust architecture. Where the internal traffic is secured with mutual TLS (mTLS) and encryption.

Namespace
This guide generally uses the namespace foxids, consider changing the namespace to suit your kubernetes environment.

Create namespace

kubectl create namespace test

List namespaces

kubectl get namespaces

Apply namespace on pod creation

kubectl apply -f xxx.yaml --namespace=test

Log
All logs from FoxIDs including errors, trace and events is written to stdout. Consider how to handle application logs and collect logs from the containers.

MongoDB Operator
Consider MongoDB Operator if you need multiple instances of MongoDB.

1. MongoDB Community Kubernetes Operator (free)
2. MongoDB Enterprise Kubernetes Operator

Redis multiple pods / cluster
Consider a scaled Redis setup if you need multiple instances of Redis.

• Redis master/replica setup in Kubernetes
• Redis on Kubernetes
• Redis Enterprise cluster on Kubernetes and architecture

Backup
Consider whether MongoDB data needs to be backed up and at what level, here are three possible solutions. It is considered less important to backup Redis.

1. Backup the persistent volume physical data store.
2. Backup with a Kubernetes Cron Job.
3. Backup is supported in MongoDB Enterprise Kubernetes Operator.

Update

FoxIDs is updated by updating each FoxIDs image to a new version, the two FoxIDs images is backwards compatible. First update the foxids/foxids image and then the foxids/foxids-control image.

It should likewise be possible to update the MongoDB image and Redis images with data in persistent volumes.

Useful commands

This is a list of commands which may be useful during deployment to view details and to make deployment changes.

Create pod

kubectl apply -f ks8-xxx.yaml

Tear down pod

kubectl delete -f ks8-xxx.yaml

List pods

kubectl get pods

Get pod description

kubectl describe pod xxx

Get pod logs

kubectl logs xxx

List deployments

kubectl get deployments

List services

kubectl get services

List secrets

kubectl get secrets

List persistent volumes

kubectl get pv

List persistent volume claims

kubectl get pvc

List ingress

kubectl get ingress

Sescribe ingress

kubectl describe ingress xxx