Kubernetes

Deploy FoxIDs in your Kubernetes (K8s) cluster or Docker Desktop with Kubernetes enabled.

This is a description of how to do a default deployment and log in for the first time as well as some considerations. It is expected that you will need to customise the yaml files to suit your needs, preferences and environment.

A FoxIDs installation is self-contained, has few external dependencies, and can achieve very high uptime with little effort. FoxIDs is updated by updating the two docker images www.foxids.com/foxids:x.x.x and www.foxids.com/foxidscontrol:x.x.x to a new version. New FoxIDs releases are backward compatible. Please review the release notes before updating. You can also find the current Docker image version listed there.

A valid licence is required to access the FoxIDs releases. From there, you can generate a secret to authenticate with the Docker image registry at www.foxids.com, which uses Basic authentication.

Prerequisites:

• You have a Kubernetes cluster or Docker Desktop with Kubernetes enabled.
• You have basic knowledge about Kubernetes.
• You have kubectl installed.

This is a list of useful commands at the end of this description.

This deployment includes:

• Two websites: one for FoxIDs and one for the FoxIDs Control (the admin Client and API) in two docker images www.foxids.com/foxids:x.x.x and www.foxids.com/foxidscontrol:x.x.x.
• The two websites are exposed on two different domains secured with automatically generated Let's Encrypt certificates.
• MongoDB is a NoSQL database and contains all data including tenants, environments and users. Deployed with the official MongoDB Docker image. You can optionally use your own PostgreSQL instance instead of MongoDB.
• By default, the cache is stored in the configured database. Redis is only needed if you run a high-throughput FoxIDs cluster with PostgreSQL. MongoDB has a built-in cache that performs well when the MongoDB instance has enough memory. The FoxIDs cache stores sequences (e.g., login and logout), handle counters to secure authentication against various attacks and data cache (Redis only) to improve performance. Redis is deployed with the official Redis Docker image.
• Logs are written to stdout where the logs can be picked up by Kubernetes. It is recommended to use OpenSearch for logging unless the FoxIDs installation is small. With OpenSearch, errors, events and traces can be viewed directly in the FoxIDs Control Client in addition to the Kubernetes logs, which makes the system much easier to administer.

Deployment

The deployment is carried out in the described order.

Get ready

Download Kubernetes yaml configuration files here and place them in a folder on your machine, e.g., C:\FoxIDs\Kubernetes.

Open a console and navigate to the Kubernetes folder.

Namespace

This guide generally uses the namespace foxids, consider changing the namespace to suit your Kubernetes environment.

Create namespace

kubectl create namespace foxids

Persistent volumes

You need persistent volumes for MongoDB, optionally OpenSearch and optionally Redis if you use Redis with PostgreSQL for a high-throughput FoxIDs cluster.

In a Kubernetes cluster use or create suitable persistent volumes and create a persistent volume claim with the name mongo-data-pvc for MongoDB, optionally one for OpenSearch with the name opensearch-data-pvc and optionally one for Redis with the name redis-data-pvc.

You might be able to use dynamic storage provisioning with StorageClass.

Create persistent volume claim for MongoDB

kubectl apply -f mongo-pvc-dynamic.yaml -n foxids

Optionally create persistent volume claim for Redis

kubectl apply -f redis-pvc-dynamic.yaml -n foxids

Optionally create persistent volume claim for OpenSearch

kubectl apply -f opensearch-pvc-dynamic.yaml -n foxids

Kubernetes in Docker Desktop

If you are using Kubernetes in Docker Desktop you can create persistent volumes on the host file system - not recommended for production.

Create persistent volume for MongoDB

kubectl apply -f mongo-pv-dockerdesktop.yaml

Create persistent volume claim for MongoDB

kubectl apply -f mongo-pvc-dockerdesktop.yaml -n foxids

Optionally create persistent volume for Redis

kubectl apply -f redis-pv-dockerdesktop.yaml

Optionally create persistent volume claim for Redis

kubectl apply -f redis-pvc-dockerdesktop.yaml -n foxids

Optionally create persistent volume for OpenSearch

kubectl apply -f opensearch-pv-dockerdesktop.yaml

Optionally create persistent volume claim for OpenSearch

kubectl apply -f opensearch-pvc-dockerdesktop.yaml -n foxids

MongoDB

Change the username and password for MongoDB in mongo-secret.yaml. The username and password are base64 encoded.

Base64 encode "the text" in a command prompt depending on your platform:

Windows

powershell "[convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(\"the text\"))"

Linux / Mac

echo -n "the text" | base64

Add the MongoDB secret

kubectl apply -f mongo-secret.yaml -n foxids

Create MongoDB
Optionally expose MongoDB on port 27017 by uncommenting the LoadBalancer

kubectl apply -f mongo-deployment.yaml -n foxids

Add a ConfigMap for the MongoDB service

kubectl apply -f mongo-configmap.yaml -n foxids

Optionally use PostgreSQL instead of MongoDB

Change the username value in postgres-username and password value in postgres-password to match your PostgreSQL instance in postgres-secret.yaml. The username and password are base64 encoded.

Add the PostgreSQL secret

kubectl apply -f postgres-secret.yaml -n foxids

Change the PostgreSQL database endpoint in postgres-db to match your PostgreSQL instance in postgres-configmap.yaml

Add a ConfigMap for the PostgreSQL service

kubectl apply -f postgres-configmap.yaml -n foxids

Optionally use Redis

Redis is only needed if you use PostgreSQL and run a high-throughput FoxIDs cluster. If you use MongoDB, MongoDB's built-in cache performs well when the MongoDB instance has enough memory.

Optionally create Redis

kubectl apply -f redis-deployment.yaml -n foxids

Optionally add a ConfigMap for the Redis service

kubectl apply -f redis-configmap.yaml -n foxids

Optionally use OpenSearch

OpenSearch is recommended for logging unless the FoxIDs installation is small. A small installation can run with Stdout logging only, where Kubernetes collects the logs. With OpenSearch, errors, events and traces can also be searched directly in the FoxIDs Control Client, which makes the system much easier to administer.

Change the password for OpenSearch in opensearch-secret.yaml. The password is base64 encoded.

Add the OpenSearch secret

kubectl apply -f opensearch-secret.yaml -n foxids

Create OpenSearch

kubectl apply -f opensearch-deployment.yaml -n foxids

Add a ConfigMap for the OpenSearch service

kubectl apply -f opensearch-configmap.yaml -n foxids

FoxIDs websites

Domains
The two FoxIDs websites are configured to use two domains that you create and manage in your DNS. Configure the foxids-deployment.yaml file with your selected domains:

• The FoxIDs site domain https://id.itfoxtec.com (two places in the file) is changed to your domain - id.my-domain.com
• The FoxIDs Control site domain https://control.itfoxtec.com is changed to your domain - control.my-domain.com

Email provider
You can optionally configure a global email provider or later configure email providers per environment. FoxIDs supports sending emails with SendGrid and SMTP. You can also configure SMS providers globally or per environment.

The global email provider is configured in the foxids-deployment.yaml file on the foxids container/pod in the env: section. This example shows how to add Outlook / Microsoft 365 with SMTP:

- name: "Settings__Smtp__FromEmail"
    value: "my@email-address.org"
- name: "Settings__Smtp__FromName" # Optional from name associated to the email address 
    value: "e.g, my company name"
- name: "Settings__Smtp__Host"
    value: "smtp.office365.com"
- name: "Settings__Smtp__Port"
    value: "587"
- name: "Settings__Smtp__Username"
    value: "my@email-address.com"
- name: "Settings__Smtp__Password"
    value: "xxxxxxx"

Important if you are using PostgreSQL
Change the database and cache configuration in foxids-deployment.yaml (two places in the file).

Select PostgreSQL as database instead of MongoDb

- name: "Settings__Options__DataStorage"
   # value: "MongoDb"
   value: "PostgreSql"  # PostgreSql database

Select PostgreSQL as cache instead of MongoDb unless you are using Redis

- name: "Settings__Options__Cache"
   # value: "MongoDb"
   value: "PostgreSql"  # Cache in PostgreSql database
   # value: "Redis"  # Cache in Redis 

Uncomment the PostgreSQL access configuration

- name: POSTGRES_USERNAME
    valueFrom:
    secretKeyRef:
        name: postgres-secret
        key: postgres-username
- name: POSTGRES_PASSWORD
    valueFrom:
    secretKeyRef:
        name: postgres-secret
        key: postgres-password
- name: POSTGRES_SERVER
    valueFrom:
    configMapKeyRef:
        name: postgres-configmap
        key: database_url
- name: "Settings__PostgreSql__ConnectionString"
    value: "Host=$(POSTGRES_SERVER);Username=$(POSTGRES_USERNAME);Password=$(POSTGRES_PASSWORD);Database=FoxIDs"

Important if you are using Redis
Change the cache configuration in foxids-deployment.yaml (two places in the file).

Select Redis as cache instead of MongoDb

- name: "Settings__Options__Cache"
   # value: "MongoDb"
   # value: "PostgreSql"  # Cache in PostgreSql database
   value: "Redis"  # Cache in Redis

Optionally cache database data in Redis

- name: "Settings__Options__DataCache"
   # value: "None"
   value: "Default"  # Cache DB data in Redis

Uncomment the Redis access configuration

- name: REDIS_SERVER
    valueFrom:
    configMapKeyRef:
        name: redis-configmap
        key: database_url
- name: "Settings__RedisCache__ConnectionString"
    value: "$(REDIS_SERVER):6379"

Important if you are using OpenSearch Change the log configuration in foxids-deployment.yaml (two places in the file).

Select OpenSearch logging instead of Stdout

- name: "Settings__Options__Log"
   # value: "Stdout"
   value: "OpenSearchAndStdoutErrors"

Uncomment the OpenSearch access configuration

- name: OPENSEARCH_PASSWORD
    valueFrom:
    secretKeyRef:
        name: opensearch-secret
        key: opensearch-password
- name: OPENSEARCH_SERVER
    valueFrom:
    configMapKeyRef:
        name: opensearch-configmap
        key: opensearch_url
- name: "Settings__OpenSearch__Nodes__0"
    value: "https://admin:$(OPENSEARCH_PASSWORD)@$(OPENSEARCH_SERVER):9200"
- name: "Settings__OpenSearch__LogLifetime"
    value: "Max180Days"
- name: "Settings__OpenSearch__AllowInsecureCertificates"
    value: "true"

OpenSearchAndStdoutErrors sends logs to OpenSearch and writes errors to stdout, so Kubernetes still receives error logs. AllowInsecureCertificates is set to true because the default OpenSearch container uses a self-signed certificate.

Deploy
Set the username to your customer ID and the password to your pull secret for the FoxIDs Docker registry. Then create the registry secret.

kubectl create secret docker-registry foxids-registry-secret --docker-server=www.foxids.com --docker-username=<customer-id> --docker-password=<pull-secret> -n foxids

Create the two FoxIDs websites

kubectl apply -f foxids-deployment.yaml -n foxids

The configuration requires an Nginx controller. You can optionally change the configuration to use another controller.

Prerequisites:

• You have Helm installed. Install Helm on Windows with this CMD command winget install Helm.Helm

Install Ingress-Nginx controller with two commands

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx --force-update
helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace

Optionally verify Ingress-Nginx installation

kubectl get pod -n ingress-nginx

If you try again in a few minutes you should get an EXTERNAL-IP

kubectl get svc -n ingress-nginx ingress-nginx-controller

DNS records to the two domains need to point to the installations IP address to enable the Let's Encrypt online validation.
The firewall needs to accept requests on port 80 and 443. Let's encrypt validates the domain ownership on port 80.

Optionally scale the Ingress-Nginx controller

kubectl scale deployment ingress-nginx-controller -n ingress-nginx --replicas=2

Install Cert-manager with two commands

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set crds.enabled=true

Optionally verify Cert-manager installation

kubectl get pods -n cert-manager

Add your email in the letsencrypt-issuer.yaml (two places) file.

Configure Let's Encrypt

kubectl apply -f letsencrypt-issuer.yaml -n foxids

The foxids-ingress-deployment.yaml file is configured with the domains:

• The FoxIDs site domain id.itfoxtec.com (two places in the file) is changed to your domain - id.my-domain.com
• The FoxIDs Control site domain control.itfoxtec.com (two places in the file) is changed to your domain - control.my-domain.com

Consider starting with Let's Encrypt in staging to avoid hitting the Let's Encrypt production rate limit (staging certificates are not trusted by the browser). Optionally select staging or production in the foxids-ingress-deployment.yaml file, default configured for production.

Add ingress with certificate bound domains

kubectl apply -f foxids-ingress-deployment.yaml -n foxids

Important! Ingress is installed with the annotations nginx.ingress.kubernetes.io/proxy-buffers-number: "4" and nginx.ingress.kubernetes.io/proxy-buffer-size: "32k" to support SAML 2.0 where HTTP responses can be quite large.

Optionally verify Ingress

kubectl get ingress -n foxids

Optionally verify certificate issuer

kubectl describe ClusterIssuer letsencrypt-production -n foxids
#staging 
# kubectl describe ClusterIssuer letsencrypt-staging -n foxids

Optionally check if the certificate is ready (READY should be True)

kubectl get certificate -n foxids

And optionally verify the certificate

kubectl describe certificate letsencrypt-production -n foxids
#staging 
# kubectl describe certificate letsencrypt-staging -n foxids

First login

Open your FoxIDs Control site domain in a browser. It should redirect to the FoxIDs site where you login with the default admin user admin@foxids.com and password FirstAccess! (you are required to change the password on first login).
You are then redirected back to the FoxIDs Control site in the master tenant. You can add more admin users in the master tenant.

Then click on the main tenant and authenticate once again with the same default admin user admin@foxids.com and password FirstAccess! (again, you are required to change the password).

The default admin user and password are the same for both the master tenant and the main tenant, but it is two different users.

You are now logged into the main tenant and can start to configure your applications and authentication methods.

Seed data

The database is automatically seeded based on the configured domains. Therefore, you need to delete the database if the domains are changed. To delete the data; You can either stop the database pod and delete the physical database folder or files. Or expose the database endpoint and open the database in MongoDB Compass (download MongoDB Compass Download (GUI)) and delete the database. Thereafter, the FoxIDs Control pod needs to be restarted to initiate a new seed process.

Advanced option: The domains can also be changed by hand in the database.

Considerations

This section lists some deployment and security considerations.

Kubernetes Service Mesh
It is recommended to use a Kubernetes Service Mesh to achieve a zero-trust architecture. Where the internal traffic is secured with mutual TLS (mTLS) and encryption.

Internal PKI / root CA
If internal traffic is terminated and re-encrypted by a proxy or service mesh with an internal root CA, configure the FoxIDs containers to trust that CA. See Kubernetes internal CA.

Namespace
This guide generally uses the namespace foxids, consider changing the namespace to suit your kubernetes environment.

Create namespace

kubectl create namespace test

List namespaces

kubectl get namespaces

Apply namespace on pod creation

kubectl apply -f xxx.yaml --namespace=test

Log
All logs from FoxIDs including errors, trace and events are written to stdout. Consider how to handle application logs and collect logs from the containers.

Rate Limiting
Configure limits on connections and transmission rates in foxids-ingress-deployment.yaml.

MongoDB Operator
Consider MongoDB Operator if you need multiple instances of MongoDB.

1. MongoDB Community Kubernetes Operator (free)
2. MongoDB Enterprise Kubernetes Operator

Redis multiple pods / cluster
Consider a scaled Redis setup if you need multiple instances of Redis.

• Redis master/replica setup in Kubernetes
• Redis on Kubernetes
• Redis Enterprise cluster on Kubernetes and architecture

Backup
Consider whether MongoDB data needs to be backed up and at what level, here are three possible solutions. It is considered less important to backup Redis.

1. Backup the persistent volume physical data store.
2. Backup with a Kubernetes Cron Job.
3. Backup is supported in MongoDB Enterprise Kubernetes Operator.

Update

FoxIDs is updated by changing the image version tags in foxids-deployment.yaml for both FoxIDs containers:

image: www.foxids.com/foxids:x.x.x
image: www.foxids.com/foxidscontrol:x.x.x

Review the release notes before updating. The release notes also list the current Docker image version. New FoxIDs releases are backward compatible.

Apply the updated deployment file:

kubectl apply -f foxids-deployment.yaml -n foxids

Kubernetes then rolls the two deployments to the new image versions.

It should likewise be possible to update the MongoDB, Redis and OpenSearch images with data in persistent volumes.

Useful commands

This is a list of commands which may be useful during deployment to view details and to make deployment changes.

Create pod

kubectl apply -f xxx.yaml -n foxids

Tear down pod

kubectl delete -f xxx.yaml -n foxids

List pods

kubectl get pods -n foxids

Get pod description

kubectl describe pod xxx -n foxids

Get pod logs

kubectl logs xxx -n foxids

List deployments

kubectl get deployments -n foxids

List services

kubectl get services -n foxids

List secrets

kubectl get secrets -n foxids

Show FoxIDs docker registry secret

kubectl get secret foxids-registry-secret -n foxids --output=yaml

List persistent volumes

kubectl get pv

List persistent volume claims

kubectl get pvc

List ingress

kubectl get ingress

Describe ingress

kubectl describe ingress xxx