Deploy Production-ready Aidbox to Kubernetes

Production-ready infrastructure
Key infrastructure elements:
​Cluster configuration — Node pool and tooling
​Database — Cloud or self-managed database
​Aidbox — Aidbox installation
​Logging — Сollect application and cluster logs
​Monitoring — Сollect, alert, and visualize cluster and application metrics
​Security — Vulnerability scanning and policy management
Cluster configuration and tooling
Recommended Kubernetes cluster configuration:
Small and medium workloads — 3 nodes X 4 VCPU 16 GB RAM
Huge workloads — 3 nodes X 8 VCPU X 64 GB RAM
Toolkit required for development and deployment:
​AWS, GCP, AZURE - Cloud provider CLI and SDK. Depends on your cloud provider:
​Kubectl - connection and cluster management
​Helm - Kubernetes package manager
​Lens - Kubernetes IDE
Optional - Development and Delivery tooling:
​Terraform - Infrastructure automation tool
​Grafana tanka - configuration utility for your Kubernetes
​Argo CD - GitOps delivery and management
​Flux - set of continuous and progressive delivery solutions for Kubernetes
Database
Managed solution
Aidbox supports all popular managed Postgresql databases. Supported versions - 13 and higher. See more details in this article — Run Aidbox on managed PostgreSQL.
​AWS RDS Aurora​
​GCP Cloud SQL for PostgreSQL​
​Azure Database for PostgreSQL​
Self-managed solution
For a self-managed solution, we recommend use AidboxDB image . This image contains all required extensions, backup tool, and pre-build replication support. Read more information in the documentation — AidboxDB.
First step — create volume
Persistent Volume
1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: db-master-data
5
  namespace: prod
6
spec:
7
  accessModes:
8
    - ReadWriteOnce
9
  resources:
10
    requests:
11
      storage: 300Gi
12
  # depend on your cloud provider. Use SSD volumes
13
  storageClassName: managed-premium
Next - create all required configs, like postgresql.conf, required container parameters and credentials.
postgresql.conf
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: db-pg-config
5
  namespace: prod
6
data:
7
  postgres.conf: |-
8
    listen_addresses = '*'
9
    shared_buffers = '2GB'
10
    max_wal_size = '4GB'
11
    pg_stat_statements.max = 500
12
    pg_stat_statements.save = false
13
    pg_stat_statements.track = top
14
    pg_stat_statements.track_utility = true
15
    shared_preload_libraries = 'pg_stat_statements'
16
    track_io_timing = on
17
    wal_level = logical
18
    wal_log_hints = on
19
    archive_command = 'wal-g wal-push %p'
20
    restore_command = 'wal-g wal-fetch %f %p'
db-config Configmap
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: db-config
5
  namespace: prod
6
data:
7
  PGDATA: /data/pg
8
  POSTGRES_DB: postgres
db-secret Secret
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: db-secret
5
  namespace: prod
6
type: Opaque
7
data:
8
  POSTGRES_PASSWORD: cG9zdGdyZXM=
9
  POSTGRES_USER: cG9zdGdyZXM=
Now we can create a database StatefulSet
Db Master StatefulSet
1
apiVersion: apps/v1
2
kind: StatefulSet
3
metadata:
4
  name: prod-db-master
5
  namespace: prod
6
spec:
7
  replicas: 1
8
  serviceName: db
9
  selector:
10
    matchLabels:
11
      service: db
12
  template:
13
    metadata:
14
      labels:
15
        service: db
16
    spec:
17
      volumes:
18
        - name: db-pg-config
19
          configMap:
20
            name: db-pg-config
21
            defaultMode: 420
22
        - name: db-dshm
23
          emptyDir:
24
            medium: Memory
25
        - name: db-data
26
          persistentVolumeClaim:
27
            claimName: db-master-data
28
      containers:
29
        - name: main
30
          image: healthsamurai/aidboxdb:14.2
31
          ports:
32
            - containerPort: 5432
33
              protocol: TCP
34
          envFrom:
35
            - configMapRef:
36
                name: db-config
37
            - secretRef:
38
                name: db-secret
39
          volumeMounts:
40
            - name: db-pg-config
41
              mountPath: /etc/configs
42
            - name: db-dshm
43
              mountPath: /dev/shm
44
            - name: db-data
45
              mountPath: /data
46
              subPath: pg
Create master database service
Database Service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: db
5
  namespace: prod
6
spec:
7
  ports:
8
    - protocol: TCP
9
      port: 5432
10
      targetPort: 5432
11
  selector:
12
    service: db
13
​
Replica installation contains all the same steps but required additional configuration
Replica DB config
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: db-replica
5
  namespace: prod
6
data:
7
  PG_ROLE: replica
8
  PG_MASTER_HOST: db-master
9
  PG_REPLICA: streaming_replica_streaming
10
  PGDATA: /data/pg
11
  POSTGRES_DB: postgres
For backups and WAL archivation we are recommended cloud-native solution WAL-G. You can find full information about configuration and usage on documentation page.
​Configure storage access — WAL-G can store backups in S3, Google Cloud Storage, Azure, or a local file system.
Recommended backup policy — Full backup every week, incremental backup every day.
Alternative solutions
A set of tools to perform HA PostgreSQL with fail and switchover, automated backups.
​Patroni — A Template for PostgreSQL HA with ZooKeeper, ETCD or Consul.
​Postgres operator — The Postgres Operator delivers an easy to run HA PostgreSQL clusters on Kubernetes.
Aidbox
First of all, you need to get Aidbox license on Aidbox user portal​
Create ConfigMap with all required config and database connection
This ConfigMap example uses our default Aidbox Configuration Project Template. It's recommended to clone this template and bind your Aidbox installation with it.
Aidbox ConfigMap
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
data:
7
  AIDBOX_BASE_URL: https://my.box.url
8
  AIDBOX_BOX_ID: aidbox
9
  AIDBOX_FHIR_VERSION: 4.0.1
10
  AIDBOX_PORT: '8080'
11
  AIDBOX_STDOUT_PRETTY: all
12
  BOX_INSTANCE_NAME: aidbox
13
  BOX_METRICS_PORT: '8765'
14
  PGDATABASE: aidbox
15
  PGHOST: db.prod.svc.cluster.local   # database address
16
  PGPORT: '5432'                      # database port
17
  BOX_PROJECT_GIT_URL: "https://github.com/Aidbox/aidbox-project-template.git"
18
  BOX_PROJECT_GIT_PROTOCOL: "https"
19
  BOX_PROJECT_GIT_TARGET__PATH: "/tmp/aidbox-project"
20
  BOX_PROJECT_GIT_CHECKOUT: "main"
21
  AIDBOX_ZEN_ENTRYPOINT: main/box
22
  AIDBOX_DEV_MODE: "false"
23
  AIDBOX_ZEN_DEV_MODE: "false"
Aidbox Secret
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
data:
7
  AIDBOX_ADMIN_PASSWORD: <admin_password>
8
  AIDBOX_CLIENT_SECRET: <root_client_password>
9
  AIDBOX_LICENSE: <JWT-LICENSE>    # JWT license from the Aidbox user portal
10
  PGUSER: <db_user>                # database username
11
  PGPASSWORD: <db_password>        # database password
12
  
13
  BOX_AUTH_KEYS_SECRET: <random_string_auth_secret>
14
  BOX_AUTH_KEYS_PRIVATE: <rsa_private_key> 
15
  BOX_AUTH_KEYS_PUBLIC: <rsa_public_key> 
16
  
17
  # or just use our samples for non-production installation
18
  # BOX_AUTH_KEYS_SECRET: "auth-key-secret"
19
  # BOX_AUTH_KEYS_PRIVATE: "-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQCRLKv0n9HPsajw3wcDH1k5DUSPPdKjxqp8h4OZKiG3wGEFYXi9\nfxBbpkQXjxGEmORi8UR4aM41kX8dd4SdMRGS1VX2AMgLEAFq354MpGBPIeJyv00y\nqV6wW0HT58+Nh+xdridDFSHkkplJFjDuQbYjfQzbSNECA31ME/GI9rGomQIDAQAB\nAoGAEYGytFecCnjtC6wHiVK71JeTIZd12fJsj4MbhWpJYeJxCMAz+l0S7MxweGtU\nNFpoKz7XUBJqcJcMvlHSBA89ZDobp3HS0R8ZDcdxossNRio3Ix1bRG7Pxnhs3R/T\nsOxlrQSgnSbg1k6M5iVSZt1ptCwch+ZLG37tD3ZvdAN0LCECQQC0IFiPJJEPauUi\neKmW4oUgBvOUVA93EqnBiv9lzk7UxrPgusFqnY02qJouDNvXXso6+FM8u9DNxSvw\nHPIuqJvhAkEAzlNYaJzoInkCS5PYTGg2f1GqRih9WHj8NUukfgbO61xT9QscM6An\n+RF8dfshU2zuaQFLTBPWrS0Nk0ZOxLFjuQJAZ4gz/sqwyiDR5RdfuscmZ3s3ZClQ\n3ksO4ZzoIXcMnoY7e888PvCh6ynLvO5NKiRkrrJu/XiikrNjBtdMaH8nYQJADkCF\nl9xW0KLJPM0+oLCGKy9J8sSzO9xHl6rc9vOjcXCUQBX/YbWLbVH+5ett9uRMZ6Z2\nPBAWwSmeiXDO2hliyQJBAI/7Gtzf1Z2O5pDgNMLkKcyX4BqsHFKFSD5Btb/zReEq\nTsr6vTvzucjJcS8843vgyhIUDtW2cu7G9BGxSfsZNCw=\n-----END RSA PRIVATE KEY-----\n"
20
  # BOX_AUTH_KEYS_PUBLIC: "-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCRLKv0n9HPsajw3wcDH1k5DUSP\nPdKjxqp8h4OZKiG3wGEFYXi9fxBbpkQXjxGEmORi8UR4aM41kX8dd4SdMRGS1VX2\nAMgLEAFq354MpGBPIeJyv00yqV6wW0HT58+Nh+xdridDFSHkkplJFjDuQbYjfQzb\nSNECA31ME/GI9rGomQIDAQAB\n-----END PUBLIC KEY-----\n"
Aidbox Deployment
Aidbox Deployment
1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
spec:
7
  replicas: 2
8
  selector:
9
    matchLabels:
10
      service: aidbox
11
  template:
12
    metadata:
13
      labels:
14
        service: aidbox
15
    spec:
16
      containers:
17
        - name: main
18
          image: healthsamurai/aidboxone:latest
19
          ports:
20
            - containerPort: 8080
21
              protocol: TCP
22
            - containerPort: 8765
23
              protocol: TCP
24
          envFrom:
25
            - configMapRef:
26
                name: aidbox
27
            - secretRef:
28
                name: aidbox
29
          livenessProbe:
30
            httpGet:
31
              path: /health
32
              port: 8080
33
              scheme: HTTP
34
            initialDelaySeconds: 20
35
            timeoutSeconds: 10
36
            periodSeconds: 10
37
            successThreshold: 1
38
            failureThreshold: 12
39
          readinessProbe:
40
            httpGet:
41
              path: /health
42
              port: 8080
43
              scheme: HTTP
44
            initialDelaySeconds: 20
45
            timeoutSeconds: 10
46
            periodSeconds: 10
47
            successThreshold: 1
48
            failureThreshold: 6
49
          startupProbe:
50
            httpGet:
51
              path: /health
52
              port: 8080
53
              scheme: HTTP
54
            initialDelaySeconds: 20
55
            timeoutSeconds: 5
56
            periodSeconds:  5
57
            successThreshold: 1
58
            failureThreshold: 4
59
​
All additional information about HA Aidbox configuration you can find in this article — HA Aidbox​
To verify that Aidbox started correctly you can check the logs:
kubectl logs -f <aidbox-pod-name>
Create the Aidbox k8s service
Aidbox service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
spec:
7
  ports:
8
    - protocol: TCP
9
      port: 80
10
      targetPort: 8080
11
  selector:
12
    service: aidbox
13
​
Ingress
A Cluster must have an ingress controller Installed.
Our recommendation is to use kubernetes Ingress NGINX Controller. As an alternative, you can use Traefic.
More additional information about Ingress in k8s can be found in this documentation — Kubernetes Service Networking​
Ingress NGINX controller
Ingress-nginx — is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.
Install Ingress NGINX
helm upgrade \
  --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace
CertManager
To provide a secure HTTPS connection you can use paid SSL certificates, issued for your domain, or use LetsEncrypt-issued certificates. In case of using LetsEcrypt, we recommend install and configure Cert Manager Operator
Install Cert Manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.10.0 \       # Or latest available version
  --set installCRDs=true
Configure Cluster Issuer:
1
apiVersion: cert-manager.io/v1
2
kind: ClusterIssuer
3
metadata:
4
  name: letsencrypt
5
spec:
6
  acme:
7
    email: hello@my-domain.com
8
    preferredChain: ''
9
    privateKeySecretRef:
10
      name: issuer-key
11
    server: https://acme-v02.api.letsencrypt.org/directory
12
    solvers:
13
      - http01:
14
          ingress:
15
            class: nginx  # Ingress class name
If you use Multibox image and want to use cert manger — you should configure DNS01 authorization to provide wildcard certificates
​https://letsencrypt.org/docs/challenge-types/#dns-01-challenge​
Ingress resource
Now you can create k8s Ingress for Aidbox deployment
Ingress
1
apiVersion: networking.k8s.io/v1
2
kind: Ingress
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
  annotations:
7
    acme.cert-manager.io/http01-ingress-class: nginx
8
    cert-manager.io/cluster-issuer: letsencrypt
9
    kubernetes.io/ingress.class: nginx
10
spec:
11
  tls:
12
    - hosts:
13
        - my.box.url
14
      secretName: aidbox-tls
15
  rules:
16
    - host: my.box.url
17
      http:
18
        paths:
19
          - path: /
20
            pathType: ImplementationSpecific
21
            backend:
22
              service:
23
                name: aidbox
24
                port:
25
                  number: 80
Now you can test ingress
curl https://my.box.url
Logging
General logging & audit information can be found in this article — Logging & Audit​
Aidbox supports integration with the following systems:
ElasticSearch — Elastic Logs and Monitoring Integration​
Loki — Grafana Loki Log management integration​
DataDog — Datadog Log management integration​
ElasticSearch integration
You can install ECK using official guide.​
Configure Aidbox and ES integration
Aidbox ConfigMap
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
data:
7
  ...
8
  AIDBOX_ES_URL = http://es-service.es-ns.svc.cluster.local
9
  AIDBOX_ES_AUTH = <user>:<password>
10
  ...
DataDog integration
Aidbox ConfigMap
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: aidbox
5
  namespace: prod
6
data:
7
  ...
8
  AIDBOX_DD_API_KEY: <Datadog API Key>
9
  ...
Monitoring
For monitoring our recommendation is to use kube prometheus stack​
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
Create Aidbox metrics service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: aidbox-metrics
5
  namespace: prod
6
  labels:
7
    operated: prometheus
8
spec:
9
  ports:
10
    - protocol: TCP
11
      port: 80
12
      targetPort: 8765
13
  selector:
14
    service: aidbox
Create ServiceMonitor config for scrapping metrics data
ServiceMonitor
1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
  labels:
5
    app.kubernetes.io/component: metrics
6
    release: kube-prometheus
7
    serviceMonitorSelector: aidbox
8
  name: aidbox
9
  namespace: kube-prometheus
10
spec:
11
  endpoints:
12
    - honorLabels: true
13
      interval: 10s
14
      path: /metrics
15
      targetPort: 8765
16
    - honorLabels: true
17
      interval: 60s
18
      path: /metrics/minutes
19
      targetPort: 8765
20
    - honorLabels: true
21
      interval: 10m
22
      path: /metrics/hours
23
      targetPort: 8765
24
  namespaceSelector:
25
    any: true
26
  selector:
27
    matchLabels:
28
      operated: prometheus
Or you can directly specify the Prometheus scrapers configuration
global:
  external_labels:
    monitor: 'aidbox'
scrape_configs:
  - job_name: aidbox
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
​
  - job_name: aidbox-minutes
    scrape_interval: 30s
    metrics_path: /metrics/minutes
    static_configs:
      - targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
​
  - job_name: aidbox-hours
    scrape_interval: 1m
    scrape_timeout: 30s                     
    metrics_path: /metrics/hours
    static_configs:
      - targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
Alternative solutions
​VictoriaMetrics — High Performance Open Source Time Series Database.
​Thanos — highly available Prometheus setup with long term storage capabilities.
​Grafana Mimir — highly available, multi-tenant, long-term storage for Prometheus.
Export the Aidbox Grafana dashboard
Aidbox metrics has integration with Grafana, which can generate dashboards and upload them to Grafana — Grafana Integration​
Additional monitoring
System monitoring:
​node exporter — Prometheus exporter for hardware and OS metrics exposed by *NIX kernels
​kube state metrics — is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects
​cadvisor — container usage metrics
PostgreSQL monitoring:
​pg_exporter — Prometheus exporter for PostgreSQL server metrics
Alerting
​Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service.
Alert rules
Alert for long-running HTTP queries with P99 > 5s in 5m interval
1
alert: SlowRequests
2
for: 5m
3
expr: histogram_quantile(0.99, sum (rate(aidbox_http_request_duration_seconds_bucket[5m])) by (le, route, instance)) > 5
4
labels: {severity: ticket}
5
annotations:
6
  title: Long HTTP query execution
7
  metric: '{{ $labels.route }}'
8
  value: '{{ $value | printf "%.2f" }}'
Alert delivery
Alert manager template for Telegram
1
global:
2
  resolve_timeout: 5m
3
  telegram_api_url: 'https://api.telegram.org/'
4
route:
5
  group_by: [alertname instance]
6
  # Default receiver
7
  receiver: <my-ops-chat>
8
  routes:
9
  # Mute watchdog alert
10
  - receiver: empty
11
    match: {alertname: Watchdog}
12
receivers:
13
- name: empty
14
- name: <my-ops-chat>
15
  telegram_configs:
16
  - chat_id: <chat-id>
17
    api_url: https://api.telegram.org
18
    parse_mode: HTML
19
    message: |-
20
      <b>[{{ .CommonLabels.instance }}] {{ .CommonLabels.alertname }}</b>
21
      {{ .CommonAnnotations.title }}
22
      {{ range .Alerts }}{{ .Annotations.metric }}: {{ .Annotations.value }}
23
      {{ end }}
24
    bot_token: <bot-token>
25
​
All other integrations you can find on AlertManager documentation page.​
Additional tools
Embedded Grafana alerts
Grafana OnCall
Security
Vulnerability and security scanners:
​Trivy operator — Kubernetes-native security toolkit.
​Trivy operator Lens extension — UI extension for Lens which provides visibility into Trivy reports
Kubernetes Policy Management:
​Kyverno OR Gatekeeper — Kubernetes policy management
Advanced:
​Datree — k8s resources linter
Run Aidbox on Kubernetes
Highly Available Aidbox
Last modified 26d ago