Deploy Production-ready Aidbox to Kubernetes
Key infrastructure elements:
Recommended Kubernetes cluster configuration:
- Small and medium workloads — 3 nodes X 4 VCPU 16 GB RAM
- Huge workloads — 3 nodes X 8 VCPU X 64 GB RAM
Toolkit required for development and deployment:
Optional - Development and Delivery tooling:
Aidbox supports all popular managed Postgresql databases. Supported versions - 13 and higher. See more details in this article — Run Aidbox on managed PostgreSQL.
For a self-managed solution, we recommend use AidboxDB image . This image contains all required extensions, backup tool, and pre-build replication support. Read more information in the documentation — AidboxDB.
First step — create volume
Persistent Volume
1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
name: db-master-data
5
namespace: prod
6
spec:
7
accessModes:
8
- ReadWriteOnce
9
resources:
10
requests:
11
storage: 300Gi
12
# depend on your cloud provider. Use SSD volumes
13
storageClassName: managed-premium
Next - create all required configs, like
postgresql.conf, required container parameters and credentials.postgresql.conf
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
name: db-pg-config
5
namespace: prod
6
data:
7
postgres.conf: |-
8
listen_addresses = '*'
9
shared_buffers = '2GB'
10
max_wal_size = '4GB'
11
pg_stat_statements.max = 500
12
pg_stat_statements.save = false
13
pg_stat_statements.track = top
14
pg_stat_statements.track_utility = true
15
shared_preload_libraries = 'pg_stat_statements'
16
track_io_timing = on
17
wal_level = logical
18
wal_log_hints = on
19
archive_command = 'wal-g wal-push %p'
20
restore_command = 'wal-g wal-fetch %f %p'
db-config Configmap
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
name: db-config
5
namespace: prod
6
data:
7
PGDATA: /data/pg
8
POSTGRES_DB: postgres
db-secret Secret
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
name: db-secret
5
namespace: prod
6
type: Opaque
7
data:
8
POSTGRES_PASSWORD: cG9zdGdyZXM=
9
POSTGRES_USER: cG9zdGdyZXM=
Now we can create a database
StatefulSetDb Master StatefulSet
1
apiVersion: apps/v1
2
kind: StatefulSet
3
metadata:
4
name: prod-db-master
5
namespace: prod
6
spec:
7
replicas: 1
8
serviceName: db
9
selector:
10
matchLabels:
11
service: db
12
template:
13
metadata:
14
labels:
15
service: db
16
spec:
17
volumes:
18
- name: db-pg-config
19
configMap:
20
name: db-pg-config
21
defaultMode: 420
22
- name: db-dshm
23
emptyDir:
24
medium: Memory
25
- name: db-data
26
persistentVolumeClaim:
27
claimName: db-master-data
28
containers:
29
- name: main
30
image: healthsamurai/aidboxdb:14.2
31
ports:
32
- containerPort: 5432
33
protocol: TCP
34
envFrom:
35
- configMapRef:
36
name: db-config
37
- secretRef:
38
name: db-secret
39
volumeMounts:
40
- name: db-pg-config
41
mountPath: /etc/configs
42
- name: db-dshm
43
mountPath: /dev/shm
44
- name: db-data
45
mountPath: /data
46
subPath: pg
Create master database service
Database Service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: db
5
namespace: prod
6
spec:
7
ports:
8
- protocol: TCP
9
port: 5432
10
targetPort: 5432
11
selector:
12
service: db
13
Replica installation contains all the same steps but required additional configuration
Replica DB config
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
name: db-replica
5
namespace: prod
6
data:
7
PG_ROLE: replica
8
PG_MASTER_HOST: db-master
9
PG_REPLICA: streaming_replica_streaming
10
PGDATA: /data/pg
11
POSTGRES_DB: postgres
For backups and WAL archivation we are recommended cloud-native solution WAL-G. You can find full information about configuration and usage on documentation page.
- Configure storage access — WAL-G can store backups in S3, Google Cloud Storage, Azure, or a local file system.
- Recommended backup policy — Full backup every week, incremental backup every day.
A set of tools to perform HA PostgreSQL with fail and switchover, automated backups.
- Postgres operator — The Postgres Operator delivers an easy to run HA PostgreSQL clusters on Kubernetes.
Create ConfigMap with all required config and database connection
Aidbox ConfigMap
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
name: aidbox
5
namespace: prod
6
data:
7
AIDBOX_BASE_URL: https://my.box.url
8
AIDBOX_BOX_ID: aidbox
9
AIDBOX_FHIR_VERSION: 4.0.1
10
AIDBOX_PORT: '8080'
11
AIDBOX_STDOUT_PRETTY: all
12
BOX_INSTANCE_NAME: aidbox
13
BOX_METRICS_PORT: '8765'
14
PGDATABASE: aidbox
15
PGHOST: db.prod.svc.cluster.local # database address
16
PGPORT: '5432' # database port
Aidbox Secret
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
name: aidbox
5
namespace: prod
6
data:
7
AIDBOX_ADMIN_ID: <admin_login>
8
AIDBOX_ADMIN_PASSWORD: <admin_password>
9
AIDBOX_CLIENT_ID: <root_client_id>
10
AIDBOX_CLIENT_SECRET: <root_client_password>
11
AIDBOX_LICENSE: <JWT-LICENSE> # JWT license from aidbox user portal
12
PGPASSWORD: <db_password> # database password
13
PGUSER: <db_user> # database username
Aidbox Deployment
Aidbox Deployment
1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: aidbox
5
namespace: prod
6
spec:
7
replicas: 2
8
selector:
9
matchLabels:
10
service: aidbox
11
template:
12
metadata:
13
labels:
14
service: aidbox
15
spec:
16
containers:
17
- name: main
18
image: healthsamurai/aidboxone:latest
19
ports:
20
- containerPort: 8080
21
protocol: TCP
22
- containerPort: 8765
23
protocol: TCP
24
envFrom:
25
- configMapRef:
26
name: aidbox
27
- secretRef:
28
name: aidbox
29
livenessProbe:
30
httpGet:
31
path: /health
32
port: 8080
33
scheme: HTTP
34
initialDelaySeconds: 20
35
timeoutSeconds: 10
36
periodSeconds: 10
37
successThreshold: 1
38
failureThreshold: 12
39
readinessProbe:
40
httpGet:
41
path: /health
42
port: 8080
43
scheme: HTTP
44
initialDelaySeconds: 20
45
timeoutSeconds: 10
46
periodSeconds: 10
47
successThreshold: 1
48
failureThreshold: 6
49
startupProbe:
50
httpGet:
51
path: /health
52
port: 8080
53
scheme: HTTP
54
initialDelaySeconds: 20
55
timeoutSeconds: 5
56
periodSeconds: 5
57
successThreshold: 1
58
failureThreshold: 4
59
To verify that Aidbox started correctly you can check the logs:
kubectl logs -f <aidbox-pod-name>
Create the Aidbox k8s service
Aidbox service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: aidbox
5
namespace: prod
6
spec:
7
ports:
8
- protocol: TCP
9
port: 80
10
targetPort: 8080
11
selector:
12
service: aidbox
13
A Cluster must have an ingress controller Installed. Our recommendation is to use kubernetes Ingress NGINX Controller. As an alternative, you can use Traefic. More additional information about Ingress in k8s you can found in this documentation — Kubernetes Service Networking
Ingress-nginx — is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.
Install Ingress NGINX
helm upgrade \
--install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace
To provide a secure HTTPS connection you can use paid SSL certificates, issued for your domain, or use LetsEncrypt-issued certificates. In case of using LetsEcrypt, we recommend install and configure Cert Manager Operator
Install Cert Manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.10.0 \ # Or latest available version
--set installCRDs=true
Configure Cluster Issuer:
1
apiVersion: cert-manager.io/v1
2
kind: ClusterIssuer
3
metadata:
4
name: letsencrypt
5
spec:
6
acme:
7
email: [email protected]-domain.com
8
preferredChain: ''
9
privateKeySecretRef:
10
name: issuer-key
11
server: https://acme-v02.api.letsencrypt.org/directory
12
solvers:
13
- http01:
14
ingress:
15
class: nginx # Ingress class name
If you use Multibox image and want to use cert manger — you should configure DNS01 authorization to provide wildcard certificates
Now you can create k8s
Ingress for Aidbox deploymentIngress
1
apiVersion: networking.k8s.io/v1
2
kind: Ingress
3
metadata:
4
name: aidbox
5
namespace: prod
6
annotations:
7
acme.cert-manager.io/http01-ingress-class: nginx
8
cert-manager.io/cluster-issuer: letsencrypt
9
kubernetes.io/ingress.class: nginx
10
spec:
11
tls:
12
- hosts:
13
- my.box.url
14
secretName: aidbox-tls
15
rules:
16
- host: my.box.url
17
http:
18
paths:
19
- path: /
20
pathType: ImplementationSpecific
21
backend:
22
service:
23
name: aidbox
24
port:
25
number: 80
Now you can test ingress
curl https://my.box.url
Aidbox supports integration with the following systems:
Configure Aidbox and ES integration
Aidbox ConfigMap
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
name: aidbox
5
namespace: prod
6
data:
7
...
8
AIDBOX_ES_URL = http://es-service.es-ns.svc.cluster.local
9
AIDBOX_ES_AUTH = <user>:<password>
10
...
Aidbox ConfigMap
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
name: aidbox
5
namespace: prod
6
data:
7
...
8
AIDBOX_DD_API_KEY: <Datadog API Key>
9
...
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
Create Aidbox metrics service
1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: aidbox-metrics
5
namespace: prod
6
labels:
7
operated: prometheus
8
spec:
9
ports:
10
- protocol: TCP
11
port: 80
12
targetPort: 8765
13
selector:
14
service: aidbox
Create ServiceMonitor config for scrapping metrics data
ServiceMonitor
1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
labels:
5
app.kubernetes.io/component: metrics
6
release: kube-prometheus
7
serviceMonitorSelector: aidbox
8
name: aidbox
9
namespace: kube-prometheus
10
spec:
11
endpoints:
12
- honorLabels: true
13
interval: 10s
14
path: /metrics
15
targetPort: 8765
16
- honorLabels: true
17
interval: 60s
18
path: /metrics/minutes
19
targetPort: 8765
20
- honorLabels: true
21
interval: 10m
22
path: /metrics/hours
23
targetPort: 8765
24
namespaceSelector:
25
any: true
26
selector:
27
matchLabels:
28
operated: prometheus
Or you can directly specify the Prometheus scrapers configuration
global:
external_labels:
monitor: 'aidbox'
scrape_configs:
- job_name: aidbox
scrape_interval: 5s
metrics_path: /metrics
static_configs:
- targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
- job_name: aidbox-minutes
scrape_interval: 30s
metrics_path: /metrics/minutes
static_configs:
- targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
- job_name: aidbox-hours
scrape_interval: 1m
scrape_timeout: 30s
metrics_path: /metrics/hours
static_configs:
- targets: [ 'aidbox-metrics.prod.svc.cluster.local:8765' ]
Aidbox metrics has integration with Grafana, which can generate dashboards and upload them to Grafana — Grafana Integration
System monitoring:
- kube state metrics — is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects
PostgreSQL monitoring:
Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service.
Alert for long-running HTTP queries with P99 > 5s in 5m interval
1
alert: SlowRequests
2
for: 5m
3
expr: histogram_quantile(0.99, sum (rate(aidbox_http_request_duration_seconds_bucket[5m])) by (le, route, instance)) > 5
4
labels: {severity: ticket}
5
annotations:
6
title: Long HTTP query execution
7
metric: '{{ $labels.route }}'
8
value: '{{ $value | printf "%.2f" }}'
Alert manager template for Telegram
1
global:
2
resolve_timeout: 5m
3
telegram_api_url: 'https://api.telegram.org/'
4
route:
5
group_by: [alertname instance]
6
# Default receiver
7
receiver: <my-ops-chat>
8
routes:
9
# Mute watchdog alert
10
- receiver: empty
11
match: {alertname: Watchdog}
12
receivers:
13
- name: empty
14
- name: <my-ops-chat>
15
telegram_configs:
16
- chat_id: <chat-id>
17
api_url: https://api.telegram.org
18
parse_mode: HTML
19
message: |-
20
<b>[{{ .CommonLabels.instance }}] {{ .CommonLabels.alertname }}</b>
21
{{ .CommonAnnotations.title }}
22
{{ range .Alerts }}{{ .Annotations.metric }}: {{ .Annotations.value }}
23
{{ end }}
24
bot_token: <bot-token>
25
- Embedded Grafana alerts
- Grafana OnCall
Vulnerability and security scanners:
Kubernetes Policy Management:
Advanced:
Last modified 1mo ago