Bulk API

Bulk export & import operations

$dump

You can dump all resources of specific type with $dump operation - GET [resource-type]/$dump - Aidbox will respond with Chunked Transfer Encoding ndjson stream. This is memory efficient operation - Aidbox just stream database cursor to socket. If your HTTP Client support processing of Chunked Encoding you can process resources in stream one by one without waiting for end of response.

GET /Patient/$dump
#response
HTTP/1.1 200 OK
Content-Type: application/ndjson
Transfer-Encoding: chunked
{"resourceType": "Patient", "id": .............}
{"resourceType": "Patient", "id": .............}
{"resourceType": "Patient", "id": .............}
{"resourceType": "Patient", "id": .............}
.........

Here is example how you can dump all patients from your box (assuming you have a client with access policy):

curl -u client:secret -H 'content-type:application/json' \
https://<box-url>/Patient/\$dump | gzip > patients.ndjson.gz

$dump-sql

Take sql query and responds with Chunked Encoded stream in CSV format. Useful to export data for analytics.

POST /$dump-sql
query: select id, resource#>>'{name,0,family}'
format: csv # ndjson; sql; elastic-bulk?
HTTP/1.1 200 OK
Content-Type: application/CSV
Transfer-Encoding: chunked
pt-1 Doe John
pt-2 Smith Mike
................

$load

You can efficiently load data into Aidbox in ndjson gz format from external web service or bucket. There are two version of $load - /$load and /[resourceType]/$load. Both operations accept body with source element, which should be publicly available url. If you want to secure your import use Signed URLs by Amazon S3 or Google Storage. As well there are two version of each operation - prefixed with /fhir - accepts data in FHIR format, and without prefix - works with Aidbox Format.

First can load multiple resource types from one ndjson file, second is more efficient, but load only for specific resource type.

Here how you can load 100 synthea Patients (see tutorial):

POST /fhir/Patient/$load
source: 'https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz'
#resp
{total: 124}

Or load the whole synthea package:

POST /$load
source: 'https://storage.googleapis.com/aidbox-public/synthea/100/all.ndjson.gz'
# resp
{CarePlan: 356, Observation: 20382, MedicationAdministration: 150, .... }

$import & /fhir/$import

$import is implementation of upcoming FHIR Bulk Import API. This is async Operation, which returns url to monitor progress. Here is self descriptive example:

POST /fhir/$import
id: synthea
inputFormat: application/fhir+ndjson
contentEncoding: gzip
mode: bulk
inputs:
- resourceType: Encounter
url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz

You post import body with id and can monitor progress of import using:

GET /BulkImportStatus/[id]

Read more