$import & /fhir/$import

$import is an implementation of the upcoming FHIR Bulk Import API. This is an asynchronous Operation, which returns url to monitor progress. There are two versions of this operation - /fhir/$import accepts data in FHIR format, /$import works with Aidbox format.

Resource requirements for all import operations:

Operation	id	resourceType
/$import	Required	Not required
/fhir/$import	Required	Not required

Keep in mind that $import does not validate inserted resources for the sake of performance. Pay attention to the structure of data you insert and use the correct URL for your data format, i.e.: use /fhir prefix for FHIR data.

Please consider using Asynchronous validation API to validate data after $import

Example

Request

POST /fhir/$import
Accept: text/yaml
Content-Type: text/yaml

id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz

Response

status: 200

Parameters

Parameter	Description
id	Identifier of the import
contentEncoding	Supports gzip or plain (non-gzipped .ndjson files)
inputs	Resources to import
update	Update history for updated resources (false by default)

You can monitor progress by using id you provided in request body.

Request

GET /BulkImportStatus/synthea

Response (Not Finished)

Status

200

Body

time:
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

Response (Finished)

Status

200

Body

time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    total: 3460
    status: finished
    duration: 530
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

Response (Failed)

Status

200

Body

time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    error: '403: Forbidden'
    status: failed
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'

If you didn't provide id in request body, you can use content-location in response header.

Result

Parameter	Type	Description
id	string	Identifier of the import
resourceType	string	Type of resource where the progress of import operation is recorded. Possible value: BulkImportStatus
meta	object
meta.createdAt	string	Timestamp string at which the resource was created
meta.lastUpdated	string	Timestamp string at which the resource was updated last time
meta.versionId	string	Version id of this resource
contentEncoding	string	gzip or plain
time	object
time.start	string	Timestamp string at which the operation started in ISO format
time.end	string	Timestamp string at which the operation was completed in ISO format. Only present after the entire import operation has been completed
type	string	Data format type to be loaded. Possible values: aidbox, fhir
inputs	object[]
inputs[].url	string	URL from which load resources
inputs[].resourceType	string	Resource type to be loaded
inputs[].status	string	Load status for each input. Only present after the operation for this input has been completed. Possible values: finished, failed
inputs[].total	integer	The number of loaded resources. Only present after the operation for this input has been completed successfully
inputs[].ts	string	Timestamp string at which the loading was completed in ISO format. Only present after the operation for this input has been completed
inputs[].duration	integer	Duration of loading in milliseconds. Only present after the operation for this input has been completed successfully
status	string	Load status for all inputs. Only present after the entire import operation has been completed. After completed, this value is always finished, regardless of whether each input is finished or failed. Possible value: finished

Note

For performance reasons $import does raw upsert into the resource table without history update. If you want to store the previous version of resources in history, please set update = true

With this flag, Aidbox will update the history for updated resources. For each resource:

• if the resource was not present in DB before the import, the import time will be the same.

• if the resource was present in DB before and it's updated during the import, it will double the time importing this resource because of the additional insert operation into the _history table.

/v2/$import on top of the Workflow Engine

Improved version of the $import operation, to enhance its reliability and performance. By implementing this operation on top of the workflow-engine, it allows the $import operation to be more reliable, continue work after restarts, and handle errors correctly. The Task API also enables the operation to accept multiple requests and execute them from a queue while simultaneously processing multiple items from the "inputs" field (with a default of two items processed simultaneously). Users can monitor the status of the operation through the monitoring.md.

In the future, the ability to list and cancel $import operations will be added, as well as detailed progress info on the operation.

Changes in the new $import API:

1. Executing more than one import with the same id is not possible. Users can omit the `id` field from the request, allowing Aidbox to generate the ID.

2. The status of the workflow can be accessed with a GET request to /v2/$import/<id> instead of /BulkImportStatus/<id>. The URL for the import status is returned in the content-location header of the $import request.

This feature is not available in Multibox

To start import make a POST request to /v2[/fhir]/$import:

Request

POST /v2/fhir/$import
Accept: text/yaml
Content-Type: text/yaml

id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz

Response

Status

200 OK

Headers

Content-Location:  /v2/$import/synthea

Parameters

Parameter	Description
id	Identifier of the import. If you don't provide this, the id will be auto-generated. You can check it on Content-Location header in the response
contentEncoding	Supports gzip or plain (non-gzipped .ndjson files)
inputs (required)	Resources to import url - URL from which load resources resourceType - Resource type to be loaded
update	Update history for updated resources (false by default)
allowedRetryCount	Set the maximum number of import retries for each input (2 by default)

To check the status of the import make a GET request to /v2/$import/<id>:

As the operation is built on top of our workflow engine, the statuses and outcomes of the files and import as a whole are inherited from #task-statuses-and-outcomes

Request

GET /v2/$import/<id>

Response (In progress)

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: in-progress
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: waiting
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: waiting
contentEncoding: gzip
status: in-progress

Response (done - succeeded)

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 0
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: succeeded
    result:
      imported-resources: 3460
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: succeeded
    result:
      imported-resources: 124
contentEncoding: gzip
status: done
outcome: succeeded
result:
  message: All input files imported, 3584 new resources loaded
  total-files: 3
  total-imported-resources: 3584

Response (done - failed)

Status

200 OK

Body

type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 225
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
contentEncoding: gzip
status: done
outcome: failed
error:
  message: >-
    Import for some files failed with an error: task 'Encounter
    https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    failed

Import local file

Sometimes you want to import local file into local Aidbox. Possible solutions for local development:

Add volume to the aidboxone container (not aidboxdb):

volumes:
- ./Encounter.ndjson.gz:/resources/Encounter.ndjson.gz
# url: file:///resources/Encounter.ndjson.gz

Use tunneling e.g. ngrok:

python3 -m http.server 
ngrok http 8000
# url: https://<...>.ngrok-free.app/Encounter.ndjson.gz