$import & /fhir/$import
$import
is an implementation of the upcoming FHIR Bulk Import API. This is an asynchronous Operation, which returns url to monitor progress. There are two versions of this operation - /fhir/$import
accepts data in FHIR format, /$import
works with Aidbox format.Operation | id | resourceType |
---|---|---|
/$import | Required | Not required |
/fhir/$import | Required | Not required |
Keep in mind that $import does not validate inserted resources for the sake of performance. Pay attention to the structure of data you insert and use the correct URL for your data format, i.e.: use /fhir prefix for FHIR data.
Request
Response
POST /fhir/$import
Accept: text/yaml
Content-Type: text/yaml
id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
status: 200
Parameter | Description |
---|---|
id | Identifier of the import |
contentEncoding | Supports gzip or plain (non-gzipped .ndjson files) |
inputs | Resources to import |
update | Update history for updated resources (false by default) |
You can monitor progress by using
id
you provided in request body.Request
Response (Not Finished)
Response (Finished)
Response (Failed)
GET /BulkImportStatus/synthea
Status
200
Body
time:
start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
resourceType: Encounter
contentEncoding: gzip
id: >-
synthea
resourceType: BulkImportStatus
meta:
lastUpdated: '2023-05-15T12:45:33.278829Z'
createdAt: '2023-05-15T12:45:33.278829Z'
versionId: '129363'
Status
200
Body
time:
end: '2023-05-15T14:45:33.820465+02:00'
start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
- ts: '2023-05-15T14:45:33.819425+02:00'
url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
total: 3460
status: finished
duration: 530
resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
synthea
resourceType: BulkImportStatus
meta:
lastUpdated: '2023-05-15T12:45:33.278829Z'
createdAt: '2023-05-15T12:45:33.278829Z'
versionId: '129363'
Status
200
Body
time:
end: '2023-05-15T14:45:33.820465+02:00'
start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
- ts: '2023-05-15T14:45:33.819425+02:00'
url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
error: '403: Forbidden'
status: failed
resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
synthea
resourceType: BulkImportStatus
meta:
lastUpdated: '2023-05-15T12:45:33.278829Z'
createdAt: '2023-05-15T12:45:33.278829Z'
versionId: '129363'
If you didn't provide
id
in request body, you can use content-location
in response header.Parameter | Type | Description |
---|---|---|
id | string | Identifier of the import |
resourceType | string | Type of resource where the progress of import operation is recorded.
Possible value: BulkImportStatus |
meta | object | |
meta.createdAt | string | Timestamp string at which the resource was created |
meta.lastUpdated | string | Timestamp string at which the resource was updated last time |
meta.versionId | string | Version id of this resource |
contentEncoding | string | gzip or plain |
time | object | |
time.start | string | Timestamp string at which the operation started in ISO format |
time.end | string | Timestamp string at which the operation was completed in ISO format.
Only present after the entire import operation has been completed |
type | string | Data format type to be loaded. Possible values: aidbox , fhir |
inputs | object[] | |
inputs[].url | string | URL from which load resources |
inputs[].resourceType | string | Resource type to be loaded |
inputs[].status | string | Load status for each input.
Only present after the operation for this input has been completed. Possible values: finished , failed |
inputs[].total | integer | The number of loaded resources.
Only present after the operation for this input has been completed successfully |
inputs[].ts | string | Timestamp string at which the loading was completed in ISO format.
Only present after the operation for this input has been completed |
inputs[].duration | integer | Duration of loading in milliseconds.
Only present after the operation for this input has been completed successfully |
status | string | Load status for all inputs. Only present after the entire import operation has been completed.
After completed, this value is always finished , regardless of whether each input is finished or failed .Possible value: finished |
For performance reasons
$import
does raw upsert into resource table without history update. If you want to store the previous version of resources in history, please set update = true
With this flag Aidbox will update history for updated resources. For each resource:
- if resource was not present in DB before the import, the import time will be the same.
- if resource was present in DB before and it's updated during the import, it will double the time importing this resource because of additional insert operation into
_history
table.
Improved version of the $import operation, to enhance its reliability and performance. By implementing this operation on top of the Workflow Engine, it allows the $import operation to be more reliable, continue work after restarts, and handle errors correctly. The Task API also enables the operation to accept multiple requests and execute them from a queue while simultaneously processing multiple items from the "inputs" field (with a default of two items processed simultaneously). Users can monitor the status of the operation through the Monitoring.
In the future, the ability to list and cancel $import operations will be added, as well as detailed progress info on the operation.
To enable new version of $import API (
/v2/$import
& /v2/fhir/$import
) set environment variable BOX_BULK__API_ENGINE=task-api
Changes in the new $import API:
- 1.Executing more than one import with the same
id
is not possible. Users can omit the `id` field from the request, allowing Aidbox to generate the ID. - 2.The status of the workflow can be accessed with a GET request to
/v2/$import/<id>
instead of/BulkImportStatus/<id>
. The URL for the import status is returned in thecontent-location
header of the $import request.
To start import make a POST request to
/v2[/fhir]/$import
:Request
Response
POST /v2/fhir/$import
Accept: text/yaml
Content-Type: text/yaml
id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
Status
200 OK
Headers
Content-Location: /v2/$import/synthea
Parameter | Description |
---|---|
id | Identifier of the import.
If you don't provide this, the id will be auto-generated. You can check it on Content-Location header in the response |
contentEncoding | Supports gzip or plain (non-gzipped .ndjson files) |
inputs (required) | Resources to import
|
update | Update history for updated resources (false by default) |
To check the status of the import make a GET request to
/v2/$import/<id>
:As the operation is built on top of our workflow engine, the statuses and outcomes of the files and import as a whole are inherited from Task Statuses and Outcomes
Request
Response (In progress)
Response (done - succeeded)
Response (done - failed)
GET /v2/$import/<id>
Status
200 OK
Body
type: fhir
inputs:
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
resourceType: Organization
status: in-progress
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
resourceType: Encounter
status: waiting
- url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
resourceType: Patient
status: waiting
contentEncoding: gzip
status: in-progress
Status
200 OK
Body
type: fhir
inputs:
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
resourceType: Organization
status: done
outcome: succeeded
result:
imported-resources: 0
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
resourceType: Encounter
status: done
outcome: succeeded
result:
imported-resources: 3460
- url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
resourceType: Patient
status: done
outcome: succeeded
result:
imported-resources: 124
contentEncoding: gzip
status: done
outcome: succeeded
result:
message: All input files imported, 3584 new resources loaded
total-files: 3
total-imported-resources: 3584
Status
200 OK
Body
type: fhir
inputs:
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
resourceType: Organization
status: done
outcome: succeeded
result:
imported-resources: 225
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
resourceType: Encounter
status: done
outcome: failed
error:
message: '403: Forbidden'
- url: >-
https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
resourceType: Patient
status: done
outcome: failed
error:
message: '403: Forbidden'
contentEncoding: gzip
status: done
outcome: failed
error:
message: >-
Import for some files failed with an error: task 'Encounter
https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
failed
Last modified 18d ago