$import & /fhir/$import

$import is an implementation of the upcoming FHIR Bulk Import API. This is an asynchronous Operation, which returns url to monitor progress. There are two versions of this operation - /fhir/$import accepts data in FHIR format, /$import works with Aidbox format.
Resource requirements for all import operations:
Operation
id
resourceType
/$import
Required
Not required
/fhir/$import
Required
Not required
Keep in mind that $import does not validate inserted resources for the sake of performance. Pay attention to the structure of data you insert and use the correct URL for your data format, i.e.: use /fhir prefix for FHIR data.
Please consider using Asynchronous validation API to validate data after $import
Example
Request
Response
POST /fhir/$import
Accept: text/yaml
Content-Type: text/yaml
​
id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
status: 200
Parameters
Parameter
Description
id
Identifier of the import
contentEncoding
Supports gzip or plain (non-gzipped .ndjson files)
inputs
Resources to import
update
Update history for updated resources (false by default)
You can monitor progress by using id you provided in request body.
Request
Response (Not Finished)
Response (Finished)
Response (Failed)
GET /BulkImportStatus/synthea
Status
200
Body
time:
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'
​
Status
200
Body
time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    total: 3460
    status: finished
    duration: 530
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'
Status
200
Body
time:
  end: '2023-05-15T14:45:33.820465+02:00'
  start: '2023-05-15T14:45:33.28722+02:00'
type: aidbox
inputs:
  - ts: '2023-05-15T14:45:33.819425+02:00'
    url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    error: '403: Forbidden'
    status: failed
    resourceType: Encounter
status: finished
contentEncoding: gzip
id: >-
  synthea
resourceType: BulkImportStatus
meta:
  lastUpdated: '2023-05-15T12:45:33.278829Z'
  createdAt: '2023-05-15T12:45:33.278829Z'
  versionId: '129363'
If you didn't provide id in request body, you can use content-location in response header.
Result
Parameter
Type
Description
id
string
Identifier of the import
resourceType
string
Type of resource where the progress of import operation is recorded.
Possible value: BulkImportStatus
meta
object
​
meta.createdAt
string
Timestamp string at which the resource was created
meta.lastUpdated
string
Timestamp string at which the resource was updated last time
meta.versionId
string
Version id of this resource
contentEncoding
string
gzip or plain
time
object
​
time.start
string
Timestamp string at which the operation started in ISO format
time.end
string
Timestamp string at which the operation was completed in ISO format.
Only present after the entire import operation has been completed
type
string
Data format type to be loaded.
Possible values: aidbox, fhir
inputs
object[]
​
inputs[].url
string
URL from which load resources
inputs[].resourceType
string
Resource type to be loaded
inputs[].status
string
Load status for each input.
Only present after the operation for this input has been completed.
Possible values: finished, failed
inputs[].total
integer
The number of loaded resources.
Only present after the operation for this input has been completed successfully
inputs[].ts
string
Timestamp string at which the loading was completed in ISO format.
Only present after the operation for this input has been completed
inputs[].duration
integer
Duration of loading in milliseconds.
Only present after the operation for this input has been completed successfully
status
string
Load status for all inputs.
Only present after the entire import operation has been completed.
After completed, this value is always finished, regardless of whether each input is finished or failed.
Possible value: finished
​
Note
For performance reasons $import does raw upsert into resource table without history update. If you want to store the previous version of resources in history, please set update = true
With this flag Aidbox will update history for updated resources. For each resource:
if resource was not present in DB before the import, the import time will be the same.
if resource was present in DB before and it's updated during the import, it will double the time importing this resource because of additional insert operation into _history table.
/v2/$import on top of the Workflow Engine
Improved version of the $import operation, to enhance its reliability and performance. By implementing this operation on top of the Workflow Engine, it allows the $import operation to be more reliable, continue work after restarts, and handle errors correctly. The Task API also enables the operation to accept multiple requests and execute them from a queue while simultaneously processing multiple items from the "inputs" field (with a default of two items processed simultaneously). Users can monitor the status of the operation through the Monitoring.
In the future, the ability to list and cancel $import operations will be added, as well as detailed progress info on the operation.
To enable new version of $import API (/v2/$import & /v2/fhir/$import) set environment variable BOX_BULK__API_ENGINE=task-api
Changes in the new $import API:
1.Executing more than one import with the same id is not possible. Users can omit the `id` field from the request, allowing Aidbox to generate the ID.
2.The status of the workflow can be accessed with a GET request to /v2/$import/<id> instead of /BulkImportStatus/<id>. The URL for the import status is returned in the content-location header of the $import request.
This feature is currently not available in Multibox​
To start import make a POST request to /v2[/fhir]/$import:
Request
Response
POST /v2/fhir/$import
Accept: text/yaml
Content-Type: text/yaml
​
id: synthea
contentEncoding: gzip
inputs:
- resourceType: Encounter
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
- resourceType: Organization
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
- resourceType: Patient
  url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
Status
200 OK
Headers
Content-Location:  /v2/$import/synthea
Parameters
Parameter
Description
id
Identifier of the import.
If you don't provide this, the id will be auto-generated. You can check it on Content-Location header in the response
contentEncoding
Supports gzip or plain (non-gzipped .ndjson files)
inputs (required)
Resources to import
url - URL from which load resources
resourceType - Resource type to be loaded
update
Update history for updated resources (false by default)
To check the status of the import make a GET request to /v2/$import/<id>:
As the operation is built on top of our workflow engine, the statuses and outcomes of the files and import as a whole are inherited from Task Statuses and Outcomes​
Request
Response (In progress)
Response (done - succeeded)
Response (done - failed)
GET /v2/$import/<id>
Status
200 OK
Body
type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: in-progress
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: waiting
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: waiting
contentEncoding: gzip
status: in-progress
Status
200 OK
Body
type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 0
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: succeeded
    result:
      imported-resources: 3460
  - url: https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: succeeded
    result:
      imported-resources: 124
contentEncoding: gzip
status: done
outcome: succeeded
result:
  message: All input files imported, 3584 new resources loaded
  total-files: 3
  total-imported-resources: 3584
Status
200 OK
Body
type: fhir
inputs:
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Organization.ndjson.gz
    resourceType: Organization
    status: done
    outcome: succeeded
    result:
      imported-resources: 225
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    resourceType: Encounter
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
  - url: >-
      https://storage.googleapis.com/aidbox-public/synthea/100/Patient.ndjson.gz
    resourceType: Patient
    status: done
    outcome: failed
    error:
      message: '403: Forbidden'
contentEncoding: gzip
status: done
outcome: failed
error:
  message: >-
    Import for some files failed with an error: task 'Encounter
    https://storage.googleapis.com/aidbox-public/synthea/100/Encounter.ndjson.gz
    failed
​

Operation	id	resourceType
`/$import`	Required	Not required
`/fhir/$import`	Required	Not required

Parameter	Description
`id`	Identifier of the import
`contentEncoding`	Supports `gzip` or `plain` (non-gzipped .ndjson files)
`inputs`	Resources to import
`update`	Update history for updated resources (false by default)

Parameter	Type	Description
`id`	string	Identifier of the import
`resourceType`	string	Type of resource where the progress of import operation is recorded. Possible value: `BulkImportStatus`
`meta`	object
`meta.createdAt`	string	Timestamp string at which the resource was created
`meta.lastUpdated`	string	Timestamp string at which the resource was updated last time
`meta.versionId`	string	Version id of this resource
`contentEncoding`	string	`gzip` or `plain`
`time`	object
`time.start`	string	Timestamp string at which the operation started in ISO format
`time.end`	string	Timestamp string at which the operation was completed in ISO format. Only present after the entire import operation has been completed
`type`	string	Data format type to be loaded. Possible values: `aidbox`, `fhir`
`inputs`	object[]
`inputs[].url`	string	URL from which load resources
`inputs[].resourceType`	string	Resource type to be loaded
`inputs[].status`	string	Load status for each input. Only present after the operation for this input has been completed. Possible values: `finished`, `failed`
`inputs[].total`	integer	The number of loaded resources. Only present after the operation for this input has been completed successfully
`inputs[].ts`	string	Timestamp string at which the loading was completed in ISO format. Only present after the operation for this input has been completed
`inputs[].duration`	integer	Duration of loading in milliseconds. Only present after the operation for this input has been completed successfully
`status`	string	Load status for all inputs. Only present after the entire import operation has been completed. After completed, this value is always `finished`, regardless of whether each input is `finished` or `failed`. Possible value: `finished`

Parameter	Description
`id`	Identifier of the import. If you don't provide this, the id will be auto-generated. You can check it on `Content-Location` header in the response
`contentEncoding`	Supports `gzip` or `plain` (non-gzipped .ndjson files)
`inputs` (required)	Resources to import `url` - URL from which load resources `resourceType` - Resource type to be loaded
`update`	Update history for updated resources (false by default)

$load & /fhir/$load

aidbox.bulk data import

Last modified 18d ago

$import & /fhir/$import

Resource requirements for all import operations:

Example

Parameters

Result

​

Note

/v2/$import on top of the Workflow Engine

Parameters

​