.ndjsonfiles on an AWS bucket directly to the Aidbox database with maximum performance.
.ndjsonis required. Files can be placed in subdirectories of any level. Files with the wrong path structure will be ignored.
bucket* defines your bucket connection string in format
thread-numdefines how many threads will process the import. The default is 4.
access-key-id* AWS key ID
secret-access-key* AWS secret key
region* AWS Bucket region
disable-idx?the default is
false. Allows to drop all indexes for resources, which data are going to be loaded. Indexes will be restored at the end of successful import. All information about dropped indexes is stored at
drop-primary-key?the default is
false. The same as the previous parameter, but drops primary key constraint for resources tables. This parameter disables all checks for duplicates for imported resources.
upsert?the default is
false, import for files with
iduniqueness constraint violation will fail with an error, if
true- records in the database will be overridden with records from import. Even when
true, it's still not allowed to have more than one record with the same id in one import file. Setting this option to true will cause a decrease in performance.
by-last-modified, the default is
optimal. Establishes the order in which the files are processed. The
optimalvalue provides the best performance.
by-last-modifiedshould be used with
thread-num = 1to guarantee a stable order of file processing.
prefixesarray of prefixes to specify which files should be processed. Example: with value
["fhir/1/", "fhir/2/Patient"]only files from directory
Patientfiles from directory
"fhir/2"will be processed.
load-from-bucketmethod, Aidbox creates
LoaderFileresource. To find out how many resources were imported from a file, check the
aidbox.bulk/load-from-bucketchecks if files from the bucket were planned to import and decides what to do:
ndjsonfile has it's related
LoaderFileresource, the loader skips this file from import
LoaderFileresource, Aidbox puts this file to the queue creating a
LoaderFileresource and relaunch