@cumulus/ingest

Ingest utilities

Stats

StarsIssuesVersionUpdatedCreatedSize
@cumulus/ingest
175139.8.0-alpha.07 days ago4 years agoMinified + gzip package size for @cumulus/ingest in KB

Readme

@cumulus/ingest

@cumulus/ingest is a collection of modules for discovering and ingesting data.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/ingest

Testing

Running tests locally requires localstack.

With localstack running, you can run tests using:

LOCALSTACK_HOST=localhost npm test

Modules

All modules are accessible using require: require('@cumulus/ingest/<MODULE_NAME>') or import: import <MODULE_NAME> from '@cumulus/ingest/<MODULE_NAME>'.

  • consumer - comsumer for SQS messages
  • crypto - provides encryption and decryption methods with a consistent API but differing mechanisms for dealing with encryption keys
  • ftp - for accessing FTP servers
  • granule - discovers and ingests granules
  • http - for accessing data via HTTP and HTTPS
  • lock - creates locks for S3 data
  • log - stringifies JS object logs for ElasticSearch indexing
  • parse-pdr - tools for validating PDRs and generating PDRD and PAN messages
  • pdr - discovers and ingests pdrs
  • queue - creates queues for ingesting data
  • recursion - handles recursion of a FTP/SFTP list operation
  • sftp - for accessing SFTP servers

Contributing

To make a contribution, please see our contributing guidelines.

If you find any bugs or have a feature request, please open an issue on github!

The npm package download data comes from npm's download counts api and package details come from npms.io.