AWS Batch Workloads

How would the AWS services listed likely be used to run batch workloads?

For migration batch workloads, the AWS pieces would commonly fit together like this:

• S3: Stores migration inputs and outputs.
• Customer CSV/XML exports uploaded to an S3 bucket
• Generated validation reports, error files, and audit artifacts written back to S3
• Potentially stores large media/assets separately from metadata

• Object keys often encode tenant, migration ID, and processing stage, e.g. s3://migration-bucket/org-123/job-456/input/records.xml

• S3 notifications: Trigger work when files arrive.

• An upload of an input file can emit an ObjectCreated event.
• Rather than processing the file directly in the notification handler, that event is usually forwarded to a queue.

• This creates an event-driven workflow: “input package arrived; create or begin a migration job.”

• SQS: Buffers and distributes units of work.

• A migration coordinator may split a large import into chunks: perhaps one message per file, collection, page of records, or 1,000-record batch.
• Worker processes poll SQS and process messages independently.
• SQS provides backpressure: if 100 customers upload data at once, messages wait safely rather than overwhelming the database or API.
• Failed messages can be retried; after repeated failures, they can move to a dead-letter queue for investigation.

• Because SQS is generally at-least-once delivery, workers must be idempotent. Processing the same batch twice should not create duplicate records or corrupt state.

• ECS / Fargate: Runs the containerized migration code.

• The Django application and migration tooling would be packaged in a Docker image.
• A Fargate task could run a Django management command such as:
  bash python manage.py run_migration --job-id=456 --chunk=12
• Fargate is useful because no server management is required, and task CPU/memory can be selected based on import size.
• For queue-driven work, a service can run one or more worker tasks that poll SQS continuously.
• For a one-off or scheduled migration, the application can launch a short-lived ECS task, which exits after the job completes.

A likely end-to-end flow:

1. A customer migration export is uploaded to S3.
2. S3 sends an event to SQS.
3. A coordinator/worker picks up the message and creates a migration-job record in PostgreSQL.
4. The job validates file structure, required fields, controlled vocabulary values, tenant ownership, and referential relationships.
5. Valid data is divided into checkpointed chunks.
6. SQS messages represent those chunks, and ECS/Fargate workers consume them.
7. Each worker transforms source fields into the target schema and writes data using Django ORM bulk operations or PostgreSQL bulk-loading techniques.
8. Workers record status, counts, errors, checkpoints, and source-to-target ID mappings in PostgreSQL.
9. Reports and rejected-record files are written to S3.
10. A React/MUI UI reads job state from an API and shows progress, warnings, downloadable error reports, and retry options.

For an interview, a strong point is to emphasize operational concerns, not just the services:

• Idempotency: use stable external IDs, unique constraints, upserts, and job/chunk status records.
• Checkpointing: persist progress so a failed task resumes from a known unit rather than restarting an entire migration.
• Retries: distinguish transient failures (network/API/database contention) from bad customer data; retry only the former automatically.
• Tenant isolation: ensure every S3 object, queue message, database query, and generated record is scoped to the correct organization.
• Observability: structured logs with migration/job/chunk/tenant IDs, CloudWatch metrics and alarms, and durable error reports.
• Controlled concurrency: avoid launching enough workers to overload PostgreSQL or downstream APIs.
• Data integrity: validate before loading where possible, use transactions at an appropriate chunk size, and reconcile expected versus imported record counts afterward.