ECS and Fargate

When would I use ECS or Fargate? How would they make sense in an ETL, data migration context?

Amazon ECS (Elastic Container Service) is AWS’s container orchestration service. You use it to run and manage Docker containers without building Kubernetes infrastructure.

AWS Fargate is a serverless compute option for ECS (and EKS): rather than provisioning and maintaining EC2 instances for containers, you specify CPU/memory requirements and AWS runs the task for you.

In practice:

• ECS on EC2: You manage the EC2 worker fleet. Better when workloads are steady, very large, require unusual host-level configuration, or cost optimization through reserved/spot capacity matters.
• ECS on Fargate: AWS manages the servers. Great for isolated, variable, scheduled, or event-driven jobs—often a strong fit for migration tooling.

How this fits ETL and data migration

A migration pipeline often has work that is too long-running, resource-intensive, or operationally distinct from a web application request.

For example, a Django app may let an operator upload a customer export and click “Start migration.” The web service should validate the request and create a migration record, but it should not parse millions of rows and load data inside the HTTP request.

Instead:

1. The customer export is uploaded to S3.
2. The application creates a MigrationJob record in PostgreSQL with status such as queued.
3. A message or event triggers an ECS Fargate task.
4. The container runs a migration command, perhaps a Django management command:
   bash python manage.py run_migration --job-id=12345
5. The task:
6. reads source files from S3,
7. parses XML/CSV/JSON,
8. validates and maps source metadata to the target schema,
9. writes valid records in batches using PostgreSQL bulk operations/upserts,
10. writes errors, checkpoints, counts, and job status back to PostgreSQL,
11. emits logs/metrics to CloudWatch.
12. The UI polls an API or receives status updates so an operator can see progress and resolve data issues.

Why Fargate is useful here

Isolation

Each organization’s migration can run in its own task with its own CPU, memory, environment variables, IAM role, and logs. A bad or unusually large import is less likely to degrade the main Django API.

Right-sized compute

Migration workloads vary widely. One customer may have 5,000 records; another may have 20 million assets and metadata rows.

Fargate tasks can be configured with appropriate CPU and memory per job or migration type. You only run them while work exists.

Reliable long-running work

HTTP requests have timeouts; web workers should remain responsive. A Fargate task can run independently for hours, subject to configured task limits and architecture.

Repeatability

The migration code, dependencies, parsers, and system libraries live in a versioned Docker image. This helps reproduce a specific migration issue:

• “Customer A ran image version migration-worker:2026.08.07.”
• “Re-run job 12345 from checkpoint 82 using the same source file and mapping version.”

Operational visibility

Each task can send logs to CloudWatch, expose metrics such as records processed/errors, and report failure through ECS task status or an event-driven workflow.

A reasonable architecture

Migration UI / Django API
        |
        v
PostgreSQL: MigrationJob(status=queued)
        |
        +--> S3: uploaded source files
        |
        v
SQS / EventBridge / Step Functions
        |
        v
ECS Fargate migration task
        |
        +--> S3 reads
        +--> PostgreSQL bulk writes / checkpoints
        +--> CloudWatch logs and metrics

Common AWS component roles

• S3: source exports, transformed intermediate files, error reports, audit artifacts.
• ECS/Fargate: executes containerized migration workers.
• SQS: buffers work and smooths bursts; useful for per-file or per-batch jobs.
• EventBridge: triggers scheduled or event-based jobs.
• Step Functions: orchestrates multi-step migrations, retries, approvals, validation, and rollback/compensation paths.
• RDS PostgreSQL/Aurora PostgreSQL: target data and migration job/checkpoint state.
• CloudWatch: logs, alarms, dashboards.

Important migration design concerns

Using Fargate does not itself make a migration safe. The worker should be designed for:

• Idempotency: rerunning a job should not create duplicates or corrupt state.
• Checkpointing: persist progress, such as source file, row offset, entity ID, or batch number.
• Batching: use bounded batch sizes for parsing and database writes; avoid loading entire files into memory.
• Transactions at the right scope: usually transactional batches rather than one giant transaction for a multi-hour migration.
• Upserts and stable external identifiers: map legacy IDs to target IDs and enforce uniqueness.
• Retry strategy: distinguish transient errors (network/database throttling) from invalid customer data.
• Dead-letter/error handling: preserve malformed records and produce actionable reports rather than failing silently.
• Concurrency limits: avoid starting enough tasks to overwhelm PostgreSQL or create tenant-level contention.
• Auditability: record source file checksum, mapping version, tool/image version, timestamps, counts, and errors.

Interview-ready summary

I’d use ECS Fargate for migration workers when the work is asynchronous, long-running, variable in resource requirements, or should be isolated from the main API. The Django application would create and track migration jobs, while a versioned container runs the ETL process: pulling files from S3, validating and transforming records, performing checkpointed bulk upserts into PostgreSQL, and reporting progress and errors. Fargate removes the need to manage worker servers, while ECS provides task execution, logging, IAM isolation, and operational controls.