Amazon SQS (Simple Queue Service) is a managed message queue. Its main job is to decouple producers of work from consumers of work.
Instead of one service directly calling another and waiting for it to finish, a producer puts a message on a queue. One or more workers consume messages later and process them asynchronously.
Producer → SQS queue → Worker(s)
For a migration system:
User uploads CSV to S3
↓
A job is created and a message is sent to SQS
↓
Fargate worker receives the message
↓
Worker validates/processes the file and updates migration status
This design lets uploads, job creation, and processing proceed independently.
What problems SQS solves
1. Decoupling services
Without a queue:
Web application → directly invokes migration worker → waits
Problems:
- The user-facing request may time out.
- The web application is tied to worker availability.
- A worker outage can cause user-facing failures.
- Scaling workers requires changing the web application’s behavior.
- A large migration can consume resources needed for normal requests.
With SQS:
Web application → enqueue migration job → respond quickly
Worker → processes job asynchronously
The web application only needs to know that the request was accepted.
2. Buffering bursts of work
Suppose 100 customers submit migration files around the same time, but your system can safely run only 10 migration workers.
SQS absorbs the burst:
100 queued jobs → 10 workers steadily process them
The queue provides backpressure. Work waits rather than overwhelming PostgreSQL, an external API, or a downstream indexing service.
In migration work, this is particularly important because bulk loads can:
- Exhaust database connections.
- Cause lock contention.
- Saturate database I/O.
- Trigger API rate limits.
- Consume CPU/memory while parsing large XML or CSV files.
You can control concurrency by limiting ECS task count, worker count, or batch throughput.
3. Reliable asynchronous delivery and retry
SQS stores messages durably and keeps them available until a consumer successfully processes them and deletes them.
A simplified lifecycle:
1. Producer sends message.
2. SQS stores it durably.
3. Consumer receives message.
4. Message becomes temporarily invisible.
5. Consumer processes it.
6. Consumer deletes the message to acknowledge success.
If the worker crashes before deletion:
1. Worker receives message.
2. SQS hides it for the visibility timeout.
3. Worker crashes or times out.
4. Visibility timeout expires.
5. SQS makes the message available again.
6. Another worker can retry it.
That produces at-least-once delivery: a message can be delivered more than once.
Important SQS concepts
Queue
A queue is the durable container holding messages awaiting processing.
Common configuration includes:
- Retention period
- Visibility timeout
- Delivery delay
- Long polling duration
- Dead-letter queue policy
- Encryption and access policy
Messages can remain in SQS for up to 14 days, depending on queue retention settings.
Message
A message contains a payload, typically JSON, plus SQS metadata.
Example:
{
"event_type": "migration.requested",
"migration_id": "mig_01JXYZ",
"tenant_id": "tenant_123",
"input": {
"bucket": "migration-inputs",
"key": "tenant_123/mig_01JXYZ/export.csv",
"version_id": "abc123"
},
"requested_at": "2026-08-07T14:25:00Z"
}
Keep messages relatively small. SQS has a 256 KB message size limit. Put large payloads—CSV/XML files, manifests, reports—in S3 and send only a reference in SQS.
Producer
The producer sends a message using SendMessage.
Potential producers include:
- A Django app after creating a migration record.
- A Lambda function triggered by an S3 upload.
- A scheduled process.
- Another worker that creates follow-up work.
A producer should usually create durable database state before or alongside enqueueing work. Otherwise, the worker may receive a message referring to a migration job that does not exist.
A common reliability concern is the dual-write problem:
1. Save migration job in PostgreSQL.
2. Send SQS message.
If step 1 succeeds and step 2 fails, the job exists but is never processed.
1. Send SQS message.
2. Save migration job in PostgreSQL.
If step 1 succeeds and step 2 fails, a worker may receive a message for a nonexistent job.
A common solution is a transactional outbox:
Within one PostgreSQL transaction:
- Create/update migration job.
- Write an outbox event row.
Separate publisher:
- Reads unsent outbox rows.
- Sends each event to SQS.
- Marks it sent.
The publisher itself can retry safely; duplicate messages are acceptable if consumers are idempotent.
Consumer / worker
The consumer polls the queue with ReceiveMessage.
When it receives a message, SQS does not immediately remove it. Instead, it hides it for the configured visibility timeout.
The worker should:
- Validate the message.
- Claim or lock the associated migration job in the database.
- Process the work.
- Persist successful state/checkpoints.
- Delete the SQS message only after it is safe to consider the work accepted or completed.
For a very large migration, the message typically triggers a durable job; it should not necessarily represent the whole job as a single uninterruptible unit.
For example:
SQS message: “start or resume migration mig_123”
Database: tracks each batch and current checkpoint
Worker: processes a bounded amount of work, then schedules/continues safely
Visibility timeout
The visibility timeout is the amount of time after receiving a message during which other consumers cannot receive it.
Example:
Queue visibility timeout: 10 minutes
12:00 — Worker A receives message.
12:00–12:10 — Message is hidden from other workers.
12:05 — Worker A succeeds and deletes it.
Or:
12:00 — Worker A receives message.
12:07 — Worker A crashes.
12:10 — Visibility timeout expires.
12:10 — Message becomes available for retry.
If processing may take longer than the visibility timeout, the worker can call ChangeMessageVisibility to extend it.
Risks:
- Too short: a job may still be running when the message reappears, resulting in concurrent duplicate processing.
- Too long: recovery from a crashed worker is slow.
- No idempotency: duplicate delivery can create duplicate records or repeated external side effects.
For batch jobs, a common design is to use a reasonably bounded unit of work per message or have the message initiate a tracked job rather than keep a message invisible for hours.
Message deletion / acknowledgement
SQS uses an explicit delete model.
Receiving a message is not acknowledgement. A successful worker must call DeleteMessage using the receipt handle returned by that specific receive operation.
If it does not delete the message, the message reappears after the visibility timeout.
Because the worker may successfully commit a database change and then fail before deletion, it must assume the message can be delivered again. That is why idempotency is central to SQS consumers.
Dead-letter queue (DLQ)
A DLQ receives messages that fail processing repeatedly.
For example:
Main migration queue:
maxReceiveCount = 5
Message fails five times:
→ moved automatically to migration-dlq
Typical reasons:
- Invalid message schema
- Missing S3 object
- Unsupported source file version
- Persistent database constraint violation
- A software bug
- An unavailable dependency
A DLQ prevents “poison messages” from being retried forever and consuming worker capacity.
Important operational practice:
- Monitor DLQ depth.
- Alert on new DLQ messages.
- Preserve enough metadata to investigate.
- Fix the underlying issue before replaying messages.
- Redrive messages deliberately after remediation.
For migration jobs, it is often useful to update a database job record to FAILED or NEEDS_ATTENTION with a meaningful error report before allowing the message to reach the DLQ.
Standard queues vs FIFO queues
Standard queues
Standard queues are the usual choice.
They provide:
- Very high throughput
- At-least-once delivery
- Best-effort ordering
“Best-effort ordering” means messages may arrive out of order, and duplicates are possible.
For independent migration jobs, this is usually fine.
Migration A requested
Migration B requested
Possible processing order:
B, A, A
Your application should not rely on order.
FIFO queues
FIFO queues provide stronger ordering and deduplication guarantees.
They support:
- Ordered processing within a message group
- Exactly-once message processing semantics within SQS’s defined deduplication window and constraints
- Lower throughput characteristics than standard queues, depending on configuration
A MessageGroupId defines the ordered stream:
MessageGroupId = tenant_123
Messages for tenant_123 are processed in order, while other tenant groups may process concurrently.
However, FIFO does not magically guarantee exactly-once business effects across PostgreSQL, S3, and external APIs. A consumer can still commit a database change and crash before deleting the message. You still need idempotent consumers and database constraints.
For migration systems, FIFO may be useful if:
- Only one migration per tenant can safely run at once.
- Operations must occur in strict order for a collection.
- Ordering requirements outweigh reduced parallelism.
Often, explicit tenant/job locking in PostgreSQL is enough, and a standard queue remains simpler.
SQS and migration-job design
A robust pattern is:
1. Django creates MigrationJob(status=PENDING).
2. Django stores the input file reference and configuration/version.
3. An outbox publisher sends {migration_id} to SQS.
4. Worker receives the message.
5. Worker atomically claims the job if it is eligible to run.
6. Worker transitions status to VALIDATING / RUNNING.
7. Worker processes batches, storing checkpoints and metrics.
8. Worker marks COMPLETED, COMPLETED_WITH_ERRORS, or FAILED.
9. Worker deletes the SQS message.
Example state model:
PENDING
→ VALIDATING
→ RUNNING
→ COMPLETED
PENDING
→ VALIDATING
→ REJECTED
RUNNING
→ PAUSED
→ RUNNING
RUNNING
→ FAILED
→ RUNNING # deliberate retry/resume
RUNNING
→ COMPLETED_WITH_ERRORS
The database is the source of truth for job state. SQS is the delivery and scheduling mechanism, not the authoritative record of whether a migration completed.
Key limitations and non-goals
SQS is not:
- A relational database or job-history system
- A workflow engine with rich dependency management
- A guarantee that a worker completed business logic exactly once
- A solution for storing large files
- A broadcast/event-stream system by itself—one queue message is generally handled by one consumer, not all consumers
If multiple independent systems need the same event, use SNS, EventBridge, or separate queues subscribed to a fan-out mechanism.
Strong concise interview answer
SQS is a durable managed queue used to decouple work producers from asynchronous consumers. It buffers bursts, lets workers scale independently, and retries work when consumers fail. A worker receives a message, which is hidden for a visibility timeout, processes it, and explicitly deletes it on success. If it crashes or does not acknowledge, the message becomes visible and may be delivered again, so standard SQS consumers must be idempotent. In a migration system, I would use SQS to trigger or schedule durable migration jobs, store job state and checkpoints in PostgreSQL, keep source files and reports in S3, and use a DLQ plus monitoring for repeatedly failing jobs.