Branch

Exactly Once vs At Least Once

Human dan

I'm interviewing for a senior engineer position and would like to do some light interview prep. The team I'd be joining is migrating and modernizing a legacy application. That application is used by organizations around the world, so it is key that the migration is smooth and painless.

Responsibilities

  • Support legacy data migration efforts
  • Prepare and validate customer data for migration
  • Execute and monitor data load activities
  • Investigate and resolve migration related activities
  • Implement fixes and enhancements to migration tooling and related application code
  • Deliver minor user interface improvements to support migration workflows.
  • Conduct code reviews and ensure code quality standards
  • Solve complex technical problems and identify practical solutions
  • Collaborate with stakeholders on migration priorities and outcomes

Qualifications

  • 4+ years of software dev experience
  • Python (Django): Strong experience building and maintaining server-side applications. Comfortable with Django management commands, ORM bulk operations, transactions, and service-layer patterns. Experience with the data migration domain (ETL, data mapping, checkpointed jobs) is a plus.
  • Data migration and ETL: Hands-on experience moving structured data between systems. Familiarity with XML/CSV parsing, metadata schemas, controlled vocabularies Controlled Vocabularies, and bulk database loads. Django and ETL Experience with data collections, DAM, or library/archive metadata is strongly preferred. Dam and libary/archive metadata
  • PostgreSQL: Solid understanding of relational databases, JSON/JSONB fields, upserts, and debugging data integrity issues at scale.
    2 branches
  • TypeScript / React: Working proficiency for minor UI updates in Next.js App router codebase. Next.js App routerComfortable reading and extending existing components; deep SSR/RSC expertise not required for this role.
  • Material UI (MUI): Familiarity with MUI V6+ for small UI changes.
  • RESTful APIs: Experience consuming and integrating with RES APIs; OpenAPI/Swagger-generated clients a plus. Generated REST ClientsUnderstanding of multi-tenant SaaS and content/metadata-heavy domains is a plus.
  • AWS and distributed workflows: Working knowledge of S3 ECS/Fargate tasks and event-driven ECS and Fargateprocessing (S3 notifications, SQS, background jobs). Experience running batch workloads in cloud environments preferred. AWS Batch Workloads
  • Problem-solving: Strong analytical skills for debugging partial-failure batch jobs, correlating error logs with source data, and proposing pragmatic fixes under migration timelines.
  • Communication: Clear written documentation of migration issues, fixes, and rerun steps for operators and engineers.
  • AI-assisted development: Comfort using AI coding tools for investigation, test scaffolding, and routine fixes; judgment to validate outputs against reports and tests.

Prompt

Create a study and interview prep plan that will only take a few hours to go through. For reference I am a seasoned python/django developer and am currently working as a DevOps engineer on AWS backed systems.

AI gpt-5.6-terra

Senior Engineer Migration Role: Focused Prep Plan (3–4 hours)

You already have the strongest foundations for this role: Python/Django plus AWS operations. Focus your time on translating that experience into data-migration-specific design, operational judgment, and clear interview narratives.

1. What the interviewers are likely evaluating

They probably do not need you to prove you can write Django models or deploy to AWS. They will be looking for whether you can safely operate in a context where:

  • Source data is inconsistent, incomplete, or unexpectedly large.
  • Migrations have partial failures and must be safely resumed.
  • A bad transformation can affect thousands or millions of records.
  • Customers need understandable status, error reports, and rerun procedures.
  • Timelines matter, but correctness, auditability, and reversibility matter more.
  • You can improve existing tooling pragmatically rather than proposing a rewrite.

Frame your answers around these principles:

  1. Make work idempotent.
  2. Validate before mutation.
  3. Use transactions at the right scope.
  4. Checkpoint and make retries safe.
  5. Preserve provenance and produce actionable errors.
  6. Separate transformation logic from I/O and orchestration.
  7. Measure and observe every stage.
  8. Prefer a safe, operable solution over an elegant but risky one.

Suggested Schedule

Time Topic Outcome
0:00–0:35 Migration architecture and terminology Speak fluently about ETL, mappings, validation, checkpoints, idempotency
0:35–1:20 Django/Postgres migration implementation Be ready for system-design and coding questions
1:20–1:50 AWS batch/distributed workflows Connect your DevOps background directly to migration operations
1:50–2:25 Metadata, XML/CSV, and data-integrity scenarios Cover domain-specific gaps
2:25–3:10 Prepare behavioral stories Have concise examples for ownership, debugging, and stakeholder communication
3:10–3:45 Practice questions / mini design exercise Rehearse concise, senior-level answers

If you only have two hours, prioritize sections 1, 2, 3, and 5.

1. Migration Concepts and Vocabulary — 35 minutes

Core migration pipeline

A solid default model:

Extract → Profile → Validate → Transform → Stage → Load → Reconcile → Report

Extract

Read from source systems: CSV/XML exports, APIs, S3 objects, legacy database dumps.

Important concerns:

  • Encoding and delimiter issues
  • Large files / streaming parsing
  • Source versioning
  • Immutable source-file retention
  • Recording file checksums and source identifiers

Profile

Understand the incoming data before loading it:

  • Row counts
  • Null rates
  • Unique-value counts
  • Date ranges
  • Invalid controlled-vocabulary values
  • Unexpected schema changes
  • Duplicates and identifier collisions

A strong phrase:

I would treat source profiling as a first-class migration stage, not just an implementation detail. It lets us identify data-quality problems before we create partial customer state in the destination system.

Validate

Distinguish two validation categories:

  • Structural validation: required columns, XML schema shape, parseability, data types, file format.
  • Business validation: required metadata, allowed vocabularies, referential integrity, tenant ownership, identifier uniqueness.

Useful distinction:

  • Fatal errors: invalid file shape, wrong tenant, incompatible schema version. Do not start the load.
  • Record-level errors: missing optional metadata, invalid vocabulary value, malformed date. Quarantine/report the record while allowing valid records to continue, depending on agreed policy.

Transform

Mapping legacy values to new-system representations:

def transform_record(source: dict, mappings: MappingConfig) -> AssetInput:
    return AssetInput(
        external_id=source["legacy_id"],
        title=clean_text(source.get("title")),
        creator=normalize_creator(source.get("author")),
        resource_type=mappings.resource_types.get(source.get("type")),
        metadata=build_metadata(source),
    )

Good design traits:

  • Pure, testable transformation functions
  • Explicit mapping/version configuration
  • Preserved original source values where needed
  • Clear handling for unknown values
  • No hidden database calls in parsing/transformation code unless necessary

Stage

For complex or high-volume work, load normalized source records into a staging table before loading production tables.

Benefits:

  • Auditability
  • Easier replay/reprocessing
  • SQL-based reconciliation
  • Separation of source parsing from target writes
  • Ability to review failures without reparsing source data

Load

Use bulk operations carefully, while preserving a way to identify which source records correspond to target records.

Reconcile

A migration is not complete when the job says “success.” It is complete when expected results match actual results.

Examples:

Source records:             100,000
Valid after validation:      99,850
Loaded successfully:         99,820
Quarantined:                     30
Unexpected failures:              0

Also reconcile:

  • Counts by collection / tenant / resource type
  • File/object counts
  • Relationships and child records
  • Checksums where files are copied
  • Metadata completeness
  • Sampled visual/UI verification

Concepts to be able to define quickly

Concept Interview-ready definition
Idempotency Running the same migration or batch more than once produces the same final state, without duplicate records or corrupt updates.
Checkpointing Persisting job and batch progress so work can resume safely after failure.
Upsert Insert a record if it does not exist; otherwise update it, usually keyed by a stable external identifier and tenant.
Reconciliation Verifying that source, staged, and target data agree in count and meaningful content after a load.
Data provenance Recording where a migrated value came from: source system, file, row/path, original identifier, transformation version, and load time.
Quarantine / dead-letter Isolating records that cannot be processed, along with actionable error context, without necessarily failing an entire migration. Dead Letter Queue
Controlled vocabulary A governed list of allowed terms, such as resource type, language, rights statement, or subject category.
Referential integrity Ensuring relationships point to valid records: e.g., assets link to a valid collection and tenant.

2. Django and PostgreSQL Prep — 45 minutes

Django management command design

They may ask how you would build or improve a migration command.

A good structure:

management command
  └── orchestration service
        ├── source reader
        ├── validator
        ├── transformer
        ├── staging/repository layer
        ├── batch loader
        ├── checkpoint repository
        └── reporting/metrics

Keep the management command thin:

class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument("--migration-id", required=True)
        parser.add_argument("--resume", action="store_true")
        parser.add_argument("--dry-run", action="store_true")

    def handle(self, *args, **options):
        service = MigrationService(...)
        result = service.run(
            migration_id=options["migration_id"],
            resume=options["resume"],
            dry_run=options["dry_run"],
        )
        self.stdout.write(self.style.SUCCESS(result.summary()))

Mention useful command capabilities:

  • --dry-run
  • --limit
  • --resume
  • --from-checkpoint
  • --tenant
  • --input-uri
  • --validate-only
  • --report-path
  • explicit confirmation for production/destructive runs

Transactions: avoid one giant transaction

A senior answer should call this out:

I generally would not wrap a multi-hour migration in one database transaction. It creates long lock durations, large rollback costs, and operational risk. I would use bounded batches, with each batch committed atomically and checkpointed only after its successful commit.

Pattern:

for batch in batched(records, size=1000):
    with transaction.atomic():
        load_batch(batch)
        save_checkpoint(migration_id, batch.last_source_offset)

Considerations:

  • A batch should be small enough to retry safely.
  • Checkpoint update must be coordinated with the completed database write.
  • External side effects—such as S3 copies or API calls—need separate handling because they cannot be rolled back by PostgreSQL.

For external effects, discuss either:

  • an outbox pattern, or
  • a persisted per-record state machine such as PENDING → LOADED → FILE_COPIED → COMPLETE.

bulk_create, bulk_update, and their tradeoffs

Know these points:

  • bulk_create() reduces round trips and is useful for new rows.
  • It may not invoke application-level behavior you would get from per-object .save() workflows; validate model and Django-version-specific behavior before relying on signals/hooks.
  • bulk_update() is useful for known existing rows but can generate large SQL statements.
  • Use sensible batch sizes and measure them.
  • Bulk writes should not bypass required validation or tenant scoping.
  • If parent IDs are required for child objects, load parents first, map IDs, then load children.

PostgreSQL upserts

Typical approach:

INSERT INTO asset (
    tenant_id,
    external_id,
    title,
    metadata,
    source_updated_at
)
VALUES (...)
ON CONFLICT (tenant_id, external_id)
DO UPDATE SET
    title = EXCLUDED.title,
    metadata = EXCLUDED.metadata,
    source_updated_at = EXCLUDED.source_updated_at
WHERE asset.source_updated_at <= EXCLUDED.source_updated_at;

Points to mention:

  • The conflict target requires an appropriate unique constraint, often (tenant_id, external_id).
  • External IDs must be stable and scoped appropriately.
  • Decide whether migrations are insert-only, update-only, or synchronizing.
  • Avoid silently overwriting newer destination edits.
  • Store migration/source version information to make update rules explicit.

JSONB

Be ready to say:

  • JSONB is useful for flexible or source-specific metadata.
  • It should not replace relational columns for frequently queried or integrity-critical fields.
  • Use GIN indexes for containment-style JSONB queries when justified by query patterns.
  • Normalize keys and data types during ingestion; otherwise JSONB becomes a consistency trap.
  • Preserve original/raw metadata separately when it is valuable for traceability.

Example:

CREATE INDEX asset_metadata_gin
ON asset
USING GIN (metadata jsonb_path_ops);

Common data integrity debugging sequence

If asked, “A migration produced missing or duplicate records. What do you do?”

  1. Pause or prevent additional runs if continuing may amplify corruption.
  2. Identify the migration version, input file/version, tenant, batch range, and time window.
  3. Compare source, staging, target, and error counts.
  4. Check idempotency/conflict-key behavior.
  5. Check checkpoint state versus actual committed database state.
  6. Inspect representative failed and duplicate records.
  7. Determine whether the issue is transformation, mapping, target constraints, concurrency, or retry behavior.
  8. Create a targeted repair plan, preferably based on migration IDs and source identifiers.
  9. Test repair in a non-production-like environment.
  10. Reconcile again and document rerun steps.

3. AWS and Distributed Workflow Prep — 30 minutes

This is a chance to make your current role highly relevant.

A credible AWS architecture

Customer export uploaded to S3
        ↓
S3 event notification
        ↓
SQS queue
        ↓
Worker orchestration / ECS Fargate task
        ↓
Validate and profile source
        ↓
Load staging + process batches into PostgreSQL
        ↓
Store reports in S3 and migration state in PostgreSQL
        ↓
Notify operators / update migration UI

Key operational concerns

S3 events are at-least-once

An event can be delivered more than once. Therefore:

  • Deduplicate by bucket/key/version ID or object checksum.
  • Ensure job creation and processing are idempotent.
  • Never assume one event equals exactly one processing attempt.

SQS visibility timeout

For long-running jobs:

  • Set an appropriate visibility timeout.
  • Extend it while the worker is alive if needed.
  • Use a dead-letter queue for repeatedly failing messages.
  • Prefer one message that triggers a tracked job rather than trying to process a huge file entirely within an SQS-message lifecycle.

ECS/Fargate

Useful operational points:

  • Use task-level CPU/memory appropriate for parsing and batch loading.
  • Pass migration ID/configuration via environment variables or task overrides.
  • Send structured logs to CloudWatch with migration ID, tenant ID, source file, batch number, and correlation ID.
  • Use task exit codes, CloudWatch alarms, and application-level status updates.
  • Ensure tasks can be stopped and resumed safely.

Exactly-once is usually not realistic end-to-end

A strong senior answer:

I would not promise exactly-once execution across S3, SQS, ECS, and PostgreSQL. I would design for at-least-once delivery and make individual operations idempotent through stable external IDs, unique constraints, persisted status, and safe retry behavior. Exactly Once vs At Least Once You are here

This branch begins here Exactly Once vs At Least Once
Human dan

What is the difference between exactly-once execution and at-least-once?

AI gpt-5.6-terra

At-least-once execution means a task or message will be processed one or more times. Duplicates are possible, so the application must tolerate retries safely.

Exactly-once execution means a task’s effect occurs only once, even if infrastructure retries, crashes, or redelivers messages.

At-least-once example

Suppose an SQS message says:

{ "migration_id": "mig_123", "file": "customers.csv" }

A worker receives it, loads records into PostgreSQL, then crashes before deleting the SQS message. Once the visibility timeout expires, SQS delivers the message again.

The migration may run twice.

That is normal at-least-once behavior. To handle it safely:

  • Use stable source/external IDs.
  • Enforce a unique constraint, e.g. (tenant_id, external_id).
  • Use upserts or insert-if-not-exists behavior.
  • Persist migration/job state and checkpoints.
  • Make side effects, such as copying an S3 object or creating a target record, safe to repeat.

The worker may execute twice, but the final database state should be correct and non-duplicated.

Exactly-once example

In an ideal exactly-once system, even if the worker crashes after loading the data but before acknowledging the message, the redelivered message would not cause the migration’s effects to happen again.

This is difficult across separate systems because there is no single transaction spanning:

  • SQS message acknowledgement
  • ECS/Fargate worker execution
  • PostgreSQL writes
  • S3 object operations
  • External REST API calls

For example, a worker cannot atomically both:

  1. Commit a PostgreSQL transaction, and
  2. Tell SQS to permanently remove the message.

If it commits the database write and crashes before the SQS delete, the message returns. If it deletes the message first and crashes before the database commit, the work may be lost.

Practical framing for an interview

A good answer:

In distributed systems, I assume delivery is at least once. Rather than relying on exactly-once execution, which is difficult or impossible across independent services, I design for effectively-once outcomes. That means idempotent handlers, durable checkpoints, unique constraints, upserts, and clear status tracking. Retries may happen, but they do not create duplicate or inconsistent customer data.

The distinction between exactly-once execution and effectively-once outcome is important:

  • Exactly-once execution: code literally runs once.
  • Effectively-once outcome: code may run repeatedly, but persistent business effects occur once.
Human dan

Can you give me a technical explanation of SQS. How it works and what kinds of problems it's trying to solve?

AI gpt-5.6-terra

Amazon SQS (Simple Queue Service) is a managed message queue. Its main job is to decouple producers of work from consumers of work.

Instead of one service directly calling another and waiting for it to finish, a producer puts a message on a queue. One or more workers consume messages later and process them asynchronously.

Producer → SQS queue → Worker(s)

For a migration system:

User uploads CSV to S3
        ↓
A job is created and a message is sent to SQS
        ↓
Fargate worker receives the message
        ↓
Worker validates/processes the file and updates migration status

This design lets uploads, job creation, and processing proceed independently.

What problems SQS solves

1. Decoupling services

Without a queue:

Web application → directly invokes migration worker → waits

Problems:

  • The user-facing request may time out.
  • The web application is tied to worker availability.
  • A worker outage can cause user-facing failures.
  • Scaling workers requires changing the web application’s behavior.
  • A large migration can consume resources needed for normal requests.

With SQS:

Web application → enqueue migration job → respond quickly
Worker → processes job asynchronously

The web application only needs to know that the request was accepted.

2. Buffering bursts of work

Suppose 100 customers submit migration files around the same time, but your system can safely run only 10 migration workers.

SQS absorbs the burst:

100 queued jobs → 10 workers steadily process them

The queue provides backpressure. Work waits rather than overwhelming PostgreSQL, an external API, or a downstream indexing service.

In migration work, this is particularly important because bulk loads can:

  • Exhaust database connections.
  • Cause lock contention.
  • Saturate database I/O.
  • Trigger API rate limits.
  • Consume CPU/memory while parsing large XML or CSV files.

You can control concurrency by limiting ECS task count, worker count, or batch throughput.

3. Reliable asynchronous delivery and retry

SQS stores messages durably and keeps them available until a consumer successfully processes them and deletes them.

A simplified lifecycle:

1. Producer sends message.
2. SQS stores it durably.
3. Consumer receives message.
4. Message becomes temporarily invisible.
5. Consumer processes it.
6. Consumer deletes the message to acknowledge success.

If the worker crashes before deletion:

1. Worker receives message.
2. SQS hides it for the visibility timeout.
3. Worker crashes or times out.
4. Visibility timeout expires.
5. SQS makes the message available again.
6. Another worker can retry it.

That produces at-least-once delivery: a message can be delivered more than once.

Important SQS concepts

Queue

A queue is the durable container holding messages awaiting processing.

Common configuration includes:

  • Retention period
  • Visibility timeout
  • Delivery delay
  • Long polling duration
  • Dead-letter queue policy
  • Encryption and access policy

Messages can remain in SQS for up to 14 days, depending on queue retention settings.

Message

A message contains a payload, typically JSON, plus SQS metadata.

Example:

{
  "event_type": "migration.requested",
  "migration_id": "mig_01JXYZ",
  "tenant_id": "tenant_123",
  "input": {
    "bucket": "migration-inputs",
    "key": "tenant_123/mig_01JXYZ/export.csv",
    "version_id": "abc123"
  },
  "requested_at": "2026-08-07T14:25:00Z"
}

Keep messages relatively small. SQS has a 256 KB message size limit. Put large payloads—CSV/XML files, manifests, reports—in S3 and send only a reference in SQS.

Producer

The producer sends a message using SendMessage.

Potential producers include:

  • A Django app after creating a migration record.
  • A Lambda function triggered by an S3 upload.
  • A scheduled process.
  • Another worker that creates follow-up work.

A producer should usually create durable database state before or alongside enqueueing work. Otherwise, the worker may receive a message referring to a migration job that does not exist.

A common reliability concern is the dual-write problem:

1. Save migration job in PostgreSQL.
2. Send SQS message.

If step 1 succeeds and step 2 fails, the job exists but is never processed.

1. Send SQS message.
2. Save migration job in PostgreSQL.

If step 1 succeeds and step 2 fails, a worker may receive a message for a nonexistent job.

A common solution is a transactional outbox:

Within one PostgreSQL transaction:
  - Create/update migration job.
  - Write an outbox event row.

Separate publisher:
  - Reads unsent outbox rows.
  - Sends each event to SQS.
  - Marks it sent.

The publisher itself can retry safely; duplicate messages are acceptable if consumers are idempotent.

Consumer / worker

The consumer polls the queue with ReceiveMessage.

When it receives a message, SQS does not immediately remove it. Instead, it hides it for the configured visibility timeout.

The worker should:

  1. Validate the message.
  2. Claim or lock the associated migration job in the database.
  3. Process the work.
  4. Persist successful state/checkpoints.
  5. Delete the SQS message only after it is safe to consider the work accepted or completed.

For a very large migration, the message typically triggers a durable job; it should not necessarily represent the whole job as a single uninterruptible unit.

For example:

SQS message: “start or resume migration mig_123”
Database: tracks each batch and current checkpoint
Worker: processes a bounded amount of work, then schedules/continues safely

Visibility timeout

The visibility timeout is the amount of time after receiving a message during which other consumers cannot receive it.

Example:

Queue visibility timeout: 10 minutes

12:00 — Worker A receives message.
12:00–12:10 — Message is hidden from other workers.
12:05 — Worker A succeeds and deletes it.

Or:

12:00 — Worker A receives message.
12:07 — Worker A crashes.
12:10 — Visibility timeout expires.
12:10 — Message becomes available for retry.

If processing may take longer than the visibility timeout, the worker can call ChangeMessageVisibility to extend it.

Risks:

  • Too short: a job may still be running when the message reappears, resulting in concurrent duplicate processing.
  • Too long: recovery from a crashed worker is slow.
  • No idempotency: duplicate delivery can create duplicate records or repeated external side effects.

For batch jobs, a common design is to use a reasonably bounded unit of work per message or have the message initiate a tracked job rather than keep a message invisible for hours.

Message deletion / acknowledgement

SQS uses an explicit delete model.

Receiving a message is not acknowledgement. A successful worker must call DeleteMessage using the receipt handle returned by that specific receive operation.

If it does not delete the message, the message reappears after the visibility timeout.

Because the worker may successfully commit a database change and then fail before deletion, it must assume the message can be delivered again. That is why idempotency is central to SQS consumers.

Dead-letter queue (DLQ)

A DLQ receives messages that fail processing repeatedly.

For example:

Main migration queue:
  maxReceiveCount = 5

Message fails five times:
  → moved automatically to migration-dlq

Typical reasons:

  • Invalid message schema
  • Missing S3 object
  • Unsupported source file version
  • Persistent database constraint violation
  • A software bug
  • An unavailable dependency

A DLQ prevents “poison messages” from being retried forever and consuming worker capacity.

Important operational practice:

  • Monitor DLQ depth.
  • Alert on new DLQ messages.
  • Preserve enough metadata to investigate.
  • Fix the underlying issue before replaying messages.
  • Redrive messages deliberately after remediation.

For migration jobs, it is often useful to update a database job record to FAILED or NEEDS_ATTENTION with a meaningful error report before allowing the message to reach the DLQ.

Standard queues vs FIFO queues

Standard queues

Standard queues are the usual choice.

They provide:

  • Very high throughput
  • At-least-once delivery
  • Best-effort ordering

“Best-effort ordering” means messages may arrive out of order, and duplicates are possible.

For independent migration jobs, this is usually fine.

Migration A requested
Migration B requested

Possible processing order:
B, A, A

Your application should not rely on order.

FIFO queues

FIFO queues provide stronger ordering and deduplication guarantees.

They support:

  • Ordered processing within a message group
  • Exactly-once message processing semantics within SQS’s defined deduplication window and constraints
  • Lower throughput characteristics than standard queues, depending on configuration

A MessageGroupId defines the ordered stream:

MessageGroupId = tenant_123

Messages for tenant_123 are processed in order, while other tenant groups may process concurrently.

However, FIFO does not magically guarantee exactly-once business effects across PostgreSQL, S3, and external APIs. A consumer can still commit a database change and crash before deleting the message. You still need idempotent consumers and database constraints.

For migration systems, FIFO may be useful if:

  • Only one migration per tenant can safely run at once.
  • Operations must occur in strict order for a collection.
  • Ordering requirements outweigh reduced parallelism.

Often, explicit tenant/job locking in PostgreSQL is enough, and a standard queue remains simpler.

SQS and migration-job design

A robust pattern is:

1. Django creates MigrationJob(status=PENDING).
2. Django stores the input file reference and configuration/version.
3. An outbox publisher sends {migration_id} to SQS.
4. Worker receives the message.
5. Worker atomically claims the job if it is eligible to run.
6. Worker transitions status to VALIDATING / RUNNING.
7. Worker processes batches, storing checkpoints and metrics.
8. Worker marks COMPLETED, COMPLETED_WITH_ERRORS, or FAILED.
9. Worker deletes the SQS message.

Example state model:

PENDING
  → VALIDATING
  → RUNNING
  → COMPLETED

PENDING
  → VALIDATING
  → REJECTED

RUNNING
  → PAUSED
  → RUNNING

RUNNING
  → FAILED
  → RUNNING       # deliberate retry/resume

RUNNING
  → COMPLETED_WITH_ERRORS

The database is the source of truth for job state. SQS is the delivery and scheduling mechanism, not the authoritative record of whether a migration completed.

Key limitations and non-goals

SQS is not:

  • A relational database or job-history system
  • A workflow engine with rich dependency management
  • A guarantee that a worker completed business logic exactly once
  • A solution for storing large files
  • A broadcast/event-stream system by itself—one queue message is generally handled by one consumer, not all consumers

If multiple independent systems need the same event, use SNS, EventBridge, or separate queues subscribed to a fan-out mechanism.

Strong concise interview answer

SQS is a durable managed queue used to decouple work producers from asynchronous consumers. It buffers bursts, lets workers scale independently, and retries work when consumers fail. A worker receives a message, which is hidden for a visibility timeout, processes it, and explicitly deletes it on success. If it crashes or does not acknowledge, the message becomes visible and may be delivered again, so standard SQS consumers must be idempotent. In a migration system, I would use SQS to trigger or schedule durable migration jobs, store job state and checkpoints in PostgreSQL, keep source files and reports in S3, and use a DLQ plus monitoring for repeatedly failing jobs.

Explore conversation