Asynchronous Processing and Message Queues

Most interactions on the web are synchronous. You make a request, the server processes it, and you get a response. Loading your Instagram feed, logging into a website, making a payment, all synchronous. The request comes in, gets handled immediately, and you see the result. Simple and intuitive.

But not everything should be done synchronously. Some operations take time, minutes, even hours. Making users wait that long for an HTTP response isn't just bad UX, it's impractical. This is where asynchronous processing comes in, and it's one of the most important patterns in system design.

#TL;DR

• Synchronous processing: Request comes in, gets handled immediately, response goes out. Most web interactions work this way
• Asynchronous processing: Request is acknowledged immediately, actual work happens in the background. Essential for long-running tasks
• Message queues (also called message brokers) enable async processing by decoupling producers from consumers
• Key benefits: Better UX for long tasks, system decoupling, load buffering, fault tolerance
• Popular options: SQS (AWS), RabbitMQ (open source)
• Message lifecycle: Consume → Process → Delete. If you crash before deleting, the message reappears
• Trade-off: Messages might be delivered more than once. Design your consumers to handle duplicates

#Synchronous vs Asynchronous

When you log into a website, the flow is straightforward:

1. You enter email and password
2. Server validates credentials
3. Server creates a session
4. You get logged in

This happens in milliseconds. The HTTP request waits for the work to complete, then returns. Synchronous.

Now consider spinning up an EC2 instance. Let's say it takes five minutes. Would you make an HTTP request and just... wait? Staring at a loading spinner for five minutes? That's terrible UX. Your HTTP connection might even timeout.

Instead, this is what should happen:

1. You click "Launch Instance"
2. Server responds immediately: "Got it, your instance is being created"
3. You can close the browser, grab coffee, do whatever
4. Behind the scenes, AWS spins up the machine
5. When it's ready, the status updates to "Running"

This is asynchronous processing. The user isn't blocked waiting for the work to complete. They get immediate feedback that their request was received, and the actual processing happens in the background.

#When to Use Async Processing

Async processing shines when you have:

Long-running tasks:

• Spinning up virtual machines (minutes)
• Video encoding/transcoding (minutes to hours)
• Large file uploads and processing
• Report generation
• Data exports

Tasks that don't need immediate results:

• Sending emails and notifications
• Updating search indexes
• Syncing data to external systems
• Analytics processing

Tasks that can fail and retry:

• Payment processing callbacks
• Webhook deliveries
• Third-party API integrations

The pattern is the same: acknowledge the request immediately, do the work in the background, update the status when done.

#The Async Processing Flow

Let's trace through an EC2 launch:

1. You click "Launch Instance" in the AWS Console
2. API server creates a database entry: instance_id: abc, status: pending
3. API server pushes a task to the message queue: "Create instance abc with these specs"
4. API immediately responds: "Instance abc is being created"
5. A worker pulls the task from the queue
6. Worker provisions the actual EC2 machine (takes 5 minutes)
7. Worker updates the database: status: running
8. Next time you refresh, you see the running instance

You weren't waiting those 5 minutes. You could close your browser, check other things, and come back later. The async flow made this possible.

#Message Queues (Message Brokers)

The key component enabling async processing is the message queue, also called a message broker. Examples include AWS SQS and RabbitMQ.

A message queue is a buffer that sits between producers (who create tasks) and consumers (who process them). It enables two services to communicate without being directly connected.

#Why Use a Message Queue?

1. Decouples Systems

Your video upload service doesn't need to know about your video processing service. It just pushes a message to the queue. The processing service pulls from the queue independently.

Video Upload ──→ Queue ──→ Video Processing
     │                           │
     │  (don't know about        │
     │   each other)             │
     └───────────────────────────┘

2. Buffers Load

Let's say you have an order service that needs to send email notifications. During a flash sale, you get 10,000 orders per minute. Your email service can only handle 100 emails per minute.

Without a queue: your email service dies, orders fail, chaos.

With a queue: messages pile up in the buffer. The email service processes them at its own pace. No one crashes. Users get their emails eventually.

interface OrderNotification {
  orderId: string;
  userEmail: string;
  orderTotal: number;
}

async function onOrderCreated(order: Order): Promise<void> {
  await sqs.sendMessage({
    QueueUrl: NOTIFICATION_QUEUE_URL,
    MessageBody: JSON.stringify({
      orderId: order.id,
      userEmail: order.userEmail,
      orderTotal: order.total,
    } satisfies OrderNotification),
  });
}

The order service doesn't wait for the email to be sent. It fires the message and moves on. The email worker picks it up when it can.

3. Message Retention

Brokers retain messages for a configurable period. SQS can hold messages for up to 14 days. If your consumer is down for maintenance, messages wait in the queue. When the consumer comes back, it picks up where it left off.

4. Redelivery on Failure

This is a critical feature. When you read a message from a queue, the message isn't immediately deleted. You have to explicitly delete it after processing.

Here's the flow:

1. Consumer reads message from queue
2. Message becomes "invisible" to other consumers (visibility timeout)
3. Consumer processes the message
4. Consumer deletes the message from queue
5. Done

But what if the consumer crashes between steps 3 and 4? The message was never deleted. After the visibility timeout expires, the message reappears in the queue. Another consumer picks it up and retries.

async function processEmailNotifications(): Promise<void> {
  while (true) {
    const response = await sqs.receiveMessage({
      QueueUrl: NOTIFICATION_QUEUE_URL,
      MaxNumberOfMessages: 1,
      WaitTimeSeconds: 20,
    });

    const message = response.Messages?.[0];
    if (!message) continue;

    try {
      const notification: OrderNotification = JSON.parse(message.Body ?? "{}");

      await emailService.send({
        to: notification.userEmail,
        subject: `Order ${notification.orderId} confirmed`,
        body: `Your order total: $${notification.orderTotal}`,
      });

      await sqs.deleteMessage({
        QueueUrl: NOTIFICATION_QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
      });
    } catch (error) {
      console.error("Failed to process message:", error);
    }
  }
}

The delete happens only after successful processing. If the process crashes mid-way, the message will be retried automatically.

#The Duplicate Message Problem

This retry mechanism has a side effect: messages can be delivered more than once.

Consider this scenario:

1. Consumer reads message
2. Consumer sends the email successfully
3. Before consumer can delete the message, it crashes
4. Message reappears in queue (visibility timeout expired)
5. Another consumer picks it up
6. Email is sent again

The user gets two emails for the same order. This is a known trade-off with message queues. Your consumers need to be idempotent, they should handle duplicate messages gracefully.

async function processEmailNotification(
  notification: OrderNotification,
): Promise<void> {
  const alreadySent = await redis.get(`email_sent:${notification.orderId}`);
  if (alreadySent) {
    console.log(
      `Email already sent for order ${notification.orderId}, skipping`,
    );
    return;
  }

  await emailService.send({
    to: notification.userEmail,
    subject: `Order ${notification.orderId} confirmed`,
    body: `Your order total: $${notification.orderTotal}`,
  });

  await redis.set(`email_sent:${notification.orderId}`, "true", "EX", 86400);
}

By tracking what you've already processed, you can safely handle duplicates without sending multiple emails.

#Real-World Example: Video Processing

Video processing is the classic async use case. When a creator uploads a 4K video to YouTube, that video needs to be converted into multiple resolutions: 360p, 480p, 720p, 1080p.

This processing can take hours for long videos. The creator shouldn't wait. Here's how it works:

Step 1: Upload

async function uploadVideo(file: File, userId: string): Promise<Video> {
  const videoId = generateId();
  const s3Key = `uploads/${videoId}/original.mp4`;

  await s3.upload({
    Bucket: VIDEO_BUCKET,
    Key: s3Key,
    Body: file,
  });

  const video = await db.video.create({
    data: {
      id: videoId,
      userId,
      status: "processing",
      originalKey: s3Key,
    },
  });

  await sqs.sendMessage({
    QueueUrl: VIDEO_PROCESSING_QUEUE,
    MessageBody: JSON.stringify({
      videoId,
      s3Key,
      resolutions: ["360p", "480p", "720p", "1080p"],
    }),
  });

  return video;
}

The API uploads the original file to S3, creates a database record with status: processing, pushes a task to the queue, and returns immediately. The creator sees "Video processing..." in their dashboard.

Step 2: Processing

Workers pull tasks from the queue:

interface VideoProcessingTask {
  videoId: string;
  s3Key: string;
  resolutions: string[];
}

async function processVideo(task: VideoProcessingTask): Promise<void> {
  const localPath = `/tmp/${task.videoId}`;

  await downloadFromS3(task.s3Key, localPath);

  for (const resolution of task.resolutions) {
    const outputPath = `${localPath}_${resolution}.mp4`;
    await transcodeVideo(localPath, outputPath, resolution);

    await s3.upload({
      Bucket: VIDEO_BUCKET,
      Key: `processed/${task.videoId}/${resolution}.mp4`,
      Body: fs.createReadStream(outputPath),
    });
  }

  await db.video.update({
    where: { id: task.videoId },
    data: { status: "ready" },
  });

  await cleanup(localPath);
}

The worker downloads the original, creates multiple versions, uploads them back to S3, and updates the status. This can take an hour. The creator doesn't care, they've moved on. When they come back, the video is ready.

#Popular Message Queue Options

#AWS SQS

SQS is AWS's managed message queue. No servers to manage, scales automatically, pay-per-use.

import {
  SQSClient,
  SendMessageCommand,
  ReceiveMessageCommand,
  DeleteMessageCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });

async function sendTask(task: VideoProcessingTask): Promise<void> {
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: QUEUE_URL,
      MessageBody: JSON.stringify(task),
    }),
  );
}

async function receiveTask(): Promise<VideoProcessingTask | null> {
  const response = await sqs.send(
    new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 1,
      WaitTimeSeconds: 20,
    }),
  );

  const message = response.Messages?.[0];
  if (!message) return null;

  return {
    ...JSON.parse(message.Body ?? "{}"),
    receiptHandle: message.ReceiptHandle,
  };
}

async function deleteTask(receiptHandle: string): Promise<void> {
  await sqs.send(
    new DeleteMessageCommand({
      QueueUrl: QUEUE_URL,
      ReceiptHandle: receiptHandle,
    }),
  );
}

Key SQS features:

• Message retention up to 14 days
• Visibility timeout (configurable)
• Dead letter queues for failed messages
• FIFO queues for ordered processing

#RabbitMQ

RabbitMQ is an open-source message broker. You run it yourself (or use a managed service). More features than SQS, more control, but more operational overhead.

import amqp from "amqplib";

async function setupRabbitMQ(): Promise<amqp.Channel> {
  const connection = await amqp.connect("amqp://localhost");
  const channel = await connection.createChannel();

  await channel.assertQueue("video_processing", { durable: true });

  return channel;
}

async function sendTask(
  channel: amqp.Channel,
  task: VideoProcessingTask,
): Promise<void> {
  channel.sendToQueue("video_processing", Buffer.from(JSON.stringify(task)), {
    persistent: true,
  });
}

async function consumeTasks(channel: amqp.Channel): Promise<void> {
  channel.consume("video_processing", async (msg) => {
    if (!msg) return;

    const task: VideoProcessingTask = JSON.parse(msg.content.toString());

    try {
      await processVideo(task);
      channel.ack(msg);
    } catch (error) {
      channel.nack(msg, false, true);
    }
  });
}

Key RabbitMQ features:

• Exchanges and routing keys for complex routing
• Multiple messaging patterns (pub/sub, work queues, RPC)
• Message acknowledgments and redelivery
• Configurable durability and persistence

#Conclusion

Async processing is essential for building responsive, scalable systems. Whenever you have a task that takes more than a few seconds, consider moving it to the background.

The pattern is straightforward:

1. Receive request
2. Create a database record (status: pending)
3. Push task to message queue
4. Return immediately with acknowledgment
5. Workers process tasks and update status
6. User checks back later or gets notified

Message queues make this possible by decoupling producers from consumers, buffering load spikes, retaining messages, and handling failures through redelivery.

The trade-off is complexity. You now have queues to monitor, workers to scale, and duplicate messages to handle. But for long-running tasks, the UX improvement is worth it. Nobody wants to stare at a loading spinner for five minutes.

Start simple: set up RabbitMQ locally, push some messages, consume them. Once you understand the flow, you'll start seeing async opportunities everywhere in your systems.