Serverless Order Processing in E-Commerce: A Practical Guide with AWS Lambda & SQS

In e-commerce, a seamless and efficient order processing system is crucial to handle customer transactions, inventory updates, and sending notifications. AWS provides a powerful set of tools, including Simple Queue Service (SQS) and Lambda, to create a serverless and scalable order processing pipeline.

In this article, we’ll walk through how to implement an order processing logic where each service, such as payment processing, inventory updates, and email notifications, is dependent on the success of the previous service. We’ll be using AWS SQS, Lambda, Node.js, and a MySQL database hosted on Amazon RDS.

The Setup

Our order processing system consists of the following components:

1. SQS Queues (PaymentStatusQueue, InventoryStatusQueue, EmailNotificationQueue )

2. Lambda Functions (PaymentServiceLambda, InventoryServiceLambda , EmailServiceLambda)

3 Express Server: Handles order placement.

4. RDS (MySQL): Stores order information.

Workflow

• A customer places an order from frontend via the Express server, which stores the order in the RDS database.
• The order is placed in the PaymentStatusQueue.
• PaymentServiceLambda is triggered by a message from PaymentStatusQueue, simulates payment processing, and pushes the result to InventoryStatusQueue.
• InventoryServiceLambda is triggered by a message from InventoryStatusQueue, updates the inventory, and sends a message to EmailNotificationQueue.
• EmailServiceLambda sends an email notification and updates the order status in the RDS database.

Let’s break it down step by step.

Step 1: Setting Up the MySQL Database on RDS

First, create an RDS instance with public access enabled, allowing inbound traffic on port 3306. Once connected to the database, run the following SQL script to set up the required schema and table:

CREATE SCHEMA `sqs-test`;

CREATE TABLE orders (
  order_id INT AUTO_INCREMENT PRIMARY KEY,
  customer_name VARCHAR(100),
  order_status VARCHAR(50),
  payment_status VARCHAR(50),
  inventory_status VARCHAR(50),
  email_sent BOOLEAN DEFAULT FALSE
);

This will create an orders table where we will store the status of each order.

Step 2: Setting Up the Express Server

We need an Express server that allows customers to place orders and pushes the payment request to the PaymentStatusQueue. First, install the required packages:

npm init -y
npm install express dotenv aws-sdk mysql2 cors

AWS_REGION=your-region
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
SQS_PAYMENT_QUEUE_URL=your-payment-queue-url
RDS_HOST=your-rds-host
RDS_USER=your-db-user
RDS_PASSWORD=your-db-password
RDS_DATABASE=sqs-test

require('dotenv').config();
const express = require('express');
const AWS = require('aws-sdk');
const mysql = require('mysql2/promise');
const cors = require('cors');
const app = express();

app.use(cors());
app.use(express.json());

const sqs = new AWS.SQS({
  region: process.env.AWS_REGION,
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
});

const db = mysql.createPool({
  host: process.env.RDS_HOST,
  user: process.env.RDS_USER,
  password: process.env.RDS_PASSWORD,
  database: process.env.RDS_DATABASE
});

app.post('/place-order', async (req, res) => {
  const { customerName } = req.body;

  // Insert order into the DB
  const [result] = await db.query('INSERT INTO orders (customer_name, order_status, payment_status, inventory_status) VALUES (?, ?, ?, ?)',
    [customerName, 'pending', 'pending', 'pending']);
  
  const orderId = result.insertId;

  // Push message to PaymentStatusQueue
  const message = {
    MessageBody: JSON.stringify({ orderId, customerName }),
    QueueUrl: process.env.SQS_PAYMENT_QUEUE_URL
  };
  
  await sqs.sendMessage(message).promise();
  res.json({ message: 'Order placed successfully', orderId });
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

This code handles order placement by saving the order in the MySQL database and sending a message to the PaymentStatusQueue.

Step 3: Creating SQS Queues in AWS

In your AWS account, create the following three standard SQS queues:

1. PaymentStatusQueue: Receives messages after a new order is placed.
2. InventoryStatusQueue: Receives messages after the payment is processed.
3. EmailNotificationQueue: Receives messages after the inventory is updated.

Make sure to copy the queue URLs and update your .env file accordingly.

Step 4: Writing Lambda Functions

We will create three Lambda functions to handle payment processing, inventory updates, and sending email notifications. Each Lambda function will be triggered by a message from the corresponding SQS queue.

PaymentServiceLambda

This Lambda function processes the payment and, if successful, sends a message to the InventoryStatusQueue:

const mysql = require('mysql2/promise');
const AWS = require('aws-sdk');

const db = mysql.createPool({
  host: process.env.RDS_HOST,
  user: process.env.RDS_USER,
  password: process.env.RDS_PASSWORD,
  database: process.env.RDS_DATABASE
});

const sqs = new AWS.SQS();

exports.handler = async (event) => {
  for (const record of event.Records) {
    let body = JSON.parse(record.body);
    const orderId = body.orderId;

    // Simulate payment processing
    await db.query('UPDATE orders SET payment_status = ? WHERE order_id = ?', ['completed', orderId]);

    // Send message to InventoryStatusQueue
    const inventoryMessage = {
      MessageBody: JSON.stringify({ orderId }),
      QueueUrl: process.env.SQS_INVENTORY_QUEUE_URL
    };

    await sqs.sendMessage(inventoryMessage).promise();
  }

  return { statusCode: 200, body: JSON.stringify('Payment processed successfully') };
};

InventoryServiceLambda

This function handles inventory updates and sends a message to the EmailNotificationQueue:

exports.handler = async (event) => {
  for (const record of event.Records) {
    let body = JSON.parse(record.body);
    const orderId = body.orderId;

    // Update inventory status
    await db.query('UPDATE orders SET inventory_status = ? WHERE order_id = ?', ['completed', orderId]);

    // Send message to EmailNotificationQueue
    const emailMessage = {
      MessageBody: JSON.stringify({ orderId }),
      QueueUrl: process.env.SQS_EMAIL_QUEUE_URL
    };

    await sqs.sendMessage(emailMessage).promise();
  }

  return { statusCode: 200, body: JSON.stringify('Inventory updated successfully') };
};

EmailServiceLambda

Finally, this function sends an email notification to the customer and updates the order status in the database:

exports.handler = async (event) => {
  for (const record of event.Records) {
    let body = JSON.parse(record.body);
    const orderId = body.orderId;

    // Simulate sending email notification
    await db.query('UPDATE orders SET email_sent = ? WHERE order_id = ?', [true, orderId]);
  }

  return { statusCode: 200, body: JSON.stringify('Email notification sent') };
};

To ensure that each Lambda function in this architecture has the necessary permissions to interact with the respective SQS queues, the execution roles for each Lambda function need to be assigned the following policies:

PaymentServiceLambda’s Execution Role Policy

This Lambda function needs to receive, process, and send messages to the PaymentQueue and InventoryQueue. The following policy ensures it can interact with these queues:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "sqs:ReceiveMessage",
    "sqs:GetQueueAttributes",
    "sqs:DeleteMessage",
    "sqs:SendMessage"
   ],
   "Resource": [
    "arn:aws:sqs:region:xxxxxxxxxxxx:PaymentQueue",
    "arn:aws:sqs:region:xxxxxxxxxxxx:InventoryQueue"
   ]
  }
 ]
}

InventoryServiceLambda’s Execution Role Policy

This Lambda function needs permissions to receive messages from InventoryQueue, process them, and send messages to the EmailQueue:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Sid": "Statement1",
   "Effect": "Allow",
   "Action": [
    "sqs:GetQueueAttributes",
    "sqs:DeleteMessage",
    "sqs:ReceiveMessage",
    "sqs:SendMessage"
   ],
   "Resource": [
    "arn:aws:sqs:region:xxxxxxxxxxxx:InventoryQueue",
    "arn:aws:sqs:region:xxxxxxxxxxxx:EmailQueue"
   ]
  }
 ]
}

EmailServiceLambda’s Execution Role Policy

The final Lambda function in the chain only needs to receive and process messages from the EmailQueue. The following policy grants the necessary permissions:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Sid": "Statement1",
   "Effect": "Allow",
   "Action": [
    "sqs:GetQueueAttributes",
    "sqs:ReceiveMessage",
    "sqs:DeleteMessage"
   ],
   "Resource": "arn:aws:sqs:us-east-1:992382749865:EmailQueue"
  }
 ]
}

Step 5: Creating a Lambda Layer

To reduce code duplication, you can create a Lambda layer that includes the mysql2 package and AWS SDK. This layer can be shared across all your Lambda functions.

Step 6: Setting Triggers

For each Lambda function, set the appropriate SQS queue as the trigger:

• PaymentServiceLambda → PaymentStatusQueue
• InventoryServiceLambda → InventoryStatusQueue
• EmailServiceLambda → EmailNotificationQueue

This ensures that each function is automatically triggered when a message is received in the respective queue.
Pros and Cons of Using Serverless Architecture with AWS SQS and Lambda

Pros

1. Scalability: This architecture scales automatically with the workload. As more orders are placed, SQS handles the message queuing and Lambda functions scale automatically without the need for manual intervention or additional infrastructure setup.
2. Cost Efficiency: Since AWS Lambda follows a pay-per-use model, you only pay for the compute time you use. Idle resources are not charged, unlike traditional server setups where you’d need to maintain running instances, even during low traffic.
3. Decoupling of Services: SQS acts as a buffer between services, allowing each part of the system (payment processing, inventory updates, and email notifications) to operate independently. This reduces the complexity of managing failures and enables easier updates to individual components.
4. Fault Tolerance: If one part of the system fails (e.g., payment service), the message will remain in the queue until it’s successfully processed, ensuring that no data or transactions are lost.
5. Simplified Maintenance: With AWS managing the infrastructure, you don’t need to worry about server maintenance, patching, or scaling. The serverless model reduces the operational burden significantly.

Cons

1. Cold Start Latency: AWS Lambda functions may experience cold start delays when they haven’t been invoked for a while. This can add a few hundred milliseconds to the processing time, which may not be ideal for time-sensitive applications.
2. Limited Execution Time: Lambda has a maximum execution time of 15 minutes per function. For long-running tasks, you might need to find alternative solutions or break them into smaller units of work.
3. Vendor Lock-In: By relying on AWS-specific services such as Lambda, SQS, and RDS, migrating your system to another cloud provider could become more complex and costly.
4. Complexity in Monitoring and Debugging: Managing a distributed serverless architecture requires robust monitoring and logging practices. AWS CloudWatch provides logs, but debugging across multiple Lambda functions can be more challenging than in a monolithic architecture.
5. Eventual Consistency: Since the services are decoupled and asynchronous, there might be slight delays between processing stages, leading to eventual consistency rather than immediate consistency. This might not be suitable for applications requiring instant updates.

Conclusion

This serverless architecture leveraging AWS SQS and Lambda offers a robust, scalable, and cost-efficient solution for processing orders in an e-commerce system. While it provides several advantages, such as automatic scaling and cost savings, it also comes with challenges, including cold start latency and the complexity of managing a distributed system.