Auto-generate CloudFormation with no-code and no commitment 🚀
Try Now

Use case: Claim Check pattern using serverless architecture

Learn how to implement the Claim Check pattern for processing large data while leveraging the benefits of managed services.
Yev Krupetsky

August 18 2021 · 20 min read

SHARE ON

Use case: Claim Check pattern using serverless architecture

In this article, we'll implement the Claim Check pattern using serverless architecture.

To demonstrate, let’s build a service that analyzes images using ML and provides labels of what it finds. We want to let users upload their images and have the results stored in a DynamoDB table.

The problem is that we can’t just pass the image data along with every API call, queue message, or other forms of communication. We may exceed message queue limits, cause large delays, and increase costs. If you think about the process, it’s most likely that only a single component needs the image data while others will just pass the data along, wasting resources.

We efficiently solve the problem by storing the file in a dedicated store until it’s needed and only passing a reference that points to it.

This pattern is available as an Altostra project template

Architecture

architecture

Application clients submit files by uploading them to the S3 bucket files—clients can be desktop applications, web APIs, functions, queues, or other methods.

Check out this post on how to upload large files to an S3 bucket.

The Lambda function handle-new-file triggers whenever a new file is uploaded to the bucket. The function validates that the file format and image size are within our acceptance criteria. For every valid image, the function will post a job message to an SQS queue. The message will contain the job details and a reference to the file stored in the bucket.

The Lambda function process-image handles messages as they appear in the SQS queue. It uses the file reference to send the image to AWS Rekognition for label detection and stores the results in the DynamoDB table results.

Setting up

I’m using Altostra to build our service. It allows me to quickly design and deploy the cloud infrastructure, freeing more time to code and test.

What you’ll need

Create the project

We’ll start by creating a new empty Altostra project.

  1. On the projects page, click Create Project.
  2. Enter a name.
  3. Click Create.

Altostra will create the new project and a Git repository for it.

Open the project on your machine

Clone the project to the local machine and open it in Visual Studio Code:

  1. On the project page that you created, click Clone, select your preferred protocol and copy the link.
  2. Clone the repository to your local machine.
  3. Open the project directory in VS Code.
clone project

Login to Altostra CLI

The CLI requires authentication to work with your Altostra account.

Simply run:

$ alto login

Building the service

For simplicity, I keep the default configuration for most of the resources and connections. You may want to makes changes to suit your needs.

Handling new files

Add the resources

  1. Add an S3 bucket to store the image files. Call it files and keep the default configuration.
  2. Add an SQS queue for job messages with reference to the image file. Call it jobs and keep the default configuration.
  3. Add a Lambda function to process new image files and create job messages. Call it handle-new-files and keep the default configuration.
handling new files

Connect the resources

  1. Connect the files bucket to the handle-new-image function—this creates a trigger connection. Adjust the connection configuration to trigger only for files whose path ends with .png .jpeg or .jpg to avoid invoking the Lambda function for unsupported file types.
    bucket trigger
  2. Connect the handle-new-image function to the jobs queue—this creates an access connection. Adjust the connection configuration to allow write-only access since it only needs post messages to the queue.
    queue access

As an optional step for error handling, you can add another SQS queue (I named it failed-jobs) and connect the jobs queue to it—it will create an errors connection. This assigns the failed-jobs queue as the DLQ for the jobs queue—I've set it to retry failed messages 3 times.

dlq settings

Submitting image for analysis

Now that we have jobs queueing up let’s create the process to submit them for analysis and store the results.

Add the resources

  1. Add a Lambda function to handle messages from the jobs queue. Call it process-message and change its timeout configuration to 30.
  2. Add a DynamoDB table to store the analysis results. Call it results and keep the default configuration.
  3. Add a Rekognition Service. The service isn't configurable, it only tells Altostra to create IAM policies to allow the Lambda function to call it.
handling new files

Connect the resources

  1. Connect the jobs queue to the process-image function.
  2. Connect the process-image function to the results table and keep the default configuration.
  3. Connect the process-image function to the Rekognition Service—this creates an access connection to allow the function access to the service.

The code

With the infrastructure in place, we can add the code to our functions.

The code in this article is for demonstration only and doesn’t cover all security, logging, configuration, error handling, and other practices that you want in production code.

handle-new-file.js

const util = require('util')
const AWS = require('aws-sdk')

const SUPPORTED_IMAGES = ["jpeg", "jpg", "png"]
const MAX_IMAGE_SIZE = 10_000_000

// Implicitly injected by Altostra
const QUEUE = process.env.QUEUE_QUEUE01

const sqs = new AWS.SQS()

module.exports.handler = async (event, context) => {
  console.log("Handling new file.", util.inspect(event, { depth: 3 }))

  const s3data = event.Records[0].s3
  const key = decodeURIComponent(s3data.object.key.replace(/\+/g, " "))

  const fileExt = key.match(/\.([^.]*)$/)
  if (!fileExt) {
    console.log(`Aborting. Can't detect file extension.`)
    return
  }

  const imageType = fileExt[1].toLowerCase()
  if (!SUPPORTED_IMAGES.includes(imageType)) {
    console.log(`Aborting. Unsupported image type: ${imageType}`)
    return
  }

  const size = s3data.object.size
  if (size > MAX_IMAGE_SIZE) {
    console.log(`Aborting, file size is ${size}, which is over the limit ${MAX_IMAGE_SIZE}`)
    return
  }

  const fileReference = {
    bucketArn: s3data.bucket.arn,
    bucketName: s3data.bucket.name,
    objectKey: s3data.object.key
  }

  const jobMessage = {
    id: s3data.object.key,
    file: fileReference,
    timestamp: Date.now()
  }

  try {
    const result = await sqs.sendMessage({
      QueueUrl: QUEUE,
      MessageBody: JSON.stringify(jobMessage)
    }).promise()

    console.log(`Job message sent.`, result.MessageId)
  } catch (err) {
    console.log(`Failed to send message to queue.`, err)
  }
}

process-image.js

const AWS = require('aws-sdk')

// Implicitly injected by Altostra
const TableName = process.env.TABLE_TABLE01

const rekognition = new AWS.Rekognition()
const ddb = new AWS.DynamoDB.DocumentClient()
const s3 = new AWS.S3()

module.exports.handler = async (event, context) => {
  for (const record of event.Records) {
    let message
    try {
      message = JSON.parse(record.body)
    } catch (err) {
      console.log(`Discarding job. Failed to JSON parse the message.`, record)
      continue
    }

    let image
    try {
      image = await s3.getObject({
        Bucket: message.file.bucketName,
        Key: message.file.objectKey
      }).promise()
    } catch (err) {
      console.log(`Failed to get image data.`, err)
      return
    }

    let labelDetectResult
    try {
      labelDetectResult = await rekognition.detectLabels({
        Image: {
          Bytes: image.Body
        },
        MaxLabels: 10
      }).promise()

      console.log(`Detected labels`, labelDetectResult.Labels)
    } catch (err) {
      console.log(`Failed to submit image for analysis.`, err)
      throw err
    }

    try {
      await ddb.put({
        TableName,
        Item: {
          pk: `${message.file.objectKey}#${message.timestamp}`,
          labels: labelDetectResult.Labels,
          file: message.file
        }
      }).promise()
    } catch (err) {
      console.log(`Failed to store results in ${TableName}.`)
      throw err
    }
  }
}

Deploy and test

Deploy a new stack

Let's deploy our project and test it. I'll deploy a new stack blog to my Lab environment.

$ alto deploy blog --env Lab --push

Test

Let's upload some images to the files bucket:

upload files

Wait a few seconds, and look at the content of the results table:

results table

Next steps

Want to give it a try? We provide a free-forever plan for developers. Create your free account today at https://app.altostra.com.

We want to hear from you! Share your Altostra experience with us on Twitter @AltostraHQ and LinkedIn and any suggestions that can make your work with Altostra even better.

Happy coding!

By submitting this form, you are accepting our Terms of Service and our Privacy Policy

Thanks for subscribing!

Ready to Get Started?

Get Started for Free

Copyright © 2021 Altostra. All rights reserved.