Making the best from RDS by using RDS Proxy

In the previous post, we covered how to set up a basic CRUD service with Aurora Cluster and start working with it. However, as your application grows, you may occasionally notice that lambdas fail to access the database.
So what is causing these problems?

Typically, when a lambda is connecting to the database, it will try to acquire a new connection to the database with each invocation. Eventually, when lambda is trying to acquire a new connection, the database may have no available resources. The lambda has to wait for previous connections to be closed, possibly leading to connection timeouts.
Lambdas may also be ended or suspended abruptly - not allowing them to close opened connections properly, thus amplifying the problem.

RDS proxy provides an out-of-process connection pool to the database, that allows many lambdas to connect, using a small number of connections that are shared between lambda invocations.

So how does it work?

Connection pooling is an optimization that enables applications to share and reuse database connections.

Opening and closing a new database connection is an expensive operation. Moreover, each database engine has its own limits on how many connections it can maintain.

By utilizing connection pooling, when a lambda (or another client) connects to the database it will try to acquire a connection from the pool. If no connections are available, a new connection will be created.
When a connection is no longer used - it is the pool's responsibility to maintain the connection and decide when to close it.

Hence, the connection pool decouples the lambda's connection acquisition from database limitations and lets us maintain a reasonable amount of database connections while letting lambdas acquire connections in a timely fashion.

To conclude, the RDS proxy gives us:

• Performance, by acquiring connections faster and using fewer resources.
• Reliability, by minimizing resource exhaustion.

Currently, AWS is supporting connecting RDS Proxies to:

• Aurora Cluster
  • Aurora MySQL
  • Aurora PostgreSQL
• RDS Instances
  • MySQL
  • PostgreSQL

Limitations:

• RDS proxy cannot be publicly accessible, even if the RDS instance or cluster is.
• RDS Proxy is not supported by Aurora Serverless.
• RDS Proxy can only be associated with a single RDS instance or cluster.

With Altostra you can now:

• Add RDS Proxy to existing Altostra project that contains an RDS Database.
• Deploy your own RDS Database and connect to it using RDS Proxy

In this blog post, we will add an RDS Proxy and connect to your RDS Database.

Prerequisite - A deployed project with an RDS database as described in How to use Aurora Cluster in your Altostra projects

Note: adding a proxy to an existing project will not re-create the database resource, so no data is lost.

Adding RDS Proxy

To add an RDS Proxy we will need to set VPC configuration and connection pool settings.

Configuring the VPC

First, we need to set VPC so RDS Proxy will have access to your database.
That means that the RDS Proxy and the database need to be in the same VPC.

The most flexible way to achieve this is to put your VPC configuration into Altostra environments parameters

and then set it in VPC configuration section on both resources

Subnets

The subnets must meet the following criteria:

• All subnets must belong to same VPC
• The target database must be accessible from that VPC
• The proxy must have access to at least two subnets

Note: An RDS Proxy can be set in more subnets than the database if these subnets meet the criteria above.
For example, you may want your database to be in specific subnets, but be accessible from more subnets via the proxy

Acquiring subnet IDs

Depending on the database configuration, the subnet IDs can be acquired from the following sources:

• If you have set a VPC configuration on the RDS database, copy them from the database's Advanced -> VPC Config
  
  
• If the database VPC configuration is not set, the database is created in your default VPC.
  Use the Default VPCs guide to get the default subnets in your default VPC
  It is recommended that you store the default VPCs as environment parameters for reusability.

Security group

RDS Proxy always uses default port, regardless of the port you chose for your database. You may use the same security group for both the RDS Proxy and the database, but you must make sure that the security group has an inbound rule for the default DB port (for the RDS proxy) as well as for the configured database port.

To get the security group ID:

1. Log in to AWS VPC Console
2. Select or create a security group in the VPC.
3. Copy the Security group ID from the security group details page.
   

Configuring the connection pool

Using the advanced configuration, you can control how the connection pooling is done by RDS proxy

• Idle client connection timeout:
  The number of seconds that a connection to the proxy can be inactive before the proxy disconnects it.
  A client connection is considered idle when the application doesn't submit a new request within the specified time after the previous request completed. The underlying database connection stays open and is returned to the connection pool. Thus, it's available to be reused for new client connections.
  Consider lowering the idle client connection timeout if you want the proxy to proactively remove stale connections. If your workload is spiking, consider raising the idle client connection timeout to save the cost of establishing connections.
  Default: 1800 (30 minutes)
• Connection borrow timeout:
  The number of seconds for a proxy to wait for a connection to become available in the connection pool. Only applies when the proxy has opened its maximum number of connections and all connections are busy with client sessions.
  Default: 120
  Constraints: 0 - 3600
• Max connections percent:
  The maximum size of the connection pool. The value is expressed as a percentage of the max_connections setting for the RDS DB instance or Aurora DB cluster used by the target group.
  Default: 100
  Constraints: 0 - 100
• Max idle connections percent:
  Controls how actively the proxy closes idle database connections in the connection pool. The value is expressed as a percentage of the max_connections setting for the RDS DB instance or Aurora DB cluster used by the target group. With a high value, the proxy leaves a high percentage of idle database connections open. A low value causes the proxy to close more idle connections and return them to the database.
  Default: Half of Max connections percent or 50
  Constraints: 0 - Max connections percent

Making things work

Now, we can add the proxy and connect a lambda to the Database through the proxy.

With Altostra, all you need is to add an RDS Proxy resource into you altostra project. The connected lambda will get Proxy's endpoint as an environment variable.

  // retrieving host and port
  const [host, port] = process.env.DB_CLUSTER01.split(':')

Summary

Amazon RDS proxy is an out-of-process connection pool that helps to improve application scalability, performance, and reliability.
With Altostra, you can achieve this with just a few clicks!