Infrastructure
Building a URL Shortener with AWS, Golang, and the AWS CDK
Discover how to leverage AWS services and GoLang for creating a scalable, serverless URL shortener. This step-by-step guide features API Gateway, DynamoDB, and Lambda functions, providing you with the knowledge to simplify web navigation.
Unraveling the Digital Knot: Simplifying URLs
In today's fast-paced digital world, efficiency and simplicity drive our online interactions. URLs are the highways of the internet, guiding us to our destinations. Yet, these pathways often become convoluted, resembling more a tangled knot than a straight line. It's here that our journey begins, with the aim of transforming unwieldy URLs into sleek, manageable links. Beyond aesthetics and convenience, these simplified URLs can also track user engagement and gather click data, providing valuable insights into the effectiveness of online content.
By harnessing the power of AWS and GoLang, we'll build a serverless URL Shortening Machine from the ground up. In this tutorial, we're going to build a service that takes long URLs and squeezes them into fewer characters to make a link that is easier to share. This tutorial will be your guide towards achieving this using AWS services, including DynamoDB for storage, API Gateway for creating RESTful endpoints, Lambda functions for serverless logic, and the AWS Cloud Development Kit in Go (CDK) to define and deploy AWS resources easily.
The Blueprint
Imagine creating a system so robust and efficient that it can handle the demands of modern web traffic, yet so simple that it streamlines your digital landscape. This is the promise of using AWS services in tandem with GoLang—a dynamic duo that offers scalability, reliability, and performance.
API Gateway: The front door for our URL shortener, handling incoming requests with grace.
DynamoDB: A NoSQL database that stores our shortened URLs, ready for rapid retrieval.
Lambda Functions: The heart of our operation, processing logic to shorten and expand URLs without the need for server management.
Prerequisites
Before we begin, ensure you have the following:
- An active AWS account
- Basic knowledge of Golang
- AWS CLI installed and configured
- AWS CDK installed
Setting Up Your URL Shortener Project with AWS CDK in Go
Let's kick things off by creating a new project directory for our URL shortener and initializing it with AWS CDK using Go as our programming language. Follow these steps in your terminal:
Let's kick things off by creating a new project directory for our URL shortener and initializing it with AWS CDK using Go as our programming language. Follow these steps in your terminal:
1. Create the Project Directory
Run the following commands to create a new directory for your project and navigate into it.
mkdir url-shortener
cd url-shortener/
2. Initialize the CDK Project
Use the AWS CDK command-line tool to initialize a new CDK app specifying Go as the language. This command scaffolds a new AWS CDK project, tailored for Go development.
cdk init app --language=go
3. Preparing the Development Environment
Before diving into the code, let's ensure that our environment is ready with all necessary dependencies. Use this command to install Go Dependencies.
go mod tidy
4. Bootstrap Your AWS Environment
To prepare your AWS account for CDK deployment, bootstrap the AWS CDK with your account details and target region. Replace {{AWS_ACCOUNT_NUMBER}} with your actual AWS account number and {{AWS_REGION}} with your desired AWS region:
cdk bootstrap aws://{{AWS_ACCOUNT_NUMBER}}/{{AWS_REGION}}
5. Configuring the Deployment Region
Next, we need to explicitly define the AWS region where our URL Shortener stack will be deployed.
- Open the url-shortener.go file and locate the env() function. This is where the deployment region and account are specified.
- Update the function to set your desired AWS region, ensuring your stack deploys where you want it.
func env() *awscdk.Environment {
return &awscdk.Environment{
// Replace {{AWS_REGION}} with your preferred AWS region.
Region: jsii.String("{{AWS_REGION}}"),
}
}
6. Deploying Your Project
Now that everything is set up, it's time to deploy your URL Shortener stack to AWS. Run the following command to deploy your project. This will create the necessary resources in your specified AWS region:
cdk deploy
After the deployment completes, you can visit the AWS CloudFormation console in your chosen region ({{AWS_REGION}}). There, you will find your new URL Shortener Stack up and running.
Great! Now let's start coding! 👨💻👩💻
Structuring Your Project
A well-organized directory structure is crucial for the manageability and scalability of your project. Here's the recommended structure for your serverless URL shortener:
url_shortener/
├── url-shortner.go (the file responsible for creating our IAC(Infrastructure as a code))
├── lambda/
│ └── get-tag/ (lambda function logic responsible for returning url short code)
│ │ └── main.go
│ └── create-tag/ (lambda function logic responsible for creating new short urls)
│ └── main.go
Step 1: Setting the Stage with AWS CDK
First things first, we lay the foundation using the AWS Cloud Development Kit (CDK). This toolkit allows us to define our cloud resources in the familiar territory of GoLang code. It's like drawing up the blueprints for a digital masterpiece.
Your journey into the infrastructure code begins with the url-shortener.go file.
AWS CDK automatically generates a skeleton for you, which serves as the foundation for your CloudFormation stack:
func NewUrlShortnerStack(scope constructs.Construct, id string, props *UrlShortnerStackProps) awscdk.Stack {
var sprops awscdk.StackProps
if props != nil {
sprops = props.StackProps
}
stack := awscdk.NewStack(scope, &id, &sprops)
// The code that defines your stack goes here
return stack
}
This function is the entry point for defining your stack's resources.
DynamoDB: The Backbone for URL Storage
First on our agenda is setting up DynamoDB, a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. We choose DynamoDB for its ability to handle high throughput and low latency, which are crucial for our URL shortening service.
// Import the AWS DynamoDB Go CDK library
"github.com/aws/aws-cdk-go/awscdk/v2/awsdynamodb"
// Define the DynamoDB table for storing shortened URLs
dynamoDBTable := awsdynamodb.NewTable(stack, jsii.String("url-shortener-dynamodb-table"), &awsdynamodb.TableProps{
PartitionKey: &awsdynamodb.Attribute{
Name: jsii.String("tag"),
Type: awsdynamodb.AttributeType_STRING,
},
})
// If you want to maintain the Database anyways you can use awscdk.RemovalPolicy_RETAIN instead
dynamoDBTable.ApplyRemovalPolicy(awscdk.RemovalPolicy_DESTROY)
We define a table with a single partition key, `tag` , which represents the shortened URL identifier. This simple yet effective schema is all we need to map shortened URLs to their original counterparts efficiently.
We will also apply a removal policy to handle the lifecycle of your database resource appropriately.
API Gateway: The Doorway to Our Service
Next, we introduce an HTTP API Gateway, serving as the entry point for our service. API Gateway acts as a front door to manage incoming requests, directing them to the appropriate backend service, in our case, our Lambda functions.
// Import the API Gateway library for the AWS Go CDK
"github.com/aws/aws-cdk-go/awscdkapigatewayv2alpha/v2"
// Define the HTTP API Gateway
urlShortenerAPI := awscdkapigatewayv2alpha.NewHttpApi(stack, jsii.String("url-shortner-http-api"), nil)
The simplicity of setting up an HTTP API with AWS CDK showcases its power, allowing us to focus more on development rather than infrastructure management.
Lambda Functions: The Muscle Behind the Magic
Lambda functions are at the heart of our serverless architecture, performing the logic required to create and retrieve shortened URLs. We define two Lambda functions: one for creating a new shortened URL (create-tag) and another for retrieving the original URL given a shortened identifier (get-tag).
// Import the Lambda and Go-specific Lambda libraries
"github.com/aws/aws-cdk-go/awscdk/v2/awslambda"
"github.com/aws/aws-cdk-go/awscdklambdagoalpha/v2"
// Environment variables for our Lambda functions
funcEnvVar := &map[string]*string{"TABLE_NAME": dynamoDBTable.TableName(), "API_DOMAIN": urlShortenerAPI.Url()}
// Define the Lambda function for creating a shortened URL
createURLFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("create-url-function"),
&awscdklambdagoalpha.GoFunctionProps{
Runtime: awslambda.Runtime_GO_1_X(),
Environment: funcEnvVar,
Entry: jsii.String("./lambda/create-tag"),
FunctionName: jsii.String("url-shortner-create"),
})
// Grant the create function write access on our DynamoDB table
dynamoDBTable.GrantWriteData(createURLFunction)
// Define the Lambda function for getting original URL
getURLFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("get-url-function"),
&awscdklambdagoalpha.GoFunctionProps{
Runtime: awslambda.Runtime_GO_1_X(),
Environment: funcEnvVar,
Entry: jsii.String("./lambda/get-tag"),
FunctionName: jsii.String("url-shortner-access"),
})
// Grant the get function read access on our DynamoDB table
dynamoDBTable.GrantReadData(getURLFunction)
Bridging API Gateway and Lambda: Seamless Integration
API Gateway stands at the forefront, directing traffic with precision. It connects user requests to the appropriate Lambda function, ensuring a smooth operation. Configuring API Gateway might seem daunting, but with our guide, it's a breeze.
- Allocate the POST “/” path => to the create-tag function
- Allocate the GET “/{tag}” path => to the get-tag function
// Import the package for API Gateway v2 integrations
"github.com/aws/aws-cdk-go/awscdkapigatewayv2integrationsalpha/v2"
// Create an HTTP Lambda integration for the create-tag function
createFunctionIntg := awscdkapigatewayv2integrationsalpha.NewHttpLambdaIntegration(
jsii.String("create-function-integration"), createURLFunction, nil)
// Create an HTTP Lambda integration for the get-tag function
getFunctionIntg := awscdkapigatewayv2integrationsalpha.NewHttpLambdaIntegration(
jsii.String("access-function-integration"), getURLFunction, nil)
// Add a POST route that listens for POST requests at the root ("/") path and forwards them to the create-tag Lambda function
urlShortenerAPI.AddRoutes(&awscdkapigatewayv2alpha.AddRoutesOptions{
Path: jsii.String("/"),
Methods: &[]awscdkapigatewayv2alpha.HttpMethod{awscdkapigatewayv2alpha.HttpMethod_POST},
Integration: createFunctionIntg})
// Add a GET route that listens for GET requests with a path parameter "{tag}" and forwards them to the get-tag Lambda function
urlShortenerAPI.AddRoutes(&awscdkapigatewayv2alpha.AddRoutesOptions{
Path: jsii.String("/{tag}"),
Methods: &[]awscdkapigatewayv2alpha.HttpMethod{awscdkapigatewayv2alpha.HttpMethod_GET},
Integration: getFunctionIntg})
By defining routes and linking them to their respective Lambda integrations, we ensure that our API Gateway efficiently routes requests to the right function based on the HTTP method and path.
This was the final piece of our infrastructure puzzle connecting our API Gateway with the Lambda functions, enabling the flow of requests and responses between the user and our backend logic. The final result in “url-shortner.go” file would look something like this:
package main
import (
"github.com/aws/aws-cdk-go/awscdk/v2"
"github.com/aws/aws-cdk-go/awscdkapigatewayv2integrationsalpha/v2"
"github.com/aws/aws-cdk-go/awscdk/v2/awsdynamodb"
"github.com/aws/aws-cdk-go/awscdk/v2/awslambda"
"github.com/aws/aws-cdk-go/awscdkapigatewayv2alpha/v2"
"github.com/aws/aws-cdk-go/awscdklambdagoalpha/v2"
"github.com/aws/constructs-go/constructs/v10"
"github.com/aws/jsii-runtime-go"
)
type UrlShortnerStackProps struct {
awscdk.StackProps
}
func NewUrlShortnerStack(scope constructs.Construct, id string, props *UrlShortnerStackProps) awscdk.Stack {
var sprops awscdk.StackProps
if props != nil {
sprops = props.StackProps
}
stack := awscdk.NewStack(scope, &id, &sprops)
// The code that defines your stack goes here
dynamoDBTable := awsdynamodb.NewTable(stack, jsii.String("url-shortener-dynamodb-table"),
&awsdynamodb.TableProps{
PartitionKey: &awsdynamodb.Attribute{
Name: jsii.String("tag"),
Type: awsdynamodb.AttributeType_STRING}})
dynamoDBTable.ApplyRemovalPolicy(awscdk.RemovalPolicy_DESTROY)
urlShortenerAPI := awscdkapigatewayv2alpha.NewHttpApi(stack, jsii.String("url-shortner-http-api"), nil)
funcEnvVar := &map[string]*string{"TABLE_NAME": dynamoDBTable.TableName(), "API_DOMAIN": urlShortenerAPI.Url()}
// URL Shortner Tag Creation function
createURLFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("create-url-function"),
&awscdklambdagoalpha.GoFunctionProps{
Runtime: awslambda.Runtime_GO_1_X(),
Environment: funcEnvVar,
Entry: jsii.String("./lambda/create-tag"),
FunctionName: jsii.String("url-shortner-create"),
})
// Grant the create function write access on our DynamoDB table
dynamoDBTable.GrantWriteData(createURLFunction)
// URL Shortner Get URL by Tag function
getURLFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("get-url-function"),
&awscdklambdagoalpha.GoFunctionProps{
Runtime: awslambda.Runtime_GO_1_X(),
Environment: funcEnvVar,
Entry: jsii.String("./lambda/get-tag"),
FunctionName: jsii.String("url-shortner-access"),
})
// Grant the get function read access on our DynamoDB table
dynamoDBTable.GrantReadData(getURLFunction)
createFunctionIntg := awscdkapigatewayv2integrationsalpha.NewHttpLambdaIntegration(
jsii.String("create-function-integration"), createURLFunction, nil)
getFunctionIntg := awscdkapigatewayv2integrationsalpha.NewHttpLambdaIntegration(
jsii.String("access-function-integration"), getURLFunction, nil)
urlShortenerAPI.AddRoutes(&awscdkapigatewayv2alpha.AddRoutesOptions{
Path: jsii.String("/"),
Methods: &[]awscdkapigatewayv2alpha.HttpMethod{awscdkapigatewayv2alpha.HttpMethod_POST},
Integration: createFunctionIntg})
urlShortenerAPI.AddRoutes(&awscdkapigatewayv2alpha.AddRoutesOptions{
Path: jsii.String("/{tag}"),
Methods: &[]awscdkapigatewayv2alpha.HttpMethod{awscdkapigatewayv2alpha.HttpMethod_GET},
Integration: getFunctionIntg})
awscdk.NewCfnOutput(stack, jsii.String("output"), &awscdk.CfnOutputProps{Value: urlShortenerAPI.Url(), Description: jsii.String("API Gateway endpoint")})
return stack
}
func main() {
defer jsii.Close()
app := awscdk.NewApp(nil)
NewUrlShortnerStack(app, "UrlShortnerStack", &UrlShortnerStackProps{
awscdk.StackProps{
Env: env(),
},
})
app.Synth(nil)
}
// env determines the AWS environment (account+region) in which our stack is to
// be deployed. For more information see: https://docs.aws.amazon.com/cdk/latest/guide/environments.html
func env() *awscdk.Environment {
return &awscdk.Environment{
Region: jsii.String("us-west-1"),
}
}
Step 2: Crafting the Lambda Logic
With our AWS services primed, we dive into the crux of the matter—Lambda functions. Written in GoLang, these functions are where the magic happens, transforming long URLs into their shorter counterparts and vice versa.
A. Create the Get-Tag Lambda function
As per our directory structure you’re going to open the url_shortener/lambda/get-tag/main.go file, where we’re going to put in the logic for fetching the original URL using the tag in the GET request, and redirect the request to its original URL.
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"os"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
var dbclient *dynamodb.Client
var table string
func main() {
lambda.Start(handler)
}
var ErrUrlNotFound = errors.New("url not found")
func handler(ctx context.Context, req events.APIGatewayV2HTTPRequest) (events.APIGatewayV2HTTPResponse, error) {
tag := req.PathParameters["tag"]
log.Println("redirect request for tag", tag)
url, err := getURL(tag)
if err != nil {
if errors.Is(err, ErrUrlNotFound) {
return events.APIGatewayV2HTTPResponse{StatusCode: http.StatusNotFound, Body: fmt.Sprintf("tag %s not found\n", tag)}, nil
}
return events.APIGatewayV2HTTPResponse{}, err
}
log.Println("redirecting to ", url)
return events.APIGatewayV2HTTPResponse{StatusCode: http.StatusFound, Headers: map[string]string{"Location": url}}, nil
}
func initDB() {
table = os.Getenv("TABLE_NAME")
if table == "" {
log.Fatal("missing environment variable TABLE_NAME")
}
cfg, _ := config.LoadDefaultConfig(context.Background())
dbclient = dynamodb.NewFromConfig(cfg)
}
func getURL(tag string) (string, error) {
initDB()
op, err := dbclient.GetItem(context.Background(), &dynamodb.GetItemInput{
TableName: aws.String(table),
Key: map[string]types.AttributeValue{
"tag": &types.AttributeValueMemberS{Value: tag}}})
if err != nil {
log.Println("failed to get url", err)
return "", err
}
if op.Item == nil {
return "", ErrUrlNotFound
}
urlAV := op.Item["url"]
url := urlAV.(*types.AttributeValueMemberS).Value
log.Println("url", url)
return url, nil
}
B. Create the Create-Tag Lambda function
As per our directory structure you’re going to open the url_shortener/lambda/create-tag/main.go file, where we’re going to put in the logic for creating a short tag for the provided long URL, and store it in our DynamoDB to be able to use it later for redirection.
package main
import (
"context"
"encoding/json"
"log"
"net/http"
"os"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/google/uuid"
)
var dbclient *dynamodb.Client
var table string
func main() {
lambda.Start(handler)
}
type Response struct {
ShortUrl string `json:"shortUrl"`
}
type UrlForm struct {
Url string `json:"url"`
}
func handler(ctx context.Context, req events.APIGatewayV2HTTPRequest) (events.APIGatewayV2HTTPResponse, error) {
body := []byte(req.Body)
var urlRequest UrlForm
json.Unmarshal(body, &urlRequest)
url := urlRequest.Url
log.Println("original url", url)
tag, err := createURL(url)
if err != nil {
log.Println("failed to generate tag for", url)
return events.APIGatewayV2HTTPResponse{}, err
}
baseUrl := os.Getenv("API_DOMAIN")
response := Response{ShortUrl: baseUrl + tag}
respBytes, err := json.Marshal(response)
if err != nil {
log.Println("failed to marshal response for", url)
return events.APIGatewayV2HTTPResponse{}, err
}
return events.APIGatewayV2HTTPResponse{StatusCode: http.StatusCreated, Body: string(respBytes)}, nil
}
func initDB() {
table = os.Getenv("TABLE_NAME")
if table == "" {
log.Fatal("missing environment variable TABLE_NAME")
}
cfg, _ := config.LoadDefaultConfig(context.Background())
dbclient = dynamodb.NewFromConfig(cfg)
}
func createURL(url string) (string, error) {
initDB()
tag := uuid.New().String()[:8]
log.Println("tag", tag)
item := make(map[string]types.AttributeValue)
item["url"] = &types.AttributeValueMemberS{Value: url}
item["tag"] = &types.AttributeValueMemberS{Value: tag}
_, err := dbclient.PutItem(context.Background(), &dynamodb.PutItemInput{
TableName: aws.String(table),
Item: item})
if err != nil {
log.Println("dynamodb put item failed")
return "", err
}
log.Printf("short url for %s - %s\n", url, tag)
return tag, nil
}
Step 3: Bringing it All Together: A Symphony of Services
The beauty of our URL shortener lies in the harmony of its components. Each element plays a vital role, from the database storing the links to the serverless functions processing them. It's a system designed for efficiency, scalability, and ease of use.
Deploying with AWS CDK
We’re going to deploy all this functionality using these simple commands:
go mod tidy
cdk deploy
Testing the Application
After deployment, test your URL shortener by sending requests to your API Gateway endpoint:
To create a short URL, send a POST request to / with the original URL in the body.
curl -X POST "https://9l0r6ktrud.execute-api.us-west-1.amazonaws.com" -H 'Content-Type: application/json' -d '{"url": "https://gameball.co"}'
>> {"shortUrl":"https://9l0r6ktrud.execute-api.us-west-1.amazonaws.com/d4b193a3"}
To use the short URL, send a GET request to /{code}, where {code} is the short code returned by the create function.
curl -i "https://9l0r6ktrud.execute-api.us-west-1.amazonaws.com/d4b193a3"
HTTP/2 302
date: Mon, 26 Feb 2024 19:18:22 GMT
content-length: 0
location: https://gameball.co
apigw-requestid: TwiPugjOSK4EMdg=
Conclusion: Navigating the Digital Highways with Ease
Congratulations!
Our journey concludes here, but yours is just beginning. Armed with AWS, GoLang, and a sprinkle of creativity, you've built your own serverless URL shortener. It's more than just a tool; it's a gateway to a more organized, accessible digital world. Embark on this adventure, and transform how we navigate the vast expanse of the internet. Your digital footprint just got a whole lot lighter!