Another attempt to consolidate some of my various published and presented content…

View My GitHub Profile

Enchilada - Orchestrating Apache Guacamole in AWS

by Keiran


The following blog post details a solution that we have delivered into some of our customers AWS organisations to provide a centralised bastion solution for their multi-account environments that provides the utmost level of security and traceability for their compliance teams.

I’m writing about it here because I think that it is a great pattern that demonstrates the power of AWS Events and serverless functions and how they can be leveraged to enhance and integrate existing software solutions that arent entirely cloud native nicely into a cloud environment.

This pattern was initially architected and developed by my collegues Yuri Litvinov and Richard Nguyen, and I’m representing their work here as it’s a pattern I regularly use for education and AWS Partner competency purposes.

So in the case of this post, I am standing on the shoulders of giants.

One final note is that this post represents an older topology of the solution, and we have uplifted the pattern to now leverage Guacamole API’s rather than working with the database directly, as well as enhancements in the event space to leverage newer capabilities where available.

Problem Statement

Within the clients AWS environment, there is a requirement that all Windows and Linux EC2 instances must be accessed via a centralised, remote access bastion solution in the shared services account and associated networks. As part of this solution, the bastion solution must have the following capabilities

Capability Description
AUDIT LOGS Every connection is logged to CloudWatch and/or CloudTrail for traceability and audit needs.
SESSION RECORDING There must be the ability to go back in time and audit the users bastion sessions to EC2 instances
SINGLE SIGN-ON AND MFA The solution must support Single Sign-On and Multi-Factor Authentication for a seamless and secure user experience.
AUTO DISCOVERY The solution must automatically discover and configure new EC2 instances in the bastion interface so they can easily be connected to by end users. In addition to this, terminated instances must also be removed from the bastion interface ensuring that the user experience is pleasant
CLIENTLESS SOLUTION Use of the solution should be done through an Easy-to-use web interface without the need for plugins

Proposed Solution

We initially investigated the use of entirely cloud native AWS services to solve this problem, specifically leveraging AWS Systems Manager Session Manager, however its current capabilities were unable to provide solutions for all the clients requirements.

As a result, we designed and delivered a solution that leveraged Apache Guacamole in conjunction with automation that uses AWS native services, we call this solution “Enchilada”.

The core components are:

High Level Solution Overview

The 2 core components of the solution are detailed below. To align with well architected principals, these components were

Guacamole Deployment

The guacamole deployment was done using the following common AWS topology in the shared services account.

Enchilada Deployment

The automated creation of connections in the Guacamole user interface leverages the Instance Discovery automation we call Enchilada.

From the diagram below, the core components function as follows, starting from a launch or terminate of an EC2 instance, and resulting in a suitable update of the Guacamole bastion configuration.

Full size Diagram Image can be found here

CloudTrail (Workload Account)

CloudTrail tracks all changes made by users (or services) in every account, whether through the console or programmatically through APIs or CLI toolsets.

In the context of Instance auto-discovery, the following Cloudtrail events that are of interest to Enchilada

CloudWatch Event Rule (Workload Account)

CloudWatch Event Rules allows us to listen to specific events from services (including CloudTrail) and specify a target where the trigger event can be published.

For Enchilada, we use a CloudWatch Event Rule against the following custom Event Pattern :

"source": ["aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
  "eventSource": [""],
  "eventName": ["RunInstances", "TerminateInstances"]

The Target for the Event Rule in this case is the Default Event Bus of the Shared Services Account, where the Guacamole solution is deployed.

Event (Workload Account)

The Event is the CloudTrail Event that will be sent to the target of the CloudWatch Rule as a JSON payload when either a RunInstances or TerminateInstances event occurs.

Default Event Bus (Shared Services Account)

All AWS Accounts have a default Event Bus that can be used cross-account. In this environment, the default event Bus Permissions are set to allow the entire AWS Organization to publish events to it.

As a result, This will provide us a stream of EC2 Launch and Terminate events from all of the workload accounts in the environment in real time in the shared services accounts default event bus.

CloudWatch Event Rule (Shared Services Account)

CloudWatch Event Rules allows us to listen to specific events from services (including CloudTrail) and specify a target where the trigger event can be published.

For Enchilada in the Shared Services Account, we use another CloudWatch Event Rule against the following custom Event Pattern.

"source": ["aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
  "eventSource": [""],
  "eventName": ["RunInstances", "TerminateInstances"]

The Target for the Event Rule is an SQS Queue in the Shared Services account (described below)

SQS (Shared Services Account)

The SQS Queue as the target will start to contain messages received from the CloudWatch Event Rule (described above) that will contain all events where instances are either created or terminated across the workload accounts in the organisation.

The SQS queue is used as the source event for the Lambda function (described below).

The messages are then batched and sent to Lambda to process more than one at a time as the Lambda’s core functionality of updating Guacamole’s database is a fairly expensive operation and must be batched where possible.

Lambda (Shared Services Account)

The Enchilada Lambda function does the heavy lifting of connecting to the Guacamole Database and performing updates.

Since the Lambda will require direct connectivity to the database, the Lambda will have to be configured to run inside the VPC with access to the DB backend of Apache Guacamole.

The Lambda function iterates through the batch of messages received from SQS and performs the following :

Customer Outcome(s)

After the deployment of this solution the customer now has

Requirements alignment

In addition to the above automation, we were able to acheive each of the other objectives as follows.