Mass AWS Account Provisioning

Project Goals

The client wanted to publish their legacy Windows software via AWS’s App Stream service and create a fully managed implementation. The design goals were that:

• Each client was separated into their own AWS account.
• This has to be rapidly testable
• No tenant provisioning could be done using Terraform or CloudFormation, it all had to be done with API calls directly, or orchestrated through System Manager. We could use whatever tools we deemed appropriate for non-client operations such as management functions.
• My team was only responsible for providing an endpoint to perform provisioning actions and getting operational information.

In order to deliver this we would need to create the following:

• A library of System Manager runbook templates that performed any infrastructure operations that were any more complex than a single operation.
• An API to initiate and inspect system actions.
• A state database to be the source of truth of what was needed and to cache statuses.

Project Deliverables

My position was to handle the API and orchestration. To do this, I did the following:

1. Design the API
   1. First, I determined the extent of the API actions that were required.
   2. Now that we knew what needed to be done, we roughly broke that process up into
      • Lambda handlers for things like getting status, initiating processes, or performing any complex data transformation
      • API actions where we could simply transform the inputs into direct AWS API calls (implemented using API Gateway templating transforms).
2. Database Design
   1. Determine all the data that needed to be stored and all questions that would be asked of the DB.
   2. The databse was designed to be a single-table DynamoDB implementation with all data stored in a single table with a single GSI to aide in lookups. This was turned into a DB diagram.
   3. I created a Typescript data layer to handle working with the DB design.
3. Runbook Creation - This was done by other team members.
4. API Design
   1. We began with an initial JSON schema document for the next sprint’s API calls. This was handed off to the front end teamso they could begin designing for it.
   2. The general API was turned into a Typescript module that included schema checking, and error handling. This made writing the API much faster than hand-coding each one.
   3. The API was added to a SAM template for deployment.
   4. System was tested with multiple deployments so developers and engineers could test.
5. Implementation - Account Management
   1. I deveopped an account pool system that allowed:
      • Requesting and confirming new accounts.
      • De-provisioning account and adding to a pool.
      • Getting accounts from a pool and assigning to a tenant.
   2. Further development was done on operational issues.

Chalenges Encountered

One of the primary challenges we faced was the need to develop custom provisioning logic, as the use of industry-standard tools like Terraform and CloudFormation was not feasible in this instance. This requirement necessitated a significant investment of time and resources to recreate the functionality of these tools, which ultimately extended the project duration.

Additionally, the client’s periodic reviews and subsequent changes to the project scope introduced variability and unpredictability to the development process. Our team demonstrated agility and adaptability in responding to these changes, ensuring that the project remained on track despite these challenges.

Project Outcomes

This project achieved significant technical milestones, including:

• Designing and implementing a streamlined DynamoDB schema to consolidate all provisioning metadata in a single table, ensuring efficient data management.
• Developing a comprehensive AWS account provisioning and management system, capable of supporting both development and production environments.
• Establishing a process for secure and efficient API development, integrating seamlessly with AWS services.

Despite these accomplishments, the project was ultimately cancelled due to internal stakeholder decisions.

Key Takeaways

This project highlighted the importance of aligning tooling and methodologies with client requirements and constraints. While our team was able to overcome the obstacles presented, we recognize the value of leveraging established tools and frameworks to streamline development and reduce project risk.

In retrospect, this project has provided valuable insights into the importance of collaboration and open communication in managing project expectations and scope. By working closely with the client and adapting to their needs, we were able to deliver a high-quality solution that met their requirements, albeit with some adjustments to the original timeline.

• The project underscored the need for flexibility and adaptability in responding to changing client requirements and constraints.
• Custom provisioning logic can be developed to meet specific client needs, but this approach may require additional time and resources.
• Effective communication and collaboration are essential in managing project scope and expectations.