Part 1 – General Overview & Account Structure.
Part 2 – Setting up the Azure and Visual Studio Online (VSO) structure and maintenance.
Part 3 – Setting up the Azure Environments with ARM and PowerShell.
Part 4 – Setting up Build Servers.
Part 5 – Setting up Continuous delivery (CD) on CI Development builds.
Part 6 - Setting up Release Manager for Dev nightly, QA, Stage and Prod deployments.
Part 1 – General Overview & Account Structure.
Over the past few months I have been working in conjunction with James Kewin consulting at a Financial Services client with the ambition to host a new greenfield project on Azure. Added to this the client wished to provide a template for migration of their existing infrastructure at a later date. In accordance with this goal James and I have spent several months of R&D in moving our enterprise client to the cloud, from a complete end-to-end Application Lifecycle Management point of view, with the latest in best practice throughout the pipeline.
This blog series is a combination of the recommendations we implemented, that address enterprise issues with a move online, and a step-by-step tutorial of how we set up Azure, Visual Studio Online, Release Manager and how we implemented many DevOps techniques such as continuous delivery, environment tear down and re-provision, and the obsession with automation that is required in this new “cloud” world.
Tip: Azure is constantly subject to change, what I write here is likely not to be valid in 6 months. Always double check current limitations and advancements. This series was first written in Jun-Sep 2015. It is also important to note that what is the correct setup for one organisation may not be correct for all, please get in touch with any comments or questions.
Azure Subscriptions & the Enterprise portal
Most of you will be aware of the Azure portal above which seems great, but how do you go about structuring this in the enterprise world of Dev, Test, Stage, Demo and production environments with the associated security lockdown and audit requirements across 100’s if not 1000’s of applications? Luckily if you have those kind of requirements you probably have a Microsoft Enterprise Agreement and as part of this Microsoft provides the Azure Enterprise portal which gives us some functionality to manage, report and plan our subscription structures. The enterprise portal adds another 2 layers on top of a standard subscription, the first being an “Account” which has a single administrator that has full control over multiple subscriptions, and the top most level being a “Department” which is a collection of multiple Accounts when we can set budgets for Azure usage.
In designing out department, account and subscription structures there were several design principles we wanted to keep in mind. These were:
200+ subscriptions (1 for every application) is impractical to maintain subscription setup overheads.
There should be room to grow
|Flexibility||Any decision now should be easy to change going forward|
|Security||Environments & Customer data should be segregated and access control applied appropriately.|
Conscious of hard Azure subscription limits (500 resource groups in a subscription etc), as well as resource limitations and scale limits.
|Costs||Hard “cut out” department limits to prevent accidental budget blowout.|
|Reporting||It should be easy for management to retrieve cost reporting at all levels.|
Constraints of Azure Department, Account and Subscription structures
It is all very well having a set of design principles, but we had to be aware of the current constraints and considerations associated with how we structure the Azure Enterprise Portal in relation to these principles. At the current time of writing (Aug 2015) these are:
- Cost Caps. Unlike an Azure public subscription, a subscription tied to an Enterprise Agreement only has hard cut out limits for your budget at the department level. All controls, and visibility, of costs are at a department level only, something your dev team is not likely to have access to. It is all too easy for a junior dev to create several whopping VM’s that costs £1,000’s per month each if you let them.
- Subscription are difficult to move between accounts, but accounts are easy to move between departments. There is no way of moving a subscription between accounts yourself and although you can contact Microsoft support and have them move your subscription to another account for you, my personal experience in doing this has not been pleasant (hint it involved a great deal of downtime). As such it is easier to assume that once a subscription is attached to an account it is fixed there until such times as Microsoft add the functionality to the Azure portal like they have for moving an account to a different department with 0 effort and 0 downtime. **UPDATE** MS have now added this ability to the enterprise portal (Oct 2015)**
- Subscription Security. Azure is very much still in “Migration” mode from the old https://manage.windowsazure.com to the new https://portal.azure.com/. While the new portal provides us Role Based Access Control (RBAC) to manage who has access to see, modify and create what asset, server, VM etc (and in turn the data associated with those assets) not all asset types are available in the new portal as yet, hence you are forced to provide your team with a greater level of access to a subscription than is perhaps desired. Notable exceptions to the new portal are Service Bus and Azure Active Directory. Hence locking down who has access to a production asset vs a dev, test, stage or demo asset within one subscription is currently not possible. Hence our environments will have to reside in multiple subscriptions for audit and security requirements, especially in financial services.
How did we implement the enterprise structure in an Azure Department/account/subscription structure?
It is important to note here that there are many ways to map your Enterprise structure into an Azure one. This is something you really want to get right from the outset, so take those extra few weeks and speak to all interested parties across the organization.
To understand why we implemented the structure below it is probably best I give some context. Our client is not too dissimilar to many medium / large enterprise organizations in that they have multiple “program of works” running simultaneously with each other, each with multiple teams/projects, each of which have their own Dev, QA, BA and Architecture teams. We wanted to find something that was going to be suitable for everyone while still allowing each “program” a deal of flexibility in how they operate.
At the current time our client also has a clear line in the sand between development and operations, anything client/public facing is the responsibility of a dedicated Operations support team. We modelled the account and subscription structure for stage/demo and production environments accordingly.
Why did we implement it this way?
Rational on Recommendations:
- A way to maintain hard account limits/cut-outs to prevent budget overrun.
- Keep departments to a minimum to reduce maintenance.
- Departments are easy to create, and move accounts between them, should the need arise later.
- Don’t optimise prematurely, but still leave room for flexibility. Two departments seem to fill budget cap requirements at present, if one program blows the budget for the full department it will be easy to split at a later date.
- Accounts split by department, to make it possible to move accounts to their own departments should it be required later.
- Accounts should be controlled by a central “Core” team, suggestion is IT Infrastructure / IT Helpdesk, as maintenance plans require to be setup per subscription (as do build server and VSO rights).
- Due to maintenance setup requirements and the manageability overhead, we wanted to keep the overall number of subscriptions small.
- Unlike Account and Department structures (which are virtual to users), switching and navigating Subscriptions is a “physical” workflow switch which means closing one subscription to get into the other. It is hence important that any developer should be able to navigate the subscription structure easily, and there is no confusion as to which application and app environments lives in which subscription. Hence, we recommended that subscriptions be kept at a department level with one subscription per environment, providing most developers with at most 2-3 subscriptions to filter between.
- “New” asset types can be fully managed at resource or resource group level by Role based access control. “Old” assets such as service bus can only be managed by named person access. To make administration and lock down of QA easier, QA and Dev have been separated into their own subscription.
- Subscription creation would be handled via a central team to ensure both naming and security consistency.
- Resource groups can be used as a security ring-fence for each client. For data protection data, and to facilitate easy deletion at a client’s request, no data will cross resource group boundaries.
- Due to security setup and for naming consistency, resource groups should be provisioned (by script) by a central team (IT Infrastructure). The resources would then be deployed by Release Manager (by script) by the individual teams.
- All assets for a given deployed customer will be contained in their own resource group for isolation purposes.
- A “QA on Demand” environment would be temporarily spun up to test feature differences between clients on each build in another resource group not attached to the main pipeline.
The Filing Cabinet Analogy for the non-tech types
All of the above can be a little heavy for tech types, far less non-tech types, when trying to establish requirements. Jamie came up with an excellent analogy to explain this to upper management and why they required to care.
The Paper (our projects):
Traditionally developers cared about developing the software, with little thought given to the underlying hardware this would be hosted on – that was a problem for architects and the infrastructure team. In our new tear down/up DevOps cloud (insert buzz word) world, developers now specify the infrastructure setup and configuration as yet another project in their solution. Azure Resource Manager templates (ARM Templates) are the next evolution of PowerShell DSC with a particular focus on Azure. ARM Templates allow developers to configure their desired azure configuration and state based on a JSON description of all assets in their solution, further enabling the constantly repeatable nature of tear down and re-provision in a DevOps context.
The Folders (Resource Groups):
Azure now provides a logical grouping container called “Resource Groups” that we can group our infrastructure assets and deployed content into (i.e. different projects, clients etc). These allow you to manage RBAC (role based access control) of which technical staff has access to administer given assets, with a couple of exceptions at the current time of writing. Resource Groups are the perfect “folder” that groups items together with associated metadata, for example we can “Tag” a resource group with any given string we like that will appear on the billing breakdown the finance department receive.
The Drawer (A subscription):
Azure has a soft limit of 800 resource groups (or folders) per subscription (soft limit as a call to MS support and you can have this increased). Our client has multiple products/projects under development within a given program, as such each subscription will contain multiple Resource Groups. We likened the subscription to the drawer of a filing cabinet.
Note: In the diagram above we mention release manager 2015 and the concept of a bounce VM. This is to deal with specific limitations that currently exist in Release Manager at the time of writing (Aug/Sep 2015), we cover this later in part 6 of this series.
The Cabinet (The account containing multiple subscriptions):
Referring back to the Enterprise Portal, multiple subscriptions are managed by 1 account owner. For example our Dev and QA subscriptions are managed by 1 account per program.
Multiple Cabinets (The Department):
The top most level of the azure enterprise portal is the grouping of departments, which is a group of 1 or more accounts. It is at this level where budget caps are maintained. These caps, when breached, will knockout everything under that department from being usable. For our requirements it was agree that it was only necessary to have 2 caps, one for non prod and one for production with a very large cap (no cap is a really bad idea, I’ve heard the stories 1st hand of organisations racking up £10,000’s bill in several days).
Cost caps do not affect the reporting of cost within that department, using “tags” we can specify down to the asset level who should be crossed billed for the usage of a given deployed resource. This was a key factor in negotiating this simplified structure within our client’s organisation.
See any gaps in our thinking?
Hopefully the above has explained the general high level principles we considered in setting up the general azure structure. The rest of this series of posts will be more “tutorial” style on exactly step by step how we achieved this.
We would love any feedback you have on any of this series, please see free to use the comments below… I will respond.