Evolving your automation with hybrid workers

Editor's Notes

• #2: I am Kieran Jacobsen, Technical Lead at Readify. For over two years, Readify has been a heavy user of Azure Automation, and Hybrid Workers. We don’t just use Azure Automation for Office 365 or Azure deployments like you might see others talk about, we use it for everything. Account creation, Dynamics CRM, file transfers, security monitoring, reporting and much more. The goal of this session is to show you Azure Automation, the different workers, and then integration with web hooks.
• #3: Azure Automation is a cloud hosted, automation as a service platform. It was initially released as a public preview in April 2014, and has since reached general availability. Azure automation includes the automation of tasks via PowerShell scripts and workflows; as well as the automation of configuration management using PowerShell DSC. Automation breathes PowerShell, PowerShell is the only thing it understands. If you can automate something with PowerShell, then you can automate it with Azure Automation. Let’s take a look at Azure Automation.
• #4: So here I have the Azure Automation account for this presentation. Accounts are the deployable unit within Azure. To use Automation, we need to create at least one automation account. Accounts isolate your automation resources, you may want to deploy multiple accounts to separate resources into logical environments. At Readify we have a production and development account. You might want separate accounts for your cloud and on premises automation; or perhaps separate accounts for different business units; finally, you might want to separate by region.   I am not going to go into solutions, essentially they are there to extend OMS.   Runbooks are a collection of tasks that perform some automated process. They might be something simple from starting a virtual machine or creating a file, they may perform complex tasks like deploying cloud infrastructure, or they could combine multiple smaller runbooks together. If I click on runbooks, I can see a list of the runbooks, and you can see some of the runbooks we will use tonight. I can see a runbooks name, modification date, authoring status and any tags associated. There are four types of runbooks, PowerShell, Graphical, PowerShell Workflow and Graphical PowerShell Workflow. From the Runbook blade, I can add runbooks, import runbooks and browse the gallery for pre created runbooks. With runbooks, we have the idea of an authoring status, they could be new, published or in edit. If I make changes to a runbooks code, I need to publish them before they will be actually executed. This separates editing from execution and allows us to create/edit, test and then publish changes to runbooks without impacting our existing processes. Let’s click on MyFirstRunbook, and look at the code. We can see its pretty simple and just outputs hello world. Let’s run it. This runbook takes optional parameters, if it didn't I would just get a confirmation dialog. We can also see its going to run this runbook on a Azure worker. Selecting OK will create a runbook job and submit it to the queue it for execution. So that has completed, we can see no errors or warnings were generated. Let's see the output. There is the output. Congratulations, you have run your first runbook!    So I mentioned Jobs earlier. Jobs are an executed instance of a runbook. Jobs contain a snapshot of the runbook and required assets at the time when it was started. Jobs get executed by workers, and have a state of either new, completed, suspended, queued, running, failed or stopped. Job history is kept for 3 months before being deleted. You can see that over the past few weeks I have run quite a number of runbook jobs.   There are a few different ways a Runbook can be executed. We can do it manually via the start button, start them using PowerShell, they can be triggered via a schedule, or triggered via a web hook.   Now let's look at assets. There are 6 types of assets.   Schedules are pretty obvious, they allow us to control when a runbook is executed. Runbooks can have multiple schedules linked to them and schedules can be linked to multiple runbooks. Schedules can be configured as a once off or reoccurring hourly, daily, weekly or monthly schedule.   Modules can be a little bit complex. Modules can be developed specifically for Azure Automation, or they could simply be PowerShell modules. Azure Automation accounts are created with some preshipped, "global modules“. At the moment the Azure Modules are the only global modules. Microsoft will periodically update these modules for you. If you don’t want them to be auto-updated, you can always import the same module yourself, and that will take precedence over the global module version of that module. You can also import modules directly from the PowerShell Gallery.   Connections specify how automation can connect to an external service or application. This could include authentication information, addressing and port information. The value of using a connection over using multiple variables is that it keeps the information together in a single place. There are three connection types available by default, but you can define custom ones as you like.   Variables are pretty self-explanatory. Variables are available to all runbooks in your account. They are useful for sharing a value between runbooks or configurations or even multiple jobs. We can specify a data type, of string, integer, datetime, Boolean, and null.   Credentials are pretty obvious, they hold username and password pairs. Certificates are also pretty obvious.   So that is all the basics done, let's look at some of the more advanced stuff.
• #5: So the runbook we ran earlier, executed on an azure worker. The Azure worker is managed entirely by Microsoft. Runbooks executed on the Azure worker are not connected to any virtual networks you have in azure, and they cannot access resources in your local data centre.   The advantage of azure workers is that we don't need to manage the server performing the automation, we don't need to set it up, maintain it or consider its availability. Microsoft handles all of this for us. Azure Workers are great if we want to perform tasks solely on cloud systems, say azure or Office 365. We even get some control over where the worker is located. Workers will always be located in the same Azure region as our account.   Unfortunately this lack of control introduces some limitations.   To begin with, we don't have any control over what the IP address of the worker is, theoretically we would probably get the same worker each time a runbook is executed, neither it, nor its ip address have any requirement to remain the same. This introduces challenges when we need to perform tasks like firewall whitelisting, or log file analysis.   At Readify I need to talk to our HR system via SFTP. Our HR system provider, only allows access via specified, whitelisted IP addresses. If I don't have any control over the worker, and its address, then how can I interact with the provider?   How about connecting to an on premise system like Active Directory? I suppose I could use Remoting, but then I possibly need to expose my Domain Controller to the Internet. Can I limit who can connect to it? Well I guess I could restrict it to all of Azures public addresses, all 8.2 million addresses, or even just to the IP addresses for a specific Azure region, but that is still a significant number.   Another issue is the make-up of the worker. What operating system? .Net framework or PowerShell version is it running? What if I want to install some 3rd party application to complete some automation? Well you are out of luck.   If you are familiar with VSTS and its build and release agents, the Azure worker is similar to the Hosted Agent pool. We get a reliable source of systems to run our automation, but we get no control over them, and as such our automation may end up being restricted.
• #6: Hybrid workers allow us to run jobs on machines that we manage, potentially located in our data centre with access to local resources.   To make use of Hybrid Workers, we need to implement Microsoft Operation Management Suite, and install and configure the Automation Solution in our workspace. With the new billing model for Log Analytics, you should still be able to complete this on the free tier. You will need to install the OMS Agent, or SCCM agent on your system for Hybrid Workers to function.   Hybrid Workers do not need any inbound connectivity, only outbound connectivity. The Agent will connect out to Azure over HTTPS, and monitor for jobs that they need to perform. This is all achieved via Azure Service Bus.   Hybrid Workers support all runbook types, and runbooks do not need any special configuration to run on a Hybrid Workers. So let’s run a job on a hybrid worker.
• #7: Run the Get Hostname runbook
• #8: Hybrid Worker groups are collections of workers, much like a server farm. Each worker needs to be a member of a worker group, and you will specify which group when you configure the agent. A group could have a single worker, or multiple workers to maintain high availability. You can also have multiple worker groups, each configured differently or placed in different parts of your environment. Hybrid Workers can only be associated with one Azure Automation Account.   When a runbook is started, you will specify which worker group it is executed on. Only one member will complete that job. You cannot specify a specific worker in a group. We can also specify the execution context, or runas account for a worker group. No matter what runbook job is sent to the group, they are all executed as the same account. Now Microsoft advertises this functionality as an alternative to having runbooks provide their own authentication to local resources; I personally see this as an option of not having jobs executed as Local System but as the least set of privileges as possible.   Let's take a look at hybrid worker groups.
• #9: Demo: Hybrid Worker Groups Demo: RunAs
• #10: Now all of this, doesn't come with its drawbacks. When we move to using hybrid workers, we gain the responsibility for their deployment, management and maintenance. With this great power, comes great responsibility.   Obviously, we now need to perform all of the typical server maintenance tasks, patching the operating system, installing PowerShell updates etc are now all of our responsibility.   PowerShell modules are not automatically deployed to hybrid workers. Unlike with Azure Automation DSC, where the required modules are pushed out to clients, you will need to install and update any modules that your runbooks require manually. It is worth pointing out that the Azure PowerShell module will be automatically installed when you configure a hybrid worker.   Another thing that may impact some users is that you cannot assign a weighting or priority to individual hybrid workers in a group. You can't for instance, have priority given to a hybrid worker in your production site over your DR site.   Previously documentation was pretty poor when it came to hybrid workers, this has changed, but be aware that there are still some inconsistencies.
• #11: Now I couldn't talk about automation, without talking about webhooks. A webhook allows us to trigger off the execution of a runbook in Azure Automation, via a single HTTPS request. Webhooks provide a fantastic way for external services to start runbooks, potentially performing automation within our data centre, without implementing a full API solution.   If you look at a lot of DevOps, Chat Ops, bots, you will see people talking about webhooks. A wide variety of systems support webhooks, including VSTS, GitHub, Office 365, OMS, Slack, HipChat, UptimeRobot, WordPress, ZenDesk, and a whole truckload of others. Webhooks give you the opportunity to integrate with a wide variety of systems you probably previously thought you couldn't integrate with.   Now integrating with webhooks is pretty simple, the caller will simply post data to the receiving endpoint over HTTPS. The HTTP body in that post will typically contain data in a JSON format.   One gotcha about webhooks in azure automation is that they don't fully leverage the built in parameter functionality. When a call is made via a webhook, all of the data is stored in a variable called $WebHookData. You don't need to specify this as a parameter, and runbooks are not required to accept it. Inside $WebHookData is the name of the webhook, the request headers and the request body. You will need to manually convert the body from JSON.   The response from a webhook isn't the output of the runbook, rather what is returned is the Azure Automation Job ID.   Let's take a look at a simple webhook.
• #12: <show them creating a new web hook, use the my first runbook> <just use invoke-webexpression -uri '' -method post to trigger it
• #13: So now we are going to look at an end to end example. I am going to have a function locally trigger off an Azure Automation runbook via a web hook. That runbook will execute on a hybrid worker, from there, its going to create an AD user account and then it will post back to a Slack channel telling me its completed.
• #14: <show them the local function, and then show them the runbook> <trigger function> <switch to slack>
• #15: Look there are plenty of ways to monitor your Azure Automation jobs. One of the easiest is via OMS. You can have automation send all runbook job status and job steams to your work space, from there you can view this information and create email triggered alerts, for instance when jobs fail or are suspended.
• #17: So I have had people ask me, time and time again, how much Azure Automation Costs. For a start, if you only need under 500 minutes, you can get that for free.   At Readify, we average about 7000 minutes of automation per month, and that figure is growing. That’s roughly 116 hours or almost 5 days or run time. All of this roughly costs us a little over 12 dollars per month.   Of course that doesn't take into account OMS licensing, which can vary depending up what solutions you chose to deploy. It also doesn't include the cost of your hybrid workers, we run Azure virtual machine A2 size for most of our workload, so the costs are fairly minimal.
• #18: So this is everything for today, I want to thank you all so much for coming. I have some links up to some useful resources. I will post the slides and links to all the content up on my blog in the next few days.   Does anyone have any questions?