Join Tiago Costa for an in-depth discussion in this video What is Azure Batch?, part of Developing Long-Running Tasks with Microsoft Azure.
- Now let's talk about what is Azure Batch and how can Azure Batch help us running our long running tasks. So, Azure Batch allows us to run large scale parallel and high performance computing jobs, also known as HPC. also known as HPC. It creates a pool of compute nodes behind the scenes for you, then it will install the workload in each one of the nodes. It also comes with a scheduling tool for jobs. So there is no cluster, no job scheduler to install, manage or even considering about scaling on the cluster side. We can manage and monitor all of this using the Azure Batch API. So we have an API up and ready that we can write code to interact with API, we can use Powershow, we can use Batch or we can even use the portal. So, some Business Cases that we have. Financial and project risk modeling using Monte Carlo simulation for example. Rendering of 3D images. Doing image analysis and processing. Media transcoding. Genetic analysis. Optical character recognition, and even data ingestion and some processing around ETL operations. A good example of this is in the US on the Microsoft stores, we have a bunch of screens with all kinds of videos. Passing all that video rendering that you have on all the stores it's done by using Azure Patch behind the scenes. This was something that usually took like two to three days, but now, according to the data, they can do the same thing in just four hours because they can really hyperscale the cluster using the Azure Batch. So, how does this work? So as you can see we have here on the diagram an application or service, and our application or service will upload input and the application to process those files into the Azure storage account. Then, on step two that we have, basically it creates a batch pool of compute nodes in the batch account so that the job will run the workload on the pool and the tasks in the job. Step three: we download the input files and the application to the batch application and the files are installed. On step four, we can start to monitor the tasks. Are they running or not? Are they almost finished or not? So all of that part. And on step five, our job has finished so we are going to upload the output of our tasks. And then on step six, we can download those files from the Azure storage account. So basically, the Azure Batch uploads the files to the storage account, the end result. And then we download those results through our application or service and of course then we need to write some code, to do something with that information. Talking about what kind of application types can we run on our Azure Batch account. There are two types, independent and parallel, that the applications are totally independent from each other, there is no dependency between the applications and we can run two, three, four, ten applications at the same time in the same account but they are not having any relation with each other. A good example of this is Monte Carlo Simulations and even video rendering. I can also have another application type, which is a tightly coupled. Tightly coupled applications, they will need to interact with each other. So usually we are going to find very useful to create nodes, so there are virtual machines that are under the nodes, that will support high-bandwidth RDMA networks. A very good example on this case, for example, AI training. Also, we can also use, for example, virtual machines that makes use of the GPU infrastructure instead of running this on a CPU.
- Creating and using Azure Storage queues
- Using Azure Service Bus queues
- Security for Azure Service Bus queues
- Creating and scheduling WebJobs
- Creating Azure functions
- Running Azure Batch workloads