What is AWS Batch?
With AWS Batch, you can run batch computing workloads on the AWS cloud. Batch computing is a common method used by developers, scientists, and engineers to access large amounts of computing resources, and AWS Batch will eliminate the uniform and arduous work of configuring and managing the required infrastructure, similar to traditional batch computing software. This service can effectively pre-configure resources in response to submitted jobs, thereby eliminating capacity limitations, reducing computing costs and delivering results quickly.
As a fully managed service, AWS Batch enables you to run batch computing workloads of any size. AWS Batch will automatically preset computing resources and optimize workload allocation based on the quantity and scale of workloads. With AWS Batch, there is no longer a need to install or manage batch computing software, allowing you to focus on analyzing results and solving problems.
Components of AWS Batch
AWS Batch is a regional service that enables you to easily run batch tasks across multiple availability zones within a single region. You can create an AWS Batch computing environment in a new or existing VPC. After the computing environment is ready and associated with the task queue, you can define the task definition to specify the Docker container image on which the task is to be run. Container images will be stored and retrieved in the container registry and may exist either inside or outside your AWS infrastructure.
Work
Work units submitted to AWS Batch (such as shell scripts, Linux executable files, or Docker container images). It has a name and runs as a containerized application on an Amazon EC2 instance in your computing environment (using the parameters you specified in the task definition). Tasks can refer to other tasks by name or by ID, and can rely on the successful completion of other tasks. For more information, please refer to the job.
Assignment Definition
The task definition specifies how the task runs; You can regard it as the blueprint of the resources in the task. You can provide IAM roles for your tasks to programmatically access other AWS resources, and also specify memory and CPU requirements. Task definitions can also control container properties, environment variables, and the mount points of persistent storage. Many specifications in the task definition can be overridden by specifying new values when submitting a single task. For more information, please refer to the job definition.
Job queue
When you submit an AWS Batch task, it will be submitted to a specific task queue, and then it will remain there until it is scheduled to the computing environment. You associate one or more computing environments with a task queue and can assign priority values to these computing environments or even across the task queue itself. For instance, you can have a high-priority queue for submitting time-sensitive tasks, as well as a low-priority queue for tasks that can be run at any time when computing resources are cheaper.
Computing environment
A computing environment is a set of managed or unmanaged computing resources used to run tasks. A managed computing environment enables you to specify the required instance types at multiple levels of detail. You can set the computing environment to use a specific type of instance, such as c4.2xlarge or m4.10xlarge, or simply specify the computing environment you wish to use the latest instance type. You can also specify the minimum, required and maximum number of Vcpus for the environment, as well as the bid percentage value on the Spot market and the target set for the VPC subnet. AWS Batch will efficiently start, manage and terminate EC2 instances as needed. You can also manage your own computing environment. In this case, you are responsible for setting up and scaling instances in the Amazon ECS cluster created for you by AWS Batch. For more information, please refer to the Computing Environment.
“Introduction”
Start using AWS Batch by creating task definitions, computing environments, and task queues in the AWS Batch Console.
The AWS Batch First Run Wizard provides you with the option to create the computing environment and job queue and submit the sample hello world job. If you have a Docker image to start in an AWS Batch, you can use this image to create job definitions and instead submit these definitions to your queue.
The assignment is stuck in the RUNNABLE state
If your computing environment contains computing resources but your job stops in the RUNNABLE state and cannot continue, it indicates that there is a problem, causing the job to be unable to be actually placed on the computing resources. The following are some common causes of this problem:
The awslogs log driver is not configured on your computing resources
AWS Batch tasks send their log information to CloudWatch Logs. To achieve this, you must configure your computing resources to use the awslogs log driver. If your computing resource AMI is based on an AMI optimized by Amazon ECS (or Amazon Linux), the driver is registered by default to the ECs-init package. If a different base AMI is used, it is necessary to ensure that when starting the Amazon ECS container agent, the AWS Logs log driver is specified as the available log driver using the ECS_AVAILABLE_LOGGING_DRIVERS environment variable. For more information, please refer to the Computing Resources AMI Specification and Creating a Computing Resources AMI.
Insufficient resources
If the CPU or memory resources specified by your job definition exceed the amount of available computing resources, your job will never be placed. For example, if your job specifies 4 GiB of memory and your available computing resources are less than this amount, you cannot place the job on these computing resources. In this case, you must reduce the memory specified in the job definition or add more computing resources to your environment. Some memory is reserved for the Amazon ECS container agent and other critical system processes. For more information, please refer to Memory Management
Without Internet access rights, computing resources cannot be obtained
Access permission is required to communicate with the Amazon ECS service terminal node. This can be accomplished either through an interface VPC terminal node or through one with a public IP address.
For more information about the interface VPC terminal node, please refer to.
If you have not configured an interface VPC terminal node and your one does not have a public IP address, they must use Network Address Translation (NAT) to provide this access. For more information, please refer to the NAT Gateway in the Amazon VPC User Guide. For more information, please refer to the tutorial: Creating a VPC with Public and Private Subnets for You.
The Amazon EC2 instance limit has been reached
The number of Amazon EC2 instances that your account can launch in the AWS region is determined by the limit of your EC2 instances. Some instance types also have limitations based on instance types. For more information about the Amazon EC2 instance limits for your account (including how to request a limit boost), please refer to the Amazon EC2 Service Limits in the Amazon EC2 User Guide (for Linux Instances).