In the ever-evolving landscape of cloud computing, agility and scalability are paramount. As organizations strive to optimize their infrastructure, the demand for flexible and dynamic solutions has surged. Among the myriad of services offered by Amazon Web Services (AWS), Elastic Container Service (ECS) stands out as a robust container management platform, facilitating the deployment and scaling of containerized applications with ease.
However, managing ECS tasks manually or statically poses significant challenges, especially in environments characterized by fluctuating workloads or diverse application requirements. Enter AWS Lambda and Simple Queue Service (SQS), two powerful services that, when combined, offer a seamless solution for creating dynamic ECS tasks on the fly.
In this blog, we embark on a journey to explore the synergy between Lambda, SQS, and ECS, unveiling the methodology behind orchestrating dynamic container deployments effortlessly. By harnessing the event-driven architecture of Lambda and the reliable message queuing of SQS, we pave the way for automated, efficient, and responsive ECS task management.
Steps:
- Create a Standard SQS queue.
- Create 5 different ECS task definitions with different vCPU and Memory configurations.
- Create a Lambda Function with python3.x runtime.
- Add SQS queue as a trigger and change the role of the lambda function role, provide SQS control role to it.
- Create an ECS cluster.
- Add the Python code in the lambda function and add ECS details in the code.
Create SQS Queue
- Create a Standard SQS queue with 150 seconds Visibility timeout.
Create an ECS Task Definition.
- In this, we are going to create 5 different ECS task definitions with the same docker image, we will give names of these definitions like scale-1, scale-2, …, scale-5,
For eg:
- scale-1 : 0.5 vCPU and 1 GB RAM
- scale-2 : 1 vCPU and 2 GB RAM
- scale-3 : 2 vCPU and 4 GB RAM
- scale-4 : 4 vCPU and 8 GB RAM
- scale-5 : 5 vCPU and 9 GB RAM
Create a Lambda Function with Python3.x Runtime.
Add SQS queue as trigger of lambda function and provide permission as well.
- Provide SQS permission to lambda first.
- Add SQS as a trigger of Lambda function.
Add Python code in the lambda function.
import json
import boto3
from datetime import datetime ##############################################################LAMBDA FUNCTION####################################################################
def lambda_handler(event, context): records = event['Records'] # ECS cluster and service information ecs_cluster = '' # Subnet IDs and Security Group ID subnet_ids = ['', ''] security_group_id = '' # mds-stag-video-cluster-arm-sg # Create ECS client ecs_client = boto3.client('ecs') for record in records: body = str(record['body']) data = json.loads(body) scale = data["scale"] mediakey_1 = data["mediaKey"] mediaid_1 = data["mediaId"] mediaSize_1 = data["pixel"] print(mediakey_1) print(mediaid_1) print(mediaSize_1) # Container Nmae container_name = "" create_ecs_task_audio(ecs_client, ecs_cluster, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("-----------------------------------------------audio tassk created-------------------------------------------------------------") if 1 <= int(scale) <= 5: if mediaSize_1 == '480': print("---------------------------------------------------------Pixel size taken to 480p---------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------480p task created----------------------------------------------------------------") elif mediaSize_1 == '720': print("---------------------------------------------------------Pixel size taken 720p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------720 task cretaed-----------------------------------------------------------------") print("---------------------------------------------------------Pixel size taken 480p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, "480", mediaid_1, mediakey_1) print("---------------------------------------------------------480 task created-----------------------------------------------------------------") else: print("---------------------------------------------------------Pixel size taken 1080------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------1080 task created----------------------------------------------------------------") print("---------------------------------------------------------Pixel size taken 480p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, "480", mediaid_1, mediakey_1) print("---------------------------------------------------------480p task created----------------------------------------------------------------") print("---------------------------------------------------------Pixel size taken 720p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, "720", mediaid_1, mediakey_1) print("---------------------------------------------------------720 task created-----------------------------------------------------------------") else: if mediaSize_1 == '480': print("---------------------------------------------------------scale =3 Pixel size taken 480p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 480p task created----------------------------------------------------------------") elif mediaSize_1 == '720': print("---------------------------------------------------------scale =3 Pixel size taken 720p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 720p task created----------------------------------------------------------------") print("---------------------------------------------------------scale =3 Pixel size taken 480p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, "480", mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 480p task created----------------------------------------------------------------") else: print("---------------------------------------------------------scale =3 Pixel size taken 1080p-----------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 1080p task created---------------------------------------------------------------") print("---------------------------------------------------------scale =3 Pixel size taken 720p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, "480", mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 720p task created----------------------------------------------------------------") print("---------------------------------------------------------scale =3 Pixel size taken 720p------------------------------------------------------------") create_ecs_task(ecs_client, ecs_cluster, "3", container_name, subnet_ids, security_group_id, '720', mediaid_1, mediakey_1) print("---------------------------------------------------------scale =3 720 task created-----------------------------------------------------------------") ####################################################### AUDIO Function ################################################################ def create_ecs_task_audio(ecs_client, ecs_cluster, audio_container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1): # Environment variables environment_variables = { 'mediaKey': mediakey_1, 'mediaId': mediaid_1, 'mediaSize': mediaSize_1 } try: create_task_response = ecs_client.run_task( cluster=ecs_cluster, launchType='FARGATE', taskDefinition='audio-converter-task', networkConfiguration={ 'awsvpcConfiguration': { 'subnets': subnet_ids, 'securityGroups': [security_group_id], 'assignPublicIp': 'ENABLED' } }, overrides={ 'containerOverrides': [ { 'name': audio_container_name, 'environment': [ {'name': key, 'value': value} for key, value in environment_variables.items() ] } ] } ) for task in create_task_response.get('tasks', []): task['createdAt'] = task.get('createdAt').isoformat() if task.get('createdAt') else None task['updatedAt'] = task.get('updatedAt').isoformat() if task.get('updatedAt') else None if create_task_response['tasks']: print(f"Created ECS task: {json.dumps(create_task_response, indent=2)}") else: print(f"Failed to create ECS task. Response: {json.dumps(create_task_response, indent=2)}") except Exception as e: print(f"Error creating ECS task: {str(e)}") ##################################################################### Video Function ################################################### def create_ecs_task(ecs_client, ecs_cluster, scale, container_name, subnet_ids, security_group_id, mediaSize_1, mediaid_1, mediakey_1): # Environment variables environment_variables = { 'mediaKey': mediakey_1, 'mediaId': mediaid_1, 'mediaSize': mediaSize_1 } try: create_task_response = ecs_client.run_task( cluster=ecs_cluster, launchType='FARGATE', taskDefinition=f'{scale}', networkConfiguration={ 'awsvpcConfiguration': { 'subnets': subnet_ids, 'securityGroups': [security_group_id], 'assignPublicIp': 'ENABLED' } }, overrides={ 'containerOverrides': [ { 'name': container_name, 'environment': [ {'name': key, 'value': value} for key, value in environment_variables.items() ] } ] } ) for task in create_task_response.get('tasks', []): task['createdAt'] = task.get('createdAt').isoformat() if task.get('createdAt') else None task['updatedAt'] = task.get('updatedAt').isoformat() if task.get('updatedAt') else None if create_task_response['tasks']: print(f"Created ECS task: {json.dumps(create_task_response, indent=2)}") else: print(f"Failed to create ECS task. Response: {json.dumps(create_task_response, indent=2)}") except Exception as e: print(f"Error creating ECS task: {str(e)}")
Note: Change ECS details with your details!
Thank you for reading the blog!
Content Copyright reserved by Author Harsh Viradia.
Discover more from Coursity
Subscribe to get the latest posts sent to your email.