December 2024 Meetup

Dishant talked about Celery and the unique challenges of running it at scale.

Celery and its challenges in production - Dishant Sethi

Asynchronous task queues like Celery have become essential in handling time-consuming or background jobs and optimising performance in production.

Running Celery at scale comes with unique challenges. In this talk, will explore topics such as:

• maintaining worker reliability
• optimising tasks
• handling errors
• handling retries effectively
• task prioritisation
• monitoring
• autoscaling based on number of tasks under heavy workload."

I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions.

When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.