Production-ready Celery worker with example tasks, retry logic, error handling, and Railway deployment support.
-
Add Redis to your Railway project
New → Database → Add Redis -
Deploy this service
New → GitHub Repo → Select this repository -
Set environment variables
REDIS_URL = ${{Redis.REDIS_URL}} WORKER_CONCURRENCY = 4 -
Deploy! Railway will automatically detect the Dockerfile
For full functionality, deploy all three templates:
- FastAPI template (triggers tasks)
- This Celery Worker template
- Celery Beat template (schedules tasks)
All should share the same Redis instance via ${{Redis.REDIS_URL}}.
Edit app/tasks.py:
@celery_app.task(name="your_task")
def your_task(data):
"""Your custom task logic"""
# Process data
result = process_data(data)
return result@celery_app.task(name="your_task", bind=True, max_retries=3)
def your_task(self, data):
try:
# Your logic
return result
except Exception as e:
# Retry with exponential backoff
raise self.retry(exc=e, countdown=2 ** self.request.retries)@celery_app.task(name="your_task", bind=True)
def your_task(self, items):
for i, item in enumerate(items):
# Update progress
self.update_state(
state='PROGRESS',
meta={'current': i, 'total': len(items)}
)
# Process item
return {"status": "completed"}Adjust based on your workload:
# For CPU-intensive tasks
WORKER_CONCURRENCY=4
# For I/O-intensive tasks
WORKER_CONCURRENCY=10In app/celery_config.py:
celery_app.conf.update(
task_soft_time_limit=300, # 5 minutes
task_time_limit=600, # 10 minutes
)Route different tasks to different queues:
celery_app.conf.task_routes = {
'app.tasks.slow_task': {'queue': 'slow'},
'app.tasks.fast_task': {'queue': 'fast'},
}from app.celery_config import celery_app
inspect = celery_app.control.inspect()
print(inspect.active()) # Active tasks
print(inspect.registered()) # Registered tasks
print(inspect.stats()) # Worker statsfrom celery.result import AsyncResult
result = AsyncResult(task_id)
print(result.state) # PENDING, STARTED, SUCCESS, FAILURE
print(result.info) # Progress info
print(result.result) # Final resultEmail sending, image processing, data import/export, report generation, API integrations, database cleanup.
# Check Redis connection
redis-cli -u $REDIS_URL ping
# Check worker logs
docker-compose logs worker
# Verify dependencies
pip list | grep celery# Check if tasks are registered
celery -A worker.celery_app inspect registered
# Check active tasks
celery -A worker.celery_app inspect active
# Check worker status
celery -A worker.celery_app status# Reduce concurrency
WORKER_CONCURRENCY=2
# Reduce max tasks per child
WORKER_MAX_TASKS_PER_CHILD=100Use batching, set timeouts, use task routing, monitor memory, enable result expiration.
GPL-3.0