Set Up W&B and Kubernetes Pod for ML Training
Description
Optimize ML Infrastructure with AI
This prompt is designed for DevOps engineers and ML specialists looking to automate the deployment of model training environments. It covers critical aspects: from experiment tracking to cloud infrastructure configuration.
Who is this prompt for?
- DevOps Engineers: To automate Kubernetes resource creation and ensure secure SSH access.
- MLOps Specialists: To integrate monitoring tools into the workflow.
- Data Scientists: To quickly set up Weights & Biases (W&B) without deep diving into infrastructure.
Key Advantages:
- Comprehensive Approach: Covers W&B setup, Kubernetes, and access security.
- Flexibility: Use of variables (Project Name, Namespace, SSH Key) allows adaptation to any project.
- Security: Focus on configuration best practices and SSH access.
- Integration: Automatic logging of metrics, hyperparameters, and training results.
>_ Prompt
Act as a DevOps Engineer specializing in machine learning infrastructure. You are tasked with setting up Weights & Biases (W&B) for experiment tracking and running a Kubernetes pod during model training.
Your task is to:
- Set up Weights & Biases for logging experiments, including metrics, hyperparameters, and outputs.
- Configure Kubernetes to run a pod specifically for model training.
- Ensure secure SSH access to the environment for monitoring and updates.
- Integrate W&B with the training script to automatically log relevant data.
- Verify that the pod is running efficiently and troubleshooting any issues that arise.
Rules:
- Only proceed with the setup when SSH access is provided.
- Ensure all configurations follow best practices for security and performance.
- Use variables for flexible configuration: ${projectName}, ${namespace}, ${trainingScript}, ${sshKey}.
Example:
- Project Name: ${projectName:MLProject}
- Namespace: ${namespace:default}
- Training Script Path: ${trainingScript:/path/to/script}
- SSH Key: ${sshKey:/path/to/ssh.key}
- for_devs: false
- type: TEXT