flea/_deploy

🚀 Nocr CI/CD Pipeline Documentation

📋 Overview

The Nocr project uses a modern, multi-pipeline CI/CD setup powered by Drone CI on Kubernetes. This document describes the 5 specialized pipelines and how to use them.

---

🎯 Pipeline Architecture

Pipeline 1: Feature Validation

Trigger: Push to feature/* or fix/* branches Purpose: Fast feedback for developers Duration: ~3-5 minutes

What it does:

• Clones repo with submodules
• Restores all NuGet packages (shared cache)
• Builds all 4 services in Release mode
• Runs unit and integration tests with Testcontainers

Example workflow:

git checkout -b feature/add-new-filter
# Make changes...
git add .
git commit -m "Add new filter functionality"
git push origin feature/add-new-filter

Drone automatically runs tests. Check results before creating PR.

---

Pipeline 2: Main Validation

Trigger: Push to main branch Purpose: Validate main branch after merge Duration: ~3-5 minutes

What it does:

• Same as Feature Validation
• Ensures main branch is always in working state

Example workflow:

# After PR is merged to main
# Pipeline runs automatically

---

Pipeline 3: Contracts-Only Publish

Trigger: Tag with commit message containing contracts_only:<service> Purpose: Fast publish of contract packages without building images Duration: ~2 minutes

What it does:

• Packs specified service contracts into NuGet packages
• Publishes to internal NuGet feed
• Skips Docker image builds

Example workflow:

# Update telegram-listener contracts
cd telegram-listener
# Make changes to Async.Api.Contracts...
git add .
git commit -m "contracts_only:telegram_listener - Add MessageEdited event"
git push origin main

# Create tag
git tag v1.2.4-contracts
git push origin v1.2.4-contracts

Supported markers:

• contracts_only:telegram_listener
• contracts_only:text_matcher
• contracts_only:users

---

Pipeline 4: Full Release

Trigger: Tag on main WITHOUT contracts_only or deploy_only in commit message Purpose: Complete release cycle Duration: ~8-10 minutes

What it does:

1. Stage 1: Publish all contracts to NuGet (parallel)
2. Stage 2: Build all Docker images with Kaniko (3 parallel streams)
3. Stage 3: Deploy to Kubernetes (only for tags matching v*)

Example workflow:

# Ready to release
git tag v1.3.0
git push origin v1.3.0

# Drone will:
# 1. Publish contracts
# 2. Build images (tagged with v1.3.0, commit SHA, and latest)
# 3. Deploy to k8s (if tag starts with 'v')

Image tags created:

• hub.musk.fun/k8s/nocr/telegram_listener:v1.3.0
• hub.musk.fun/k8s/nocr/telegram_listener:abc1234 (commit SHA)
• hub.musk.fun/k8s/nocr/telegram_listener:latest

---

Pipeline 5: Deploy-Only

Trigger: Tag with commit message containing deploy_only: Purpose: Fast deploy of already-built images Duration: ~1 minute

What it does:

• Skips building
• Deploys specified images to Kubernetes
• Useful for rolling back or promoting existing images

Example workflow:

# Deploy existing images
git commit --allow-empty -m "deploy_only: Deploy v1.2.9"
git tag v1.2.9-deploy
git push origin v1.2.9-deploy

---

🛠️ Deployment Scripts

All deployment scripts are located in _deploy/scripts/:

deploy.sh

Purpose: Deploy services to Kubernetes Usage:

./deploy.sh <tag> <commit-sha>
./deploy.sh v1.3.0 abc1234

Features:

• Updates deployment manifests with new image tags
• Applies manifests to cluster
• Waits for rollouts to complete with timeout
• Runs health checks after deployment
• Shows pod status

rollback.sh

Purpose: Rollback deployments to previous version Usage:

# Rollback single service
./rollback.sh telegram-listener

# Rollback all services
./rollback.sh all

Features:

• Shows revision history
• Performs kubectl rollout undo
• Waits for rollback to complete
• Runs health checks after rollback

health-check.sh

Purpose: Check health of all Nocr services Usage:

./health-check.sh

Checks:

• Pod status (Running/Ready)
• Health endpoints (/health)
• Recent events for failed pods

---

📦 Optimizations

Shared NuGet Cache

All pipelines use a shared temp volume for NuGet packages:

• First dotnet restore downloads packages
• Subsequent builds reuse cached packages
• ~60% faster than individual restores per service

Parallel Execution

• Contract publishing: 3 services in parallel
• Docker builds: 3 parallel streams
• Independent operations never block each other

Kaniko Caching

All Kaniko builds use:

• --cache=true - Layer caching enabled
• --cache-repo=hub.musk.fun/k8s/cache/* - Shared cache repo
• --compressed-caching=true - Faster cache transfer

---

🧪 Testcontainers Support

Feature and Main validation pipelines include Docker-in-Docker service for Testcontainers:

services:
  - name: docker
    image: docker:27-dind
    privileged: true

Tests can use Testcontainers to spin up real databases, message queues, etc.

---

🔒 Required Secrets

Configure these in Drone:

• hub_username - Docker registry username
• hub_password - Docker registry password
• nuget_musk_api_key - NuGet feed API key

---

📊 Pipeline Decision Tree

Push to feature/* → Feature Validation (build + test)
Push to main → Main Validation (build + test)

Tag + "contracts_only:" → Contracts Publish
Tag + "deploy_only:" → Deploy Only
Tag (no markers) → Full Release (contracts → images → deploy)

---

🎓 Best Practices

1. Feature Branches

   • Always create feature branches for new work
   • Let CI validate before merging to main

2. Contracts Changes

   • Use contracts_only: for quick contract updates
   • Other services can update references immediately

3. Release Process

   • Tag only from main branch
   • Use semantic versioning (v1.2.3)
   • Tags starting with v auto-deploy to k8s

4. Emergency Rollback

   # Quick rollback via deploy-only
   git commit --allow-empty -m "deploy_only: Rollback to v1.2.8"
   git tag v1.2.8-rollback
   git push origin v1.2.8-rollback
   
   # Or use rollback script directly on the cluster
   kubectl exec -it deploy-pod -- bash
   cd /flea/_deploy/scripts
   ./rollback.sh all
   

5. Monitoring Deployments

   • Watch Drone UI for pipeline progress
   • Check pod logs: kubectl logs -f deployment/telegram-listener -n nocr
   • Run health checks: ./_deploy/scripts/health-check.sh

---

🐛 Troubleshooting

Pipeline stuck on "Waiting for contracts"

Cause: Contract publish failed Solution: Check NuGet feed, verify API key

Docker build fails with "unauthorized"

Cause: Invalid registry credentials Solution: Update hub_username and hub_password secrets

Tests fail with "Cannot connect to Docker daemon"

Cause: Testcontainers can't reach Docker-in-Docker service Solution: Check DOCKER_HOST environment variable is set correctly

Deployment fails with "ImagePullBackOff"

Cause: Image not found in registry Solution: Verify image was built and pushed successfully in previous step

---

📚 Additional Resources

• Drone CI Documentation
• Kaniko Documentation
• Testcontainers for .NET
• Kubernetes Deployments