Added some debug info while dry run.

2025-10-14 17:17:32 +04:00 · 2025-10-14 17:17:32 +04:00 · da53ef5727
commit da53ef5727
parent 4938092b5c
9 changed files with 676 additions and 15 deletions
--- a/.gitignore
+++ b/.gitignore
@ -4,6 +4,7 @@
 !README.md
 .secrets/
 .env
-!.env.template
+.nocr.env
+!.nocr.env.example
 .mono/
 WTelegram.session
--- a/.nocr.env.example
+++ b/.nocr.env.example
@ -0,0 +1,43 @@
+# ===========================================
+# Environment Variables for Docker Compose
+# ===========================================
+# This file provides example environment variables for running the system with Docker Compose.
+#
+# USAGE:
+# 1. Copy this file to .nocr.env in the same directory
+# 2. Fill in your actual values (removing placeholder text)
+# 3. Run: docker-compose up
+#
+# SECURITY NOTE:
+# The .nocr.env file is gitignored and should NEVER be committed to version control.
+# For K8s deployments, use Secrets instead of .nocr.env files.
+# ===========================================
+
+# -----------------
+# Telegram Listener
+# -----------------
+# WTelegram API credentials - obtain from https://my.telegram.org/apps
+
+# Your Telegram API ID (numeric)
+WTelegramClientOptions__ApiId=YOUR_API_ID_HERE
+
+# Your Telegram API Hash (alphanumeric string)
+WTelegramClientOptions__ApiHash=YOUR_API_HASH_HERE
+
+# Your phone number with country code (e.g., 79167310711)
+WTelegramClientOptions__PhoneNumber=YOUR_PHONE_NUMBER_HERE
+
+# -----------------
+# Telegram Client (Bot)
+# -----------------
+# Bot token from @BotFather on Telegram
+
+# Telegram Bot API Token
+TelegramBotOptions__Token=YOUR_BOT_TOKEN_HERE
+
+# -----------------
+# Optional: Debug Mode
+# -----------------
+# Set to "true" to enable configuration debug logging on service startup
+# This will print masked configuration values to console for troubleshooting
+# NOCR_DEBUG_MODE=true
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -73,20 +73,58 @@ cd users && ./src/Nocr.Users.Migrator/AddMigration.sh MyMigrationName

 ## Configuration

-Services use layered appsettings configuration:
- `appsettings.json` - base settings
- `appsettings.Development.json` - local development
- `appsettings.DockerCompose.json` - Docker Compose environment  
- `appsettings.protected.json` - sensitive settings (not in repo)
+**📖 See [CONFIGURATION.md](CONFIGURATION.md) for detailed configuration guide.**
+
+### Quick Start
+
+The system uses ASP.NET Core's layered configuration with **Environment Variables having highest priority**.
+
+**Configuration priority (lowest → highest):**
+1. `appsettings.json` (base settings, committed)
+2. `appsettings.{Environment}.json` (environment-specific, some committed via `.example` files)
+3. User Secrets (Development only)
+4. **Environment Variables (ALWAYS WINS)**
+
+### Three Deployment Modes
+
+1. **VS Code (Local Development)**
+   - Copy `appsettings.Development.json.example` → `appsettings.Development.json` in each service
+   - Or use environment variables in `.vscode/launch.json`
+   - Start infrastructure: `docker-compose up nocr-rabbitmq nocr-text-matcher-db nocr-users-db -d`
+
+2. **Docker Compose (Local Full Stack)**
+   - Copy `.nocr.env.example` → `.nocr.env` in project root
+   - Fill in your Telegram API credentials and Bot token
+   - Run: `docker-compose up`
+
+3. **Kubernetes (Production)**
+   - Create K8s Secrets (do NOT use `.nocr.env`)
+   - Reference secrets in deployment manifests via `envFrom`
+
+### Debug Mode
+
+Enable configuration debug logging on startup:
+```bash
+export NOCR_DEBUG_MODE=true
+```
+
+This prints masked configuration values to help troubleshoot issues.
+
+### Important Notes
+
+- ❌ **Never commit secrets** - use `.example` files as templates
+- ✅ Environment variables override all appsettings files
+- ✅ All services use `.example` files for documentation
+- ✅ Docker Compose uses `.nocr.env` file (gitignored)

 ## Service Ports

 When running with docker-compose:
- telegram-client: 4999
- telegram-listener: 5000  
- text-matcher: 5001
- users: 4998
- RabbitMQ: 5672 (AMQP), 15672 (Management UI)
+- telegram-client: 5050 (http://localhost:5050/health)
+- telegram-listener: 5040 (http://localhost:5040/health)
+- text-matcher: 5041 (http://localhost:5041/health)
+- users: 5042 (http://localhost:5042/health)
+- RabbitMQ: 5672 (AMQP), 15672 (Management UI - http://localhost:15672, admin/admin)
 - MariaDB: 3316 (text-matcher), 3326 (users)

 ## Key Technologies
--- a/CONFIGURATION.md
+++ b/CONFIGURATION.md
@ -0,0 +1,563 @@
+# Configuration Guide
+
+This document describes how to configure the NOCR microservices system for different deployment scenarios.
+
+## Table of Contents
+
+1. [Configuration System Overview](#configuration-system-overview)
+2. [Deployment Scenarios](#deployment-scenarios)
+   - [Local Development (VS Code)](#local-development-vs-code)
+   - [Docker Compose](#docker-compose)
+   - [Kubernetes](#kubernetes)
+3. [Configuration Priority](#configuration-priority)
+4. [Service-Specific Settings](#service-specific-settings)
+5. [Debug Mode](#debug-mode)
+6. [Troubleshooting](#troubleshooting)
+
+---
+
+## Configuration System Overview
+
+The system uses ASP.NET Core's layered configuration approach with the following sources (listed from lowest to highest priority):
+
+1. `appsettings.json` - Base configuration (committed to git)
+2. `appsettings.{Environment}.json` - Environment-specific settings (some committed, some not)
+3. User Secrets - Development-only secrets (stored locally, not in git)
+4. **Environment Variables** - **Highest priority** (overrides everything)
+
+### Important Notes
+
+- **Never commit secrets to git**
+- Use `.example` files as templates
+- Environment Variables always win in priority
+- Each service has its own configuration requirements
+
+---
+
+## Deployment Scenarios
+
+### Local Development (VS Code)
+
+For debugging services individually in VS Code:
+
+#### Step 1: Create Environment-Specific Configuration
+
+For each service you want to run, create `appsettings.Development.json` from the example:
+
+```bash
+# Telegram Listener
+cd telegram-listener/src/Nocr.TelegramListener.Host
+cp appsettings.Development.json.example appsettings.Development.json
+
+# Text Matcher
+cd text-matcher/src/Nocr.TextMatcher.Host
+cp appsettings.Development.json.example appsettings.Development.json
+
+# Users
+cd users/src/Nocr.Users.Host
+cp appsettings.Development.json.example appsettings.Development.json
+
+# Telegram Client
+cd telegram-client/src/Nocr.TelegramClient.Host
+cp appsettings.Development.json.example appsettings.Development.json
+```
+
+#### Step 2: Fill in Your Values
+
+Edit each `appsettings.Development.json` file and replace placeholder values with your actual credentials.
+
+**Example for Telegram Listener:**
+```json
+{
+  "RebusRabbitMqOptions": {
+    "ConnectionString": "amqp://admin:admin@localhost:5672/"
+  },
+  "WTelegramClientOptions": {
+    "ApiId": "22101230",
+    "ApiHash": "c72f884d8eb84cb7134a14362bff060b",
+    "PhoneNumber": "79167310711"
+  }
+}
+```
+
+#### Step 3: Alternative - Use Environment Variables
+
+Instead of creating `appsettings.Development.json`, you can use environment variables in `.vscode/launch.json`:
+
+```json
+{
+  "name": "Launch Service",
+  "type": "coreclr",
+  "request": "launch",
+  "env": {
+    "ASPNETCORE_ENVIRONMENT": "Development",
+    "RebusRabbitMqOptions__ConnectionString": "amqp://admin:admin@localhost:5672/",
+    "WTelegramClientOptions__ApiId": "22101230",
+    "WTelegramClientOptions__ApiHash": "c72f884d8eb84cb7134a14362bff060b",
+    "WTelegramClientOptions__PhoneNumber": "79167310711"
+  }
+}
+```
+
+**Note:** Use double underscores `__` to represent nested configuration sections in environment variables.
+
+#### Step 4: Start Infrastructure
+
+You need RabbitMQ and databases running:
+
+```bash
+# Start only infrastructure services
+docker-compose up nocr-rabbitmq nocr-text-matcher-db nocr-users-db -d
+```
+
+#### Step 5: Debug in VS Code
+
+Press F5 or use the Run and Debug panel to start your service.
+
+---
+
+### Docker Compose
+
+For running the entire system locally with Docker:
+
+#### Step 1: Create .nocr.env File
+
+```bash
+cd /path/to/project/root
+cp .nocr.env.example .nocr.env
+```
+
+#### Step 2: Edit .nocr.env
+
+Open `.nocr.env` and fill in your actual credentials:
+
+```bash
+# Telegram Listener - get from https://my.telegram.org/apps
+WTelegramClientOptions__ApiId=22101230
+WTelegramClientOptions__ApiHash=c72f884d8eb84cb7134a14362bff060b
+WTelegramClientOptions__PhoneNumber=79167310711
+
+# Telegram Client Bot - get from @BotFather
+TelegramBotOptions__Token=1234567890:ABCdefGHIjklMNOpqrsTUVwxyz
+
+# Optional: Enable debug logging
+# NOCR_DEBUG_MODE=true
+```
+
+#### Step 3: Create DockerCompose Configuration Files
+
+Each service needs `appsettings.DockerCompose.json`:
+
+```bash
+# For each service
+cd telegram-listener/src/Nocr.TelegramListener.Host
+cp appsettings.DockerCompose.json.example appsettings.DockerCompose.json
+
+cd text-matcher/src/Nocr.TextMatcher.Host
+cp appsettings.DockerCompose.json.example appsettings.DockerCompose.json
+
+cd users/src/Nocr.Users.Host
+cp appsettings.DockerCompose.json.example appsettings.DockerCompose.json
+
+cd telegram-client/src/Nocr.TelegramClient.Host
+cp appsettings.DockerCompose.json.example appsettings.DockerCompose.json
+```
+
+**Note:** These files contain Docker network hostnames (e.g., `nocr-rabbitmq:5672` instead of `localhost:5672`).
+
+#### Step 4: Start All Services
+
+```bash
+docker-compose up
+```
+
+Or build and start:
+
+```bash
+docker-compose up --build
+```
+
+#### Step 5: Verify Services are Running
+
+- Telegram Listener: http://localhost:5040/health
+- Text Matcher: http://localhost:5041/health
+- Users: http://localhost:5042/health
+- Telegram Client: http://localhost:5050/health
+- RabbitMQ Management: http://localhost:15672 (admin/admin)
+
+---
+
+### Kubernetes
+
+For production deployment on Kubernetes:
+
+#### Step 1: Create Kubernetes Secrets
+
+**Do NOT use `.nocr.env` file in K8s.** Instead, create Secrets:
+
+```bash
+# Create namespace
+kubectl create namespace nocr
+
+# Create secrets for Telegram Listener
+kubectl create secret generic telegram-listener-secrets \
+  --from-literal=WTelegramClientOptions__ApiId=22101230 \
+  --from-literal=WTelegramClientOptions__ApiHash=c72f884d8eb84cb7134a14362bff060b \
+  --from-literal=WTelegramClientOptions__PhoneNumber=79167310711 \
+  -n nocr
+
+# Create secrets for Telegram Client
+kubectl create secret generic telegram-client-secrets \
+  --from-literal=TelegramBotOptions__Token=1234567890:ABCdefGHIjklMNOpqrsTUVwxyz \
+  -n nocr
+```
+
+#### Step 2: Reference Secrets in Deployment
+
+Example Kubernetes deployment manifest:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: telegram-listener
+  namespace: nocr
+spec:
+  template:
+    spec:
+      containers:
+      - name: telegram-listener
+        image: nocr-telegram-listener:latest
+        env:
+        - name: ASPNETCORE_ENVIRONMENT
+          value: "Production"
+        envFrom:
+        - secretRef:
+            name: telegram-listener-secrets
+```
+
+#### Step 3: Create appsettings.Production.json
+
+Create production config files with K8s service names:
+
+```json
+{
+  "RebusRabbitMqOptions": {
+    "ConnectionString": "amqp://admin:admin@rabbitmq-service:5672/"
+  }
+}
+```
+
+**Note:** Secrets from environment variables will override these values.
+
+#### Step 4: Deploy
+
+```bash
+kubectl apply -f deployment/
+```
+
+---
+
+## Configuration Priority
+
+Understanding priority is crucial when troubleshooting configuration issues.
+
+### Priority Order (Lowest → Highest)
+
+1. ⬇️ `appsettings.json` (base)
+2. ⬆️ `appsettings.{Environment}.json` (e.g., Development, DockerCompose, Production)
+3. ⬆️ User Secrets (Development only)
+4. ⬆️⬆️ **Environment Variables (ALWAYS WINS)**
+
+### Example Scenario
+
+If you have:
+
+- `appsettings.json`: `"ConnectionString": ""`
+- `appsettings.DockerCompose.json`: `"ConnectionString": "amqp://admin:admin@nocr-rabbitmq:5672/"`
+- Environment Variable: `RebusRabbitMqOptions__ConnectionString=amqp://admin:admin@localhost:5672/`
+
+**Result:** Environment variable wins! Connection will use `localhost:5672`.
+
+This is why we removed `appsettings.protected.json` - it was overriding Docker Compose settings incorrectly.
+
+---
+
+## Service-Specific Settings
+
+### Telegram Listener
+
+**Required Configuration:**
+
+```json
+{
+  "RebusRabbitMqOptions": {
+    "ConnectionString": "amqp://user:pass@host:port/"
+  },
+  "WTelegramClientOptions": {
+    "ApiId": "YOUR_API_ID",
+    "ApiHash": "YOUR_API_HASH",
+    "PhoneNumber": "YOUR_PHONE_NUMBER"
+  }
+}
+```
+
+**How to obtain Telegram credentials:**
+1. Visit https://my.telegram.org/apps
+2. Log in with your phone number
+3. Create a new application
+4. Copy `api_id` and `api_hash`
+
+---
+
+### Text Matcher
+
+**Required Configuration:**
+
+```json
+{
+  "ConnectionStrings": {
+    "TextMatcherContext": "server=host;port=3306;database=nocr_text_matcher;uid=root;pwd=password"
+  },
+  "RebusRabbitMqOptions": {
+    "ConnectionString": "amqp://user:pass@host:port/"
+  }
+}
+```
+
+**Ports by environment:**
+- Development: `localhost:3316`
+- Docker Compose: `nocr-text-matcher-db:3306`
+- Kubernetes: `text-matcher-db-service:3306`
+
+---
+
+### Users
+
+**Required Configuration:**
+
+```json
+{
+  "ConnectionStrings": {
+    "UsersContext": "server=host;port=3306;database=nocr_users;uid=root;pwd=password"
+  }
+}
+```
+
+**Ports by environment:**
+- Development: `localhost:3326`
+- Docker Compose: `nocr-users-db:3306`
+- Kubernetes: `users-db-service:3306`
+
+---
+
+### Telegram Client
+
+**Required Configuration:**
+
+```json
+{
+  "RebusRabbitMqOptions": {
+    "ConnectionString": "amqp://user:pass@host:port/"
+  },
+  "UsersRestEaseOptions": {
+    "BasePath": "http://users-service:8080"
+  },
+  "TextMatcherRestEaseOptions": {
+    "BasePath": "http://text-matcher-service:8080"
+  },
+  "TelegramBotOptions": {
+    "Token": "YOUR_BOT_TOKEN"
+  }
+}
+```
+
+**How to obtain Bot Token:**
+1. Open Telegram and find @BotFather
+2. Send `/newbot` command
+3. Follow instructions to create your bot
+4. Copy the token provided
+
+---
+
+## Debug Mode
+
+Enable debug mode to print masked configuration values on startup.
+
+### How to Enable
+
+Set environment variable:
+
+```bash
+export NOCR_DEBUG_MODE=true
+```
+
+Or in `.nocr.env`:
+
+```bash
+NOCR_DEBUG_MODE=true
+```
+
+Or in `docker-compose.yml`:
+
+```yaml
+environment:
+  NOCR_DEBUG_MODE: "true"
+```
+
+### Example Output
+
+When debug mode is enabled, you'll see masked configuration on startup:
+
+```
+=== [NOCR_DEBUG] Configuration Values ===
+[NOCR_DEBUG] RebusRabbitMqOptions:
+[NOCR_DEBUG]   ConnectionString: amqp://admin:***@nocr-rabbitmq:5672/
+[NOCR_DEBUG]   InputQueueName: nocr.telegram.listener.queue
+[NOCR_DEBUG]   DirectExchangeName: nocr.direct
+[NOCR_DEBUG]   TopicsExchangeName: nocr.topics
+[NOCR_DEBUG] WTelegramClientOptions:
+[NOCR_DEBUG]   ApiId: 22101230
+[NOCR_DEBUG]   ApiHash: c7...0b
+[NOCR_DEBUG]   PhoneNumber: 79...11
+=== [NOCR_DEBUG] End Configuration ===
+```
+
+**Security:** Passwords and secrets are automatically masked to protect sensitive data.
+
+---
+
+## Troubleshooting
+
+### Problem: Service can't connect to RabbitMQ
+
+**Symptoms:**
+```
+Connection refused to nocr-rabbitmq:5672
+```
+
+**Solution:**
+1. Check if RabbitMQ is running: `docker-compose ps nocr-rabbitmq`
+2. Enable debug mode: `NOCR_DEBUG_MODE=true`
+3. Verify ConnectionString in debug output
+4. For Docker Compose, ensure hostname is `nocr-rabbitmq:5672`, NOT `localhost:5672`
+5. Check if environment variable is overriding appsettings
+
+---
+
+### Problem: Service can't connect to database
+
+**Symptoms:**
+```
+Unable to connect to any of the specified MySQL hosts
+```
+
+**Solution:**
+1. Check database is running: `docker-compose ps nocr-text-matcher-db`
+2. Enable debug mode to see masked connection string
+3. Verify hostname in connection string:
+   - Docker Compose: `nocr-text-matcher-db:3306`
+   - Local dev: `localhost:3316`
+4. Wait for database healthcheck to pass (may take 10-30 seconds on first start)
+
+---
+
+### Problem: Configuration values not applied
+
+**Symptoms:**
+- Settings in `appsettings.Development.json` are ignored
+- Service uses wrong configuration
+
+**Solution:**
+1. Check `ASPNETCORE_ENVIRONMENT` is set correctly:
+   - VS Code: `Development`
+   - Docker Compose: `DockerCompose`
+   - K8s: `Production`
+2. Enable debug mode to see which values are loaded
+3. Check for environment variables overriding your settings
+4. Remember: **Environment Variables always win!**
+
+---
+
+### Problem: Secrets exposed in logs
+
+**Solution:**
+- Debug mode automatically masks sensitive values
+- Never log configuration in production without masking
+- Review debug output format in `Startup.cs`:
+  - Passwords: `amqp://user:***@host`
+  - Secrets: `ab...yz` (first 2 + last 2 chars only)
+
+---
+
+### Problem: Missing .nocr.env file
+
+**Symptoms:**
+```
+docker-compose up fails with missing environment variables
+```
+
+**Solution:**
+1. Copy example file: `cp .nocr.env.example .nocr.env`
+2. Fill in your actual values
+3. Make sure `.nocr.env` is in the project root (same level as `docker-compose.yml`)
+
+---
+
+## Quick Reference
+
+### Environment Variable Naming Convention
+
+ASP.NET Core uses double underscores `__` to represent nested JSON structure:
+
+```bash
+# JSON: { "Section": { "Key": "Value" } }
+# Environment Variable:
+Section__Key=Value
+
+# Example:
+WTelegramClientOptions__ApiId=12345
+RebusRabbitMqOptions__ConnectionString=amqp://localhost
+```
+
+### File Locations
+
+```
+flea/                                   # Project root
+├── .nocr.env                          # Your secrets (gitignored)
+├── .nocr.env.example                  # Template (committed)
+├── docker-compose.yml                 # References .nocr.env
+│
+├── telegram-listener/
+│   └── src/Nocr.TelegramListener.Host/
+│       ├── appsettings.json                        # Base (committed)
+│       ├── appsettings.Development.json            # Local dev (gitignored)
+│       ├── appsettings.Development.json.example    # Template (committed)
+│       ├── appsettings.DockerCompose.json          # Docker (gitignored)
+│       └── appsettings.DockerCompose.json.example  # Template (committed)
+│
+└── [same structure for text-matcher, users, telegram-client]
+```
+
+---
+
+## Security Best Practices
+
+1. ✅ **Never commit secrets** - Use `.example` files as templates
+2. ✅ **Use environment variables** for sensitive data in production
+3. ✅ **Use K8s Secrets** for Kubernetes deployments
+4. ✅ **Enable debug mode** only when troubleshooting
+5. ✅ **Rotate credentials** regularly
+6. ✅ **Use User Secrets** for local development (optional)
+7. ❌ **Don't commit** `.nocr.env` or `appsettings.*.json` (except `.example` files)
+8. ❌ **Don't use** `appsettings.protected.json` (removed from code)
+
+---
+
+## Additional Resources
+
+- [ASP.NET Core Configuration Docs](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/)
+- [Docker Compose Environment Variables](https://docs.docker.com/compose/environment-variables/)
+- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+- [Telegram API Documentation](https://core.telegram.org/api)
+- [Telegram Bot API](https://core.telegram.org/bots/api)
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -7,6 +7,10 @@ services:
      dockerfile: src/Nocr.TelegramClient.Host/Dockerfile
    ports:
      - 5050:8080
+    # IMPORTANT: Create .nocr.env file in project root with your secrets
+    # See .nocr.env.example for reference
+    env_file:
+      - .nocr.env
    environment:
      ASPNETCORE_ENVIRONMENT: DockerCompose
    depends_on:
@ -27,6 +31,10 @@ services:
      dockerfile: src/Nocr.TelegramListener.Host/Dockerfile
    ports:
      - 5040:8080
+    # IMPORTANT: Create .nocr.env file in project root with your secrets
+    # See .nocr.env.example for reference
+    env_file:
+      - .nocr.env
    environment:
      ASPNETCORE_ENVIRONMENT: DockerCompose
      session_pathname: '/app/WTelegram.session'
@ -49,6 +57,10 @@ services:
      dockerfile: src/Nocr.TextMatcher.Host/Dockerfile
    ports:
      - 5041:8080
+    # IMPORTANT: Create .nocr.env file in project root with your secrets
+    # See .nocr.env.example for reference
+    env_file:
+      - .nocr.env
    environment:
      ASPNETCORE_ENVIRONMENT: DockerCompose
    depends_on:
@ -84,6 +96,10 @@ services:
      dockerfile: src/Nocr.Users.Host/Dockerfile
    ports:
      - 5042:8080
+    # IMPORTANT: Create .nocr.env file in project root with your secrets
+    # See .nocr.env.example for reference
+    env_file:
+      - .nocr.env
    environment:
      ASPNETCORE_ENVIRONMENT: DockerCompose
    depends_on:
--- a/2
+++ b/2
@ -1 +1 @@
-Subproject commit db7b973182d0b4872f8b5de251a1754186f7b146
+Subproject commit 19bae3071e9e886bca230c6cc1541041327db2ef
--- a/2
+++ b/2
@ -1 +1 @@
-Subproject commit 08dd95a9865b85b138eda9c4a34fcb1542ce4e50
+Subproject commit f88eb2c6e32f14eedcbf43704f6bc1eb2b8c254e
--- a/2
+++ b/2
@ -1 +1 @@
-Subproject commit 5045e0826efeec5baebc58cff949bf171a6071d4
+Subproject commit 1623dbfe101ca7b07966ecd2c2f59b1313bdf6d0
--- a/2
+++ b/2
@ -1 +1 @@
-Subproject commit a2750a154ca3fe224a8d2d87a10b58fc643393b6
+Subproject commit e3e26392b66d624e7c6628bbc7d2fa567ca4f6c4