Feature</th>	How Plain Text Helps</th></tr></thead>
WhatsApp logging</strong></td>	AI parses “swiggy 500” → appends text to file</td></tr>
Receipt OCR</strong></td>	Extract data → format as ledger → append</td></tr>
Bank CSV import</strong></td>	Transform CSV → ledger format → append</td></tr>
Transaction editing</strong></td>	Find by `gullak:id</code> → text replacement</td></tr>`
Undo/history</strong></td>	Git handles it for free</td></tr>
Backup</strong></td>	`cp main.ledger backup.ledger</code></td></tr>`
Migration</strong></td>	It’s text. There’s nothing to migrate.</td></tr> </tbody></table> The Trade-off#</a></h2> Of course, you lose ACID transactions, database indexes, and complex SQL queries. But for personal finance, the scale makes these unnecessary trade-offs.</p> You likely have 10-50 transactions a month.</li> Running `ledger bal</code> on 10 years of data takes less than 100ms.</li>` `The simplicity is</em> the feature.</li> </ul>`Conclusion#</a></h2> Adopting the ledger format turns Gullak from “yet another expense app” into a thin AI layer over your permanent financial record. By decoupling the data from the application logic, we ensure that the data remains accessible, hackable, and enduring.</p> CLIs are the New AI Interfaces 2026-01-31T00:00:00+00:00 The industry is currently obsessed with defining standards for how Large Language Models (LLMs) should interact with software. We see a proliferation of SDKs, function calling schemas, and protocols like MCP (Model Context Protocol). They all aim to solve the same problem: bridging the gap between natural language intent and deterministic code execution.</p> But we might be reinventing the wheel.</p> The most effective tools for AI agents aren’t those wrapped in heavy “AI-native” integration layers. They are the tools that adhere to a philosophy established forty years ago: the command-line interface.</p> The Unix Philosophy as an AI Protocol#</a></h2> An LLM’s native tongue is text. It reasons in tokens, generates strings, and parses patterns. The Unix philosophy, which emphasizes small tools, plain text interfaces, and standard streams, is accidentally the perfect protocol for AI interaction.</p> Consider the anatomy of a well-behaved CLI:</p> Discovery:</strong> tool --help</code> explains capabilities without hallucination.</li> Structure:</strong> tool --json</code> provides deterministic output for parsing.</li> Composition:</strong> Pipes (\|</code>) allow complex workflows to be assembled on the fly.</li> </ul> When you give an agent access to a robust CLI, you don’t need to define 50 separate function schemas. You give it a shell and a single instruction: “Figure it out using --help</code>.”</p> Context Economy: Lazy vs. Eager Loading#</a></h2> The current approach to agent tooling often involves dumping massive JSON schemas into the context window. Connecting to a standard MCP server might load dozens of tool definitions, involving thousands of tokens describing every possible parameter, before the user has even asked a question. This is “eager loading,” and it is expensive in terms of both latency and context window utilization.</p> A CLI-driven approach is “lazy loaded.”</p> The agent starts with zero knowledge of the tool’s internals. It burns zero tokens on schema definitions. Only when tasked with a specific goal does it invoke man</code> or --help</code>. It retrieves exactly the information needed to construct the command, executes it, and parses the result. This reflects the professional intuition of a senior engineer. We rarely memorize documentation. Instead, we prioritize the ability to quickly discover and apply the specific flags required for the task at hand.</p> Leveraging the Skills Pattern#</a></h2> To bridge the gap between a raw CLI and an agent’s reasoning, we can leverage the Skills</a> pattern. This is an emerging standard for agent-based systems where capabilities are documented as self-contained units of knowledge.</p> Instead of writing a Python wrapper that maps an API to a function call, you provide a Markdown file that explains when and why to use a specific CLI command. The agent uses this as a semantic index.</p> Here is a snippet from a logchef.md</code> skill:</p> ---</span></span> n</span>ame</span>:</span> l</span>ogchef</span></span> d</span>escription</span>:</span> Q</span>uery application logs via LogChef CLI. Use for investigating production incidents and analyzing traffic patterns.</span></span> ---</span></span> </span> ##</span> Common Workflows</span></span> </span> \|</span> Goal</span> \|</span> Command Pattern</span> \|</span></span> \|</span> --------------</span> \|</span> ---------------------</span> \|</span></span> \|</span> Error Analysis</span> \|</span> `</span>logchef sql "..."</span>`</span> \|</span></span> \|</span> Live Tail</span> \|</span> `</span>logchef query '...'</span>`</span> \|</span></span> </span> ##</span> Example: Error Rates by Minute</span></span> </span> To visualize error spikes, use aggregation:</span></span> </span> ```</span>sql</span></span> logchef </span>sql</span> "</span>SELECT toStartOfMinute(_timestamp) as ts, count() as errors</span></span> FROM logs.app_logs WHERE service='api-gateway' AND level='ERROR'</span></span> GROUP BY ts ORDER BY ts DESC LIMIT 60</span>"</span> --</span>output json</span></span> ```</span></span></code></pre> </p> When I ask an agent to “check for error spikes in the API gateway,” Claude identifies that this skill is relevant to the request and loads it on-demand. It sees the example, adapts the SQL query to the current context, and executes the CLI command. The Markdown file serves as a few-shot prompt, teaching the model how to use the tool effectively without rigid code constraints.</p> I maintain similar skill sets for AWS, Kubernetes, and Nomad. The AWS skill doesn’t wrap boto3; it simply documents useful aws ec2</code> and aws cloudwatch</code> commands.</p> The Developer Experience: uv</code> and Single-File CLIs#</a></h2> When a CLI doesn’t exist, the barrier to creating one has never been lower. Modern Python tooling, specifically uv</code> with its inline script metadata, allows us to treat CLIs as disposable, single-file artifacts.</p> I recently needed an agent to manage my Trello board. Rather than fighting with the Trello API documentation or looking for an abandoned library, I had the agent generate a CLI wrapper:</p> #</span>!/usr/bin/env -S uv run --script</span></span> #</span> /// script</span></span> #</span> requires-python = ">=3.11"</span></span> #</span> dependencies = ["typer", "httpx", "rich"]</span></span> #</span> ///</span></span> </span> import</span> typer</span></span> import</span> httpx</span></span> import</span> json</span></span> </span> app</span> =</span> typer</span>.</span>Typer</span>(</span>)</span></span> </span> @</span>app</span>.</span>command</span>(</span>)</span></span> def</span> list_cards</span>(</span>list_id</span>:</span> str</span>,</span> format</span>:</span> str</span> =</span> "</span>table</span>"</span>)</span>:</span></span> """</span>Fetch all cards from a specific list.</span>"""</span></span> #</span> Implementation details...</span></span></code></pre> This script is self-contained. It defines its own dependencies. It implements --help</code> and --json</code> automatically via typer</code>. It took minutes to generate and immediately unlocked Trello capabilities for the agent.</p> The SaaS Imperative#</a></h2> The strategic takeaway for SaaS founders and platform engineers is significant. Your CLI is no longer just a developer convenience; it is your primary AI API.</p> We are moving past the era where a REST API and a web dashboard are sufficient. If your product lacks a terminal interface, you are locking out the growing workforce of AI agents.</p> Browser Automation</strong> is brittle, slow, and breaks with every UI update.</li> Direct API Integration</strong> puts the burden of schema management on the user.</li> CLIs</strong> offer a stable, discoverable, and composable interface that agents can learn and use autonomously.</li> </ul> The “hobby” CLI wrappers built by enthusiasts, such as those for Notion, Jira, or Spotify, are no longer just developer conveniences. They are becoming critical infrastructure. They provide the stable, text-based interface required for agents to interact with these platforms reliably.</p> If you want your platform to be AI-ready, don’t just build an MCP server. Build a great CLI. Make sure it supports --json</code>. Write good man pages. The agents will figure out the rest.</p> Logchef v1.0: The Journey to a Real Log Viewer 2025-12-22T00:00:00+00:00 About eight months ago I wrote about Logchef</a> – a log viewer I’d been building to scratch my own itch with log exploration at work. Back then it was basically a nicer way to query ClickHouse without writing raw SQL every time. Today I’m shipping v1.0, and it’s evolved into something I didn’t quite expect.</p> </a> </div> Let me walk through the major features that made it to 1.0 and some of the engineering decisions behind them.</p> Alerting with Alertmanager Integration#</a></h2> In that first post, I mentioned alerting as a “roadmap” item. It always felt like the obvious next step – you find a pattern in your logs, you want to know when it happens again.</p> </a> </div> But building it took longer than expected. My first attempt was a “rooms” system – a home-grown notification router with its own email, Slack, and webhook channels. I got it working, then stared at the code for notification deduplication, grouping, silencing, and escalation. All problems that Alertmanager</a> has already solved and battle-tested in production for years.</p> So I ripped out rooms and integrated Alertmanager instead. Now Logchef just fires alerts to Alertmanager, and you get all the routing logic – Slack, PagerDuty, email, webhooks, silencing, grouping, inhibition – without me reinventing it poorly.</p> The workflow is simple: write a LogchefQL or SQL query, set a threshold (e.g., “fire if count > 100”), pick a frequency, configure severity and labels. Logchef runs your query on schedule, evaluates the threshold, and if it triggers, fires an alert. Alert history is stored with execution logs so you can debug why something fired (or didn’t).</p> LogchefQL: From Toy Parser to Production Backend#</a></h2> The query language I wrote about originally was pretty basic – just filters that compiled to SQL on the frontend. Over the months it grew into something more capable, but more importantly, I rewrote the entire parser in Go and moved it to the backend. This also opens the door for a CLI tool later – same parser, same query language, different interface.</p> Here’s what LogchefQL looks like now:</p> namespace</span>=</span>"</span>prod</span>"</span> AND</span> level=</span>"</span>error</span>"</span> \|</span> message,</span> trace_id</span></span></code></pre> The pipe operator (\|</code>) selects specific columns instead of SELECT </code>:</p> msg.level</span>=</span>"</span>ERROR</span>"</span> \|</span> timestamp,</span> msg.request.method</span></span></code></pre> Dot notation handles nested JSON fields. If your logs have a log_attributes</code> Map column with nested data:</p> log_attributes.user.name</span> =</span> "</span>john</span>"</span></span></code></pre> For keys that contain dots (common in OTEL-style logs), use quoted field syntax:</p> log_attributes.</span>"</span>http.status_code</span>"</span> ></span>=</span> 500</span></span></code></pre>Why Move Parsing to the Backend?#</a></h3> The original frontend parser was TypeScript. It worked, but had problems:</p> Inconsistency</strong>: The frontend generated SQL, but the backend had no idea what that SQL meant. Validation happened in two places.</p> </li> Type-awareness</strong>: ClickHouse has Map</code>, JSON</code>, LowCardinality</code>, and various string types. The frontend didn’t know the schema, so it couldn’t generate optimal SQL for each column type. For a Map(String, String)</code> column, you want mapContains()</code> or ['key']</code> access. For JSON</code>, you want JSONExtractString()</code>. For regular String</code>, it’s a simple comparison.</p> </li> Debugging hell</strong>: When a query failed, was it the parser? The SQL generator? ClickHouse syntax? Everything happened client-side, invisible to server logs.</p> </li> </ol> The new architecture is cleaner:</p> Frontend</span> Backend</span></span> \|</span> \|</span></span> \|</span> ---</span> LogchefQL</span> query</span> -</span>-</span>></span> \|</span></span> \|</span> \|</span> --</span>></span> Parse</span> (Go</span>)</span></span> \|</span> \|</span> --</span>></span> Validate</span> against</span> schema</span></span> \|</span> \|</span> --</span>></span> Generate</span> type-aware</span> SQL</span></span> \|</span> <</span>--</span> SQL</span> +</span> results</span> -</span>----</span> \|</span> --</span>></span> Execute</span> on</span> ClickHouse</span></span></code></pre> The backend exposes three endpoints: /logchefql/translate</code> (returns the SQL for “View as SQL”), /logchefql/validate</code> (real-time validation with debouncing), and /logchefql/query</code> (parse, validate, execute, return results).</p> Moving parsing to the backend also made the field sidebar implementation cleaner – the same schema-aware code that generates WHERE clauses can filter field values based on your current query.</p> The Field Sidebar#</a></h2> If you’ve used Kibana, you know the interaction: click a field, see its top values, click a value to add it as a filter. It’s the fastest way to explore logs when you don’t know exactly what you’re looking for.</p> </a> </div> Building this for ClickHouse required solving a few problems:</p> High-Cardinality Fields#</a></h3> You can’t just run SELECT DISTINCT field FROM logs</code> on a table with billions of rows. String fields like trace_id</code> would take forever and return millions of values.</p> The solution is a hybrid loading strategy based on column types:</p> LowCardinality and Enum fields</strong>: Auto-load values when the sidebar opens. These are designed for fields with limited distinct values.</li> String fields</strong>: Require an explicit click. A badge shows the count is unknown until you ask.</li> Complex types (Map, Array, Tuple, JSON)</strong>: Excluded. You can’t have meaningful “distinct values” for a JSON blob.</li> </ul> Progressive Loading#</a></h3> Each field loads in parallel (max 4 concurrent) with a 15-second timeout. One slow or failed field doesn’t block others – you get a retry button for that specific field.</p> Query Context#</a></h3> The sidebar respects your current query. If you’ve filtered to level="error"</code>, the field values update to show only values from error logs. This happens through the backend – the field values endpoint accepts the current LogchefQL query and applies it as a WHERE clause filter. Same parser, same SQL generator, consistent results.</p> Query Cancellation#</a></h2> Hit Esc and it cancels the query in ClickHouse. Without this, pressing “Cancel” would just hide the spinner – the query kept running on the server, burning resources.</p> The implementation uses ClickHouse’s query ID feature:</p> SELECT</span> </span> FROM</span> logs </span>WHERE</span> ...</span></span> SETTINGS query_id </span>=</span> '</span>logchef-abc123</span>'</span></span></code></pre> When you hit Esc, the frontend calls a cancellation endpoint that runs:</p> KILL</span> QUERY </span>WHERE</span> query_id </span>=</span> '</span>logchef-abc123</span>'</span></span></code></pre> The original query returns an error, the UI clears, ClickHouse frees resources. Simple, but requires plumbing the query ID through every execution path.</p> AI Query Assistant#</a></h2> “Write a query that finds slowest endpoints by p99” actually works. The AI generates LogchefQL or SQL based on natural language and your table schema.</p> </a> </div> Under the hood it uses go-openai</a>, so any OpenAI-compatible endpoint works – OpenAI, Ollama, vLLM, whatever you prefer. The system prompt includes your table schema so the model knows what fields exist.</p> There’s also an MCP server</a> that exposes Logchef to AI assistants like Claude Desktop, Cursor, or any MCP-compatible client. Instead of context-switching between your AI chat and the log viewer, you can ask directly:</p> “What log sources do I have access to?”</li> “Find all 500 errors in the last hour from the web service”</li> “Show me a histogram of log volume over the past day”</li> “What are the most common error messages in the database logs?”</li> </ul> The MCP server handles discovery (teams, sources, schemas), querying (full ClickHouse SQL), analysis (histograms, saved queries), and even admin operations. It’s a separate binary that runs alongside Logchef – configure it once, and your AI assistant can query your logs through natural conversation.</p> Compact View for Terminal Lovers#</a></h2> </a> </div> Not everyone wants a table. The compact view is a terminal-style display that shows logs as formatted text with syntax highlighting. Denser and faster to scan for certain debugging workflows.</p> Query Variables#</a></h2> Use {{namespace}}</code> in your query, and an input field appears automatically. Great for saved queries that teams want to reuse with different parameters.</p> This was a community contribution from @songxuanqing</a>. The implementation detects {{variable}}</code> patterns in the query text and renders input fields dynamically.</p> Team Management and RBAC#</a></h2> </a> </div> Logchef supports multi-tenancy with role-based access. Teams can have multiple data sources, and users can be members of multiple teams with different roles:</p> Admin</strong>: Full access, can manage team members and sources</li> Editor</strong>: Can create/edit saved queries and collections</li> Viewer</strong>: Read-only access to query and explore logs</li> </ul> This integrates with OIDC for SSO, so you can use your existing identity provider.</p> Admin UI for Runtime Config#</a></h2> Configure stuff without touching config files. The admin settings panel lets you change AI configuration, Alertmanager connection, authentication settings, and query timeouts.</p> </a> </div> This was a migration from config files to database-backed settings. On first boot, Logchef seeds the database from config.toml</code>. After that, the UI takes over and changes are stored in SQLite. Backward compatible – existing config files still work, the UI just overrides them at runtime. No more SSH-ing into production to bump a timeout.</p> Prometheus Metrics#</a></h2> A /metrics</code> endpoint exposes query execution times, error rates, active queries, and other operational data. There’s a pre-built Grafana dashboard</a> for monitoring Logchef itself.</p> What’s Not in 1.0#</a></h2> Some things didn’t make the cut:</p> Live tail</strong>: Streaming logs in real-time. Still on the roadmap.</li> Dashboarding</strong>: Multiple visualizations on one page. Logchef is query-focused; for dashboards, you probably want Grafana with ClickHouse as a datasource.</li> </ul> Calling It 1.0#</a></h2> Calling something “1.0” is weird. There’s no clear line where software becomes “ready.” But I’ve been using Logchef daily at work for months now, and it’s at the point where I trust it. The rough edges are mostly smoothed out. The architecture feels right.</p> Building tools you use yourself is different. You’re the first to hit the rough edges, so you fix them. Slower than building for imaginary users, but the result is something you actually want to use.</p> Thanks again to Kailash</a> for the early direction (schema-agnostic was his idea), and to everyone at Zerodha who’s been using this and giving feedback. Thanks to @songxuanqing</a> for query variables and other contributors for docs and bug fixes.</p> Demo</a> \| Docs</a> \| GitHub</a> \| v1.0.0 Release</a></p> Fin!</p> Fixing a CIBIL Score Disaster with AI 2025-12-04T00:00:00+00:00 About a month ago, I downloaded my CIBIL report expecting a routine check. Instead, I found loans from lenders I had never interacted with, written-off accounts, overdues from fintechs I had never installed, and even two-wheeler loan enquiries. I don’t even ride a bike.</p> My credit score had collapsed to under 680. I stared at the report trying to understand how this could happen.</p> The Root Cause: A Wrong Date of Birth#</a></h2> Buried in my profile section was the problem: my date of birth was wrong. Not a typo, but a completely different year.</p> Because of this mismatch, CIBIL’s system had paired my PAN and mobile number with someone else’s DOB, effectively merging two individuals’ credit histories into one report. The accounts mapped to me included:</p> Aditya Birla Capital</strong>: Short-term personal loan marked as doubtful/substandard</li> Clix Capital</strong>: A loan marked written-off (₹50,000+)</li> Poonawalla Fincorp</strong>: Personal loans with delayed payments</li> Ring / Kissht</strong>: Unsecured digital loans</li> InCred</strong>: Personal loan I never took</li> Dhani Loans</strong>: BNPL-style loan with unrecognized activity</li> Axio (Capital Float)</strong>: Old consumer loan</li> KrazyBee</strong>: Various short-term loans</li> Transactree</strong>: Small-ticket personal loan</li> Multiple enquiries from HDFC, ICICI, IDFC First, Shriram Finance, and others</li> </ul> Some were written-off, others 90+ days overdue, others still active. On paper, I looked like a serial defaulter.</p> Using AI to Understand the Problem#</a></h2> I opened ChatGPT and uploaded the entire PDF with a simple prompt: identify everything wrong in this report.</p> Within minutes, it had mapped every suspicious account, flagged which ones didn’t match my history, highlighted the incorrect DOB, and explained why CIBIL systems mis-map accounts when demographic data is inconsistent.</p> More usefully, it drafted formal dispute letters citing relevant RBI regulations and prepared lender-specific escalations with the right legal language. It felt like having a credit compliance team on demand.</p> The Dispute Process#</a></h2> With the AI-drafted communications as a starting point, I sent disputes to CIBIL and direct emails to each lender. The key was being specific: every email included the CIBIL report control number, the exact account identifiers from the report, and references to specific RBI regulations.</p> For example, when writing to Poonawalla Fincorp about a co-lending arrangement with Kissht, the email included:</p> CIBIL Report details:</strong></p> Control Number: [REDACTED]</li> Downloaded on: [DATE]</li> Where your name appears: “POONAFIN – Personal Loan – Account No. [REDACTED]”</li> Delinquency trail in history: DPD values 35 / 62 / 93 / 124 during [MONTHS]</li> </ul> I reiterate that I have never applied for, signed, or availed any facility from Poonawalla/Kissht. This appears to be erroneous mapping / data contamination.</p> </blockquote> The emails also cited the relevant regulations explicitly:</p> Under Section 45-A(2) of the Credit Information Companies (Regulation) Act 2005 and Para 7.2.2 & 8.1.3 of the RBI Master Directions on Credit Information Companies (2021), please verify this record against your origination/KYC systems. If the record is not verifiable or was created with misused/incorrect KYC, immediately instruct TransUnion CIBIL to delete/correct the entry.</p> </blockquote> This kind of precise, regulation-backed language gets results. Vague complaints are likely to get ignored or deprioritized. Specific complaints with control numbers, account IDs, and regulatory citations get escalated to teams that can actually fix things.</p> For co-lending cases (common with fintechs like Kissht, Ring, etc.), I learned to CC both parties and explicitly request a “consolidated correction” so the entry gets fully removed rather than bouncing between two institutions.</p> When initial responses were slow, I sent reminders that referenced the original complaint number and the 30-day statutory deadline:</p> This is a reminder regarding my complaint Ref No. [REDACTED]. The acknowledgement stated that the issue would be resolved by [DATE], yet I have not received any confirmation.</p> Failure to resolve within the statutory period will leave me with no option but to escalate to the RBI Integrated Ombudsman.</p> </blockquote> CIBIL started closing disputes. One by one, accounts were removed. Eight fraudulent accounts were purged in the first wave.</p> But there was a catch: even after the fraudulent accounts were removed, my DOB was still wrong. CIBIL kept closing my DOB correction disputes without actually fixing the underlying data. Their responses were templated and generic, treating it like a lender issue when DOB is actually a CIBIL demographic field that they control directly.</p> This required escalating to the Nodal Officer with a sharper tone:</p> My Date of Birth correction dispute has been closed twice, yet my DOB remains incorrect in every new CIBIL report. This is a CIBIL demographic field — it is not lender-controlled and should have been corrected immediately once KYC was submitted.</p> Because of this incorrect DOB, my profile was wrongly merged with another individual’s data. Although many wrong accounts have been removed, the root cause remains uncorrected — the wrong DOB is still mapped, and therefore the risk of future wrongful linkages still exists.</p> </blockquote> Only after escalating to the Nodal Officer did the DOB finally get corrected. Once that happened, the system stopped associating the other person’s accounts with my profile. It was an algorithmic identity collision, and fixing the DOB resolved it.</p> The Outcome#</a></h2> My latest CIBIL report shows the correct date of birth, zero fraudulent loans, no written-off or overdue accounts, and a score back in a healthy range. Only my actual accounts remain.</p> What I Learned#</a></h2> Credit bureaus are not infallible. A single incorrect demographic detail (in my case, a mismatched DOB) can cause wrong loan mappings, score drops, false delinquencies, and a complete distortion of your financial identity.</p> The resolution required documentation, persistence with escalations, and an understanding of RBI regulations. AI made the last part significantly easier. Instead of spending hours researching dispute procedures and drafting formal letters, I could focus on gathering the right documents and following up with the right people.</p> If you haven’t checked your CIBIL report recently, it’s worth verifying that your basic details are correct: DOB, PAN, address, mobile, email. One wrong field can create problems that take weeks to untangle.</p> Fin!</p> AI and Home-Cooked Software 2025-10-05T00:00:00+00:00 Everyone is worried that AI will replace programmers. They’re missing the real revolution: AI is turning everyone into one.</p> I’ve been noticing a new pattern: people with deep domain knowledge but no coding experience are now building their own tools. Armed with AI assistants, they can create custom workflows in a matter of days, bypassing traditional development cycles. Are these solutions production-ready? Not even close. But they solve urgent, specific problems, and that’s what matters. Tasks that once required weeks of specialized training are quickly becoming weekend projects.</p> This trend is happening even within the AI companies themselves. Anthropic, for example, shared how their own teams use Claude to accelerate their work. Crucially, this isn’t limited to developers. Their post details how non-technical staff now build their own solutions</a> and create custom automations, providing a powerful real-world example of this new paradigm.</p> Home-Cooked Software#</a></h2> Why search for a generic tool when you can build exactly what you need? This question leads to what I call ‘home-cooked software’: small, personal applications we build for ourselves, tailored to our specific needs. Robin Sloan</a> beautifully describes building an app as making “a home-cooked meal,” while Maggie Appleton</a> writes about “barefoot developers” creating software outside traditional industry structures.</p> What’s new isn’t the concept but the speed and accessibility. With AI, a custom export format, a specific workflow, or the perfect integration is now an afternoon’s work. We’re entering an unprecedented era where the barrier between wanting a tool and having it has nearly vanished.</p> But let’s be clear: the journey from a prototype to a production-ready application is as challenging as ever. In my experience, an AI can churn out a first draft in a few hours, which gets you surprisingly far. But the devil is in the details, and the last stretch of the journey – handling edge cases, ensuring security, and debugging subtle issues – can stretch into weeks. This distinction is crucial. AI isn’t replacing programmers; it’s creating millions of people who can build simple tools. There’s a significant difference.</p> The New Economics#</a></h2> AI is fundamentally reshaping the economics of building software. Before AI, even a simple tool required a significant time investment in learning programming basics, understanding frameworks, and debugging. Only tools with broad appeal or critical importance justified the effort. Now, that effort is measured in hours, not months, and the primary barrier is no longer technical knowledge, but imagination and a clear understanding of one’s own needs.</p> This doesn’t apply to complex or security-critical systems, where deep expertise remains essential. But for the long tail of personal utilities, automation scripts, and custom workflows, the math has changed completely. I’m talking about solving all those minor irritations that pile up: the script to reformat a specific CSV export, the dashboard showing exactly the three metrics you care about, or a script that pulls data from a personal project management tool to sync with an obscure time-tracking app.</p> These tools might be held together with digital duct tape, but they solve real problems for real people. And increasingly, that’s all that matters.</p> The Hidden Costs#</a></h2> But this newfound capability isn’t free. It comes with what I call the “AI Tax”: a set of hidden costs that are rarely discussed.</p> First, prompt engineering can be surprisingly time-consuming, especially for tasks of moderate complexity. While simple requests are often straightforward, anything more nuanced can become an iterative dialogue. You prompt, the AI generates a flawed output, you clarify the requirements, and it returns a new version that misses a different detail. It’s a classic 80/20 scenario: you get 80% of the way there with a simple prompt, but achieving the final 20% of correctness requires a disproportionate amount of effort in refining, correcting, and clarifying your intent to the model.</p> Second, there’s the verification burden. Every line of AI-generated code is a plausible-looking liability. It may pass basic tests, only to fail spectacularly in production with an edge case you never considered. AI learned from the public internet, which means it absorbed all the bad code along with the good. SQL injection vulnerabilities, hardcoded secrets, race conditions—an AI will happily generate them all with complete confidence.</p> Perhaps the most frustrating aspect is “hallucination debugging”: the uniquely modern challenge of troubleshooting plausible-looking code that relies on APIs or methods that simply don’t exist. Your codebase becomes a patchwork of different AI-generated styles and patterns. Six months later, it’s an archaeological exercise to determine which parts you wrote and which parts an AI contributed.</p> But the most significant danger is that AI enables you to build systems you don’t fundamentally understand. When that system inevitably breaks, you lack the foundational knowledge to debug it effectively.</p> Building for One#</a></h2> Despite these challenges, there’s something profoundly liberating about building software just for yourself. Instead of just sketching out ideas, I’ve started building these small, specific tools. For this blog, I wanted a simple lightbox for images; instead of pulling in a heavy external library, I had Claude write a 50-line JavaScript snippet that did exactly what I needed. I built a simple, single-page compound interest calculator</a> tailored for my own financial planning. To save myself from boilerplate at work, I created prom2grafana</a>, a tool that uses an LLM to convert Prometheus metrics into Grafana dashboards.</p> Ten years ago, I might have thought about generalizing these tools, making them useful for others, perhaps even starting an open source project. Today? I just want a tool that works exactly how I think. I don’t need to handle anyone else’s edge cases or preferences. Home-cooked software doesn’t need product-market fit—it just needs to fit you.</p> We’re witnessing the emergence of a new software layer. At the base are the professionally-built, robust systems that power our world: databases, operating systems, and rock-solid frameworks. In the middle are commercial applications built for broad audiences. And at the top, a new layer is forming: millions of tiny, personal tools that solve individual problems in highly specific ways.</p> This top layer is messy, fragile, and often incomprehensible to anyone but its creator. It’s also incredibly empowering. Creating simple software is becoming as accessible as writing. And just as most writing isn’t professional literature, most of this new software won’t be professional-grade. That’s not just okay; it’s the point.</p> The implications are profound. Subject-matter experts can now solve their own problems without waiting for engineering resources, and tools can be hyper-personalized to a degree that is impossible for commercial software. This unlocks a wave of creativity, completely unconstrained by the need to generalize or find a market.</p> Yes, there are legitimate concerns. Security is a real risk, though the profile changes when a tool runs locally on personal data with no external access. We’re creating personal technical debt, but when a personal tool breaks, the owner is the only one affected. They can choose to fix it, rebuild it, or abandon it without impacting anyone else. Organizations, on the other hand, will soon have to grapple with the proliferation of incompatible personal tools and establish new patterns for managing them.</p> But these challenges pale in comparison to the opportunities. The barrier between user and creator is dissolving. We’re entering the age of home-cooked software, where building your own tool is becoming as natural as cooking your own meal.</p> The kitchen is open. What will you cook?</p> State of My Homelab 2025 2025-10-04T00:00:00+00:00 Introduction#</a></h2> For the past five years, I have maintained a homelab in various configurations. This journey has served as a practical exploration of different technologies, from Raspberry Pi clusters running K3s</a> to a hybrid cloud setup</a> and eventually a cloud-based Nomad setup</a>. Each iteration provided valuable lessons, consistently highlighting the operational benefits of simplicity.</p> This article details the current state of my homelab. A primary motivation for this build was to dip my toes into “actual” homelabbing—that is, maintaining a physical server at home. The main design goal was to build a dedicated, reliable, and performant server that is easy to maintain. This led me to move away from complex container orchestrators like Kubernetes in favor of a more straightforward Docker Compose workflow. I will cover the hardware build, software architecture, and the rationale behind the key decisions.</p> Hardware Configuration#</a></h2> After considerable research, I selected components to balance performance, power efficiency, and cost. The server is designed for 24/7 operation in a home environment, making noise and power consumption important considerations.</p> The Build#</a></h3> Component</th> Choice</th> Price</th></tr></thead> CPU</strong></td> AMD Ryzen 5 7600X (6-core, 4.7 GHz)</td> $167.58</td></tr> CPU Cooler</strong></td> ARCTIC Liquid Freezer III Pro 360</td> $89.99</td></tr> Motherboard</strong></td> MSI B650M Gaming Plus WiFi</td> $225.83</td></tr> RAM</strong></td> Kingston FURY Beast 32GB DDR5-6000</td> $136.99</td></tr> Boot Drive</strong></td> WD Blue SN580 500GB NVMe</td> $88.76</td></tr> Storage 1</strong></td> WD Red Plus 4TB (5400 RPM)</td> $99.99</td></tr> Storage 2</strong></td> Seagate IronWolf Pro 4TB (7200 RPM)</td> $150.00</td></tr> Case</strong></td> ASUS Prime AP201 MicroATX</td> $89.99</td></tr> PSU</strong></td> Corsair SF750 (80+ Platinum)</td> $169.99</td></tr> Total</strong></td> </td> $1,219.12</strong></td></tr> </tbody></table> </a> </a> </a> </div> Component Rationale#</a></h3> CPU</strong>: The Ryzen 5 7600X provides a strong price-to-performance ratio. Its 6 cores offer ample headroom for concurrent containerized workloads and future experimentation.</li> Storage</strong>: The boot drive is a 500GB NVMe for fast OS and application performance. The primary storage consists of two 4TB HDDs in a BTRFS RAID 1</strong> configuration. To mitigate the risk of correlated failures, I chose drives from different manufacturers (WD and Seagate) purchased at different times.</li> RAM</strong>: 32GB of DDR5-6000 provides sufficient memory for a growing number of services without risking contention.</li> Case & PSU</strong>: The ASUS Prime AP201 is a compact MicroATX case with a clean aesthetic suitable for a home office. The Corsair SF750 (80+ Platinum) PSU was chosen for its efficiency and to provide capacity for a future GPU for local LLM or transcoding workloads.</li> </ul> System Architecture & Deployment#</a></h2> My previous setups involved Kubernetes and Nomad, but the operational overhead proved unnecessary for my use case. I have since standardized on a Git-based, Docker Compose workflow that prioritizes simplicity and transparency.</p> Directory Structure and “Stacks”#</a></h3> The core of the system is a Git repository that holds all configurations. Each service is defined as a self-contained “stack” in its own directory. The structure is organized by machine, making it easy to manage multiple environments:</p> homelab/</span></span> ├──</span> deploy.sh</span> #</span> Main deployment script</span></span> ├──</span> justfile</span> #</span> Task runner for common commands</span></span> └──</span> machines/</span></span> ├──</span> floyd-homelab-1/</span> #</span> Primary home server</span></span> │</span> ├──</span> config.sh</span> #</span> SSH and deployment settings</span></span> │</span> └──</span> stacks/</span></span> │</span> ├──</span> immich/</span></span> │</span> │</span> └──</span> docker-compose.yml</span></span> │</span> └──</span> paperless/</span></span> │</span> └──</span> docker-compose.yml</span></span> └──</span> floyd-pub-1/</span> #</span> Public-facing VPS</span></span> ├──</span> config.sh</span></span> └──</span> stacks/</span></span> ├──</span> caddy/</span></span> └──</span> ntfy/</span></span></code></pre> This modular approach allows me to manage each application’s configuration, including its docker-compose.yml</code> and any related files, as an independent unit.</p> Deployment Workflow#</a></h3> Deployments are handled by a custom deploy.sh</code> script, with a justfile</code> providing a convenient command-runner interface. The process is fundamentally simple:</p> Sync</strong>: rsync</code> copies the specified stack’s directory from the local Git repository to a REMOTE_BASE_PATH</code> (e.g., /opt/homelab</code>) on the target machine.</li> Execute</strong>: ssh</code> runs the appropriate docker compose</code> command on the remote machine.</li> </ol> Each machine’s connection settings (SSH_HOST</code>, SSH_USER</code>, REMOTE_BASE_PATH</code>) are defined in its machines/<name>/config.sh</code> file. This file can also contain pre_deploy</code> and post_deploy</code> hooks for custom actions.</p> The justfile</code> makes daily operations trivial:</p> #</span> Deploy a single stack to a machine</span></span> just</span> deploy-stack</span> floyd-homelab-1</span> immich</span></span> </span> #</span> View the logs for a stack</span></span> just</span> logs</span> floyd-homelab-1</span> immich</span></span> </span> #</span> Test a deployment without making changes</span></span> just</span> dry-run</span> floyd-homelab-1</span></span></code></pre> </p> This system provides fine-grained control over deployments, with support for actions like up</code>, down</code>, restart</code>, pull</code>, and recreate</code> (which also removes persistent volumes).</p> Container & Configuration Patterns#</a></h3> To keep the system consistent, I follow a few key patterns:</p> Data Persistence</strong>: Instead of using Docker named volumes, I use host bind mounts. All persistent data for a service is stored in a dedicated directory on the host, typically /data/<service-name></code>. This makes backups and data management more transparent.</li> Reverse Proxy Network</strong>: The Caddy stack defines a shared Docker network called public_proxy</code>. Other stacks that need to be exposed to the internet are configured to join this network. This allows Caddy to discover and proxy them without exposing their ports on the host machine. I have written about this pattern in detail in a previous post</a>.</li> Port Exposure</strong>: Services behind the reverse proxy use the expose</code> directive in their docker-compose.yml</code> to make ports available to Caddy within the Docker network. I avoid binding ports directly with ports</code> unless absolutely necessary.</li> </ul> Multi-Machine Topology#</a></h2> The homelab comprises three distinct machines to provide isolation and redundancy.</p> floyd-homelab-1 (Primary Server)</strong>: The core of the homelab, running on the AMD hardware detailed above. It runs data-intensive personal services (e.g., Immich, Paperless-ngx) and is accessible only via the Tailscale network.</li> floyd-pub-1 (Public VPS)</strong>: A small cloud VPS that hosts public-facing services requiring high availability, such as DNS utilities, analytics, and notification relays.</li> floyd-monitor-public (Monitoring VPS)</strong>: A small Hetzner VM running Gatus for health checks. Its independence ensures that I am alerted if the primary homelab or home network goes offline.</li> </ul> This distributed setup isolates my home network from the public internet and ensures that critical public services remain online even if the home server is down for maintenance.</p> Hosted Services#</a></h2> The following is a breakdown of the services, or “stacks,” running on each machine. A few key services that are central to the homelab are detailed further in the next section.</p> floyd-homelab-1 (Primary Server)#</a></h3> Actual</a></strong>: A local-first personal finance and budgeting tool.</li> Caddy</a></strong>: A powerful, enterprise-ready, open source web server with automatic HTTPS.</li> Gitea</a></strong>: A Git service for personal projects.</li> Glance</a></strong>: A dashboard for viewing all my feeds and data in one place.</li> Immich</a></strong>: A photo and video backup solution, directly from my mobile phone.</li> Karakeep</a></strong>: An app for bookmarking everything, with AI-based tagging and full-text search.</li> Owntracks</a></strong>: A private location tracker for recording my own location data.</li> Paperless-ngx</a></strong>: A document management system that transforms physical documents into a searchable online archive.</li> Silverbullet</a></strong>: A Markdown-based knowledge management and note-taking tool.</li> </ul> floyd-monitor-public (Monitoring VPS)#</a></h3> Caddy</a></strong>: Reverse proxy for the services on this node.</li> </ul> floyd-pub-1 (Public VPS)#</a></h3> Beszel-agent</a></strong>: The agent for the Beszel monitoring platform.</li> Caddy</a></strong>: Reverse proxy for the services on this node.</li> Cloak</a></strong>: A service to securely share sensitive text with others.</li> Doggo</a></strong>: A command-line DNS Client for Humans, written in Golang.</li> Ntfy</a></strong>: A self-hosted push notification service.</li> prom2grafana</a></strong>: A tool to convert Prometheus metrics to Grafana dashboards and alert rules using AI.</li> Umami</a></strong>: A simple, fast, privacy-focused alternative to Google Analytics.</li> </ul> Service Highlights#</a></h2> Technitium: A Powerful DNS Server#</a></h3> I came across Technitium DNS after seeing a recommendation from @oddtazz</a>, and it has been a revelation. For anyone who wants more than just basic ad blocking from their DNS server, it’s a game-changer. It serves as both a recursive and authoritative server, meaning I don’t need a separate tool like unbound</code> to resolve from root hints. The level of configuration is incredible—from DNSSEC, custom zones, and SOA records to fine-grained caching control.</p> The UI is a bit dated, but that’s a minor point for me given the raw power it provides. It is a vastly underrated tool for any homelabber who wants to go beyond Pi-hole or AdGuard Home.</p> </a> </div> Beszel: Lightweight Monitoring#</a></h3> For a long time, I felt that monitoring a homelab meant spinning up a full Prometheus and Grafana stack. Beszel is the perfect antidote to that complexity. It provides exactly what I need for basic node monitoring—CPU, memory, disk, and network usage—in a simple, lightweight package.</p> It’s incredibly easy to set up and provides a clean, real-time view of my servers without the overhead of a more complex system. For a simple homelab monitoring setup, it’s hard to beat.</p> </a> </div> Gatus: External Health Checks#</a></h3> While Beszel monitors the servers from the inside, Gatus watches them from the outside. Running on an independent Hetzner VM, its job is to ensure my services are reachable from the public internet. It validates HTTP status codes, response times, and more.</p> This separation is crucial; if my entire home network goes down, Gatus is still online to send an alert to my phone. It’s the final piece of the puzzle for robust monitoring, ensuring I know when things are broken even if the monitoring service itself is part of the outage.</p> </a> </div> Storage and Backup Strategy#</a></h2> Data integrity and recoverability are critical. My strategy is built on layers of redundancy and encryption.</p> Storage: BTRFS RAID 1 + LUKS Encryption#</a></h3> I chose BTRFS for its modern features:</p> Checksumming</strong>: Protects against silent data corruption.</li> Copy-on-Write</strong>: Enables instantaneous, low-cost snapshots.</li> Transparent Compression</strong>: zstd</code> compression saves space without significant performance overhead.</li> </ul> The two 4TB drives are mirrored in a RAID 1 array, providing redundancy against a single drive failure. The entire array is encrypted using LUKS2, with the key stored on the boot SSD for automatic mounting. This protects data at rest in case of physical theft or drive disposal.</p> Mount options in /etc/fstab</code>:</p> /dev/mapper/crypt-sda</span> /mnt/storage</span> btrfs</span> defaults,noatime,compress=zstd</span> 0</span> 2</span></span></code></pre>Backup: Restic + Cloudflare R2#</a></h3> RAID does not protect against accidental deletion, file corruption, or catastrophic failure. My backup strategy follows the 3-2-1 rule.</p> Daily, automated backups are managed by systemd timers running restic</code>. Backups are encrypted and sent to Cloudflare R2, providing an off-site copy. R2 was chosen for its zero-cost egress, which is a significant advantage for restores.</p> The backup script covers critical application data and the Docker Compose configurations:</p> BACKUP_PATHS</span>=</span>(</span></span> "</span>/mnt/storage</span>"</span> #</span> All application data</span></span> "</span>/home/karan/stacks</span>"</span> #</span> Docker Compose configs</span></span> )</span></span></code></pre> Each backup run reports its status to a healthchecks.io</a> endpoint, which sends a push notification on failure. I must appreciate its generous free tier, which is more than sufficient for my needs.</p> </a> </div> Conclusion#</a></h2> This homelab represents a shift in philosophy from exploring complexity to valuing simplicity and reliability. The upfront hardware investment of ~$1,200 is offset by eliminating recurring cloud hosting costs and providing complete control over my data and services.</p> For those considering a homelab, my primary recommendation is to start with a simple, well-understood foundation. A reliable machine with a solid backup strategy is more valuable than a complex, hard-to-maintain cluster. The goal is to build a system that serves your needs, not one that you serve.</p> TIL: WireGuard's Misleading "No Route to Host" Error 2025-07-30T00:00:00+00:00 I recently spent some time debugging a WireGuard tunnel that was acting weird. The handshake was successful, pings worked perfectly, but any TCP connection failed with connect: no route to host</code>.</p> Classic misleading error message. The routing was fine.</p> The Setup#</a></h2> Server with a public IP running WireGuard (wg0</code>) with IP 10.100.0.1/24</code>. Client connects and gets assigned 10.100.0.2/32</code>. I wanted to proxy TCP traffic from the server to a service running on the client at 10.100.0.2:7778</code>.</p> The Investigation#</a></h2> Diagnostics showed contradictory results:</p> Routing worked fine:</strong> Server routing table correctly directed 10.100.0.0/24</code> traffic to wg0</code>. Pings were successful:</p> #</span> On the server</span></span> $</span> ping</span> -</span>c</span> 3</span> 10.100.0.2</span></span> PING</span> 10.100.0.2</span> (10.100.0.2</span>) 56(</span>84</span>) bytes of data.</span></span> 64</span> bytes</span> from</span> 10.100.0.2:</span> icmp_seq=</span>1</span> ttl=</span>64</span> time=</span>150</span> ms</span></span> .</span>.</span>.</span></span></code></pre> TCP failed immediately:</strong></p> #</span> On the server</span></span> $</span> curl</span> -</span>v</span> http://10.100.0.2:7778</span></span> </span> Trying 10.100.0.2:7778...</span></span> </span> connect to 10.100.0.2 port 7778 from 10.100.0.1 port 59812 failed: No route to host</span></span></code></pre> The key insight: ICMP</code> was being treated differently than TCP</code>. This pointed to a firewall issue, not routing. The “no route to host” error was the kernel interpreting an ICMP “Destination Unreachable” message from the remote peer.</p> But when I ran tcpdump</code> on the client, things got stranger:</p> #</span> On the client</span></span> $</span> sudo</span> tcpdump</span> -</span>i</span> any</span> -</span>n</span> '</span>host 10.100.0.1</span>'</span></span> </span> #</span> Output when the server tries to connect</span></span> 17:36:03.043147</span> wg0</span> In</span> IP</span> 10.100.0.1.14808</span> ></span> 10.100.0.2.7778:</span> Flags</span> [S</span>], seq 324784341, win 42780, ...</span></span></code></pre> The TCP SYN</code> packet arrived successfully through wg0</code>. But no response. No SYN-ACK</code> (success), no ICMP</code> error (rejection). The packet was being silently dropped.</p> The Culprit: firewalld</code>#</a></h2> The client was running Arch Linux with firewalld</code>. My mistake was trying to manage firewall rules with iptables</code> commands in the WireGuard PostUp</code> script. While iptables</code> was installed, firewalld</code> was the active manager, using nftables</code> as its backend.</p> When a new interface like wg0</code> comes up, firewalld</code> needs to know which “zone” it belongs to. If unassigned, it gets handled by a restrictive default policy that silently DROP</code>s unsolicited TCP packets while allowing ICMP (pings).</p> The Fix#</a></h2> Don’t add iptables</code> rules. Just assign the WireGuard interface to the right firewalld</code> zone. For internal tunnels, trusted</code> works well.</p> On the client</strong>:</p> sudo</span> firewall-cmd</span> -</span>-permanent</span> -</span>-zone=trusted</span> -</span>-add-interface=wg0</span></span> sudo</span> firewall-cmd</span> -</span>-reload</span></span></code></pre> TCP connections worked instantly after this.</p> TL;DR:</strong> If WireGuard pings work but TCP fails with “no route to host”, it’s probably a client firewall issue. On firewalld</code> systems, assign the WireGuard interface to the right zone instead of messing with iptables</code>.</p> Fin!</p> Announcing Logchef 2025-04-27T00:00:00+00:00 So, for the last 3-4 months, I’ve been busy building Logchef</a>. This tool basically grew straight out of my day job managing logs at Zerodha</a>, where I’ve been managing logs for almost half a decade. I wanted to share a bit about how Logchef came to be.</p> Like many, we journeyed through the complexities of ELK (a management nightmare) and found its OSS fork, OpenSearch, didn’t quite hit the mark for us either. We eventually found solid ground with Clickhouse, as detailed on our tech blog: Logging at Zerodha</a>.</p> Challenges Faced with Metabase#</a></h2> However, as I noted in that post, while Metabase served us well for analytics, it wasn’t the ideal UI specifically tailored for log analysis against Clickhouse:</p> “While Metabase has served us well so far, there is certainly room for improvement, especially regarding a more tailored UI for Clickhouse… we plan to continue exploring potential solutions.”</p> </blockquote> Here’s a distilled version of the common pain points we experienced:</p> Ad-hoc Querying Was Painful:</strong> Writing raw Clickhouse SQL in Metabase for quick log searches felt cumbersome and slow. Even modifying existing complex query templates was error-prone – a tiny syntax mistake could lead to minutes spent debugging the query itself, especially stressful during production incidents.</li> Disconnect Between Visualization and Raw Logs:</strong> A common workflow is to visualize trends (e.g., errors over time) and then drill down into the specific logs causing those trends. In Metabase, this often meant writing two separate queries</em> – one for aggregation/visualization and another (often rewritten from scratch) just to see the raw log lines. Metabase’s row limits (around 2k) further complicated viewing the full context of raw logs after filtering. The intuitive “slice and drill-down” experience many log tools offer was missing.</li> UI/UX Annoyances:</strong> Several smaller but cumulative issues added friction: difficulty selecting precise time ranges like “last 6 hours,” viewing logs immediately surrounding a relevant event, columns getting truncated (...</code>), and limited timestamp precision display in results. Though there are some workarounds, they often felt like band-aids rather than solutions.</li> </ul> TL;DR: Metabase interface wasn’t optimized for the specific task of log exploration. Debugging sessions that should</em> have taken minutes were stretching significantly longer. Querying and exploring logs felt clunkier than it needed to be.</p> And one fine day, I decided to stop just wishing for a better tool and start building one:</strong></p> </p> Logchef#</a></h2> When I first started prototyping, I kept the scope pretty tight: just build a viewer for the standard OTEL schema</a>. OTEL’s flexible enough, but a quick chat with Kailash</a> sparked what turned out to be a game-changing idea: make Logchef schema-agnostic. And that really became the core concept.</p> Basically, Logchef lets you connect it straight to your existing Clickhouse log tables, no matter their structure. All it really needs is a timestamp field (DateTime</code> or DateTime64</code>). Bring your own custom schemas, stick with the OTEL standard, or even adapt it to your own needs. Logchef doesn’t force you into a specific format. From what I’ve seen, not many tools offer this kind of plug-and-play flexibility with existing tables today.</p> Logchef is designed as a specialized query and visualization layer sitting on top of Clickhouse. Logchef intentionally</em> excludes log collection and ingestion. Why reinvent the wheel when excellent tools like Vector, Fluentbit, Filebeat, etc., already handle this reliably? Logchef focuses purely on exploring the logs once it’s in Clickhouse.</p> Stack#</a></h2> Backend: Written in Go for performance and concurrency.</li> Metadata Storage: Uses SQLite for lightweight management of users, teams, Clickhouse source connections, and query collections. It’s simple and perfectly suited for this kind of a metadata store.</li> Frontend: An interactive log viewer with Vue.js and styled with shadcn/ui and Tailwind CSS. I also implemented a simple search syntax for common filtering tasks (e.g., status=200 and path~"/api/"</code>). This involved writing a tokenizer and parser that translates this syntax into efficient ClickHouse SQL conditions optimised for querying logs. Building this parser, validator, and integrating it smoothly with the Monaco editor for syntax highlighting was a significant effort but quite happy with the end result.</li> </ul> Setting Up the Public Demo (demo.logchef.app</code>)#</a></h2> </p> I wanted a public demo instance</a> so people could try Logchef easily. Setting this up involved a few specific tweaks compared to a standard deployment, all managed within the Docker Compose setup:</p> Generating Dummy Data:</strong> A log viewer isn’t much use without logs! Instead of ingesting real data, I configured vector</code> using its demo_logs</code> source type. This continuously generates realistic-looking syslog and HTTP access logs and pushes them into the demo Clickhouse instance (syslogs</code> and http_logs</code> tables). It gives users immediate data to query without any setup on their part.</p> #</span> vector.toml snippet</span></span> [</span>sources</span>.</span>generate_syslog</span>]</span></span> type</span> =</span> "</span>demo_logs</span>"</span></span> format</span> =</span> "</span>syslog</span>"</span></span> interval</span> =</span> 0.3</span> #</span> Generate logs frequently</span></span> </span> [</span>sinks</span>.</span>clickhouse_syslog</span>]</span></span> #</span> ... config to send to Clickhouse ...</span></span> table</span> =</span> "</span>syslogs</span>"</span></span></code></pre></li> Securing Admin Endpoints (Demo Mode):</strong> Since this is a public, shared instance, I wanted to prevent users from making potentially disruptive changes via the API (like deleting sources or teams). I used Caddy</code> as the reverse proxy and configured it to intercept requests to admin-specific API routes (like /api/v1/admin/</code>) and block any method other than GET</code>. If someone tries a POST</code>, PUT</code>, or DELETE</code> to these endpoints, Caddy returns a 403 Forbidden</code> directly. This keeps the demo environment stable.</p> Caddyfile snippet (conceptual)</span></span> handle /api/v1/admin/ {</span></span> @block_methods method POST PUT DELETE PATCH</span></span> respond @block_methods `{"error":"Operation not permitted in demo mode"}` 403</span></span> reverse_proxy logchef:8125 # Forward GET requests</span></span> }</span></span></code></pre></li> Improving Demo Login UX:</strong> Logchef uses OIDC for authentication. For the demo, I’m running Dex</code> as the OIDC provider. To make it completely frictionless for users, I didn’t want them needing to sign up or guess credentials. I simply customized Dex’s theme template for the login page to explicitly display the static demo username (demo@logchef.app</code>) and password (password</code>) right there. It’s a small UX tweak (again, thanks to Kailash for the idea!), but it means anyone landing on the demo can log in instantly.</p> <!--</span> Dex login template snippet </span>--></span></span> <</span>div</span> class</span>=</span>"</span>dex-info-box</span>"</span>></span></span> <</span>strong</span>></span>Demo Credentials:</span></</span>strong</span>></span><</span>br</span>></span></span> Username: </span><</span>code</span>></span>demo@logchef.app</span></</span>code</span>></span><</span>br</span>></span></span> Password: </span><</span>code</span>></span>password</span></</span>code</span>></span></span> </</span>div</span>></span></span> <</span>input</span> ...</span> value</span>=</span>"</span>demo@logchef.app</span>"</span> ...</span>/></span></span> <</span>input</span> ...</span> type</span>=</span>"</span>password</span>"</span> value</span>=</span>"</span>password</span>"</span> ...</span>/></span></span></code></pre></li> </ol> What’s Next?#</a></h2> Logchef is already being used internally, but the journey towards a full v1.0 release continues this year. The roadmap</a> includes exciting additions like:</p> Alerting: Trigger notifications based on query results.</li> Live Tail Logs: Stream logs in real-time.</li> Enhanced Dashboarding: More powerful visualization capabilities.</li> </ul> Logchef is open source (AGPLv3), and community involvement is welcomed. You can check out the Demo</a> or view the code on GitHub</a>.</p> If you have more ideas or features you’d like to see, please reach out on GitHub</a> or email me</a>! I’m always open to suggestions and feedback.</p> Honestly, building Logchef has been incredibly rewarding. It started as a way to fix something that bugged me (and others!), and seeing it turn into a tool I’m genuinely excited about feels great.</p> I couldn’t have done it alone, though. I’m really grateful to my friends and colleagues who jumped in with feedback along the way. Huge thanks to Kailash</a> for the constant support and encouragement, and to Vivek</a>, Sarat</a>, and Rohan</a> for testing the early builds and offering great suggestions.</p> Finally, a big thank you to my wife, who patiently endured my late-night coding sessions. Her support means the world to me <3</p> Fin!</p> Trying out NixOS 2025-03-23T10:15:00+05:30 I’ve been introduced to Nix by my colleagues at work. Being a Linux user for over a decade and a serial distro hopper, I was curious to learn more about it. I’d seen Nix mentioned before, but the comments about its steep learning curve made me wonder if the effort was worth it. I decided to give it a try by reading this excellent beginner’s guide</a> however got bored very quickly and decided to “learn on the fly”. I spun up a VM in my homelab to install NixOS using their official GUI installer image</a>.</p> Installation & First Impressions#</a></h2> The installation was as straightforward as any other Linux distro. NixOS is a declarative operating system that leverages the Nix functional package manager and a rich ecosystem of Nix packages. The flexibility is mind-blowing: you can configure everything—from user accounts and SSH keys to $SHELL</code> config and plugins entirely through code.</p> Once installed, the first place you’d want to poke around is the /etc/nixos</code> directory, which contains two essential configuration files:</p> hardware-configuration.nix</code>: Generated during installation (or regenerated with commands like nixos-generate-config</code>), it has hardware-specific details such as filesystem mount points, disk configurations, kernel modules etc. See an example file here</a>.</p> </li> configuration.nix</code>: This is the most</em> important file you want to start editing with. Here you define system-wide settings like timezone, locale, user accounts, and networking. Everything is declared in one place, making your system’s state reproducible.</p> </li> </ul> First Configuration Changes#</a></h3> When I opened the terminal, I immediately noticed that vim</code> wasn’t installed. So, I updated my configuration.nix</code> to include the packages I needed:</p> environment</span>.</span>systemPackages</span> =</span> with</span> pkgs</span>;</span> [</span></span> git</span></span> vim</span></span> ]</span>;</span></span></code></pre> After saving, I ran:</p> sudo</span> nixos-rebuild</span> switch</span></span></code></pre> This rebuilds the system using the new declarative configuration.</p> Version Control & Flakes#</a></h2> Next, I wanted to set up version control for my Nix configurations. The key takeaway is that while the system’s state is revertable in NixOS, your personal data (which includes configuration.nix</code>) isn’t automatically backed up. You must manage your own version history for your Nix configs. Since I was tweaking with no knowledge of Nix, having a version history was crucial.</p> I moved my /etc/nixos</code> configs to ~/Code/nixos-configs</code> and initialized a Git repository:</p> #</span> Create repo in home directory (better than root-owned /etc/nixos)</span></span> mkdir</span> ~/nixos-config</span></span> cp</span> -</span>r</span> /etc/nixos/</span></span> ~/nixos-config/</span></span> cd</span> ~/nixos-config</span></span> </span> #</span> Initialize Git</span></span> git</span> init</span></span> git</span> add</span> .</span></span> git</span> commit</span> -</span>m</span> "</span>Initial NixOS configuration</span>"</span></span> </span> #</span> Add GitHub remote</span></span> git</span> remote</span> add</span> origin</span> https://github.com/username/nixos-config.git</span></span> git</span> push</span> -</span>u</span> origin</span> main</span></span></code></pre> Here’s how flake.nix</code> looks:</p> {</span></span> description</span> =</span> "</span>NixOS configuration for Karan's homelab, servers, and personal dev machines</span>"</span>;</span></span> </span> inputs</span> =</span> {</span></span> nixpkgs</span>.</span>url</span> =</span> "</span>github:nixos/nixpkgs/nixos-unstable</span>"</span>;</span></span> </span></span> #</span> Add agenix as an input</span></span> agenix</span> =</span> {</span></span> url</span> =</span> "</span>github:ryantm/agenix</span>"</span>;</span></span> inputs</span>.</span>nixpkgs</span>.</span>follows</span> =</span> "</span>nixpkgs</span>"</span>;</span></span> }</span>;</span></span> #</span> Optionally add other inputs like home-manager</span></span> #</span> home-manager = {</span></span> #</span> url = "github:nix-community/home-manager/release-24.11";</span></span> #</span> inputs.nixpkgs.follows = "nixpkgs";</span></span> #</span> };</span></span> }</span>;</span></span> </span> outputs</span> =</span> {</span> self</span>,</span> agenix</span>,</span> nixpkgs</span>,</span> ...</span> }</span>@</span>inputs</span>:</span> {</span></span> #</span> Make agenix available as a package</span></span> packages</span>.</span>x86_64-linux</span>.</span>agenix</span> =</span> agenix</span>.</span>packages</span>.</span>x86_64-linux</span>.</span>default</span>;</span></span> </span></span> nixosConfigurations</span>.</span>work</span> =</span> nixpkgs</span>.</span>lib</span>.</span>nixosSystem</span> {</span></span> system</span> =</span> "</span>x86_64-linux</span>"</span>;</span></span> modules</span> =</span> [</span></span> ./configuration.nix</span></span> agenix</span>.</span>nixosModules</span>.</span>default</span> #</span> Add agenix module</span></span> ]</span>;</span></span> }</span>;</span></span> }</span>;</span></span> }</span></span></code></pre>A Note on Flakes#</a></h3> Flakes are an experimental (although widely adopted in the community) feature in Nix that bring reproducibility, composability, and a standardized structure to your configurations and package definitions. They allow you to declare all inputs (like nixpkgs, home-manager, or other repositories) and outputs (such as system configurations, packages, or development shells) in a single file. Flakes also create a lock file (flake.lock</code>) that pins your dependencies to specific revisions, ensuring that your builds remain reproducible over time.</p> I learned the hard way that—even for local configurations you must commit your files. Otherwise, you may see errors like:</p> path</span> '</span>/nix/store/...source/flake.nix</span>'</span> does</span> not</span> exist</span></span></code></pre> Even if you’re using local paths and have no intention to push to git</code>, you still need git init</code> && git add</code> for flakes to work. From whatever google-fu I did, it seems this requirement is to ensure that flakes can reliably reference the exact content in your configuration. I am sure there might be good reasons for it (as I said before, I’ve skipped RTFMing altogether ^_^), but atleast the errors can be more verbose/helpful.</p> And why I skipped docs: Remember, we’re on a mission to get things up and running with Nix and then later</em> spend time about reading their internals if it actually proves to be a valuable experiment.</p> Switching Channels#</a></h2> While installing packages, I noticed some packages were quite outdated. That’s when I learned about NixOS channels. Think of channels as analogous to LTS releases. For faster updates, you can switch to the unstable</code> channel. Although the name sounds intimidating, it simply means you’ll receive more frequent package updates.</p> To do this, you can edit your flake.nix</code> and switch the URL to an unstable channel:</p> inputs</span> =</span> {</span></span> -</span> nixpkgs.url</span> =</span> "</span>github:NixOS/nixpkgs/nixos-24.11</span>"</span>;</span> #</span> Stable channel</span></span> +</span> nixpkgs.url</span> =</span> "</span>github:NixOS/nixpkgs/nixos-unstable</span>"</span>;</span> #</span> Unstable channel</span></span> }</span>;</span></span></code></pre>Firmware Updates#</a></h2> After setting up packages, it was time to configure firmware updates using fwupd—essential</code> for keeping your hardware up to date.</p> I asked Claude to help me for a quick setup. Here’s what I did:</p> {</span> config</span>,</span> pkgs</span>,</span> ...</span> }</span>:</span></span> </span> {</span></span> services</span>.</span>fwupd</span>.</span>enable</span> =</span> true</span>;</span></span> }</span></span></code></pre> Then run a rebuild:</p> sudo</span> nixos-rebuild</span> switch</span></span></code></pre> Once enabled, you can use the fwupdmgr</code> command-line tool to manage firmware updates:</p> #</span> Refresh metadata and check for available updates</span></span> fwupdmgr</span> refresh</span></span> fwupdmgr</span> get-updates</span></span> #</span> Install available firmware updates</span></span> fwupdmgr</span> update</span></span></code></pre>Fine Tuning#</a></h2> I also tweaked some settings for the Nix package manager to optimize builds, caching, and overall performance. Here’s a snippet from my configuration:</p> #</span> Nix package manager optimizations</span></span> nix</span> =</span> {</span></span> settings</span> =</span> {</span></span> #</span> Optimize store to remove duplicate files</span></span> auto-optimise-store</span> =</span> true</span>;</span></span> </span> #</span> Allow building multiple derivations in parallel</span></span> max-jobs</span> =</span> "</span>auto</span>"</span>;</span></span> </span> #</span> Number of parallel build tasks per job</span></span> cores</span> =</span> 0</span>;</span> #</span> 0 means use all available cores</span></span> </span> #</span> Use the binary cache aggressively</span></span> substituters</span> =</span> [</span></span> "</span><https://cache.nixos.org></span>"</span></span> "</span><https://nix-community.cachix.org></span>"</span></span> "</span><https://nixpkgs-wayland.cachix.org></span>"</span></span> ]</span>;</span></span> </span> #</span> Optimize fetching from GitHub</span></span> connect-timeout</span> =</span> 5</span>;</span></span> </span> #</span> Prevent unneeded rebuilds</span></span> commit-lockfile-summary</span> =</span> "</span>Update flake.lock</span>"</span>;</span></span> }</span>;</span></span> </span> #</span> Garbage collection settings</span></span> gc</span> =</span> {</span></span> automatic</span> =</span> true</span>;</span></span> dates</span> =</span> "</span>weekly</span>"</span>;</span></span> options</span> =</span> "</span>--delete-older-than 30d</span>"</span>;</span></span> }</span>;</span></span> </span> #</span> Optimize builds using different build cores</span></span> buildCores</span> =</span> 0</span>;</span> #</span> 0 means use all available cores</span></span> </span> #</span> Enable flakes and modern Nix command features</span></span> extraOptions</span> =</span> '</span>'</span></span> experimental-features</span> =</span> nix-command</span> flakes</span></span> warn-dirty</span> =</span> false</span></span> keep-going</span> =</span> true</span></span> log-lines</span> =</span> 20</span></span> '</span>'</span>;</span></span> }</span>;</span></span></code></pre>Escape Hatches#</a></h2> So far things seems all rosy. Within just spending a couple of minutes - I had a perfectly working machine for myself - and the best part - all reproducible with a single command. I was starting to see why people who use NixOS preach about it so much.</p> However, not everything is smooth when you deviate from the happy path. For instance, I use Aider</a> for LLM assisted programming, but the version on Nixpkgs was about three minor versions</a> behind. Typically for any other software, I wouldn’t have cared so much - however with these LLM tools, a lot changes rapidly and I didn’t want to stay behind. Besides, it seemed like a fun exercise in getting my hands dirty by installing a Python package on NixOS which turned out to be quite tricky because Nix is absurdly obsessive about fully isolated builds.</p> Here’s an example flake that I used for attempting to install Aider with uv</code> in a dev shell (which didn’t work btw):</p> {</span></span> description</span> =</span> "</span>Aider development environment</span>"</span>;</span></span> </span> inputs</span> =</span> {</span></span> nixpkgs</span>.</span>url</span> =</span> "</span>github:NixOS/nixpkgs/nixos-unstable</span>"</span>;</span></span> flake-utils</span>.</span>url</span> =</span> "</span>github:numtide/flake-utils</span>"</span>;</span></span> }</span>;</span></span> </span> outputs</span> =</span> {</span> self</span>,</span> nixpkgs</span>,</span> flake-utils</span> }</span>:</span></span> flake-utils</span>.</span>lib</span>.</span>eachDefaultSystem</span> (</span>system</span>:</span></span> let</span></span> pkgs</span> =</span> nixpkgs</span>.</span>legacyPackages</span>.</span>${</span>system</span>}</span>;</span></span> in</span></span> {</span></span> devShell</span> =</span> pkgs</span>.</span>mkShell</span> {</span></span> buildInputs</span> =</span> with</span> pkgs</span>;</span> [</span></span> python312</span></span> uv</span></span> ]</span>;</span></span> shellHook</span> =</span> ''</span></span> export PATH="$HOME/.local/bin:$PATH"</span></span> ''</span>;</span></span> }</span>;</span></span> }</span></span> )</span>;</span></span> }</span></span></code></pre> Entering the dev shell with nix develop</code> and installing Aider with uv</code>:</p> uv</span> tool</span> install</span> -</span>-force</span> -</span>-python</span> python3.12</span> aider-chat@latest</span></span></code></pre> However, I ran into this error:</p> "</span>/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/litellm_core_uti</span></span> ls/llm_cost_calc/utils.py</span>"</span>,</span> line</span> 9,</span> in</span> <</span>modul</span>e</span>></span></span> from</span> litellm.utils</span> import</span> get_model_info</span></span> File</span></span> "</span>/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/utils.py</span>"</span>,</span> line</span></span> 53,</span> in</span> <</span>modul</span>e</span>></span></span> from</span> tokenizers</span> import</span> Tokenizer</span></span> File</span></span> "</span>/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/tokenizers/__init__.py</span>"</span>,</span></span> line</span> 78,</span> in</span> <</span>modul</span>e</span>></span></span> from</span> .tokenizers</span> import</span> (</span></span> ImportError:</span> libstdc++.so.6:</span> cannot</span> open</span> shared</span> object</span> file:</span> No</span> such</span> file</span> or</span> directory</span></span></code></pre> The error indicated that Aider was missing a required dependency libstdc++.so.6</code> which is a part of the C++ standard library needed by the tokenizers package. To fix this, I added stdenv.cc.cc.lib</code> (and even gcc</code> to be on the safer side) to my buildInputs</code>. This is because while uv</code> installs Python packages, it doesn’t handle system-level dependencies. In a Nix environment, every dependency, including system libraries, must be explicitly specified.</p> Frankly, Python’s packaging ecosystem is still a mess. Although tools like uv</code> help, achieving a completely isolated build, especially when shared libraries are involved is challenging. I wish the Python community would put more effort into resolving these issues.</p> While I was able to make aider</code> work by explicitly adding all the dependencies, I faced another outdate package: code-cursor</code>. Since this is a full blown electron app, I didn’t wish to package this myself.</p> After some frustration, I tried using Distrobox</a> as recommended by a colleague</a>. Distrobox lets you run containers that feel almost like a native OS by managing user IDs, host mounts, network interfaces, and more. I used an Arch Linux image, installed cursor-bin</code> from the AUR, and everything worked fine. Well mostly:</p> Fonts were missing. So, if I want to use custom fonts in my IDE - I need to have them installed in the container as well.</li> Since my fish</code> shell config had export EDITOR=nvim</code>, I had to install neovim</code> in the container as well, otherwise, I’d get an error when trying to git commit</code> etc. There’s an option to customise the shell</a> in distrobox, but for whatever reason (that I didn’t want to debug), it didn’t work for me.</li> </ul> Yet, something still felt off. The whole point of using NixOS is to achieve a fully declarative and reproducible setup. Resorting to an escape hatch like Distrobox undermines that goal. So I was very conflicted about this. I’m sure there’s a better way to handle these situations, and I should probably read the docs to find out.</p> Final Thoughts#</a></h2> I’m definitely sold on running NixOS, especially when managing multiple systems. With a single declarative file (configuration.nix</code>), duplicating your setup across machines becomes effortless. No more “documenting” (or rather forgetting</em> to document and keeping it updated)- as the config is the single source of truth.</p> Fun fact: I even messed up my NixOS build by misconfiguring the hardware-configuration.nix</code>, and my system became unusable even after a reboot, it couldn’t mount the filesystem on the correct device. In other distros, that would have sent me into panic mode, but with NixOS, all I had to do was revert to the previous working state, and everything was fine. That was so cool!</p> I’m definitely considering moving my homelab to NixOS in the coming few days because I honestly see the value for a server setup. I often set up my personal server and then forget everything I’ve done and I’m always scared of touching or creating a new server from scratch. I even created a small shell script installer</a> to help me for getting a base system ready. But like this shell script or even tools such as Ansible - they are all idempotent in nature. However in Nix, if I remove a certain piece from the configuration, there isn’t a trace of it left on the system. That makes it truly declarative and reproducible - unlike Ansible where you can still have some parts of the old setup.</p> However, for my primary machine at work, I’ll wait on the sidelines until the packages I depend on resolve their dependency issues and I get a chance to read up more on the escape hatches I tried to see if there’s a more streamlined way of doing things. I might be missing a lot of fundamental details since I skipped the docs entirely to get my hands dirty. But now that I see the value of a declarative system and especially how easy it is to roll back the machine to a previously known good state, I’m motivated to read up more on this and might post an update to this blog.</p> Fin!</p> Automating Badminton Game Alerts 2025-03-04T00:10:55+05:30 I’ve been playing badminton more regularly since the start of 2025 - almost 4-5 days a week. I recently moved to a new part of the city, which meant I couldn’t play with my old friends anymore. PlayO</a> has been super helpful for finding games with new people. On PlayO, a host creates a game and up to 6 people can join one court for a one-hour badminton doubles session.</p> However, on hectic days I would often forget to check for badminton games, only to find them fully booked later. I wanted to automate this process by creating a small script that would send me scheduled alerts about today’s game availability, allowing me to book slots before they filled up. I drew inspiration from Matt’s</a> post where he did something similar.</p> Thankfully, PlayO has a public API endpoint to retrieve a list of available games: https://api.playo.io/activity-public/list/location</code>.</p> You can send a POST</code> request to this URL with these parameters for filtering:</p> {</span></span> "</span>lat</span>"</span>:</span> 12.9783692</span>,</span></span> "</span>lng</span>"</span>:</span> 77.6408356</span>,</span></span> "</span>cityRadius</span>"</span>:</span> 5</span>,</span></span> "</span>gameTimeActivities</span>"</span>:</span> false</span>,</span></span> "</span>page</span>"</span>:</span> 0</span>,</span></span> "</span>lastId</span>"</span>:</span> "</span>"</span>,</span></span> "</span>sportId</span>"</span>:</span> [</span>"</span>SP5</span>"</span>]</span>,</span></span> "</span>booking</span>"</span>:</span> false</span>,</span></span> "</span>date</span>"</span>:</span> [</span>"</span>2025-03-04T11:03:17.260Z</span>"</span>]</span></span> }</span></span></code></pre> It returns a list of activities matching these filters. One such activity looks like:</p> {</span></span> "</span>userInfo</span>"</span>:</span> [</span></span> {</span></span> "</span>profilePicUrl</span>"</span>:</span> "</span>https://playov2.gumlet.io/profiles/redacted.511716.jpg</span>"</span>,</span></span> "</span>fName</span>"</span>:</span> "</span>Redacted</span>"</span>,</span></span> "</span>lName</span>"</span>:</span> "</span>"</span>,</span></span> "</span>karma</span>"</span>:</span> 2800</span></span> }</span>,</span></span> {</span></span> "</span>profilePicUrl</span>"</span>:</span> "</span>https://playov2.gumlet.io/profiles/redacted-redacted.png</span>"</span>,</span></span> "</span>fName</span>"</span>:</span> "</span>redacted</span>"</span>,</span></span> "</span>lName</span>"</span>:</span> "</span>N</span>"</span>,</span></span> "</span>karma</span>"</span>:</span> 499</span></span> }</span></span> ]</span>,</span></span> "</span>isPlayoGame</span>"</span>:</span> false</span>,</span></span> "</span>skill</span>"</span>:</span> "</span>Intermediate & above</span>"</span>,</span></span> "</span>sportName</span>"</span>:</span> "</span>Badminton</span>"</span>,</span></span> "</span>shortListed</span>"</span>:</span> false</span>,</span></span> "</span>joineeList</span>"</span>:</span> [</span></span> "</span>7f3cf298-3324-4fc2-96ad-b0f00093cd8f</span>"</span>,</span></span> "</span>250572a2-555d-4a77-94f0-452142c08f81</span>"</span>,</span></span> "</span>cc3b9eb6-a3b5-4c26-8605-0486fa000a4b</span>"</span>,</span></span> "</span>8d5d4299-950b-4011-a7ac-b466b1c00e84</span>"</span>,</span></span> "</span>235ae56d-6f4f-4106-9304-fb38e7d4add8</span>"</span></span> ]</span>,</span></span> "</span>isPlaypalPlaying</span>"</span>:</span> false</span>,</span></span> "</span>lat</span>"</span>:</span> 12.976394040119704</span>,</span></span> "</span>lng</span>"</span>:</span> 77.63644146986815</span>,</span></span> "</span>location</span>"</span>:</span> "</span>Game Theory - Double Road Indiranagar, Indiranagar</span>"</span>,</span></span> "</span>joineeCount</span>"</span>:</span> 6</span>,</span></span> "</span>status</span>"</span>:</span> -1</span>,</span></span> "</span>sportsPlayingMode</span>"</span>:</span> {</span></span> "</span>name</span>"</span>:</span> "</span>"</span>,</span></span> "</span>icon</span>"</span>:</span> "</span>"</span></span> }</span>,</span></span> "</span>maxPlayers</span>"</span>:</span> 7</span>,</span></span> "</span>full</span>"</span>:</span> false</span>,</span></span> "</span>price</span>"</span>:</span> 0</span>,</span></span> "</span>startTime</span>"</span>:</span> "</span>2025-03-04T13:30:00.000Z</span>"</span>,</span></span> "</span>endTime</span>"</span>:</span> "</span>2025-03-04T15:30:00.000Z</span>"</span>,</span></span> "</span>minSkill</span>"</span>:</span> 3</span>,</span></span> "</span>maxSkill</span>"</span>:</span> 5</span>,</span></span> "</span>skillSet</span>"</span>:</span> true</span>,</span></span> "</span>booking</span>"</span>:</span> false</span>,</span></span> "</span>bookingId</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> 0</span>,</span></span> "</span>venueId</span>"</span>:</span> "</span>82af038f-058c-4b2f-bc3d-3a47910d4f97</span>"</span>,</span></span> "</span>venueName</span>"</span>:</span> "</span>Game Theory - Double Road Indiranagar, Indiranagar</span>"</span>,</span></span> "</span>activityType</span>"</span>:</span> "</span>regular</span>"</span>,</span></span> "</span>isOnline</span>"</span>:</span> false</span>,</span></span> "</span>groupId</span>"</span>:</span> "</span>"</span>,</span></span> "</span>groupName</span>"</span>:</span> "</span>"</span>,</span></span> "</span>currencyTxt</span>"</span>:</span> "</span>INR</span>"</span>,</span></span> "</span>strictSkill</span>"</span>:</span> true</span>,</span></span> "</span>date</span>"</span>:</span> "</span>2025-03-04T00:00:00.000Z</span>"</span>,</span></span> "</span>hostId</span>"</span>:</span> "</span>redacted</span>"</span>,</span></span> "</span>sportId</span>"</span>:</span> "</span>SP5</span>"</span>,</span></span> "</span>timing</span>"</span>:</span> 2</span>,</span></span> "</span>id</span>"</span>:</span> "</span>e2ee9f62-c9b6-472b-aea2-b0c52dd7c525</span>"</span>,</span></span> "</span>distance</span>"</span>:</span> 0.5249236963063415</span>,</span></span> "</span>courtInfo</span>"</span>:</span> "</span>"</span>,</span></span> "</span>sponsored</span>"</span>:</span> false</span>,</span></span> "</span>groups</span>"</span>:</span> [</span>]</span></span> }</span></span></code></pre> Using the above response, I filtered for games where:</p> full</code> is false</code> (This indicates that joineeCount == maxPlayer</code> is not true, meaning spots are still available to join)</li> startTime</code> and endTime</code> fall within 7-8PM IST</li> </ul> I also wanted to add a feature to send these details to Telegram for convenient notifications. I then vibe coded</a> with Claude 3.7 to create a Python script to automate this whole process. Impressively, it produced a working script pretty much in a one-shot prompt, though I had to make a few minor tweaks. I quite like Simon Willison’s approach</a> of using uv</code> to build one-shot tools. Managing dependencies, virtual environments, etc. is still a pain point in Python, but using uv</code> feels like magic by comparison.</p> #</span> /// script</span></span> #</span> requires-python = ">=3.12"</span></span> #</span> dependencies = [</span></span> #</span> "click",</span></span> #</span> "requests",</span></span> #</span> "pytz",</span></span> #</span> "rich",</span></span> #</span> "python-dateutil",</span></span> #</span> "python-telegram-bot",</span></span> #</span> ]</span></span> #</span> ///</span></span> </span> import</span> click</span></span> import</span> requests</span></span> import</span> json</span></span> import</span> datetime</span></span> import</span> pytz</span></span> import</span> os</span></span> import</span> sys</span></span> from</span> rich</span>.</span>console</span> import</span> Console</span></span> from</span> rich</span>.</span>table</span> import</span> Table</span></span> from</span> dateutil</span> import</span> parser</span></span> from</span> telegram</span> import</span> Bot</span>,</span> InputMediaPhoto</span></span> from</span> telegram</span>.</span>constants</span> import</span> ParseMode</span></span> from</span> io</span> import</span> BytesIO</span></span> import</span> asyncio</span></span> </span> console</span> =</span> Console</span>(</span>)</span></span> </span> @</span>click</span>.</span>command</span>(</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--lat</span>"</span>,</span> default</span>=</span>12.9783692</span>,</span> help</span>=</span>"</span>Latitude for search</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--lng</span>"</span>,</span> default</span>=</span>77.6408356</span>,</span> help</span>=</span>"</span>Longitude for search</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--radius</span>"</span>,</span> default</span>=</span>50</span>,</span> help</span>=</span>"</span>City radius in km</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--sport</span>"</span>,</span> default</span>=</span>"</span>SP5</span>"</span>,</span> help</span>=</span>"</span>Sport ID (default: SP5 for Badminton)</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--start-time</span>"</span>,</span> default</span>=</span>"</span>19:00</span>"</span>,</span> help</span>=</span>"</span>Desired start time (24-hour format HH:MM)</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--end-time</span>"</span>,</span> default</span>=</span>"</span>20:00</span>"</span>,</span> help</span>=</span>"</span>Desired end time (24-hour format HH:MM)</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--timezone</span>"</span>,</span> default</span>=</span>"</span>Asia/Kolkata</span>"</span>,</span> help</span>=</span>"</span>Your timezone</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--verbose</span>"</span>,</span> is_flag</span>=</span>True</span>,</span> help</span>=</span>"</span>Show detailed information including exact UTC/IST times</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--include-full</span>"</span>,</span> is_flag</span>=</span>True</span>,</span> help</span>=</span>"</span>Include games that are full</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--telegram</span>"</span>,</span> is_flag</span>=</span>True</span>,</span> help</span>=</span>"</span>Send results to Telegram</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--telegram-token</span>"</span>,</span> envvar</span>=</span>"</span>TELEGRAM_BOT_TOKEN</span>"</span>,</span> help</span>=</span>"</span>Telegram Bot Token (or set TELEGRAM_BOT_TOKEN env var)</span>"</span>)</span></span> @</span>click</span>.</span>option</span>(</span>"</span>--telegram-chat-id</span>"</span>,</span> envvar</span>=</span>"</span>TELEGRAM_CHAT_ID</span>"</span>,</span> help</span>=</span>"</span>Telegram Chat ID (or set TELEGRAM_CHAT_ID env var)</span>"</span>)</span></span> def</span> find_games</span>(</span>lat</span>,</span> lng</span>,</span> radius</span>,</span> sport</span>,</span> start_time</span>,</span> end_time</span>,</span> timezone</span>,</span> verbose</span>,</span> include_full</span>,</span> telegram</span>,</span> telegram_token</span>,</span> telegram_chat_id</span>)</span>:</span></span> """</span>Find available badminton games on Playo matching your criteria.</span>"""</span></span> #</span> Get today's date in the specified timezone</span></span> local_tz</span> =</span> pytz</span>.</span>timezone</span>(</span>timezone</span>)</span></span> now</span> =</span> datetime</span>.</span>datetime</span>.</span>now</span>(</span>local_tz</span>)</span></span> today_date</span> =</span> now</span>.</span>strftime</span>(</span>"</span>%Y-%m-</span>%d</span>T%H:%M:%S.</span>%f</span>Z</span>"</span>)</span></span> </span> #</span> Parse desired time window</span></span> try</span>:</span></span> desired_start</span> =</span> datetime</span>.</span>datetime</span>.</span>strptime</span>(</span>start_time</span>,</span> "</span>%H:%M</span>"</span>)</span>.</span>time</span>(</span>)</span></span> desired_end</span> =</span> datetime</span>.</span>datetime</span>.</span>strptime</span>(</span>end_time</span>,</span> "</span>%H:%M</span>"</span>)</span>.</span>time</span>(</span>)</span></span> except</span> ValueError</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[bold red]Error:[/bold red] Invalid time format. Please use HH:MM (24-hour format).</span>"</span>)</span></span> return</span></span> </span> console</span>.</span>print</span>(</span>f</span>"</span>[bold green]Searching for badminton games around your location...[/bold green]</span>"</span>)</span></span> console</span>.</span>print</span>(</span>f</span>"</span>Looking for games between [bold]</span>{</span>start_time</span>}</span>[/bold] and [bold]</span>{</span>end_time</span>}</span>[/bold] IST today</span>"</span>)</span></span> </span> if</span> verbose</span>:</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[dim]Search parameters: lat=</span>{</span>lat</span>}</span>, lng=</span>{</span>lng</span>}</span>, radius=</span>{</span>radius</span>}</span>km[/dim]</span>"</span>)</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[dim]Current time in </span>{</span>timezone</span>}</span>: </span>{</span>now</span>.</span>strftime</span>(</span>'</span>%Y-%m-</span>%d</span> %H:%M:%S</span>'</span>)</span>}</span>[/dim]</span>"</span>)</span></span> </span> #</span> Prepare API request</span></span> url</span> =</span> "</span>https://api.playo.io/activity-public/list/location</span>"</span></span> payload</span> =</span> {</span></span> "</span>lat</span>"</span>:</span> lat</span>,</span></span> "</span>lng</span>"</span>:</span> lng</span>,</span></span> "</span>cityRadius</span>"</span>:</span> radius</span>,</span></span> "</span>gameTimeActivities</span>"</span>:</span> False</span>,</span></span> "</span>page</span>"</span>:</span> 0</span>,</span></span> "</span>lastId</span>"</span>:</span> "</span>"</span>,</span></span> "</span>sportId</span>"</span>:</span> [</span>sport</span>]</span>,</span></span> "</span>booking</span>"</span>:</span> False</span>,</span></span> "</span>date</span>"</span>:</span> [</span>today_date</span>]</span></span> }</span></span> </span> headers</span> =</span> {</span></span> "</span>Content-Type</span>"</span>:</span> "</span>application/json</span>"</span>,</span></span> "</span>User-Agent</span>"</span>:</span> "</span>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36</span>"</span></span> }</span></span> </span> try</span>:</span></span> response</span> =</span> requests</span>.</span>post</span>(</span>url</span>,</span> headers</span>=</span>headers</span>,</span> json</span>=</span>payload</span>)</span></span> response</span>.</span>raise_for_status</span>(</span>)</span></span> data</span> =</span> response</span>.</span>json</span>(</span>)</span></span> </span> if</span> data</span>.</span>get</span>(</span>"</span>requestStatus</span>"</span>)</span> !=</span> 1</span> or</span> "</span>data</span>"</span> not</span> in</span> data</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[bold red]Error:[/bold red] Failed to get valid response from Playo API</span>"</span>)</span></span> return</span></span> </span> #</span> Process activities</span></span> activities</span> =</span> data</span>[</span>"</span>data</span>"</span>]</span>.</span>get</span>(</span>"</span>activities</span>"</span>,</span> [</span>]</span>)</span></span> if</span> not</span> activities</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[yellow]No badminton activities found for today[/yellow]</span>"</span>)</span></span> return</span></span> </span> #</span> Filter activities based on criteria</span></span> matching_games</span> =</span> [</span>]</span></span> </span> for</span> activity</span> in</span> activities</span>:</span></span> #</span> Convert UTC times to local timezone</span></span> start_time_utc</span> =</span> parser</span>.</span>parse</span>(</span>activity</span>[</span>"</span>startTime</span>"</span>]</span>)</span></span> end_time_utc</span> =</span> parser</span>.</span>parse</span>(</span>activity</span>[</span>"</span>endTime</span>"</span>]</span>)</span></span> </span> start_time_local</span> =</span> start_time_utc</span>.</span>astimezone</span>(</span>local_tz</span>)</span></span> end_time_local</span> =</span> end_time_utc</span>.</span>astimezone</span>(</span>local_tz</span>)</span></span> </span> #</span> Print all times in debug mode</span></span> #</span> console.print(f"DEBUG: {activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST (UTC: {start_time_utc.strftime('%H:%M')})")</span></span> </span> #</span> Convert time objects correctly for comparison</span></span> start_hour</span> =</span> start_time_local</span>.</span>hour</span></span> start_minute</span> =</span> start_time_local</span>.</span>minute</span></span> </span> #</span> Convert desired times to hours and minutes for easier comparison</span></span> desired_start_hour</span> =</span> desired_start</span>.</span>hour</span></span> desired_start_minute</span> =</span> desired_start</span>.</span>minute</span></span> desired_end_hour</span> =</span> desired_end</span>.</span>hour</span></span> desired_end_minute</span> =</span> desired_end</span>.</span>minute</span></span> </span> #</span> Check if this game starts at 7PM (19:00) and ends at 8PM (20:00)</span></span> is_time_match</span> =</span> False</span></span> </span> #</span> Get duration in minutes</span></span> duration_minutes</span> =</span> (</span>(</span>end_time_local</span>.</span>hour</span> </span> 60</span> +</span> end_time_local</span>.</span>minute</span>)</span> -</span></span> (</span>start_time_local</span>.</span>hour</span> </span> 60</span> +</span> start_time_local</span>.</span>minute</span>)</span>)</span></span> </span> #</span> Check if start time is 7PM (with small tolerance)</span></span> if</span> (</span>start_hour</span> ==</span> desired_start_hour</span> and</span></span> start_minute</span> >=</span> desired_start_minute</span> and</span></span> start_minute</span> <</span> desired_start_minute</span> +</span> 10</span>)</span>:</span> #</span> Allow a small window of 10 minutes</span></span> </span> #</span> Check if duration is approximately 1 hour (between 50-70 minutes)</span></span> if</span> 50</span> <=</span> duration_minutes</span> <=</span> 70</span>:</span></span> is_time_match</span> =</span> True</span></span> </span> #</span> Check if there are available slots</span></span> is_available</span> =</span> (</span></span> not</span> activity</span>.</span>get</span>(</span>"</span>full</span>"</span>,</span> True</span>)</span> and</span></span> (</span>activity</span>.</span>get</span>(</span>"</span>maxPlayers</span>"</span>,</span> 0</span>)</span> ==</span> -</span>1</span> or</span></span> activity</span>.</span>get</span>(</span>"</span>joineeCount</span>"</span>,</span> 0</span>)</span> <</span> activity</span>.</span>get</span>(</span>"</span>maxPlayers</span>"</span>,</span> 0</span>)</span>)</span></span> )</span></span> </span> #</span> When verbose, print time details for each game to help debug</span></span> if</span> verbose</span>:</span></span> time_info</span> =</span> f</span>"</span>[dim]</span>{</span>activity</span>.</span>get</span>(</span>'</span>location</span>'</span>,</span> '</span>Unknown</span>'</span>)</span>}</span> - Start: </span>{</span>start_time_local</span>.</span>strftime</span>(</span>'</span>%H:%M</span>'</span>)</span>}</span> IST (</span>{</span>start_time_utc</span>.</span>strftime</span>(</span>'</span>%H:%M</span>'</span>)</span>}</span> UTC), </span>"</span> +</span> \</span></span> f</span>"</span>End: </span>{</span>end_time_local</span>.</span>strftime</span>(</span>'</span>%H:%M</span>'</span>)</span>}</span> IST, Duration: </span>{</span>duration_minutes</span>}</span> min, </span>"</span> +</span> \</span></span> f</span>"</span>Time match: </span>{</span>'</span>Yes</span>'</span> if</span> is_time_match</span> else</span> '</span>No</span>'</span>}</span>, Available: </span>{</span>'</span>Yes</span>'</span> if</span> is_available</span> else</span> '</span>No</span>'</span>}</span>[/dim]</span>"</span></span> console</span>.</span>print</span>(</span>time_info</span>)</span></span> </span> #</span> Both conditions must be true</span></span> if</span> is_time_match</span> and</span> is_available</span>:</span></span> matching_games</span>.</span>append</span>(</span>{</span></span> "</span>id</span>"</span>:</span> activity</span>[</span>"</span>id</span>"</span>]</span>,</span></span> "</span>location</span>"</span>:</span> activity</span>[</span>"</span>location</span>"</span>]</span>,</span></span> "</span>venue_name</span>"</span>:</span> activity</span>.</span>get</span>(</span>"</span>venueName</span>"</span>,</span> "</span>N/A</span>"</span>)</span>,</span></span> "</span>start</span>"</span>:</span> start_time_local</span>.</span>strftime</span>(</span>"</span>%I:%M %p</span>"</span>)</span>,</span></span> "</span>end</span>"</span>:</span> end_time_local</span>.</span>strftime</span>(</span>"</span>%I:%M %p</span>"</span>)</span>,</span></span> "</span>players</span>"</span>:</span> f</span>"</span>{</span>activity</span>.</span>get</span>(</span>'</span>joineeCount</span>'</span>,</span> 0</span>)</span>}</span>/</span>{</span>activity</span>.</span>get</span>(</span>'</span>maxPlayers</span>'</span>,</span> '</span>unlimited</span>'</span>)</span>}</span>"</span>,</span></span> "</span>host</span>"</span>:</span> activity</span>.</span>get</span>(</span>"</span>userInfo</span>"</span>,</span> [</span>{</span>}</span>]</span>)</span>[</span>0</span>]</span>.</span>get</span>(</span>"</span>fName</span>"</span>,</span> "</span>Unknown</span>"</span>)</span>,</span></span> "</span>skill</span>"</span>:</span> activity</span>.</span>get</span>(</span>"</span>skill</span>"</span>,</span> "</span>Any</span>"</span>)</span>,</span></span> "</span>price</span>"</span>:</span> f</span>"</span>{</span>activity</span>.</span>get</span>(</span>'</span>price</span>'</span>,</span> 0</span>)</span>}</span> {</span>activity</span>.</span>get</span>(</span>'</span>currencyTxt</span>'</span>,</span> '</span>INR</span>'</span>)</span>}</span>"</span></span> }</span>)</span></span> </span> #</span> Display results</span></span> if</span> matching_games</span>:</span></span> table</span> =</span> Table</span>(</span>title</span>=</span>f</span>"</span>Available Badminton Games (</span>{</span>len</span>(</span>matching_games</span>)</span>}</span> matches found)</span>"</span>)</span></span> </span> table</span>.</span>add_column</span>(</span>"</span>Location</span>"</span>,</span> style</span>=</span>"</span>cyan</span>"</span>)</span></span> table</span>.</span>add_column</span>(</span>"</span>Time</span>"</span>,</span> style</span>=</span>"</span>green</span>"</span>)</span></span> table</span>.</span>add_column</span>(</span>"</span>Players</span>"</span>,</span> style</span>=</span>"</span>yellow</span>"</span>)</span></span> table</span>.</span>add_column</span>(</span>"</span>Host</span>"</span>,</span> style</span>=</span>"</span>magenta</span>"</span>)</span></span> table</span>.</span>add_column</span>(</span>"</span>Skill Level</span>"</span>,</span> style</span>=</span>"</span>blue</span>"</span>)</span></span> table</span>.</span>add_column</span>(</span>"</span>Link</span>"</span>,</span> style</span>=</span>"</span>bright_blue</span>"</span>)</span></span> </span> for</span> game</span> in</span> matching_games</span>:</span></span> table</span>.</span>add_row</span>(</span></span> f</span>"</span>{</span>game</span>[</span>'</span>venue_name</span>'</span>]</span>}</span>"</span>,</span></span> f</span>"</span>{</span>game</span>[</span>'</span>start</span>'</span>]</span>}</span> - </span>{</span>game</span>[</span>'</span>end</span>'</span>]</span>}</span>"</span>,</span></span> game</span>[</span>"</span>players</span>"</span>]</span>,</span></span> game</span>[</span>"</span>host</span>"</span>]</span>,</span></span> game</span>[</span>"</span>skill</span>"</span>]</span>,</span></span> f</span>"</span>https://playo.co/match/</span>{</span>game</span>[</span>'</span>id</span>'</span>]</span>}</span>"</span></span> )</span></span> </span> console</span>.</span>print</span>(</span>table</span>)</span></span> </span> #</span> Send to Telegram if requested</span></span> if</span> telegram</span>:</span></span> if</span> not</span> telegram_token</span> or</span> not</span> telegram_chat_id</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[bold red]Error:[/bold red] Telegram token and chat ID are required for Telegram notifications</span>"</span>)</span></span> console</span>.</span>print</span>(</span>"</span>[dim]Set them with --telegram-token and --telegram-chat-id or via environment variables[/dim]</span>"</span>)</span></span> else</span>:</span></span> try</span>:</span></span> send_to_telegram</span>(</span>matching_games</span>,</span> telegram_token</span>,</span> telegram_chat_id</span>)</span></span> console</span>.</span>print</span>(</span>"</span>[green]Results sent to Telegram successfully![/green]</span>"</span>)</span></span> except</span> Exception</span> as</span> e</span>:</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[bold red]Error sending to Telegram:[/bold red] </span>{</span>e</span>}</span>"</span>)</span></span> else</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[yellow]No games found matching your criteria[/yellow]</span>"</span>)</span></span> if</span> telegram</span> and</span> telegram_token</span> and</span> telegram_chat_id</span>:</span></span> try</span>:</span></span> asyncio</span>.</span>run</span>(</span>send_telegram_message</span>(</span></span> "</span>No badminton games found matching your criteria for today.</span>"</span>,</span></span> telegram_token</span>,</span></span> telegram_chat_id</span></span> )</span>)</span></span> console</span>.</span>print</span>(</span>"</span>[green]Empty results notification sent to Telegram[/green]</span>"</span>)</span></span> except</span> Exception</span> as</span> e</span>:</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[bold red]Error sending to Telegram:[/bold red] </span>{</span>e</span>}</span>"</span>)</span></span> </span> except</span> requests</span>.</span>RequestException</span> as</span> e</span>:</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[bold red]Error:[/bold red] Failed to connect to Playo API: </span>{</span>e</span>}</span>"</span>)</span></span> except</span> json</span>.</span>JSONDecodeError</span>:</span></span> console</span>.</span>print</span>(</span>"</span>[bold red]Error:[/bold red] Failed to parse API response</span>"</span>)</span></span> except</span> Exception</span> as</span> e</span>:</span></span> console</span>.</span>print</span>(</span>f</span>"</span>[bold red]Error:[/bold red] An unexpected error occurred: </span>{</span>e</span>}</span>"</span>)</span></span> </span> def</span> send_to_telegram</span>(</span>games</span>,</span> token</span>,</span> chat_id</span>)</span>:</span></span> """</span>Send game information to Telegram as a nicely formatted message.</span>"""</span></span> if</span> not</span> games</span>:</span></span> return</span></span> </span> #</span> Create a formatted message for Telegram</span></span> message</span> =</span> "</span>🏸 Available Badminton Games* 🏸</span>\n</span>\n</span>"</span></span> </span> for</span> i</span>,</span> game</span> in</span> enumerate</span>(</span>games</span>,</span> 1</span>)</span>:</span></span> message</span> +=</span> f</span>"</span></span>{</span>i</span>}</span>. </span>{</span>game</span>[</span>'</span>venue_name</span>'</span>]</span>}</span></span>\n</span>"</span></span> message</span> +=</span> f</span>"</span>⏰ </span>{</span>game</span>[</span>'</span>start</span>'</span>]</span>}</span> - </span>{</span>game</span>[</span>'</span>end</span>'</span>]</span>}</span>\n</span>"</span></span> message</span> +=</span> f</span>"</span>👥 Players: </span>{</span>game</span>[</span>'</span>players</span>'</span>]</span>}</span>\n</span>"</span></span> message</span> +=</span> f</span>"</span>👤 Host: </span>{</span>game</span>[</span>'</span>host</span>'</span>]</span>}</span>\n</span>"</span></span> message</span> +=</span> f</span>"</span>🎯 Skill: </span>{</span>game</span>[</span>'</span>skill</span>'</span>]</span>}</span>\n</span>"</span></span> message</span> +=</span> f</span>"</span>🔗 [Join Game](https://playo.co/match/</span>{</span>game</span>[</span>'</span>id</span>'</span>]</span>}</span>)</span>\n</span>\n</span>"</span></span> </span> #</span> Send the message</span></span> asyncio</span>.</span>run</span>(</span>send_telegram_message</span>(</span>message</span>,</span> token</span>,</span> chat_id</span>)</span>)</span></span> </span> async</span> def</span> send_telegram_message</span>(</span>message</span>,</span> token</span>,</span> chat_id</span>)</span>:</span></span> """</span>Send a message to Telegram using the Bot API.</span>"""</span></span> bot</span> =</span> Bot</span>(</span>token</span>=</span>token</span>)</span></span> await</span> bot</span>.</span>send_message</span>(</span></span> chat_id</span>=</span>chat_id</span>,</span></span> text</span>=</span>message</span>,</span></span> parse_mode</span>=</span>ParseMode</span>.</span>MARKDOWN</span>,</span></span> disable_web_page_preview</span>=</span>False</span></span> )</span></span> </span> </span> if</span> __name__</span> ==</span> "</span>__main__</span>"</span>:</span></span> find_games</span>(</span>)</span></span></code></pre> The script outputs a beautiful output:</p> </p> Telegram:</p> </p> Scheduling#</a></h2> I wanted this script to run reliably every day and used GitHub Actions for that. GitHub Actions felt like the path of least resistance as I didn’t have to worry about keeping a server running or getting alerts if something crashed. For a small personal script like this, it was the perfect “set it and forget it” solution.</p> n</span>ame</span>:</span> B</span>adminton Game Checker Base</span></span> on</span>:</span></span> s</span>chedule</span>:</span></span> #</span> Run Monday to Friday at 12:00 PM IST (6:30 AM UTC)</span></span> -</span> c</span>ron</span>:</span> "</span>30 6 * * 1-5</span>"</span></span> </span> j</span>obs</span>:</span></span> c</span>heck-games</span>:</span></span> r</span>uns-on</span>:</span> u</span>buntu-latest</span></span> </span> e</span>nv</span>:</span></span> T</span>ELEGRAM_BOT_TOKEN</span>:</span> $</span>{{ secrets.TELEGRAM_BOT_TOKEN }}</span></span> T</span>ELEGRAM_CHAT_ID</span>:</span> $</span>{{ secrets.TELEGRAM_CHAT_ID }}</span></span> L</span>ATITUDE</span>:</span> $</span>{{ inputs.latitude }}</span></span> L</span>ONGITUDE</span>:</span> $</span>{{ inputs.longitude }}</span></span> R</span>ADIUS</span>:</span> $</span>{{ inputs.radius }}</span></span> S</span>PORT_ID</span>:</span> $</span>{{ inputs.sport_id }}</span></span> T</span>IMEZONE</span>:</span> $</span>{{ inputs.timezone }}</span></span> S</span>TART_TIME</span>:</span> $</span>{{ inputs.start_time }}</span></span> E</span>ND_TIME</span>:</span> $</span>{{ inputs.end_time }}</span></span> </span> s</span>teps</span>:</span></span> -</span> n</span>ame</span>:</span> C</span>heckout code</span></span> u</span>ses</span>:</span> a</span>ctions/checkout@v4</span></span> </span> -</span> n</span>ame</span>:</span> S</span>et up Python</span></span> u</span>ses</span>:</span> a</span>ctions/setup-python@v5</span></span> w</span>ith</span>:</span></span> p</span>ython-version</span>:</span> "</span>3.12</span>"</span></span> c</span>ache</span>:</span> "</span>pip</span>"</span></span> </span> -</span> n</span>ame</span>:</span> I</span>nstall dependencies</span></span> r</span>un</span>:</span> \|</span></span> python -m pip install --upgrade pip</span></span> pip install uv</span></span> </span> -</span> n</span>ame</span>:</span> R</span>un game check</span></span> r</span>un</span>:</span> \|</span></span> echo "Checking for games from $START_TIME to $END_TIME"</span></span> uv run finder.py \</span></span> --lat "$LATITUDE" \</span></span> --lng "$LONGITUDE" \</span></span> --radius "$RADIUS" \</span></span> --sport "$SPORT_ID" \</span></span> --timezone "$TIMEZONE" \</span></span> --start-time "$START_TIME" \</span></span> --end-time "$END_TIME" \</span></span> --telegram</span></span> e</span>nv</span>:</span></span> T</span>ELEGRAM_BOT_TOKEN</span>:</span> $</span>{{ secrets.TELEGRAM_BOT_TOKEN }}</span></span> T</span>ELEGRAM_CHAT_ID</span>:</span> $</span>{{ secrets.TELEGRAM_CHAT_ID }}</span></span></code></pre> I used GitHub Actions inputs to configure the variables for my script. Found this feature to be quite neat for scheduling different crons for weekday/weekends.</p> </p> Summary#</a></h2> For small quality-of-life improvements - solving your own specific problems with custom scripts tailored exactly to your needs - gotta love the LLMs man. We’re gonna see more and more of such “personal tooling” in future as the entry to barrier for coding is lowered with LLMs. The democratization of coding through LLMs means people (even non-technical ones) can focus on “describing” the problem well, rather than struggling with implementation details. Being able to articulate what you want clearly becomes the primary skill - yes, it’s a skill issue if you can’t prompt well, but it’s far more accessible than learning programming from scratch.</p> Fin!</p> Cleaning up Notes with LLM 2025-01-03T10:15:00+05:30 My Obsidian vault has gotten quite messy over time. I’ve been dumping notes without proper frontmatter, tags were all over the place, and some notes didn’t even have proper titles! I needed a way to clean this up without spending hours manually organizing everything.</p> I’d been playing around with Claude’s API lately, and thought – hey, why not use an LLM to analyze my notes and add proper frontmatter? After all, that’s what these AI models are good at – understanding context and categorizing stuff.</p> I wrote a small Python script using the llm</a> library (which is pretty neat btw) to do just this. Here’s what it looks like:</p> import</span> llm</span></span> import</span> os</span></span> import</span> yaml</span></span> import</span> datetime</span></span> from</span> pathlib</span> import</span> Path</span></span> import</span> re</span></span> </span> class</span> ObsidianNoteProcessor</span>:</span></span> def</span> __init__</span>(</span>self</span>,</span> notes_dir</span>,</span> model_name</span>=</span>"</span>claude-3.5-sonnet</span>"</span>)</span>:</span></span> self</span>.</span>notes_dir</span> =</span> Path</span>(</span>notes_dir</span>)</span></span> self</span>.</span>model</span> =</span> llm</span>.</span>get_model</span>(</span>model_name</span>)</span></span> </span></span> def</span> extract_existing_frontmatter</span>(</span>self</span>,</span> content</span>)</span>:</span></span> """</span>Extract existing frontmatter if present.</span>"""</span></span> frontmatter_pattern</span> =</span> r</span>'</span>^</span>---</span>\n</span>(</span>.</span>?</span>)</span>\n</span>---</span>\n</span>'</span></span> match</span> =</span> re</span>.</span>match</span>(</span>frontmatter_pattern</span>,</span> content</span>,</span> re</span>.</span>DOTALL</span>)</span></span> </span></span> if</span> match</span>:</span></span> try</span>:</span></span> return</span> yaml</span>.</span>safe_load</span>(</span>match</span>.</span>group</span>(</span>1</span>)</span>)</span>,</span> content</span>[</span>match</span>.</span>end</span>(</span>)</span>:</span>]</span></span> except</span> yaml</span>.</span>YAMLError</span>:</span></span> return</span> {</span>}</span>,</span> content</span></span> return</span> {</span>}</span>,</span> content</span></span> </span> def</span> generate_prompt</span>(</span>self</span>,</span> content</span>)</span>:</span></span> """</span>Generate a prompt for the LLM to analyze the note content.</span>"""</span></span> return</span> f</span>"""</span>Analyze the following note content and extract/infer the following properties:</span></span> 1. A clear title (if not present, generate from content)</span></span> 2. Relevant categories based on the content</span></span> 3. Appropriate tags (include 'inbox' if content seems draft-like)</span></span> 4. Status (Draft/In Progress/Complete) based on content completeness</span></span> 5. Priority (Low/Medium/High) based on content importance</span></span> 6. A brief description summarizing the content</span></span> </span> Note content:</span></span> {</span>content</span>}</span></span> </span> Return ONLY the YAML frontmatter without any code block markers. Use this exact format (omit fields if not applicable):</span></span> title: <title></span></span> category: <category></span></span> tags:</span></span> - tag1</span></span> - tag2</span></span> status: <status></span></span> priority: <priority></span></span> description: <description></span>"""</span></span> </span> def</span> clean_llm_response</span>(</span>self</span>,</span> response_text</span>)</span>:</span></span> """</span>Clean up the LLM response to ensure proper YAML.</span>"""</span></span> #</span> Remove yaml code block markers if present</span></span> response_text</span> =</span> response_text</span>.</span>strip</span>(</span>)</span></span> if</span> response_text</span>.</span>startswith</span>(</span>'</span>```yaml</span>'</span>)</span>:</span></span> response_text</span> =</span> response_text</span>.</span>split</span>(</span>'</span>\n</span>'</span>,</span> 1</span>)</span>[</span>1</span>]</span></span> if</span> response_text</span>.</span>endswith</span>(</span>'</span>```</span>'</span>)</span>:</span></span> response_text</span> =</span> response_text</span>.</span>rsplit</span>(</span>'</span>\n</span>'</span>,</span> 1</span>)</span>[</span>0</span>]</span></span> return</span> response_text</span>.</span>strip</span>(</span>)</span></span> </span> def</span> process_note</span>(</span>self</span>,</span> file_path</span>)</span>:</span></span> """</span>Process a single note file.</span>"""</span></span> try</span>:</span></span> with</span> open</span>(</span>file_path</span>,</span> '</span>r</span>'</span>,</span> encoding</span>=</span>'</span>utf-8</span>'</span>)</span> as</span> f</span>:</span></span> content</span> =</span> f</span>.</span>read</span>(</span>)</span></span> </span></span> #</span> Extract existing frontmatter and content</span></span> existing_frontmatter</span>,</span> main_content</span> =</span> self</span>.</span>extract_existing_frontmatter</span>(</span>content</span>)</span></span> </span></span> #</span> Generate and execute prompt</span></span> response</span> =</span> self</span>.</span>model</span>.</span>prompt</span>(</span>self</span>.</span>generate_prompt</span>(</span>main_content</span>)</span>)</span></span> response_text</span> =</span> self</span>.</span>clean_llm_response</span>(</span>response</span>.</span>text</span>(</span>)</span>)</span></span> </span></span> try</span>:</span></span> new_frontmatter</span> =</span> yaml</span>.</span>safe_load</span>(</span>response_text</span>)</span></span> if</span> not</span> isinstance</span>(</span>new_frontmatter</span>,</span> dict</span>)</span>:</span></span> print</span>(</span>f</span>"</span>Warning: Invalid response format for </span>{</span>file_path</span>.</span>name</span>}</span>"</span>)</span></span> new_frontmatter</span> =</span> {</span>}</span></span> except</span> yaml</span>.</span>YAMLError</span> as</span> e</span>:</span></span> print</span>(</span>f</span>"</span>YAML parsing error for </span>{</span>file_path</span>.</span>name</span>}</span>"</span>)</span></span> print</span>(</span>f</span>"</span>Response text was:</span>\n</span>{</span>response_text</span>}</span>"</span>)</span></span> raise</span> e</span></span> </span></span> #</span> Merge with existing frontmatter, preferring existing values</span></span> merged_frontmatter</span> =</span> {</span></span>new_frontmatter</span>,</span> </span>existing_frontmatter</span>}</span></span> </span></span> #</span> Add date if not present</span></span> if</span> '</span>date</span>'</span> not</span> in</span> merged_frontmatter</span>:</span></span> merged_frontmatter</span>[</span>'</span>date</span>'</span>]</span> =</span> datetime</span>.</span>date</span>.</span>today</span>(</span>)</span>.</span>isoformat</span>(</span>)</span></span> </span></span> #</span> Generate new note content</span></span> new_content</span> =</span> "</span>---</span>\n</span>"</span></span> new_content</span> +=</span> yaml</span>.</span>dump</span>(</span>merged_frontmatter</span>,</span> sort_keys</span>=</span>False</span>,</span> allow_unicode</span>=</span>True</span>)</span></span> new_content</span> +=</span> "</span>---</span>\n</span>\n</span>"</span></span> new_content</span> +=</span> main_content</span>.</span>strip</span>(</span>)</span></span> </span></span> #</span> Write back to file</span></span> with</span> open</span>(</span>file_path</span>,</span> '</span>w</span>'</span>,</span> encoding</span>=</span>'</span>utf-8</span>'</span>)</span> as</span> f</span>:</span></span> f</span>.</span>write</span>(</span>new_content</span>)</span></span> </span></span> print</span>(</span>f</span>"</span>✓ Processed: </span>{</span>file_path</span>.</span>name</span>}</span>"</span>)</span></span> </span></span> except</span> Exception</span> as</span> e</span>:</span></span> print</span>(</span>f</span>"</span>✗ Error processing </span>{</span>file_path</span>.</span>name</span>}</span>: </span>{</span>str</span>(</span>e</span>)</span>}</span>"</span>)</span></span> </span> def</span> process_vault</span>(</span>self</span>)</span>:</span></span> """</span>Process all markdown files in the vault.</span>"""</span></span> print</span>(</span>"</span>Starting Obsidian vault cleanup...</span>"</span>)</span></span> </span></span> for</span> file_path</span> in</span> self</span>.</span>notes_dir</span>.</span>glob</span>(</span>'</span>/.md</span>'</span>)</span>:</span></span> self</span>.</span>process_note</span>(</span>file_path</span>)</span></span> </span></span> print</span>(</span>"</span>\n</span>Vault cleanup completed!</span>"</span>)</span></span> </span> def</span> main</span>(</span>)</span>:</span></span> #</span> Set up the model key if not already configured</span></span> model</span> =</span> llm</span>.</span>get_model</span>(</span>"</span>claude-3.5-sonnet</span>"</span>)</span></span> if</span> not</span> hasattr</span>(</span>model</span>,</span> '</span>key</span>'</span>)</span>:</span></span> api_key</span> =</span> os</span>.</span>getenv</span>(</span>'</span>ANTHROPIC_API_KEY</span>'</span>)</span></span> if</span> not</span> api_key</span>:</span></span> raise</span> ValueError</span>(</span>"</span>Please set ANTHROPIC_API_KEY environment variable</span>"</span>)</span></span> model</span>.</span>key</span> =</span> api_key</span></span> </span></span> #</span> Initialize and run the processor</span></span> notes_dir</span> =</span> "</span>/Users/karan/Notes/Obsidian/The Wall/Notes</span>"</span></span> processor</span> =</span> ObsidianNoteProcessor</span>(</span>notes_dir</span>)</span></span> processor</span>.</span>process_vault</span>(</span>)</span></span> </span> if</span> __name__</span> ==</span> "</span>__main__</span>"</span>:</span></span> main</span>(</span>)</span></span></code></pre> </p> The script is pretty straightforward – it reads each markdown file, extracts any existing frontmatter (because I don’t want to lose that!), and then asks Claude to analyze the content and generate appropriate frontmatter. It adds stuff like title, category, tags, status, priority.</p> What I love about this approach is that it’s contextual</em>. Unlike regex-based approaches or keyword matching, the LLM actually understands what the note is about and can categorize it properly. A note about “Setting up BTRFS on Arch” automatically gets tagged with “linux”, “filesystem”, “arch” without me having to maintain a predefined list of tags. The categorization is probably better than what I’d have done manually at 2 AM while organizing my notes!</p> Fin!</p> 2024: A Year In Review 2024-12-23T12:00:00+05:30 2024 was indeed an important year for me as it marked several significant milestones. Quite happy with how this year was! Here’s my reflection on this memorable year.</p> Life#</a></h2> Got married to the prettiest and dearest Saumya 💗</li> Did my first international trip, exploring Europe</li> Bought a fun toy - Maruti Jimny 4x4</li> Relocated to Bangalore after working from home for 4+ years since Covid</li> Attended several amazing concerts: Indian Ocean</li> Parvaaz</li> Blackstratblues</li> Anand Bhaskar Collective</li> Bandland</li> </ul> </li> </ul> Travel#</a></h2> Switzerland</li> Italy</li> Austria</li> Binsar</li> Cochin</li> Ajmer</li> Ranthambore</li> Pondicherry</li> Chennai</li> </ul> Projects#</a></h2> Released v1.0.0 of Doggo</a> - It hit frontpage of HN</a> as well!</li> Built an expense tracker app - Gullak</a></li> Made a lot of small utility apps: lil</a> - URL shortener</li> silencer</a> - Prometheus alerts <> Mattermost bridge</li> toru</a> - Go modules proxy with caching</li> junbi</a> - Server Setup and Hardening Tool</li> Ovenly Delights</a> - Small bakery shop website</li> nomcfg</a> - Nomad config generator</li> clx</a> - Generate CLI commands using AI for common ops</li> </ul> </li> Started working on a log analytics app - full focus on that in 2025. Read more</a></li> </ul> This year has been truly transformative, bringing together personal joy, professional growth, and exciting adventures. Looking forward to what 2025 has in store!</p> How I use LLMs 2024-10-30T00:00:00+00:00 Just yesterday, GitHub announced</a> integrating Claude 3.5 Sonnet with Copilot. Interesting times ahead. In my experience, Claude has been remarkably better than the GPT-4 family of models for programming tasks. I’ve tried a bunch of tools like Cursor, Continue.dev but finally settled with Aider</a> for most of my tasks. In this post, I want to write about my workflow of using Aider when working on small coding tasks.</p> Aider is an open source Python CLI which supports multiple models, including Claude 3.5 Sonnet. Aider describes itself as “AI pair programming in your terminal”. The tool integrates git</code> quite well in its workflow so it can edit files, create new files, and track all changes via git. In case you want to revert, simply reverting the commit or using the /undo</code> shortcut would do the same.</p> The tool has multiple modes that serve different purposes:</p> /ask</code>: Use it when you simply want to chat with the model about the codebase or explain some pieces of it. This mode won’t touch your files. It’s great for understanding existing code or getting explanations.</li> /architect</code>: Use it to discuss a broad overall idea. The model will propose some changes to your files. You can further chat and tune it to your preferences.</li> /code</code>: This will directly edit your files and commit them.</li> </ul> My typical workflow involves running Aider in a terminal while keeping VSCode open for manual code review. I often use the --no-auto-commits</code> flag to view the diffs before committing. Despite advances in LLM technology, I believe they haven’t yet reached the stage where they can fully understand your team’s coding style guides, and I prefer not to have a certain style forced upon me. Manually tweaking portions of AI-generated functions still proves helpful and saves considerable time.</p> To begin, aider --sonnet</code> would open the interactive window where you can begin writing prompts.</p> </p> To add context, you need to add files using commands like /add main.py</code>. What makes Aider powerful is its control over the LLM context - you can /add</code> or /drop</code> source code, or even /reset</code> to drop all files and start with a fresh context. This granular control helps manage the context window effectively.</p> A really cool thing about it is that it gives an approximate idea of the number of tokens (cost) associated with each prompt. I find it useful to remove unnecessary files from the context window, which not only helps in getting sharper, more accurate responses but also helps with the costs. There’s a nice /tokens</code> command which will show the cost of sending each file added in context with the prompt.</p> </p> I find the Aider + Claude 3.5 combo works really well when you have a narrow-scoped, well-defined task. For example, this is the prompt I used on a codebase I was working on:</p> Theme preference is not preserved when reloading pages or navigating to new pages. We should store this setting in localStorage. Please implement using standard best practices.</p> </blockquote> </p> Under the hood, Aider uses tree-sitter</a> to improve code generation and provide rich context about your codebase. Tree-sitter parses your code into an Abstract Syntax Tree (AST), which helps Aider understand the structure and relationships in your code. Unlike simpler tools that might just grep through your codebase, tree-sitter understands the actual syntax of your programming language.</p> It can identify function definitions, class declarations, variable scopes, and their relationships</li> It extracts full function signatures and type information</li> It builds a dependency graph showing how different parts of your code relate to each other</li> It helps rank the importance of different code sections based on how often they’re referenced</li> </ul> This means when you’re working on a task, Aider isn’t just blindly sending your entire codebase to the LLM. Instead, it creates an optimized “repository map” that fits within your token budget (default is 1k tokens, adjustable via --map-tokens</code>). This map focuses on the most relevant pieces of your code, making sure the LLM understands the context without wasting tokens.</p> Aider’s approach to AI pair programming feels natural and productive. Here are some example prompts where it helped me build stuff in less than a minute:</p> Modify fetch method in store/store.go to filter out expired entries</p> </blockquote> Write a k6 load test script to benchmark the POST /submit</code> endpoint and simulate real-world traffic patterns</p> </blockquote> Create a Makefile, Dockerfile, goreleaser.yml for my Go binary. Target platforms: arm64 and amd64</p> </blockquote> </p> I prefer to invoke aider</code> with a few extra flags:</p> aider</span> -</span>-no-auto-commits</span> -</span>-cache-prompts</span> -</span>-cache-keepalive-pings</span> 12</span> -</span>-no-suggest-shell-commands</span></span></code></pre> Make sure to go through the Tips</a> page to effectively try out Aider on your existing projects.</p> Fin!</p> Self Hosting Outline Wiki 2024-09-20T00:00:00+00:00 I recently discovered Outline</a> a collaborative knowledge base. I wanted to self-host it on my server, but the mandatory auth provider requirement was off-putting. My server is on a private encrypted network (Tailscale) that only my approved devices in the tailnet can access, so I don’t really need authentication for my personal single-use apps. I found a few guides using Authelia/Keycloak, but these are heavy-duty applications that would consume a lot of resources (DBs, caches, proxies, and whatnot) just to have an OIDC provider for Outline.</p> There had to be a simpler way, right? Enter Dex</a>. As recommended by my friend and colleague Chinmay</a>, it turned out to be quite easy.</p> Here’s the full docker-compose.yml</code> setup you need to get Outline up and running on your local instance!</p> s</span>ervices</span>:</span></span> </span> o</span>utline</span>:</span></span> i</span>mage</span>:</span> d</span>ocker.getoutline.com/outlinewiki/outline:latest</span></span> e</span>nv_file</span>:</span> .</span>/docker.env</span></span> p</span>orts</span>:</span></span> -</span> "</span>3000:3000</span>"</span></span> v</span>olumes</span>:</span></span> -</span> s</span>torage-data:/var/lib/outline/data</span></span> d</span>epends_on</span>:</span></span> -</span> p</span>ostgres</span></span> -</span> r</span>edis</span></span> e</span>nvironment</span>:</span></span> P</span>GSSLMODE</span>:</span> d</span>isable</span></span> </span> r</span>edis</span>:</span></span> i</span>mage</span>:</span> r</span>edis</span></span> e</span>nv_file</span>:</span> .</span>/docker.env</span></span> p</span>orts</span>:</span></span> -</span> "</span>6379:6379</span>"</span></span> h</span>ealthcheck</span>:</span></span> t</span>est</span>:</span> [</span> "</span>CMD</span>"</span>,</span> "</span>redis-cli</span>"</span>,</span> "</span>ping</span>"</span> ]</span></span> i</span>nterval</span>:</span> 1</span>0s</span></span> t</span>imeout</span>:</span> 3</span>0s</span></span> r</span>etries</span>:</span> 3</span></span> </span> p</span>ostgres</span>:</span></span> i</span>mage</span>:</span> p</span>ostgres</span></span> e</span>nv_file</span>:</span> .</span>/docker.env</span></span> p</span>orts</span>:</span></span> -</span> "</span>5432:5432</span>"</span></span> v</span>olumes</span>:</span></span> -</span> d</span>atabase-data:/var/lib/postgresql/data</span></span> h</span>ealthcheck</span>:</span></span> t</span>est</span>:</span> [</span> "</span>CMD</span>"</span>,</span> "</span>pg_isready</span>"</span>,</span> "</span>-d</span>"</span>,</span> "</span>outline</span>"</span>,</span> "</span>-U</span>"</span>,</span> "</span>user</span>"</span> ]</span></span> i</span>nterval</span>:</span> 3</span>0s</span></span> t</span>imeout</span>:</span> 2</span>0s</span></span> r</span>etries</span>:</span> 3</span></span> e</span>nvironment</span>:</span></span> P</span>OSTGRES_USER</span>:</span> '</span>user</span>'</span></span> P</span>OSTGRES_PASSWORD</span>:</span> '</span>pass</span>'</span></span> P</span>OSTGRES_DB</span>:</span> '</span>outline</span>'</span></span> </span> d</span>ex</span>:</span></span> i</span>mage</span>:</span> d</span>exidp/dex:v2.35.3</span></span> p</span>orts</span>:</span></span> -</span> "</span>5556:5556</span>"</span></span> v</span>olumes</span>:</span></span> -</span> .</span>/dex:/etc/dex</span></span> c</span>ommand</span>:</span> [</span> "</span>dex</span>"</span>,</span> "</span>serve</span>"</span>,</span> "</span>/etc/dex/config.yaml</span>"</span> ]</span></span> </span> v</span>olumes</span>:</span></span> s</span>torage-data</span>:</span></span> d</span>atabase-data</span>:</span></span> </span></code></pre> You’ll need to add the following env variables as well</p> NODE_ENV</span>=</span>p</span>r</span>o</span>d</span>u</span>c</span>t</span>i</span>o</span>n</span></span> SECRET_KEY</span>=</span>y</span>o</span>u</span>r</span>-</span>k</span>e</span>y</span></span> UTILS_SECRET</span>=</span>y</span>o</span>u</span>r</span>-</span>k</span>e</span>y</span></span> DATABASE_URL</span>=</span>p</span>o</span>s</span>t</span>g</span>r</span>e</span>s</span>:</span>/</span>/</span>u</span>s</span>e</span>r</span>:</span>p</span>a</span>s</span>s</span>@</span>p</span>o</span>s</span>t</span>g</span>r</span>e</span>s</span>:</span>5</span>4</span>3</span>2</span>/</span>o</span>u</span>t</span>l</span>i</span>n</span>e</span></span> PGSSLMODE</span>=</span>d</span>i</span>s</span>a</span>b</span>l</span>e</span></span> REDIS_URL</span>=</span>r</span>e</span>d</span>i</span>s</span>:</span>/</span>/</span>r</span>e</span>d</span>i</span>s</span>:</span>6</span>3</span>7</span>9</span></span> URL</span>=</span>h</span>t</span>t</span>p</span>:</span>/</span>/</span>l</span>o</span>c</span>a</span>l</span>h</span>o</span>s</span>t</span>:</span>3</span>0</span>0</span>0</span></span> PORT</span>=</span>3</span>0</span>0</span>0</span></span> FILE_STORAGE</span>=</span>l</span>o</span>c</span>a</span>l</span></span> FILE_STORAGE_LOCAL_ROOT_DIR</span>=</span>/</span>v</span>a</span>r</span>/</span>l</span>i</span>b</span>/</span>o</span>u</span>t</span>l</span>i</span>n</span>e</span>/</span>d</span>a</span>t</span>a</span></span> FILE_STORAGE_UPLOAD_MAX_SIZE</span>=</span>2</span>6</span>2</span>1</span>4</span>4</span>0</span>0</span>0</span></span> OIDC_CLIENT_ID</span>=</span>o</span>u</span>t</span>l</span>i</span>n</span>e</span></span> OIDC_CLIENT_SECRET</span>=</span>o</span>u</span>t</span>l</span>i</span>n</span>e</span>-</span>s</span>e</span>c</span>r</span>e</span>t</span></span> OIDC_AUTH_URI</span>=</span>h</span>t</span>t</span>p</span>:</span>/</span>/</span>l</span>o</span>c</span>a</span>l</span>h</span>o</span>s</span>t</span>:</span>5</span>5</span>5</span>6</span>/</span>d</span>e</span>x</span>/</span>a</span>u</span>t</span>h</span></span> OIDC_TOKEN_URI</span>=</span>h</span>t</span>t</span>p</span>:</span>/</span>/</span>d</span>e</span>x</span>:</span>5</span>5</span>5</span>6</span>/</span>d</span>e</span>x</span>/</span>t</span>o</span>k</span>e</span>n</span></span> OIDC_USERINFO_URI</span>=</span>h</span>t</span>t</span>p</span>:</span>/</span>/</span>d</span>e</span>x</span>:</span>5</span>5</span>5</span>6</span>/</span>d</span>e</span>x</span>/</span>u</span>s</span>e</span>r</span>i</span>n</span>f</span>o</span></span> OIDC_USERNAME_CLAIM</span>=</span>p</span>r</span>e</span>f</span>e</span>r</span>r</span>e</span>d</span>_</span>u</span>s</span>e</span>r</span>n</span>a</span>m</span>e</span></span> OIDC_DISPLAY_NAME</span>=</span>D</span>e</span>x</span></span> OIDC_SCOPES</span>=</span>o</span>p</span>e</span>n</span>i</span>d</span> profile</span> email</span></span> FORCE_HTTPS</span>=</span>f</span>a</span>l</span>s</span>e</span></span> ENABLE_UPDATES</span>=</span>t</span>r</span>u</span>e</span></span> WEB_CONCURRENCY</span>=</span>1</span></span> DEBUG</span>=</span>h</span>t</span>t</span>p</span></span> LOG_LEVEL</span>=</span>i</span>n</span>f</span>o</span></span></code></pre> And finally, to configure Dex, we need the following config:</p> i</span>ssuer</span>:</span> h</span>ttp://localhost:5556/dex</span></span> </span> s</span>torage</span>:</span></span> t</span>ype</span>:</span> s</span>qlite3</span></span> c</span>onfig</span>:</span></span> f</span>ile</span>:</span> /</span>var/dex/dex.db</span></span> </span> w</span>eb</span>:</span></span> h</span>ttp</span>:</span> 0</span>.0.0.0:5556</span></span> </span> s</span>taticClients</span>:</span></span> -</span> i</span>d</span>:</span> o</span>utline</span></span> r</span>edirectURIs</span>:</span></span> -</span> "</span>http://localhost:3000/auth/oidc.callback</span>"</span></span> n</span>ame</span>:</span> "</span>Outline</span>"</span></span> s</span>ecret</span>:</span> o</span>utline-secret</span></span> </span> o</span>auth2</span>:</span></span> s</span>kipApprovalScreen</span>:</span> true</span></span> </span> e</span>nablePasswordDB</span>:</span> true</span></span> </span> s</span>taticPasswords</span>:</span></span> -</span> e</span>mail</span>:</span> "</span>admin@example.com</span>"</span></span> h</span>ash</span>:</span> "</span>$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W</span>"</span></span> u</span>sername</span>:</span> "</span>admin</span>"</span></span> u</span>serID</span>:</span> "</span>08a8684b-db88-4b73-90a9-3cd1661f5466</span>"</span></span></code></pre> Voilà! With docker compose up</code>, you’ll have an Outline server ready to go. You can log in using the admin</code> user.</p> </p> </p> Building an expense tracker app 2024-06-14T00:00:00+00:00 A couple of weeks ago, I decided to start logging and tracking my expenses. The goal was not to record every minor purchase but to gain a general insight into where my money was going. In this post, I’ll dive deep into the behind-the-scenes of building Gullak</a>—an expense tracker app with a dash of AI</em> (yes).</p> </iframe> </div> </p> Why#</a></h2> My wife and I have a simple system for tracking our expenses during trips: we use Apple Notes to maintain a day-wise record, jotting down a one-liner for each expense under the date. This straightforward method has proven effective in keeping tabs on our spending habits while traveling.</p> For instance, during our last Europe trip, we recorded our daily expenses. After returning home, I was eager to analyze our spending patterns. I copied all these items into Google Sheets to analyse the top categories that I spent on during the trip.</p> </p> I decided to develop a simple expense tracker app that automatically categorizes expenses into various groups like food, travel, shopping, etc. I believed this was a practical use case for leveraging an LLM paired with Function calling</a> to parse and categorize expenses.</p> Initial Prototype#</a></h2> The first step involved designing a prompt to capture user input about their spending. I picked up go-openai</a> library and experimented with it.</p> Almost a year ago, I had developed a small bot for personal use, which provided a JSON output detailing the macronutrients and calories in specific food items, storing this information in Metabase. However, this was during the early days of API access provided by OpenAI. Due to occasionally unsatisfactory and inconsistent responses (despite instructions like “MUST RETURN JSON OR 1000 CATS WILL DE SOMEWHERE”), it wasn’t entirely reliable.</p> Function calling addresses two main limitations of traditional language model responses:</p> Inconsistent response format</strong>: Without function calling, responses from language models can be unstructured and inconsistent, requiring complex validation and parsing logic on the application side.</li> Lack of external data integration</strong>: Language models are typically limited to the knowledge they were trained on, making it challenging to provide answers based on real-time or external data.</li> </ul> It’s important to note that the LLM does not actually execute</em> any functions. Rather, we create a structure for the LLM to follow in its responses. The LLM would then generate a response with the content as a stringified JSON object following the schema provided in the function definiton.</p> I created a function called categorize_expense</code>. This function takes a list of transactions as parameters, with each transaction having properties like transaction_date</code>, amount</code>, category</code>, and description</code>.</p> Here’s what this looks like:</p> fnCategorizeExpenses</span> :=</span> openai</span>.</span>FunctionDefinition</span>{</span></span> Name</span>:</span> "</span>categorize_expense</span>"</span>,</span></span> Description</span>:</span> "</span>Categorize expenses from the given input.</span>"</span>,</span></span> Parameters</span>:</span> jsonschema</span>.</span>Definition</span>{</span></span> Type</span>:</span> jsonschema</span>.</span>Object</span>,</span></span> Properties</span>:</span> map</span>[</span>string</span>]</span>jsonschema</span>.</span>Definition</span>{</span></span> "</span>transactions</span>"</span>:</span> {</span></span> Type</span>:</span> jsonschema</span>.</span>Array</span>,</span></span> Description</span>:</span> "</span>List of items purchased</span>"</span>,</span></span> Items</span>:</span> &</span>jsonschema</span>.</span>Definition</span>{</span></span> Type</span>:</span> jsonschema</span>.</span>Object</span>,</span></span> Properties</span>:</span> map</span>[</span>string</span>]</span>jsonschema</span>.</span>Definition</span>{</span></span> "</span>transaction_date</span>"</span>:</span> {</span></span> Type</span>:</span> jsonschema</span>.</span>String</span>,</span></span> Description</span>:</span> "</span>Date of transaction in ISO 8601 format (e.g., 2021-09-01) if specified else today's date.</span>"</span>,</span></span> }</span>,</span></span> "</span>amount</span>"</span>:</span> {</span></span> Type</span>:</span> jsonschema</span>.</span>Number</span>,</span></span> Description</span>:</span> "</span>Amount of the item</span>"</span>,</span></span> }</span>,</span></span> "</span>category</span>"</span>:</span> {</span></span> Type</span>:</span> jsonschema</span>.</span>String</span>,</span></span> Description</span>:</span> "</span>One word category of the expense (e.g., food, travel, entertainment)</span>"</span>,</span></span> }</span>,</span></span> "</span>description</span>"</span>:</span> {</span></span> Type</span>:</span> jsonschema</span>.</span>String</span>,</span></span> Description</span>:</span> "</span>Concise and short description of the item</span>"</span>,</span></span> }</span>,</span></span> }</span>,</span></span> Required</span>:</span> [</span>]</span>string</span>{</span>"</span>transaction_date</span>"</span>,</span> "</span>amount</span>"</span>,</span> "</span>category</span>"</span>,</span> "</span>description</span>"</span>}</span>,</span></span> }</span>,</span></span> }</span>,</span></span> }</span>,</span></span> Required</span>:</span> [</span>]</span>string</span>{</span>"</span>transactions</span>"</span>}</span>,</span></span> }</span>,</span></span> }</span></span></code></pre> The response from this API call can then be unmarshalled into a struct.</p> var</span> transactions</span> models</span>.</span>Transactions</span></span> </span> if</span> err</span> :=</span> json</span>.</span>Unmarshal</span>(</span>[</span>]</span>byte</span>(</span>toolCall</span>.</span>Function</span>.</span>Arguments</span>)</span>,</span> &</span>transactions</span>)</span>;</span> err</span> !=</span> nil</span> {</span></span> return</span> err</span></span> }</span></span></code></pre> The next step was to determine exactly how users would provide input. I considered various methods that would make entering expenses as straightforward as my approach with Apple Notes and decided to create a Telegram bot.</p> </p> I developed a Telegram bot that would parse the expenses and save them to a SQLite database. I explored tools like evidence.dev</a>, a nice platform for creating frontends using the database as the sole source of truth. However, I encountered an issue where it could not correctly parse date values (see GitHub issue</a>). Ultimately, I returned to my reliable old friend—Metabase.</p> However, I faced two main challenges with this approach:</p> Privacy Concerns</strong>: Telegram does not offer the option to create a private bot; all bots generated through BotFather are public. To restrict access, I considered adding session tokens, but this approach was unsatisfactory. If I planned to distribute this bot, implementing a token-based, DIY authentication system on Telegram did not seem appropriate.</p> </li> Fixing Bad Entries</strong>: To correct erroneous entries, I had to manually update the SQLite table. As I intended to share this bot with my wife, I needed a more user-friendly workflow. Manually raw dogging UPDATE</code> SQL queries was not the most user-friendly solution.</p> </li> </ul> After a day or two of experimenting, I decided to build a small frontend for now.</p> Building Frontend#</a></h2> </p> As a backend developer, my core expertise is NOT JavaScript, and I strongly dislike the JS ecosystem. Obviously there’s no dearth of choices when it comes to frameworks, however for this project I wanted to stay away from the hype and choose a stack that is simple to use and productive (for me) out of the box. Having used Vue.js in production in the past, I feel it ticks those boxes for me as it comes bundled with a router, store, and all the niceties, and it has excellent</strong> documentation. After reading a refresher on the new Vue3 composition API syntax, I hit the ground running.</p> I find Tailwind CSS ideal for someone like me who prefers not to write CSS or invent class names. It’s a heavily debated topic online, but it’s important to pick our battles. An issue I encountered while researching UI frameworks was that Vue.js seems to have fewer options compared to React, likely due to its lower popularity. After some google-fu, I discovered a promising project called shadcn-vue</a>, an unofficial community led port of the shadcn/ui</a> React library.</p> The cool thing about this library is that it doesn’t come bundled as a package, meaning there’s no way to install it as a dependency</em>. Instead, it gets added directly to your source code, encouraging you to tweak it the way you like.</p> I believe it’s an excellent starting point for anyone looking to build their own design system from scratch, as it allows for customization of both appearance and behavior. It might have been overkill for my simple UI, but I thought, what the heck, if side projects aren’t for exploring new things, what’s the point of it all? 😄</p> Database#</a></h2> For the database, I opted for SQLite. It’s perfect for a small project like this since the database is just a single file, making it easier to manage. Initially, I used the popular driver mattn/go-sqlite3</a>, but I found that the CGO-free alternative modernc/sqlite</a> works just as well.</p> I also experimented with sqlc</a> for the first time. For those unfamiliar, sqlc</code> generates type-safe Go code from your raw SQL queries. It handles all the boilerplate database code needed to retrieve results, scan them into a model, manage transactions, and more. sqlc makes it seem like you’re getting the best of both worlds (ORM + raw SQL).</p> Here’s an example query:</p> --</span> name: CreateTransaction :many</span></span> --</span> Inserts a new transaction into the database.</span></span> INSERT INTO</span> transactions (created_at, transaction_date, amount, currency, category, </span>description</span>, confirm)</span></span> VALUES</span> (?, ?, ?, ?, ?, ?, ?)</span></span> RETURNING </span></span>;</span></span></code></pre> Using sqlc generate</code>, it generates the following code:</p> //</span> Code generated by sqlc. DO NOT EDIT.</span></span> //</span> versions:</span></span> //</span> sqlc v1.26.0</span></span> //</span> source: queries.sql</span></span> </span> package</span> db</span></span> </span> import</span> (</span></span> "</span>context</span>"</span></span> "</span>database/sql</span>"</span></span> "</span>time</span>"</span></span> )</span></span> </span> const</span> createTransaction</span> =</span> `</span>-- name: CreateTransaction :many</span></span> INSERT INTO transactions (created_at, transaction_date, amount, currency, category, description, confirm)</span></span> VALUES (?, ?, ?, ?, ?, ?, ?)</span></span> RETURNING id, created_at, transaction_date, currency, amount, category, description, confirm</span></span> `</span></span> </span> type</span> CreateTransactionParams</span> struct</span> {</span></span> CreatedAt</span> time</span>.</span>Time</span> `</span>json:"created_at"</span>`</span></span> TransactionDate</span> time</span>.</span>Time</span> `</span>json:"transaction_date"</span>`</span></span> Amount</span> float64</span> `</span>json:"amount"</span>`</span></span> Currency</span> string</span> `</span>json:"currency"</span>`</span></span> Category</span> string</span> `</span>json:"category"</span>`</span></span> Description</span> string</span> `</span>json:"description"</span>`</span></span> Confirm</span> bool</span> `</span>json:"confirm"</span>`</span></span> }</span></span> </span> //</span> Inserts a new transaction into the database.</span></span> func</span> (</span>q </span></span>Queries</span>)</span> CreateTransaction</span>(</span>ctx</span> context</span>.</span>Context</span>,</span> arg</span> CreateTransactionParams</span>)</span> (</span>[</span>]</span>Transaction</span>,</span> error</span>)</span> {</span></span> rows</span>,</span> err</span> :=</span> q</span>.</span>query</span>(</span>ctx</span>,</span> q</span>.</span>createTransactionStmt</span>,</span> createTransaction</span>,</span></span> arg</span>.</span>CreatedAt</span>,</span></span> arg</span>.</span>TransactionDate</span>,</span></span> arg</span>.</span>Amount</span>,</span></span> arg</span>.</span>Currency</span>,</span></span> arg</span>.</span>Category</span>,</span></span> arg</span>.</span>Description</span>,</span></span> arg</span>.</span>Confirm</span>,</span></span> )</span></span> if</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> err</span></span> }</span></span> defer</span> rows</span>.</span>Close</span>(</span>)</span></span> items</span> :=</span> [</span>]</span>Transaction</span>{</span>}</span></span> for</span> rows</span>.</span>Next</span>(</span>)</span> {</span></span> var</span> i</span> Transaction</span></span> if</span> err</span> :=</span> rows</span>.</span>Scan</span>(</span></span> &</span>i</span>.</span>ID</span>,</span></span> &</span>i</span>.</span>CreatedAt</span>,</span></span> &</span>i</span>.</span>TransactionDate</span>,</span></span> &</span>i</span>.</span>Currency</span>,</span></span> &</span>i</span>.</span>Amount</span>,</span></span> &</span>i</span>.</span>Category</span>,</span></span> &</span>i</span>.</span>Description</span>,</span></span> &</span>i</span>.</span>Confirm</span>,</span></span> )</span>;</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> err</span></span> }</span></span> items</span> =</span> append</span>(</span>items</span>,</span> i</span>)</span></span> }</span></span> if</span> err</span> :=</span> rows</span>.</span>Close</span>(</span>)</span>;</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> err</span></span> }</span></span> if</span> err</span> :=</span> rows</span>.</span>Err</span>(</span>)</span>;</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> err</span></span> }</span></span> return</span> items</span>,</span> nil</span></span> }</span></span></code></pre>Apple Shortcuts#</a></h2> Similar to my Apple Notes approach, I wanted to create a shortcut that would allow me to log expenses quickly. I created a simple shortcut that would prompt me to enter the expenses and send an HTTP POST request to Gullak’s API server. I then open the dashboard once in a while to confirm/edit these unconfirmed transactions.</p> You can read more about setting up the Shortcut in your Apple devices here</a>.</p> Proudly, Not a Weekend Project#</a></h2> For every “I could do this in a weekend” comment, yes, this project is straightforward—a “CRUD GPT” wrapper that isn’t complicated to build. Yet, it took me over a month to develop. I spent less than an hour most days on this project, instead of cramming it into an all-nighter weekend project - an approach I want to move away from. Slow and steady efforts compound, outlasting quick, sporadic bursts. I’m pleased to balance this with my full-time job without burning out.</p> Ideas for the Future#</a></h2> Initially, I didn’t set out to build a comprehensive budgeting app, just an expense logger, as that was my primary need. However, if usage increases and the tool proves helpful in reducing unnecessary spending, I’m open to adding more features. Some possibilities include a subscription tracker, integration with budgeting tools like YNAB or Actual through their APIs, and monthly reports sent via email. The best part is that you own complete data, as the data is stored locally on your device so you can also export it anytime and build other integrations on top of it.</p> Feel free to open a GitHub issue or reach out</a> if you have any suggestions or feedback. I’m excited to see where this project goes!</p> Update (Feb 2026)#</a></h2> I’ve rewritten Gullak to use plain-text ledger files. Read more about why in the follow-up post: Why Plain-Text Ledger is Powerful for Gullak</a>.</p> A Random Act of Kindness 2024-04-21T00:00:00+00:00 Last month, I did a wonderful trip travelling through the scenic landscapes of Switzerland. My wife and I were in Lucerne and had scheduled a day trip to Mt. Titlis for the next day but were wondering what to do that evening. After strolling along the Chapel Bridge and enjoying an amazing lunch by the waterfront, my wife and I collectively decided to book a Lake Lucerne cruise for the evening. It seemed like the perfect setup for a romantic date night, or so I thought!</p> </p> We got the tickets from the information booth for the evening and went back to our hotel to freshen up and relax for a bit.</p> We arrived exactly at 6:45 PM, as mentioned on our tickets, and started to wait. Except, there were only the two of us waiting. We waited for probably half an hour, until 7:15 PM, which was the departure time. Knowing how precise the Swiss transport system usually is, I sensed something fishy. Luckily, I spotted Martin, a staff member of the cruise company, and asked him about it. He looked puzzled by my question and informed me that there was no cruise scheduled for today. Yep, not today, not tomorrow, and not for the rest of the weekend. It was Easter time, and all the cruise trips were cancelled. In fact, he was quite as puzzled as I was as to how the lady at the ticket counter even gave us the tickets for today. However, he told me that he couldn’t do much and suggested writing an email for a refund. I was a bit sad as this dashed our evening plans, but I thought, fine… Shit happens. It’s not the end of the world!</p> And then, out of nowhere, by pure serendipity, my wife spotted another member of the cruise—this time, in fact, the captain! She began to tell the captain about our ordeal. The captain, a very warm and kind lady, listened to my wife patiently and understood our plight! She apologized on behalf of her company and immediately offered a refund in cash for the tickets that we had purchased. We were quite happy, as following up on emails, etc., was something that I was not particularly excited about looking forward to during the other half of my trip. So we said yes, except she needed to go to the ATM to draw cash. We waited for her and chatted with Martin about random stuff! He told us some really fun stories about how Toblerone’s iconic packaging no longer features the Matterhorn mountain</a>. He also shared practical advice on how to safeguard oneself against pickpockets in Italy(we were gonna visit it soon), and reminisced about his life in Zermatt before relocating to Lucerne.</p> Anyway, the wait stretched longer than usual, and I found myself wondering about her whereabouts. And then finally, we saw her approaching us. She told us that she hadn’t found a working ATM nearby, and had to go a bit far. But she didn’t come back empty-handed; she also brought us macaroons as a token of her apologies, a gesture that was incredibly thoughtful. She handed us 200 CHF in cash, covering more than the 180 CHF we had paid for our tickets. When we attempted to return the excess 20 CHF, she firmly refused to take it back. Despite our insistence on returning the excess 20 CHF, she remained steadfast, refusing to accept it. She encouraged us to use the extra money to enjoy a few drinks, suggesting it as a consolation for our evening plans being spoiled by the canceled cruise.</p> </p> I am glad the cruise didn’t happen. Life has its own ways of revealing that kindness exists in every corner of the world, and serendipity can lead to the most memorable encounters! For most, it may not be a huge thing, but for me, it was a profoundly touching and generous act from a stranger who simply chose to be kind without any ulterior motives.</p> The lesson: Be kind to others, do no harm, and always pay it forward. :)</p> Fin!</p> Travelling with Tailscale 2024-03-27T00:00:00+00:00 I have an upcoming trip to Europe, which I am quite excited about. I wanted to set up a Tailscale exit node to ensure that critical apps I depend on, such as banking portals continue working from outside the country. Tailscale provides a feature called “Exit nodes”. These nodes can be setup to route all traffic (0.0.0.0/0, ::/0) through them.</p> I deployed a tiny DigitalOcean droplet in BLR</code> region and setup Tailscale as an exit node. The steps are quite simple and can be found here</a>.</p> $</span> echo</span> '</span>net.ipv4.ip_forward = 1</span>'</span> \|</span> sudo</span> tee</span> -</span>a</span> /etc/sysctl.d/99-tailscale.conf</span></span> $</span> echo</span> '</span>net.ipv6.conf.all.forwarding = 1</span>'</span> \|</span> sudo</span> tee</span> -</span>a</span> /etc/sysctl.d/99-tailscale.conf</span></span> $</span> sudo</span> sysctl</span> -</span>p</span> /etc/sysctl.d/99-tailscale.conf</span></span> $</span> sudo</span> tailscale</span> up</span> -</span>-advertise-exit-node</span></span></code></pre> The node is now advertised as an exit node, and we can confirm that from the output of tailscale status</code>:</p> $</span> sudo</span> tailscale</span> status</span> </span></span> 100.78.212.33</span> pop-os</span> mr-karan@</span> linux</span> -</span></span> 100.75.180.88</span> homelab</span> mr-karan@</span> linux</span> -</span></span> 100.100.191.57</span> iphone</span> mr-karan@</span> iOS</span> offline</span></span> 100.123.189.14</span> karans-macbook-pro</span> mr-karan@</span> macOS</span> offline</span></span> 100.104.67.7</span> lab</span> mr-karan@</span> linux</span> offline</span></span> 100.108.220.87</span> tailscale-exit</span> mr-karan@</span> linux</span> active</span>;</span> exit</span> node</span>;</span> direct</span> 167.71.236.222:41641,</span> tx</span> 21540</span> rx</span> 17356</span></span></code></pre> On the client side, I was able to start Tailscale and configure it to send all the traffic to the exit node with:</p> sudo</span> tailscale</span> up</span> -</span>-exit-node=100.108.220.87</span></span></code></pre> We can confirm that the traffic is going via the exit node by checking our public IP from this device:</p> ➜</span> curl</span> https://ipinfo.io</span> </span></span> {</span></span> "</span>ip</span>"</span>:</span> "</span>167.x.x.222</span>"</span>,</span></span> "</span>city</span>"</span>:</span> "</span>Doddaballapura</span>"</span>,</span></span> "</span>region</span>"</span>:</span> "</span>Karnataka</span>"</span>,</span></span> "</span>country</span>"</span>:</span> "</span>IN</span>"</span>,</span></span> "</span>loc</span>"</span>:</span> "</span>13.2257,77.5750</span>"</span>,</span></span> "</span>org</span>"</span>:</span> "</span>AS14061 DigitalOcean, LLC</span>"</span>,</span></span> "</span>postal</span>"</span>:</span> "</span>560100</span>"</span>,</span></span> "</span>timezone</span>"</span>:</span> "</span>Asia/Kolkata</span>"</span>,</span></span> "</span>readme</span>"</span>:</span> "</span>https://ipinfo.io/missingauth</span>"</span></span> }</span> </span></span></code></pre> However, I encountered a minor issue since I needed to bring my work laptop for on-call duties, in case any critical production incidents required my attention during my travels. At my organization, we use Netbird as our VPN, which, like Tailscale, creates a P2P overlay network between different devices.</p> The problem was that all 0.0.0.0 traffic was routed to the exit node, meaning the internal traffic meant for Netbird to access internal sites on our private AWS VPC network was no longer routed via the Netbird interface.</p> Netbird automatically propagates a bunch of IP routing rules when connected to the system. These routes are to our internal AWS VPC infrastructure. For example:</p> 10.0.0.0/16</span> via</span> 100.107.12.215</span> dev</span> wt0</span></span></code></pre> Here, wt0</code> is the Netbird interface. So, for example, any IP like 10.0.1.100</code> will go via this interface. To verify this:</p> $</span> ip</span> route</span> get</span> 10.0.1.100</span></span> 10.0.1.100</span> dev</span> wt0</span> src</span> 100.107.12.215</span> uid</span> 1000</span> </span></span></code></pre> However, after connecting to the Tailscale exit node, this was no longer the case. Now, even the private IP meant to be routed via Netbird was being routed through Tailscale:</p> $</span> ip</span> route</span> get</span> 10.0.1.100</span></span> 10.0.1.100</span> dev</span> tailscale0</span> table</span> 52</span> src</span> 100.78.212.33</span> uid</span> 1000</span> </span></span></code></pre> Although Tailscale nodes allow for the selective whitelisting of CIDRs to route only the designated network packets through them, my scenario was different. I needed to selectively bypass certain CIDRs and route all other traffic through the exit nodes. I came across a relevant GitHub issue</a>, but unfortunately, it was closed due to limited demand.</p> This led me to dig deeper into understanding how Tailscale propagates IP routes, to see if there was a way for me to add custom routes with a higher priority.</p> Initially, I examined the IP routes for Tailscale. Typically, one can view the route table list using ip route</code>, which displays the routes in the default</code> and main</code> tables. However, Tailscale uses routing table 52 for its routes, instead of the default or main table.</p> $</span> ip</span> route</span> show</span> table</span> 52</span> </span></span> </span> default</span> dev</span> tailscale0</span> </span></span> 100.75.180.88</span> dev</span> tailscale0</span> </span></span> .</span>.</span>.</span> others</span> ...</span></span> throw</span> 127.0.0.0/8</span> </span></span> 192.168.29.0/24</span> dev</span> tailscale0</span> </span></span></code></pre> A few notes on the route table:</p> default dev tailscale0</code> is the default route for this table. Traffic that doesn’t match any other route in this table will be sent through the tailscale0</code> interface. This ensures that any traffic not destined for a more specific route will go through the Tailscale network.</p> </li> throw 127.0.0.0/8</code>: This is a special route that tells the system to “throw” away traffic destined for 127.0.0.0/8 (local host addresses) if it arrives at this table, effectively discarding it before it reaches the local routing table.</p> </li> </ul> We can see the priority of these IP rules are evaluated using ip rule show</code>:</p> ➜</span> ip</span> rule</span> show</span> </span></span> 0:</span> from</span> all</span> lookup</span> local</span></span> 5210:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> lookup</span> main</span></span> 5230:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> lookup</span> default</span></span> 5250:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> unreachable</span></span> 5270:</span> from</span> all</span> lookup</span> 52</span></span> 32766:</span> from</span> all</span> lookup</span> main</span></span> 32767:</span> from</span> all</span> lookup</span> default</span></span></code></pre> This command lists all the current policy routing rules, including their priority (look for the pref or priority value). Each rule is associated with a priority, with lower numbers having higher priority.</p> By default, Linux uses three main routing tables:</p> Local (priority 0)</li> Main (priority 32766)</li> Default (priority 32767)</li> </ul> Since Netbird already propagates the IP routes in the main routing table, we only need to add a higher priority rule to lookup in the main</code> table before Tailscale takes over.</p> $</span> sudo</span> ip</span> rule</span> add</span> to</span> 10.0.0.0/16</span> pref</span> 5000</span> lookup</span> main</span></span></code></pre> Now, our ip rule</code> looks like:</p> $</span> ip</span> rule</span> show</span> </span></span> 0:</span> from</span> all</span> lookup</span> local</span></span> 5000:</span> from</span> all</span> to</span> 10.0.0.0/16</span> lookup</span> main</span></span> 5210:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> lookup</span> main</span></span> 5230:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> lookup</span> default</span></span> 5250:</span> from</span> all</span> fwmark</span> 0x80000/0xff0000</span> unreachable</span></span> 5270:</span> from</span> all</span> lookup</span> 52</span></span> 32766:</span> from</span> all</span> lookup</span> main</span></span> 32767:</span> from</span> all</span> lookup</span> default</span></span></code></pre> To confirm whether the packets for destination 10.0.0.0/16</code> get routed via wt0</code> instead of tailscale0</code>, we can use the good ol’ ip route get</code>:</p> $</span> ip</span> route</span> get</span> 10.0.1.100</span> </span></span> 10.0.1.100</span> dev</span> wt0</span> src</span> 100.107.12.215</span> uid</span> 1000</span></span></code></pre> Perfect! This setup allows us to route all our public traffic via exit node and only the internal traffic meant for internal AWS VPCs get routed via Netbird VPN.</p> Since, these rules are ephemeral and I wanted to add a bunch of similar network routes, I created a small shell script to automate the process of adding/deleting rules:</p> #!</span>/bin/bash</span></span> </span> #</span> Function to add IP rules for specified CIDRs</span></span> add</span>(</span>)</span> {</span></span> echo</span> "</span>Adding IP rules...</span>"</span></span> sudo</span> ip</span> rule</span> add</span> to</span> 10.0.0.0/16</span> pref</span> 5000</span> lookup</span> main</span></span> #</span> ... others ...</span></span> }</span></span> </span> #</span> Function to remove IP rules based on preference numbers</span></span> remove</span>(</span>)</span> {</span></span> echo</span> "</span>Removing IP rules...</span>"</span></span> sudo</span> ip</span> rule</span> del</span> pref</span> 5000</span></span> #</span> ... others ....</span></span> }</span></span> </span> #</span> Check the first argument to determine which function to call</span></span> case</span> $</span>1</span> in</span></span> a</span>d</span>d</span>)</span></span> add</span></span> ;;</span></span> r</span>e</span>m</span>o</span>v</span>e</span>)</span></span> remove</span></span> ;;</span></span> )</span></span> echo</span> "</span>Invalid argument: </span>$</span>1</span>"</span></span> echo</span> "</span>Usage: </span>$</span>0</span> add\|remove</span>"</span></span> exit</span> 1</span></span> ;;</span></span> esac</span></span></code></pre> Fin!</p> One Billion Row Challenge in Go 2024-01-10T00:00:00+00:00 Earlier this week, I had stumbled upon 1brc</a>, which presents a fun task: loading a huge text file (1 billion lines) in Java as quickly as possible.</p> The One Billion Row Challenge (1BRC) is a fun exploration of how far modern Java can be pushed for aggregating one billion rows from a text file. Utilize all your virtual threads, leverage SIMD, optimize your GC, or employ any other technique to create the fastest implementation for this task!</p> </blockquote> The challenge is mainly about Java, but I thought to do the same in my preferred language: Go. This post is about how I did several iterations to my Go program to reduce the time and discuss the main techniques used in each iteration to make it faster.</p> I was able to create a solution which takes ~20s</strong> to read, parse and calculate stats for 1bn lines on my Apple M2 (10 vCPU, 32GB RAM). There are some insane</a> solutions that people have come up with, be sure to check out GitHub Discussions</a> to go through them!</p> Prerequisites#</a></h2> To generate the text file for these measurements, follow the steps outlined here</a>.</p> After running the commands, I have a measurements.txt</code> on my file system:</p> Example output after running the commands:</p> ➜</span> 1brc-go</span> git:</span>(</span>main</span>)</span> du</span> -</span>sh</span> measurements.txt</span></span> 13G</span> measurements.txt</span></span> ➜</span> 1brc-go</span> git:</span>(</span>main</span>)</span> tail</span> measurements.txt</span></span> Mek</span>'</span>ele;13.3</span></span> Kampala;50.8</span></span> Dikson;-3.7</span></span> Dodoma;20.3</span></span> San Diego;7.1</span></span> Chihuahua;20.3</span></span> Ngaoundéré;24.2</span></span> Toronto;12.7</span></span> Wrocław;12.6</span></span> Singapore;14.4</span></span></code></pre>Ultra minimalistic example of reading a file#</a></h2> Let’s take a look at a basic Go code to read and parse the above file. We’ll also calculate stats on the fly.</p> package</span> main</span></span> </span> import</span> (</span></span> "</span>bufio</span>"</span></span> "</span>fmt</span>"</span></span> "</span>os</span>"</span></span> "</span>sort</span>"</span></span> "</span>strconv</span>"</span></span> "</span>strings</span>"</span></span> )</span></span> </span> type</span> Measurement</span> struct</span> {</span></span> Station</span> string</span></span> Temp</span> float64</span></span> }</span></span> </span> type</span> Stats</span> struct</span> {</span></span> Min</span>,</span> Mean</span>,</span> Max</span> float64</span></span> }</span></span> </span> func</span> main</span>(</span>)</span> {</span></span> //</span> Open the file.</span></span> file</span>,</span> err</span> :=</span> os</span>.</span>Open</span>(</span>"</span>measurements.txt</span>"</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> defer</span> file</span>.</span>Close</span>(</span>)</span></span> </span> //</span> Map to hold the temperatures for each station.</span></span> stationTemps</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>[</span>]</span>float64</span>)</span></span> </span> scanner</span> :=</span> bufio</span>.</span>NewScanner</span>(</span>file</span>)</span></span> for</span> scanner</span>.</span>Scan</span>(</span>)</span> {</span></span> //</span> Parse each line into a Measurement struct.</span></span> parts</span> :=</span> strings</span>.</span>Split</span>(</span>scanner</span>.</span>Text</span>(</span>)</span>,</span> "</span>;</span>"</span>)</span></span> temp</span>,</span> _</span> :=</span> strconv</span>.</span>ParseFloat</span>(</span>parts</span>[</span>1</span>]</span>,</span> 64</span>)</span></span> stationTemps</span>[</span>parts</span>[</span>0</span>]</span>]</span> =</span> append</span>(</span>stationTemps</span>[</span>parts</span>[</span>0</span>]</span>]</span>,</span> temp</span>)</span></span> }</span></span> </span> //</span> Calculate min, mean, and max for each station.</span></span> results</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> for</span> station</span>,</span> temps</span> :=</span> range</span> stationTemps</span> {</span></span> min</span>,</span> max</span>,</span> sum</span> :=</span> temps</span>[</span>0</span>]</span>,</span> temps</span>[</span>0</span>]</span>,</span> 0</span>.</span>0</span></span> for</span> _</span>,</span> t</span> :=</span> range</span> temps</span> {</span></span> if</span> t</span> <</span> min</span> {</span></span> min</span> =</span> t</span></span> }</span></span> if</span> t</span> ></span> max</span> {</span></span> max</span> =</span> t</span></span> }</span></span> sum</span> +=</span> t</span></span> }</span></span> mean</span> :=</span> sum</span> /</span> float64</span>(</span>len</span>(</span>temps</span>)</span>)</span></span> results</span>[</span>station</span>]</span> =</span> Stats</span>{</span>Min</span>:</span> min</span>,</span> Mean</span>:</span> mean</span>,</span> Max</span>:</span> max</span>}</span></span> }</span></span> </span> //</span> Sort the stations and format the output.</span></span> var</span> stations</span> [</span>]</span>string</span></span> for</span> station</span> :=</span> range</span> results</span> {</span></span> stations</span> =</span> append</span>(</span>stations</span>,</span> station</span>)</span></span> }</span></span> sort</span>.</span>Strings</span>(</span>stations</span>)</span></span> </span> fmt</span>.</span>Print</span>(</span>"</span>{</span>"</span>)</span></span> for</span> i</span>,</span> station</span> :=</span> range</span> stations</span> {</span></span> r</span> :=</span> results</span>[</span>station</span>]</span></span> fmt</span>.</span>Printf</span>(</span>"</span>%s</span>=</span>%.1f</span>/</span>%.1f</span>/</span>%.1f</span>"</span>,</span> station</span>,</span> r</span>.</span>Min</span>,</span> r</span>.</span>Mean</span>,</span> r</span>.</span>Max</span>)</span></span> if</span> i</span> <</span> len</span>(</span>stations</span>)</span>-</span>1</span> {</span></span> fmt</span>.</span>Print</span>(</span>"</span>, </span>"</span>)</span></span> }</span></span> }</span></span> fmt</span>.</span>Println</span>(</span>"</span>}</span>"</span>)</span></span> }</span></span></code></pre> On running the above program, we get the following output:</p> {Chihuahua=20.3/20.3/20.3, Dikson=-3.7/-3.7/-3.7, Dodoma=20.3/20.3/20.3, Kampala=50.8/50.8/50.8, Mek'ele=13.3/13.3/13.3, Ngaoundéré=24.2/24.2/24.2, San Diego=7.1/7.1/7.1, Singapore=14.4/14.4/14.4, Toronto=12.7/12.7/12.7, Wrocław=12.6/12.6/12.6}</span></span></code></pre> This approach works well for small, simple files. However, there are certain restrictions:</p> It reads the file line by line using a scanner. Reading and processing a billion rows is time-consuming.</li> Each operation, even if small, adds up when repeated a billion times. This includes string splitting, type conversion, error checking, and appending to a slice.</li> Additionally, we need to consider the potential of hitting the max Disk IOPS limit if we perform too many file operations per second.</li> </ul> Before we proceed to optimize this further, let’s establish a baseline performance of 100 million</em> lines first:</p> $ </span>wc</span> -</span>l</span> measurements</span>.</span>txt</span></span> 100000000</span> measurements</span>.</span>txt</span></span> $ </span>time</span> go</span> run</span> main</span>.</span>go</span></span> go</span> run</span> main</span>.</span>go</span> 18.44s</span> user</span> 0.83s</span> system</span> 100</span>%</span> cpu</span> 19</span>.</span>135</span> total</span></span></code></pre> Baseline: It takes approximately 19s to read and calculate stats from 100 mn lines.</p> There’s a lot of room to optimize it further, let’s go through them one by one.</p> Iteration 1: Producer-Consumer Pattern#</a></h2> The concept involves reading multiple lines simultaneously in the producer Goroutine and then dispatching these batches to worker Goroutines. We can establish a worker pool to implement a producer-consumer pattern. Producers read lines from the file and send them to a channel. Consumers retrieve lines from the channel, parse the data, and calculate the minimum, mean, and maximum temperatures for each station.</p> func</span> main</span>(</span>)</span> {</span></span> numWorkers</span> :=</span> runtime</span>.</span>NumCPU</span>(</span>)</span></span> runtime</span>.</span>GOMAXPROCS</span>(</span>numWorkers</span>)</span></span> </span> linesChan</span> :=</span> make</span>(</span>chan</span> string</span>,</span> 1000000</span>)</span></span> resultsChan</span> :=</span> make</span>(</span>chan</span> map</span>[</span>string</span>]</span>Stats</span>,</span> numWorkers</span>)</span></span> </span> //</span> Start worker goroutines</span></span> var</span> wg</span> sync</span>.</span>WaitGroup</span></span> for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> numWorkers</span>;</span> i</span>++</span> {</span></span> wg</span>.</span>Add</span>(</span>1</span>)</span></span> go</span> worker</span>(</span>linesChan</span>,</span> resultsChan</span>,</span> &</span>wg</span>)</span></span> }</span></span> </span> //</span> Read the file and send lines to the workers</span></span> go</span> func</span>(</span>)</span> {</span></span> file</span>,</span> err</span> :=</span> os</span>.</span>Open</span>(</span>measurementsFile</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> defer</span> file</span>.</span>Close</span>(</span>)</span></span> </span> scanner</span> :=</span> bufio</span>.</span>NewScanner</span>(</span>file</span>)</span></span> for</span> scanner</span>.</span>Scan</span>(</span>)</span> {</span></span> linesChan</span> <-</span> scanner</span>.</span>Text</span>(</span>)</span></span> }</span></span> close</span>(</span>linesChan</span>)</span></span> }</span>(</span>)</span></span> </span> //</span> Collect results from workers</span></span> wg</span>.</span>Wait</span>(</span>)</span></span> close</span>(</span>resultsChan</span>)</span></span> </span> //</span> Aggregate results</span></span> finalResults</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> for</span> workerResult</span> :=</span> range</span> resultsChan</span> {</span></span> for</span> station</span>,</span> stats</span> :=</span> range</span> workerResult</span> {</span></span> finalStats</span> :=</span> finalResults</span>[</span>station</span>]</span></span> finalStats</span>.</span>Min</span> =</span> min</span>(</span>finalStats</span>.</span>Min</span>,</span> stats</span>.</span>Min</span>)</span></span> finalStats</span>.</span>Max</span> =</span> max</span>(</span>finalStats</span>.</span>Max</span>,</span> stats</span>.</span>Max</span>)</span></span> finalStats</span>.</span>Mean</span> =</span> (</span>finalStats</span>.</span>Mean</span></span>float64</span>(</span>finalStats</span>.</span>Count</span>)</span> +</span> stats</span>.</span>Mean</span></span>float64</span>(</span>stats</span>.</span>Count</span>)</span>)</span> /</span> float64</span>(</span>finalStats</span>.</span>Count</span>+</span>stats</span>.</span>Count</span>)</span></span> finalStats</span>.</span>Count</span> +=</span> stats</span>.</span>Count</span></span> finalResults</span>[</span>station</span>]</span> =</span> finalStats</span></span> }</span></span> }</span></span> </span> //</span> Print results</span></span> printStats</span>(</span>finalResults</span>)</span></span> }</span></span> </span> func</span> worker</span>(</span>linesChan</span> <-</span>chan</span> string</span>,</span> resultsChan</span> chan</span><-</span> map</span>[</span>string</span>]</span>Stats</span>,</span> wg</span> </span>sync</span>.</span>WaitGroup</span>)</span> {</span></span> defer</span> wg</span>.</span>Done</span>(</span>)</span></span> </span> stationStats</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> for</span> line</span> :=</span> range</span> linesChan</span> {</span></span> parts</span> :=</span> strings</span>.</span>Split</span>(</span>line</span>,</span> "</span>;</span>"</span>)</span></span> temp</span>,</span> err</span> :=</span> strconv</span>.</span>ParseFloat</span>(</span>parts</span>[</span>1</span>]</span>,</span> 64</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> continue</span></span> }</span></span> </span> stats</span> :=</span> stationStats</span>[</span>parts</span>[</span>0</span>]</span>]</span></span> stats</span>.</span>Count</span>++</span></span> stats</span>.</span>Min</span> =</span> min</span>(</span>stats</span>.</span>Min</span>,</span> temp</span>)</span></span> stats</span>.</span>Max</span> =</span> max</span>(</span>stats</span>.</span>Max</span>,</span> temp</span>)</span></span> stats</span>.</span>Mean</span> +=</span> (</span>temp</span> -</span> stats</span>.</span>Mean</span>)</span> /</span> float64</span>(</span>stats</span>.</span>Count</span>)</span></span> stationStats</span>[</span>parts</span>[</span>0</span>]</span>]</span> =</span> stats</span></span> }</span></span> </span> resultsChan</span> <-</span> stationStats</span></span> }</span></span> </span> func</span> min</span>(</span>a</span>,</span> b</span> float64</span>)</span> float64</span> {</span></span> if</span> a</span> ==</span> 0</span> \|\|</span> a</span> ></span> b</span> {</span></span> return</span> b</span></span> }</span></span> return</span> a</span></span> }</span></span> </span> func</span> max</span>(</span>a</span>,</span> b</span> float64</span>)</span> float64</span> {</span></span> if</span> a</span> <</span> b</span> {</span></span> return</span> b</span></span> }</span></span> return</span> a</span></span> }</span></span> </span> func</span> printStats</span>(</span>statsMap</span> map</span>[</span>string</span>]</span>Stats</span>)</span> {</span></span> var</span> stations</span> [</span>]</span>string</span></span> for</span> station</span> :=</span> range</span> statsMap</span> {</span></span> stations</span> =</span> append</span>(</span>stations</span>,</span> station</span>)</span></span> }</span></span> sort</span>.</span>Strings</span>(</span>stations</span>)</span></span> </span> fmt</span>.</span>Print</span>(</span>"</span>{</span>"</span>)</span></span> for</span> i</span>,</span> station</span> :=</span> range</span> stations</span> {</span></span> stats</span> :=</span> statsMap</span>[</span>station</span>]</span></span> fmt</span>.</span>Printf</span>(</span>"</span>%s</span>=</span>%.1f</span>/</span>%.1f</span>/</span>%.1f</span>"</span>,</span> station</span>,</span> stats</span>.</span>Min</span>,</span> stats</span>.</span>Mean</span>,</span> stats</span>.</span>Max</span>)</span></span> if</span> i</span> <</span> len</span>(</span>stations</span>)</span>-</span>1</span> {</span></span> fmt</span>.</span>Print</span>(</span>"</span>, </span>"</span>)</span></span> }</span></span> }</span></span> fmt</span>.</span>Println</span>(</span>"</span>}</span>"</span>)</span></span> }</span></span></code></pre>Results#</a></h3> The concurrent version, unexpectedly, resulted in almost a 3x decrease in performance.</p> go</span> run</span> main</span>.</span>go</span> 84.15s</span> user</span> 101.34s</span> system</span> 342</span>%</span> cpu</span> 54</span>.</span>225</span> total</span></span></code></pre> Where did we go wrong? This is a classic case where the overhead of concurrency mechanisms outweighs their benefits. In our current implementation, each line is sent to the channel individually, which is likely less efficient than batching lines for processing. This means that for a file with a large number of lines, there will be an equally large number of channel send operations. Each channel operation involves locking and unlocking, which can be costly, especially in a high-frequency context.</p> Iteration 2: Batch processing of lines#</a></h2> In this version we are Batching the lines before sending to the worker which will significantly reduce the overhead of channel communication.</p> Batch Processing</strong>: Each batch contains batchSize</code></strong> lines. This reduces the frequency of channel operations (both sending and receiving), as well as the overhead associated with these operations.</p> </li> Efficient Worker Utilization</strong>: With batch processing, each worker goroutine spends more time processing data and less time interacting with channels. This reduces the overhead of context switching and synchronization, making the processing more efficient.</p> </li> </ol> const</span> (</span></span> batchSize</span> =</span> 1000000</span> //</span> Number of lines per batch</span></span> )</span></span> </span> //</span> ...</span></span> scanner</span> :=</span> bufio</span>.</span>NewScanner</span>(</span>file</span>)</span></span> var</span> batch</span> [</span>]</span>string</span></span> for</span> scanner</span>.</span>Scan</span>(</span>)</span> {</span></span> batch</span> =</span> append</span>(</span>batch</span>,</span> scanner</span>.</span>Text</span>(</span>)</span>)</span></span> if</span> len</span>(</span>batch</span>)</span> >=</span> batchSize</span> {</span></span> batchesChan</span> <-</span> batch</span></span> batch</span> =</span> nil</span> //</span> Start a new batch</span></span> }</span></span> }</span></span> //</span> Send any remaining lines in the last batch</span></span> if</span> len</span>(</span>batch</span>)</span> ></span> 0</span> {</span></span> batchesChan</span> <-</span> batch</span></span> }</span></span> close</span>(</span>batchesChan</span>)</span></span> </span> //</span> ...</span></span> func</span> worker</span>(</span>batchesChan</span> <-</span>chan</span> [</span>]</span>string</span>,</span> resultsChan</span> chan</span><-</span> map</span>[</span>string</span>]</span>Stats</span>,</span> wg</span> </span>sync</span>.</span>WaitGroup</span>)</span> {</span></span> defer</span> wg</span>.</span>Done</span>(</span>)</span></span> </span> stationStats</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> for</span> batch</span> :=</span> range</span> batchesChan</span> {</span></span> for</span> _</span>,</span> line</span> :=</span> range</span> batch</span> {</span></span> //</span> Process the line ...</span></span> }</span></span> }</span></span> </span> resultsChan</span> <-</span> stationStats</span></span> }</span></span> //</span> ...</span></span></code></pre>Results#</a></h3> The improvement from iteration 2 to iteration 3 is quite remarkable, thanks to efficiently batching the lines together and reducing the number of channel ops.</p> go</span> run</span> main.go</span> 30.02s</span> user</span> 0.67s</span> system</span> 476%</span> cpu</span> 6.442</span> total</span></span></code></pre> So far, we’ve reduced the time to about 6.5s which is a great start and improvement of our baseline version of 19s. However, we’re making quite a few extra memory allocations and the focus of next iteration should be to reduce that.</p> Iteration 3: Reducing memory allocations#</a></h2> A batch slice is pre-allocated with a capacity of batchSize</code></strong> and reused for each batch of lines.</li> After sending a batch to the channel, the slice is reset to zero length (batch = batch[:0]</code>), but the underlying array is retained and reused.</li> </ul> //</span> Read the file and send batches of lines to the workers</span></span> go</span> func</span>(</span>)</span> {</span></span> file</span>,</span> err</span> :=</span> os</span>.</span>Open</span>(</span>measurementsFile</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> panic</span>(</span>err</span>)</span></span> }</span></span> defer</span> file</span>.</span>Close</span>(</span>)</span></span> </span> scanner</span> :=</span> bufio</span>.</span>NewScanner</span>(</span>file</span>)</span></span> batch</span> :=</span> make</span>(</span>[</span>]</span>string</span>,</span> 0</span>,</span> batchSize</span>)</span> //</span> Pre-allocate with capacity</span></span> </span> for</span> scanner</span>.</span>Scan</span>(</span>)</span> {</span></span> line</span> :=</span> scanner</span>.</span>Text</span>(</span>)</span></span> </span> //</span> Reuse the batch slice by appending to it until it reaches the batch size</span></span> batch</span> =</span> append</span>(</span>batch</span>,</span> line</span>)</span></span> </span> if</span> len</span>(</span>batch</span>)</span> >=</span> batchSize</span> {</span></span> batchesChan</span> <-</span> batch</span></span> batch</span> =</span> batch</span>[</span>:</span>0</span>]</span> //</span> Reset the slice without allocating new memory</span></span> }</span></span> }</span></span> //</span> Send any remaining lines in the last batch</span></span> if</span> len</span>(</span>batch</span>)</span> ></span> 0</span> {</span></span> batchesChan</span> <-</span> batch</span></span> }</span></span> close</span>(</span>batchesChan</span>)</span></span> }</span>(</span>)</span></span></code></pre>Results#</a></h3> Down to 5.3s!</p> go</span> run</span> main.go</span> 25.43s</span> user</span> 0.53s</span> system</span> 485%</span> cpu</span> 5.346</span> total</span></span></code></pre>Iteration 3 (cont): Further reducing memory allocations#</a></h2> Avoiding strings.Split</code>: Instead of using strings.Split</code>, which allocates a new slice for each line, we can use strings.Index</code> to find the delimiter and manually slice the string. strings.Split</code> typically creates a new slice for each split part, leading to more memory usage and subsequent GC overhead.</li> </ul> for</span> batch</span> :=</span> range</span> batchesChan</span> {</span></span> for</span> _</span>,</span> line</span> :=</span> range</span> batch</span> {</span></span> delimiterIndex</span> :=</span> strings</span>.</span>Index</span>(</span>line</span>,</span> "</span>;</span>"</span>)</span></span> if</span> delimiterIndex</span> ==</span> -</span>1</span> {</span></span> continue</span> //</span> Delimiter not found, skip this line</span></span> }</span></span> </span> station</span> :=</span> line</span>[</span>:</span>delimiterIndex</span>]</span></span> </span> tempStr</span> :=</span> line</span>[</span>delimiterIndex</span>+</span>1</span>:</span>]</span></span> temp</span>,</span> err</span> :=</span> strconv</span>.</span>ParseFloat</span>(</span>tempStr</span>,</span> 64</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> continue</span> //</span> Invalid temperature value, skip this line</span></span> }</span></span> </span> stats</span> :=</span> stationStats</span>[</span>station</span>]</span></span> stats</span>.</span>Count</span>++</span></span> stats</span>.</span>Min</span> =</span> min</span>(</span>stats</span>.</span>Min</span>,</span> temp</span>)</span></span> stats</span>.</span>Max</span> =</span> max</span>(</span>stats</span>.</span>Max</span>,</span> temp</span>)</span></span> stats</span>.</span>Mean</span> +=</span> (</span>temp</span> -</span> stats</span>.</span>Mean</span>)</span> /</span> float64</span>(</span>stats</span>.</span>Count</span>)</span></span> stationStats</span>[</span>station</span>]</span> =</span> stats</span></span> }</span></span> }</span></span></code></pre>Results#</a></h3> The time has further decreased from 5.3s to 4.8s with these changes.</p> go</span> run</span> main</span>.</span>go</span> 15.69s</span> user</span> 0.44s</span> system</span> 332</span>%</span> cpu</span> 4</span>.</span>853</span> total</span></span></code></pre>Iteration 4: Read file in chunks#</a></h2> In this version, the file is read in chunks, and each chunk is processed to ensure it contains complete lines. The processChunk</code> function is used to separate valid data from leftover data in each chunk. Chunk size can be controlled with command line args as well.</p> func</span> main</span>(</span>)</span> {</span></span> //</span> ....</span></span> const</span> chunkSize</span> =</span> 256</span> </span> 1024</span> //</span> 256 KB</span></span> buf</span> :=</span> make</span>(</span>[</span>]</span>byte</span>,</span> chunkSize</span>)</span></span> leftover</span> :=</span> make</span>(</span>[</span>]</span>byte</span>,</span> 0</span>,</span> chunkSize</span>)</span></span> </span> go</span> func</span>(</span>)</span> {</span></span> for</span> {</span></span> bytesRead</span>,</span> err</span> :=</span> file</span>.</span>Read</span>(</span>buf</span>)</span></span> if</span> bytesRead</span> ></span> 0</span> {</span></span> //</span> Copy the chunk to a new slice, because the</span></span> //</span> buffer will be reused in the next iteration.</span></span> chunk</span> :=</span> make</span>(</span>[</span>]</span>byte</span>,</span> bytesRead</span>)</span></span> copy</span>(</span>chunk</span>,</span> buf</span>[</span>:</span>bytesRead</span>]</span>)</span></span> //</span> Process the chunk. The returned leftover will be processed in the next iteration.</span></span> validChunk</span>,</span> newLeftover</span> :=</span> processChunk</span>(</span>chunk</span>,</span> leftover</span>)</span></span> leftover</span> =</span> newLeftover</span></span> //</span> Send the valid chunk to the processing goroutine.</span></span> if</span> len</span>(</span>validChunk</span>)</span> ></span> 0</span> {</span></span> wg</span>.</span>Add</span>(</span>1</span>)</span></span> go</span> processChunkData</span>(</span>validChunk</span>,</span> resultsChan</span>,</span> &</span>wg</span>)</span></span> }</span></span> }</span></span> if</span> err</span> !=</span> nil</span> {</span></span> break</span></span> }</span></span> }</span></span> wg</span>.</span>Wait</span>(</span>)</span></span> close</span>(</span>resultsChan</span>)</span></span> }</span>(</span>)</span></span> //</span> ...</span></span> }</span></span> </span> </span> func</span> processChunk</span>(</span>chunk</span>,</span> leftover</span> [</span>]</span>byte</span>)</span> (</span>validChunk</span>,</span> newLeftover</span> [</span>]</span>byte</span>)</span> {</span></span> firstNewline</span> :=</span> -</span>1</span></span> lastNewline</span> :=</span> -</span>1</span></span> //</span> Find the first and last newline in the chunk.</span></span> for</span> i</span>,</span> b</span> :=</span> range</span> chunk</span> {</span></span> if</span> b</span> ==</span> '</span>\n</span>'</span> {</span></span> if</span> firstNewline</span> ==</span> -</span>1</span> {</span></span> firstNewline</span> =</span> i</span></span> }</span></span> lastNewline</span> =</span> i</span></span> }</span></span> }</span></span> if</span> firstNewline</span> !=</span> -</span>1</span> {</span></span> validChunk</span> =</span> append</span>(</span>leftover</span>,</span> chunk</span>[</span>:</span>lastNewline</span>+</span>1</span>]</span>...</span>)</span></span> newLeftover</span> =</span> make</span>(</span>[</span>]</span>byte</span>,</span> len</span>(</span>chunk</span>[</span>lastNewline</span>+</span>1</span>:</span>]</span>)</span>)</span></span> copy</span>(</span>newLeftover</span>,</span> chunk</span>[</span>lastNewline</span>+</span>1</span>:</span>]</span>)</span></span> }</span> else</span> {</span></span> newLeftover</span> =</span> append</span>(</span>leftover</span>,</span> chunk</span>...</span>)</span></span> }</span></span> return</span> validChunk</span>,</span> newLeftover</span></span> }</span></span> </span> func</span> processChunkData</span>(</span>chunk</span> [</span>]</span>byte</span>,</span> resultsChan</span> chan</span><-</span> map</span>[</span>string</span>]</span>Stats</span>,</span> wg</span> </span>sync</span>.</span>WaitGroup</span>)</span> {</span></span> defer</span> wg</span>.</span>Done</span>(</span>)</span></span> </span> stationStats</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> scanner</span> :=</span> bufio</span>.</span>NewScanner</span>(</span>strings</span>.</span>NewReader</span>(</span>string</span>(</span>chunk</span>)</span>)</span>)</span></span> </span> for</span> scanner</span>.</span>Scan</span>(</span>)</span> {</span></span> line</span> :=</span> scanner</span>.</span>Text</span>(</span>)</span></span> </span> //</span> Find the index of the delimiter</span></span> delimiterIndex</span> :=</span> strings</span>.</span>Index</span>(</span>line</span>,</span> "</span>;</span>"</span>)</span></span> if</span> delimiterIndex</span> ==</span> -</span>1</span> {</span></span> continue</span> //</span> Delimiter not found, skip this line</span></span> }</span></span> </span> //</span> Extract the station name and temperature string</span></span> station</span> :=</span> line</span>[</span>:</span>delimiterIndex</span>]</span></span> tempStr</span> :=</span> line</span>[</span>delimiterIndex</span>+</span>1</span>:</span>]</span></span> </span> //</span> Convert the temperature string to a float</span></span> temp</span>,</span> err</span> :=</span> strconv</span>.</span>ParseFloat</span>(</span>tempStr</span>,</span> 64</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> continue</span> //</span> Invalid temperature value, skip this line</span></span> }</span></span> </span> //</span> Update the statistics for the station</span></span> stats</span>,</span> exists</span> :=</span> stationStats</span>[</span>station</span>]</span></span> if</span> !</span>exists</span> {</span></span> stats</span> =</span> Stats</span>{</span>Min</span>:</span> temp</span>,</span> Max</span>:</span> temp</span>}</span></span> }</span></span> stats</span>.</span>Count</span>++</span></span> stats</span>.</span>Min</span> =</span> min</span>(</span>stats</span>.</span>Min</span>,</span> temp</span>)</span></span> stats</span>.</span>Max</span> =</span> max</span>(</span>stats</span>.</span>Max</span>,</span> temp</span>)</span></span> stats</span>.</span>Mean</span> +=</span> (</span>temp</span> -</span> stats</span>.</span>Mean</span>)</span> /</span> float64</span>(</span>stats</span>.</span>Count</span>)</span></span> stationStats</span>[</span>station</span>]</span> =</span> stats</span></span> }</span></span> </span> //</span> Send the computed stats to resultsChan</span></span> resultsChan</span> <-</span> stationStats</span></span> }</span></span></code></pre> In addition to this, I moved the aggregateStats</code> to a separate Goroutine as well:</p> aggWg</span>.</span>Add</span>(</span>1</span>)</span></span> finalResults</span> :=</span> make</span>(</span>map</span>[</span>string</span>]</span>Stats</span>)</span></span> </span> //</span> Start a separate goroutine for aggregation</span></span> go</span> func</span>(</span>)</span> {</span></span> defer</span> aggWg</span>.</span>Done</span>(</span>)</span></span> for</span> workerResult</span> :=</span> range</span> resultsChan</span> {</span></span> for</span> station</span>,</span> stats</span> :=</span> range</span> workerResult</span> {</span></span> finalStats</span>,</span> exists</span> :=</span> finalResults</span>[</span>station</span>]</span></span> if</span> !</span>exists</span> {</span></span> finalResults</span>[</span>station</span>]</span> =</span> stats</span></span> continue</span></span> }</span></span> finalStats</span>.</span>Min</span> =</span> min</span>(</span>finalStats</span>.</span>Min</span>,</span> stats</span>.</span>Min</span>)</span></span> finalStats</span>.</span>Max</span> =</span> max</span>(</span>finalStats</span>.</span>Max</span>,</span> stats</span>.</span>Max</span>)</span></span> totalCount</span> :=</span> finalStats</span>.</span>Count</span> +</span> stats</span>.</span>Count</span></span> finalStats</span>.</span>Mean</span> =</span> (</span>finalStats</span>.</span>Mean</span></span>float64</span>(</span>finalStats</span>.</span>Count</span>)</span> +</span> stats</span>.</span>Mean</span></span>float64</span>(</span>stats</span>.</span>Count</span>)</span>)</span> /</span> float64</span>(</span>totalCount</span>)</span></span> finalStats</span>.</span>Count</span> =</span> totalCount</span></span> finalResults</span>[</span>station</span>]</span> =</span> finalStats</span></span> }</span></span> }</span></span> }</span>(</span>)</span></span></code></pre>Results#</a></h3> We’re down from 4.8s to just 2.1s to read/parse/process 100mn lines!</p> ./bin/1brc.bin</span> -</span>-file=input.txt</span> -</span>-chunksize=1048576</span> 17.58s</span> user</span> 0.77s</span> system</span> 837%</span> cpu</span> 2.190</span> total</span></span></code></pre>Summary#</a></h2> Basic File Reading and Parsing (Baseline)</strong>:</p> Time</strong>: 19s (baseline).</li> Key Change</strong>: Sequentially reading and processing each line.</li> Speedup</strong>: N/A (baseline).</li> </ul> </li> Producer-Consumer Pattern</strong>:</p> Time</strong>: 54.225s.</li> Key Change</strong>: Implemented concurrent line processing with producer-consumer pattern.</li> Speedup</strong>: -185% (slower than baseline).</li> </ul> </li> Batch Processing of Lines</strong>:</p> Time</strong>: 6.442s.</li> Key Change</strong>: Batched lines before processing, reducing channel communication.</li> Speedup</strong>: +66% (compared to baseline).</li> </ul> </li> Reducing Memory Allocations - Iteration 1</strong>:</p> Time</strong>: 5.346s.</li> Key Change</strong>: Reused batch slices and reduced memory allocations.</li> Speedup</strong>: +72% (compared to baseline).</li> </ul> </li> Reducing Memory Allocations - Iteration 2 (Avoiding strings.Split</code>)</strong>:</p> Time</strong>: 4.853s.</li> Key Change</strong>: Replaced strings.Split</code> with manual slicing for efficiency.</li> Speedup</strong>: +75% (compared to baseline).</li> </ul> </li> Read File in Chunks</strong>:</p> Time</strong>: 2.190s.</li> Key Change</strong>: Processed file in chunks and optimized aggregation.</li> Speedup</strong>: +87% (compared to baseline).</li> </ul> </li> </ol> Final Run#</a></h2> I’m quite satisfied with the final version for now. We can now proceed to test it with 1 billion lines. However it’s evidently CPU-bound, as we spawn N workers for N CPUs.</p> I experimented with different chunk sizes, and here are the results from each run:</p> Chunk Size</th> Time</th></tr></thead> 512.00 KB</td> 23.756s</td></tr> 1.00 MB</td> 21.798s</td></tr> 16.00 MB</td> 20.693s</td></tr> 32.00 MB</td> 19.501s</td></tr> </tbody></table> Tweaking the chunk size doesn’t significantly impact performance, as processing larger chunks takes longer.</p> TL;DR: On an average and with multiple runs it takes approx 20s</strong> with the final iteration for 1bn lines.</p> Checkout the full code on my GitHub</a>.</p> Potential Improvements#</a></h2> This project was not only fun but also a great opportunity to revisit and refine many Go concepts. There are several ideas to contemplate for further improving this version’s timings:</p> I haven’t yet considered using mmap</code>, but I believe it could substantially speed things up.</li> To delve even deeper, custom line parsing functions, especially for converting string</code> to float64</code>, could offer improvements.</li> Employing custom hashing functions (perhaps FnV) might aid in faster map lookups.</li> </ul> Fin!</p> Making sad servers happy 2023-11-06T00:00:00+00:00 Introduction to SadServers#</a></h2> Recently, I stumbled upon sadservers</a>, a platform described as “Like LeetCode for Linux”. The premise is: you are given access to a full remote Linux server with a pre-configured problem. Your mission is to diagnose and fix the issues in a fixed time window.</p> With the goal of documenting my journey through these challenges and sharing the knowledge gained, I decided to not only tackle these puzzles but also to record my solutions in a video format. The format is twofold in its purpose: it allows me to reflect on my problem-solving approach and provides a resource for others who may encounter similar problems, whether in real-world scenarios or in preparation for an SRE/DevOps interview.</p> The Learning Curve#</a></h2> Each server presented a different issue, from misconfigured network settings to services failing to start, from permission issues to resource overutilization. One server, for instance, had a failing database service because of a disk full partition. The cause? Stale backup files. Another had a web server throwing errors because of incorrect file permissions.</p> Recording the Solutions#</a></h2> The video recordings start with an introduction to the problem and my initial thoughts. Viewers can see my screen as I work through the issue, making the troubleshooting process transparent and educational. The commentary explains my thought process, the tools/CLI utilities used, and the solutions applied.</p> Part 1#</a></h3> </iframe> Part 2#</a></h3> </iframe> Part 3#</a></h3> </iframe> Conclusion#</a></h2> For those looking to enhance their Linux troubleshooting skills, sadservers.com is a gold mine. It’s an excellent preparation ground for anyone aiming to step into the SRE/DevOps field or wanting to keep their skills sharp.</p> As I continue to record and share these troubleshooting escapades, I invite you to subscribe, comment with your insights, or even suggest what types of challenges you’d like to see addressed next.</p> Nomad can do everything that K8s can 2023-07-23T05:34:47+00:00 This blog post is ignited by the following Twitter exchange</a>:</p> </p> I don’t take the accusation of unsubstantiated</em> argument, especially on a technical topic lightly. I firmly believe in substantiated arguments and hence, here I am, elaborating on my stance. If found mistaken, I am open to corrections and revise my stance.</p> Some Historical Context#</a></h3> In my professional capacity, I have run and managed several K8s clusters (using AWS EKS) for our entire team of devs (been there done that</em>). The most complex piece of our otherwise simple and clean stack was K8s and we’d been longing to find a better replacement. None of us knew whether that would be Nomad or anything else. But we took the chance and we have reached a stage where we can objectively argue that, for our specific workloads, Nomad has proven to be a superior tool compared to K8s.</p> Building with Nomad#</a></h2> Nomad presents a fundamental building block approach to designing your own services. It used to be true that Nomad was primarily a scheduler, and for serious production workloads, you had to rely on Consul for service discovery and Vault for secret management. However, this scenario has changed as Nomad now seamlessly integrates these features, making them first-class citizens in its environment. Our team replaced our HashiCorp stack with just Nomad, and we never felt constrained in terms of what we could accomplish with Consul/Vault. While these tools still hold relevance for larger clusters managed by numerous teams, they are not necessary for our use case.</p> Deconstructing Infrastructure#</a></h2> Kubernetes employs a declarative state for every operation in the cluster, essentially operating as a reconciliation mechanism to keep everything in check. In contrast, Nomad requires dealing with fewer components, making it appear lacking compared to K8s’s concept of everything being a “resource.” However, that is far from the truth.</p> Ingress</strong>: We run a set of HAProxy on a few nodes which act as “L7 LBs”. Configured with Nomad services, they can do the routing based on Host headers.</li> DNS</strong>: To provide external access to a service without using a proxy, we developed a tool</a> that scans all services registered in the cluster and creates a corresponding DNS record on AWS Route53.</li> Monitoring</strong>: Ah my fav. You wanna monitor your K8s cluster. Sure, here’s kube-prometheus</a>, prometheus-operator</a>, kube-state-metrics</a>. Choices, choices. Enough to confuse you for days. Anyone who’s ever deployed any of these, tell me why this thing needs such a monstrosity setup of CRDs and operators. Monitoring Nomad is such a breeze, 3 lines of HCL</a> config and done.</li> Statefulsets</strong>: It’s 2023 and the irony is rich - the recommended way to run a database inside K8s is… not to run it inside K8s at all. In Nomad, we run a bunch of EC2 instances and tag them as db</code> nodes. The DBs don’t float around as containers to random nodes. And there’s no CSI plugin reaching for a storage disk in AZ-1 when the node is basking in AZ-2. Running a DB on Nomad feels refreshingly like running it on an unadorned EC2 instance.</li> Autoscale</strong>: All our client nodes (except for the db</code> nodes) are ephemeral and part of AWS’s Auto Scaling Groups (ASGs). We use ASG rules for the horizontal scaling of the cluster. While Nomad does have its own autoscale, our preference is to run large instances dedicated to specific workloads, avoiding a mix of different workloads on the same machine.</li> </ul> Over abstraction of Kubernetes#</a></h2> One of my primary critiques of K8s is its hidden complexities. While these abstractions might simplify things on the surface, debugging becomes a nightmare when issues arise. Even after three years of managing K8s clusters, I’ve never felt confident dealing with databases or handling complex networking problems involving dropped packets.</p> You might argue that it’s about technical chops, which I won’t disagree with - but then do you want to add value to the business by getting shit done or do you want to be the resident K8s whiz at your organization?</p> Consider this: How many people do you know who run their own K8s clusters? Even the K8s experts themselves preach about running prod clusters on EKS/GKE etc. How many fully leverage all that K8s has to offer? How many are even aware of all the network routing intricacies managed by kube-proxy? If these queries stir up clouds of uncertainty, it’s possible you’re sipping the Kubernetes Kool-Aid without truly comprehending the recipe, much like I found myself doing at one point</p> Nomad: Not Perfect, But Simpler#</a></h2> Now, if you’re under the impression that I’m singing unabashed praises for Nomad, let me clarify - Nomad has its share of challenges. I’ve personally encountered and reported several. However, the crucial difference lies in Nomad’s lesser degree of abstraction, allowing for a comprehensive understanding of its internals. For instance, we encountered service reconciliation issues</a> with a particular Nomad version. However, we could query the APIs, identify the problem, and write a bash script to resolve and reconcile it. It wouldn’t have been possible when there are too many moving parts in the system and we don’t know where to even begin debugging.</p> The YAML hell</a> is all too well known to all of us. In K8s, writing job manifests required a lot of effort (by the developers who don’t work with K8s all day) and were very complex to understand. It felt “too verbose” and involved copy pasting large blocks from the docs and trying to make things work. Compare that to HCL, it feels much nicer to read and shorter. Things are more straightforward to understand.</p> I’ve not even touched upon the nice-ities on Nomad yet. Like better humanly understandable ACLs? Cleaner and simpler job spec, which defines the entire job in one file? A UI which actually shows everything about your cluster, nodes, and jobs? Not restricting your workloads to be run as Docker containers? A single binary which powers all of this?</p> The central question this post aims to raise is: What can K8s do that Nomads can’t, especially considering the features people truly need? My perspectives are informed not only by my organization but also through interactions with several other organizations at various meetups and conferences. Yet, I have rarely encountered a use case that could only be managed by K8s. While Nomad isn’t a panacea for all issues, it’s certainly worth a try. Reducing the complexity of your tech stack can prove beneficial for your applications and, most importantly, your developers.</p> At this point, K8s enjoys immense industry-wide support, while Nomad remains the unassuming newcomer. This contrast is not a negative aspect, per se. Large organizations often gravitate towards complexity and the opportunity to engage more engineers. However, if simplicity were the primary goal, the prevailing sense of overwhelming complexity in the infrastructure and operations domain wouldn’t be as pervasive.</p> Conclusion#</a></h2> I hope my arguments provide a more comprehensive perspective and address the earlier critique of being unsubstantiated.</p> Update#</a></h2> INFO</strong> Darren has responded to this blog post. You can read the response on Twitter</a>.</p> </p> </blockquote> Fin!</p> Storing AWS Pinpoint Logs 2023-07-21T05:24:06+00:00 At $dayjob, we use AWS Pinpoint to send out SMS to our customers. We’ve also written a detailed blog post</a> on how we use Clickhouse</a> + vector</a> stack for our logging needs. We additionally wanted to store the delivery logs generated by the Pinpoint service. But like with anything else in AWS, even simpler tasks like these usually tend to piggyback on other counterparts of AWS - in this case, it happens to be AWS Kinesis. All the delivery logs which contain metadata about SMS delivery are streamed to Kinesis.</p> Our setup involves configuring Pinpoint with Amazon Kinesis Data Firehose stream. Firehose is an ETL service that helps stream events to other persistent stores. Firehose supports multiple such output sinks and in our case we use HTTP</code> sink.</p> This is what the flow looks like:</p> Pinpoint -> Kinesis Firehose -> Vector HTTP -> Clickhouse</code></p> Ingesting Data#</a></h2> On the HTTP server side, we used vector</code>’s aws_kinesis_firehose</a> source. Compared to just using http</a> source, here are the differences I found:</p> Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value X-Amz-Firehose-Access-Key</code> header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the http</code> source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether X-Amz-Firehose-Access-Key</code> is present in the headers and do a value comparison with our key.</p> </li> Has native support for base64</code> decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the http</code> source. So, basically, this is how the server receives the payload:</p> {</span></span> "</span>requestId</span>"</span>:</span> "</span>6a14a06b-6eae-4218-...</span>"</span>,</span></span> "</span>timestamp</span>"</span>:</span> 1689766125971</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> {</span></span> "</span>data</span>"</span>:</span> "</span>eyJld...</span>"</span></span> }</span>,</span></span> {</span></span> "</span>data</span>"</span>:</span> "</span>eyJldmVudF9...</span>"</span></span> }</span></span> ]</span></span> }</span></span></code></pre> The value of the payload is a base64 encoded value of the JSON Object</a> of an SMS event. However, the aws_kinesis_firehose</code> source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using aws_kinesis_firehose</code> source:</p> {</span></span> "</span>message</span>"</span>:</span> "</span>{</span>\"</span>event_type</span>\"</span>:</span>\"</span>_SMS.SUCCESS</span>\"</span>,</span>\"</span>event_timestamp</span>\"</span>:1689827914426,</span>\"</span>arrival_timestamp</span>\"</span>:1689827917659,</span>\"</span>event_version</span>\"</span>:</span>\"</span>3.1</span>\"</span>,</span>\"</span>application</span>\"</span>:{</span>\"</span>app_id</span>\"</span>:</span>\"</span>redacted</span>\"</span>,</span>\"</span>sdk</span>\"</span>:{}},</span>\"</span>client</span>\"</span>:{</span>\"</span>client_id</span>\"</span>:</span>\"</span>redacted</span>\"</span>},</span>\"</span>device</span>\"</span>:{</span>\"</span>platform</span>\"</span>:{}},</span>\"</span>session</span>\"</span>:{},</span>\"</span>attributes</span>\"</span>:{</span>\"</span>sender_request_id</span>\"</span>:</span>\"</span>redacted</span>\"</span>,</span>\"</span>destination_phone_number</span>\"</span>:</span>\"</span>+91xxx</span>\"</span>,</span>\"</span>record_status</span>\"</span>:</span>\"</span>DELIVERED</span>\"</span>,</span>\"</span>iso_country_code</span>\"</span>:</span>\"</span>IN</span>\"</span>,</span>\"</span>mcc_mnc</span>\"</span>:</span>\"</span>xxx</span>\"</span>,</span>\"</span>number_of_message_parts</span>\"</span>:</span>\"</span>1</span>\"</span>,</span>\"</span>message_id</span>\"</span>:</span>\"</span>redacted</span>\"</span>,</span>\"</span>message_type</span>\"</span>:</span>\"</span>Transactional</span>\"</span>,</span>\"</span>origination_phone_number</span>\"</span>:</span>\"</span>redactedORG</span>\"</span>},</span>\"</span>metrics</span>\"</span>:{</span>\"</span>price_in_millicents_usd</span>\"</span>:xx.0},</span>\"</span>awsAccountId</span>\"</span>:</span>\"</span>redacted</span>\"</span>}</span>\n</span>"</span>,</span></span> "</span>request_id</span>"</span>:</span> "</span>6dd45388-xxx</span>"</span>,</span></span> "</span>source_arn</span>"</span>:</span> "</span>arn:aws:firehose:ap-south-1:redacted:deliverystream/redacted</span>"</span>,</span></span> "</span>source_type</span>"</span>:</span> "</span>aws_kinesis_firehose</span>"</span>,</span></span> "</span>timestamp</span>"</span>:</span> "</span>2023-07-20T04:39:38.772Z</span>"</span></span> }</span></span></code></pre> This makes it straightforward because now we just have to parse the JSON inside the message</code> key and do transformations on that object. If it was http</code> source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least.</p> </li> </ul> Here’s the vector config so far:</p> [</span>sources</span>.</span>firehose</span>]</span></span> #</span> General</span></span> type</span> =</span> "</span>aws_kinesis_firehose</span>"</span></span> address</span> =</span> "</span>127.0.0.1:9000</span>"</span></span> store_access_key</span> =</span> false</span></span> access_keys</span> =</span> [</span>"</span>superdupersecret</span>"</span>]</span></span> </span> #</span> Use it for debugging</span></span> [</span>sinks</span>.</span>console</span>]</span></span> type</span> =</span> "</span>firehose</span>"</span></span> inputs</span> =</span> [</span>"</span>format_pinpoint_logs</span>"</span>]</span></span> encoding</span>.</span>codec</span> =</span> "</span>json</span>"</span></span> </span></code></pre>Formatting the data#</a></h2> Now that we have a pipeline which sends and receives data, we can process the events and transform them into a schema that is more desirable. Since we require the events to be queryable in a Clickhouse DB, this is the schema we have:</p> CREATE</span> TABLE</span> default</span>.pinpoint_logs (</span></span> `</span>_timestamp</span>`</span> DateTime</span>(</span>'</span>Asia/Kolkata</span>'</span>),</span></span> `</span>app_id</span>`</span> LowCardinality(String),</span></span> `</span>event_type</span>`</span> LowCardinality(String),</span></span> `</span>record_status</span>`</span> LowCardinality(String),</span></span> `</span>origination_phone_number</span>`</span> String,</span></span> `</span>message_id</span>`</span> String,</span></span> `</span>destination_phone_number</span>`</span> String,</span></span> `</span>arrival_timestamp</span>`</span> DateTime</span>(</span>'</span>Asia/Kolkata</span>'</span>),</span></span> `</span>event_timestamp</span>`</span> DateTime</span>(</span>'</span>Asia/Kolkata</span>'</span>),</span></span> `</span>meta</span>`</span> Nullable(String)</span></span> )</span></span> ENGINE </span>=</span> MergeTree</span></span> PARTITION</span> BY</span> toYYYYMM(_timestamp)</span></span> ORDER BY</span> _timestamp</span></span> SETTINGS index_granularity </span>=</span> 8192</span>;</span></span></code></pre> To achieve the above format, we can use VRL to parse and format our SMS events:</p> [</span>transforms</span>.</span>format_pinpoint_logs</span>]</span></span> type</span> =</span> "</span>remap</span>"</span> </span></span> inputs</span> =</span> [</span>"</span>firehose</span>"</span>]</span> </span></span> source</span> =</span> '''</span></span> # Decode the JSON message and set ingestion timestamp</span></span> .message = parse_json!(.message)</span></span> .ingestion_timestamp = .timestamp</span></span> </span> # Convert timestamps from Unix to DateTime</span></span> .event_timestamp = from_unix_timestamp!(.message.event_timestamp, unit:"milliseconds")</span></span> .arrival_timestamp = from_unix_timestamp!(.message.arrival_timestamp, unit:"milliseconds")</span></span> </span> # Extract keys to top level and remove from attributes</span></span> .record_status = del(.message.attributes.record_status)</span></span> .origination_phone_number = del(.message.attributes.origination_phone_number)</span></span> .destination_phone_number = del(.message.attributes.destination_phone_number)</span></span> .message_id = del(.message.attributes.message_id)</span></span> </span> # Encode the remaining attributes as JSON string</span></span> .attr = encode_json(.message.attributes)</span></span> </span> # Format Payload for Clickhouse</span></span> . = {</span></span> "_timestamp": .ingestion_timestamp,</span></span> "arrival_timestamp": .arrival_timestamp,</span></span> "event_timestamp": .event_timestamp,</span></span> "app_id": .message.application.app_id,</span></span> "event_type": .message.event_type,</span></span> "record_status": .record_status,</span></span> "message_id": .message_id,</span></span> "origination_phone_number": .origination_phone_number,</span></span> "destination_phone_number": .destination_phone_number,</span></span> "meta": .attr</span></span> }</span></span> '''</span></span> </span></code></pre> Plugging this, we have a clean JSON object for each SMS event. The only thing now we need to add is an output sink to Clickhouse:</p> [</span>sinks</span>.</span>clickhouse</span>]</span></span> type</span> =</span> "</span>clickhouse</span>"</span></span> inputs</span> =</span> [</span>"</span>format_pinpoint_logs</span>"</span>]</span></span> skip_unknown_fields</span> =</span> true</span></span> compression</span> =</span> "</span>gzip</span>"</span></span> database</span> =</span> "</span>default</span>"</span></span> endpoint</span> =</span> "</span>http://127.0.0.1:8123</span>"</span></span> table</span> =</span> "</span>pinpoint_logs</span>"</span></span> encoding</span>.</span>timestamp_format</span> =</span> "</span>unix</span>"</span></span> batch</span>.</span>max_bytes</span> =</span> 1049000</span> #</span> 1 MB</span></span> batch</span>.</span>timeout_secs</span> =</span> 5</span></span> buffer</span>.</span>max_size</span> =</span> 268435488</span></span> buffer</span>.</span>type</span> =</span> "</span>disk</span>"</span></span> buffer</span>.</span>when_full</span> =</span> "</span>block</span>"</span></span></code></pre> Perfect! On running this pipeline with vector -c config.toml</code> we can see the consumption the records</p> Hope this short post was useful if you’ve to do anything similar!</p> Fin!</p> Bridge Networking in Nomad 2023-07-17T05:45:04+00:00 To set the stage, it’s crucial to understand what we mean by “bridge networking”. In a nutshell, it is a type of network connection in Linux that allows virtual interfaces, like the ones used by virtual machines and containers, to share a physical network interface.</p> With Nomad, when a task is allocated, it creates a network namespace with its own network stack. Within this, a virtual ethernet (veth) pair is established, one end of which is assigned to the network namespace of the allocation, and the other remains in the host namespace.</p> </p> The Network Journey</strong>#</a></h2> To illustrate this practically, let’s assume a packet is sent from a task within an allocation. The packet would first be received by the local end of the veth pair, it would then traverse to the other end residing in the host’s namespace. From there, it is sent to the bridge on the host (in this case, the “nomad” bridge), which finally sends the packet out to the world via the host’s physical network interface (typically “eth0” or equivalent in your machine).</p> The journey of a packet from the outside world to a task inside an allocation is the exact mirror image. The packet reaches “eth0” first, then the nomad bridge, it is then forwarded to the appropriate veth interface in the host’s namespace. From there, it crosses over to the other end of the veth pair in the allocation’s network namespace and finally gets routed to the destination task.</p> To bridge or not to#</a></h2> Let’s take a look at the following jobspec which is for deploying my tiny side project - Cloak</a> on Nomad</p> job</span> "cloak"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> </span> group</span> "redis"</span> {</span></span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> }</span></span> }</span></span> </span> service</span> {</span></span> name</span> =</span> "</span>cloak-redis</span>"</span></span> port</span> =</span> "</span>redis</span>"</span></span> provider</span> =</span> "</span>nomad</span>"</span></span> }</span></span> </span> </span> task</span> "redis"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> </span> config</span> {</span></span> image</span> =</span> "</span>redis:7</span>"</span></span> advertise_ipv6_address</span> =</span> false</span></span> </span> ports</span> =</span> [</span></span> "</span>redis</span>"</span>,</span></span> ]</span></span> </span> volumes</span> =</span> [</span></span> "</span>/data/cloak/redis:/data</span>"</span>,</span></span> ]</span></span> }</span></span> </span> resources</span> {</span></span> cpu</span> =</span> 500</span> #</span> MHz</span></span> memory</span> =</span> 256</span> #</span> MB</span></span> }</span></span> }</span></span> }</span></span> </span> group</span> "cloak"</span> {</span></span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "cloak"</span> {</span></span> static</span> =</span> 7000</span></span> to</span> =</span> 7000</span></span> }</span></span> }</span></span> </span> task</span> "cloak"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> config</span> {</span></span> image</span> =</span> "</span>ghcr.io/mr-karan/cloak:v0.2.0</span>"</span></span> command</span> =</span> "</span>--config=config.toml</span>"</span></span> ports</span> =</span> [</span></span> "</span>cloak</span>"</span>,</span></span> ]</span></span> }</span></span> </span> template</span> {</span></span> data</span> =</span> <<</span>EOH</span></span> # Configuration for 1 redis instances, as assigned via rendezvous hashing.</span></span> {{$allocID := env "NOMAD_ALLOC_ID" -}}</span></span> {{range nomadService 1 $allocID "cloak-redis"}}</span></span> CLOAK_REDIS__address={{ .Address }}:{{ .Port }}</span></span> {{- end}}</span></span> EOH</span></span> destination</span> =</span> "</span>secrets/file.env</span>"</span></span> env</span> =</span> true</span></span> }</span></span> </span> </span> resources</span> {</span></span> cpu</span> =</span> 500</span> #</span> MHz</span></span> memory</span> =</span> 700</span> #</span> MB</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Our focus should be on the network.mode</code> stanza. To illustrate what happens behind the scenes when an alloc runs in network.mode=host</code> (host network), we can run the above job.</p> On the machine, we can see that port 7000 (static) and port 27042 (dynamic) are allocated on the host network interface (eth0):</p> </p> We can also see the port and process details using ss</code>:</p> sudo</span> ss</span> -</span>ltpn</span> '</span>sport = :7000</span>'</span></span> State</span> Recv-Q</span> Send-Q</span> Local</span> Address:Port</span> Peer</span> Address:Port</span> Process</span></span> LISTEN</span> 0</span> 4096</span> 95.216.165.210:7000</span> 0.0.0.0:</span></span> users:</span>(</span>(</span>"</span>docker-proxy</span>"</span>,pid</span>=</span>6</span>7</span>0</span>6</span>8</span>,</span>f</span>d</span>=</span>4</span>)</span>)</span></span></code></pre> This config is more suitable for specific workloads - like load balancers or similar deployments where you want to expose the network interface on the host. It’s also helpful for applications running outside of Nomad on that host to connect via the host network interface.</p> However, typically in a job where you want to connect to multiple different allocs - you’d want to set up a bridge network. This generally avoids exposing the workload on the host network directly. It’s a typical setup where you want to put applications behind a reverse proxy (NGINX/Caddy).</p> Let’s change network.mode=bridge</code> in the above job spec and see the changes.</p> $</span> nomad</span> job</span> plan</span> cloak.nomad</span></span> </span> +/-</span> Job:</span> "</span>cloak</span>"</span></span> +/-</span> Task</span> Group:</span> "</span>cloak</span>"</span> (1</span> create/destroy</span> update</span>)</span></span> +</span> Network</span> {</span></span> Hostname:</span> "</span>"</span></span> +</span> MBits:</span> "</span>0</span>"</span></span> +</span> Mode:</span> "</span>bridge</span>"</span></span> +</span> Static</span> Port</span> {</span></span> +</span> HostNetwork:</span> "</span>default</span>"</span></span> +</span> Label:</span> "</span>cloak</span>"</span></span> +</span> To:</span> "</span>7000</span>"</span></span> +</span> Value:</span> "</span>7000</span>"</span></span> }</span></span> }</span></span> -</span> Network</span> {</span></span> Hostname:</span> "</span>"</span></span> -</span> MBits:</span> "</span>0</span>"</span></span> -</span> Mode:</span> "</span>host</span>"</span></span> -</span> Static</span> Port</span> {</span></span> -</span> HostNetwork:</span> "</span>default</span>"</span></span> -</span> Label:</span> "</span>cloak</span>"</span></span> -</span> To:</span> "</span>7000</span>"</span></span> -</span> Value:</span> "</span>7000</span>"</span></span> }</span></span> }</span></span> Task:</span> "</span>cloak</span>"</span></span> </span> +/-</span> Task</span> Group:</span> "</span>redis</span>"</span> (1</span> create/destroy</span> update</span>)</span></span> +</span> Network</span> {</span></span> Hostname:</span> "</span>"</span></span> +</span> MBits:</span> "</span>0</span>"</span></span> +</span> Mode:</span> "</span>bridge</span>"</span></span> +</span> Dynamic</span> Port</span> {</span></span> +</span> HostNetwork:</span> "</span>default</span>"</span></span> +</span> Label:</span> "</span>redis</span>"</span></span> +</span> To:</span> "</span>6379</span>"</span></span> }</span></span> }</span></span> -</span> Network</span> {</span></span> Hostname:</span> "</span>"</span></span> -</span> MBits:</span> "</span>0</span>"</span></span> -</span> Mode:</span> "</span>host</span>"</span></span> -</span> Dynamic</span> Port</span> {</span></span> -</span> HostNetwork:</span> "</span>default</span>"</span></span> -</span> Label:</span> "</span>redis</span>"</span></span> -</span> To:</span> "</span>6379</span>"</span></span> }</span></span> }</span></span> Task:</span> "</span>redis</span>"</span></span></code></pre> Now we don’t see the ports forwarded on the host network:</p> </p> Similarly, ss</code> also shows no process listening on the host network</p> </p> IPTables and Routing</strong>#</a></h2> To understand what happened when we switched the networking mode to bridge</code>, we need to take a look at the Nomad iptables</code> magic which comes into play when using bridge</code> network.</p> I pulled up the iptables</code> and saw specific rules under the chains CNI-FORWARD</code> and NOMAD-ADMIN</code>. These rules, in essence, allow all traffic to and from the allocation’s network namespace.</p> $</span> sudo</span> iptables</span> -</span>L</span> CNI-FORWARD</span></span> Chain</span> CNI-FORWARD</span> (1</span> references</span>)</span></span> target</span> prot</span> opt</span> source</span> destination</span> </span></span> NOMAD-ADMIN</span> all</span> -</span>-</span> anywhere</span> anywhere</span> /</span></span> CNI</span> firewall</span> plugin</span> admin</span> overrides</span> </span>/</span></span> ACCEPT</span> all</span> -</span>-</span> anywhere</span> 172.26.64.5</span> ctstate</span> RELATED,ESTABLISHED</span></span> ACCEPT</span> all</span> -</span>-</span> 172.26.64.5</span> anywhere</span> </span></span> ACCEPT</span> all</span> -</span>-</span> anywhere</span> 172.26.64.6</span> ctstate</span> RELATED,ESTABLISHED</span></span> ACCEPT</span> all</span> -</span>-</span> 172.26.64.6</span> anywhere</span></span> </span> sudo</span> iptables</span> -</span>L</span> NOMAD-ADMIN</span></span> Chain</span> NOMAD-ADMIN</span> (1</span> references</span>)</span></span> target</span> prot</span> opt</span> source</span> destination</span> </span></span> ACCEPT</span> all</span> -</span>-</span> anywhere</span> 172.26.64.0/20</span></span></code></pre> Nomad uses 172.26.64.0/20</code> as the default subnet</a> for the bridge network. The IPs 172.26.64.5</code> and 172.26.64.6</code> are assigned to 2 different allocs in this CIDR. The iptables</code> rules allow complete traffic to flow on this subnet.</p> To check the routing,ip route</code> command can be used.</p> $</span> ip</span> route</span> show</span> 172.26.64.0/20</span></span> 172.26.64.0/20</span> dev</span> nomad</span> proto</span> kernel</span> scope</span> link</span> src</span> 172.26.64.1</span></span></code></pre> It uses the nomad</code> network interface for routing packets related to the default bridge network.</p> Using nsenter</code> we can find more details about the network namespace created for an alloc. Let’s find details about the redis</code> alloc:</p> sudo</span> nsenter</span> -</span>t</span> $(</span>pgrep</span> redis</span>)</span> -</span>-net</span> ip</span> addr</span></span> </span> 1:</span> lo:</span> <</span>LOOPBACK,UP,LOWER_U</span>P</span>></span> mtu</span> 65536</span> qdisc</span> noqueue</span> state</span> UNKNOWN</span> group</span> default</span> qlen</span> 1000</span></span> link/loopback</span> 00:00:00:00:00:00</span> brd</span> 00:00:00:00:00:00</span></span> inet</span> 127.0.0.1/8</span> scope</span> host</span> lo</span></span> valid_lft</span> forever</span> preferred_lft</span> forever</span></span> 2:</span> eth0@if113:</span> <</span>BROADCAST,MULTICAST,UP,LOWER_U</span>P</span>></span> mtu</span> 1500</span> qdisc</span> noqueue</span> state</span> UP</span> group</span> default</span> </span></span> link/ether</span> 76:47:6d:49:00:c0</span> brd</span> ff:ff:ff:ff:ff:ff</span> link-netnsid</span> 0</span></span> inet</span> 172.26.64.5/20</span> brd</span> 172.26.79.255</span> scope</span> global</span> eth0</span></span> valid_lft</span> forever</span> preferred_lft</span> forever</span></span> </span></code></pre> We can see that one end of the pair is eth0</code> (container’s default gateway) which is connected to a network interface with an index 113</code>. For the tunnel to actually work, the veth pair should also exist on the host:</p> $</span> ip</span> a</span></span> 113:</span> veth3402deda@if2:</span> <</span>BROADCAST,MULTICAST,UP,LOWER_U</span>P</span>></span> mtu</span> 1500</span> qdisc</span> noqueue</span> master</span> nomad</span> state</span> UP</span> group</span> default</span> </span></span> link/ether</span> 3a:85:1b:37:75:17</span> brd</span> ff:ff:ff:ff:ff:ff</span> link-netnsid</span> 0</span></span> inet6</span> fe80::3885:1bff:fe37:7517/64</span> scope</span> link</span> </span></span> valid_lft</span> forever</span> preferred_lft</span> forever</span></span></code></pre> So, when we see veth3402deda@if2</code> in the host’s network namespace (with the index 113</code>), and then we see eth0@if113</code> inside the Redis container, we can infer that these two interfaces form a veth</code> pair: veth3402deda@if2</code> on the host side and eth0</code> inside the container. This connection enables the container to communicate with the external network through the host’s network stack.</p> Capturing packets#</a></h2> We can capture TCP packets on the veth</code> interface to see the routing work:</p> sudo</span> tcpdump</span> -</span>i</span> veth971858d5</span> -</span>n</span></span> tcpdump:</span> verbose</span> output</span> suppressed,</span> use</span> -</span>v</span>[</span>v</span>]</span>...</span> for</span> full</span> protocol</span> decode</span></span> listening</span> on</span> veth971858d5,</span> link-type</span> EN10MB</span> (Ethernet</span>), snapshot length 262144 bytes</span></span> 10:51:27.801319</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [S</span>], seq 1331933249, win 65495, options </span>[</span>mss 65495,sackOK,TS val </span>248300785</span> ecr 0,nop,wscale 7</span>]</span>, length 0</span></span> 10:51:27.801549</span> IP</span> 172.26.64.6.7000</span> ></span> 172.26.64.1.35826:</span> Flags</span> [S.</span>], seq 107697826, ack 1331933250, win 65160, options </span>[</span>mss 1460,sackOK,TS val </span>3965422857</span> ecr 248300785,nop,wscale 7</span>]</span>, length 0</span></span> 10:51:27.801616</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [.</span>], ack 1, win 512, options </span>[</span>nop,nop,TS val </span>248300785</span> ecr 3965422857</span>]</span>, length 0</span></span> 10:51:27.801737</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [P.</span>], seq 1:79, ack 1, win 512, options </span>[</span>nop,nop,TS val </span>248300786</span> ecr 3965422857</span>]</span>, length 78</span></span> 10:51:27.801751</span> IP</span> 172.26.64.6.7000</span> ></span> 172.26.64.1.35826:</span> Flags</span> [.</span>], ack 79, win 509, options </span>[</span>nop,nop,TS val </span>3965422858</span> ecr 248300786</span>]</span>, length 0</span></span> 10:51:27.802022</span> IP</span> 172.26.64.6.7000</span> ></span> 172.26.64.1.35826:</span> Flags</span> [P.</span>], seq 1:4097, ack 79, win 509, options </span>[</span>nop,nop,TS val </span>3965422858</span> ecr 248300786</span>]</span>, length 4096</span></span> 10:51:27.802059</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [.</span>], ack 4097, win 491, options </span>[</span>nop,nop,TS val </span>248300786</span> ecr 3965422858</span>]</span>, length 0</span></span> 10:51:27.802120</span> IP</span> 172.26.64.6.7000</span> ></span> 172.26.64.1.35826:</span> Flags</span> [P.</span>], seq 4097:5396, ack 79, win 509, options </span>[</span>nop,nop,TS val </span>3965422858</span> ecr 248300786</span>]</span>, length 1299</span></span> 10:51:27.802135</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [.</span>], ack 5396, win 502, options </span>[</span>nop,nop,TS val </span>248300786</span> ecr 3965422858</span>]</span>, length 0</span></span> 10:51:27.803484</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [F.</span>], seq 79, ack 5396, win 512, options </span>[</span>nop,nop,TS val </span>248300787</span> ecr 3965422858</span>]</span>, length 0</span></span> 10:51:27.803567</span> IP</span> 172.26.64.6.7000</span> ></span> 172.26.64.1.35826:</span> Flags</span> [F.</span>], seq 5396, ack 80, win 509, options </span>[</span>nop,nop,TS val </span>3965422859</span> ecr 248300787</span>]</span>, length 0</span></span> 10:51:27.803597</span> IP</span> 172.26.64.1.35826</span> ></span> 172.26.64.6.7000:</span> Flags</span> [.</span>], ack 5397, win 512, options </span>[</span>nop,nop,TS val </span>248300787</span> ecr 3965422859</span>]</span>, length 0</span></span> 10:53:08.523431</span> IP</span> 172.26.64.6.53042</span> ></span> 95.216.165.210.27372:</span> Flags</span> [.</span>], ack 2169295212, win 501, options </span>[</span>nop,nop,TS val </span>735542538</span> ecr 4133067854</span>]</span>, length 0</span></span> 10:53:08.523551</span> IP</span> 95.216.165.210.27372</span> ></span> 172.26.64.6.53042:</span> Flags</span> [.</span>], ack 1, win 509, options </span>[</span>nop,nop,TS val </span>4133379150</span> ecr 735231242</span>]</span>, length 0</span></span> 10:53:08.523554</span> IP</span> 95.216.165.210.27372</span> ></span> 172.26.64.6.53042:</span> Flags</span> [.</span>], ack 1, win 509, options </span>[</span>nop,nop,TS val </span>4133379150</span> ecr 735231242</span>]</span>, length 0</span></span> 10:53:08.523562</span> IP</span> 172.26.64.6.53042</span> ></span> 95.216.165.210.27372:</span> Flags</span> [.</span>], ack 1, win 501, options </span>[</span>nop,nop,TS val </span>735542538</span> ecr 4133379150</span>]</span>, length 0</span></span> </span></code></pre> To summarize the output, we can see that the log is showing a TCP connection between 172.26.64.1 (source) and 172.26.64.6 (destination), specifically on port 7000. 172.26.64.1</code> happens to be the gateway for nomad</code> subnet.</p> Summary#</a></h2> Hope this post clarified some networking internals and behind the scenes magic when using Nomad bridge networking. Refer to my other post - Nomad networking explained</a> for a practical breakdown of all the different ways to expose and connect applications in a Nomad cluster.</p> Fin!</p> Analyzing credit card transactions with GPT and Python 2023-04-30T07:08:02+00:00 You know those budget freaks? People who log and categorise every Rupee they’ve spent over the month? The financially sane people? I am definitely not one and I suck at it.</p> I moved cities a couple of months back and had some big ticket spends off late, mostly financed by credit card. I wanted an easy way to list down where all I’ve spent most money and spot some recurring expenses so I can be better prepared for them from next month. I’ve found that broadly keeping an idea of things where you spend money works for me (v/s the two extremes - completely blind or logging every small transactions). Of course I know people who make budgeting a habit but I only wish I was consistent enough to do that.</p> Anyway, I downloaded the statement in CSV format from my bank.</p> Initially I thought I’d use some simple Excel to make sense of this but I realised how bad my Excel skills really are. I got an idea to dump the CSV file to ChatGPT (yay privacy) and ask questions. It kinda sucked at it and gave wrong answers for a lot of questions and also started to hallucinate data which wasn’t even present in the CSV.</p> The next most obvious step would be to write a simple script and parse it. I wanted to experiment if ChatGPT could do this entire exercise of writing the script and the relevant code for the analysis I wanted to perform.</p> Here’s the initial prompt I gave:</p> I am pasting a CSV file which also includes the first row as header. Write Python code to parse this:</span></span> </span> ,,Transaction Date,Details,,,Amount (INR),,,Reference Number,,</span></span> ,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,7477468633,,</span></span> ... other data ...</span></span></code></pre> It returned the following code:</p> import</span> pandas</span> as</span> pd</span></span> from</span> io</span> import</span> StringIO</span></span> </span> csv_data</span> =</span> """</span>\</span></span> ,,Transaction Date,Details,,,Amount (INR),,,Reference Number,,</span></span> ,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,747746xxxx,,</span></span> # ... (the rest of your CSV data)</span></span> ,,03/04/2023,"DIGITAL LOUNGE, GURGAON, IN",,,799.00 Dr.,,,738201xxx,,</span>"""</span></span> </span> #</span> Read the CSV data into a pandas DataFrame</span></span> data</span> =</span> StringIO</span>(</span>csv_data</span>)</span></span> df</span> =</span> pd</span>.</span>read_csv</span>(</span>data</span>,</span> header</span>=</span>1</span>)</span></span> </span> #</span> Clean up the DataFrame (remove unnecessary columns)</span></span> df</span> =</span> df</span>.</span>dropna</span>(</span>axis</span>=</span>1</span>,</span> how</span>=</span>"</span>all</span>"</span>)</span></span> </span> #</span> Rename columns</span></span> df</span>.</span>columns</span> =</span> [</span></span> "</span>Transaction Date</span>"</span>,</span></span> "</span>Details</span>"</span>,</span></span> "</span>Amount (INR)</span>"</span>,</span></span> "</span>Reference Number</span>"</span></span> ]</span></span> </span> print</span>(</span>df</span>)</span></span></code></pre> Looking at this, I was a bit impressed as it figured the CSV contains some useless empty columns and it removed (without me giving any information about it).</p> I also asked it to modify the code to read the file locally from disk and it swapped StringIO</code> with the path to CSV file:</p> #</span> Provide the file path to your local CSV file</span></span> file_path</span> =</span> "</span>your_file.csv</span>"</span></span> </span> #</span> Read the CSV data into a pandas DataFrame</span></span> df</span> =</span> pd</span>.</span>read_csv</span>(</span>file_path</span>,</span> header</span>=</span>1</span>)</span></span></code></pre> </p> Next, I prompted to do some analysis on it:</p> Write some more code to do the analysis to answer these questions:</span></span> </span> - How many total debit and credit entries</span></span> - What is the total amount of debits and total amount of credits</span></span> - Which vendor have I spent the most?</span></span> - How much have I spent on `SWIGGY` ?</span></span></code></pre> It returned some one-liners to answer each question:</p> #</span> Filter debit and credit entries</span></span> debit_entries</span> =</span> df</span>[</span>df</span>[</span>"</span>Amount (INR)</span>"</span>]</span> ></span> 0</span>]</span></span> credit_entries</span> =</span> df</span>[</span>df</span>[</span>"</span>Amount (INR)</span>"</span>]</span> <</span> 0</span>]</span></span> </span> #</span> Count the total number of debit and credit entries</span></span> total_debit_entries</span> =</span> len</span>(</span>debit_entries</span>)</span></span> total_credit_entries</span> =</span> len</span>(</span>credit_entries</span>)</span></span> </span> #</span> Calculate the total amount of debits and credits</span></span> total_debit_amount</span> =</span> debit_entries</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span></span> total_credit_amount</span> =</span> credit_entries</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span></span> </span> #</span> Find the vendor with the most spending</span></span> most_spent_vendor</span> =</span> debit_entries</span>.</span>groupby</span>(</span>"</span>Details</span>"</span>)</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span>.</span>idxmin</span>(</span>)</span></span> </span> #</span> Calculate the total amount spent on SWIGGY</span></span> swiggy_spending</span> =</span> debit_entries</span>[</span>debit_entries</span>[</span>"</span>Details</span>"</span>]</span>.</span>str</span>.</span>contains</span>(</span>"</span>SWIGGY</span>"</span>,</span> case</span>=</span>False</span>)</span>]</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span></span> </span> #</span> Calculate the total amount spent on AMAZON RECHARGES</span></span> amazon_recharges_spending</span> =</span> debit_entries</span>[</span>debit_entries</span>[</span>"</span>Details</span>"</span>]</span>.</span>str</span>.</span>contains</span>(</span>"</span>AMAZON RECHARGES</span>"</span>,</span> case</span>=</span>False</span>)</span>]</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span></span> </span> #</span> Print the results</span></span> print</span>(</span>f</span>"</span>Total debit entries: </span>{</span>total_debit_entries</span>}</span>"</span>)</span></span> print</span>(</span>f</span>"</span>Total credit entries: </span>{</span>total_credit_entries</span>}</span>"</span>)</span></span> </span> print</span>(</span>f</span>"</span>Total debit amount: </span>{</span>total_debit_amount</span>}</span>"</span>)</span></span> print</span>(</span>f</span>"</span>Total credit amount: </span>{</span>total_credit_amount</span>}</span>"</span>)</span></span> </span> print</span>(</span>f</span>"</span>Vendor with the most spending: </span>{</span>most_spent_vendor</span>}</span>"</span>)</span></span> </span> print</span>(</span>f</span>"</span>Total amount spent on SWIGGY: </span>{</span>swiggy_spending</span>}</span>"</span>)</span></span></code></pre> At this point, I know that this will fail because we’ve not cleaned up the data. The column Amount (INR)</code> needs cleaning up. I prompted ChatGPT to write a function to clean this column:</p> The amount column as `21,620.54 Dr.` to represent that Rs 21,620.54 is debited. And `73,500.00 Cr.` to represent that Rs 73,500.00 is credited. Can you write a function to apply this transformation on the `Amount (INR)` column?</span></span></code></pre> It responded with:</p> #</span> Clean up the data</span></span> def</span> clean_amount</span>(</span>value</span>)</span>:</span></span> if</span> '</span> Cr.</span>'</span> in</span> value</span>:</span></span> value</span> =</span> value</span>.</span>replace</span>(</span>'</span> Cr.</span>'</span>,</span> '</span>'</span>)</span></span> return</span> -</span>float</span>(</span>value</span>.</span>replace</span>(</span>'</span>,</span>'</span>,</span> '</span>'</span>)</span>)</span></span> elif</span> '</span> Dr.</span>'</span> in</span> value</span>:</span></span> value</span> =</span> value</span>.</span>replace</span>(</span>'</span> Dr.</span>'</span>,</span> '</span>'</span>)</span></span> return</span> float</span>(</span>value</span>.</span>replace</span>(</span>'</span>,</span>'</span>,</span> '</span>'</span>)</span>)</span></span> </span> df</span>[</span>"</span>Amount (INR)</span>"</span>]</span> =</span> df</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>apply</span>(</span>clean_amount</span>)</span></span> </span> #</span> Now proceed with the analysis code</span></span> </span></code></pre> Perfect! After transforming the amounts, I ran the analysis code:</p> </p> The next prompt I gave was to analyse the spending in various categories.</p> In the above CSV file can you guess the categories of each merchant based on the `Details`? And write the code for grouping the spends on various categories. If you're unable to figure out the category for the transaction, you can use a "Misc" column to bucket them.</span></span></code></pre> It actually did an okayish job at this and ignored a lot of vendors which I think it could have guessed easily:</p> def</span> categorize</span>(</span>details</span>)</span>:</span></span> details</span> =</span> details</span>.</span>lower</span>(</span>)</span></span> if</span> "</span>swiggy</span>"</span> in</span> details</span>:</span></span> return</span> "</span>Food Delivery</span>"</span></span> elif</span> "</span>amazon recharges</span>"</span> in</span> details</span>:</span></span> return</span> "</span>Recharge</span>"</span></span> elif</span> "</span>uber</span>"</span> in</span> details</span> or</span> "</span>ola</span>"</span> in</span> details</span>:</span></span> return</span> "</span>Cab</span>"</span></span> elif</span> "</span>zomato</span>"</span> in</span> details</span>:</span></span> return</span> "</span>Food Delivery</span>"</span></span> elif</span> "</span>netflix</span>"</span> in</span> details</span> or</span> "</span>prime video</span>"</span> in</span> details</span>:</span></span> return</span> "</span>Streaming</span>"</span></span> else</span>:</span></span> return</span> "</span>Misc</span>"</span></span> </span> #</span> Create a new column called "Category" by applying the categorize function to the "Details" column</span></span> df</span>[</span>"</span>Category</span>"</span>]</span> =</span> df</span>[</span>"</span>Details</span>"</span>]</span>.</span>apply</span>(</span>categorize</span>)</span></span> </span> #</span> Group the transactions by category and calculate the total amount spent in each category</span></span> grouped</span> =</span> df</span>[</span>df</span>[</span>"</span>Amount (INR)</span>"</span>]</span> ></span> 0</span>]</span>.</span>groupby</span>(</span>"</span>Category</span>"</span>)</span>[</span>"</span>Amount (INR)</span>"</span>]</span>.</span>sum</span>(</span>)</span></span> </span> print</span>(</span>grouped</span>)</span></span></code></pre> I decided to give some manual inputs to it to refine the function.</p> I can help you with categorising it further.</span></span> `Blink` - Grocery</span></span> `Indigo` - Travel</span></span> `PEGS` - Alcohol</span></span> `Nike` - Shopping</span></span> </span> Also note that the cateorgy name matching should be case-insensitive.</span></span></code></pre> And it modified the Python snippet to add these rules.</p> </p> Mixed reactions looking at this. Happy that I could practically get exactly the result I had in mind in just 10 minutes without writing any code. Sad because damn I need to limit those empty calories from next month (famous last words).</p> Next, I wanted to see if my spends on weekends are higher or not. (I don’t expect them to be, but you never know).</p> Prompt:</p> Can you also write the code to figure how much spends I've done grouped by the day of week?</span></span></code></pre> Response:</p> #</span> Convert the 'Transaction Date' column to a datetime format</span></span> df</span>[</span>'</span>Transaction Date</span>'</span>]</span> =</span> pd</span>.</span>to_datetime</span>(</span>df</span>[</span>'</span>Transaction Date</span>'</span>]</span>)</span></span> </span> #</span> Create a new column called "Day of Week" containing the day of the week for each transaction</span></span> df</span>[</span>'</span>Day of Week</span>'</span>]</span> =</span> df</span>[</span>'</span>Transaction Date</span>'</span>]</span>.</span>dt</span>.</span>day_name</span>(</span>)</span></span> </span> #</span> Group the transactions by the "Day of Week" and calculate the total amount spent on each day of the week</span></span> grouped_by_day</span> =</span> df</span>[</span>df</span>[</span>"</span>Amount (INR)</span>"</span>]</span> ></span> 0</span>]</span>.</span>groupby</span>(</span>'</span>Day of Week</span>'</span>)</span>[</span>'</span>Amount (INR)</span>'</span>]</span>.</span>sum</span>(</span>)</span>.</span>abs</span>(</span>)</span></span> </span> #</span> Sort the grouped data by the total amount spent in descending order</span></span> sorted_grouped_by_day</span> =</span> grouped_by_day</span>.</span>sort_values</span>(</span>ascending</span>=</span>False</span>)</span></span> </span> print</span>(</span>sorted_grouped_by_day</span>)</span></span></code></pre> </p> It was a fun 10-15 min exercise to figure out my spending habits based on the last month’s statement. I intend to do this for the next couple of months and then it would make sense to write more queries which would show trend-lines of spends in various categories over time.</p> Honestly, I just loved how ChatGPT made this task so seemingly simple. It’s not that I can’t myself write the code for these kind of simple analysis. It’s the sheer power at your hand to go from ideation phase to answer within seconds. And I think that’s why I love it so much. I didn’t have to go through Pandas docs (because I don’t use that in my day job so it’s quite normal to not know various syntax/functions that I could use) and I’d grok through different StackOverflow questions to achieve what I wanted to. And maybe imagining all this resistance on a Sunday morning would have meant that I never got to write the script in the first place.</p> Fin!</p> The curious case of missing and duplicate logs 2023-02-16T08:27:13+00:00 At work, we use a Vector</a> pipeline for processing and shipping logs to Clickhouse</a>. We also self-host our SMTP servers and recently started using Haraka SMTP</a>. While Haraka is excellent in raw performance and throughput, it needed an external logging plugin for audit and compliance purposes. I wrote haraka-plugin-outbound-logger</a> to log basic metadata like timestamps/subject/SMTP response in a JSON file.</p> The plan was to dump these logs into a file and use Vector’s file source for reading them and doing the further transformation. However, things went differently than I had planned. There were mainly two issues propped up due to bad Vector configuration.</p> Missing logs#</a></h3> The vector configuration to read the file looked like this:</p> [</span>sources</span>.</span>outbound_logger</span>]</span></span> type</span> =</span> "</span>file</span>"</span></span> include</span> =</span> [</span>"</span>/var/log/haraka/outbound/.log</span>"</span>]</span></span> read_from</span> =</span> "</span>beginning</span>"</span></span> #</span> Remove the files after 24 hours. Vector must have permission to delete these files.</span></span> remove_after_secs</span> =</span> 86400</span></span> fingerprint</span>.</span>strategy</span> =</span> "</span>device_and_inode</span>"</span></span> </span></code></pre> Vector has a handy configuration of automagically</em> deleting the file if the file hasn’t received any new write in the configured time interval. So remove_after_secs=86400</code> specifies if the file hasn’t had any new writes since 24h, it can delete it. It made sense to configure because our workload was a shorter process. It was a batch job done once every N days.</p> When the file didn’t receive any new writes after 24h, vector</code> deleted the file as expected. However, the plugin continued logging into the same</em> file handler, even for newer batch jobs. As a result, the file didn’t receive any new logs and was empty.</p> I created a minimal POC to reproduce this seemingly strange issue:</p> var</span> pino</span> =</span> require</span>(</span>'</span>pino</span>'</span>)</span>;</span></span> </span> //</span> Initialise pino js logger and inject in the plugin context.</span></span> var</span> opts</span> =</span> {</span></span> name</span>:</span> '</span>outbound_logger</span>'</span>,</span></span> level</span>:</span> '</span>debug</span>'</span>,</span></span> //</span> uses the ISO time format.</span></span> timestamp</span>:</span> pino</span>.</span>stdTimeFunctions</span>.</span>isoTime</span>,</span></span> formatters</span>:</span> {</span></span> level</span>:</span> (</span>label</span>)</span> =></span> {</span></span> return</span> {</span> level</span>:</span> label</span> }</span>;</span></span> }</span>,</span></span> }</span>,</span></span> }</span></span> </span> pilog</span> =</span> pino</span>(</span>opts</span>,</span> pino</span>.</span>destination</span>(</span>`</span>${</span>__dirname</span>}</span>/app.log</span>`</span>)</span>)</span></span> </span> pilog</span>.</span>info</span>(</span>'</span>this is a first message</span>'</span>)</span></span> setTimeout</span>(</span>(</span>)</span> =></span> pilog</span>.</span>info</span>(</span>'</span>this message should get logged</span>'</span>)</span>,</span> 10000</span>)</span></span> pilog</span>.</span>info</span>(</span>'</span>this message will be recorded as well</span>'</span>)</span></span></code></pre> This snippet logs to app.log</code>.</p> {</span>"</span>level</span>"</span>:</span>"</span>info</span>"</span>,</span>"</span>time</span>"</span>:</span>"</span>2023-02-16T06:45:04.031Z</span>"</span>,</span>"</span>pid</span>"</span>:</span>206573</span>,</span>"</span>hostname</span>"</span>:</span>"</span>pop-os</span>"</span>,</span>"</span>name</span>"</span>:</span>"</span>outbound_logger</span>"</span>,</span>"</span>msg</span>"</span>:</span>"</span>this is a first message</span>"</span>}</span></span> {</span>"</span>level</span>"</span>:</span>"</span>info</span>"</span>,</span>"</span>time</span>"</span>:</span>"</span>2023-02-16T06:45:04.031Z</span>"</span>,</span>"</span>pid</span>"</span>:</span>206573</span>,</span>"</span>hostname</span>"</span>:</span>"</span>pop-os</span>"</span>,</span>"</span>name</span>"</span>:</span>"</span>outbound_logger</span>"</span>,</span>"</span>msg</span>"</span>:</span>"</span>this message will be recorded as well</span>"</span>}</span></span> </span></code></pre> During the 10s time interval, I deleted the file from the disk rm app.log</code> to mimic the behaviour of remove_after_secs</code>. I expected the file to get re-created and this message should get logged</code> logged by the above script.</p> However, that didn’t</strong> happen. The script didn’t complain about a missing file, either. I was perplexed and did some google-fu and found the following via Stackoverflow</a>:</p> The writes actually do not fail.When you delete a file that is open in another program you are deleting a named link to that file’s inode. The program that has it open still points to that inode. It will happily keep writing to it, actually writing to disk. Only now you don’t have a way to look it at, because you deleted the named reference to it. (If there were other references, e.g. hard links, you would still be able to!).</p> </blockquote> This is exactly what was happening in production. When vector</code> deleted the file (as configured via remove_after_secs</code> ), the plugin didn’t know about it and kept writing to the same inode. This was a major TIL moment for me.</p> Fix</strong>: The fix was simple enough; I removed remove_after_secs</code> from Vector’s config. To address the problem of the file not growing unbounded forever, I created a logrotate</code> config:</p> /opt/app/logs/app.log {</span></span> daily</span></span> rotate 15</span></span> dateext</span></span> dateformat -%Y-%m-%d.log</span></span> delaycompress</span></span> compress</span></span> notifempty</span></span> missingok</span></span> copytruncate</span></span> }</span></span></code></pre> Some notes:</p> copytruncate</code> is useful in this context. It copies the existing file to a new one, which now becomes a stale one. The current (active) file will be truncated to zero bytes. E.g., if app.log</code> it is rotated, logrotate will copy the file to a new file app-2023-02-05.log</code> and then truncate the existing one to zero bytes.</li> delaycompress</code> will not compress the logs until the next rotation happens. This is useful if vector</code> hasn’t finished processing the log and can continue to do that.</li> </ul> Duplicate Logs#</a></h3> Now after fixing the case of missing logs, I found myself in the opposite problem - now the logs were duplicated</em> on Clickhouse.</p> My mental state at that moment couldn’t be more accurately described than this meme:</p> </p> To add more context, before developing my plugin for logging email delivery in Haraka, we used another plugin (acharkizakaria/haraka-plugin-accounting-files</a>) to get these logs. This plugin records the metadata to CSV files. Still, there were some issues in properly escaping the subject lines (if the subject had a comma, that was incorrectly parsed); hence, the log file had inconsistent output. To address these issues, I found writing another plugin from scratch that outputs to a fixed JSON schema is better.</p> As seen above, vector</code>’s file source was configured like the following for reading CSV files. The only change here is that remove_after_secs</code> is gone after fixing issue #1.</p> [</span>sources</span>.</span>outbound_logger</span>]</span></span> type</span> =</span> "</span>file</span>"</span></span> include</span> =</span> [</span>"</span>/var/log/haraka/outbound/.log</span>"</span>]</span></span> read_from</span> =</span> "</span>beginning</span>"</span></span> fingerprint</span>.</span>strategy</span> =</span> "</span>device_and_inode</span>"</span></span></code></pre> Vector “fingerprints</a>” the file source, as it keeps the checkpoint of how many bytes it has read for each file in its own “disk buffer”. This buffer is helpful if Vector crashes so it can restart reading the file from where it last stopped.</p> There are two strategies for fingerprinting that vector uses:</p> The checksum</code> strategy uses CRC check on the first N lines of the file.</li> The device_and_inode strategy</code> uses the disk’s actual inode location to identify the file uniquely.</li> </ul> As I was using a different plugin which logged to a CSV file, the checksum strategy did not work in my context. Since vector fingerprints, the first few bytes (usually just enough for a header of a CSV), all the CSV files in that disk would have the same title, and Vector would not read all of them. To work around this, I changed the fingerprint.starategy = "device_and_inode"</code> so Vector uniquely identifies all CSV files by their inode path. (In hindsight, I should have just used checksum</code> with a higher value of fingerprint.lines</a> value.)</p> The mistake this time was when I switched to a JSON log file, I continued with the device_and_inode</code> strategy. This isn’t a problem if there is no log rotation setup. Since I did configure logrotate</code> to fix issue #1, as you would have guessed, copytruncate</code> created another log file, and because I was using device_and_inode</code> strategy, vector thought this was a “new” file to be watched and processed. So now I had duplicate entries from this new file, which is technically just an older log-rotated file.</p> The fix:</strong></p> [</span>sources</span>.</span>outbound_logger</span>.</span>fingerprint</span>]</span></span> lines</span> =</span> 14</span></span> strategy</span> =</span> "</span>checksum</span>"</span></span> ignored_header_bytes</span> =</span> 2048</span></span></code></pre> I switched back to the default checksum</code> strategy and adjusted the thresholds for lines/header bytes to account for JSON logs. The same is also documented</a> very clearly in Vector and it was my RTFM moment.</p> This strategy avoids the common pitfalls associated with using device and inode names since inode names can be reused across files. This enables Vector to properly tail files across various rotation strategies.</p> </blockquote> Phew! I am glad after these fixes; vector</code> is durably and reliably processing all the logs and logrotate</code> is happily working in conjunction as well. I hope documenting my learnings about these production issues would help someone with the same problems.</p> Fin!</p> Structured logging in Go with slog 2023-02-15T07:29:53+00:00 A few months ago, a proposal</a> for adding a structured logging library in Go was introduced by Jonathan Amsterdam</a>. At present, Go has a minimal and bare-bones log package which works all right for basic use cases. However, the current library has a few shortcomings that this proposal aims to solve:</p> Emitting logs with different severity/levels</li> Structured output: Makes parsing of logs harder</li> Logging a set of common fields/attributes</li> Difficult to have a log object inside libraries as each service could have its log implementation.</li> </ul> As a result, many code bases have their wrappers around the log package. Additionally, there are plenty of 3rd party libraries to choose from - including logf</a> (which my work colleagues and I built at Zerodha).</p> This article is about how to get started with slog</a> for logging in Go applications.</p> NOTE</strong> Since slog is currently in the proposal state and hasn’t yet merged in the official library, the API could change in future.</p> </p> </blockquote> Architecture of slog#</a></h2> At a higher level, slog contains three main entities:</p> Logger</strong>: The user-facing API for interacting with slog. All the public methods are defined on the Logger object.</li> Record</strong>: Contains information about the log event itself. A standard record will have timestamp, level and message fields as default. Additional attributes and metadata like caller info can be added to the Record.</li> Handlers</strong>: A handler is an interface</em> implementation. The Logger object passes the Record to a handler, and the handler can choose whatever it wants to do with the Record. This is a common approach in Go libraries, where a “provider” can be abstracted in handling that task. Currently, slog ships with two handlers: JSON and logfmt</a>. Some projects have also created handlers for zap/logrus (popular 3rd party libraries).</li> </ul> Initialization#</a></h3> This snippet initializes a Text Handler, which produces logfmt</code> format messages on os.Stdout</code>.</p> package</span> main</span></span> </span> import</span> (</span></span> "</span>os</span>"</span></span> </span> "</span>golang.org/x/exp/slog</span>"</span></span> )</span></span> </span> func</span> main</span>(</span>)</span> {</span></span> log</span> :=</span> slog</span>.</span>New</span>(</span>slog</span>.</span>NewTextHandler</span>(</span>os</span>.</span>Stdout</span>)</span>)</span></span> log</span>.</span>Info</span>(</span>"</span>Hello world</span>"</span>)</span></span> </span> fakeErr</span> :=</span> os</span>.</span>ErrNotExist</span></span> log</span>.</span>Error</span>(</span>"</span>something went wrong</span>"</span>,</span> fakeErr</span>,</span> "</span>file</span>"</span>,</span> "</span>/tmp/abc.txt</span>"</span>)</span></span> }</span></span> </span></code></pre> Log output:</p> time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>9</span>:</span>5</span>8</span>:</span>1</span>0</span>.</span>6</span>1</span>5</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>I</span>N</span>F</span>O</span> msg</span>=</span>"</span>Hello world</span>"</span></span> time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>9</span>:</span>5</span>8</span>:</span>1</span>0</span>.</span>6</span>1</span>5</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>E</span>R</span>R</span>O</span>R</span> msg</span>=</span>"</span>something went wrong</span>"</span> file</span>=</span>/</span>t</span>m</span>p</span>/</span>a</span>b</span>c</span>.</span>t</span>x</span>t</span> err</span>=</span>"</span>file does not exist</span>"</span></span></code></pre>Customizing#</a></h3> You’ll notice that the caller information isn’t exposed by default. The reason could be that finding the stack trace of the calling line is a bit expensive operation. However, for libraries/apps which need it can do that by customizing the handler:</p> func</span> main</span>(</span>)</span> {</span></span> handler</span> :=</span> slog</span>.</span>HandlerOptions</span>{</span>AddSource</span>:</span> true</span>}</span></span> log</span> :=</span> slog</span>.</span>New</span>(</span>handler</span>.</span>NewTextHandler</span>(</span>os</span>.</span>Stdout</span>)</span>)</span></span> </span> log</span>.</span>Info</span>(</span>"</span>Hello world</span>"</span>)</span></span> }</span></span> </span></code></pre> Log Output:</p> time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>2</span>:</span>1</span>7</span>:</span>5</span>3</span>.</span>7</span>4</span>2</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>I</span>N</span>F</span>O</span> source</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>k</span>a</span>r</span>a</span>n</span>/</span>C</span>o</span>d</span>e</span>/</span>P</span>e</span>r</span>s</span>o</span>n</span>a</span>l</span>/</span>s</span>l</span>o</span>g</span>-</span>e</span>x</span>a</span>m</span>p</span>l</span>e</span>s</span>/</span>m</span>a</span>i</span>n</span>.</span>g</span>o</span>:</span>1</span>4</span> msg</span>=</span>"</span>Hello world</span>"</span></span></code></pre>Attributes#</a></h3> Sometimes, it’s helpful to append specific metadata to each log line which will help in aggregating/filtering with a central log-collecting agent. E.g., you can export a component key for each sub-service of your primary application.</p> func</span> main</span>(</span>)</span> {</span></span> log</span> :=</span> slog</span>.</span>New</span>(</span>slog</span>.</span>NewTextHandler</span>(</span>os</span>.</span>Stdout</span>)</span>)</span>.</span>With</span>(</span>"</span>component</span>"</span>,</span> "</span>demo</span>"</span>)</span></span> log</span>.</span>Info</span>(</span>"</span>Hello world</span>"</span>)</span></span> }</span></span> </span></code></pre> Log Output:</p> time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>2</span>:</span>2</span>1</span>:</span>5</span>0</span>.</span>2</span>3</span>1</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>I</span>N</span>F</span>O</span> msg</span>=</span>"</span>Hello world</span>"</span> component</span>=</span>d</span>e</span>m</span>o</span></span></code></pre>Nested Keys#</a></h3> So far, we’ve seen flat keys in the log message. It may be helpful to group together specific keys together and form a nested object. In JSON, that would mean a top-level object with different fields inside. However, in logfmt</code>, it would-be parent.child</code> format.To use nested keys, slog.Group</code> can be used. This example uses http</code> as the top-level key, and all its associated fields will be nested inside.</p> log</span>.</span>Info</span>(</span>"</span>Hello world</span>"</span>,</span> slog</span>.</span>Group</span>(</span>"</span>http</span>"</span>,</span></span> slog</span>.</span>String</span>(</span>"</span>method</span>"</span>,</span> "</span>GET</span>"</span>)</span>,</span></span> slog</span>.</span>Int</span>(</span>"</span>status</span>"</span>,</span> 200</span>)</span>,</span></span> slog</span>.</span>Duration</span>(</span>"</span>duration</span>"</span>,</span> 250</span>)</span>,</span></span> slog</span>.</span>String</span>(</span>"</span>method</span>"</span>,</span> "</span>GET</span>"</span>)</span>,</span></span> slog</span>.</span>String</span>(</span>"</span>path</span>"</span>,</span> "</span>/api/health</span>"</span>)</span>)</span>)</span></span> </span></code></pre>time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>2</span>:</span>3</span>0</span>:</span>4</span>3</span>.</span>1</span>3</span>0</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>I</span>N</span>F</span>O</span> msg</span>=</span>"</span>Hello world</span>"</span> component</span>=</span>d</span>e</span>m</span>o</span> http.method</span>=</span>G</span>E</span>T</span> http.status=</span>200</span> http.duration=250ns</span> http.method=GET</span> http.path=/api/health</span></span></code></pre>Configurable Handlers#</a></h3> JSON logs are daunting and tedious to read when locally developing applications. However, it’s a great fit for machine parsing of the logs. logfmt</code> hits the sweet spot for being machine parseable and human-readable.However, thanks to the powerful “interface” implementation approach, it’s easy to switch to any handler via user-configurable methods (like config files/env variables):</p> package</span> main</span></span> </span> import</span> (</span></span> "</span>os</span>"</span></span> </span> "</span>golang.org/x/exp/slog</span>"</span></span> )</span></span> </span> func</span> main</span>(</span>)</span> {</span></span> var</span> (</span></span> env</span> =</span> os</span>.</span>Getenv</span>(</span>"</span>APP_ENV</span>"</span>)</span></span> handler</span> slog</span>.</span>Handler</span></span> )</span></span> </span> switch</span> env</span> {</span></span> case</span> "</span>production</span>"</span>:</span></span> handler</span> =</span> slog</span>.</span>NewJSONHandler</span>(</span>os</span>.</span>Stdout</span>)</span></span> default</span>:</span></span> handler</span> =</span> slog</span>.</span>NewTextHandler</span>(</span>os</span>.</span>Stdout</span>)</span></span> }</span></span> </span> log</span> :=</span> slog</span>.</span>New</span>(</span>handler</span>)</span></span> log</span>.</span>Info</span>(</span>"</span>Hello world</span>"</span>)</span></span> }</span></span> </span></code></pre>$</span> go</span> run</span> main.go</span></span> time</span>=</span>2</span>0</span>2</span>3</span>-</span>0</span>2</span>-</span>1</span>5</span>T</span>1</span>2</span>:</span>3</span>9</span>:</span>4</span>5</span>.</span>5</span>4</span>3</span>+</span>0</span>5</span>:</span>3</span>0</span> level</span>=</span>I</span>N</span>F</span>O</span> msg</span>=</span>"</span>Hello world</span>"</span></span> $</span> APP_ENV=production</span> go</span> run</span> main.go</span></span> {</span>"</span>time</span>"</span>:</span>"</span>2023-02-15T12:39:53.523477544+05:30</span>"</span>,</span>"</span>level</span>"</span>:</span>"</span>INFO</span>"</span>,</span>"</span>msg</span>"</span>:</span>"</span>Hello world</span>"</span>}</span></span> </span></code></pre>Summary#</a></h2> slog</code> is an excellent proposal, and it’s high time Go gets its official structured logging library. The API is designed to be easy to use, and a clear path is given for users wanting high-performance/zero-allocs by creating their handlers and making these performance improvements.</p> Fin</p> Building a CoreDNS plugin 2023-01-05T00:00:00+05:30 CoreDNS is an extensible DNS server (which is actually a fork of Caddy v1) that can be used to serve DNS records for a domain. It is written in Go and is very easy to extend. It has a plugin system that allows you to write your own plugins to extend its functionality. In this post, I will be writing a plugin for CoreDNS that will allow it to serve DNS records for Nomad services.</p> I recently came across a niche use case</a> which required me to use a resolver address for querying Nomad services. Currently Nomad native service discovery is only possible via consul-template</code> (which renders static block of address/port of services) and HTTP API. I felt adding a DNS interface would be a nice add-on.</p> Rather than writing and implementing all the boring crux of a DNS server, it’s better to extend on an existing server. CoreDNS fits in well here, it’s also used by K8s for service discovery. CoreDNS has an extensible plugin system which allows you to chain multiple different plugins for handling a request. Stuff like logs/metrics/caching comes for free with CoreDNS in form of a plugin.</p> CoreDNS docs have a handy guide</a> on how to write a plugin from scratch, so I won’t cover that again here. These are just my short notes on how I locally developed the plugin, tested and some problems I encountered during the process.</p> Developing the plugin#</a></h2> Firstly, you need to clone CoreDNS repo. Then, using the example</a> plugin provided, you can create a new repository for your own plugin.</p> To make CoreDNS aware about this plugin, you need to add it to the plugin.cfg</code> file. This file is used by the build script to generate the plugin list. The order of plugins matter here as they define how the request is handled. For example, if you want to log all the requests, you need to add the log</code> plugin before your plugin.</p> To add an external plugin this is the format used:</p> nomad:github.com/mr-karan/coredns-nomad</span></span></code></pre> However, since we are developing the plugin locally, we need to add a replace</code> directive in go.mod</code> file to point to the local plugin directory.</p> replace</span> github</span>.</span>com</span>/</span>mr</span>-</span>karan</span>/</span>coredns</span>-</span>nomad</span> =</span>></span> .</span>.</span>/</span>.</span>.</span>/</span>coredns</span>-</span>nomad</span></span></code></pre> Next, you can run make</code> in coredns</code> repository. It’ll build the binary and place it in coredns</code> directory. You can run this binary to test your plugin. To check if the plugin indeed exists in the binary, you can use the following command</p> ./coredns</span> -</span>plugins</span> \|</span> grep</span> nomad</span></span></code></pre>Handling requests#</a></h2> The ServeDNS</code> function is used to handle the DNS request by the plugin. It takes a context.Context</code> and a dns.ResponseWriter</code> as arguments. The dns.ResponseWriter</code> is used to write the response back to the client. The ServeDNS</code> function returns an int</code> which is the status code of the response. The status code is used by the next plugin in the chain to determine if it should handle the request or not.</p> Since the nomad</code> plugin expects a query in format of service.namespace.nomad</code>, it validates the query and extracts the service name and namespace from it. If the query is invalid, it returns dns.RcodeServerFailure</code> status code. If the query is valid, it queries the Nomad API for the service and returns the response.</p> func</span> (</span>n </span>Nomad</span>)</span> ServeDNS</span>(</span>ctx</span> context</span>.</span>Context</span>,</span> w</span> dns</span>.</span>ResponseWriter</span>,</span> r</span> </span>dns</span>.</span>Msg</span>)</span> (</span>int</span>,</span> error</span>)</span> {</span></span> state</span> :=</span> request</span>.</span>Request</span>{</span>W</span>:</span> w</span>,</span> Req</span>:</span> r</span>}</span></span> qname</span> :=</span> state</span>.</span>Name</span>(</span>)</span></span> qtype</span> :=</span> state</span>.</span>QType</span>(</span>)</span></span> </span> //</span> Split the query name with a `.` as the delimiter and extract namespace and service name.</span></span> //</span> If the query is not for a Nomad service, return.</span></span> qnameSplit</span> :=</span> dns</span>.</span>SplitDomainName</span>(</span>qname</span>)</span></span> if</span> len</span>(</span>qnameSplit</span>)</span> <</span> 3</span> \|\|</span> qnameSplit</span>[</span>2</span>]</span> !=</span> "</span>nomad</span>"</span> {</span></span> return</span> plugin</span>.</span>NextOrFailure</span>(</span>n</span>.</span>Name</span>(</span>)</span>,</span> n</span>.</span>Next</span>,</span> ctx</span>,</span> w</span>,</span> r</span>)</span></span> }</span></span> namespace</span> :=</span> qnameSplit</span>[</span>1</span>]</span></span> serviceName</span> :=</span> qnameSplit</span>[</span>0</span>]</span></span> </span> ...</span></span> </span></code></pre> The plugin handles A,AAAA and SRV record requests currently. Since A/AAAA records can only contain an IP address, SRV records can be used to advertise the port number.</p> //</span> Check the query type to format the appriopriate response.</span></span> switch</span> qtype</span> {</span></span> case</span> dns</span>.</span>TypeA</span>:</span></span> m</span>.</span>Answer</span> =</span> append</span>(</span>m</span>.</span>Answer</span>,</span> &</span>dns</span>.</span>A</span>{</span></span> Hdr</span>:</span> header</span>,</span></span> A</span>:</span> addr</span>,</span></span> }</span>)</span></span> case</span> dns</span>.</span>TypeAAAA</span>:</span></span> m</span>.</span>Answer</span> =</span> append</span>(</span>m</span>.</span>Answer</span>,</span> &</span>dns</span>.</span>AAAA</span>{</span></span> Hdr</span>:</span> header</span>,</span></span> AAAA</span>:</span> addr</span>,</span></span> }</span>)</span></span> </span> ...</span></span> </span></code></pre>Caching#</a></h3> While some coredns plugins have an in-built support for caching the records to avoid a lookup to Nomad server everytime (which can get expensive), I decided to skip the caching implementation. This is because coredns</code> itself has a cache</code> plugins which supports a lot of various options for controlling the cache. In my testing, just using this cache</code> plugin was sufficient to avoid Nomad lookups each time a query came in.</p> Testing the plugin#</a></h2> I created a fake HTTP test server and added the URI paths which the Nomad Go client uses to query the Nomad API. This way I could test the plugin without having to run a Nomad cluster locally.</p> //</span> Setup a fake Nomad server.</span></span> nomadServer</span> :=</span> httptest</span>.</span>NewServer</span>(</span>http</span>.</span>HandlerFunc</span>(</span>func</span>(</span>w</span> http</span>.</span>ResponseWriter</span>,</span> r</span> </span>http</span>.</span>Request</span>)</span> {</span></span> switch</span> r</span>.</span>URL</span>.</span>Path</span> {</span></span> default</span>:</span></span> t</span>.</span>Errorf</span>(</span>"</span>Not implemented: </span>%v</span>"</span>,</span> r</span>.</span>URL</span>.</span>Path</span>)</span></span> return</span></span> case</span> "</span>/v1/service/example</span>"</span>:</span></span> w</span>.</span>Write</span>(</span>[</span>]</span>byte</span>(</span>`</span>[{"Address":"1.2.3.4","Namespace":"default","Port":23202,"ServiceName":"example"}]</span>`</span>)</span>)</span></span> case</span> "</span>/v1/service/fakeipv6</span>"</span>:</span></span> w</span>.</span>Write</span>(</span>[</span>]</span>byte</span>(</span>`</span>[{"Address":"1:2:3::4","Namespace":"default","Port":8000,"ServiceName":"fakeipv6"}]</span>`</span>)</span>)</span></span> case</span> "</span>/v1/service/multi</span>"</span>:</span></span> w</span>.</span>Write</span>(</span>[</span>]</span>byte</span>(</span>`</span>[{"Address":"1.2.3.4","Namespace":"default","Port":25395,"ServiceName":"multi"},{"Address":"1.2.3.5","Namespace":"default","Port":20888,"ServiceName":"multi"},{"Address":"1.2.3.6","Namespace":"default","Port":26292,"ServiceName":"multi"}]</span>`</span>)</span>)</span></span> case</span> "</span>/v1/service/nonexistent</span>"</span>:</span></span> w</span>.</span>Write</span>(</span>[</span>]</span>byte</span>(</span>`</span>[]</span>`</span>)</span>)</span></span> }</span></span> }</span>)</span>)</span></span></code></pre>Usage Example#</a></h2> Here are some examples of how this plugin works. The Corefile I’ve used is:</p> nomad:1053 {</span></span> errors</span></span> debug</span></span> health</span></span> log</span></span> nomad {</span></span> address http://127.0.0.1:4646</span></span> ttl 10</span></span> }</span></span> prometheus :9153</span></span> cache 30</span></span> }</span></span></code></pre> On running coredns</code>, it connects to a local Nomad agent which is running at http://127.0.0.1:4646</code>. I’m running a redis</code> job in Nomad, so I can query the service using the following command:</p> nomad</span> service</span> info</span> -</span>namespace=default</span> redis</span> </span></span> Job</span> ID</span> Address</span> Tags</span> Node</span> ID</span> Alloc</span> ID</span></span> redis</span> 192.168.29.76:25395</span> [</span>] 9e02c85b 95170495</span></span> redis</span> 192.168.29.76:20888</span> [</span>] 9e02c85b a1cf923c</span></span> redis</span> 192.168.29.76:26292</span> [</span>] 9e02c85b a9d1181a</span></span></code></pre> Now, the same query can also be handled using the DNS server running by coredns</code>:</p> doggo</span> redis.default.nomad</span> @tcp://127.0.0.1:1053</span></span> NAME</span> TYPE</span> CLASS</span> TTL</span> ADDRESS</span> NAMESERVER</span> </span></span> redis.default.nomad.</span> A</span> IN</span> 10s</span> 192.168.29.76</span> 127.0.0.1:1053</span> </span></span> redis.default.nomad.</span> A</span> IN</span> 10s</span> 192.168.29.76</span> 127.0.0.1:1053</span> </span></span> redis.default.nomad.</span> A</span> IN</span> 10s</span> 192.168.29.76</span> 127.0.0.1:1053</span></span></code></pre> Quering an SRV record is also possible:</p> dig</span> +noall</span> +answer</span> +additional</span> redis.default.nomad</span> @127.0.0.1</span> -</span>p</span> 1053</span> SRV</span></span> redis.default.nomad.</span> 10</span> IN</span> SRV</span> 10</span> 10</span> 25395</span> redis.default.nomad.</span></span> redis.default.nomad.</span> 10</span> IN</span> SRV</span> 10</span> 10</span> 20888</span> redis.default.nomad.</span></span> redis.default.nomad.</span> 10</span> IN</span> SRV</span> 10</span> 10</span> 26292</span> redis.default.nomad.</span></span> redis.default.nomad.</span> 10</span> IN</span> A</span> 192.168.29.76</span></span> redis.default.nomad.</span> 10</span> IN</span> A</span> 192.168.29.76</span></span> redis.default.nomad.</span> 10</span> IN</span> A</span> 192.168.29.76</span></span></code></pre>Code#</a></h2> You can checkout the source code here</a>.</p> Fin!</p> Writing a disk-based key-value store in Golang 2022-12-18T00:00:00+05:30 I’d been mulling around reading a computer science paper and implementing a project based on it. Distributed systems, Networking and Databases are some of the things that fascinate me a lot. However, I had been looking to implement a more approachable project to avoid getting inundated initially. And I happened to chance upon the Bitcask paper</a> through Avinash’s project: CaskDB</a>.</p> After giving a quick read of this reasonably short paper, I decided to write a Golang implementation of the same, as it looked like an exciting project. If you’re interested in checking out the complete project, checkout BarrelDB</a>.</p> Bitcask is a disk-based key-value storage engine designed for fast read and write operations. It is mainly in production use by Riak (which is a distributed database) as one of the storage engines</a>. Bitcask under the hood has a straightforward yet clever design. It writes to the file in an append-only mode. This means that writes are performed only by appending to the end of the file, thus avoiding the need to perform any random disk I/O seek.</p> Let’s look at various components of Bitcask:</p> Format of the record#</a></h2> </p> CRC: Stores the checksum of the value to ensure data consistency</li> Timestamp: Timestamp in UNIX format, stored as int32.</li> Expiry: If the record has an expiry defined, then the timestamp, in UNIX format, is stored as int32.</li> Key Size: Size of the key in bytes</li> Value Size: Size of the value in bytes</li> Key</li> Value</li> </ul> This additional metadata stored alongside the key/value is represented with a fixed-width header. Each field is represented as int32</code>, so the total size of the header is 45 = 20 bytes. Here’s the code which encodes and decodes this record:</p> type</span> Record</span> struct</span> {</span></span> Header</span> Header</span></span> Key</span> string</span></span> Value</span> [</span>]</span>byte</span></span> }</span></span> </span> //</span> Header represents the fixed width fields present at the start of every record.</span></span> type</span> Header</span> struct</span> {</span></span> Checksum</span> uint32</span></span> Timestamp</span> uint32</span></span> Expiry</span> uint32</span></span> KeySize</span> uint32</span></span> ValSize</span> uint32</span></span> }</span></span> </span> //</span> Encode takes a byte buffer, encodes the value of header and writes to the buffer.</span></span> func</span> (</span>h </span></span>Header</span>)</span> encode</span>(</span>buf</span> </span>bytes</span>.</span>Buffer</span>)</span> error</span> {</span></span> return</span> binary</span>.</span>Write</span>(</span>buf</span>,</span> binary</span>.</span>LittleEndian</span>,</span> h</span>)</span></span> }</span></span> </span> //</span> Decode takes a record object decodes the binary value the buffer.</span></span> func</span> (</span>h </span></span>Header</span>)</span> decode</span>(</span>record</span> [</span>]</span>byte</span>)</span> error</span> {</span></span> return</span> binary</span>.</span>Read</span>(</span>bytes</span>.</span>NewReader</span>(</span>record</span>)</span>,</span> binary</span>.</span>LittleEndian</span>,</span> h</span>)</span></span> }</span></span></code></pre> The record is encoded in the binary format before storing it on the disk.</p> Datafile#</a></h2> A “datafile” (term used for the DB file on disk) is an append-only record of all the write operations. An instance of Bitcask can have several datafiles. However, there’s only one “active” datafile. In BarrelDB, a goroutine runs in the background at regular intervals to check if the size of the active DB file has crossed the threshold and then rotates the active file. It appends this DB file to the list of “stale” data files. All the new writes only happen to the “active” data file, and the stale files are merged as a part of the “compaction” process (described later in the post).</p> </p> Here’s how a datafile</code> is represented:</p> type</span> DataFile</span> struct</span> {</span></span> sync</span>.</span>RWMutex</span></span> </span> writer</span> </span>os</span>.</span>File</span></span> reader</span> </span>os</span>.</span>File</span></span> id</span> int</span></span> </span> offset</span> int</span></span> }</span></span></code></pre> It contains different handlers for writing and reading the file. The reason we have 2 file handlers instead of re-using the same one is that the writer</code> is only opened in an “append-only” mode. In addition, since the active file can be rotated, the writer can be set to nil</code>, ensuring no new writes ever happen on that file.</p> writer</span>,</span> err</span> :=</span> os</span>.</span>OpenFile</span>(</span>path</span>,</span> os</span>.</span>O_APPEND</span>\|</span>os</span>.</span>O_CREATE</span>\|</span>os</span>.</span>O_WRONLY</span>,</span> 0644</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> fmt</span>.</span>Errorf</span>(</span>"</span>error opening file for writing db: </span>%w</span>"</span>,</span> err</span>)</span></span> }</span></span> </span> //</span> Create a reader for reading the db file.</span></span> reader</span>,</span> err</span> :=</span> os</span>.</span>Open</span>(</span>path</span>)</span></span> if</span> err</span> !=</span> nil</span> {</span></span> return</span> nil</span>,</span> fmt</span>.</span>Errorf</span>(</span>"</span>error opening file for reading db: </span>%w</span>"</span>,</span> err</span>)</span></span> }</span></span></code></pre>KeyDir#</a></h2> In addition to storing the file on disk, Bitcask also stores additional metadata, which defines how to retrieve the record. This hashtable is a map of keys with this metadata and is referred to as KeyDir</code>. An important point to note here is that the value</code> is never</em> stored in this map. This makes it possible for Bitcask to handle datasets more significant than what the RAM can hold.</p> //</span> KeyDir represents an in-memory hash for faster lookups of the key.</span></span> //</span> Once the key is found in the map, the additional metadata, like the offset record</span></span> //</span> and the file ID is used to extract the underlying record from the disk.</span></span> //</span> Advantage is that this approach only requires a single disk seek of the db file</span></span> //</span> since the position offset (in bytes) is already stored.</span></span> type</span> KeyDir</span> map</span>[</span>string</span>]</span>Meta</span></span> </span> //</span> Meta represents some additional properties for the given key.</span></span> //</span> The actual value of the key is not stored in the in-memory hashtable.</span></span> type</span> Meta</span> struct</span> {</span></span> Timestamp</span> int</span></span> RecordSize</span> int</span></span> RecordPos</span> int</span></span> FileID</span> int</span></span> }</span></span></code></pre> Here, RecordPos</code> tells the record’s position offset (in bytes) in the entire file. Since the position of the record is stored in memory along with the key, the retrieval of the key doesn’t require more than a single</em> disk seek. Bitcask achieves really low latency even with many keys in the database. A file system read-ahead cache also helps boost the performance and comes for free - no need to design a separate caching mechanism.</p> </p> Compaction#</a></h2> As we looked at previously, a datafile is simply an append-only sequence of writes. Any modification of the key is merely a new record appended to the datafile. KeyDir overwrites the entry of the key with the new metadata, which contains the new location of the record. Thus all reads will automatically return the updated value.</p> Deletes are handled similarly by writing a “tombstone” record for the key. When the user requests the key after it’s been deleted, BarrelDB can check whether that value equals the tombstone value and return an appropriate error.</p> As you would have guessed, our database will grow unbounded if we don’t perform any garbage cleanup. The datafiles need to be pruned for deleting expired/deleted records and merging all stale files in a single active file - to keep the number of opened files in check. All of these processes are together called “Compaction”.</p> Let’s take a look at how each of these compaction routine works under the hood:</p> Merge#</a></h3> The merge process iterates over all the keys inside KeyDir and fetches their value. The value could come from a stale file as well. Once the new keys/values are updated, it writes them to a new active file. All the old file handlers are closed, and the stale files are deleted from the disk. The KeyDir is updated similarly since the new records live in a different position/file.</p> Hints File#</a></h3> Bitcask paper describes a way of creating a “hints” file initially loaded in the database for faster startup time. This file is essential to bootstrap KeyDir after a cold startup. This avoids iterating over all data files and reading their values sequentially. In BarrelDB, gob</code> encoding is used to dump the KeyDir</code> map as a gob</code> dump.</p> //</span> generateHints encodes the contents of the in-memory hashtable</span></span> //</span> as `gob` and writes the data to a hints file.</span></span> func</span> (</span>b </span></span>Barrel</span>)</span> generateHints</span>(</span>)</span> error</span> {</span></span> path</span> :=</span> filepath</span>.</span>Join</span>(</span>b</span>.</span>opts</span>.</span>dir</span>,</span> HINTS_FILE</span>)</span></span> if</span> err</span> :=</span> b</span>.</span>keydir</span>.</span>Encode</span>(</span>path</span>)</span>;</span> err</span> !=</span> nil</span> {</span></span> return</span> err</span></span> }</span></span> </span> return</span> nil</span></span> }</span></span></code></pre> During the startup, BarrelDB checks the presence of a .hints</code> file, decodes this gob dump, and loads the data in KeyDir</code>.</p> Removing expired keys#</a></h3> A goroutine runs at a configurable interval to check if the value of the key has expired. If it has, it deletes the entry from KeyDir. During the following merge process, since this entry won’t be present in KeyDir, it’ll automatically be removed when the new datafile is created.</p> To check if the key has expired, a simple check, like comparing their timestamps in UNIX epoch format, is enough: time.Now().Unix() > int64(r.Header.Expiry)</code>.</p> Redis Server#</a></h2> In addition to using BarrelDB as a Go library, I also implemented a redis-compatible server. I found tidwall/redcon</a> as an easy-to-use library to create a Redis-compatible server for Go applications. All I’d do was wrap BarrelDB API methods and define handlers for SET</code> / GET</code>.</p> I was able to use redis-cli</code> and connect to the BarrelDB server:</p> 127.0.0.1:6379</span>></span> set</span> hello</span> world</span></span> OK</span></span> 127.0.0.1:6379</span>></span> get</span> hello</span></span> "</span>world</span>"</span></span></code></pre>Benchmarks#</a></h2> You can check the repo</a> for the actual benchmarks. However, I’d like to point out some inferences of the results from redis-benchmark</code>.</p> First, let’s send 100000 requests to the server using 50 parallel clients. This command creates a unique key for each SET</code> operation.</p> redis-benchmark</span> -</span>p</span> 6379</span> -</span>c</span> 50</span> -</span>t</span> set</span> -</span>n</span> 100000</span> -</span>r</span> 100000000</span></span> </span> Summary:</span></span> throughput</span> summary:</span> 145985.41</span> requests</span> per</span> second</span></span> latency</span> summary</span> (msec</span>):</span></span> avg</span> min</span> p50</span> p95</span> p99</span> max</span></span> 0.179</span> 0.016</span> 0.183</span> 0.207</span> 0.399</span> 1.727</span></span></code></pre> So, 140k requests per second is not bad at all for a disk-based KV. But the exciting thing to note here is that the performance is predictable even if you increase the load by increasing clients:</p> redis-benchmark</span> -</span>p</span> 6379</span> -</span>c</span> 200</span> -</span>t</span> set</span> -</span>n</span> 100000</span> -</span>r</span> 100000000</span></span> </span> Summary:</span></span> throughput</span> summary:</span> 140845.08</span> requests</span> per</span> second</span></span> latency</span> summary</span> (msec</span>):</span></span> avg</span> min</span> p50</span> p95</span> p99</span> max</span></span> 0.718</span> 0.224</span> 0.711</span> 0.927</span> 1.183</span> 5.775</span></span></code></pre> If we increase the number of requests (by 5x) as well, the throughput looks almost the same:</p> redis-benchmark</span> -</span>p</span> 6379</span> -</span>c</span> 200</span> -</span>t</span> set</span> -</span>n</span> 500000</span> -</span>r</span> 100000000</span></span> </span> Summary:</span></span> throughput</span> summary:</span> 138350.86</span> requests</span> per</span> second</span></span> latency</span> summary</span> (msec</span>):</span></span> avg</span> min</span> p50</span> p95</span> p99</span> max</span></span> 0.748</span> 0.056</span> 0.711</span> 0.879</span> 1.135</span> 63.135</span></span></code></pre> This magic is all because of the way Bitcask uses Log structured hash table (just append-only records for writing data). Even with a lot of records, all it has to do is to write to the end of the file, which avoids any expensive I/O operations.</p> Summary#</a></h2> Overall, I am happy with BarrelDB</code> implementation as I covered everything described in the paper. This project had excellent learning outcomes for me. I spent a lot of time coming up with a design for structuring different components and their API methods and handling all the edge scenarios during the compaction process. Although, full credit to Bitcask as it kept its design so elegant and minimal yet achieved some significant numbers in the benchmarks. This is also a reminder that simple need not necessarily mean less powerful.</p> I look forward to implementing a distributed KV store by adding support for multiple BarrelDB nodes connected via Raft. For now, gonna enjoy some chai and release this project to the WWW :)</p> Fin!</p> Logging on Nomad with Vector 2022-09-04T00:10:55+05:30 Application orchestrators like Nomad</a>, Kubernetes etc., allow you to deploy multiple applications on the same host. Logs are stored on the underlying node wherever the applications are run. However, it’s pretty common to treat these instances as ephemeral if they’re a part of an autoscaling group. Therefore depending on the node’s availability to search these logs is not a practical idea as the node can go down anytime. Moreover, in most cases, access to these nodes is limited to cluster administrators, not the maintainers (developers/application owners). In such cases, a log shipping</em> agent must ship and aggregate logs from all cluster nodes and store them centrally (like Elasticsearch, Clickhouse, Loki).</p> I’d recommend reading this excellent post</a> by Adrian, who has explained how to set up a Vector</a> logging pipeline for applications running with docker</code> task driver and ship the logs to Loki. For the applications running using docker</code> task driver, Nomad piggybacks</a> to docker daemon for configuring logging options. Docker daemon supports many logging options; in my experience, the journald</code> log driver works reliably well.</p> However, this post is about tasks not using docker</code> but any other driver (e.g. raw_exec</code> and exec</code>). Nomad doesn’t provide many configuration options for logging for these drivers. The biggest issue is that Nomad logs the application’s stdio/stderr</code> stream to the log directory as-is</em> without annotating any metadata about the task. This means that if you’ve multiple applications running on one host, the log shipping agent will not be able to identify</em> which application’s logs are being ingested.</p> Consider this as an example. We’re running a simple web server using exec</code> driver:</p> job</span> "http"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> </span> group</span> "app"</span> {</span></span> count</span> =</span> 1</span></span> network</span> {</span></span> mode</span> =</span> "</span>bridge</span>"</span></span> port</span> "python-http"</span> {</span></span> to</span> =</span> "</span>8888</span>"</span></span> }</span></span> }</span></span> </span> task</span> "server"</span> {</span></span> driver</span> =</span> "</span>exec</span>"</span></span> </span> config</span> {</span></span> command</span> =</span> "</span>/usr/bin/python3</span>"</span></span> args</span> =</span> [</span>"</span>-m</span>"</span>,</span> "</span>http.server</span>"</span>,</span> "</span>8888</span>"</span>]</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Once the alloc is running, we can find its IP address using:</p> nomad</span> alloc</span> status</span> 1d05d64b</span> \|</span> grep</span> -</span>A</span> 3</span> '</span>Allocation Addresses</span>'</span></span> Allocation</span> Addresses</span> (mode</span> =</span> "</span>bridge</span>"</span>)</span></span> Label</span> Dynamic</span> Address</span></span> </span>python-http yes 192.168.29.76:31775 -</span>></span> 8888</span></span></code></pre> On sending an HTTP request using cURL</code> we can see the logs that this webserver generated:</p> curl</span> -</span>i</span> 192.168.29.76:31775</span></span> HTTP/1.0</span> 200</span> OK</span></span> Server:</span> SimpleHTTP/0.6</span> Python/3.10.4</span></span> Date:</span> Sun,</span> 04</span> Sep</span> 2022</span> 06:18:45</span> GMT</span></span> Content-type:</span> text/html</span>;</span> charset</span>=</span>u</span>t</span>f</span>-</span>8</span></span> Content-Length:</span> 869</span></span> .</span>.</span>.</span></span></code></pre> Nomad stores the logs inside the applications’ allocation directory, inside /opt/nomad/data/alloc</code>. To see the logs for the above allocation ID, we can use:</p> tail -f /opt/nomad/data/alloc/1d05d64b-4b59-3c65-8261-760499d9e4f6/alloc/logs/server.stderr.0tail -f server.stderr.0 </span></span> 192.168.29.76 - - [04/Sep/2022 11:48:26] "GET / HTTP/1.1" 200 -</span></span> 192.168.29.76 - - [04/Sep/2022 11:48:45] "GET / HTTP/1.1" 200 -</span></span></code></pre>Enriching logs with metadata#</a></h2> As you can see, these logs are precisely what the python3 -m http.server</code> command generates. Ideally, Nomad should have enriched these logs with metadata about the allocation ID, job name, namespace, the node it’s running on, etc., as noted in this GitHub issue</a>.</p> However, since that’s not yet available, I brainstormed a few different options:</p> Approach 1: Sidecar#</a></h3> The first approach was to run vector</code> as a sidecar next to the main task. This is the simplest option, to begin with. Vector can be independently configured to add metadata for the logs collected from the allocation directory of the group. However, as with every sidecar deployment, there’s a lot of extra resource usage. For every 10 different groups, reserving resources for 10 vector agents quickly eats up your available CPU/Memory of the underlying node. A more critical downside, though, was asking every developer to also configure a Vector sidecar job. And to keep all these configs in sync to ensure they’re unified across namespaces is also another headache. Due to these reasons, I discarded this option early on. However, suppose your deployment scale (managing applications) is relatively smaller. In that case, this is actually not a bad idea.</p> Approach 2: Events Stream#</a></h3> My next option was to listen to the Nomad events stream</a> and generate a “vector” configuration template to collect logs and enrich them with metadata from the Events Stream. I developed v0.1 of nomad-vector-logger</a> based on this concept. Since I’ve written a wrapper to collect events from Nomad using nomad-events-sink</a> it was relatively straightforward to extend it to generate a vector.toml</code> config. However, after testing in prod for a few days, I noticed that relying on the events</code> stream is unreliable. Nomad events are not WebSocket based (as of yet). It’s a simple long polling mechanism which sends events to a Go channel as and when they happen. What happens when you miss an event? What happens when you run nomad system gc</code>, which clears the events index? These were some of the challenges I faced with this v0.1 approach. There needs to be some sort of “reconcile” mechanism that periodically runs. A reconciliation loop that lists all allocations using the HTTP API can help whenever there are missing events.</p> Approach 3: Enrichment Tables#</a></h3> I also posted about the above program in Vector’s discord group (they’re super active+helpful folks) and discussed this daemon with them. They also suggested a simpler alternative: generating a CSV of running allocations instead of a .toml</code> config. Vector has support for Enrichment Tables</a> which means that it can “lookup” a CSV file to find a particular row and enrich the log event with the information found from the CSV. This seemed a super cool idea, and I developed v0.2 using this. Super thankful to Vector maintainers for giving me this idea!</p> </p> However, this approach had a few “subtle” drawbacks that I found:</p> vector</code> doesn’t support live-reloading if the CSV file changes. vector</code> has support for watching a config file for changes or sending a SIGHUP</code> to reload. However, that only works for vector’s own config files. Since the CSV file is an external file, vector cannot watch</em> it for changes. I came up with an ugly bash script hack and compared the md5</code> hash of the file in a while</code> loop and if it changed, then send a SIGHUP</code> to vector. All I can say is it’s ugly, but it works</em>. If you wish to see it, it’s available here</a> in all it’s glory.</li> The most significant issue was the chance of losing logs for the initial 10-20s of a new allocation. The above shell script had a sleep(10)</code> because md5sum</code> can be a bit CPU intensive to keep frequently doing. Vector sees a new allocation and starts ingesting events. It tries to look up the CSV row by the allocation ID, but it doesn’t find it yet</em> in the CSV file, complains about it, and drops the log event. Thus, I had to drop the CSV idea in search to find another more reliable approach to this. For people interested in this approach, you can checkout the csv</code> branch here</a>.</li> </ul> Approach 4: Periodic Reconciliation Loop#</a></h3> The final v0.3.0 solution, which IMHO fixed all the above issues, was:</p> Skip Nomad events stream. Since I have to build a reconciliation loop anyway, listening to events is just extra work without tangible benefits.</li> I used a background Goroutine channel to periodically refresh the list of allocations running on that node. Even if I fetched this list once every 30s or so, it’s OK because Vector will start ingesting logs once</em> the config gets generated. It will start reading the file from the beginning</em>. So logs aren’t lost even if I templated the config much later after the alloc began running.</li> I added the support to delay</em> the removal of allocation from the file. If an allocation is stopped (e.g., a new version is deployed or the job restarted), the program doesn’t immediately</em> removes the allocation from the config file. The user can set a delay</em> period which works like a cooling down period. In this period, one can assume</em> that Vector would have finished sending all logs to the upstream. In case the application generates too many logs faster than what the upstream sink can accept (e.g. if the upstream Elasticsearch gets slower). Suppose we remove the allocation _immediately</em> whenever it stops. In that case, there’s a probability that Vector wouldn’t have read the file to the end</em>. This cooling period helps to ensure that doesn’t happen. This is not a fool-proof situation but should cover most cases unless the upstream sink is dead for many hours.</li> </ul> How it works#</a></h3> Now that we’ve covered a few different approaches and the pros/cons of each let’s see how nomad-vector-logger</code> works. Essentially nomad-vector-logger</code> is meant to be deployed inside a Nomad cluster as a system</code></a> job. A system job in Nomad runs on each</em> node. Whenever a new node gets added to the cluster, Nomad’s scheduler schedules a copy of this program on that new node automatically. This is the equivalent of a “Daemonset” in K8s.</p> nomad-vector-logger</code> uses Nomad’s HTTP API</a> to query all nodes’ running allocations. Once it gets the list, it adds it to an internal map and signals to generate a config.</p> The final config that is templated out looks like this:</p> [</span>sources</span>.</span>source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy</span>]</span></span> type</span> =</span> "</span>file</span>"</span></span> include</span> =</span> [</span> "</span>/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy</span>"</span> ]</span></span> line_delimiter</span> =</span> "</span>\n</span>"</span></span> read_from</span> =</span> "</span>beginning</span>"</span></span> </span> [</span>transforms</span>.</span>transform_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy</span>]</span></span> type</span> =</span> "</span>remap</span>"</span></span> inputs</span> =</span> [</span>"</span>source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy</span>"</span>]</span></span> source</span> =</span> '''</span></span> # Store Nomad metadata.</span></span> .nomad.namespace = "default"</span></span> .nomad.node_name = "pop-os"</span></span> .nomad.job_name = "nginx"</span></span> .nomad.group_name = "nginx"</span></span> .nomad.task_name = "proxy"</span></span> .nomad.alloc_id = "64a2f9fd-e003-0bb3-b5cd-838125283a06"</span></span> '''</span></span></code></pre> For people unfamiliar with vector, it’s essentially doing 2 things:</p> Get logs from a “file” source</em>. The file path comes from nomad-vector-logger</code> (where all the logs for proxy</code> task are located)</li> It adds a JSON object nomad</code> with relevant keys.</li> </ul> Vector pipeline</em> will send this event to another “transformer” which can further process the log event (for eg parsing it as logfmt</code> or JSON etc) and then finally send it to an upstream sink</em> like Loki/Elasticsearch etc.</p> Here’s an example of the before/after of a log line shown above in this post:</p> Before#</a></h4> </p> After#</a></h4> </p> Perfect! We’ve annotated the same log event with Nomad metadata, and Vector will be able to identify these logs. If you’re interested in a complete setup on deploying this to Nomad, take a look at dev setup</a> which contains a Nomad jobspec to deploy nomad-vector-logger</code> as a sidecar with vector</code> as the main task.</p> Hope this post helped you start configuring a logging pipeline for applications running with non-docker task drivers.</p> Fin!</p> Don't use which 2022-08-08T00:00:00+05:30 which</code> is a non-standard/non-POSIX compliant program. I faced many issues getting which</code> to work in a chroot environment (Nomad).</p> So basically, which</code> is a simple shell script program to find out the dependency by searching the $PATH</code> (which is what makes it less deterministic). It’s also somehow symlinked 3 levels deep in Debian:</p> $</span> ls</span> -</span>laht</span> /usr/bin/which</span></span> lrwxrwxrwx</span> 1</span> root</span> root</span> 23</span> Apr</span> 26</span> 22:25</span> /usr/bin/which</span> -</span>></span> /etc/alternatives/which</span></span> $</span> ls</span> -</span>laht</span> /etc/alternatives/which</span></span> lrwxrwxrwx</span> 1</span> root</span> root</span> 26</span> Apr</span> 26</span> 22:25</span> /etc/alternatives/which</span> -</span>></span> /usr/bin/which.debianutils</span></span></code></pre> Now, notice this:</p> $</span> which</span> ls</span></span> ls:</span> aliased</span> to</span> ls</span> -</span>-color=tty</span></span> </span> $</span> /usr/bin/which.debianutils</span> ls</span></span> /usr/bin/ls</span></span></code></pre> Both are the same programs. However, why is the output different? This is because which</code> is apparently a shell built-in zsh</code>, that is why:</p> #</span> zsh</span></span> which</span> which</span></span> which</span>:</span> shell</span> built-in</span> command</span></span> </span> #</span> bash</span></span> which</span> which</span></span> /usr/bin/which</span></span></code></pre> The inconsistency happens because zsh</code> treats which</code> as a shell built-in when it’s apparently not one.</p> This</a> article has some more details on why which</code> is bad</em> and how the Debian team is slowly deprecating it from being a part of debianutils anymore.</p> When I invoked which</code> from inside a Nomad chroot, it complained that I didn’t have /bin/sh</code> (because I was using a custom chroot mount). I started looking hard for alternatives because this silly utility already wasted too much of my time.</p> What to use#</a></h2> Use command -v</code>. It’s a shell built-in, so it avoids a dependency on an external binary (unlike which</code>).</p> Usage example:#</a></h3> if</span> !</span> command</span> -</span>v</span> aws</span> ></span> /dev/null</span>;</span> then</span></span> echo</span> "</span>Can't find 'aws' executable. Aborted.</span>"</span></span> exit</span> 1</span></span> fi</span></span></code></pre>References#</a></h2> I found the following posts while digging which</code> and its alternatives.</p> which-not-posix</a></li> why-not-use-which-what-to-use-then</a></li> </ul> Fin</p> Attending IndiaFOSS 2022 2022-08-07T00:00:00+05:30 On June 28, I received a text from Anand</a> nudging me to submit a CFP for IndiaFOSS</a>. Until then, I’d not planned to attend IndiaFOSS because travelling to participate in a conference didn’t seem enticing. However, the thought of visiting Bangalore (a city where I spent five years before Covid took the joy from our lives) stayed in my mind for the rest of the day. Call it serendipity or just that little external motivation by Anand; I decided to submit a proposal</a> on Self Hosting with Nomad. IndiaFOSS would be the first physical conference I would attend after the pandemic. I love attending conferences because you get to meet and interact with many people (I somehow dislike the term “networking with people” because it’s heavily connoted with marketing/sales pitches).</p> IndiaFOSS is a v2 of a small pilot conference IndiaOS</a> that happened in 2020. Around 650 people attended IndiaFOSS, and the execution scale was much bigger this time. IndiaFOSS is a 100% volunteer-driven conference, yet everything was quite professionally managed. My friend Dhiraj</a> travelled from Chennai to attend the conference, and I hung out with him for most of the next two days.</p> Filling our tummies with a scrumptious breakfast of Thatte Idlis</em>, we headed to attend the keynote by Rushabh</a>, who gave an excellent overview of FOSS United and its journey so far. There were a series of exciting talks right after - especially by Kovid</a>, a prolific FOSS contributor from India. Kovid’s software is widely used across the globe. He shared the story of how calibre</a> evolved from a bunch of scripts written to get around Sony’s DRM to become the most sought-after ebook management tool that it is now. I’ve myself used calibre in one of my initial projects</a>(back when I was still learning Python, please don’t judge that codebase). Next was the much-awaited interview of Rudra Saraswat</a> - the 12-year-old whiz kid</em> who’s a primary contributor to Ubuntu Unity Remix and many other projects. The entire auditorium was stunned to witness how this bright young kid manages his studies, extra-curricular and FOSS projects with a laser-sharp focus.</p> I spent some time reviewing my slides which were scheduled right after an excellent talk on hpw the Kubernetes release team works - by Nabarun</a>. I was pleasantly surprised to see many people interested in Self Hosting/Nomad. You can check out the slides here</a>.</p> It was time for lunch, and I finally got to meet Vinayak</a>, Bibhas</a>, Gaurav</a> and Raghav</a> with whom I’ve been interacting in a small Discord group since so many months! This is what pandemic years took away from us, sigh!</p> After lunch, Kailash</a> presented his fantastic talk on languages, dictionaries, and his FOSS project - dictpress</a>. Kailash was anticipating that people would be drowsy after lunch, but I doubt anyone would want to miss this talk. He gave a glimpse of the monumental work done by Datuk KJ Joseph and V. Krishna in the field of dictionaries and how dictpress makes it easier for anyone to host a dictionary website for any language.</p> The rest of the day was spent hanging out in the lobby, meeting fellow hackers</em>, and chit-chatting. I consider myself an introvert, but I guess with developers, I tend to speak up openly because we tend to ignore any small-talks :P. Rainy evenings and a weekend meant all the Peak Bengaluru</em> memes about not finding cabs were experienced first-hand.</p> Day 2 had quite a good lineup of talks as well. Krutika</a> shared the designer’s dilemma of contributing to FOSS Projects. Nemo</a> talked about how we could get a FOSS UPI app and touched upon the openness of UPI. We even did an open space around Self Hosting where many of us self-hosted enthusiasts came together and discussed different topics/problems we encountered while self-hosting. It was my first time interacting with Abhas</a>, who runs Deeproot Linux</a>. He shared his wisdom and experiences from self-hosting email servers, media servers and just about anything. I was pretty motivated by his dedication.</p> The best part about IndiaFOSS was that this wasn’t a conference meant just for techies. The talks highlighted every domain of FOSS- policy, design, languages, and hardware rather than just keeping it limited to software. I was happy to see a conference not peddled with any “corporate” product talks. This is what all FOSS conferences should look like, I guess?</p> I look forward to attending (and hopefully volunteering) next year’s conference!</p> </p> Fin</p> Debugging DNS Issues in Nomad 2022-07-14T00:00:00+05:30 At work, my colleagues and I stumbled upon a hair-pulling networking issue involving a specific problem when connecting to a Kafka cluster. We use the franz-go</a> library in our Golang applications to interact with an external Kafka cluster. These Go apps are hosted inside a Nomad</a> cluster and running with the exec</a> driver.</p> The Issue#</a></h2> Solving issues where a specific condition happens only</em> sometimes is terribly difficult to debug because you need to reproduce the bug in a controlled environment. Our issue was that writing the first</strong> message to a Kafka topic took unusually high time (>5 seconds) while the writes of subsequent messages were instant. The following messages were instant for the next 30s, and then the write again took >5s.</p> The request flow looks something like this:</p> Go app in Nomad cluster (bridge mode) -> Kafka node in a Kafka cluster</em></p> Both apps are inside the same VPC running on AWS EC2 instances. The Nomad task is running with network.mode=bridge</code>, which means that there are some iptables</code> rules configured to do SNAT/DNAT translation to forward packets from the default bridge network (nomad</code>) to the default ethernet interface (ens5</code>).</p> We have a couple of other Nomad clusters in our environment and regularly use external EC2 instances to communicate, and we’ve never observed any slowdowns in our existing applications. This behaviour seemed something specific to Kafka. However, we discovered that we could not reproduce the high wait time issue in Kafka when the message was sent from a task running as host</em> mode. So now, we had two conflicting things which made this issue strange:</p> Issue happens in bridge</code> mode, not in host</code> mode.</li> Issue happens only with Kafka nodes, not with any external services - even in bridge mode.</li> </ul> We spent a lot of time dissecting our apps, turning on TRACE</code> level logging for Kafka clusters and enabling debug mode in franz-go</code>. One of the significant challenges for us was that even an idle Kafka cluster could be super chatty</strong>, producing ~1Mn records for a few minutes when run with TRACE</code>. Cutting through the noise and finding the exact point where the slowdown happened turned out to be more difficult than expected.</p> However, with the debug logs in franz-go</code>, we’d arrived at a breakthrough which helped us narrow down the issue. We saw the write_wait</code> time to be ~5s in the logs emitted</a> by franz-go</code>. The subsequent messages had write_wait</code> as low as a few microseconds. What was puzzling was why Kafka took so much time to wait before writing the bytes to the underlying socket.</p> We forked the franz-go library and added a bunch of our custom logs to figure out where and why exactly the slowdown happens. One issue in the logs emitted by franz-go was that no timestamps were attached to the logger. We added that and deployed the binaries with the patched version. This time we immediately found the logs, which pointed that it took ~5s from the time it initiated the connection</a> to the Kafka broker node and the time it was able to connect to the node</a>.</p> The Fix#</a></h2> The node’s address was an internal hostname kafka-abc.private-zone.internal</code>. We postulated that it could be a DNS resolver issue. We did a dig kafka-abc.private-zone.internal</code> and instantly got the record. Maybe it’s cached? We decided to verify the /etc/resolv.conf</code> till we waited for the TTL of the record to expire. Opening this seemingly innocent /etc/resolv.conf</code> revealed what the issue was DNS indeed.</p> nameserver bad</span></span> nameserver good</span></span></code></pre> We had an unreachable</em> nameserver address in the nameserver</code> list. The first message has ~5s write timeout because /etc/resolv.conf</code> has a 5s default timeout in case the nameserver is unreachable. Go’s DNS resolver picked up the second resolver, which cached the DNS records until the record’s TTL (30 seconds). Subsequent Kafka write messages on the topic worked without any issues in the TTL window. When the DNS record expires, rinse, and repeat. We later found the source of the bad nameserver came from our Nomad client initialising script.</p> Takeaways#</a></h2> We use a custom Nomad client initialising script in our Nomad clusters to populate the chroot_env</code>. Since, by default, the chroot</a> provided by Nomad copies the/usr</code>/ directory, we found that it increased the initial startup time of the alloc. It made sense to customise the chroot_env</code> with the list of binaries and config files we would need.</p> One of the configs happens to be /etc/resolv.conf</code>, which DNS resolvers use to resolve queries. On the host, we have systemd-resolve</code> running and /etc/resolv.conf</code> is configured with the stub resolver address. However, since that address (127.0.0.53</code>) is unreachable by the bridge</em> network in Nomad, we mount our custom config, which looks like:</p> nameserver 10.100.0.2</span></span> options edns0 trust-ad</span></span> search ap-south-1.compute.internal</span></span></code></pre> The nameserver</code> represents the AWS R53 resolver running in all VPCs if configured with enableDnsSupport</a> in VPC settings.</p> The above config ensures that all DNS queries for the tasks inside the bridge network go directly to the AWS R53 resolver. The resolver can do DNS Lookups for the private zones associated with the R53 in that VPC and forward all other hostnames to their own upstream DNS resolver.</p> Fun Fact#</a></h3> If you don’t mount a custom /etc/resolv.conf</code>, then DNS resolution is broken by default in any Nomad exec task. You can quickly reproduce it with this task definition:</p> job</span> "sleep"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> group</span> "sleep"</span> {</span></span> count</span> =</span> 1</span></span> task</span> "sleep"</span> {</span></span> driver</span> =</span> "</span>exec</span>"</span></span> config</span> {</span></span> command</span> =</span> "</span>bash</span>"</span></span> args</span> =</span> [</span>"</span>-c</span>"</span>,</span> "</span>sleep infinity</span>"</span>]</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Run a Nomad agent in -dev</code> mode:</p> nomad</span> agent</span> -</span>dev</span></span> nomad</span> run</span> sleep.nomad</span></span></code></pre> When you exec inside the alloc:</p> $</span> nomad</span> alloc</span> exec</span> -</span>i</span> -</span>t</span> -</span>task</span> sleep</span> 8b4a0a82</span> /bin/sh</span></span> $</span> cat</span> /etc/resolv.conf</span></span> nameserver</span> 127.0.0.1</span></span> $</span> dig</span> mrkaran.dev</span></span> ;</span>;</span> communications</span> error</span> to</span> 127.0.0.1#53:</span> connection</span> refused</span></span></code></pre> I believe Nomad should bootstrap some DNS resolver or a relevant iptables</code> rule for the exec tasks so that DNS can be resolved by default without the need to mount a custom config. For comparison, a docker</code> also bootstraps the container with a customisable /etc/resolv.conf</code>, and the settings can be specified either at runtime and fallback to the global settings in /etc/docker/daemon.json</code>.</p> I hope this post helps someone solve these weird DNS issues when running tasks with the bridge network and exec task driver in the Nomad cluster.</p> Like they say:</p> </p> Fin!</p> Understanding Networking in Nomad 2022-05-14T00:10:55+05:30 Nomad</a> is a general-purpose cluster orchestrator and scheduler. Up until Nomad 1.3</a> was released, it had no native support for discovering other applications running in the cluster. This is sort of a very elementary requirement when scheduling tasks in a cluster. Nomad relies on Consul to discover other “services” and has first class support for registering and fetching service records which makes things easier. Consul provides the records via various mechanisms such as a REST API, DNS and Consul Templates which render the exact IP/Port of the service in a Go template that can be injected into your application.</p> I’ve been using Nomad since quite some time (both at work and for my self hosted instance) however I’ve often tripped when it comes to Networking. Nomad has a lot of simple concepts and it all “clicks” once you understand and recognise various patterns that can be used to connect the applications. A major learning curve for someone new to Nomad and trying to integrate Consul is that the person now has to first understand how Consul works, deploy a Consul cluster and this creates a lot of friction amongst newcomers to Nomad. Nomad 1.3 solves a part of this issue (i.e. no need to run Consul for basic service discovery) and is a great fit for just getting started with Nomad based networking. However, in this post I’d like to go through all the different networking patterns I’ve known or used in production and make an attempt at simplifying these concepts for Nomad beginners.</p> I'll be running a single node Nomad on my dev machine. The instructions to do that can be seen here</a>. </p> Scenario 1: Expose an application on the host#</a></h2> </p> We’ll start off with the simplest usecase: You have a redis</code> container and you want to expose that to the host. The docker run</code> equivalent to what we wanna do is:</p> docker</span> run</span> -</span>-rm</span> -</span>p=6379</span> redis:7</span></span></code></pre> This command exposes a dynamic port on your host. To see what exactly is the port number, you can do docker ps</code> and find out an output similar to 0.0.0.0:49153->6379/tcp</code> under PORTS</code>.</p> To verify, I can use redis-cli</code> to connect to this host:</p> $</span> redis-cli</span> -</span>p</span> 49153</span> </span></span> 127.0.0.1:49153</span>></span> ping</span></span> PONG</span></span></code></pre> Fantastic! Now, let’s stop this container and see how we can do the same in Nomad.</p> job</span> "redis"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> </span> group</span> "redis"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> }</span></span> }</span></span> </span> task</span> "redis"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> config</span> {</span></span> image</span> =</span> "</span>redis:7</span>"</span></span> ports</span> =</span> [</span>"</span>redis</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 256</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> This is a barebones definition of how to run Redis on Nomad. We’re not dealing with any volume mounts, custom config etc here - the idea is to only learn networking concepts in Nomad!</p> Save the above file as job.nomad</code> and deploy to the cluster with:</p> nomad</span> run</span> job.nomad</span></span></code></pre> Within a few lines of config, we have a Docker container running, which exposes a dynamic port 23522</code>.</p> </p> We can connect to it via redis-cli</code> on our host:</p> $</span> redis-cli</span> -</span>p</span> 23522</span></span> 127.0.0.1:23522</span>></span> ping</span></span> PONG</span></span></code></pre> NOTE: It’s important to have ports</code> in your task.config</code> section. Nomad passes this information to the docker</code> daemon running on the host. So unless you specify which ports to advertise in the container, it won’t know whether to expose 6379 or not.</p> This can be easily verified with docker inspect</code>:</p> #</span> without `ports`</span></span> $</span> docker</span> inspect</span> fc32a4ffd148</span> -</span>f</span> "</span>{{json .NetworkSettings.Ports }}</span>"</span> \|</span> jq</span> '</span>.</span>'</span></span> {</span></span> "</span>6379/tcp</span>"</span>:</span> null</span></span> }</span></span> </span> #</span> with `ports`</span></span> $</span> docker</span> inspect</span> 0421101d0228</span> -</span>f</span> "</span>{{json .NetworkSettings.Ports }}</span>"</span> \|</span> jq</span> '</span>.</span>'</span></span> {</span></span> "</span>6379/tcp</span>"</span>:</span> [</span></span> {</span></span> "</span>HostIp</span>"</span>:</span> "</span>127.0.0.1</span>"</span>,</span></span> "</span>HostPort</span>"</span>:</span> "</span>31930</span>"</span></span> }</span></span> ],</span></span> "</span>6379/udp</span>"</span>:</span> [</span></span> {</span></span> "</span>HostIp</span>"</span>:</span> "</span>127.0.0.1</span>"</span>,</span></span> "</span>HostPort</span>"</span>:</span> "</span>31930</span>"</span></span> }</span></span> ]</span></span> }</span></span></code></pre>Exposing Static ports#</a></h3> A less common scenario is to bind an application to a static port on the host. We’ll cover an example of when you want to do that a bit later, but this is generally not widely used because in a cluster your application can “float” around and the idea of sticking to a port is not useful. However, there’s a way for us to do that by simply adding a static</code> line in our port</code> block:</p> network</span> {</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> static</span> =</span> 6379</span></span> }</span></span> }</span></span></code></pre> </p> When we deploy the same file again, we can see the port allocation has changed from dynamic to the static port we assigned. It’s your job to ensure no other applications are listening on that same interface and port because that’s bound to cause conflicts.</p> Scenario 2: Communicate to Redis within the same group#</a></h2> For this scenario, we are going to assume there’s a Go application that needs to talk to Redis. However, in this scenario, the Redis is sort of like an ephemeral cache, so it’s okay to deploy both of them in the same Task Group</em>.</p> If you don't know the difference between a Task and Group, here's a very primitive explanation but please read the docs for more clarity. A Group can contain multiple tasks. What's important to know here is that a group will always have its own shared network namespace. This means, that if you have 2 tasks in the group, they both will have access to the same network namespace. This allows both tasks to talk to each other on the same network interface. </p> job</span> "hello"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> </span> group</span> "app"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "app"</span> {</span></span> to</span> =</span> 8080</span></span> static</span> =</span> 8080</span></span> }</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> static</span> =</span> 6379</span></span> }</span></span> }</span></span> </span> task</span> "redis"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> config</span> {</span></span> network_mode</span> =</span> "</span>host</span>"</span></span> image</span> =</span> "</span>redis:7</span>"</span></span> ports</span> =</span> [</span>"</span>redis</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 256</span></span> }</span></span> }</span></span> </span> </span> task</span> "app"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> env</span> {</span></span> DEMO_REDIS_ADDR</span> =</span> "</span>${</span>NOMAD_ADDR_redis</span>}</span>"</span></span> }</span></span> </span> config</span> {</span></span> network_mode</span> =</span> "</span>host</span>"</span></span> image</span> =</span> "</span>mrkaran/hello-app:1.0.0</span>"</span></span> ports</span> =</span> [</span>"</span>app</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 512</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Key Points</strong>:</p> You can see we have defined task</code> app and task redis</code> under the same group, app</code>. This means that Nomad will co-locate both of these tasks on the same client (because they tend to share not just the same network namespace but a common allocation directory as well - which makes it super easy to share files across tasks).</li> We are using NOMAD_ADDR_redis</code> to get the IP:Port combination for the redis</code> task. This gets injected at runtime by Nomad. You can find a list of runtime variables here</a>.</li> This is ideal for quick tests/dev setup where you don’t want the overhang of Service Discovery etc and want to connect to your applications in the least friction possible.</li> </ul> The above config is suitable if you’re migrating from docker-compose</code> based environments, you can use this template for your services. The biggest limitation of this approach is that it’s using a host network so it’s not possible to set up any kind of Access Controls on it. This effectively means that nothing prevents any other application on the cluster to talk to these ports.</p> Scenario 3: Communicate across different groups#</a></h2> Task Groups are useful if you have related</em> tasks (like the init task where you wanna fetch the files before the task starts). But a drawback of using group</code> is that you can’t scale the tasks independently. In the above example, we placed Redis and App in the same group, but that means if you increase count</code> of same group to scale the app, you end up scaling Redis containers too. This is undesirable as Redis may not to scale proportionally to app.</p> The way to do create multiple groups is to split the tasks into their own individual groups:</p> job</span> "hello"</span> {</span></span> datacenters</span> =</span> [</span>"</span>dc1</span>"</span>]</span></span> type</span> =</span> "</span>service</span>"</span></span> </span> group</span> "app"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "app"</span> {</span></span> to</span> =</span> 8080</span></span> static</span> =</span> 8080</span></span> }</span></span> }</span></span> </span> task</span> "app"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> env</span> {</span></span> DEMO_REDIS_ADDR</span> =</span> "</span>localhost:6379</span>"</span></span> }</span></span> </span> config</span> {</span></span> image</span> =</span> "</span>mrkaran/hello-app:1.0.0</span>"</span></span> ports</span> =</span> [</span>"</span>app</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 512</span></span> }</span></span> }</span></span> }</span></span> </span> group</span> "redis"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> static</span> =</span> 6379</span></span> }</span></span> }</span></span> </span> task</span> "redis"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> config</span> {</span></span> image</span> =</span> "</span>redis:7</span>"</span></span> ports</span> =</span> [</span>"</span>redis</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 256</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> When you submit this job, you get 2 allocation IDs (each group creates one alloc</code>). The key point here is that both of these groups have their own network namespace. So, we don’t really have any way to reach the other application (we can’t really rely on the host network, because there’s no guarantee that both of these groups will be deployed on the same node).</p> In the previous example, we saw how Nomad exposes runtime variables that contained information about all tasks in other groups. But now since the groups are separate, the app</code> container has no idea about redis</code> (or vice-versa):</p> env</span> \|</span> grep</span> NOMAD</span></span> NOMAD_REGION</span>=</span>g</span>l</span>o</span>b</span>a</span>l</span></span> NOMAD_CPU_LIMIT</span>=</span>4</span>7</span>0</span>0</span></span> NOMAD_IP_app</span>=</span>1</span>2</span>7</span>.</span>0</span>.</span>0</span>.</span>1</span></span> NOMAD_JOB_ID</span>=</span>h</span>e</span>l</span>l</span>o</span></span> NOMAD_TASK_NAME</span>=</span>a</span>p</span>p</span></span> NOMAD_SECRETS_DIR</span>=</span>/</span>s</span>e</span>c</span>r</span>e</span>t</span>s</span></span> NOMAD_CPU_CORES</span>=</span>1</span></span> NOMAD_NAMESPACE</span>=</span>d</span>e</span>f</span>a</span>u</span>l</span>t</span></span> NOMAD_ALLOC_INDEX</span>=</span>0</span></span> NOMAD_ALLOC_DIR</span>=</span>/</span>a</span>l</span>l</span>o</span>c</span></span> NOMAD_JOB_NAME</span>=</span>h</span>e</span>l</span>l</span>o</span></span> NOMAD_HOST_IP_app</span>=</span>1</span>2</span>7</span>.</span>0</span>.</span>0</span>.</span>1</span></span> NOMAD_SHORT_ALLOC_ID</span>=</span>a</span>9</span>d</span>a</span>7</span>2</span>d</span>c</span></span> NOMAD_DC</span>=</span>d</span>c</span>1</span></span> NOMAD_ALLOC_NAME</span>=</span>h</span>e</span>l</span>l</span>o</span>.</span>a</span>p</span>p</span>[</span>0</span>]</span></span> NOMAD_PORT_app</span>=</span>8</span>0</span>8</span>0</span></span> NOMAD_GROUP_NAME</span>=</span>a</span>p</span>p</span></span> NOMAD_PARENT_CGROUP</span>=</span>n</span>o</span>m</span>a</span>d</span>.</span>s</span>l</span>i</span>c</span>e</span></span> NOMAD_TASK_DIR</span>=</span>/</span>l</span>o</span>c</span>a</span>l</span></span> NOMAD_HOST_PORT_app</span>=</span>8</span>0</span>8</span>0</span></span> NOMAD_MEMORY_LIMIT</span>=</span>5</span>1</span>2</span></span> NOMAD_ADDR_app</span>=</span>1</span>2</span>7</span>.</span>0</span>.</span>0</span>.</span>1</span>:</span>8</span>0</span>8</span>0</span></span> NOMAD_ALLOC_PORT_app</span>=</span>8</span>0</span>8</span>0</span></span> NOMAD_ALLOC_ID</span>=</span>a</span>9</span>d</span>a</span>7</span>2</span>d</span>c</span>-</span>9</span>4</span>f</span>c</span>-</span>6</span>3</span>1</span>5</span>-</span>b</span>b</span>3</span>7</span>-</span>6</span>3</span>c</span>b</span>e</span>e</span>f</span>1</span>5</span>3</span>b</span>9</span></span> NOMAD_HOST_ADDR_app</span>=</span>1</span>2</span>7</span>.</span>0</span>.</span>0</span>.</span>1</span>:</span>8</span>0</span>8</span>0</span></span></code></pre>Service Discovery#</a></h3> This is where things get interesting. The app</code> group needs to discover</em> redis</code> before connecting to it. There are multiple ways to do that, but we’ll cover 2 standard ways which are more common.</p> Using Nomad native service discovery#</a></h3> </p> This is a feature launched in Nomad 1.3. Up until this release, Nomad had to rely on Consul for this. But with native service discovery built in Nomad, things are much simpler. Let’s make the following changes to our job file. In each group, we’ll add a service</code> definition:</p> group</span> "app"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "app"</span> {</span></span> to</span> =</span> 8080</span></span> }</span></span> }</span></span> </span> service</span> {</span></span> name</span> =</span> "</span>app</span>"</span></span> provider</span> =</span> "</span>nomad</span>"</span></span> port</span> =</span> "</span>app</span>"</span></span> }</span></span> //</span> task is the same</span></span> }</span></span> </span> group</span> "redis"</span> {</span></span> count</span> =</span> 1</span></span> </span> network</span> {</span></span> mode</span> =</span> "</span>host</span>"</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> }</span></span> }</span></span> </span> service</span> {</span></span> name</span> =</span> "</span>redis</span>"</span></span> provider</span> =</span> "</span>nomad</span>"</span></span> port</span> =</span> "</span>redis</span>"</span></span> }</span></span> //</span> task is the same</span></span> }</span></span></code></pre> So, we added a new service</code> block and got rid of static</code> ports. Well, there’s no need to bind to static ports when we’re using service discovery.</p> After submitting the job, we can use the nomad service list</code> command to ensure the services are registered with Nomad.</p> nomad</span> service</span> list</span> </span></span> Service</span> Name</span> Tags</span></span> app</span> [</span>]</span></span> redis</span> [</span>]</span></span></code></pre> To find out details about a particular service, we can use nomad service info</code>:</p> $</span> nomad</span> service</span> info</span> app</span> </span></span> Job</span> ID</span> Address</span> Tags</span> Node</span> ID</span> Alloc</span> ID</span></span> hello</span> 127.0.0.1:29948</span> [</span>] d92224a5 5f2ac51f</span></span> $</span> nomad</span> service</span> info</span> redis</span></span> Job</span> ID</span> Address</span> Tags</span> Node</span> ID</span> Alloc</span> ID</span></span> hello</span> 127.0.0.1:22300</span> [</span>] d92224a5 8078c9a6</span></span></code></pre> Perfect! We can see the dynamic port assignment in each of the services. To use this config in our app, we will template it:</p> task</span> "app"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> template</span> {</span></span> data</span> =</span> <<</span>EOH</span></span> {{ range nomadService "redis" }}</span></span> DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}</span></span> {{ end }}</span></span> EOH</span></span> </span> destination</span> =</span> "</span>secrets/config.env</span>"</span></span> env</span> =</span> true</span></span> }</span></span> </span> config</span> {</span></span> image</span> =</span> "</span>mrkaran/hello-app:1.0.0</span>"</span></span> ports</span> =</span> [</span>"</span>app</span>"</span>]</span></span> }</span></span> </span> resources</span> {</span></span> cores</span> =</span> 1</span></span> memory</span> =</span> 512</span></span> }</span></span> }</span></span></code></pre> We added the template</code> stanza which will interpolate the env variables in the container. We loop over nomadService</code> and get the address and port of the redis</code> service. This makes it convenient for tasks on other nodes to discover each other.</p> Using Consul Service Discovery#</a></h3> </p> Just by tweaking provider</code> in our service</code> block, we can use the Consul agent for service discovery.</p> service</span> {</span></span> name</span> =</span> "</span>app</span>"</span></span> provider</span> =</span> "</span>consul</span>"</span></span> port</span> =</span> "</span>app</span>"</span></span> }</span></span> </span> </span> task</span> "app"</span> {</span></span> driver</span> =</span> "</span>docker</span>"</span></span> </span> template</span> {</span></span> data</span> =</span> <<</span>EOH</span></span> {{ range service "redis" }}</span></span> DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}</span></span> {{ end }}</span></span> EOH</span></span></code></pre> Ensure that you're running Consul and have connected Nomad to it. Please refer to docs</a> for the same. </p> Since now we are using consul</code> for registering services, we have to loop over service</code> instead of nomadService</code>. The rest of the things remain pretty much the same. I really like how with just 2 lines of code you can switch between Nomad/Consul for discovering services.</p> Now, of course, there are certain advantages</strong> to using Consul:</p> You can query the address of the service with DNS:</li> </ul> doggo</span> redis.service.consul</span> @tcp://127.0.0.1:8600</span></span> NAME</span> TYPE</span> CLASS</span> TTL</span> ADDRESS</span> NAMESERVER</span> </span></span> redis.service.consul.</span> A</span> IN</span> 0s</span> 172.20.10.3</span> 127.0.0.1:8600</span> </span></span></code></pre> Define health checks. Since it’s a new feature, health checks on Nomad service aren’t there but there’s a GitHub issue</a> open for the same.</li> </ul> Update (2024)</strong>: Native health checks are now available in Nomad since version 1.4 (released October 2022). This means you can define health checks directly in your Nomad service definitions without requiring Consul. You can read more about this in the Nomad documentation</a>.</p> </blockquote> Accessible by applications outside Nomad. In case consul</code> is used by other applications outside of the Nomad cluster, they can still get their address (using DNS or REST APIs)</li> </ul> However, Nomad native service discovery is perfect for local setups and even smaller use-cases in production because it eliminates the need of running Consul in your stack which is a big thing!</p> Scenario 4: Restricting access to certain namespaces#</a></h2> </p> In all the above scenarios, we found that the service gets exposed to the local Nomad client. In case you’re running multiple namespaces on your cluster, you’d like to not expose them at all. In addition, you may want to express fine-grained controls on which application can access a particular service. All of this is possible via a Service Mesh</strong>. Nomad provides a way to have a “service mesh” via Consul Connect. Consul Connect can do mTLS and service authorization. Under the hood, it’s an Envoy proxy that runs alongside your app (sidecar is a fancy way to say that). The consul</code> agent configures an Envoy configuration for you so it’s all pretty seamless.</p> For this to work, the first thing we need is a bridge</code> network mode. This network model is actually a CNI plugin and needs to be installed separately in /opt/cni/bin</code>. Follow the steps mentioned here</a>.</p> network</span> {</span></span> mode</span> =</span> "</span>bridge</span>"</span></span> port</span> "redis"</span> {</span></span> to</span> =</span> 6379</span></span> }</span></span> }</span></span></code></pre> The service in redis</code> is called as a Consul Connect Ingress</em>:</p> service</span> {</span></span> name</span> =</span> "</span>redis</span>"</span></span> provider</span> =</span> "</span>consul</span>"</span></span> port</span> =</span> "</span>6379</span>"</span></span> connect</span> {</span></span> sidecar_service</span> {</span>}</span></span> }</span></span> }</span></span></code></pre> It’s an empty block because we don’t need to define any upstream here. The rest of the values will be default values.</p> Next, we create a service for our app</code> and that is a Consul Connect Egress</em>:</p> service</span> {</span></span> name</span> =</span> "</span>app</span>"</span></span> provider</span> =</span> "</span>consul</span>"</span></span> port</span> =</span> "</span>app</span>"</span></span> connect</span> {</span></span> sidecar_service</span> {</span></span> proxy</span> {</span></span> upstreams</span> {</span></span> destination_name</span> =</span> "</span>redis</span>"</span></span> local_bind_port</span> =</span> 6379</span></span> }</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Here we define an upstream for redis</code>. If you notice closely, we are using a port number in Consul Connect Ingress</em>. For some reason, if you use a named port instead of 6379 it doesn’t work. I am not entirely sure if it’s a bug or it’s intended to work like this.</p> So here, when the app</code> wants to talk to redis, it talks to localhost:6379</code> which is the local port that the Envoy sidecar is listening to. We can verify that using netstat</code>:</p> $</span> netstat</span> -</span>tulpvn</span></span> Active</span> Internet</span> connections</span> (only</span> servers</span>)</span></span> Proto</span> Recv-Q</span> Send-Q</span> Local</span> Address</span> Foreign</span> Address</span> State</span> PID/Program</span> name</span> </span></span> tcp</span> 0</span> 0</span> 127.0.0.2:19001</span> 0.0.0.0:</span></span> LISTEN</span> -</span> </span></span> tcp</span> 0</span> 0</span> 0.0.0.0:23237</span> 0.0.0.0:</span></span> LISTEN</span> -</span> </span></span> tcp</span> 0</span> 0</span> 127.0.0.1:6379</span> 0.0.0.0:</span></span> LISTEN</span> -</span> </span></span> tcp6</span> 0</span> 0</span> :::8080</span> :::</span></span> LISTEN</span> 1/./hello.bin</span></span></code></pre> The traffic is sent from this port to the other Envoy proxy on a port that it advertises (and Consul automagically configured). That Envoy proxy further sends the traffic to the redis</code> container on port 6379. The proxied traffic is securley encrypted via mTLS and authorized (via Consul Intentions - not covered in this post).</p> Scenario 5: Exposing services to end-user#</a></h2> </p> In the first scenario, we discussed using static ports. Well, it turns out it’s super helpful if you want to define a Traffic Ingress service. Unlike K8s, Nomad doesn’t have any Ingress Controllers, so the best way is to deploy these web proxies on each node as a system job</a> (which means it’s ensured to run on every client node) and bind them to a static port (say 443/80). Then, configure your Load Balancers and register all the Nomad nodes as target IPs and their ports would be the static port you define. These Ingress proxies (like HAProxy/Nginx) can then be used to talk to your application via any of the patterns we’ve mentioned above.</p> Typically, you’d want to use a “Host-Based” routing pattern for your ingress proxy to make a routing decision.</p> For eg, in case you have an a.example.org</code> DNS record pointing to an ALB. Now, when the request comes to the ALB, it forwards to any one of the NGINX/HAProxy. For HAProxy to correctly route the traffic to `a service, you can use a “Host” header.</p> Summary#</a></h2> These were some of the common networking patterns that I’m aware of. Since some of these concepts are not really straightforward I hope the explanation helped in bringing some clarity.</p> There’s much more to this topic like Consul Gateways and multiple kind of CNIs which tweak how networking happens in the cluster but those are some really advanced topics that are out of the scope for this post.</p> Fin!</p> How I expose services while self hosting 2022-02-10T00:00:00+00:00 I’ve often been asked how to expose public and private services running on DigitalOcean droplets/RPis when self hosting apps. Most people don’t have access to a static IP for their RPis. I felt I’d summarize my approach to this in this blog post and hope it’ll be useful for others trying to do the same.</p> Tools I use#</a></h3> Tailscale</a>: I love this and I’ve written about it in the past</a>. I am one of the early adopters, using this for 2 years now and while it can be replaced with other similar Wireguard based mesh services that have popped up recently, I still like the UX of this application a lot. In simpler words, it just works!</em></li> Caddy</a>: I use Caddy as a webserver to host some static websites and as a reverse proxy for all the applications I’ve deployed. It makes the SSL setup a breeze, the config is nice to read as well. There’s nothing about Caddy to not like it if you’re not in the business of dealing with super high concurrent traffic on your website (in which case NGINX/HAProxy/Envoy et al might be worth looking at). I’ve never used Traefik myself, and while I know it makes the routing simple with Docker labels, I think my current approach with Caddy is also similar and easier for me to use without having to learn a new config.</li> Docker</a>: All the services are containerized and I make special use of Docker networks, which I’ll show in the post.</li> </ul> Setup Overview#</a></h2> </p> I use 2 instances of Caddy for my setup:</p> Public: This is to reverse proxy all the public-facing websites.</li> Private: This is to reverse proxy all the internal websites.</li> </ul> The docker-compose</code> looks like this:</p> v</span>ersion</span>:</span> "</span>3.7</span>"</span></span> </span> s</span>ervices</span>:</span></span> c</span>addy_public</span>:</span></span> ...</span></span> p</span>orts</span>:</span></span> -</span> "</span><do_floating_ip>:80:80</span>"</span></span> -</span> "</span><do_floating_ip>:443:443</span>"</span></span> n</span>etworks</span>:</span></span> -</span> p</span>ublic</span></span> ...</span></span> </span> c</span>addy_internal</span>:</span></span> ...</span></span> p</span>orts</span>:</span></span> -</span> "</span>100.111.91.100:80:80</span>"</span></span> -</span> "</span>100.111.91.100:443:443</span>"</span></span> n</span>etworks</span>:</span></span> -</span> i</span>nternal</span></span> ...</span></span> </span> n</span>etworks</span>:</span></span> p</span>ublic</span>:</span></span> n</span>ame</span>:</span> c</span>addy_public</span></span> i</span>nternal</span>:</span></span> n</span>ame</span>:</span> c</span>addy_internal</span></span></code></pre> This is where most of the magic lies. I use 2 Docker networks caddy_public</code> and caddy_internal</code>. Both these networks are configured as Bridge Networks. The containers connected to the same bridge network can even reach other containers using internal DNS.</p> The published</em> port section is the one of importance here.</p> In the internal</code> caddy instance, the TCP port 80 in the container is mapped to port 80 on the Docker host for connections to host IP 100.111.91.100</code> (this is a private IP and belongs to the CGNAT space</a>). The same is done for caddy_public</code> where instead of Tailscale IP, the Floating IP of the DigitalOcean droplet is used.</p> Next comes the part where we’ll attach these networks to our applications. docker-compose</code> by default creates a user-defined bridge network if you leave networks</code> unspecified. However, if you want more granular control, you can specify the networks in the Compose spec itself.</p> Here’s an example of Plausible</code> compose spec which is exposed publicly:</p> p</span>lausible_events_db</span>:</span></span> i</span>mage</span>:</span> y</span>andex/clickhouse-server:21.3.2.5</span></span> n</span>etworks</span>:</span></span> -</span> p</span>lausible</span></span> </span> p</span>lausible</span>:</span></span> i</span>mage</span>:</span> p</span>lausible/analytics:latest</span></span> n</span>etworks</span>:</span></span> -</span> w</span>eb</span></span> -</span> p</span>lausible</span></span> </span> n</span>etworks</span>:</span></span> w</span>eb</span>:</span></span> n</span>ame</span>:</span> c</span>addy_public</span></span> e</span>xternal</span>:</span> true</span></span> p</span>lausible</span>:</span></span> n</span>ame</span>:</span> p</span>lausible</span></span></code></pre> Here you can see that the ClickHouse container is only attached to the plausible</code> network. This plausible</code> network is scoped only to these services defined in this file.</p> We can exec</code> inside the caddy_public</code> container and find out the IP of plausible</code> and verify if the network is correctly configured and reachable:</p> $</span> host</span> plausible</span></span> plausible</span> has</span> address</span> 172.20.0.3</span></span> $</span> curl</span> plausible:8000</span></span> <</span>html><body</span>></span>You are being </span><</span>a</span> href</span>=</span>"</span>/login</span>"</span>></span>r</span>e</span>d</span>i</span>r</span>e</span>c</span>t</span>e</span>d</span><</span>/</span>a</span>></span>.</span><</span>/</span>b</span>o</span>d</span>y</span>><</span>/</span>h</span>t</span>m</span>l</span>></span>/</span>s</span>r</span>v</span> #</span> </span></span></code></pre> Some notes on this setup:</p> You’ll also note that I haven’t published</em> the port 8000 anywhere. Publishing is only required if you want to forward the traffic from your host network to the docker network (via the bridge). But here, since both are attached to the same network (caddy_public</code>), that is not required anymore.</li> This also means that the only way someone can reach port 8000 on Plausible is only via the Caddy container (which is firewall restricted to Cloudflare IPs.)</li> The DB container doesn’t need to be accessed at all from Caddy, so we’ve not attached the web</code> network there</li> </ul> Here’s another example of exposing an internal service, which works on the same principles:</p> g</span>rafana</span>:</span></span> i</span>mage</span>:</span> g</span>rafana/grafana:8.3.4</span></span> n</span>etworks</span>:</span></span> -</span> m</span>onitoring</span></span> -</span> i</span>nternal</span></span> </span> n</span>etworks</span>:</span></span> i</span>nternal</span>:</span></span> n</span>ame</span>:</span> c</span>addy_internal</span></span> e</span>xternal</span>:</span> true</span></span> m</span>onitoring</span>:</span></span> n</span>ame</span>:</span> m</span>onitoring</span></span></code></pre> Here, the Grafana container is attached to caddy_internal</code> network. Since the caddy_internal</code> container only publishes ports on the Tailscale IP, anyone who is not inside the Tailscale network will not be able to access this. Tailscale can do much more by setting up ACL rules per device for each user, but since I am the only user, I’ve not configured ACL rules on it yet.</p> Hope this approach was simplistic enough. I follow this pattern across all the applications I self-host and honestly pretty happy with it</p> Fin!</p> Using ClickHouse Keeper for Replication 2021-12-17T08:10:55+05:30 ClickHouse is an extremely performant columnar DB used for fast analytical processing. ClickHouse supports data replication</a> using Apache Zookeeper which needs to be deployed externally. While Zookeeper works well after you’ve tuned it properly, it was still an additional maintenance overhead. The good news is that you don’t have to worry about this anymore.</p> ClickHouse 21.12 release annoucement</a> mentions ClickHouse Keeper</code> as feature complete. Keeper</a> is a replacement for Zookeeper, written in C++ and uses the RAFT algorithm for consensus across different nodes in the cluster. It also has some nice improvements over Zookeeper, such as compressed snapshots/logs of the state changes in the cluster and the ability to run it inside</em> the server binary itself. I decided to spin up a local Clickhouse cluster and test out the new clickhouse-keeper</code> feature.</p> For our local setup, we’ll set up 4 nodes</strong>. We’ll create 2 shards</strong> to distribute our data and each shard will have 2 replicas each</strong>. The setup looks something like this:</p> </p> We need to run clickhouse-keeper</code> only on 3 of these nodes to ensure a quorum. Here’s a sample docker-compose.yml</code> to spin up these nodes as containers:</p> v</span>ersion</span>:</span> "</span>3.7</span>"</span></span> </span> x</span>-clickhouse-defaults</span>:</span> &</span>clickhouse-defaults</span></span> r</span>estart</span>:</span> u</span>nless-stopped</span></span> i</span>mage</span>:</span> y</span>andex/clickhouse-server:21.12.2.17</span></span> u</span>limits</span>:</span></span> n</span>proc</span>:</span> 65535</span></span> n</span>ofile</span>:</span></span> s</span>oft</span>:</span> 262144</span></span> h</span>ard</span>:</span> 262144</span></span> </span> s</span>ervices</span>:</span></span> c</span>lickhouse-blue-1</span>:</span></span> <<</span>:</span> </span>clickhouse-defaults</span></span> c</span>ontainer_name</span>:</span> c</span>lickhouse-blue-1</span></span> h</span>ostname</span>:</span> c</span>lickhouse-blue-1</span></span> p</span>orts</span>:</span></span> -</span> 9</span>000:9000</span></span> -</span> 8</span>123:8123</span></span> -</span> 9</span>181:9181</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> c</span>h-blue-1-data</span></span> t</span>arget</span>:</span> /</span>var/lib/clickhouse</span></span> -</span> "</span>./configs/gen/clickhouse-blue-1:/etc/clickhouse-server/config.d/</span>"</span></span> </span> c</span>lickhouse-blue-2</span>:</span></span> <<</span>:</span> </span>clickhouse-defaults</span></span> c</span>ontainer_name</span>:</span> c</span>lickhouse-blue-2</span></span> h</span>ostname</span>:</span> c</span>lickhouse-blue-2</span></span> p</span>orts</span>:</span></span> -</span> 9</span>001:9000</span></span> -</span> 8</span>124:8123</span></span> -</span> 9</span>182:9181</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> c</span>h-blue-2-data</span></span> t</span>arget</span>:</span> /</span>var/lib/clickhouse</span></span> -</span> "</span>./configs/gen/clickhouse-blue-2:/etc/clickhouse-server/config.d/</span>"</span></span> </span> c</span>lickhouse-green-1</span>:</span></span> <<</span>:</span> </span>clickhouse-defaults</span></span> c</span>ontainer_name</span>:</span> c</span>lickhouse-green-1</span></span> h</span>ostname</span>:</span> c</span>lickhouse-green-1</span></span> p</span>orts</span>:</span></span> -</span> 9</span>002:9000</span></span> -</span> 8</span>125:8123</span></span> -</span> 9</span>183:9181</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> c</span>h-green-1-data</span></span> t</span>arget</span>:</span> /</span>var/lib/clickhouse</span></span> -</span> "</span>./configs/gen/clickhouse-green-1:/etc/clickhouse-server/config.d/</span>"</span></span> </span> c</span>lickhouse-green-2</span>:</span></span> <<</span>:</span> </span>clickhouse-defaults</span></span> c</span>ontainer_name</span>:</span> c</span>lickhouse-green-2</span></span> h</span>ostname</span>:</span> c</span>lickhouse-green-2</span></span> p</span>orts</span>:</span></span> -</span> 9</span>003:9000</span></span> -</span> 8</span>126:8123</span></span> -</span> 9</span>184:9181</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> c</span>h-green-2-data</span></span> t</span>arget</span>:</span> /</span>var/lib/clickhouse</span></span> -</span> "</span>./configs/gen/clickhouse-green-2:/etc/clickhouse-server/config.d/</span>"</span></span> </span> v</span>olumes</span>:</span></span> c</span>h-blue-1-data</span>:</span></span> c</span>h-blue-2-data</span>:</span></span> c</span>h-green-1-data</span>:</span></span> c</span>h-green-2-data</span>:</span></span></code></pre> clickhouse-keeper</code> runs only if the <keeper_config></code> section is present inside the config. Here’s a sample config:</p> <</span>clickhouse</span>></span></span> <</span>keeper_server</span>></span></span> <</span>tcp_port</span>></span>9181</span></</span>tcp_port</span>></span></span> <</span>server_id</span>></span>${SERVER_ID}</span></</span>server_id</span>></span></span> <</span>log_storage_path</span>></span>/var/lib/clickhouse/coordination/log</span></</span>log_storage_path</span>></span></span> <</span>snapshot_storage_path</span>></span>/var/lib/clickhouse/coordination/snapshots</span></</span>snapshot_storage_path</span>></span></span> </span> <</span>coordination_settings</span>></span></span> <</span>operation_timeout_ms</span>></span>10000</span></</span>operation_timeout_ms</span>></span></span> <</span>session_timeout_ms</span>></span>30000</span></</span>session_timeout_ms</span>></span></span> <</span>raft_logs_level</span>></span>trace</span></</span>raft_logs_level</span>></span></span> </</span>coordination_settings</span>></span></span> </span> <</span>raft_configuration</span>></span></span> <</span>server</span>></span></span> <</span>id</span>></span>1</span></</span>id</span>></span></span> <</span>hostname</span>></span>clickhouse-blue-1</span></</span>hostname</span>></span></span> <</span>port</span>></span>9234</span></</span>port</span>></span></span> </</span>server</span>></span></span> <</span>server</span>></span></span> <</span>id</span>></span>2</span></</span>id</span>></span></span> <</span>hostname</span>></span>clickhouse-blue-2</span></</span>hostname</span>></span></span> <</span>port</span>></span>9234</span></</span>port</span>></span></span> </</span>server</span>></span></span> <</span>server</span>></span></span> <</span>id</span>></span>3</span></</span>id</span>></span></span> <</span>hostname</span>></span>clickhouse-green-1</span></</span>hostname</span>></span></span> <</span>port</span>></span>9234</span></</span>port</span>></span></span> </</span>server</span>></span></span> </</span>raft_configuration</span>></span></span> </</span>keeper_server</span>></span></span> </</span>clickhouse</span>></span></span></code></pre> There are some other configs required for Clickhouse to discover other nodes and enable replication. You can find a working example in this repo</a>.</p> Verifying Cluster State#</a></h2> Once the containers are configured and running, we can verify if the replication is working as intended:</p> Let’s first check if the keeper</code> daemon is running by:</p> echo</span> ruok</span> \|</span> nc</span> 127.0.0.1</span> 9181</span></span> imok</span></span></code></pre> ruok</code> is a part of Four Letter Commands</a> that are mostly used to diagnose Keeper’s client/server.</p> To ensure that clickhouse-server</code> is aware of the keeper</code> cluster, we can query the system.zookeeper</code> table:</p> SELECT</span> </span></span> FROM</span> system</span>.</span>zookeeper</span></span> WHERE</span> path</span> =</span> '</span>/</span>'</span></span> FORMAT Vertical</span></span> </span> Query id: 287d3c2d</span>-</span>b93f</span>-</span>4d48</span>-</span>b335</span>-</span>6df2f89a8ab3</span></span> </span> Row</span> 1</span>:</span></span> ──────</span></span> name</span>: clickhouse</span></span> value</span>: </span></span> czxid: </span>3</span></span> mzxid: </span>3</span></span> ctime: </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>11</span>:</span>05</span></span> mtime: </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>11</span>:</span>05</span></span> version</span>: </span>0</span></span> cversion: </span>1</span></span> aversion: </span>0</span></span> ephemeralOwner: </span>0</span></span> dataLength: </span>0</span></span> numChildren: </span>1</span></span> pzxid: </span>4</span></span> path</span>: </span>/</span></span></code></pre> If you don’t see any results in the system.zookeeper</code> table, then re-check if zookeeper</code> section is present inside the config. This config tells ClickHouse how to discover keeper nodes.</p> We can also see if our cluster is configured correctly with:</p> SELECT</span></span> host_name,</span></span> host_address,</span></span> replica_num</span></span> FROM</span> system</span>.</span>clusters</span></span> WHERE</span> cluster </span>=</span> '</span>events</span>'</span></span> </span> Query id: a4bacfa1</span>-</span>d3aa</span>-</span>482f</span>-</span>b8b2</span>-</span>30b05442a173</span></span> </span> ┌─host_name──────────┬─host_address─┬─replica_num─┐</span></span> │ clickhouse</span>-</span>blue</span>-</span>1</span> │ </span>172</span>.</span>19</span>.</span>0</span>.</span>5</span> │ </span>1</span> │</span></span> │ clickhouse</span>-</span>blue</span>-</span>2</span> │ </span>172</span>.</span>19</span>.</span>0</span>.</span>3</span> │ </span>2</span> │</span></span> │ clickhouse</span>-</span>green</span>-</span>1</span> │ </span>172</span>.</span>19</span>.</span>0</span>.</span>4</span> │ </span>1</span> │</span></span> │ clickhouse</span>-</span>green</span>-</span>2</span> │ </span>172</span>.</span>19</span>.</span>0</span>.</span>2</span> │ </span>2</span> │</span></span> └────────────────────┴──────────────┴─────────────┘</span></span></code></pre> (Here events</code> is our cluster name specified in the remote_servers</code> section of the config.)</p> Inserting Sample Data#</a></h2> Let’s create a DB and add some data to the DB. We need to ensure that our data is split across shards and we can query all shards using a central view</em>.</p> Cluster Schema#</a></h3> CREATE</span> DATABASE</span> app</span> ON</span> CLUSTER </span>'</span>events</span>'</span>;</span></span> </span> CREATE</span> TABLE</span> app</span>.events_local </span>ON</span> CLUSTER </span>'</span>{cluster}</span>'</span> (</span></span> time</span> DateTime</span>,</span></span> event_id Int32,</span></span> uuid UUID</span></span> )</span></span> ENGINE </span>=</span> ReplicatedMergeTree(</span>'</span>/clickhouse/tables/{cluster}/{shard}/{table}</span>'</span>, </span>'</span>{replica}</span>'</span>)</span></span> PARTITION</span> BY</span> toYYYYMM(</span>time</span>)</span></span> ORDER BY</span> (event_id);</span></span> </span> CREATE</span> TABLE</span> app</span>.events_main </span>ON</span> CLUSTER </span>'</span>{cluster}</span>'</span> AS</span> app</span>.</span>events_local</span></span> ENGINE </span>=</span> Distributed</span>(</span>'</span>{cluster}</span>'</span>, app, events_local, </span>rand</span>(</span>));</span></span></code></pre> What’s happening here:</p> We’ll create a sample database app</code> with a single table events_local</code>.</li> We’re using ReplicatedMergeTree</code> here as that tells ClickHouse</code> to automatically replicate the data inside the table when it’s inserted. Properties like cluster</code>/shard</code>/replica</code> are automatically populated from the server’s macros</em>. It’s a handy feature that allows you to execute this command just once</em> on the cluster and it automatically populates the relevant config in each server.</li> </ul> </li> Finally we create a Distributed</code> table to perform our INSERT</code>/SELECT</code> operations in a central place. It’s possible to manually insert data to particular replicas, but since Clickhouse supports load balancing across shards, it’s preferred to create a table with a Distributed</code> engine and let that happen automatically. Write operations can be sharded based on a particular column name, but here we are simply using the rand()</code> function, which splits the write randomly to different shards.</li> All read operations are parallelized and Clickhouse selects one replica from each shard to query the data from.</li> </ul> </li> We use ON CLUSTER</code> keyword to indicate that this query has to be run on all servers which are part of the cluster events</code>.</li> </ol> It’s now time to insert some random data, for which you can use this command:</p> INSERT INTO</span> app</span>.</span>events_main</span> VALUES</span> (</span>now</span>(</span>)</span>, </span>rand</span>(</span>1</span>), generateUUIDv4</span>(</span>)</span>);</span></span></code></pre> We can now query the table and see if our records are present:</p> SELECT</span> </span></span> FROM</span> app</span>.</span>events_main</span></span> </span> ┌────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>17</span> │ </span>1888949839</span> │ 3c33305e</span>-</span>8ea0</span>-</span>4ac1</span>-</span>a07a</span>-</span>667465ec9a85 │</span></span> └─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>-</span>689113926</span> │ 2a944d71</span>-</span>73f3</span>-</span>4491</span>-</span>b851</span>-</span>4f0e6f296e44 │</span></span> └─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌────────────────</span>time</span>─┬──event_id─┬─uuid─────────────────────────────────┐</span></span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>15</span> │ </span>415899002</span> │ 5980299f</span>-</span>1594</span>-</span>4c17</span>-</span>8eb5</span>-</span>e512f15ecf34 │</span></span> └─────────────────────┴───────────┴──────────────────────────────────────┘</span></span> ┌────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>1303963476</span> │ 0b27d5bd</span>-</span>6315</span>-</span>4937</span>-</span>82a4</span>-</span>18199e5eebb7 │</span></span> └─────────────────────┴────────────┴──────────────────────────────────────┘</span></span></code></pre> Now, you must be wondering how to check if the data is replicated and how the data is distributed across our shards. For that, we can use the handy remote()</code> function:</p> SELECT</span> </span></span> FROM</span></span> (</span></span> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.2</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> UNION ALL</span></span> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.3</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> UNION ALL</span></span> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.4</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> UNION ALL</span></span> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.5</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> )</span></span> </span> Query id: 34a81447</span>-</span>a31b</span>-</span>4915</span>-</span>9bd8</span>-</span>7a6e17bb0860</span></span> </span> ┌─hostName</span>(</span>)</span>────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>blue</span>-</span>2</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>17</span> │ </span>1888949839</span> │ 3c33305e</span>-</span>8ea0</span>-</span>4ac1</span>-</span>a07a</span>-</span>667465ec9a85 │</span></span> └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>blue</span>-</span>1</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>17</span> │ </span>1888949839</span> │ 3c33305e</span>-</span>8ea0</span>-</span>4ac1</span>-</span>a07a</span>-</span>667465ec9a85 │</span></span> └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬──event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>2</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>15</span> │ </span>415899002</span> │ 5980299f</span>-</span>1594</span>-</span>4c17</span>-</span>8eb5</span>-</span>e512f15ecf34 │</span></span> └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>2</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>-</span>689113926</span> │ 2a944d71</span>-</span>73f3</span>-</span>4491</span>-</span>b851</span>-</span>4f0e6f296e44 │</span></span> └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>2</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>1303963476</span> │ 0b27d5bd</span>-</span>6315</span>-</span>4937</span>-</span>82a4</span>-</span>18199e5eebb7 │</span></span> └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>1</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>1303963476</span> │ 0b27d5bd</span>-</span>6315</span>-</span>4937</span>-</span>82a4</span>-</span>18199e5eebb7 │</span></span> └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬──event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>1</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>15</span> │ </span>415899002</span> │ 5980299f</span>-</span>1594</span>-</span>4c17</span>-</span>8eb5</span>-</span>e512f15ecf34 │</span></span> └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘</span></span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬───event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>1</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 09</span>:</span>36</span>:</span>16</span> │ </span>-</span>689113926</span> │ 2a944d71</span>-</span>73f3</span>-</span>4491</span>-</span>b851</span>-</span>4f0e6f296e44 │</span></span> └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</span></span></code></pre> Perfect! We can see our data is sharded as some parts of it exists in green</code> and blue</code>. We can also see that for each record, we have 2 entries thus confirming that ReplicatedMergeTree</code> is doing its job!</p> Additional Scenarios#</a></h2> All is well and good so far, but to ensure that our cluster setup is resilient we need to introduce our cluster to some Non-Happy</em> scenarios.</p> Stop a replica node#</a></h3> </p> ❯</span> docker-compose</span> stop</span> clickhouse-green-2</span> </span></span> Stopping</span> clickhouse-green-2</span> ...</span> done</span></span></code></pre> Now, let’s query our records:</p> SELECT</span> count</span>(</span></span>)</span></span> FROM</span> app</span>.</span>events_main</span></span> </span> ┌─</span>count</span>(</span>)─┐</span></span> │ </span>4</span> │</span></span> └─────────┘</span></span></code></pre> That seems right. We’re able to access all our data with just one replica down.</p> Let’s try inserting data and bring back the replica to see if it got automatically replicated or not:</p> INSERT INTO</span> app</span>.</span>events_main</span> VALUES</span> (</span>now</span>(</span>)</span>, </span>rand</span>(</span>1</span>), generateUUIDv4</span>(</span>)</span>);</span></span> </span> Ok.</span></span></code></pre>SELECT</span> </span></span> FROM</span> app</span>.</span>events_main</span></span> ORDER BY</span> time</span> DESC</span></span> LIMIT</span> 1</span></span> </span> ┌────────────────</span>time</span>─┬──event_id─┬─uuid─────────────────────────────────┐</span></span> │ </span>2021</span>-</span>12</span>-</span>17</span> 10</span>:</span>09</span>:</span>10</span> │ </span>375826335</span> │ 8dbbc9cf</span>-</span>8f00</span>-</span>4cc3</span>-</span>a4fb</span>-</span>0f17a68340e5 │</span></span> └─────────────────────┴───────────┴──────────────────────────────────────┘</span></span></code></pre> Let’s start the replica again:</p> ❯</span> docker-compose</span> start</span> clickhouse-green-2</span></span> Starting</span> clickhouse-green-2</span> ...</span> done</span></span></code></pre> On querying the replica to see if it has the data:</p> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.5</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> ORDER BY</span> time</span> DESC</span></span> LIMIT</span> 1</span></span> </span> Query id: 422041dc</span>-</span>afe8</span>-</span>4f28</span>-</span>9659</span>-</span>6d58726f8c90</span></span> </span> ┌─hostName</span>(</span>)</span>─────────┬────────────────</span>time</span>─┬──event_id─┬─uuid─────────────────────────────────┐</span></span> │ clickhouse</span>-</span>green</span>-</span>2</span> │ </span>2021</span>-</span>12</span>-</span>17</span> 10</span>:</span>09</span>:</span>10</span> │ </span>375826335</span> │ 8dbbc9cf</span>-</span>8f00</span>-</span>4cc3</span>-</span>a4fb</span>-</span>0f17a68340e5 │</span></span> └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘</span></span></code></pre> Perfect! The record 375826335</code> automatically got replicated once the replica was healthy.</p> Stop a Keeper Node#</a></h3> </p> We’ll stop a server instance that is running the clickhouse-keeper</code> process. By doing this, we’ll also be killing a replica, but that is okay.</p> ❯</span> docker-compose</span> stop</span> clickhouse-blue-2</span></span> Stopping</span> clickhouse-blue-2</span> ...</span> done</span></span></code></pre> Let’s insert some data:</p> INSERT INTO</span> app</span>.</span>events_main</span> VALUES</span> (</span>now</span>(</span>)</span>, </span>rand</span>(</span>1</span>), generateUUIDv4</span>(</span>)</span>);</span></span> </span> Ok</span></span></code></pre> We can check the server information of the other 2 keeper</code> nodes:</p> $</span> echo</span> stat</span> \|</span> nc</span> 127.0.0.1</span> 9181</span> \|</span> grep</span> Mode</span></span> Mode:</span> follower</span></span> $</span> echo</span> stat</span> \|</span> nc</span> 127.0.0.1</span> 9183</span> \|</span> grep</span> Mode</span></span> Mode:</span> leader</span></span></code></pre> So, one of the keeper</code> nodes has elected itself to be the leader and we have no problems in the setup so far. However, if we stop another keeper</code> node, then there will be only keeper</code> node remaining in the setup and to avoid a Split Brain</em> issue, it won’t be able to elect itself as the leader.</p> </p> What happens then? Only one way to find out:</p> ❯</span> docker-compose</span> stop</span> clickhouse-blue-1</span></span> Stopping</span> clickhouse-blue-1</span> ...</span> done</span></span></code></pre>INSERT INTO</span> app</span>.</span>events_main</span> VALUES</span> (</span>now</span>(</span>)</span>, </span>rand</span>(</span>1</span>), generateUUIDv4</span>(</span>)</span>);</span></span> </span> Received exception </span>from</span> server</span> (</span>version</span> 21</span>.</span>12</span>.</span>2</span>):</span></span> Code: </span>242</span>. DB::Exception: Received </span>from</span> localhost:</span>9000</span>. DB::Exception: </span>Table</span> is</span> in</span> readonly</span> mode (zookeeper </span>path</span>: </span>/</span>clickhouse</span>/</span>tables</span>/</span>events</span>/</span>green</span>/</span>table</span>). (TABLE_IS_READ_ONLY)</span></span></code></pre> Ah! So, we can still query for the data (which will be incomplete since blue</em> shard is completely down), but we cannot insert any new data at all. We can even verify this by querying for the health of the keeper</code> node:</p> $</span> echo</span> mntr</span> \|</span> nc</span> localhost</span> 9183</span> </span></span> This</span> instance</span> is</span> not</span> currently</span> serving</span> requests%</span> </span></span></code></pre>Add a new shard#</a></h3> </p> Alright, so our tests so far have been quite good and show that the cluster is resilient to failures as long as there exists a keeper</code> node running as leader</code> mode in the quorum. Now, let’s see how to add a new shard. We’ll extend our docker-compose.yml</code> to add a new orange</code> shard:</p> c</span>lickhouse-orange-1</span>:</span></span> <<</span>:</span> </span>clickhouse-defaults</span></span> c</span>ontainer_name</span>:</span> c</span>lickhouse-orange-1</span></span> h</span>ostname</span>:</span> c</span>lickhouse-orange-1</span></span> p</span>orts</span>:</span></span> -</span> 9</span>004:9000</span></span> -</span> 8</span>127:8123</span></span> -</span> 9</span>185:9181</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> c</span>h-orange-1-data</span></span> t</span>arget</span>:</span> /</span>var/lib/clickhouse</span></span> -</span> "</span>./configs/gen/clickhouse-orange-1:/etc/clickhouse-server/config.d/</span>"</span></span></code></pre> Inside our remote_servers.xml</code>, we’ll add the orange</code> shard as well:</p> <</span>shard</span>></span></span> <</span>internal_replication</span>></span>true</span></</span>internal_replication</span>></span></span> <</span>replica</span>></span></span> <</span>host</span>></span>clickhouse-orange-1</span></</span>host</span>></span></span> <</span>port</span>></span>9000</span></</span>port</span>></span></span> </</span>replica</span>></span></span> </</span>shard</span>></span></span></code></pre> That’s pretty much it. Let’s start the new replica:</p> docker-compose</span> up</span></span></code></pre> Let’s insert some data and query if the shard is getting data or not:</p> SELECT</span></span> hostName</span>(</span>)</span>,</span></span> </span></span> FROM</span> remote</span>(</span>'</span>172.20.0.2</span>'</span>, </span>'</span>app</span>'</span>, </span>'</span>events_local</span>'</span>)</span></span> </span> Received exception </span>from</span> server</span> (</span>version</span> 21</span>.</span>12</span>.</span>2</span>):</span></span> Code: </span>519</span>. DB::Exception: Received </span>from</span> localhost:</span>9000</span>. DB::Exception: All attempts </span>to</span> get</span> table</span> structure failed. </span>Log</span>: </span></span> </span> Code: </span>279</span>. DB::NetException: All </span>connection</span> tries failed. </span>Log</span>: </span></span> </span> There </span>is</span> no</span> table</span> `</span>app</span>`</span>.</span>`</span>events_local</span>`</span> on</span> server</span>: </span>172</span>.</span>20</span>.</span>0</span>.</span>2</span>:</span>9000</span></span> </span> . (ALL_CONNECTION_TRIES_FAILED) (</span>version</span> 21</span>.</span>12</span>.</span>2</span>.</span>17</span> (official build))</span></span> </span> . (NO_REMOTE_SHARD_AVAILABLE)</span></span></code></pre> Houston, we have a problem.</p> > There is no table `app`.`events_local` on server: 172.20.0.2:9000</span></span></code></pre> This is mentioned in the ClickHouse docs on Replication</a>:</p> CREATE, DROP, ATTACH, DETACH and RENAME queries are executed on a single server and are not replicated:</p> </blockquote> This means that although we’d run CREATE DATABASE</code> and CREATE TABLE</code> commands using ON CLUSTER</code> which executes on all servers, but since the orange-1</code> node is introduced after</em> we ran that command, we need to manually create the DB and Table here. We have to execute the below commands inside</em> the orange-1</code> replica:</p> CREATE</span> DATABASE</span> app</span>;</span></span> </span> CREATE</span> TABLE</span> app</span>.events_local (</span></span> time</span> DateTime</span>,</span></span> event_id Int32,</span></span> uuid UUID</span></span> )</span></span> ENGINE </span>=</span> ReplicatedMergeTree(</span>'</span>/clickhouse/tables/{cluster}/{shard}/{table}</span>'</span>, </span>'</span>{replica}</span>'</span>)</span></span> PARTITION</span> BY</span> toYYYYMM(</span>time</span>)</span></span> ORDER BY</span> (event_id);</span></span></code></pre> That’s all that is required. Adding a new replica is also the same process.</p> Summary#</a></h2> Hope this tutorial helped you figure out how to use clickhouse-keeper</code> to set up a distributed ClickHouse cluster DB. clickhouse-keeper</code> is still a relatively new feature as the docs mention already, but given that it solves operations overhead of running a Zookeeper cluster, it’s worth checking it out.</p> References#</a></h3> https://www.youtube.com/watch?v=abhcCRW09Ac</a> Slides used in above talk: https://presentations.clickhouse.com/meetup54/keeper.pdf</a></li> </ul> </li> https://clickhouse.com/docs/en/operations/clickhouse-keeper</a></li> https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration/test_keeper_multinode_simple</a></li> https://github.com/ClickHouse/ClickHouse/issues/2161</a></li> </ul> For the full code/config samples, you can check out the repo</a>.</p> Fin!</p> Updates#</a></h3> If you have setup RBAC on your cluster, make sure you add <user></code> and <password></code> fields to the <remote_server></code> configuration.</li> </ul> Breaking Software and Getting Older 2021-12-09T08:10:55+05:30 Recently, I’d posted on Twitter that my feed has become messy</em> overtime. Maybe I followed some accounts that were of no interest, maybe Twitter’s algos don’t really know what to show to me. Whatever, I wanted a fresh start. And, I’ve done this in past. I’ve removed all the accounts that I follow but they don’t follow me back. This little trick is helpful to start afresh while still not offending your friends ;)</p> </p> I’d written a really simple Python script to do the job! It looks like:</p> import</span> tweepy</span></span> </span> SCREEN_NAME</span> =</span> '</span>mrkaran_</span>'</span></span> CONSUMER_KEY</span> =</span> '</span>'</span></span> CONSUMER_SECRET</span> =</span> '</span>'</span></span> ACCESS_TOKEN</span> =</span> '</span>'</span></span> ACCESS_TOKEN_SECRET</span> =</span> '</span>'</span></span> </span> auth</span> =</span> tweepy</span>.</span>OAuthHandler</span>(</span>CONSUMER_KEY</span>,</span> CONSUMER_SECRET</span>)</span></span> auth</span>.</span>set_access_token</span>(</span>ACCESS_TOKEN</span>,</span> ACCESS_TOKEN_SECRET</span>)</span></span> api</span> =</span> tweepy</span>.</span>API</span>(</span>auth</span>)</span></span> </span> followers</span> =</span> api</span>.</span>followers_ids</span>(</span>SCREEN_NAME</span>)</span></span> friends</span> =</span> api</span>.</span>friends_ids</span>(</span>SCREEN_NAME</span>)</span></span> </span> for</span> f</span> in</span> friends</span>:</span></span> if</span> f</span> not</span> in</span> followers</span>:</span></span> print</span>(</span>"</span>Unfollow </span>{0}</span>?</span>"</span>.</span>format</span>(</span>api</span>.</span>get_user</span>(</span>f</span>)</span>.</span>screen_name</span>)</span>)</span></span> api</span>.</span>destroy_friendship</span>(</span>f</span>)</span></span></code></pre> I’d last ran this in 2019 or so. That is dinosaur ages in the world of software. All I wanted to do was run this goddamn script again, 2 years later. Seems to be too much of an ask? Apparently tweepy</code> the library used here to interact with Twitter’s APIs had a major release</a> with lots of breaking changes. They’ve internally migrated to start using v2 Twitter API. So, when I naively ran pip install tweepy</code>, my code threw:</p> ❯ </span>python</span> main</span>.</span>py</span> </span></span> Traceback</span> (</span>most</span> recent</span> call</span> last</span>)</span>:</span></span> File</span> "</span>/home/karan/Code/Personal/twitter-unfollow/main.py</span>"</span>,</span> line</span> 13</span>,</span> in</span> <</span>module</span>></span></span> followers</span> =</span> api</span>.</span>followers_ids</span>(</span>SCREEN_NAME</span>)</span></span> AttributeError</span>:</span> '</span>API</span>'</span> object</span> has</span> no</span> attribute</span> '</span>followers_ids</span>'</span></span></code></pre> So, some method name changed. But that’s not all. The whole auth process including the initialisation of the API object changed as well. I’d spent some ~15 minutes grokking the docs but frustrated because 1) I don’t give a shit about v2/v1 APIs. 2) I just want to carry</em> on with whatever I was doing. Why is this shit taking more time than I can care to give this to?</p> I’d have cared enough if it was a side-project I maintained or something that I used daily. A utility like this which gets used once in a couple of years, will</strong> see more such breaking changes in future. Why, then should I spend migrating to v2 APIs, when after 2 years, v3 APIs would have broken my code again? What is the damn point? Why can’t software just keep working without troubling their users?</p> In the end, I just installed the last version that works with v1</code> APIs and ran the script.</p> I get breaking changes, I totally do. And I’ve no qualms with Tweepy. They did what they had to, in order to be compatible with v2. I am just angry/sad at the whole ecosystem of “Move fast, break things”.</p> Please. Slow. Down.</em></p> So that the rest of us who have a life can enjoy it and not spend an entire weekend migrating across versions!</p> Sigh</p> Debugging issues with Packer and Ansible in Docker 2021-11-16T00:00:00+05:30 Today I faced an issue that questioned my sanity. Since I didn’t find many “related” issues on StackOverflow/Google-fu except a lone GitHub thread where a kind stranger hinted at what could be the issue, I am writing about it here in the hopes (although I don’t wish this torture on anyone) that it helps someone!</p> To give some context, I am running a Gitlab CI</a> job that bakes an AMI</a> using Packer</a>. Packer can use different kinds of provisioners to do configure stuff on the host and then prepare the image. I am using Ansible provisioner</a> to install and configure Consul</a>. At this point you may think this post is sponsored by Hashicorp by the sheer mention of all their products, but I assure you that is not the case.</p> Anyway, so this role works locally but it fails on the damn Gitlab CI. Classic case of Works on my machine, Ops problem now</a>. These kinds of issues although are particularly exciting for me because they give me a chance to dig down deeper in the internals and slowly peel apart layers to figure out where the “drift” between local and CI is happening.</p> Here’s the relevant Packer snippet (Oh and this week as I updated myself with new Packer releases, it’s now possible to write Packer config with HCL and not just JSON anymore! Yayie. Again a reminder: not a sponsored post).</p> build</span> {</span></span> sources</span> =</span> [</span>"</span>source.amazon-ebs.golden-ami</span>"</span>]</span></span> provisioner</span> "ansible"</span> {</span></span> playbook_file</span> =</span> "</span>${</span>var</span>.</span>playbook_file</span>}</span>"</span></span> extra_arguments</span> =</span> [</span>"</span>--tags</span>"</span>,</span> "</span>install</span>"</span>,</span> "</span>-e</span>"</span>,</span> "</span>ansible_python_interpreter=/usr/bin/python3</span>"</span>]</span></span> ansible_env_vars</span> =</span> [</span>"</span>ANSIBLE_LOCAL_TEMP=$HOME/.ansible/tmp</span>"</span>,</span> "</span>ANSIBLE_REMOTE_TEMP=$HOME/.ansible/tmp</span>"</span>]</span></span> galaxy_file</span> =</span> "</span>${</span>var</span>.</span>galaxy_file</span>}</span>"</span></span> inventory_file_template</span> =</span> "</span>[consul_instances]</span>\n</span>{{ .HostAlias }} ansible_host={{ .Host }} ansible_user={{ .User }} ansible_port={{ .Port }}</span>\n</span>"</span></span> }</span></span> }</span></span></code></pre> Running packer build</code> in CI results in a failure of a task defined in the Ansible playbook. The task simply creates a new group:</p> -</span> n</span>ame</span>:</span> A</span>dd Consul group</span></span> g</span>roup</span>:</span></span> n</span>ame</span>:</span> "</span>{{ consul_group }}</span>"</span></span> s</span>tate</span>:</span> p</span>resent</span></span> w</span>hen</span>:</span></span> -</span> c</span>onsul_manage_group \| bool</span></span></code></pre> group</code></a> runs groupapp</code> command behind the scenes and you should be in the list of sudoers to actually create new groups. Since I already have become: true</code> and become_user: root</code> in my playbook that requirement is fulfilled. Moreover, this task runs just fine in the local as I mentioned above. While running in CI, I see the following error:</p> amazon-ebs.golden-ami: TASK [consul : Add Consul group] **********************************************</span></span> amazon-ebs.golden-ami: fatal: [default]: FAILED! => {"changed": false, "msg": "groupadd: Permission denied.\ngroupadd: cannot lock /etc/group; try again later.\n", "name": "consul"}</span></span></code></pre> Erhm, okayyy. That looks like a permission error. But why does this not happen in my local was the question eating me up.</p> Now was the time to start from the ground up and dissect different things going here. I will add a small hint though: The Gitlab CI runner is a Docker-based</a> runner. That means all the commands like packer build</code> etc happen inside a Docker container. I am using hashicorp/packer:light</code> image, which is an Alpine based image containing just the packer</code> executable.</p> I tried to run the container locally with:</p> docker</span> run</span> -</span>v</span> `</span>pwd</span>`</span>:/app</span> -</span>-rm</span> -</span>it</span> -</span>-entrypoint=</span>'</span>'</span> hashicorp/packer:light</span> sh</span></span></code></pre> And yes. When I ran packer build</code>, I could replicate the issue here! But wait. More questions than answers. Ansible would run this groupadd</code> command on the remote host, right? Why does Ansible care if it’s inside a container or not? So, I created a really simple playbook to reproduce this further.</p> - name: Assemble Consul cluster</span></span> hosts: localhost</span></span> any_errors_fatal: true</span></span> become: true</span></span> become_user: root</span></span> tasks:</span></span> - name: Add Consul group</span></span> group:</span></span> name: debug_consul</span></span> state: present</span></span></code></pre> I ran this ansible-playbook test.yml</code> inside the container and… it worked! Okay, now it’s becoming clear. We aren’t executing ansible-playbook</code> directly, it’s being wrapped by Packer. So this is clearly getting messed up by Packer. This time I did find a GitHub issue</a> where people were asking about similar issues and this person described it well:</p> </p> I opened Packer docs</a> again and that’s when I read this:</p> We recommend against running Packer as root; if you do then you won’t be able to successfully run your Ansible playbook as root; become: yes will fail.</p> </blockquote> WTF!!! This was right there in the docs, hiding in plain sight. Sheesh!</p> Okay, so I can not run my playbook with become: true</code> with the packer Image (which uses root user). Time to fix that by building a custom image. That is because Gitlab CI won’t let me change the user</a> without hacking stuff. And a custom image also allows me to ditch Alpine for Ubuntu, which is what I prefer.</p> FROM ubuntu:latest</span></span> </span> RUN apt-get update && apt-get install -y \</span></span> python3 \</span></span> git \</span></span> curl \</span></span> unzip \</span></span> python3-pip \</span></span> && rm -rf /var/lib/apt/lists/</span></span> </span> RUN curl -o packer.zip https://releases.hashicorp.com/packer/1.7.8/packer_1.7.8_linux_amd64.zip</span></span> RUN unzip packer.zip</span></span> RUN mv packer /usr/local/bin</span></span> </span> RUN pip3 install ansible</span></span> </span> RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 ubuntu</span></span> USER ubuntu</span></span> </span> WORKDIR /tmp</span></span> </span> ENV PATH="$HOME/.local/bin:$PATH"</span></span> </span> WORKDIR /app</span></span></code></pre> Using this custom image, the packer build worked just fine. Fun day indeed (/s).</p> Fin!</p> Load testing with K6 2021-10-29T00:00:00+05:30 This week I was occupied with optimising a Golang program I’d written at work. I wanted a way to reproduce the issue under heavy load on my development environment and Load Tests are a good way to do that.</p> The service in question is a RESTful API so it’s relatively easy to use any HTTP load test tools. The endpoint had an input parameter uuid</code> which accepted a valid UUIDv4</code> as the input. To my surprise, this was not so straightforward with hey</a> (which is my tool of choice for simple tests) and ab</a>. While it was possible to write an external script to do that, I thought to look around at some “scriptable” alternatives. I found wrk</a> which allowed me to write custom Lua modules. Now, I didn’t want to lose my focus from the main task which was load testing my service to write Lua, so I didn’t use wrk</code> but it’s still a pretty decent option (and very very</em> fast, at that).</p> Hello k6!#</a></h2> </p> Some more Google-fu resulted in me finding k6</a>. I’d never heard of this but after exploring the GitHub repo</a> and the docs it looks like a pretty active project.</p> So, k6</code> basically allows you to write scriptable tests which allow you to test a variety of scenarios</a>. The scripts are written in Javascript and treated as ES6 Modules for extensibility. k6</code> has a concept of Virtual Users</a> to mimic a real-world user. Each VU runs the “script” in an isolated self-contained JS runtime using Goja</a>. Now obviously at this point, if speed is your utmost concern to generate very heavy load tests, I guess wrk</code> is your only real choice as invoking a JS runtime inside Go won’t be super fast. But for most use-cases and people, like my case, this will just be fine.</p> Basic Usage#</a></h2> Anyway, I quickly grokked the docs and copy-pasted some examples and modified them to what I needed. I was able to run a basic load test running very quickly and admired the simplicity here. It generated some p90, p95 etc stats which were helpful to look at. Here’s a basic example of how the script looks:</p> import</span> http</span> from</span> '</span>k6/http</span>'</span>;</span></span> import</span> {</span> check</span>,</span> sleep</span> }</span> from</span> '</span>k6</span>'</span>;</span></span> import</span> {</span> Rate</span> }</span> from</span> '</span>k6/metrics</span>'</span>;</span></span> import</span> {</span> uuidv4</span> }</span> from</span> "</span>https://jslib.k6.io/k6-utils/1.0.0/index.js</span>"</span>;</span></span> </span> export</span> const</span> errorRate</span> =</span> new</span> Rate</span>(</span>'</span>errors</span>'</span>)</span>;</span></span> </span> export</span> default</span> function</span> (</span>)</span> {</span></span> const</span> url</span> =</span> '</span>https://httpbin.org/post</span>'</span>;</span></span> const</span> params</span> =</span> {</span></span> headers</span>:</span> {</span></span> '</span>Content-Type</span>'</span>:</span> '</span>application/x-www-form-urlencoded</span>'</span>,</span></span> }</span>,</span></span> }</span>;</span></span> </span> const</span> data</span> =</span> {</span></span> custname</span>:</span> "</span>hello</span>"</span>,</span></span> comments</span>:</span> uuidv4</span>(</span>)</span>,</span></span> }</span>;</span></span> check</span>(</span>http</span>.</span>post</span>(</span>url</span>,</span>data</span>,</span> params</span>)</span>,</span> {</span></span> '</span>status is 200</span>'</span>:</span> (</span>r</span>)</span> =></span> r</span>.</span>status</span> ==</span> 200</span>,</span></span> }</span>)</span> \|\|</span> errorRate</span>.</span>add</span>(</span>1</span>)</span>;</span></span> </span> sleep</span>(</span>0</span>.</span>5</span>)</span>;</span></span> }</span></span></code></pre> To run it:</strong></p> k6</span> run</span> -</span>d</span> 10s</span> -</span>u</span> 10</span> httpbin_load.js</span></span></code></pre> (Here -d</code> is for the duration to run the test and -u</code> is to specify Virtual User</code>)</p> Explanation:</strong></p> It’s an HTTP POST</code> request with some form data to https://httpbin.org/post</a></li> We use the uuid</a> function because the JS stdlib is great</em> at providing basic helper methods (/s)</li> We define a check for HTTP status code as 200. Later we’ll see how to add more real-world checks under heavy load.</li> We have a sleep</a> function to pause a little bit before each iteration. This is pretty important as leaving sleep</code> is akin to a user pressing F5 on a browser non-stop and you’d probably not want your load tests to be that aggressive. Read docs</a> for more info.</li> </ul> Output:</strong></p> ❯ k6 run -d 30s -u 10 test.js </span></span> </span> /\ \|‾‾\| /‾‾/ /‾‾/ </span></span> /\ / \ \| \|/ / / / </span></span> / \/ \ \| ( / ‾‾\ </span></span> / \ \| \|\ \ \| (‾) \| </span></span> / __________ \ \|__\| \__\ \_____/ .io</span></span> </span> execution: local</span></span> script: test.js</span></span> output: -</span></span> </span> scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):</span></span> * default: 10 looping VUs for 30s (gracefulStop: 30s)</span></span> </span> </span> running (0m30.6s), 00/10 VUs, 387 complete and 0 interrupted iterations</span></span> default ✓ [======================================] 10 VUs 30s</span></span> </span> ✓ status is 200</span></span> </span> checks.........................: 100.00% ✓ 387 ✗ 0 </span></span> data_received..................: 318 kB 10 kB/s</span></span> data_sent......................: 78 kB 2.5 kB/s</span></span> http_req_blocked...............: avg=28.65ms min=210ns med=857ns max=1.1s p(90)=1.49µs p(95)=1.68µs </span></span> http_req_connecting............: avg=7.77ms min=0s med=0s max=301.24ms p(90)=0s p(95)=0s </span></span> http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms </span></span> { expected_response:true }...: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms </span></span> http_req_failed................: 0.00% ✓ 0 ✗ 387 </span></span> http_req_receiving.............: avg=177.35µs min=39.43µs med=170.18µs max=604.22µs p(90)=259.75µs p(95)=285.73µs</span></span> http_req_sending...............: avg=268.41µs min=48.62µs med=216.61µs max=7.95ms p(90)=334.94µs p(95)=448.21µs</span></span> http_req_tls_handshaking.......: avg=15.87ms min=0s med=0s max=615.25ms p(90)=0s p(95)=0s </span></span> http_req_waiting...............: avg=253.61ms min=215.34ms med=230.77ms max=826.57ms p(90)=316.98ms p(95)=324.69ms</span></span> http_reqs......................: 387 12.666038/s</span></span> iteration_duration.............: avg=784ms min=717ms med=732.43ms max=1.92s p(90)=820.35ms p(95)=926.87ms</span></span> iterations.....................: 387 12.666038/s</span></span> vus............................: 10 min=10 max=10</span></span> vus_max........................: 10 min=10 max=10</span></span></code></pre> Things to look for:</strong></p> From the above output, I think these 2 metrics are the most important to look at:</p> http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms </span></span> http_reqs......................: 387 12.666038/s</span></span></code></pre> We see the total requests sent in 30s were 387</code> and the p95</code> response time is 325.1ms</code>.</p> Testing some real-world scenarios#</a></h2> This was a really simple example but we can add some more scenarios to mimic real-world checks. Let’s tweak the script to</p> Go from 1 to 10 users in 10s.</li> Stay at 10 users for 5s.</li> Ramp down to 1 user for the next 15s.</li> Have a threshold of not exceeding 500ms as p95.</li> Have a threshold for the count of non 200 OK</code> responses.</li> </ul> The above script now becomes:</p> import</span> http</span> from</span> '</span>k6/http</span>'</span>;</span></span> import</span> {</span> check</span>,</span> sleep</span> }</span> from</span> '</span>k6</span>'</span>;</span></span> import</span> {</span> Rate</span> }</span> from</span> '</span>k6/metrics</span>'</span>;</span></span> import</span> {</span> uuidv4</span> }</span> from</span> "</span>https://jslib.k6.io/k6-utils/1.0.0/index.js</span>"</span>;</span></span> </span> export</span> const</span> errorRate</span> =</span> new</span> Rate</span>(</span>'</span>non_200_requests</span>'</span>)</span>;</span></span> </span> export</span> let</span> options</span> =</span> {</span></span> stages</span>:</span> [</span></span> //</span> Ramp-up from 1 to 10 VUs in 10s.</span></span> {</span> duration</span>:</span> "</span>10s</span>"</span>,</span> target</span>:</span> 10</span> }</span>,</span></span> </span> //</span> Stay at rest on 10 VUs for 5s.</span></span> {</span> duration</span>:</span> "</span>5s</span>"</span>,</span> target</span>:</span> 10</span> }</span>,</span></span> </span> //</span> Linearly ramp down from 10 to 0 VUs over the last 15s.</span></span> {</span> duration</span>:</span> "</span>15s</span>"</span>,</span> target</span>:</span> 0</span> }</span></span> ]</span>,</span></span> thresholds</span>:</span> {</span></span> //</span> We want the 95th percentile of all HTTP request durations to be less than 500ms</span></span> "</span>http_req_duration</span>"</span>:</span> [</span>"</span>p(95)<500</span>"</span>]</span>,</span></span> //</span> Thresholds based on the custom metric `non_200_requests`.</span></span> "</span>non_200_requests</span>"</span>:</span> [</span></span> //</span> Global failure rate should be less than 1%.</span></span> "</span>rate<0.01</span>"</span>,</span></span> //</span> Abort the test early if it climbs over 5%.</span></span> {</span> threshold</span>:</span> "</span>rate<=0.05</span>"</span>,</span> abortOnFail</span>:</span> true</span> }</span>,</span></span> ]</span>,</span></span> }</span>,</span></span> }</span>;</span></span> </span> export</span> default</span> function</span> (</span>)</span> {</span></span> const</span> url</span> =</span> '</span>https://httpbin.org/post</span>'</span>;</span></span> const</span> params</span> =</span> {</span></span> headers</span>:</span> {</span></span> '</span>Content-Type</span>'</span>:</span> '</span>application/x-www-form-urlencoded</span>'</span>,</span></span> }</span>,</span></span> }</span>;</span></span> </span> const</span> data</span> =</span> {</span></span> custname</span>:</span> "</span>hello</span>"</span>,</span></span> comments</span>:</span> uuidv4</span>(</span>)</span>,</span></span> }</span>;</span></span> check</span>(</span>http</span>.</span>post</span>(</span>url</span>,</span>data</span>,</span> params</span>)</span>,</span> {</span></span> '</span>status is 200</span>'</span>:</span> (</span>r</span>)</span> =></span> r</span>.</span>status</span> ==</span> 200</span>,</span></span> }</span>)</span> \|\|</span> errorRate</span>.</span>add</span>(</span>1</span>)</span>;</span></span> </span> sleep</span>(</span>Math</span>.</span>random</span>(</span>)</span> </span> 1</span> +</span> 1</span>)</span>;</span> //</span> Random sleep between 1s and 2s.</span></span> }</span></span></code></pre> Run with k6 run test.js</code>:</p> 2.21.0 on ☁️ (ap-south-1) took 24s </span></span> ❯ k6 run test.js</span></span> </span> /\ \|‾‾\| /‾‾/ /‾‾/ </span></span> /\ / \ \| \|/ / / / </span></span> / \/ \ \| ( / ‾‾\ </span></span> / \ \| \|\ \ \| (‾) \| </span></span> / __________ \ \|__\| \__\ \_____/ .io</span></span> </span> execution: local</span></span> script: test.js</span></span> output: -</span></span> </span> scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):</span></span> default: Up to 10 looping VUs for 30s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)</span></span> </span> </span> running (0m30.3s), 00/10 VUs, 105 complete and 0 interrupted iterations</span></span> default ✓ [======================================] 00/10 VUs 30s</span></span> </span> ✓ status is 200</span></span> </span> checks.........................: 100.00% ✓ 105 ✗ 0 </span></span> data_received..................: 126 kB 4.2 kB/s</span></span> data_sent......................: 25 kB 841 B/s</span></span> http_req_blocked...............: avg=73.69ms min=292ns med=761ns max=1.02s p(90)=1.54µs p(95)=688.76ms</span></span> http_req_connecting............: avg=21.67ms min=0s med=0s max=245.46ms p(90)=0s p(95)=223.77ms</span></span> ✓ http_req_duration..............: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms</span></span> { expected_response:true }...: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms</span></span> http_req_failed................: 0.00% ✓ 0 ✗ 105 </span></span> http_req_receiving.............: avg=177.85µs min=103.82µs med=163.76µs max=366.5µs p(90)=235.86µs p(95)=266.39µs</span></span> http_req_sending...............: avg=258.17µs min=96.92µs med=215.88µs max=958.67µs p(90)=410.05µs p(95)=487.18µs</span></span> http_req_tls_handshaking.......: avg=50.22ms min=0s med=0s max=614.14ms p(90)=0s p(95)=460.3ms </span></span> http_req_waiting...............: avg=252.29ms min=215.17ms med=229.69ms max=559.86ms p(90)=298.42ms p(95)=406.52ms</span></span> http_reqs......................: 105 3.471037/s</span></span> iteration_duration.............: avg=1.84s min=1.22s med=1.85s max=3.02s p(90)=2.22s p(95)=2.46s </span></span> iterations.....................: 105 3.471037/s</span></span> vus............................: 1 min=1 max=10</span></span> vus_max........................: 10 min=10 max=10</span></span></code></pre> We can see that all the checks passed without breaching any thresholds we’d set.</p> Some important points</strong>:</p> In my local environment, I stress tested my service with 10k VUs which is quite a high number for the service but it was good to see it hold under extreme conditions as well. An important thing to note if you are spawning many VUs is that ulimit</code> number should be high. This is described in their docs</a> as well.</p> </li> To debug the HTTP response you can run with --http-debug="full"</code> flag and get the verbose output for debugging.</p> </li> </ul> Summary#</a></h2> I’ve barely scratched the surface of what this tool does. You can export metrics to various data sources, add a lot more checks on the response code, use it with GRPC or WebSockets as well.</p> Overall pretty happy with this tool and I am going to use more of it for future projects.</p> References#</a></h3> https://k6.io/blog/comparing-best-open-source-load-testing-tools/</a></li> https://k6.io/our-beliefs/#simple-testing-is-better-than-no-testing</a></li> https://k6.io/docs/</a></li> https://github.com/grafana/k6</a></li> </ul> Fin</p> Analyzing CoreDNS logs with Clickhouse and Vector 2021-06-05T00:00:00+05:30 I’ve been toying around Clickhouse</a> and Vector</a> at my day job and find both of these tools pretty interesting at what they do. A short summary for those unaware of these tools:</p> Vector</strong> helps you build a pipeline for collecting, transforming and processing different kinds of observability data (logs and metrics).</li> Clickhouse</strong> is a columnar based DB used as a warehousing tool for generating reports and analytics.</li> </ul> Now, for the context, I use coredns</code> on my personal dev machine as it supports Split DNS (routing certain zones to a particular resolver) which I need for accessing internal domains at work. Yep, systemd-resolved</code> can also do this, but I find coredns</code> easier to configure and manage with OpenVPN as well.</p> Anyway, so one random evening, I got the idea of dumping CoreDNS Logs to Clickhouse. Maaaybe</em> I was still hungover from the Vector/Clickhouse work I was doing at work but nevertheless I was interested in it.</p> Overview#</a></h2> Gather logs from CoreDNS.</li> Transform the logs with regex</code> and construct a payload for Clickhouse.</li> Write a schema for the logs table.</li> Dump to Clickhouse.</li> </ul> Here’s how the pipeline looks:</p> </p> Collecting Logs#</a></h2> First, let’s look at how the raw logs are structured by CoreDNS:</p> [</span>INFO</span>]</span> 127.0.0.1:55678 - 21963 </span>"</span>A IN xpui.app.spotify.com. udp 38 false 512</span>"</span> NXDOMAIN qr,rd,ra 121 0.061416978s</span></span> [</span>INFO</span>]</span> 127.0.0.1:59333 - 22742 </span>"</span>AAAA IN incoming.telemetry.mozilla.org. udp 48 false 512</span>"</span> NOERROR qr,aa,rd,ra 106 0.049235139s</span></span> [</span>INFO</span>]</span> 127.0.0.1:39609 - 47247 </span>"</span>AAAA IN ping.archlinux.org. udp 36 false 512</span>"</span> NOERROR qr,rd,ra 140 0.056721154s</span></span></code></pre> Vector provides a variety of sources</a> to collect these logs. Since I am running coredns</code> as a Docker container, the following config shows how to collect logs from a particular container:</p> #</span> Filter coredns logs from Docker logs</span></span> [</span>sources</span>.</span>coredns_logs</span>]</span></span> type</span> =</span> "</span>docker_logs</span>"</span> #</span> required</span></span> docker_host</span> =</span> "</span>unix:///var/run/docker.sock</span>"</span></span> include_images</span> =</span> [</span>"</span>coredns/coredns</span>"</span>]</span> #</span> optional, no default</span></span></code></pre> The above config basically specifies a Docker Host variable and an image name filter. Vector talks to the Docker API over a unix</code> socket and gathers metadata about the container (like container_created_at</code>, container_name</code>, label</code> etc).</p> After collecting logs from Vector, it enriches with the following metadata.</p> {</span></span> "</span>container_created_at</span>"</span>:</span> "</span>2021-06-04T14:18:03.967143133Z</span>"</span>,</span></span> "</span>container_id</span>"</span>:</span> "</span>00c5c4d36ea5b4772b517d3cca7d397c92f72be2a2bf45bb8c430f717fbd331e</span>"</span>,</span></span> "</span>container_name</span>"</span>:</span> "</span>coredns_coredns_1</span>"</span>,</span></span> "</span>host</span>"</span>:</span> "</span>iris</span>"</span>,</span></span> "</span>image</span>"</span>:</span> "</span>coredns/coredns</span>"</span>,</span></span> "</span>label</span>"</span>:</span> {</span></span> "</span>com.docker.compose.config-hash</span>"</span>:</span> "</span>928d71143c2af6553d551dbbf14140304d53f92378746454fbfeb0382a896d5b</span>"</span>,</span></span> "</span>com.docker.compose.container-number</span>"</span>:</span> "</span>1</span>"</span>,</span></span> "</span>com.docker.compose.oneoff</span>"</span>:</span> "</span>False</span>"</span>,</span></span> "</span>com.docker.compose.project</span>"</span>:</span> "</span>coredns</span>"</span>,</span></span> "</span>com.docker.compose.project.config_files</span>"</span>:</span> "</span>/home/karan/Code/Infra/coredns/hydra-vpn-compose.yml</span>"</span>,</span></span> "</span>com.docker.compose.project.working_dir</span>"</span>:</span> "</span>/home/karan/Code/Infra/coredns</span>"</span>,</span></span> "</span>com.docker.compose.service</span>"</span>:</span> "</span>coredns</span>"</span>,</span></span> "</span>com.docker.compose.version</span>"</span>:</span> "</span>1.29.2</span>"</span></span> }</span>,</span></span> "</span>message</span>"</span>:</span> "</span>[INFO] 127.0.0.1:38266 - 20341 </span>\"</span>A IN open.spotify.com. udp 34 false 512</span>\"</span> NOERROR qr,rd,ra 160 0.300268123s</span>"</span>,</span></span> "</span>source_type</span>"</span>:</span> "</span>docker</span>"</span>,</span></span> "</span>stream</span>"</span>:</span> "</span>stdout</span>"</span>,</span></span> "</span>timestamp</span>"</span>:</span> "</span>2021-06-04T16:13:07.454601872Z</span>"</span></span> }</span></span></code></pre> (NOTE: I am using console</a> sink to dump these logs to STDOUT</code>. It’s pretty handy for inspecting logs).</p> #</span> Print parsed logs to stdout</span></span> [</span>sinks.print</span>]</span></span> type</span> =</span> "</span>console</span>"</span></span> inputs</span> =</span> [</span>"</span>coredns_logs</span>"</span>]</span></span> encoding.codec</span> =</span> "</span>json</span>"</span></span></code></pre> As you can see from the above JSON object, Vector has transformed the log with its own Data Model. The log that we care about now is inside .message</code> key. It’s nice to have other metadata as well.</p> Transforming the logs#</a></h3> Our objectives at this step:</p> Discard unused fields. We don’t really care about container metadata for this mini-project.</li> Parse the message</code> field with regex</code> so they can be stored in individual columns in our Clickhouse table.</li> </ul> Now, CoreDNS can emit two kinds of logs (INFO</code> and ERROR</code>). The error usually happens when the upstream resolver is unreachable or there’s an issue with any of the CoreDNS plugins.</p> We need to write regex for both cases:</p> INFO</code> logs:</li> </ol> \[(?P</span><level>[^</span>]]+)]</span>\s</span>(</span>?</span>P</span><</span>server_addr</span>></span>[</span>^:</span>]</span>+):(</span>?</span>P</span><</span>server_port</span>></span>\S</span>+)</span>\s</span>+-</span>\s</span>+</span>(</span>?</span>P</span><</span>id</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>type</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>class</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>name</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>proto</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>size</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>do</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>bufsize</span>></span>[</span>^</span>"</span>]+)</span>"</span>\s</span>+</span>(</span>?</span>P</span><</span>rcode</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>rflags</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>rsize</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>duration</span>></span>[</span>\d</span>\.</span>]</span>+</span>)</span>.</span></span></span></code></pre> </p> ERROR</code> logs:</li> </ol> \[(?P</span><level>ERROR</span>)]</span>\s</span>+</span>(</span>?</span>P</span><</span>component</span>></span>plugin</span>\/</span>errors</span>)</span>:</span>\s</span>+</span>(</span>?</span>P</span><</span>code</span>></span>\S</span>)</span>+</span>\s</span>+</span>(</span>?</span>P</span><</span>name</span>></span>\S</span>+</span>)</span>\s</span>+</span>(</span>?</span>P</span><</span>type</span>></span>[</span>^:</span>]</span></span>)</span>:</span>\s</span>+</span>(</span>?</span>P</span><</span>error_msg</span>></span>.</span></span>)</span></span></code></pre> </p> Combining a bunch of other things to remove some fields and constructing the final payload, the config looks like this:</p> </span> #</span> Parse coredns logs</span></span> [</span>transforms</span>.</span>parse_logs</span>]</span></span> type</span> =</span> "</span>remap</span>"</span></span> inputs</span> =</span> [</span>"</span>coredns_logs</span>"</span>]</span></span> source</span> =</span> '''</span></span> # parse the log event.</span></span> ts = .timestamp</span></span> log,err = parse_regex(.message,r'\[(?P<level>[^]]+)]\s(?P<server_addr>[^:]+):(?P<server_port>\S+)\s+-\s+(?P<id>\S+)\s+"(?P<type>\S+)\s+(?P<class>\S+)\s+(?P<name>\S+)\s+(?P<proto>\S+)\s+(?P<size>\S+)\s+(?P<do>\S+)\s+(?P<bufsize>[^"]+)"\s+(?P<rcode>\S+)\s+(?P<rflags>\S+)\s+(?P<rsize>\S+)\s+(?P<duration>[\d\.]+).')</span></span> if err !=null {</span></span> # capture the error log. If the error log also fails to get parsed, the log event is dropped.</span></span> log = parse_regex!(.message,r'\[(?P<level>ERROR)]\s+(?P<component>plugin/errors):\s+(?P<code>\S)+\s+(?P<name>\S+)\s+(?P<type>[^:]):\s+(?P<error_msg>.)')</span></span> }</span></span> . = log</span></span> # add timestamp</span></span> .timestamp = ts</span></span> # remove fields we dont care about</span></span> del(.do)</span></span> '''</span></span> drop_on_error</span> =</span> true</span></span></code></pre> Apart from regex matching, we store the timestamp</code> as received from Vector (since CoreDNS logs don’t contain any timestamp information). We delete some fields that we don’t care about.</p> Vector uses a powerful DSL (called VRL</a>) to do such kind of transformations on the fly. It has a lot of functions to do almost any kind of transformation to your original event payload. You can invoke vector vrl</code> from the terminal and get a shell to write the above transformations and debug quickly. It proved to be really useful when dealing with such a long regex pattern.</p> Storing in Clickhouse#</a></h2> Finally we get to the part where we need to dump these logs to our Clickhouse DB. Here’s the schema for the table where we will be storing these records:</p> CREATE</span> DATABASE</span> IF</span> NOT</span> EXISTS</span> `</span>coredns</span>`</span>;</span></span> </span> CREATE</span> TABLE</span> IF</span> NOT</span> EXISTS</span> `</span>coredns</span>`</span>.</span>`</span>logs</span>`</span> (</span></span> `</span>timestamp</span>`</span> DateTime</span>(</span>'</span>Asia/Kolkata</span>'</span>),</span></span> `</span>bufsize</span>`</span> Int32,</span></span> `</span>class</span>`</span> LowCardinality(String),</span></span> `</span>duration</span>`</span> Float64,</span></span> `</span>id</span>`</span> Int32,</span></span> `</span>level</span>`</span> LowCardinality(String),</span></span> `</span>name</span>`</span> String,</span></span> `</span>proto</span>`</span> LowCardinality(String),</span></span> `</span>rcode</span>`</span> LowCardinality(String),</span></span> `</span>rflags</span>`</span> String,</span></span> `</span>server_addr</span>`</span> String,</span></span> `</span>server_port</span>`</span> Int32,</span></span> `</span>rsize</span>`</span> Int32,</span></span> `</span>size</span>`</span> Int32,</span></span> `</span>type</span>`</span> LowCardinality(String)</span></span> ) ENGINE </span>=</span> MergeTree</span>(</span>)</span></span> PARTITION</span> BY</span> toYYYYMM(</span>timestamp</span>)</span></span> ORDER BY</span> toYYYYMMDD(</span>timestamp</span>)</span></span> TTL </span>timestamp</span> +</span> INTERVAL </span>1</span> WEEK</span>;</span></span></code></pre> Key things to note:</strong></p> LowCardinality</code></a> is used for columns where the data is predictable to reduce the disk space used.</li> Clickhouse uses the sort key as the primary key if unspecified. This is the default behaviour.</li> TTL for the records is set to 1 week. After 1 week all the records will be purged. Since this is my dev machine, I don’t really care about a higher TTL.</li> This also means that the partition is never really put to use since I am partitioning by month but logs are being deleted every week. At this scale, it doesn’t really make sense to even have it, but I just included it for posterity.</li> </ul> UPDATE</strong>:</p> Clickhouse on Twitter clarified</a> that ORDER BY timestamp</code> will have better performance in this context. Usually if your queries are “last 1h”, “last 5m” based, it is better to not store the the sort key as YYYYMMDD</code> format.</p> </p> Now, we need to instruct Vector to send these logs to Clickhouse:</p> </span> [</span>sinks</span>.</span>coredns_logs_clickhouse_output</span>]</span></span> type</span> =</span> "</span>clickhouse</span>"</span></span> inputs</span> =</span> [</span>"</span>parse_logs</span>"</span>]</span></span> compression</span> =</span> "</span>gzip</span>"</span></span> database</span> =</span> "</span>coredns</span>"</span></span> endpoint</span> =</span> "</span>http://localhost:8123</span>"</span></span> table</span> =</span> "</span>logs</span>"</span></span> encoding</span>.</span>timestamp_format</span> =</span> "</span>unix</span>"</span></span> batch</span>.</span>timeout_secs</span> =</span> 10</span></span></code></pre> Clickhouse offers an HTTP API (which is running on port 8123 by default). Vector takes the input from the previous step (parse_logs</code> transformation) and sends it to Clickhouse over the HTTP interface. Clickhouse stores datetimes in UNIX, so before sending the data, Vector can encode certain fields in the payload to a different data type as well (isn’t that cool 😎)</p> Query Examples#</a></h2> I’ve been running this pipeline for 3-4 days, so I have a decent amount of data collected to show for the blog post.</p> Total Count of Queries</p> SELECT</span> count</span>(</span></span>)</span></span> FROM</span> coredns</span>.</span>logs</span></span> </span> ┌─</span>count</span>(</span>)─┐</span></span> │ </span>16774</span> │</span></span> └─────────┘</span></span></code></pre></li> Top Query Types</p> SELECT</span></span> count</span>(</span></span>) </span>AS</span> total,</span></span> type</span></span> FROM</span> coredns</span>.</span>logs</span></span> GROUP BY</span> type</span></span> ORDER BY</span> total </span>DESC</span></span> LIMIT</span> 5</span></span> </span> ┌─total─┬─</span>type</span>─┐</span></span> │ </span>9931</span> │ A │</span></span> │ </span>6852</span> │ AAAA │</span></span> └───────┴──────┘</span></span></code></pre></li> Top Query Names</p> SELECT</span></span> count</span>(</span></span>) </span>AS</span> total,</span></span> name</span></span> FROM</span> coredns</span>.</span>logs</span></span> GROUP BY</span> name</span></span> ORDER BY</span> total </span>DESC</span></span> LIMIT</span> 5</span></span> </span> ┌─total─┬─</span>name</span>────────────────────────────┐</span></span> │ </span>2513</span> │ </span>ping</span>.</span>archlinux</span>.org. │</span></span> │ </span>1868</span> │ </span>ws</span>.</span>todoist</span>.com. │</span></span> │ </span>1011</span> │ </span>incoming</span>.</span>telemetry</span>.</span>mozilla</span>.</span>org</span>. │</span></span> │ </span>802</span> │ </span>vortex</span>.</span>data</span>.</span>microsoft</span>.</span>com</span>. │</span></span> │ </span>707</span> │ logs</span>-</span>01</span>.</span>loggly</span>.</span>com</span>. │</span></span> └───────┴─────────────────────────────────┘</span></span></code></pre></li> Max/Min duration</p> SELECT</span></span> max</span>(</span>duration) </span>AS</span> max,</span></span> min</span>(</span>duration) </span>AS</span> min</span></span> FROM</span> coredns</span>.</span>logs</span></span> FORMAT Vertical</span></span> </span> Row</span> 1</span>:</span></span> ──────</span></span> max: </span>4</span>.</span>056606352</span></span> min: </span>0</span>.</span>000020837</span></span></code></pre></li> Top TLDs Queried</p> SELECT</span></span> count</span>(</span>name</span>) </span>AS</span> total,</span></span> topLevelDomain(</span>substring</span>(</span>name</span>, </span>1</span>, </span>-</span>1</span>)) </span>AS</span> tld</span></span> FROM</span> coredns</span>.</span>logs</span></span> GROUP BY</span> tld</span></span> ORDER BY</span> total </span>DESC</span></span> LIMIT</span> 10</span></span> </span> ┌─total─┬─tld──┐</span></span> │ </span>10666</span> │ com │</span></span> │ </span>3950</span> │ org │</span></span> │ </span>671</span> │ net │</span></span> │ </span>346</span> │ so │</span></span> │ </span>288</span> │ tech │</span></span> │ </span>279</span> │ </span>io</span> │</span></span> │ </span>190</span> │ co │</span></span> │ </span>167</span> │ dev │</span></span> │ </span>82</span> │ arpa │</span></span> │ </span>43</span> │ </span>in</span> │</span></span> └───────┴──────┘</span></span></code></pre></li> </ul> Well, that’s all I could think of really. If you’ve some more interesting analysis to get from this data, let me know!</p> Summary#</a></h2> The intention behind writing this post is to give an overview of how the entire log collection and processing pipeline works. Using Vector has been an amazing experience however the sad bit is that I don’t know Rust and I cannot contribute to some of the issues I’ve opened (even though they are presumably trivial). Maybe I should pick up Rust, finally? 🤭</p> Thanks for reading!</p> How I take Notes 2021-04-14T08:10:55+05:30 Over the past 2-3 years, note-taking apps have become all the rage. Note-taking is an extremely subjective topic and a lot of it depends on the individual’s workflow. There’s no one size fits all and maybe that justifies the ever-expanding landscape of such apps. I’ve tried a few popular ones (Notion, Roam Research) in the recent past but never really quite stuck to any after the initial hype phase.</p> I even collaborated with @iamd3vil</a> to make our own version</a> of the Zettelkasten based note-taking app. I found the Zettelkasten system to be really useful on paper but then again, I didn’t use it after a few weeks.</p> Tried the old school way of Bullet Journal (and I did end up liking it quite a lot) but it was not so practical in many cases (like documenting code snippets, URLs etc).</p> Disgruntled with all the options, I just had a simple folder on my laptop with some markdown files in it. It was a stop-gap solution until I found something better.</p> TL;DR</strong>: I’ve been through a pendulum phase of finding a new note-taking app, wasting time to set it up the “proper way” and then just end up not using it.</p> Enter Joplin#</a></h2> </p> Thanks to @shantanugoel</a> who introduced me to Joplin</a>. He’s a heavy user</a> of it as well and that gave me some confidence to try it out. I initially disliked it because of not-so-great-looking</em> UI theme and how it essentially looked just</em> an editor. Admittedly, I was proven wrong quickly in my initial judgement. As I gave more time to it, I noticed I kept coming back to Joplin “naturally” and stuck through it because it’s so damn simple to use. Notion, for people who have tried it would know it complicates a lot of simple tasks. You need to create databases to render a simple table, every component is a “block” (a new page) and yes it’s slow (although they are working on it to make it better</a>, just to be fair). Notion focuses a lot on team collaboration features, which I didn’t need for my “personal” note-taking system.</p> TL;DR</strong>: Joplin is fast, it’s open-source, it’s based on Markdown, and it’s simple to use. A tool you just forget that it exists, because it becomes a natural extension to your workflow. It has a great plugin system that you can use to extend it and build your own utilities on top of it. The search is based on sqlite3</code> FTS which is pretty awesome!</p> Workflow#</a></h2> Joplin revolves around the concept of notebooks</strong>. Notebooks are broader categories for your content and you can nest multiple subnotebooks for specific categories.</p> I’ve the following notebooks and subnotebooks in my Joplin setup:</p> - Bookmarks</span></span> - Twitter Threads</span></span> - HN Threads</span></span> - Articles</span></span> - Design Inspo</span></span> - Youtube Videos</span></span> - Inbox</span></span> - Links</span></span> - Adhoc Notes</span></span> - Personal</span></span> - Finance</span></span> - Dev Setup</span></span> - OSS Ideas</span></span> - Self Hosted Setup</span></span> - Work</span></span> - Org-Stuff</span></span> - Redacted</span></span></code></pre> Over the week, I primarily use the Inbox/Adhoc Notes</code> notebook as a brain dump. I don’t focus much on the structure, the aim is to get the content out and stored. I’m also someone who doesn’t like to keep more than 5 browser tabs open at any time, so I use Links</code> notebook with Joplin’s Web Clipper</a> service to store these links to read later.</p> Every weekend, I clean up these notebooks to achieve “Inbox Zero”. The idea is to move all these ad-hoc notes to their proper notebooks, annotated with tags. All the useful links are moved in the appropriate Bookmarks/...</code> notebook as well. This helps me find stuff quicker at a later time.</p> I heavily use Tags in all my notebooks, which allows me to have a unified view of different kind of stuff I’ve. For example “golang” tag in my Work notes and Personal notes, allows me to see all the “golang” stuff together in one place.</p> For stuff that can be shared publicly, I basically copy-paste those notes in my public wiki</a> as well. This allows me to share snippets/commands with others, which Joplin cannot do.</p> Synchronisation#</a></h2> Joplin provides a bunch of different sync</a> options. I’ve tried Dropbox, Nextcloud and AWS S3 targets in the past but off late there’s a new sync option, Joplin Server</a> which provides native sync for Joplin files. I found this option to be the best so far because Dropbox/OneNote etc have API limits and syncing on an initial device with lots of notes will be time-consuming.</p> I self-host this Joplin Sync Server on my server and have configured the Android app and the Desktop app to use this server endpoint as the sync target</code>. So far so good, although it’s relatively a newer sync option so it’s pertinent to have alternate backups.</p> Backups and Export#</a></h2> Joplin stores all the files locally on a device in a sqlite3</code> DB. It can export notes in markdown/HTML format however the file names are all named with the id</code> of the note (and not really the title of the note). I found this to be a bit of a drawback, however, one can quickly whip up a small Python script to fix this using the Joplin API.</p> Joplin also has the option to export all of the notes and notebooks with their metadata (Geolocation, creation time etc), tags in a custom format called Joplin Export File (JEX). This option is pretty convenient to re-import in a new Joplin installation as well.</p> Sidenote for people using Joplin Server: Once https://github.com/laurent22/joplin/issues/4836 gets resolved, it’ll be possible to do joplin sync</code> and cron it, just like other sync targets.</p> Support#</a></h2> Joplin is 100% FOSS and is actively developed by @laurent22</code></a> and a few other regular contributors. I contribute $5/mo</code> to laurent22 via Github Sponsors. It’s more or less the same amount that most note-taking apps charge for personal use as well, so this is just me expressing gratitude for building such a lovely software for the world to use it. I’m not sure of the motivations of laurent22</code> behind building this and I don’t wanna incorrectly assume anything either, but I guess some amount of financial incentive makes the whole deal sustainable for the open-source ecosystem.</p> Thanks for reading! It’s been around a year that I am using Joplin and I posted this blog post only after really really using it a lot.</p> I’d love to know about your note-taking setups too, so please reach out to me on the usual channels that I’m available on and feel free to discuss!</p> Fin!</p> (Bonus Section) Why Not Obsidian#</a></h4> Yes, Obsidian is comparable to Joplin in a lot of ways. However there’s a term in the license</a> of Obsidian for personal use that makes it impossible</strong> to use it for your work stuff:</p> You need to pay for Obsidian if and only if you use it for revenue-generating, work-related activities in a company that has two or more people. Get a commercial license for each user if that’s the case</p> </blockquote> I don’t have a problem for “paying” for software but such kinda licenses are just BS.</p> Running Nomad for home server 2021-02-14T08:10:55+05:30 It’s been a long time since I’ve written a post on Hydra (my home server). I use Hydra as a testbed to learn new tools, workflows and it just gives me joy to self-host applications while learning something in return.</p> History#</a></h2> A brief history of how Hydra’s</a> setup evolved over time:</p> 2019</a>:</p> A pretty minimal K3s setup deployed on 2 RPi4 nodes. I couldn’t continue with this setup because: Some of the apps didn’t have ARM-based image (this was 2019, pre M1 hype era).</li> Didn’t want to risk deploying persistent workloads on RPi.</li> A lot of tooling to deploy workloads was missing (storing env variables for eg.).</li> It was so boring to write YAML (that I also did at work). Didn’t give me joy.</li> </ul> </li> </ul> 2020 First Half</a>:</p> RPi 2x Nodes + K3s + DO Droplet. Tailscale for networking. This was a considerable step up from the previous setup. I deployed a DO node and added Node Labels</a> to deploy persistent workloads on DO Node only.</li> I used my own tooling Kubekutr</a> + Kustomize which helped with version control of my configs.</li> Took quite a bit of time to onboard new services. Got lazy, didn’t host much apart from initial 3-4 applications.</li> Writing long YAMLs. No joy.</li> </ul> </li> </ul> 2020 Second Half:</p> Single node on DO. Terraform for deploying Docker containers. I believe the third iteration nailed it for me. I kept the setup super simple, used Terraform for deploying workloads as Docker containers.</li> Used Terraform extensively for setting up the node, Cloudflare records, DO firewall rules.</li> Time to onboard new services reduced from a couple of hours to a few minutes. This was a huge win for me. I deployed around 10-15 new services to try it out on the server directly.</li> Writing HCL is actually a much better experience than YAML.</li> </ul> </li> </ul> Why Nomad#</a></h2> </p> Around a month back, Kailash</a> had asked about feedback on Nomad</a>. We at Zerodha</a> (India’s largest stock broker) are evaluating it to migrate our services to Nomad from Kubernetes (more on this later). It was almost 2 years since I last saw Nomad so it was definitely worth re-evaluating (esp since it hit 1.0 recently). I wanted to try out Nomad to answer a personal curiosity: What does it do differently than Kubernetes?</em> No better way than actually getting hands dirty, right?!</p> After following the brief tutorials from the official website</a> I felt confident to try it for actual workloads. In my previous setup, I was hosting quite a few applications (Pihole, Gitea, Grafana etc) and thought it’ll be a nice way to learn how Nomad works by deploying the same services in the Nomad cluster. And I came in with zero expectations, I already had a nice setup which was reliable and running for me. My experience with a local Nomad cluster was joyful, I was able to quickly go from 0->1 in less than 30 minutes. This BTW is a strong sign of how easy Nomad is to get started with as compared to K8s. The sheer amount of different concepts you’ve to register in your mind before you can even deploy a single container in a K8s cluster is bizarre. Nomad takes the easy way out here and simplified the concepts for developers into just three things:</p> job</span></span> \_ group</span></span> \_ task</span></span></code></pre> Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed.</li> Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You’ll want to use Groups for use-cases like a logging sidecar, reverse proxies etc.</li> Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java VM etc, defining the mount points, env variables, ports to be exposed etc.</li> </ul> If you’re coming from K8s you can think of Task as a Pod and Group as a Replicaset. There’s no equivalent to Job in K8s. BUT! The coolest part? You don’t have to familiarise yourself with all different types of Replicasets (Deployments, Daemonsets, Statefulsets) and different ways of configuring them.</p> Want to make a normal job as a periodic job in Nomad? Simply add the following block to your existing Job:</p> periodic</span> {</span></span> cron</span> =</span> "</span>@daily</span>"</span></span> }</span></span></code></pre> You want to make a service run as a batch job (on all Nomad nodes – the equivalent of Daemonset in K8s)? Simply make the following change to your existing job:</p> -</span>type="service"</span></span> +</span>type="batch"</span></span></code></pre> You see this</strong> is what I mean by the focus on UX. There are many many such examples which will leave a nice smile on your face if you’re coming from K8s background.</p> I’d recommend reading Internal Architecture</a> of Nomad if you want to understand this in-depth.</p> Architecture#</a></h2> Tech stack for Hydra:</p> Tailscale VPN: Serves as a mesh layer between my laptop/mobile and DO server. Useful for exposing internal services.</li> Caddy for reverse proxying and automatic SSL setup for all services. I run 2 instances of Caddy: Internal: Listens on Tailscale Network Interface. Reverse proxies all private services.</li> Public: Listens on DO’s Public IPv4 network interface. Reverse proxies all public-facing services.</li> </ul> </li> Terraform: Primary component to have IaC (Infra as Code). Modules to manage: Cloudflare DNS Zone and Records</li> DO Droplet, Firewall rules, SSH Keys, Floating IPs etc.</li> Nomad Jobs. Used for running workloads after templating env variables, config files in Nomad job files.</li> </ul> </li> </ul> Complexity of Nomad vs Kubernetes#</a></h2> </a></p> Nomad shines because it follows the UNIX philosophy of “Make each program do one thing well”. To put simply, Nomad is just</em> a workload orchestrator. It only is concerned about things like Bin Packing, scheduling decisions.</p> If you’re running heterogeneous workloads, running a server (or a set of servers) quickly becomes expensive. Hence orchestrators tend to make sense in this context. They tend to save costs by making it efficient to run a vast variety of workloads. This is all an orchestrator has to do really.</p> Nomad doesn’t interfere in your DNS setup, Service Discovery, secrets management mechanisms and pretty much anything else. If you read some of the posts at Kubernetes Failure Stories</a>, the most common reason for outages is Networking (DNS, ndots etc). A lot of marketing around K8s never talks about these things.</p> I always maintain “Day 0 is easy, Day N is the real test of your skills”. Anyone can deploy a workload to a K8s cluster, it’s always the Day N operations which involve debugging networking drops, mysterious container restarts, proper resource allocations and other such complex issues that require real skills and</strong> effort. It’s not as easy as kubectl apply -f</code> and my primary gripe is with people who miss out on this in their “marketing” pitches (obvious!).</p> When to use Nomad#</a></h2> Nomad hits the sweet spot of being operationally easy and functional. Nomad is a great choice if you want to:</p> Run not just containers but other forms of workloads.</li> Increase developer productivity by making it easier to deploy/onboard new services.</li> Consistent experience of deployment by testing the deployments locally.</li> (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp.</li> </ul> Nomad is available as a single binary. If you want to try it locally, all you need is sudo nomad agent -dev</code> and you’ll have a Nomad Server, Client running in dev mode along with a UI. This makes it easy for the developers to test out the deployments locally because there’s very little configuration difference between this and production deployment. Not to forget it’s super easy to self-host Nomad clusters. I’m yet to meet anyone who self hosts K8s clusters in production without a dedicated team babysitting it always.</p> Once you eliminate the “blackbox” components from your stack, life becomes easier for everyone.</p> When to not use Nomad#</a></h2> If you’re relying on custom controllers and operators. Operator Pattern</a> is a new way of managing large complex distributed systems (like databases, job queues etc). There are a lot of community built operators which help in reducing the effort to run these services. However, all of these are tied deeply into the “Kubernetes” ecosystem. If you find yourself running any of such operators, it’ll be tough (not impossible) to translate the same in Nomad ecosystem.</li> </ul> I genuinely</em> cannot think of any other reason to not use Nomad!</p> Practical Scenarios#</a></h2> Since I migrated a couple of workloads from my DO docker containers setup to Nomad, I’d demonstrate a few use cases which might be helpful if you want to start migrating your services to Nomad</p> Accessing a Web service with Reverse Proxy#</a></h3> Context: I’m running Caddy as a reverse proxy for all the services. Since we discussed earlier, Nomad only</strong> is concerned about scheduling, so how exactly do you do Service Discovery? You need Consul (or something like Consul, Nomad has no hard restrictions) to register a service name with it’s IP Address. Here’s how you can do that:</p> In the .task</code> section of your Nomad job spec, you need to register the service name with the port you’re registering and additional tags as metadata (optional):</p> service</span> {</span></span> name</span> =</span> "</span>gitea-web</span>"</span></span> tags</span> =</span> [</span>"</span>gitea</span>"</span>,</span> "</span>web</span>"</span>]</span></span> port</span> =</span> "</span>http</span>"</span></span> }</span></span></code></pre> Nomad’s template</a> uses consul-template</code> behind the scenes. This is a small utility which continuously watches for Consul/Vault keys and provides the ability to reload/restart your workloads if any of those keys change. It can also be used to discover</em> the address of the service registered in Consul. So here’s an example of Caddyfile</code> using Consul Template functions to pull the IP address of the upstream gitea-web</code> service:</p> git</span>.</span>mrkaran</span>.</span>dev</span> {</span></span> {</span>{</span> range service </span>"</span>gitea-web</span>"</span> }</span>}</span></span> reverse_proxy </span>{</span>{</span> .Address </span>}</span>}</span>:</span>{</span>{</span> .Port </span>}</span>}</span></span> {</span>{</span> end </span>}</span>}</span></span> }</span></span></code></pre> When a job is submitted to Nomad, a rendered template is mounted inside the container. You can define actions on what to do when the values change. For eg on a redeployment of Gitea container, the address will most likely change. We’d like Caddy to automatically restart with the new address configured in the Caddyfile in that case:</p> template</span> {</span></span> data</span> =</span> <<</span>EOF</span></span> ${</span>caddyfile_public</span>}</span></span> EOF</span></span> </span> destination</span> =</span> "</span>configs/Caddyfile</span>"</span> #</span> Rendered template.</span></span> </span> change_mode</span> =</span> "</span>restart</span>"</span></span> }</span></span></code></pre> Using change_mode</code></a> we can either send a signal</code> or restart the task altogether.</p> Binding to different network interfaces#</a></h3> I run a public instance of Gitea but I wanted to restrict the SSH access only to my Tailscale network. Nomad has an interesting feature host_network</code></a> which lets you bind different ports of a task on different network interfaces.</p> network</span> {</span></span> port</span> "http"</span> {</span></span> to</span> =</span> 3000</span></span> }</span></span> </span> port</span> "ssh"</span> {</span></span> to</span> =</span> 22</span></span> </span> #</span> Need a static assignment for SSH ops.</span></span> static</span> =</span> 4222</span></span> </span> #</span> SSH port on the host only exposed to Tailscale IP.</span></span> host_network</span> =</span> "</span>tailscale</span>"</span></span> }</span></span> }</span></span></code></pre>Templating Env Variables#</a></h3> NOTE</strong>: This is not</strong> recommended for production.</p> Nomad doesn’t have any templating functionalities, so all the config must be sourced from Consul and secrets should be sourced from Vault. However in the time constraint I had, I wanted to understand Nomad and Consul better and use Vault at a later stage</a>. I needed a way to interpolate the env variables. This is where Terraform comes into picture:</p> resource</span> "nomad_job"</span> "app"</span> {</span></span> jobspec</span> =</span> templatefile</span>(</span>"</span>${</span>path</span>.</span>module</span>}</span>/conf/shynet.nomad</span>"</span>,</span> {</span></span> shynet_django_secret_key</span> =</span> var.shynet_django_secret_key,</span></span> shynet_postgresql_password</span> =</span> var.shynet_postgresql_password</span></span> }</span>)</span></span> hcl2</span> {</span></span> enabled</span> =</span> true</span></span> }</span></span> }</span></span></code></pre> We can pass the variables from Terraform (which can be sourced by TF_VAR_</code> in your local env) to the Nomad job spec. Inside the job spec we can use env</code> to make it available to our task:</p> env</span> {</span></span> DB_PASSWORD</span> =</span> "</span>${</span>shynet_postgresql_password</span>}</span>"</span></span> DJANGO_SECRET_KEY</span> =</span> "</span>${</span>shynet_django_secret_key</span>}</span>"</span></span> }</span></span></code></pre>Running a backup job on the host#</a></h3> I use restic</code> to take periodic backups of my server and upload to Backblaze B2. Since Nomad supports running tasks as a different isolated environment (chroot</code>) using exec</code> driver and even without isolation using raw_exec</code> driver, I wanted to give that a try. I’ve to resort using raw_exec</code> driver here because /data</code> file path on my host was not available to the chroot’ed environment.</p> job</span> "restic"</span> {</span></span> datacenters</span> =</span> [</span>"</span>hydra</span>"</span>]</span></span> type</span> =</span> "</span>batch</span>"</span></span> </span> periodic</span> {</span></span> cron</span> =</span> "</span>0 3 * </span>"</span></span> time_zone</span> =</span> "</span>Asia/Kolkata</span>"</span></span> prohibit_overlap</span> =</span> true</span></span> }</span></span> ...</span></span> task</span> "backup"</span> {</span></span> driver</span> =</span> "</span>raw_exec</span>"</span></span> </span> config</span> {</span></span> #</span> Since `/data` is owned by `root`, restic needs to be spawned as `root`. </span></span> </span> #</span> `raw_exec` spawns the process with which `nomad` client is running (`root` i.e.).</span></span> command</span> =</span> "</span>$${NOMAD_TASK_DIR}/restic_backup.sh</span>"</span></span> }</span></span> }</span></span> ...</span></span> }</span></span></code></pre> You can follow the rest of the config here</a>.</p> Scope of Improvements#</a></h2> Nomad has been an absolute joy to work with. However, I’ve spotted a few rough edge cases which I believe one should be aware of:</p> host_network</code> property sometimes gets ignored when doing a modification to service</code>. I’ve opened an issue</a> upstream but looks like other people are facing similar behaviours here</a> and here</a>.</li> host_network</code> as of present cannot</a> bind to a floating IP address (DigitalOcean/GCP etc). I’ve to resort to using my droplet’s public IPv4 address for now.</li> I tried using Consul Connect (service mesh with mTLS) but looks like again because of host_network</code>, I’m unable</a> to use it.</li> Nomad CLI can definitely be improved</a> for a much more consistent experience. I particularly missed using kubectl</code> when using nomad</code>.</li> </ul> That apart, I ended up sending a PR</a> to upstream addressing a CLI arg ordering issue.</p> Gotchas:#</a></h3> On a Nomad server already</em> bootstrapped, if you try changing server.bind_addr</code>, it won’t have any effect. I almost pulled my hair debugging this, ultimately deleting the data_dir</code> of the server resolved the issue for me.</li> I’m running DB and</em> the App together as a single “group” in my setup configs. Don’t do this in production. Whenever you restart the job, the group will restart both the containers. The side effect of this is pretty interesting: Since we use Consul to fetch the DB Host, the app may start before the DB boots up and</em> registers its new address with Consul. I will fix the dependency in a future version but since I’m running fewer workloads and there are automatic retries, it’s okay enough for me to keep it like this.</li> </ul> Community#</a></h2> Nomad’s community is pretty small compared to Kubernetes. However, the folks are super responsive on Gitter</a>, Discourse</a> and Github Issues. A few noteworthy mentions:</p> @the-maldridge</a> helped me with my doubts in Gitter.</li> @tgross</a> who is super responsive on Github issues and does an excellent job at housekeeping the issues.</li> @shantanugadgil</a> who is also pretty active in the community.</li> </ul> Nomad’s ecosystem is still in its nascent stage and I believe there are a lot of contribution opportunities for folks interested in Golang, Ops, Distributed Systems to contribute to Nomad. The codebase of Nomad is approachable and there are quite a few key areas which can be contributed to:</p> Docs: More examples, practical use cases.</li> Nomad Job files: There are many helm charts available to follow best practices. Something similar in Nomad will definitely be interesting.</li> Nomad Gotchas: Since K8s is widely used and has a much larger adoption, it’s only natural that the failure stories of K8s are highlighted a lot. Nomad being a pretty smaller community, we need more debugging and “things that went wrong” reference materials. You learn more from failures than 101 setup guides :)</li> </ul> Final Thoughts#</a></h2> I think I’m sold</em> on Nomad. I’ve used Kubernetes in prod for 2 years but if you were to ask me to write a Deployment spec from scratch (without Googling/kubectl help) I won’t be able to. After writing Nomad configs, I just can’t think of the sheer amount of boilerplate that K8s requires to get an application running.</p> Nomad is also a simpler piece to keep in your tech stack. Sometimes it’s best to keep things simple when you don’t really achieve any benefits from the complexity.</p> Nomad offers less</em> than Kubernetes and it’s a feature, not a bug.</p> Fin!</p> Discussions#</a></h4> HackerNews</a></li> Lobster</a></li> Twitter</a></li> </ul> Setup Gitlab Runner with AWS ECR 2021-01-29T08:10:55+05:30 There are some things you expect to just work. Sadly trying to make Gitlab Runner</a> with AWS ECR</a> turned out to be quite a daunting task and the little documentation in this area doesn’t help. There’s even a 4 years old issue</a> and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been.</p> Anyway, since I spent a lot of time figuring out how to make a Private Registry work with a cross-account ECR, I’m documenting these steps hoping it’ll help someone someday :).</p> The Problem#</a></h3> There are mainly 2 seemingly same but different problems when it comes to using ECR. Let’s discuss both of them separately:</p> Pulling a private image from ECR using the Docker Executor. For eg, if your gitlab-ci.yml</code> looks like:</li> </ul> t</span>est-pull</span>:</span></span> i</span>mage</span>:</span> $</span>PRIVATE_ECR_IMAGE</span></span> s</span>cript</span>:</span></span> -</span> e</span>cho "Hello World!"</span></span></code></pre> In this case, the Docker Executor needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE</code>.</p> Pulling a private image inside</em> the job. For eg, if you’re using Kaniko:</li> </ul> d</span>ocker-build</span>:</span></span> i</span>mage</span>:</span> g</span>cr.io/kaniko-project/executor:debug</span></span> s</span>cript</span>:</span></span> -</span> \|</span></span> /kaniko ...</span></span> # Inside this step, we use a PRIVATE_ECR_IMAGE defined in our `Dockerfile`.</span></span></code></pre> In this case, Kaniko</a> needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE</code>.</p> NOTE</strong> I prefer Kaniko over DIND</a> as it is faster, doesn’t require running the privileged container, caching is simplified, and is in general a lot simpler to setup.</p> The Solution#</a></h3> So, for the first case, where you want to authenticate the Docker Executor to AWS ECR, you’ll need 2 things:</p> Setup DOCKER_AUTH_CONFIG</code> environment variable to { "credsStore": "ecr-login" }</code> in the config.toml</code> of the runner. For eg:</li> </ol> [[runners]]</span></span> name = "Test"</span></span> url = "https://gitlab.internal/"</span></span> token = "REDACTED"</span></span> executor = "docker"</span></span> environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]</span></span></code></pre> Now, we’ve specified the Credential Store for Docker, but we don’t have this binary docker-credential-ecr-login</code> in our runner. AWS provides amazon-ecr-credential-helper</a> which is a neat way of automatically authenticating with AWS ECR based on your Access Keys/IAM role. What does automatic mean here? So, the normal docker login</code> is a basic auth command, where if you’ve to log in to ECR, you need to do something like:</li> </ol> aws</span> ecr</span> get-login-password</span> -</span>-region</span> region</span> \|</span> docker</span> login</span> -</span>-username</span> AWS</span> -</span>-password-stdin</span> aws_account_id.dkr.ecr.region.amazonaws.com</span></span></code></pre> This is problematic because the authorization token is valid for 12 hours. Further, you’ve to log in to multiple registry IDs separately. Managing this is a nightmare, so Docker instead of just relying on Basic Auth, came up with a neat mechanism: docker-credential-helpers</a>. This allows you to keep your secret tokens in your Keystore. A new credential helper can be written in Go</code> which implements the credentials.Helper</code> interface. This is what amazon-ecr-credential-helper</a> does by offering various ways like AWS IAM Roles, Assumed Roles, Access Keys, etc to authenticate with ECR.</p> This is where I stumbled the most. I downloaded the binary from the Github Releases</a> but this binary is statically compiled with muslc</code> libraries.</p> However gitlab/gitlab-runner</code> is based on the ubuntu</code> docker image, so the above binary never worked. The strangest thing was the unhelpful error message that sh</code> returned as explained in this post</a>.</p> To make things easier, I baked my own gitlab-runner</code> image with the above binary compiled inside the image using go get</code>:</p> FROM</span> ubuntu:20.04 </span>AS</span> build</span></span> ENV</span> DEBIAN_FRONTEND=noninteractive </span></span> RUN</span> : \</span></span> && apt-get update \</span></span> && apt-get install --no-install-recommends -y git golang-go ca-certificates \</span></span> && rm -rf /var/lib/apt/lists/ \</span></span> ;</span></span> RUN</span> go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login</span></span> WORKDIR</span> /build</span></span> RUN</span> mv /root/go/bin/docker-credential-ecr-login .</span></span> </span> FROM</span> gitlab/gitlab-runner:v13.8.0 </span>AS</span> deploy</span></span> COPY</span> --from=build /build/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login</span></span></code></pre> The above image bakes in the docker-credential-ecr-login</code> binary and also puts it under /usr/local/bin</code> so it’ll be available under $PATH</code> to the Docker engine.</p> With the above 2 things, if the runner’s server (EC2 instance/K8s pod) has access to the ECR image, it should be able to pull.</p> Now coming to the 2nd problem, where we wanted kaniko</code> to authenticate to ECR, things are a bit simpler:</p> Kaniko comes with docker-credential-ecr-login</code> baked in. All you need to do is add the following to ~/.docker/config.json</code> as explained here</a>.</li> </ul> { "credsStore": "ecr-login" }</span></span></code></pre> Next, we need to mount the AWS Credentials to Kaniko’s image so that it can use AWS SDK to perform a login to ECR. We do that by using volumes</code> of the runner</code>:</p> [runners.docker]</span></span> volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]</span></span></code></pre> This mounts the aws-credentials</code> file from the host inside the container which the runner spawned (kaniko</code> in this case).</p> A sample aws-credentials</code> file if you’re using a cross-account access can look like:</p> [default]</span></span> role_arn=arn:aws:iam::ACCOUNT_ID:role/assume-role-{{ROLE_NAME}}</span></span> credential_source=Ec2InstanceMetadata</span></span> region=ap-south-1</span></span></code></pre> You can put in your normal AWS Keys or leave them blank if you want to use your IAM Role. With mounting the AWS Credentials inside Kaniko’s container, you can authenticate to cross-account ECRs as well (once you set up the whole assumed-role/trusted entities flow).</p> Sample Runner#</a></h3> If you want to take a look at how the complete flow looks like:</p> Start the Runner service using docker-compose</code>:</li> </ul> v</span>ersion</span>:</span> '</span>3.7</span>'</span></span> </span> s</span>ervices</span>:</span></span> </span> t</span>est</span>:</span></span> #</span> As explained above, this image has `docker-credentials-ecr-login` baked in.</span></span> i</span>mage</span>:</span> {</span>{</span>A</span>CCOUNT_ID</span>}</span>.</span>dkr.ecr.ap-south-1.amazonaws.com/custom/gitlab-runner:13.8.0</span></span> r</span>estart</span>:</span> a</span>lways</span></span> v</span>olumes</span>:</span></span> #</span> Automatically created; config.toml.</span></span> - </span>'</span>./test/runner-config:/etc/gitlab-runner</span>'</span></span> #</span> Mount AWS Credentials for Docker Executor to authenticate.</span></span> - </span>'</span>./test/aws-credentials:/root/.aws/credentials:ro</span>'</span></span> #</span> Mount Docker Socket so that executor can communicate with it.</span></span> - </span>'</span>/var/run/docker.sock:/var/run/docker.sock</span>'</span></span></code></pre> Register a new runner:</li> </ul> docker-compose exec test register</span></span></code></pre> Fill in the basic info and edit ./test/runner-config.toml</code> with the following options:</p> [[runners]]</span></span> name = "Test"</span></span> executor = "docker"</span></span> environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]</span></span> [runners.docker]</span></span> volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]</span></span></code></pre>Conclusion#</a></h2> Honestly, this was a lot of trial and error to figure out how to use private images with Gitlab. Some important links and references that helped me figure this out:</p> https://gitlab.com/bmares/gitlab-runner-ecr-auth-example/</a></li> https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_84649153</a></li> </ul> Fin!</p> Designing a simple Job Queue in Golang 2020-12-01T08:10:55+05:30 In this post we’ll see how to create a simple job queue in Golang. There are tonnes of libraries and posts out there doing overly complicated stuff, however if your need is pretty minimal or you want to understand the concepts from ground up, this post aims to do just that and nothing more.</p> We’ll be using concepts like WaitGroup</a>, Channels</a> and Contexts</a> to build our own Job Queuing mechanism. It primarily involves 2 components:</p> Queue</strong>: A queue which has a list of items waiting to be processed.</p> </li> Worker</strong>: A worker</em> constantly listening to that queue and processing</em> the events as desired.</p> </li> </ul> With these 2 main ideas behind us, let us create our sample structure:</p> package</span> dispatch</span></span> </span> type</span> Dispatcher</span> interface</span> {</span></span> //</span> Push takes an Event and pushes to a queue.</span></span> Push</span>(</span>Event</span>)</span> error</span></span> //</span> Run spawns the workers and waits indefinitely for</span></span> //</span> the events to be processed.</span></span> Run</span>(</span>)</span></span> }</span></span> </span> //</span> EventDispatcher represents the datastructure for an</span></span> //</span> EventDispatcher instance. This struct satisfies the</span></span> //</span> Dispatcher interface.</span></span> type</span> EventDispatcher</span> struct</span> {</span></span> Opts</span> Options</span></span> Queue</span> chan</span> models</span>.</span>Notification</span></span> Finished</span> bool</span></span> }</span></span> </span> //</span> Options represent options for EventDispatcher.</span></span> type</span> Options</span> struct</span> {</span></span> MaxWorkers</span> int</span> //</span> Number of workers to spawn.</span></span> MaxQueueSize</span> int</span> //</span> Maximum length for the queue to hold events.</span></span> }</span></span> </span> //</span> NewEventDispatcher initialises a new event dispatcher.</span></span> func</span> NewEventDispatcher</span>(</span>opts</span> Options</span>)</span> (</span>Dispatcher</span>)</span> {</span></span> return</span> EventDispatcher</span>{</span></span> Opts</span>:</span> opts</span>,</span></span> Queue</span>:</span> make</span>(</span>chan</span> Event</span>,</span> opts</span>.</span>MaxQueueSize</span>)</span>,</span></span> Finished</span>:</span> false</span>,</span></span> }</span></span> }</span></span></code></pre>Pushing to Queue#</a></h3> Now that we have our basic structure ready, let’s write a function to push events to queue.</p> A queue is simply a channel. We have created a new queue of size MaxQueueSize</code> while initialising the EventDispatcher</code>.</p> Queue: make(chan Event, opts.MaxQueueSize)</span></span></code></pre> To push events into it, we’ll simply do: d.Queue <- event</code>. This adds a new item (event</code>) of type Event</code> to our queue.</p> //</span> Push adds a new event payload to the queue.</span></span> func</span> (</span>d </span></span>EventDispatcher</span> Push</span>(</span>event</span> Event</span>)</span> error</span> {</span></span> if</span> d</span>.</span>Finished</span> {</span></span> return</span> errors</span>.</span>New</span>(</span>`</span>queue is closed</span>`</span>)</span></span> }</span></span> d</span>.</span>Queue</span> <-</span> event</span></span> return</span> nil</span></span> }</span></span></code></pre>Listening to Queue#</a></h3> So the client is calling Push()</code> on our EventDispatcher</code> and events are being pushed in the channel. But there’s no one reading from this channel so far. Let’s fix that by spawning workers, who will listen on the channel indefinitely</em> and process the events:</p> for</span> {</span></span> select</span> {</span></span> case</span> event</span> <-</span> d</span>.</span>Queue</span>:</span></span> event</span>.</span>Process</span>(</span>)</span></span> }</span></span> }</span></span></code></pre> In the above snippet, we are simply looping indefinitely to scan through all items in the queue. event <- d.Queue</code> is basically fetching the item from the channel and assigning a value to it.</p> event.Process()</code> is a dummy function but it basically indicates that whatever processing that needs to be done should be handled here.</p> Right now, you’ll be wondering two things:</p> If this is an infinite loop, how do we guarantee it runs forever?</li> How do I spawn more workers if I need concurrency?</li> </ul> To address these problems, let’s add in WaitGroups and GoRoutines to our mix.</p> WaitGroups will help us keep a count of workers which have been spawned and until each one of them finishes processing, wait groups will keep blocking indefinitely using wg.Wait()</code>.</p> And to bring in more workers, we’ll simply spawn them with GoRoutines:</p> go</span> func</span>(</span>)</span> {</span></span> for</span> {</span></span> select</span> {</span></span> case</span> event</span> <-</span> d</span>.</span>Queue</span>:</span></span> event</span>.</span>Process</span>(</span>)</span></span> }</span></span> }</span></span> }</span>(</span>)</span></span></code></pre> Now, spawning n</code> Goroutines is just a matter of a simple for loop over this:</p> for</span> i</span>:=</span>0</span>;</span> i</span><</span>d</span>.</span>Opts</span>.</span>MaxWorkers</span>;</span> i</span>++</span>{</span></span> wg</span>.</span>Add</span>(</span>1</span>)</span> //</span> Add a wait group for each worker</span></span> go</span> func</span>(</span>)</span> {</span></span> for</span> {</span></span> select</span> {</span></span> case</span> event</span> <-</span> d</span>.</span>Queue</span>:</span></span> event</span>.</span>Process</span>(</span>)</span></span> }</span></span> }</span></span> }</span>(</span>)</span></span> }</span></span></code></pre> Perfect! But hang on! We have missed a critical thing. How do we handle cancellations? For eg, when your program shuts down, we should clean up all the Goroutines spawned and process the remaining messages in queue. For that, we need to listen to a Cancellation</code> channel. The only purpose of this channel is to listen for SIGINT or SIGTERM signals and whenever either of them is received, we should flush our events.</p> Here’s how the client would initialise a context:</p> //</span> Create a channel to relay `SIGINT` and `SIGTERM` signals.</span></span> closeChan</span> :=</span> make</span>(</span>chan</span> os</span>.</span>Signal</span>,</span> 1</span>)</span></span> signal</span>.</span>Notify</span>(</span>closeChan</span>,</span> os</span>.</span>Interrupt</span>,</span> syscall</span>.</span>SIGTERM</span>)</span></span> ctx</span>,</span> cancel</span> :=</span> context</span>.</span>WithCancel</span>(</span>context</span>.</span>Background</span>(</span>)</span>)</span></span></code></pre> And in the main thread, the client would block on closeChan</code> channel like:</p> //</span> Listen on close channel indefinitely until a</span></span> //</span> `SIGINT` or `SIGTERM` is received.</span></span> <-</span>closeChan</span></span> //</span> Cancel the context to gracefully shutdown.</span></span> cancel</span>(</span>)</span></span></code></pre> When cancel()</code> is called, it does something special. It passes a value to ctx.Done()</code> channel. We can listen to this channel in the .Run()</code> function and flush pending events accordingly:</p> case</span> <-</span> ctx</span>.</span>Done</span>(</span>)</span>:</span></span> //</span> Ensure no new messages are added.</span></span> d</span>.</span>Finished</span> =</span> true</span></span> //</span> Flush all events.</span></span> e</span>.</span>Flush</span>(</span>)</span></span> //</span> This Goroutine has finished processing.</span></span> wg</span>.</span>Done</span>(</span>)</span></span></code></pre> Stitching all pieces together, we finally have:</p> package</span> dispatch</span></span> </span> type</span> Dispatcher</span> interface</span> {</span></span> //</span> Push takes an Event and pushes to a queue.</span></span> Push</span>(</span>Event</span>)</span> error</span></span> //</span> Run spawns the workers and waits indefinitely for</span></span> //</span> the events to be processed.</span></span> Run</span>(</span>)</span></span> }</span></span> </span> //</span> EventDispatcher represents the datastructure for an</span></span> //</span> EventDispatcher instance. This struct satisfies the</span></span> //</span> Dispatcher interface.</span></span> type</span> EventDispatcher</span> struct</span> {</span></span> Opts</span> Options</span></span> Queue</span> chan</span> models</span>.</span>Notification</span></span> Finished</span> bool</span></span> }</span></span> </span> //</span> Options represent options for EventDispatcher.</span></span> type</span> Options</span> struct</span> {</span></span> MaxWorkers</span> int</span> //</span> Number of workers to spawn.</span></span> MaxQueueSize</span> int</span> //</span> Maximum length for the queue to hold events.</span></span> }</span></span> </span> //</span> NewEventDispatcher initialises a new event dispatcher.</span></span> func</span> NewEventDispatcher</span>(</span>opts</span> Options</span>)</span> (</span>Dispatcher</span>)</span> {</span></span> return</span> EventDispatcher</span>{</span></span> Opts</span>:</span> opts</span>,</span></span> Queue</span>:</span> make</span>(</span>chan</span> Event</span>,</span> opts</span>.</span>MaxQueueSize</span>)</span>,</span></span> Finished</span>:</span> false</span>,</span></span> }</span></span> }</span></span> </span> //</span> Push adds a new event payload to the queue.</span></span> func</span> (</span>d </span></span>EventDispatcher</span> Push</span>(</span>event</span> Event</span>)</span> error</span> {</span></span> if</span> d</span>.</span>Finished</span> {</span></span> return</span> errors</span>.</span>New</span>(</span>`</span>queue is closed</span>`</span>)</span></span> }</span></span> d</span>.</span>Queue</span> <-</span> event</span></span> return</span> nil</span></span> }</span></span> </span> //</span> Run spawns workers and listens to the queue</span></span> //</span> It's a blocking function and waits for a cancellation</span></span> //</span> invocation from the Client.</span></span> func</span> (</span>d </span></span>EventDispatcher</span> Run</span>(</span>ctx</span> context</span>.</span>Context</span>)</span> {</span></span> wg</span> :=</span> sync</span>.</span>WaitGroup</span>{</span>}</span></span> for</span> i</span> :=</span> 0</span>;</span> i</span> <</span> d</span>.</span>Opts</span>.</span>MaxWorkers</span>;</span> i</span>++</span> {</span></span> wg</span>.</span>Add</span>(</span>1</span>)</span> //</span> Add a wait group for each worker</span></span> //</span> Spawn a worker</span></span> go</span> func</span>(</span>)</span> {</span></span> for</span> {</span></span> select</span> {</span></span> case</span> <-</span>ctx</span>.</span>Done</span>(</span>)</span>:</span></span> //</span> Ensure no new messages are added.</span></span> d</span>.</span>Finished</span> =</span> true</span></span> //</span> Flush all events</span></span> e</span>.</span>Flush</span>(</span>)</span></span> wg</span>.</span>Done</span>(</span>)</span></span> return</span></span> case</span> e</span> <-</span> d</span>.</span>Queue</span>:</span></span> e</span>.</span>Process</span>(</span>)</span></span> }</span></span> }</span></span> }</span>(</span>)</span></span> }</span></span> wg</span>.</span>Wait</span>(</span>)</span></span> }</span></span> </span> //</span> Push adds a new event payload to the queue.</span></span> func</span> (</span>d </span></span>EventDispatcher</span> Push</span>(</span>event</span> Event</span>)</span> error</span> {</span></span> if</span> d</span>.</span>Finished</span> {</span></span> return</span> errors</span>.</span>New</span>(</span>`</span>queue is closed</span>`</span>)</span></span> }</span></span> d</span>.</span>Queue</span> <-</span> event</span></span> return</span> nil</span></span> }</span></span></code></pre> This post doesn’t cover how to flush or process the events as these are implementation specific details. This is a pretty barebones structure and you can modify the code according to your usecase.</p> Fin!</p> Analysis Paralysis - CrashLoopBackoff 2020-11-27T08:10:55+05:30 Have you ever been stuck in a situation where you’ve spent countless hours digging through lot of options, overthinking their pros/cons and having a tough time to decide?</p> If yes, I’ve been in the same boat myself and the more I think about this, I think it’s causing some serious issues in my productivity. Since I’ve started to self host a lot of services, I’m overthinking and overanalysing the software choices available to do a certain task. For eg, a personal wiki, sounds such a simple and common use case but I’ve personally tried countless choices. I’ve read and searched for these discussions on lobster/HN/Reddit discussions boards and that has been a real time sink for me, now that I look back. I’ll try an option for maybe a week or two at best and then jump on to something else in search of the perfect</em> solution.</p> It’s hard for me to stick to a solution. And it’s even more embarassing when you tell your friends about the “latest new distro” you’re using, announce it to the world on Twitter and when someone asks, “hey how’s that OS working for you” after a month, you tell them “Oh, I switched from it last week”. I’m not making any of this up, I’ve been in these shoes myself and honestly it’s not a good feeling.</p> In the search of perfect</em> I tend to lose sight of things that are important. Things which are slow, boring but they work</em> are more reliable than a software which changes fast and breaks often. I want to fail cheap and quickly rather than spending a lot of time and still eventually failing.</p> No, this post doesn’t have any answers on how to fix this problem. Well, if I myself knew then I’d not be writing this but giving a TEDx talk on it. These are just my observations and something that I intend to actively fix.</p> Boring is better. I want to stick to stable choices when it comes to trivial things in life. I don’t wanna seek excitement or joy from switching to a cool new distro. Instead, by actually doing some important project at my work or building my side projects, something that gives me a genuine feeling of happiness. After all grass always seems to look greener on the other side.</p> I don’t usually pen down my thoughts like these, but I want to refer to this public post as a reminder for myself the next time I’m about to make the same mistake.</p> Fin.</p> UPDATE</strong> (2021-11-02): There’s a really nice video</a> I came across today on the same subject. I’d highly recommend you to give this a listen if you found the above post relatable.</p> Migrating my blog to Zola 2020-11-08T08:10:55+05:30 I’ve been writing on this blog for about 2 years now. This has been the longest I’ve stuck on to the same technology stack</em> for my blog. I’ve previously jumped from a Jekyll based static site to a Medium blog before finally settling for Hugo</a>.</p> I’ve been using Hugo since 2018 but I don’t recall as to why</em> I went ahead with it. Maybe it was increasingly popular at that time and everyone touted Hugo as the</em> solution to Static Site Generator (referred to as SSG from here on). There are 1000s of SSGs and at least a dozen of websites which lists all the SSGs out there. This is crazy by any standards. Hugo started as a generic blog generator but over the years it has become a website generator</em>. It’s no longer aimed at people who just want to have a small little static website/blog but supports all the use cases for people building full-fledged static websites. IMHO these two goals are overarching however this has resulted in a simple project to become incredibly complex over time.</p> Tipping Point#</a></h3> Anyway, so I wanted to change the look of the homepage on my website so I decided to look at Hugo’s documentation. Hugo’s documentation is great for someone who knows what exactly are they looking for. The documentation is so huge that you simply cannot grok it in one evening. I had zero ideas on how to customise the damn homepage of my blog and after spending hours buried in the documentation I was able to kind of figure the solution but it was unintuitive, to say the least. Apparently, to override any template from the theme, you have to mirror the directory structure of the theme in your root directory. Which meant, I needed to look at the source code of the theme, figure out the project structure, copy-paste all the folder names and put my override of index.html</code> there. Which, BTW magically</strong> overrides it. This whole magic thing is BS and I am being strongly opinionated here.</p> There is more than one way to do something in Hugo. Different theme authors use different styles, which makes the whole thing even more complex. It also means for my customisations to work across themes, well you guessed it right: it’s impossible</strong>.</p> Recently I discovered that I was unable to preview my Hugo website locally without internet because I had a Twitter shortcode</a> in one of my blog post (which makes an API call to Twitter to render a nice card preview). The site completely failed to render instead of just logging a warning. Bollocks.</p> The tipping point for me, however, was when the theme I was using stopped working with the latest version of Hugo at that point. So, picture this – You make dozens of custom changes and then one update just breaks</em> your website. Now not only you have to fix your shit but the theme you were using, you’ve to make upstream changes to the theme or maintain your own fork. And no, this is not a one-off experience. Hugo upgrades are a joke, they are known to break very very often.</p> I was done at this point. I didn’t want to deal with this BS of continuously fighting the generator for my blog.</p> A fresh change#</a></h3> Being a practitioner of Yak Shaving</a>, I discussed the idea of a “tinyhugo” with Kailash</a> and Sarat</a>. We’d arrived at a spec and I started writing some code to pander to my NIH syndrome.</p> However, I was still not convinced that a simpler solution doesn’t exist. I spent countless hours exploring other alternatives. I’d used Lektor</a>, Pelican</a>, Eleventy</a> before finally stumbling upon Zola</a> from HN/Lobster discussions. I’ve got to say, the landing page gave a fresh</em> feeling - one that I’ve not seen with any other alternatives. In fact quite opposite to the Eleventy landing page which looks like an over-engineered piece of software to generate websites (Not hating on it, there might be use cases for it, but the JS tooling and dependency system is something that I would not want to touch with a 10ft pole).</p> Zola’s primary appeal to me was that like Hugo it’s extremely fast and comes as a single binary no dependency package. I looked at the docs the first impression was they are concise enough to get a basic idea. Zola is strongly opinionated, even to the extent of dictating a project structure and sometimes filenames too. I actually preferred this over the magic</em> Hugo does. In less than 2 hours I was able to port the home page of my blog (and tweak it to my liking) in Zola. I decided to abandon my own tinyhugo</code> attempt because for the very fact Zola fits my needs very well.</p> The thing that I really loved about Zola is how it enforces a separation between Section</a> and Pages</a>. The section represents a “collection” of posts. So a blog</em> can be a section, and I can have another section called “Book Reviews”. I could easily tell Zola where to look for the templates by specifying the same in content/book_reviews/_index.md</code>. I don’t have to read Hugo docs or do Google-fu</em> to figure this out, it’s right there in the docs and very apparent.</p> For the record, I still don’t know how to customise different templates for different sections in Hugo, but I couldn’t care less.</p> Migration#</a></h3> The migration was pretty straightforward – I had to copy the content folder</code>s of my blog (which are just a bunch of .md</code> files) and replace YAML</code> frontmatter to TOML</code>. There were a few variable changes that I needed to do manually but since they were a manageable 20-25 posts, I did it by hand. I could potentially automate but then rabbit deep in the rabbit hole of Yak Shaving. The good part was that I was able to retain the same URL structure for my new blog because the URL scheme was based on the file paths.</p> I spent some time porting hugo-ink</a> to Zola and did minor CSS tweaks to it. Zola uses the Tera language for templating and it’s much more pleasing to eyes than the Go Template syntax. Zola comes with pretty neat features like Search, RSS/Atom Feeds, Syntax Highlighting and SASS->CSS Processors.</p> What took me time however was to figure out how to get opengraph</code> tags in each page. Hugo provides nifty template</a> for this use case but Zola is pretty barebones like that. People who care a lot about SEO need to spend some extra efforts here.</p> Future#</a></h3> Zola is still a pretty new kid on the block but the author shares the same frustration about Hugo:</p> it personally drives me insane, to the point of writing my own template engine and static site generator. Yes, this is a bit biased. – Source</a></p> </blockquote> This also reflects in the issues/PRs I’ve seen for Zola and the author is opinionated about not adding features which would make Zola complicated. Overall I am very happy with the switch and it was long due. I feel more confident in tweaking certain sections of my website. I plan to open-source the current theme in the next few days.</p> You can read the Source Code</a> of this website if you’d like to explore how this website is built.</p> Fin!</p> Import existing Route53 records in Terraform 2020-10-18T08:10:55+05:30 Terraform has a straightforward way of importing existing records (managed outside Terraform) via terraform import</code> command. The usage is documented here</a> and works well if you have a handful of records to import. However when you work with custom Terraform modules and</em> have a whole bunch of records to be imported, you’d look out ways to script</strong> the entire workflow. I did this a few weeks back at work and thought to share a solution which works well for my usecase.</p> How it works#</a></h3> The task consists of 3 parts:</p> 1. Import all existing records in a hosted zone using AWS CLI</code>.#</a></h4> aws</span> route53</span> list-resource-record-sets</span> -</span>-hosted-zone-id</span> XXX</span> ></span> data/company-tld.json</span></span></code></pre>#</span> Loads the zone records in a dict</span></span> def</span> load_records</span>(</span>zone_file</span>=</span>ZONE_FILE</span>)</span>:</span></span> with</span> open</span>(</span>zone_file</span>)</span> as</span> record_file</span>:</span></span> data</span> =</span> json</span>.</span>load</span>(</span>record_file</span>)</span></span> return</span> data</span></span></code></pre>2. Import the record in Terraform state.#</a></h4> To do this, Terraform CLI comes with an import</code> command. However for import</code> to work, you need to have a resource declaration in your Terraform file already.</p> From the official documentation</a>:</p> Because of this, prior to running terraform import it is necessary to write manually a resource configuration block for the resource, to which the imported object will be mapped.</p> </blockquote> To overcome this restriction, we will create a dummy.tf</code> and programatically write the configuration block for each record.</p> #</span> Writes the dummy Terraform template which is required</span></span> #</span> before `terraform import` runs.</span></span> def</span> template_dummy_file</span>(</span>resource_name</span>)</span>:</span></span> add_dummy_record</span> =</span> Template</span>(</span></span> """</span></span> resource "aws_route53_record" "$resource_name" {</span></span> # (resource arguments)</span></span> }</span></span> """</span></span> )</span></span> dummy_file_path</span> =</span> path</span>.</span>join</span>(</span>TERRAFORM_DIR</span>,</span> "</span>dummy.tf</span>"</span>)</span></span> with</span> open</span>(</span>dummy_file_path</span>,</span> "</span>a</span>"</span>)</span> as</span> f</span>:</span></span> f</span>.</span>write</span>(</span>add_dummy_record</span>.</span>substitute</span>(</span>resource_name</span>=</span>resource_name</span>)</span>)</span></span></code></pre> AWS Route53 module can import aws_route53_record</code> as decsribed here</a>. We will run this command as a subprocess.</p> #</span> Shells out `terraform import` command in the host OS.</span></span> def</span> terraform_import</span>(</span>resource_name</span>,</span> resource_type</span>)</span>:</span></span> import_command</span> =</span> f</span>"</span>terraform import aws_route53_record.</span>{</span>resource_name</span>}</span> {</span>ZONE_ID</span>}</span>_</span>{</span>resource_name</span>}</span>_</span>{</span>resource_type</span>}</span>"</span></span> run</span>(</span>import_command</span>,</span> shell</span>=</span>True</span>,</span> check</span>=</span>True</span>)</span></span></code></pre>3. (Optional) Move Resources in a Module#</a></h4> In case you are using a Module to manage AWS Route53 resources, you’ll need to move the declaration from resource</code> to module</code> configuration block. This is described more in detail here</a>.</p> The module declaration/naming would depend on how the module is configured. To demonstrate, the module I use internally requires the name to be of the format resource_name-resource_type</code>. To achieve this, you can call terraform state mv</code> as a subprocess:</p> #</span> Shells out `terraform state mv` command in the host OS.</span></span> def</span> terraform_move</span>(</span>resource_name</span>,</span> resource_type</span>)</span>:</span></span> mv_command</span> =</span> f</span>"</span>terraform state mv aws_route53_record.</span>{</span>resource_name</span>}</span> 'module.</span>{</span>MODULE_NAME</span>}</span>.aws_route53_record.route53_record[</span>\"</span>{</span>resource_name</span>}</span>-</span>{</span>resource_type</span>}</span>\"</span>]</span>'</span>"</span></span> run</span>(</span>mv_command</span>,</span> shell</span>=</span>True</span>,</span> check</span>=</span>True</span>)</span></span></code></pre> That’s it! Running terraform plan</code> should now show you the changes and if you imported every record correctly you should not see any drift</em> from the real world state.</p> You can view the entire script here:</p> import</span> json</span></span> from</span> os</span> import</span> getenv</span>,</span> path</span></span> from</span> string</span> import</span> Template</span></span> from</span> subprocess</span> import</span> run</span></span> from</span> sys</span> import</span> exit</span></span> </span> </span> ZONE_ID</span> =</span> getenv</span>(</span>"</span>ZONE_ID</span>"</span>)</span></span> MODULE_NAME</span> =</span> getenv</span>(</span>"</span>MODULE_NAME</span>"</span>)</span></span> ZONE_FILE</span> =</span> getenv</span>(</span>"</span>ZONE_FILE</span>"</span>)</span></span> TERRAFORM_DIR</span> =</span> getenv</span>(</span>"</span>TERRAFORM_DIR</span>"</span>)</span></span> </span> #</span> Returns the variable key if not present in ENV.</span></span> def</span> check_env_vars</span>(</span>)</span>:</span></span> if</span> not</span> ZONE_FILE</span>:</span></span> return</span> "</span>$ZONE_FILE</span>"</span></span> if</span> not</span> ZONE_ID</span>:</span></span> return</span> "</span>$ZONE_ID</span>"</span></span> if</span> not</span> MODULE_NAME</span>:</span></span> return</span> "</span>$MODULE_NAME</span>"</span></span> if</span> not</span> TERRAFORM_DIR</span>:</span></span> return</span> "</span>$TERRAFORM_DIR</span>"</span></span> return</span> "</span>"</span></span> </span> </span> #</span> Loads the zone records in a dict</span></span> def</span> load_records</span>(</span>zone_file</span>=</span>ZONE_FILE</span>)</span>:</span></span> with</span> open</span>(</span>zone_file</span>)</span> as</span> record_file</span>:</span></span> data</span> =</span> json</span>.</span>load</span>(</span>record_file</span>)</span></span> return</span> data</span></span> </span> </span> #</span> Writes the dummy Terraform template which is required</span></span> #</span> before `terraform import` runs.</span></span> def</span> template_dummy_file</span>(</span>resource_name</span>)</span>:</span></span> add_dummy_record</span> =</span> Template</span>(</span></span> """</span></span> resource "aws_route53_record" "$resource_name" {</span></span> # (resource arguments)</span></span> }</span></span> """</span></span> )</span></span> dummy_file_path</span> =</span> path</span>.</span>join</span>(</span>TERRAFORM_DIR</span>,</span> "</span>dummy.tf</span>"</span>)</span></span> with</span> open</span>(</span>dummy_file_path</span>,</span> "</span>a</span>"</span>)</span> as</span> f</span>:</span></span> f</span>.</span>write</span>(</span>add_dummy_record</span>.</span>substitute</span>(</span>resource_name</span>=</span>resource_name</span>)</span>)</span></span> </span> </span> #</span> Shells out `terraform import` command in the host OS.</span></span> def</span> terraform_import</span>(</span>resource_name</span>,</span> resource_type</span>)</span>:</span></span> import_command</span> =</span> f</span>"</span>terraform import aws_route53_record.</span>{</span>resource_name</span>}</span> {</span>ZONE_ID</span>}</span>_</span>{</span>resource_name</span>}</span>_</span>{</span>resource_type</span>}</span>"</span></span> run</span>(</span>import_command</span>,</span> shell</span>=</span>True</span>,</span> check</span>=</span>True</span>)</span></span> </span> </span> #</span> Shells out `terraform state mv` command in the host OS.</span></span> def</span> terraform_move</span>(</span>resource_name</span>,</span> resource_type</span>)</span>:</span></span> mv_command</span> =</span> f</span>"</span>terraform state mv aws_route53_record.</span>{</span>resource_name</span>}</span> 'module.</span>{</span>MODULE_NAME</span>}</span>.aws_route53_record.route53_record[</span>\"</span>{</span>resource_name</span>}</span>-</span>{</span>resource_type</span>}</span>\"</span>]</span>'</span>"</span></span> run</span>(</span>mv_command</span>,</span> shell</span>=</span>True</span>,</span> check</span>=</span>True</span>)</span></span> </span> </span> if</span> __name__</span> ==</span> "</span>__main__</span>"</span>:</span></span> missing</span> =</span> check_env_vars</span>(</span>)</span></span> if</span> missing</span>:</span></span> exit</span>(</span>f</span>"</span>Required env variable </span>{</span>missing</span>}</span> is missing.</span>"</span>)</span></span> records</span> =</span> load_records</span>(</span>)</span></span> for</span> i</span> in</span> records</span>.</span>get</span>(</span>"</span>ResourceRecordSets</span>"</span>)</span>:</span></span> resource_name</span> =</span> i</span>.</span>get</span>(</span>"</span>Name</span>"</span>)</span></span> resource_type</span> =</span> i</span>.</span>get</span>(</span>"</span>Type</span>"</span>)</span></span> template_dummy_file</span>(</span>resource_name</span>)</span></span> terraform_import</span>(</span>resource_name</span>,</span> resource_type</span>)</span></span> terraform_move</span>(</span>resource_name</span>,</span> resource_type</span>)</span></span> print</span>(</span>f</span>"</span>Imported </span>{</span>resource_name</span>}</span>"</span>)</span></span></code></pre> Hope this tiny Python script helps you transition your AWS Route53 records neatly and effortlessly!</p> Fin!</p> Setting up a RIPE Atlas Probe 2020-10-03T08:10:55+05:30 Twitter is an amazing thing! For all the shitposting and meme-ing that’s done there, there are some really cool people you get to interact with which wouldn’t have been possible IRL. I happened to stumble upon Swapneel</a> last year in Bangalore at a meetup. He occasionally posts about RIPE Atlas</a> Probes measurements on his Twitter feed and that made me curious and learn more about the Atlas Probe. Call it the Baader-Meinhof effect but just when I was reading his blog post</a> on RIPE Atlas, I found about a Hasgeek workshop</a> which is being conducted by Swapneel himself and well, that’s how I basically got interested in setting up a probe myself.</p> What is RIPE Atlas Probe#</a></h2> A Probe</a> is a device used to measure various metrics like DNS, SSL/TLS, NTP, ping, traceroute etc for an upstream target. A network of such probes is useful not just to internet researchers, network engineers, ISP operators, public activists but also to common folks like me who are simply curious about the network/WWW in general. The idea behind this is simple – you get some virtual credits for hosting a probe. You can make use of these credits to run your own measurements to gain insights about the health of your network. The data from these measurements are made publicly available by RIPE NCC and anyone can see these results.</p> There are 2 ways to host a Probe: Software</strong> and Hardware</strong>. Since the start of this year, RIPE Atlas is available as software packages for various platforms. Before this, the only way to set up a Probe was to host a Hardware probe by applying for one at RIPE NCC website. These hardware probes are not shipping to India since quite some time as Swapneel mentioned and if you’re in India, your only choice, for now, is to set up a software probe. A few key differences in the 2 kinds of probes:</p> Initial Setup</strong>: Hardware probe is a plug-n-play device, no setup required. That makes it attractive for a lot of people, to just get a small device and host it.</li> Spare Compute</strong>: For hosting a probe it is recommended that the probe stays online for as long as possible. This means you need to have a spare compute lying around, like a RaspebrryPi, a VM, a server etc. It’s not really recommended to host a probe on your laptop/desktop - something which has ad-hoc usage pattern. For people who don’t have any spare compute, a hardware probe is more suited for them.</li> Tampering Result</strong>: With the software probe, there are easy ways of tampering the measurement result, which might give a bad data point to people running the measurements. For eg, a software probe could tamper with the DNS queries for a particular upstream and fool you to believe that there’s a problem with the uplink or the upstream target.</li> </ul> You can read more about Probes here</a> and this FAQ</a> section is highly recommended to clear all the basic doubts you might have about Probes.</p> Setting up a Software Probe in a Container#</a></h2> Well, Containers are</em> awesome! It’s easier to do a docker run</code> than figure out the installation instructions for your OS, ensure dependencies are installed and insert 10 other things you have to do</em>. So, I decided to use jamesits/ripe-atlas</a> Docker image to host a probe on my RPi 4.</p> However, in case you don’t want to use Containers for a reason best known to you, you can visit the official docs</a> and find out instructions for your platform.</p> You can follow the below steps to set up a probe via Docker:</p> 1. Create a RIPE NCC Account#</a></h4> You need to register for a RIPE NCC Access Account before proceeding further. Visit the registration</a> page and create an account.</p> 2. Start the container#</a></h4> docker</span> run</span> -</span>-detach</span> -</span>-restart=always</span> -</span>-log-opt</span> max-size=10m</span> \</span></span> -</span>-cap-add=SYS_ADMIN</span> -</span>-cap-add=NET_RAW</span> -</span>-cap-add=CHOWN</span> \</span></span> -</span>-mount</span> type=tmpfs,destination=/var/atlasdata,tmpfs-size=64M</span> \</span></span> -</span>v</span> /var/atlas-probe/etc:/var/atlas-probe/etc</span> \</span></span> -</span>v</span> /var/atlas-probe/status:/var/atlas-probe/status</span> \</span></span> -</span>e</span> RXTXRPT=yes</span> \</span></span> -</span>-name</span> ripe-atlas</span> -</span>-hostname</span> "</span>$(</span>hostname</span> -</span>-fqdn</span>)</span>"</span> \</span></span> jamesits/ripe-atlas:latest</span></span></code></pre> NOTE</strong>: In the above case, since I am running the Probe on my RPi which is why I am using latest-armv7l</code> tag for the jamesits/ripe-atlas</code> image. In case you are doing this on an amd64</code> machine, you should use the latest</code> tag only. You can find more options for the container here</a>.</p> 3. Apply for a Software Probe#</a></h4> You need to register for a software probe here</a> and fill in the details as mentioned below.</p> </p> ASN</strong>: You can find the ASN of your public IP by visiting ip2asn.com</a>. You need to enter your public IP here which you can find it by visiting ifconfig.co</a>.</li> </ul> </p> Public Key</strong>: You need to fill the public key generated by your Probe:</li> </ul> cat</span> /var/atlas-probe/etc/probe_key.pub</span></span></code></pre> Note</strong>: We are mounting the /var/atlas-probe/etc</code> on the host path inside the container at /var/atlas-probe/etc</code>. This ensures that if you stop/remove the container, your public/private key pair isn’t lost and there’s no need to regenerate/apply for a Probe again. However in case you lose this, the Probe will attempt to generate a new pair and you might need to change the public key at that point of time. So be careful to take a backup of the generated key and store somewhere safe.</p> After submitting the form, you will receive an email which will have information about your Probe ID.</p> </p> 4. Wait patiently#</a></h4> After filling the form you need to wait roughly for 15 minutes</strong> for your probe to be added to the global network of Atlas Probes. You will receive an email saying your Probe is live now.</p> </p> Creating Measurements#</a></h2> It’s pretty simple to create measurements, both via UI and a Python library</a> developed by the RIPE community. However, for the ease and convenience I will demonstrate how to create a traceroute measurement from the UI itself:</p> Note</strong>: You get around 15 Credits/minute for hosting a probe. In case you just deployed a probe you might not have enough credits to perform this measurement. You can come back later to do this when you have sufficient credits.</p> Visit the dashboard</a> and headover to Measurement</a> tab. On clicking the Create a Measurement</code> button you will be presented a form with various options. Let’s try a Traceroute to mrkaran.dev</code> for this measurement:</p> </p> To limit the number of probes, I am going to select all Probes in India, by clicking on +New Set -wizard</code> option in the Probe Selection tab.</p> </p> After submitting the form, you will be presented with a Measurement ID which you can use to download the data at a later point as well.</p> </p> Visit the Measurement ID page and wait for a couple of minutes for all probes to perform the measurements:</p> </p> That’s it! You can explore other types of Measurements and different options while performing them to customise your use case.</p> Summary#</a></h2> Atlas Probes are pretty easy to host and they are crucial in giving valuable insights about how different clients connect to upstreams and spot issues in Networks. If you can host a probe, you should seriously consider doing that and help increase the number of probes in the network. As of today, the number of probes in India is just around 113 and that is seriously very less.</p> If you need any help hosting one, don’t hesitate to contact me on my Twitter.</p> Oh, and for the record, I am hosting 2 Probes currently. Probe 1000991</a> hosted on my local RPi connected to my home network (ASN24560) and Probe 1001117</a> hosted on a DigitalOcean droplet in Bengaluru (ASN14061).</p> </p> Fin!</p> Monitoring my home network 2020-09-18T08:10:55+05:30 I like monitoring stuff</em>. That’s what I do at work and when my home ISP started giving me random problems and I decided it would be nice to monitor my home network as well. There are a couple of ways to go around this, a very popular and OSS solution is SmokePing</a>. SmokePing is written in Perl and is used to visualise network latencies. It’s quite a great solution but for my current stack which involves Prometheus and Grafana, it meant I had to deploy a standalone tool separate from my monitoring stack - something which I wanted to avoid.</p> </p> So, I looked for other solutions and luckily happened to stumble upon oddtazz</a> in one of the common Telegram groups where he shared his solution for the above: Telegraf ICMP plugin</a> and Grafana. This is exactly what I’ve been looking for but for some reason, I had wrongly assumed Telegraf needs InfluxDB to store the data. Googling a bit more, I found Telegraf supports</a> Prometheus format (amongst a huge list of others!) but this wasn’t so clear in their docs.</p> I decided to run a Telegraf agent in my RPi connected to my home router over LAN and scrape metrics using Prometheus and visualise graphs in Grafana! For the non-patient readers, here’s what my dashboard looks like!:</p> </p> </p> Setup#</a></h2> To get started, we need to download Telegraf</a> and configure the Ping</a> plugin. Telegraf has the concept of Plugins</strong> for Input, Output, Aggregating and Processing. What this basically means is that you can configure multiple input plugins like DNS, ICMP, HTTP and export the data of these plugins in a format of your choice with Output plugins. This makes Telegraf extermely extensible, you could write a plugin (in Go) of your choice if you fancy that as well!</p> Here’s what my telegraf.conf</code> looks like:</p> #</span> Input plugins</span></span> </span> #</span> Ping plugin</span></span> [[</span>inputs</span>.</span>ping</span>]]</span></span> urls</span> =</span> [</span>"</span>mrkaran.dev</span>"</span>,</span> "</span>tailscale.mrkaran.dev</span>"</span>,</span> "</span>floyd.mrkaran.dev</span>"</span>,</span> "</span>1.1.1.1</span>"</span>,</span> "</span>kite.zerodha.com</span>"</span>,</span> "</span>google.com</span>"</span>,</span> "</span>reddit.com</span>"</span>,</span> "</span>twitter.com</span>"</span>,</span> "</span>amazon.in</span>"</span>,</span> "</span>zerodha.com</span>"</span>]</span></span> count</span> =</span> 4</span></span> ping_interval</span> =</span> 1.0</span></span> timeout</span> =</span> 2.0</span></span> </span> #</span> DNS plugin</span></span> [[</span>inputs</span>.</span>dns_query</span>]]</span></span> servers</span> =</span> [</span>"</span>100.101.134.59</span>"</span>]</span></span> domains</span> =</span> [</span>"</span>mrkaran.dev</span>"</span>,</span> "</span>tailscale.mrkaran.dev</span>"</span>,</span> "</span>floyd.mrkaran.dev</span>"</span>,</span> "</span>1.1.1.1</span>"</span>,</span> "</span>kite.zerodha.com</span>"</span>,</span> "</span>google.com</span>"</span>,</span> "</span>reddit.com</span>"</span>,</span> "</span>twitter.com</span>"</span>,</span> "</span>amazon.in</span>"</span>,</span> "</span>zerodha.com</span>"</span>]</span></span> </span> #</span> Output format plugins</span></span> [[</span>outputs</span>.</span>prometheus_client</span>]]</span></span> listen</span> =</span> "</span>:9283</span>"</span></span> metric_version</span> =</span> 2</span></span></code></pre> Firstly, so nice to see an Ops</em> tool not</strong> using YAML</code>. Kudos to Telegraf for that. I’d love to see other tools follow suit.</p> Getting back to the configuration part, input.plugin</code> is a list of plugins that can be configured and I have configured the Ping and DNS plugin in my config. The output</code> is in Prometheus format so it can be scraped and ingested by Prometheus’ time-series DB.</p> Running Telegraf#</a></h3> With the above config in place, let’s try running the agent and see what metrics we get. I am using official</a> Docker image to run the agent with the following config:</p> docker</span> run</span> -</span>-name</span> telegraf-agent</span> -</span>-restart</span> always</span> -</span>d</span> -</span>p</span> 9283:9283</span> -</span>v</span> $</span>PWD</span>/telegraf.conf:/etc/telegraf/telegraf.conf:ro</span> telegraf</span></span></code></pre> After running the above command, you should be able to see the metrics at localhost:9283/metrics</code></p> $</span> curl</span> localhost:9283/metrics</span> \|</span> head</span></span> %</span> Total</span> %</span> Received</span> %</span> Xferd</span> Average</span> Speed</span> Time</span> Time</span> Time</span> Current</span></span> Dload</span> Upload</span> Total</span> Spent</span> Left</span> Speed</span></span> 0</span> 0</span> 0</span> 0</span> 0</span> 0</span> 0</span> 0</span> -</span>-:--:--</span> -</span>-:--:--</span> -</span>-:--:--</span> 0#</span> HELP</span> dns_query_query_time_ms</span> Telegraf</span> collected</span> metric</span></span> #</span> TYPE dns_query_query_time_ms untyped</span></span> dns_query_query_time_ms</span>{</span>d</span>c</span>=</span>"</span>floyd</span>"</span>,domain</span>=</span>"</span>amazon.in</span>"</span>,host</span>=</span>"</span>work</span>"</span>,rack</span>=</span>"</span>work</span>"</span>,rcode</span>=</span>"</span>NOERROR</span>"</span>,record_type</span>=</span>"</span>NS</span>"</span>,result</span>=</span>"</span>success</span>"</span>,server</span>=</span>"</span>100.101.134.59</span>"</span>}</span> 124.096472</span></span> dns_query_query_time_ms</span>{</span>d</span>c</span>=</span>"</span>floyd</span>"</span>,domain</span>=</span>"</span>google.com</span>"</span>,host</span>=</span>"</span>work</span>"</span>,rack</span>=</span>"</span>work</span>"</span>,rcode</span>=</span>"</span>NOERROR</span>"</span>,record_type</span>=</span>"</span>NS</span>"</span>,result</span>=</span>"</span>success</span>"</span>,server</span>=</span>"</span>100.101.134.59</span>"</span>}</span> 136.793673</span></span> dns_query_query_time_ms</span>{</span>d</span>c</span>=</span>"</span>floyd</span>"</span>,domain</span>=</span>"</span>kite.zerodha.com</span>"</span>,host</span>=</span>"</span>work</span>"</span>,rack</span>=</span>"</span>work</span>"</span>,rcode</span>=</span>"</span>NOERROR</span>"</span>,record_type</span>=</span>"</span>NS</span>"</span>,result</span>=</span>"</span>success</span>"</span>,server</span>=</span>"</span>100.101.134.59</span>"</span>}</span> 122.780946</span></span> dns_query_query_time_ms</span>{</span>d</span>c</span>=</span>"</span>floyd</span>"</span>,domain</span>=</span>"</span>mrkaran.dev</span>"</span>,host</span>=</span>"</span>work</span>"</span>,rack</span>=</span>"</span>work</span>"</span>,rcode</span>=</span>"</span>NOERROR</span>"</span>,record_type</span>=</span>"</span>NS</span>"</span>,result</span>=</span>"</span>success</span>"</span>,server</span>=</span>"</span>100.101.134.59</span>"</span>}</span> 137.915851</span></span> dns_query_query_time_ms</span>{</span>d</span>c</span>=</span>"</span>floyd</span>"</span>,domain</span>=</span>"</span>twitter.com</span>"</span>,host</span>=</span>"</span>work</span>"</span>,rack</span>=</span>"</span>work</span>"</span>,rcode</span>=</span>"</span>NOERROR</span>"</span>,record_type</span>=</span>"</span>NS</span>"</span>,result</span>=</span>"</span>success</span>"</span>,server</span>=</span>"</span>100.101.134.59</span>"</span>}</span> 111.097483</span></span></code></pre> Perfect! Now, we’re all set to configure Prometheus to scrape the metrics from this target. In order to do that you need to add a new Job</a>:</p> -</span> j</span>ob_name</span>:</span> "</span>ispmonitor</span>"</span></span> s</span>crape_interval</span>:</span> 6</span>0s</span></span> s</span>tatic_configs</span>:</span></span> -</span> t</span>argets</span>:</span> [</span>"</span>100.94.241.54:9283</span>"</span>]</span> #</span> RPi telegraf Agent</span></span></code></pre> In the above config, I am plugging my Tailscale IP assigned to my RPi on the port where Telegraf agent is bound to. This is one of the many</strong> reasons why Tailscale is so bloody awesome! I can connect different components in my network to each other without setting up any particular firewall rules, exposing ports on a case by case basis.</p> Sidenote</strong>: If you haven’t read Tailscale’s amazing</strong> NAT Traversal blog post</a>, do yourself a favour and check it out after you finish reading this one ofcourse!</p> Anyway, coming back to our Prometheus setup, we can see the metrics being ingested:</p> </p> Show me the graphs#</a></h2> Now comes the exciting bit – making pretty</strong> graphs. First, let’s discuss what’s the most important data I can extract out of Ping</code> and DNS</code> plugins. These plugins export decent amount of data, but a good rule of thumb while making dashboards is to optimise signal v/s noise ratio. We’ll do that by filtering out only the metrics that we care for.</p> Let’s checkout all the metrics exported by Ping</code> plugin:</p> $</span> curl</span> localhost:9283/metrics</span> \|</span> grep</span> ping</span> \|</span> grep</span> TYPE</span></span> #</span> TYPE ping_average_response_ms untyped</span></span> #</span> TYPE ping_maximum_response_ms untyped</span></span> #</span> TYPE ping_minimum_response_ms untyped</span></span> #</span> TYPE ping_packets_received untyped</span></span> #</span> TYPE ping_packets_transmitted untyped</span></span> #</span> TYPE ping_percent_packet_loss untyped</span></span> #</span> TYPE ping_result_code untyped</span></span> #</span> TYPE ping_standard_deviation_ms untyped</span></span> #</span> TYPE ping_ttl untyped</span></span></code></pre> Perfect! So, from the above list of metrics, the most important ones for us are:</p> ping_average_response_ms</code>: Avg RTT for a packet</li> ping_maximum_response_ms</code>: Max RTT for a packet</li> ping_percent_packet_loss</code>: % of packets lost on the way</li> </ul> With just the above 3 metrics, we can answer questions like:</p> Is my ISP suffering an outage?</strong></li> </ul> If yes, ping_percent_packet_loss</code> should be unusually higher than normal. This usually happens when the ISP has routing is borked and that causes the packet to be routed in a less optimized way and as a side effect packet loss becomes one of the key metrics to measure here.</p> Is the upstream down?</strong></li> </ul> If yes, ping_average_response_ms</code> over a recent window should be higher than a window compared to a previous time range when things were fine and dandy. This can either mean 2 things: Either your ISP isn’t routing correctly to the said upstream or the CDN/Region where your upstream is faced an outage. This is quite a handy metric for me to monitor!</p> How many times have your friends complained “xyz.com</code> isn’t working for me” and when you try to load, it’s fine from your end? There are a lot of actors at play but ping</code> is usually the most simple and quickest way to detect whether an issue persists or not. Of course, this doesn’t work for hosts which block ICMP packets altogether. They are not rare either, like netflix.com</code> and github.com</code> both block ICMP probes for example. For my use case, this wasn’t a major issue as I was able to still probe a decent amount of upstreams all over the world.</p> With that out of the way, let’s break the dashboard into different components and see what goes behind them.</p> Ping Response Panel#</a></h3> </p> To plot this, simply choose a Stat</code> visualisation with the query ping_average_response_ms{url="$url"}</code>. Repeat this panel for the variable $url</code> and you should be able to generate a nice row view like this.</p> Additonally you can choose Thresholds and the Unit to be displayed in the panel with these options.</p> </p> Ping Response Time Graph#</a></h3> The next graph is interesting, it lets me visualise the avg, min, max ping response time as well as the % packet loss plotted on the Y2 (right Y) axis.</p> </p> Availability Panel#</a></h3> An interesting query to calculate uptime (just in the context whether the upstream is reachable) is:</p> 100 - avg_over_time(ping_percent_packet_loss[2m])</span></span></code></pre> Since I scrape metrics at an interval of 1m</code>(in order to not ping too frequently and disrupt my actual browsing experience), in this query I am averaging the data points for the metric ping_percent_packet_loss</code> in a [2m]</code> window.</p> </p> DNS Response Time Graph#</a></h3> We can similarly query the DNS response time by visualising the average response time for a DNS query. This might be useful only to people self-hosting their DNS servers.</p> </p> Conclusion#</a></h2> So with a pretty simple and minimal OSS solution, I was able to setup monitoring for my home network! Over the last few days whenever my ISP had slightest of trouble, I can correlate it with my metrics! I mean I still can’t do anything about it cause the other person on ISP’s customer support is “Did you try rebooting your router” – the quintessential solution to all tech problems. Wish we could reboot this entire damn 2020 as well, but one could hope!</p> Shoot me for any questions on my Twitter @mrkaran_</a> :)</p> Fin!</p> Using Tailscale for home server 2020-04-23T08:10:55+05:30 For those of you who are new to my blog, I had written about my home server setup</a> (hydra</code>) a few months back. Since then I’ve tweaked the setup a bit and made some changes to how I organise/deploy applications inside my cluster. I’ll be talking about the updates made so far and the reason behind them.</p> A brief overview of what has changed from hydra</code> v0.1 to hydra</code> v0.2:</p> Replaced Wireguard</strong> with Tailscale</strong></p> </li> Added a new worker node (residing in DigitalOcean, Bengaluru) to the existing RPi k3s cluster.</p> </li> Shifted from PiHole to Adguard DNS upstreaming to Unbound.</p> </li> Containerised all workloads and deployed on K8s (using k3s</code> platform).</p> </li> </ul> The setup looks something like this now:</p> </p> Let’s take a quick look at some of the above things in detail!</p> Using Tailscale#</a></h2> So, before we jump to why I started using Tailscale, let’s address a few things about Wireguard which bothered me.</p> Adding new devices is a real PITA and quite often laziness kicks in to generate a QR code or Private/Public key pair. So, for those of you unaware, Wireguard needs a client config to add the server endpoint and it’s public key for the client to communicate with the server. You need to add the client’s private key on the server-side as well for the exchange of encrypted packets to happen. Doing all of this manually has been one of the reasons I’ve not added/updated my devices as regularly as I’d like to do. For eg, on my recently bought iPad, I just haven’t bothered to do all of this, cause ugh I am lazy?</p> </li> Having a central VPN server to talk to my RPi from my local network just doesn’t seem</em> right you know? Especially when both devices are literally lying in the same room, sending the packets to a server somewhere in Electronic City (DO, blr region) from JP Nagar (where I live) feels like totally unnecessary. I really needed a mesh network to reduce wasted bandwidth and latencies. Having a central VPN server is also a SPOF, not that my home-server runs any mission-critical workloads, but still good to avoid where we can. Where else can I flex my ops-chops skills if not my home server, eh?</p> </li> </ul> I started looking at different mesh VPN setups and Tailscale attracted me the most. I heard about Tailscale first time when I saw bradfitz’s</a> post about leaving Google and joining Tailscale on HN. Interestingly, Tailscale is built on top of wireguard-go</a> (the userspace implementation of Wireguard</a> kernel module). Since I was already familiar with Wireguard and had been using it for almost a year, I got curious on Tailscale.</p> Tailscale basically sets up a point-to-point</em> encrypted network</strong> across all your devices. What that means is there’s no “relay” (like Wireguard server) and clients can talk to each other directly</em>. While it may seem that it is easy to set this up with a bunch of shell scripting-foo (just add all Wireguard peers in each peer config), Tailscale attempts to do a lot</em> of the heavy lifting by making the network seamless and handling authentication.</p> Coordination Server#</a></h3> So, when we add a new device to Wireguard server, we basically need to generate a private/public key pair as explained above. When you’re setting up Tailscale agent, it does all of this in the background and asks you to authorise to a central co-ordination server. Now, this server is only used to exchange information about each peer in your network. The tailscale agent uploads the private/public key information of the peer you are currently on and any time any new peer joins the network, all of the agents’ configurations are updated real-time. The coordination server periodically checks for any new changes and pushes the updates to the agent on each node.</p> The real juice is that they authenticate via OAuth, OIDC or SAML. Which means that you can use your existing 2FA configurations to authenticate a new node in the network. However, this might be a point of concern for some users, but I chose convenience here. Also, since the traffic flows through the nodes directly, and is encrypted there’s not much to worry here. I’ve been following Tailscale closely and they do plan to opensource the co-ordination server in future, so maybe when that happens I’ll self-host it.</p> NAT Traversal#</a></h3> Apart from handling the auth, Tailscale does a pretty good job at handling connectivity across different network topologies. So, basically, if you want to peer 2 devices over the public internet you’d need to perform NAT Traversal. There are several NAT hole punching techniques which allow traversing NAT but since they are not standardised and sometimes due to erratic NAT behaviour, it poses quite a challenge to do it seamlessly</em>. And, I’m not even talking about roaming networks as of yet.</p> Tailscale agents can perform NAT traversal using ICE</a> and STUN</a>. What all of this practically means is if you’re sitting in a cafe somewhere and you want to access any of your internal service it is</strong> possible without messing around any firewall ports :D</p> TL;DR</strong>: I decided to give Tailscale a shot and was quite impressed by how easy it was to setup. You can refer to the official docs for more instructions but I was able to set it up across 2 RPis and have them talk to each other under 15 minutes. I think that’s quite an awesome feat in itself. The only sad bit is they don’t have an Android app yet, so I am eagerly waiting for it.</p> Hybrid K3s Cluster#</a></h2> So, I am running a k3s</code> cluster on my RPi. At that time I was still using DO for running Pihole, Unbound, public DNSCrypt resolver etc. I decided to standardise the ad-hoc deployments to manage them efficiently. It also allowed me to play around with more on K8s which was my original goal behind buying these Pi was.</p> Now, I’ve 2RPi nodes, the k8s master node runs on the 4GB RPi while the 2GB variant serves as a worker node. I decided to get a bit fancy with my setup and hooked up the k3s</code> agent installation script on a DO node andta-da!</em> I have a multi-arch(amd64</code> and arm64</code>), hybrid (bare metal and cloud) K8s cluster ready! I think if I sprinkle some ML/AI + Bitcoins in the above setup, I’m all set to raise VC funding for hydra</code>.</p> I wanted to learn Terraform</a> as part of my work at my org as well, so I created + managed the entire droplet through terraform</code>. The script</a> has modules to provision a droplet, attach an elastic IP, configure firewall rules and add my SSH key to the server. Quite easy to manage and I am generally a hater of GUIs, so Terraform is indeed a blessing in disguise.</p> I know some of my opinions are a bit strong but don’t worry I get meme’d/burnt for this almost every day by my work colleagues.</p> </blockquote> Pihole to Adguard#</a></h2> While Pihole works really well for blocking Ads it had some features lacking, particularly DOT and DOH support out of the box</strong>. I decided I’ll shift to Adguard as the codebase is in Go - something which I am a bit familiar with and also the UI feels a bit sleek and refreshing too!</p> </p> Accessing internal services#</a></h3> A major challenge for me was however to configure access to internal services on the K8s cluster. Since I have bare metal worker nodes, it’s not possible to deploy a cloud load balancer. For now, I went with a really simple old school solution, to expose an Nginx proxy front-ending all my services through NodePort. I am planning to look at Traefik or Istio for this but I wanted to just shipit! at this point.</p> Here’s a very basic example of an nginx</code> config for ip.mrkaran.dev</code> that I run on my cluster:</p> server {</span></span> server_name ip.mrkaran.dev;</span></span> # reverse proxy</span></span> location / {</span></span> proxy_pass http://100.96.239.6:30506; # tailscale IP, connecting to NodePort service</span></span> include fluff/proxy.conf;</span></span> }</span></span> }</span></span></code></pre> 30506</code> port is exposed by the Service</code> object backing up the pods for that application. Since a NodePort</code> service is available on any K8s node, you can give the Tailscale IP for any node and the routing will be handled by kube-proxy</code>.</p> Challenges faced#</a></h2> Mesh or mess#</a></h3> Setting all of this up didn’t come without its own challenges or hurdles. Right off the bat, the first problem was that I was seeing really high latencies from my RPi node to DO node through Tailscale. Now, since both the nodes are physically in Bangalore and they are connecting to each other (or that’s what I had presumed), I didn’t expect latencies to be as high as 500-600ms. Bollocks!</p> Eventually, I’d figured thanks to my super restrictive rules on the firewall of DO node, I had blocked all inbound UDP connection. That means NAT hole punching through STUN is simply not possible. In such cases, Tailscale forwards all packets to an encrypted TURN server(called DERP - Designated Encrypted Relay for Packets), which basically are TCP encrypted relays. Tailscale manages this network of DERPs and the one I got connected to was somewhere in the USA.</p> Bottom line, I was all pumped up to talk to a DO node from my RPi node but as it turns</em> out (no pun intended) my packets were flowing through the USA! Ah so bad. Anyway, opening up UDP from the Tailscale subnet, fixed the issue and latencies were back to being sub 10ms. Yay!</p> Overlay over an overlay#</a></h3> Next up, were problems with K3s networking from the DO node to RPi node. The DO node was in a NotReady</code> state because the agent couldn’t reach the server:</p> Apr</span> 14</span> 09:00:33</span> hydra-control</span> k3s[19746]:</span> I0414</span> 09:00:33.306650</span> 19746</span> log.go:172]</span> http:</span> TLS</span> handshake</span> error</span> from</span> 100.97.222.106:51516:</span> read</span> tcp</span> 100.96.239.6:6443</span>-</span>></span>100.97.222.106:51516:</span> read:</span> connection</span> reset</span> by</span> peer</span></span></code></pre> Through some trial and error and reading the docs</a>, I figured that flannel</code> is running as a CNI in k3s</code>. Now the problem is flannel itself is an overlay network. But… Tailscale already is an overlay network (Wireguard) so the packets are not being routed correctly and being dropped halfway in the master node (I am guessing the DNAT/SNAT translation botched up here).</p> The trick was to just change</a> flannel backend to run in the host namespace only. That solved the above issue for me.</p> However, I still had one more issue. The DO node’s public</strong> IP was being advertised, while the agent was running on Tailscale network interface so the master was never able to reach the agent. Similarly, when the agent tried to communicate with the server, the private IP of the node was being advertised. Setting the --node-external-ip <tailscale-ip></code> in k3s</code> config seemed to have fixed the problem.</p> Now all of the nodes in the cluster had proper Tailscale IPs advertised and the node went to Ready</code> state, at last!</p> Who let the DNS out#</a></h3> So, I’ve a chicken and egg problem in my setup. Since my laptop</code> runs a Tailscale agent and whenever I boot up my systemm, Tailscale</code> attempts to posts logs to log.tailsclae.io</code> and fails</strong> to start if it cannot. The problem here is who resolves the DNS for me?</p> I run a local DNS server with CoreDNS</a> forwarding my queries to Adguard. Now if I can’t reach Adguard (since Tailscale agent hasn’t initialised), how am I supposed to resolve log.tailscale.io</code>? I did what any sane guy would do, write a simple hacky</em> bash script:</p> #!</span>/bin/bash</span></span> sudo</span> chattr</span> -</span>i</span> /etc/resolv.conf</span></span> sudo</span> echo</span> '</span>nameserver 1.1.1.1</span>'</span> ></span> /etc/resolv.conf</span></span> echo</span> "</span>changed dns server to 1.1.1.1</span>"</span></span> sudo</span> tailscale</span> up</span></span> sudo</span> echo</span> '</span>nameserver 127.0.0.1</span>'</span> ></span> /etc/resolv.conf</span></span> echo</span> "</span>changed dns server back to 127.0.0.1</span>"</span></span> sudo</span> chattr</span> +i</span> /etc/resolv.conf</span></span></code></pre> Yes, it’s quite insane. Also, I’ve not been able to figure out how to stop NetworkManager</code> from changing my /etc/resolv.conf</code> so I rely on a hack (documented in Arch official docs</a>), to lock</code> the file so any process cannot modify it. Quirky, but works!</p> Storage#</a></h3> Unfortunately, I don’t have any external HDD/SSD so I am postponing running any stateful workloads till I get one soon (whenever lockdown gets over in my area). I plan to deploy an NFS server so I can run stateful workloads across any node and have redundancy in form of cloud backups. I’ve also heard cool things about Longhorn</a> but unfortunately, it doesn’t have ARM support</a>.</p> Final Thoughts#</a></h2> Well, I am quite stoked by my current setup right now. Learnt a bunch of cool things around NAT traversal, zero-trust networking, realised the old days of LAN were so much better (not that I am that old to have experienced, my first internet connection was rather broadband at home, not even dial-up). Tailscale opens up a lot of new opportunities for corporate VPNs and it is something definitely to watch out for as they continue improving the product.</p> Also, I was super elated when bradfitz</a> himself had commented on my tweet, not gonna lie!</p> {{< tweet 1249552822674149376 >}}</p> Credits#</a></h3> First, a major thanks to my friend sarat</a> with whom I pair debugged a lot of the above issues and since he also runs a home server, he was my go-to person to ask doubts!</p> Here are some links that I collected while setting all of this up and might be useful as references if you’re planning a similar setup!</p> How Tailscale Works</a></li> What are these 100.x.y.z addresses?</a></li> k3s config reference</a></li> P2P across NAT</a></li> Remembering the LAN</a></li> Cloudskew</a> (wonderful tool for creating architecture diagrams)</li> </ul> If you’ve any questions please find me on Twitter @mrkaran</a>. You can find all of the setup on GitHub</a>.</p> Till next time, if we survive the pandemic!</p> DNS Lookups in Kubernetes 2020-02-02T10:57:55+05:30 One of the primary advantages of deploying workloads in Kubernetes is seamless application discovery. Intra cluster communication becomes easy with the concept of Service</a> which represents a Virtual IP backing a set of Pod IPs. For example, if vanilla</code> service needs to talk to a chocolate</code> service, it can directly use the Virtual IP for chocolate</code>. Now the question is who resolves the DNS query for chocolate</code> and how?</p> DNS resolution is configured in Kubernetes cluster through CoreDNS</a>. The kubelet configures each Pod’s /etc/resolv.conf</code> to use the coredns pod as the nameserver</code>. You can see the contents of /etc/resolv.conf</code> inside any pod, they’ll look something like:</p> search</span> hello.svc.cluster.local</span> svc.cluster.local</span> cluster.local</span></span> nameserver</span> 10.152.183.10</span></span> options</span> ndots:5</span></span></code></pre> This config is used by DNS clients to forward the DNS queries to a DNS server. resolv.conf</code> is the resolver configuration file which has information about:</p> nameserver</strong> : Where the DNS queries are forwarded to. In our case this is the address of CoreDNS</code> service.</p> </li> search</strong>: Represents the search path for a particular domain. Interestingly google.com</code> or mrkaran.dev</code> is not FQDN (fully qualified domain name). A standard convention that most DNS resolvers follow is that if a domain ends with .</code> (representing the root zone), the domain is considered to be FQDN. Some resolvers try to act smart and append the .</code> themselves. So mrkaran.dev.</code> is an FQDN but mrkaran.dev</code> is not.</p> </li> ndots</strong>: This is the most interesting value and is the highlight</em> of this post. ndots</code> represents the threshold value of the number of dots in a query name to consider it as a “fully qualified” domain name. More on this later, as we discover the flow of DNS lookup.</p> </li> </ul> </p> Let’s check what happens when we query for mrkaran.dev</code> in a pod.</p> $</span> nslookup</span> mrkaran.dev</span></span> Server:</span> 10.152.183.10</span></span> Address:</span> 10.152.183.10#53</span></span> </span> Non-authoritative</span> answer:</span></span> Name:</span> mrkaran.dev</span></span> Address:</span> 157.230.35.153</span></span> Name:</span> mrkaran.dev</span></span> Address:</span> 2400:6180:0:d1::519:6001</span></span></code></pre> For this experiment, I’ve also turned on CoreDNS logging level to all</code> which makes it highly verbose. Let’s look at the logs of coredns</code> pod:</p> [</span>INFO</span>]</span> 10.1.28.1:35998 - 11131 </span>"</span>A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512</span>"</span> NXDOMAIN qr,aa,rd 146 0.000263728s</span></span> [</span>INFO</span>]</span> 10.1.28.1:34040 - 36853 </span>"</span>A IN mrkaran.dev.svc.cluster.local. udp 47 false 512</span>"</span> NXDOMAIN qr,aa,rd 140 0.000214201s</span></span> [</span>INFO</span>]</span> 10.1.28.1:33468 - 29482 </span>"</span>A IN mrkaran.dev.cluster.local. udp 43 false 512</span>"</span> NXDOMAIN qr,aa,rd 136 0.000156107s</span></span> [</span>INFO</span>]</span> 10.1.28.1:58471 - 45814 </span>"</span>A IN mrkaran.dev. udp 29 false 512</span>"</span> NOERROR qr,rd,ra 56 0.110263459s</span></span> [</span>INFO</span>]</span> 10.1.28.1:54800 - 2463 </span>"</span>AAAA IN mrkaran.dev. udp 29 false 512</span>"</span> NOERROR qr,rd,ra 68 0.145091744s</span></span></code></pre> Whew. So 2 things piqued my interest here:</p> The query iterates through all search paths until the answer contains a NOERROR</code> code (which the DNS clients understand and store it as the result). NXDOMAIN</code> simply indicates no record found for that domain name. Since mrkaran.dev</code> isn’t an FQDN (according to ndots=5</code> setting), the resolver looks at search path and determines the order of query.</p> </li> A</code> and AAAA</code> records are fired parallelly. This is because single-request</code> option in /etc/resolv.conf</code> has a default configuration to perform parallel IPv4 and IPv6 lookups. You can disable this using single-request</code> option.</p> </li> </ul> Note</em>: glibc</code> can be configured to send these requests sequentially but musl</code> cannot, so Alpine users must take note.</p> Playing around with ndots#</a></h3> Let’s play around with ndots</code> a bit more and see how it behaves. The idea is simple, for the DNS client to know whether a domain is an absolute one or not is through this ndots</code> setting. For example, if you query for google</code> simply, how will the DNS client know if this is an absolute domain. If you set ndots</code> as 1</code>, the DNS client will say “oh, google</code> doesn’t have even one 1 dot, let me try going through search list. However, if you query for google.com</code>, the search list will be completely ignored since the query name satisfies the ndots</code> threshold (At least one dot).</p> We can see this by actually doing it:</p> $</span> cat</span> /etc/resolv.conf</span></span> options</span> ndots:1</span></span></code></pre>$</span> nslookup</span> mrkaran</span></span> Server:</span> 10.152.183.10</span></span> Address:</span> 10.152.183.10#53</span></span> </span> </span></span> server can</span>'</span>t find mrkaran: NXDOMAIN</span></span></code></pre> CoreDNS logs:</p> [</span>INFO</span>]</span> 10.1.28.1:52495 - 2606 </span>"</span>A IN mrkaran.hello.svc.cluster.local. udp 49 false 512</span>"</span> NXDOMAIN qr,aa,rd 142 0.000524939s</span></span> [</span>INFO</span>]</span> 10.1.28.1:59287 - 57522 </span>"</span>A IN mrkaran.svc.cluster.local. udp 43 false 512</span>"</span> NXDOMAIN qr,aa,rd 136 0.000368277s</span></span> [</span>INFO</span>]</span> 10.1.28.1:53086 - 4863 </span>"</span>A IN mrkaran.cluster.local. udp 39 false 512</span>"</span> NXDOMAIN qr,aa,rd 132 0.000355344s</span></span> [</span>INFO</span>]</span> 10.1.28.1:56863 - 41678 </span>"</span>A IN mrkaran. udp 25 false 512</span>"</span> NXDOMAIN qr,rd,ra 100 0.034629206s</span></span></code></pre> Since mrkaran</code> didn’t specify any .</code> so the search list was used to find the answer.</p> Note</em>: ndots</code> value is silently capped to 15</code> and is 5</code> in Kubernetes as default.</p> Handling this in Production#</a></h3> If your application is of the nature that makes a lot of external network calls, the DNS can become a bottleneck in case of heavy traffic since a lot of extra queries are made before the real DNS query is even fired. It’s quite uncommon to see applications appending the root zone in the domain names, but that can be considered as a hack. So instead of using api.twitter.com</code>, you can hardcode your application to include api.twitter.com.</code> which would force the DNS clients to do an authoritative lookup directly on the absolute domain.</p> Alternatively, since K8s 1.14, the dnsConfig</code> and dnsPolicy</code> feature gates have become stable. So while deploying a pod you can specify ndots</code> setting to something lesser, say 3</code> or if you want to be really aggressive you can turn it down to 1</code>. The consequences of this will be that every intra-node communication now has to include the full domain. This is one of the classic tradeoffs where you have to choose between performance and portability. If the app doesn’t demand super low latencies, I guess you need not worry about this at all since DNS results are cached internally too.</p> References#</a></h3> I got to know about this peculiarity first, in a K8s meetup</a> which I went to, last weekend where the folks mentioned about having to deal with this.</p> Here are some additional links you can read on the web:</p> Explainer on why ndots=5 in kubernetes</a></li> Great read on how ndots affects application performance</a></li> musl and glibc resolver inconsistencies</a></li> </ul> Note</em>: I’m particularly not using dig</code> in this post. dig</code> apparently automatically adds a .</code> (root zone identifier) to make the domain an FQDN one without</strong> even first going through the search path. I’ve mentioned about this briefly in one of my older</a> posts. Nonetheless, it’s quite surprising to see that you need to give a flag to make it behave in what seems to be a standard way.</p> It’s always DNS, isn’t it ;)#</a></h4> Fin!</p> Update#</a></h3> 2020-12-24</strong>:</p> In case you want to play around with ndots</code> and DNS more, I’ve worked on a DNS Client which lets you tweak these params on the fly. Feel free to checkout doggo</a> if interested!</p> Resize PVC in Kubernetes 2020-01-14T08:10:55+05:30 Well, the title is self explanatory so let’s begin!</p> First off, we need to ensure that the StorageClass</code> which was used to provision the PVC has the correct configuration. From the official docs</a>:</p> You can only expand a PVC if its storage class’s allowVolumeExpansion field is set to true.</p> </blockquote> So, let’s inspect our storage class:</p> $</span> kubectl</span> get</span> sc</span> #</span> sc is short for storageclass</span></span> NAME</span> PROVISIONER</span> AGE</span></span> gp2</span> (default</span>) kubernetes.io/aws-ebs 8d</span></span> </span> $</span> kubectl</span> describe</span> sc/gp2</span></span> #</span> output redacted to focus only on the field we're concerned with</span></span> Name:</span> gp2</span></span> AllowVolumeExpansion:</span> True</span></span></code></pre> If AllowVolumeExpansion</code> is set to True</code> you can skip the below step. If it’s not true, you need edit the field allowVolumeExpansion</code> as true</code>:</p> a</span>piVersion</span>:</span> s</span>torage.k8s.io/v1</span></span> k</span>ind</span>:</span> S</span>torageClass</span></span> m</span>etadata</span>:</span></span> a</span>nnotations</span>:</span></span> s</span>torageclass.kubernetes.io/is-default-class</span>:</span> "</span>true</span>"</span></span> n</span>ame</span>:</span> g</span>p2</span></span> p</span>arameters</span>:</span></span> f</span>sType</span>:</span> e</span>xt4</span></span> t</span>ype</span>:</span> g</span>p2</span></span> p</span>rovisioner</span>:</span> k</span>ubernetes.io/aws-ebs</span></span> r</span>eclaimPolicy</span>:</span> D</span>elete</span></span> v</span>olumeBindingMode</span>:</span> W</span>aitForFirstConsumer</span></span> a</span>llowVolumeExpansion</span>:</span> true</span></span></code></pre> Once the StorageClass</code> config is correct, all we need to do is update with the new size.</p> So, for example if size of the PVC was 15GB</code> orginally:</p> s</span>pec</span>:</span></span> r</span>esources</span>:</span></span> r</span>equests</span>:</span></span> s</span>torage</span>:</span> 1</span>5Gi</span></span></code></pre> To update it to 30GB</code>, you simply need to edit spec.resources.requests</code> field:</p> s</span>pec</span>:</span></span> r</span>esources</span>:</span></span> r</span>equests</span>:</span></span> s</span>torage</span>:</span> 3</span>0Gi</span></span></code></pre> We now need to “apply” the updated PVC manifest.</p> $</span> kubectl</span> apply</span> -</span>f</span> pvc.yml</span></span></code></pre> Let’s take a look at the PVC:</p> $</span> kubectl</span> describe</span> pvc/myclaim</span></span> #</span> output redacted for brevity</span></span> .</span>.</span>.</span></span> Conditions:</span></span> Type</span> Status</span> LastProbeTime</span> LastTransitionTime</span> Reason</span> Message</span></span> ----</span> -</span>-----</span> -</span>----------------</span> -</span>-----------------</span> -</span>-----</span> -</span>------</span></span> FileSystemResizePending</span> True</span> Mon,</span> 01</span> Jan</span> 0001</span> 00:00:00</span> +0000</span> Tue,</span> 14</span> Jan</span> 2020</span> 20:52:21</span> +0530</span> Waiting</span> for</span> user</span> to</span> (re-</span>)start a pod to finish file system resize of volume on node.</span></span> .</span>.</span>.</span></span></code></pre> So, basically what FileSystemResizePending</code> means is that while PVC is in use</em>, we have to either restart or delete the underlying Pod using the PVC. At the time of writing this, ExpandInUsePersistentVolumes</code> is still in beta</a> and has to be enabled as a feature gate. Sadly, EKS is still on 1.14</code> (while the world has moved to 1.17, such sloooow</em> release cycles!), so I couldn’t enable this in my case.</p> Once the pod is restarted, the expanded disk is automagically available! Let’s verify this:</p> kc get pvc/myclaim -o=jsonpath="{.status.capacity.storage}"</span></span> 30Gi</span></span></code></pre> Now, compare this with the standard way of resizing an EBS volume in EC2 instance. You need to first modify</a> the volume size using AWS EBS API and then in the EC2 instance, use a combination of growpart</code> and resize2fs</code> to extend</a> the resized volume. This sounds much more cumbersome than simply updating the storage field in PVC manifest!</p> Fin!</p> sshuttle - A better ssh tunnel 2020-01-12T09:57:55+05:30 The Motivation#</a></h2> Sometime back I had to access a Kubernetes API server which was firewalled to a private VPC network. I didn’t want to setup a separate bastion instance just to access this cluster, cause TBH bastions are kinda redundant in K8s as every task can be performed through the client-server APIs using kubectl</code>. So, all I needed was access to this API server from a trusted network in a secure way. Thanks to my friend @sarat</a>, I got to know about sshuttle</a>. sshuttle</code> is quite unique in the sense that it’s not really a VPN but acts like one (for most practical purposes). sshuttle</code> lets you access an internal network through a trusted node inside the VPC, without you having to deal with the mess of port forwarding or VPNs.</p> The basic idea is pretty simple, sshuttle</code> starts a local python</code> server in your host machine and creates iptables</code> rules to route the destination packets of the specified CIDR blocks to this local server. At the server, the packets are multiplexed over an ssh</code> session and sent to the server. The server disassembles the multiplexed packet and the routes them to upstream. So, basically this is a clever hack to avoid TCP over TCP (which again is a mess on unreliable networks). Multiplexed streams on ssh</code> is just a single stateful TCP connection (as compared to VPN connections which are stateless). Now you must be wondering, how come the target server disassembles the packets. Yes, there needs to be some kind of sshuttle</code> daemon running which does that for you. This is where sshuttle</code> does some magic, it automagically</em> deploys a python script on your target host to perform this task. So yes, for sshuttle</code> to work, both the client and target need to have python </code> and iptables</code> installed.</p> Usage#</a></h3> sshuttle -r user@port x.x.x.x</code></p> All the packets routed to the CIDR block will now go through sshuttle</code> daemon, since it configured iptables</code> rules for them. Also, sshuttle</code> starts a local python server on your host machine. You can see it using netstat</code>:</p> $</span> sudo</span> netstat</span> -</span>tunapl</span> \|</span> grep</span> python</span></span> Proto</span> Recv-Q</span> Send-Q</span> Local</span> Address</span> Foreign</span> Address</span> State</span> PID/Program</span> name</span></span> tcp</span> 0</span> 0</span> 127.0.0.1:12300</span> 0.0.0.0:</span></span> LISTEN</span> 27425/python</span></span></code></pre> There’s a python</code> server listening on port 12300</code> in my host machine. To actually verify, this indeed is started by sshuttle</code>, you can use pstree -p \| less</code> and search for sshuttle</code>. Here you can see sshuttle</code> did indeed start a python</code> server and the PID (27425</code>) matches with the one we saw in netstat</code> command.</p> -zsh(13201</span>)---sshuttle(</span>27425</span>)-+-ssh(</span>27446</span>)---ssh(</span>27447</span>)</span></span> `</span>-sudo(27427</span>)---python(</span>27445</span>)</span></span></code></pre> You can even forward DNS queries with the --dns</code> flag. This is super helpful if you have something like Route53 to host your DNS records on a private zone (for eg tld like .internal</code>).</p> Better than SSH tunnels?#</a></h3> Yes, you can also port forward with ssh using: ssh -nNT -L <local-port>:{upstream-host}:{upstream-port} user@remote</code></p> The problem with ssh</code> tunnels is that they experience frequent packet loss on a normal WiFi connection and it’s quite frustrating to deal with them. Moreover, sometimes you need access to multiple ports in your private network which requires you to explictly provide them with -L</code> flag which I find it as cumbersome. Also, you cannot forward DNS queries (over UDP) since ssh</code> can only do TCP.</p> sshuttle</code> has made my life so simple!</p> Fin!</p> Use netcat for port scanning 2020-01-11T08:10:55+05:30 Quite often you’d need to check if a port on a target node is opened or blocked by firewall. I’ve always used telnet</code> to test that but it has a few drawbacks:</p> Need to use dirty hacks</a> in shell scripts to auto close the connection. Also, telnet outputs some errors to /dev/stdout</code> instead of the standard /dev/stderr</code> which makes it harder to use in scripts.</p> </li> Non standard implementation across different OSes. On Alpine Linux (mostly used in containers), if you install telnet using the /busybox-extras</code> package, the behaviour is different from what it is on standard Ubuntu/Arch environments. I’ve even faced weird issues on Alpine where telnet will simply wait endlessley for the connection to be established, while netcat would not indicate any issues.</p> </li> Telnet is actually a protocol and the telnet-client initiates the negotiation</a> with the server before a connection is established.</p> </li> </ul> So, after all these issues, I looked at other tools to eventually replace telnet</code> with something better. I tried nmap</code> which is also a port scanner, but is unreliable since a lot of hipster sysadmins drinking the security koolaid block port scanning tools like these. I wanted a dependable tooling and after a bit of Google-fu, I stumbled across netcat</code>.</p> netcat</code> is basically a swiss army knife to perform all kind of ops with TCP/UDP. You can create a file server, chat client/server, TCP client etc. We are simply interested in the port scanning abilities of this for this blog post, so let’s actually see how to use it for the same.</p> Note</strong>: Install netcat-openbsd</code> as it is a rewritten version of netcat-traditional</code> with some more bells and whistles.</p> The basic syntax for port scanning looks like:</p> nc -z host port</code></p> -z</code> tells nc to not send any data, just scan</em> for any process listening on the target port. This is much better (and faster) than telnet</code> client initiating a connection with the upstream.</p> To make it more usable however, let’s pepper our command with some helpful flags:</p> nc -vz -w 3 host port</code></p> -v</code> turns on verbose mode which outputs diagnostic messages. -w</code> adds the timeout for the connection to be established. If you want to set a timeout in telnet</code> there’s a hack</a> for it.</p> You can even supply a range of ports to netcat like:</p> nc -vz -w 3 host 8000-9000</code></p> Quick Tip</em>: You can also give an alias for port instead of the number. For example:</p> $</span>nc</span> -vz -w 3 google.com https</span></span> Connection</span> to</span> google.com</span> 443</span> port</span> [tcp/https</span>] succeeded</span>!</span></span> </span> $</span>nc</span> -vz -w 3 google.com ssh</span></span> nc:</span> connect</span> to</span> google.com</span> port</span> 22</span> (tcp</span>) timed out: Operation now in progress</span></span> nc:</span> connect</span> to</span> google.com</span> port</span> 22</span> (tcp</span>) failed: Network is unreachable</span></span></code></pre> Hope this post pretty much sums up the usage of netcat for port scanning! Read the man page</a> for more info.</p> Fin!</p> kubectl wait 2020-01-01T08:10:55+05:30 For the longest time I’ve had these commands in my .gitlab-ci.yml</code> file for a K8s CD pipeline:</p> .</span>.</span>.</span></span> -</span> kubectl</span> apply</span> -</span>k</span> overlays/prod</span></span> -</span> echo</span> "</span>Waiting for 15 seconds for pods to be restarted</span>"</span> &&</span> sleep</span> 15</span></span> -</span> kubectl</span> get</span> po</span></span> .</span>.</span>.</span></span></code></pre> So, basically I apply the changes to cluster using kubectl apply</code> and wait for arbitary decided time (15 seconds) to see the pod status</code>, hoping by that time the new deployments would have been active and old pods would be deleted. As the traditional SRE saying goes Hope is not a strategy</em> this was clearly hacky and I knew it back then, just didn’t priortise enough to find a replacement. Recently got to know about kubectl wait</code> and woah, this is exactly what I needed. I can wait till either the condition is true or a timeout happens, whichever is earlier. This is so much better than the previous hack</em>.</p> kubectl wait --for=condition=available --timeout=60s --all deployments</span></span></code></pre> Here the condition</code> depends on the resource you are selecting. You can see the values for Conditions</code> using kubectl describe <resource></code>. For eg, for deployment and pods:</p> $</span> kc</span> describe</span> deployments/{deployment_name}</span> \|</span> grep</span> Conditions</span> -</span>A</span> 5</span></span> </span> Conditions:</span></span> Type</span> Status</span> Reason</span></span> ----</span> -</span>-----</span> -</span>-----</span></span> Progressing</span> True</span> NewReplicaSetAvailable</span></span> Available</span> True</span> MinimumReplicasAvailable</span></span></code></pre>$</span> kc</span> describe</span> pods/{pod_name}</span> \|</span> grep</span> Conditions</span> -</span>A</span> 5</span></span> </span> Conditions:</span></span> Type</span> Status</span></span> Initialized</span> True</span></span> Ready</span> True</span></span> ContainersReady</span> True</span></span> PodScheduled</span> True</span></span></code></pre> So, now you’ll set the value for condition</code> according to your choice. This will be pretty useful in CI/CD pipelines. That’s pretty much it.</p> Unrelated, but I thought about doing more such short</em> posts and be consistent with more of writing. If you liked the short and precise format or have any feedback on it, do reach out to me on Twitter</a>.</p> Happy New Year :)</p> Fin!</p> Introducing kubekutr 2019-12-30T08:10:55+05:30 </p> kubekutr</a> was born out of my frustration of organising K8s resource manifests files/directories. For the uninitiated, K8s lets you hold the state of your cluster declaratively</a> as “manifest” files. K8s does so by a lot of asynchronous control loops to check whether the desired state matches the real world state and in case of drifts, it resets back to the user desired state (I oversimplified this, but hope you get the drift ;)). These files are predominantly YAML</code> but there’s support for JSON</code> as well. Anyway, to create these manifest files for a production level project is quite a bit of manual labour. The API spec of every resource in Kubernetes is quite daunting</a> and overwhelming. There are tools like Helm which abstract away the complexity of these YAML with it’s own templating system. There are quite a lot of these charts available for 3rd party apps here</a>. The idea is you populate the Chart with just your own config “values” and you’ve a deployment ready in no time. Admittedly this works quite well for something you want to take out for a quick spin but personally I am not quite a fan of hiding away the complexity with a magic</em> layer. Also, the problem with Helm was the “Chart” (and templates) still had to be written by someone if you have a bespoke</a> application. Helm is more geared towards common off the shelf apps like DBs, Key Value stores, web proxies etc.</p> I found out kustomize</a> few months back and quite happy with it’s approach towards managing manifests. The basic idea behind kustomize</code> is that you create a base</em> and any kind of “customisations” must come as overlays</em>. This is such a powerful technique over wrangling templates. kustomize</code> A common approach is to name these overlays</em> based on the environment. For example, dev</code> deployment can have replicas: 1</code> for a pod, but prod</code> can apply a “patch” to update with repliacs: 3</code>. This way of separating two environments helps a lot when you follow GitOps</a> approach of deployment. All fine and good, until I realised I spent way too much time on copy-pasting the bases for different projects and manually editing these files for the new project config.</p> Then I did what any other programmer would do, spend some more time to automate :P And that is how kubekutr</code> was born. (Quite an anticlimax I know!)</p> kubekutr</code> is a really simple tool to bootstrap a Kustomize base</code>. kubekutr</code> reads a config file, templates out different resources and produces them as YAML files. Now, I know a lot of you reading this would be going Another damn templating solution</em> in your mind and while that reaction is warranted, given that we have 200+ tools</a> in the community (everyone trying to solve similar problems in their own ways), I legit</em> could not find a simple enough tool which would let the boring part of scaffolding a base out of my way and let me focus on what’s more important: the actual deployment. Hence I just decided to roll out my own solution which is the best one according to IKEA effect</a> (just kidding</em>).</p> Workflow#</a></h3> So, let’s say you need to create a Nginx deployment, the kubekutr</code> config.yml</code> would look something like this:</p> deployments:</span></span> - name: nginx</span></span> replicas: 1</span></span> labels:</span></span> - name: 'service: nginx'</span></span> containers:</span></span> - name: nginx</span></span> image: 'nginx:latest'</span></span> portInt: 80</span></span> portName: nginx-port</span></span> services:</span></span> - name: nginx</span></span> type: ClusterIP</span></span> port: 80</span></span> targetPort: 80</span></span> labels:</span></span> - name: 'service: nginx'</span></span> selectors:</span></span> - name: 'service: nginx'</span></span></code></pre> To create the base</code>:</p> kubekutr -c config.yml scaffold -o nginx-deployment</code></p> nginx-deployment</code> folder is initialised and you can view deployments/nginx.yml</code> and service/nginx.yml</code> which kubekutr</code> created.</p> $ tree nginx-deployment</span></span> </span> \|-- base</span></span> \| \|-- deployments</span></span> \| \| `-- nginx.yml</span></span> \| \|-- ingresses</span></span> \| \|-- services</span></span> \| \| `-- nginx.yml</span></span> \| `-- statefulsets</span></span></code></pre>$ cat base/deployments/nginx.yml</span></span> </span> apiVersion: apps/v1</span></span> kind: Deployment</span></span> metadata:</span></span> name: nginx</span></span> labels:</span></span> service: nginx</span></span> spec:</span></span> replicas: 1</span></span> selector:</span></span> matchLabels:</span></span> service: nginx</span></span> template:</span></span> metadata:</span></span> labels:</span></span> service: nginx</span></span> spec:</span></span> containers:</span></span> - name: nginx</span></span> image: nginx:latest</span></span> ports:</span></span> - containerPort: 80</span></span> name: nginx-port</span></span></code></pre>$ cat base/services/nginx.yml</span></span> </span> apiVersion: v1</span></span> kind: Service</span></span> metadata:</span></span> name: nginx</span></span> labels:</span></span> service: nginx</span></span> spec:</span></span> ports:</span></span> - port: 80</span></span> targetPort: 80</span></span> protocol: TCP</span></span> type: ClusterIP</span></span> selector:</span></span> service: nginx</span></span></code></pre> You can now use the generated folder as a Kustomize base</code>.</p> Non Goals#</a></h3> kubekutr</code> isn’t meant to replace the existing tools, it’s just a real simple cookie cutter approach to kustomize</code> bases and that’s pretty much it. kustomize</code> is native to Kubernetes and exposes the full API spec to end users. I feel that is much more better approach than templating solutions, the users must be exposed to the standard conventions rather than a random tool’s own config fields. The benefits are the same conventions can then be used across a wide variety of tools (like kubekutr</code>) and users are in better control of the underlying resources. Adding a layer of magic also makes it harder to debug when shit goes down. Hence kubekutr</code> chose kustomize</code> to do all the heavy lifting of managing manifests.</p> There’s a lot of scope of improvements, but I wanted to just Ship It! and get some initial feedback. Let me know your thoughts on this :)</p> Fin!</p> A quick primer on dig 2019-11-11T10:57:55+05:30 Dig is a DNS lookup utility developed by BIND</a> which helps a lot while troubleshooting DNS issues (which are more common than you probably think #hugops). I use dig</code> fairly often and thought to write an introductory guide on how you can use dig</code> with some practical examples that’ll help you dig</code> through DNS issues faster (sorry for the lame pun, couldn’t resist.)</p> Basics#</a></h2> The most basic and common usage for dig</code> is to query the authorative servers for a particular domain and retrieve the IP. If it’s an IPv4 then you should be looking at A</code> record, while if it’s IPv6 then AAAA</code> record is your friend. Let’s see the DNS records for the site you’re currently on:</p> ➜</span> ~</span> dig</span> mrkaran.dev</span></span> </span> ;</span> <<</span>>></span> DiG</span> 9.10.6</span> <<</span>>></span> mrkaran.dev</span></span> ;; global options: +cmd</span></span> ;; Got answer:</span></span> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23292</span></span> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</span></span> </span> ;; OPT PSEUDOSECTION:</span></span> ; EDNS: version: 0, flags:; udp: 1220</span></span> ;; QUESTION SECTION:</span></span> ;mrkaran.dev. IN A</span></span> </span> ;; ANSWER SECTION:</span></span> mrkaran.dev. 60 IN A 206.189.89.118</span></span> </span> ;; Query time: 6 msec</span></span> ;; SERVER: 127.0.0.1#53(127.0.0.1)</span></span> ;; WHEN: Tue Oct 29 23:13:31 IST 2019</span></span> ;; MSG SIZE rcvd: 67</span></span> </span></code></pre> This is the most basic example for dig</code>. Let’s explore some of the additional options.</p> Keep it short#</a></h3> dig +short</code> keeps the information to bare minimum and only displays the ANSWER</code>.</p> dig</span> +short</span> mrkaran.dev</span></span> 206.189.89.118</span></span></code></pre>Nameserver details#</a></h3> If you want to find the Nameserver</code> for your DNS records, you can use the query type ns</code>.</p> $</span> dig</span> mrkaran.dev</span> ns</span> +short</span></span> alec.ns.cloudflare.com.</span></span> cruz.ns.cloudflare.com.</span></span></code></pre> ns</code> is one of the many query types you can use to indicate which type of DNS record you want to fetch. Default is A</code> record which returns the IPv4 address of the domain (unless it’s a root domain, in which case the default query type is NS</code>). Some other examples of query types are mx</code>, AAAA</code>, TXT</code> etc.</p> Fun Fact: ANY</code> query type has become obsolete</a> as per the new RFC8482</a> and DNS operators can choose to not respond to this query. The reason for this is that the payload response size for an ANY</code> query is quite huge (since it has to return all type of DNS records) and this could affect the performance of authoritative servers in case of a DNS amplification</a> attack.</p> Using different DNS server#</a></h3> Let’s say you want to switch to a different resolver, you can use @</code> followed by the address of your DNS server.</p> $</span> dig</span> mrkaran.dev</span> @9.9.9.9</span></span></code></pre>Reverse DNS Lookup#</a></h3> This one’s actually pretty cool. dig -x</code> lets you query the IP and retrieve the hostname details for that IP.</p> dig</span> -</span>x</span> 206.189.89.118</span></span></code></pre>Multiple queries#</a></h3> You can input a list of domain names and pass the file with the arg -f</code> to dig.</p> $</span> cat</span> digfile</span></span> mrkaran.dev</span></span> joinmastodon.org</span></span> zoho.com</span></span></code></pre> To list down all MX records for the domains in a file, you can use something like:</p> $</span> dig</span> -</span>f</span> digfile</span> +noall</span> mx</span> +answer</span></span> mrkaran.dev.</span> 242</span> IN</span> MX</span> 10</span> mx.zoho.in.</span></span> mrkaran.dev.</span> 242</span> IN</span> MX</span> 20</span> mx2.zoho.in.</span></span> mrkaran.dev.</span> 242</span> IN</span> MX</span> 50</span> mx3.zoho.in.</span></span> joinmastodon.org.</span> 21599</span> IN</span> MX</span> 10</span> in1-smtp.messagingengine.com.</span></span> joinmastodon.org.</span> 21599</span> IN</span> MX</span> 20</span> in2-smtp.messagingengine.com.</span></span> zoho.com.</span> 299</span> IN</span> MX</span> 10</span> smtpin.zoho.com.</span></span> zoho.com.</span> 299</span> IN</span> MX</span> 20</span> smtpin2.zoho.com.</span></span> zoho.com.</span> 299</span> IN</span> MX</span> 50</span> smtpin3.zoho.com.</span></span></code></pre>Search List#</a></h3> I learnt this recently while debugging a DNS issue in one of the Kubernetes pods. Dig doesn’t use search paths by default, so if you have a service say redis</code> inside a namespace dig won’t fetch any result:</p> $</span> dig</span> redis</span> +short</span></span> #</span> empty output, indicates no record found</span></span></code></pre> This is because a service name in Kubernetes is of the form service.namespace.svc.cluster.local</code>. So, we should actually be querying for redis.myns.svc.cluster.local</code> and we’ll get our result. But isn’t that too long and painful (sorry for the pun) to type?</p> So, there’s another option +search</code> which can be used to find all domains matching the search path defined in /etc/resolv.conf</code> namesever configurations.</p> $</span> cat</span> /etc/resolv.conf</span></span> nameserver</span> 10.100.0.10</span></span> search</span> myns.svc.cluster.local</span> svc.cluster.local</span> cluster.local</span></span></code></pre> We can now query for redis</code> with this search list:</p> dig</span> redis</span> +search</span> +short</span></span> 10.100.32.73</span></span></code></pre>DNSSec Validation#</a></h3> dig</code> even lets you validate the DNS records you received using DNSSEC</code> validation.</p> $</span> dig</span> mrkaran.dev</span> +dnssec</span></span> ;</span> <<</span>>></span> DiG</span> 9.10.6</span> <<</span>>></span> mrkaran.dev</span> +dnssec</span></span> ;; global options: +cmd</span></span> ;; Got answer:</span></span> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36275</span></span> ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1</span></span> </span> ;; OPT PSEUDOSECTION:</span></span> ; EDNS: version: 0, flags: do; udp: 1452</span></span> ;; QUESTION SECTION:</span></span> ;mrkaran.dev. IN A</span></span> </span> ;; ANSWER SECTION:</span></span> mrkaran.dev. 20 IN A 178.128.17.49</span></span> mrkaran.dev. 20 IN RRSIG A 13 2 20 20191112173050 20191110153050 34505 mrkaran.dev. Tl3zD6EqfVRvZi79ahePQcAXnbSUY9ZEYx/KwXnDUyonlrCKuBHzIYYC MJoVns410+sOwbIrcAdLgx+eiMYqRQ==</span></span> </span> ;; Query time: 65 msec</span></span> ;; SERVER: 1.1.1.1#53(1.1.1.1)</span></span> ;; WHEN: Mon Nov 11 22:01:01 IST 2019</span></span> ;; MSG SIZE rcvd: 163</span></span></code></pre> The important bit to note here is the ad</code> flag set which represents Authenticated Data. The records will only be returned if the validation succeeds (unless you also specify +cd</code> which indicates Checking Disabled flag.)</p> On a server which doesn’t have DNSSEC enabled, you can see no records are returned with the +dnssec</code> flag.</p> $</span> dig</span> dnssec-failed.org</span> +dnssec</span></span> ;</span> <<</span>>></span> DiG</span> 9.10.6</span> <<</span>>></span> dnssec-failed.org</span> +dnssec</span></span> ;; global options: +cmd</span></span> ;; Got answer:</span></span> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19886</span></span> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0</span></span> </span> ;; QUESTION SECTION:</span></span> ;dnssec-failed.org. IN A</span></span> </span> ;; Query time: 335 msec</span></span> ;; SERVER: 1.1.1.1#53(1.1.1.1)</span></span> ;; WHEN: Mon Nov 11 22:03:50 IST 2019</span></span> ;; MSG SIZE rcvd: 35</span></span></code></pre> That pretty much broadly covers some practical examples with dig</code>. I will soon write a detailed post on how DNSSEC</code> validation works and why it needs to be mainstream.</p> Fin!</p> GitOps approach to Continuous Delivery for Kubernetes 2019-11-11T10:57:55+05:30 In this post, I’d like to share my experience and learnings about configuring a deployment pipeline for Kubernetes. I’ll be using Gitlab CI/CD and AWS EKS to demonstrate the concept, but the core idea remains the same: all changes must come declaratively from a single source of truth</em>. GitOps</code> is a relatively newer term in the town but goes back to the fundamentals of Infra as Code</em>.</p> GitOps fundamentally is an operating model to perform tasks on Kubernetes related to deployments, configuration, secrets and monitoring workloads. All kind of changes must be performed via a single place, which happens to be a git</code> repo. Benefits of that are what basically benefits of version controlling the code is. So why treat infra as any different? git</code> happens to be the single source of truth for your infra, rollbacks are easy as reverting to last known good configuration and every change can be observed/verified.</p> Goals#</a></h2> A lot of tutorials/blog posts hitherto cover a very basic scenario where they do kubectl apply</code> and voila the deployment’s live. However we all know things are very different (to say the least) in production, so this post will cover all aspects of deployment:</p> Creating Manifests</li> Environment Promotion</li> Handling config and secrets</li> Authorization of CI/CD in the cluster</li> </ul> Basics#</a></h3> A GitOps workflow looks like:</p> Push vs Pull#</a></h3> There are 2 approaches to how you can handle deployments to a cluster. In a Pull based approach, the cluster runs a synch controller program which continmioously syncs the state of cluster with a Git repo. Any changes you make to the Git repo will be synced automatically in cluster. The idea is that there should be no drift in the desired state via Git repo and the actual state of cluster. Flux, Argo are good tools if you want a Pull based pipeline. The merits of Pull based pipeline is it’s more secure, since the deployment is actually happening inside cluster and no external sytstem needs to communicate to your production infra. The demerits are sometimes you’ve to wait for the changes to be synced (every controller runs these sync process in a loop with a sleep which can be configured). Also using any kind of preprocessing tools like Kustomize becomes difficult, since Flux just syncs the state and applies those changes. Handling of secrets is yet another concern, you need to look . And finally GitOps is a relatively newer tech in market so GitOps tooling is still nascent and like with any other relatively (non battle tested) software you’re gonna find bugs.</p> Push Approach however is a traditional CD approach, where the CD server talks to the cluster and applies changes through commands. In context of normal EC2 deployments, those commands could be SSH into server, running ansible playbook etc. In context of K8s however kubectl</code> does the magic for us. The CD server needs to talk to the K8s API server and run kubectl commands to change the cluster state. The merits of this approach are you can run all sorts of commands inside deployment pipeline and make it fully customisable. Handling of secrets also can be handled natively (like Gitlab env variables) or encrypted in git</code>. The demerits is that your production cluster is now exposed to your CD server.</p> Overall, if you have an airgapped CD server with no inbound ports open, access controll the user auth to CD, I found the Push approach to be more preferable. YMMV.</p> Writing the pipeline#</a></h2> I’ve created a docker image</a> eks-gitops</code> which I’ll be using throughout the pipeline. This container image contains popular tools like kustomize</code>, kubeval</code> etc and scripts to configure access to cluster using kubectl</code> using aws-iam-authenticator</code>. I’ve written more about how RBAC works inside EKS here</a>.</p> Excerpt from .gitla-ci.yml</code></em>:</p> #</span> Use this as base image for all jobs unless overriden</span></span> d</span>efault</span>:</span></span> i</span>mage</span>:</span></span> n</span>ame</span>:</span> m</span>rkaran/eks-gitops:latest</span></span> e</span>ntrypoint</span>:</span> [</span>"</span>/bin/sh</span>"</span>,</span> "</span>-c</span>"</span>]</span></span> #</span>## Pipeline</span></span> s</span>tages</span>:</span></span> -</span> v</span>alidate</span></span> -</span> d</span>eploy</span></span></code></pre>Prepare the manifests#</a></h3> I use Kustomize</code> to prepare the manifests. Advantage of Kustomize</code> is writing template free YAMLs but still be able to customise them heavily using overlays. For different environments, you can apply certain changes like increasing resource requests, adding more storage, while keeping the base same.</p> Here’s a folder structure (from a real GitOps repo) I follow for manifests:</p> .</span></span> ├──</span> base</span></span> │ </span> ├──</span> deployments</span></span> │ </span> │ </span> ├──</span> app.yml</span></span> │ </span> │ </span> ├──</span> celery.yml</span></span> │ </span> │ </span> └──</span> nginx.yml</span></span> │ </span> ├──</span> ingresses</span></span> │ </span> │ </span> └──</span> web.yml</span></span> │ </span> ├──</span> kustomization.yaml</span></span> │ </span> ├──</span> services</span></span> │ </span> │ </span> ├──</span> app.yml</span></span> │ </span> │ </span> ├──</span> nginx.yml</span></span> │ </span> │ </span> └──</span> redis-headless.yml</span></span> │ </span> ├──</span> statefulsets</span></span> │ </span> │ </span> └──</span> redis.yml</span></span> │ </span> └──</span> volumes</span></span> │ </span> └──</span> redis.yml</span></span> ├──</span> kubekutter.yml</span></span> ├──</span> Makefile</span></span> ├──</span> overlays</span></span> │ </span> ├──</span> dev</span></span> │ </span> └──</span> prod</span></span> │ </span> ├──</span> configs</span></span> │ </span> │ </span> ├──</span> app-config.env</span></span> │ </span> │ </span> └──</span> app-nginx.conf</span></span> │ </span> ├──</span> kustomization.env.yml</span></span> │ </span> ├──</span> namespace.yml</span></span> │ </span> ├──</span> patches</span></span> │ </span> │ </span> ├──</span> configure-configmap-volume.yml</span></span> │ </span> │ </span> ├──</span> modify-alb.yml</span></span> │ </span> │ </span> └──</span> resource-limits.yml</span></span> │ </span> └──</span> rbac.yml</span></span> └──</span> README.md</span></span></code></pre> Shameless Plug</strong>: I created kubekutr</code> which makes managing of these manifests using kustomize</code> a breeze.</p> Some things to note here:</p> Inside base/</code>, I keep all the base</code> resources required for the app to run. The resources can be Service</code>, Deployment</code>, Ingress</code> etc.</li> Inside overlays</code> there are multiple folders for different environment. This is very crucial as we want to separate the production config with a UAT config. Last-mile configuration to the base becomes very easy with this folder structure, since you only now need to build the manifests by targetting a specific folder in CI.</li> Inside overlays/{env}/patches</code> are all the “patches” you want to do to the base resource. Think like replica count, ALB subnets (since different env can be in different VPCs), increasing resource limits and stuff like that.</li> rbac.yml</code> abd namespace.yml</code> is the only missing piece because it’s like a chicken and egg problem. I cannot deploy directly (at first go) from a CI/CD if I don’t have a namespace created since the CD server is configured only has limited namespaced restricted access. So unless I create a namespace, add proper RBAC for the CD server I cannot do any deployments from CD. Note however this is only a first time step, which I guess is okay.</li> </ul> Lint yo manifest#</a></h3> I’m using kubeval</code> to lint the manifests. The manifests have to prepared by kustomize</code>. CI_ENVIRONMENT_NAME</code> is set by Gitlab</code> when you specify an environment for a job. Don’t sweat about this part, I’ll describe it more as we proceed.</p> #</span> Validate the yaml using kubeval</span></span> .</span>lint</span>:</span></span> e</span>xtends</span>:</span> .</span>prepare-manifest</span></span> s</span>tage</span>:</span> v</span>alidate</span></span> s</span>cript</span>:</span></span> -</span> e</span>cho "Linting manifest for ${CI_ENVIRONMENT_NAME}"</span></span> -</span> k</span>ustomize build overlays/$CI_ENVIRONMENT_NAME --load_restrictor none \| kubeval</span></span></code></pre>Setup environment#</a></h3> You can define environment name as:</p> #</span> Create an environment to record all jobs for this env</span></span> .</span>prod</span>:</span> &</span>prod</span></span> e</span>nvironment</span>:</span></span> n</span>ame</span>:</span> p</span>rod</span></span> u</span>rl</span>:</span> h</span>ttps://prod.site</span></span> .</span>dev</span>:</span> &</span>dev</span></span> e</span>nvironment</span>:</span></span> n</span>ame</span>:</span> d</span>ev</span></span> u</span>rl</span>:</span> h</span>ttps://dev.site</span></span></code></pre> Any job which is to be executed in a particular environment can include this variable and CI_ENVIRONMENT_NAME</code> will be automatically set.</p> A cool feature of Gitlab is that you can restrict Variables scoped to the environment they are defined in.</p> Configure Secrets#</a></h3> All secrets are defined as Environment Variables in Gitlab CD pipeline. While running the job, the runner has access to these variables and with the help of secretGenerator</code> in kustomize</code>, the Secret</code> is created.</p> I use secretGenerator</code> because any time a K8s secret changes, kustomize appends with a new suffix, which makes the Replication Controller believe that they deployment has changed. So a new deployment is automatically triggered.</p> Authenticate to cluster#</a></h3> EKS uses aws-iam-authenticator and uses IAM access roles to allow the cluster to perform actions. Since this is a push based pipeline, you need to allow the access from your CD server to port 443.</p> Deploy changes#</a></h3> This is as simple as kubectl apply</code> which configures all the changes and diff between cluster and real world state.</p> Here’s a full gitops repo if you’re interested in checking it out:</p> Intro to RBAC in EKS 2019-11-01T18:10:55+05:30 EKS uses a custom authenticator tool called “aws-iam-authenticator”. The basic idea is to make the auth flow in EKS easier by using the tools you already use in AWS.</p> </p> To wrap your head around the flow, consider three separate entities:</p> A kubernetes resource (entire namespace, specific pods/configs etc)</li> An action (get, watch, list, create, delete etc)</li> An IAM role/user created on AWS</li> </ul> Your usecase might be to give an IAM role access to a Kubernetes namespace for example with certain restricted actions.</p> Since K8s is basically client-server communication with the API server, we must need to perform the following two things for every single API request which goes to the control plane:</p> Authentication</em></li> </ul> kubectl</code> talks to the API server using the token generated by aws-iam-authenticator</code>. The API server passes on this info to the AWS servers which validated if the originating call is coming from a valid IAM user or not. If not, the access is denied from K8s.</p> Authorization</em></li> </ul> Once the IAM user is validated, the IAM user needs to be mapped to a user to perform Authorization so K8s API server can know whether the action/resource requested for is to be allowed or not. This is where the aws-auth-cm.yml</code> comes into the picture. It is basically a map of all IAM users with internal K8s groups or users created. The roles are assosciated to these users/groups so once the mapping is done, K8s API server can know what to do with the API request.</p> How does EKS know my IAM?#</a></h3> If you configured your KUBECONFIG</code> correctly using aws eks update-kubeconfig</code> then you’ll find the below lines in your config file. This basically runs a aws cli</code> command to find your IAM user/role configured on your system to produce a token.</p> exec:</span></span> apiVersion: client.authentication.k8s.io/v1alpha1</span></span> args:</span></span> - token</span></span> - -i</span></span> - eks-zero-public</span></span> command: aws-iam-authenticator</span></span></code></pre>Show me the YAML already#</a></h3> To create a Role and RoleBinding</li> </ul> apiVersion: rbac.authorization.k8s.io/v1</span></span> kind: Role</span></span> metadata:</span></span> namespace: coolapp</span></span> name: fullaccess</span></span> rules:</span></span> - apiGroups: [""]</span></span> resources: [""]</span></span> verbs: [""]</span></span> - apiGroups: ["apps"]</span></span> resources: [""]</span></span> verbs: [""]</span></span> - apiGroups: ["batch"]</span></span> resources: [""]</span></span> verbs: [""]</span></span> - apiGroups: ["extensions"]</span></span> resources: [""]</span></span> verbs: [""]</span></span> - apiGroups: ["autoscaling"]</span></span> resources: [""]</span></span> verbs: [""]</span></span> ---</span></span> kind: RoleBinding</span></span> apiVersion: rbac.authorization.k8s.io/v1</span></span> metadata:</span></span> name: 10xdevs-fullaccess</span></span> namespace: coolapp</span></span> subjects:</span></span> - kind: Group</span></span> name: 10xdevs</span></span> namespace: coolapp</span></span> roleRef:</span></span> kind: Role</span></span> name: fullaccess</span></span> apiGroup: rbac.authorization.k8s.io</span></span></code></pre> So, for all the normal folks who don’t grok YAML as fast as 10x devops engineers, I am basically creating a Role fullaccess</code> with all permissions for a namespace coolapp</code> (innovative, ikr). Then I am binding</code> this Role to a Group called 10xdevs</code> so that the group is allocated the role which has permissions. We will use this group 10xdevs</code> to map our AWS user now.</p> To create the map of IAM Role/User ARN with the above Role</li> </ul> Edit your aws-auth-cm.yml</code> and add the below stuff:</p> apiVersion: v1</span></span> kind: ConfigMap</span></span> metadata:</span></span> name: aws-auth</span></span> namespace: kube-system</span></span> data:</span></span> mapRoles ... <skipping> ...</span></span> </span> mapUsers: \|</span></span> - userarn: arn:aws:iam::<account-id>:user/<user-name></span></span> username: 10xdevs-fullaccess # RoleBinding name created in previous step</span></span> groups:</span></span> - 10xdevs # Group name created in previous step</span></span></code></pre>Verify it yourself#</a></h3> kubectl auth can-i create pods --all-namespaces</code>: should fail</em></p> </li> kubectl auth can-i create pods -n coolapp</code>: should work</em></p> </li> </ul> Fin!</p> Kubernetes cluster on RPi 2019-09-22T08:10:55+05:30 So, I got hold of 2 Raspberry Pi4 (still limited stocks in India) recently and wanted to build a Kubernetes cluster. Don’t ask why cause that would be pointless. I’ve little experience with a managed Kubernetes workload (Amazon EKS, which btw deserves its post 😝) but never really played around any of the K8s internal stuff yet. In this post, I’ll show you how I got a lightweight Kubernetes distro: K3s up and running.</p> k3s</a> is a pretty great Kubernetes distro which passes the K8s Conformance tests. On ARM architectures, you’re pretty much resource-bounded and you want the resource footprint of your infra to be as minimal as possible. In k3s there are a few changes such as the persistent layer of K8s is replaced with SQLite instead of etcd, unusable (legacy/alpha) features of K8s are removed and cloud-provider plugins are not bundled (but can be installed separately). All of this together means just 40MB of a binary to run the cluster and ~250MB of memory usage on an idle cluster. Awesome, team Rancher</a> :)</p> </p> Automation is the key and even though I have just 2 nodes on RPi, I Ansible-ized</em> the setup which I am hoping would save time in future if I add more nodes to the setup.</p> Hardware#</a></h2> 1x RPi4 4GB</a> and 1x RPi4 2GB</a> variant</li> 2x Samsung EVO micro SD Card</a></li> 2x USBC Cables</a></li> 1x Amker Power Port</a> (don’t compromise on the power supply, give enough juice so RPi doesn’t throttle)</li> 1x TP-Link Network Switch</a> (my router has only 1 usable LAN port)</li> 2x CAT5 LAN cables</a> (keeping it basic, you can get fancy flat LAN cables if you wish to)</li> </ul> This is how the final setup looks like:</p> {{< tweet 1175659256764219392 >}}</p> Setting up RPi#</a></h3> I downloaded Raspbian Buster Lite</a> because it’s the easiest to setup. Next step is to flash the SD card and for that, I used Etcher</a>.</p> </p> To enable SSH access, you need to create an empty file ssh</code> in the root volume.</p> sudo</span> touch</span> /boot/ssh</span></span></code></pre> Once all sorted, we can use Ansible to set up the basic OS stuff, like changing the default password, enabling password-less SSH login, timezone & locale settings, changing hostname etc. I’ll be sharing relevant Ansible snippets, if interested you can check out the complete playbook at mr-karan/hydra repo</a>.</p> We need to enable container features on the RPi so that containerd</code> can run. Containers like Docker make use of cgroups</code> (Linux kernel feature) which allows them to put resource limits on container processes like CPU and Memory. To enable cgroups</code>, you need to edit /boot/cmdline.txt</code>.</p> -</span> n</span>ame</span>:</span> A</span>dd cgroup directives to boot command line config</span></span> l</span>ineinfile</span>:</span></span> p</span>ath</span>:</span> /</span>boot/cmdline.txt</span></span> r</span>egexp</span>:</span> '</span>((.)+?)(\scgroup_\w+=\w+)$</span>'</span></span> l</span>ine</span>:</span> '</span>\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory</span>'</span></span> b</span>ackrefs</span>:</span> yes</span></span></code></pre> Quick Tip: If you’re going to use RPi as a headless server (not connecting with any monitor) you can reduce the GPU Memory to lowest possible (16M)</p> -</span> n</span>ame</span>:</span> S</span>et GPU memory split to 16 MB</span></span> l</span>ineinfile</span>:</span></span> p</span>ath</span>:</span> /</span>boot/config.txt</span></span> l</span>ine</span>:</span> "</span>gpu_mem=16</span>"</span></span> c</span>reate</span>:</span> yes</span></span></code></pre>Deploy K3s cluster#</a></h3> Next, we’ll come to the actual stuff, where we’ll download K3s binary and run as a systemd service. There’s a handy shell script</a> to bootstrap the cluster provided by the Rancher team which setups the whole thing in one command. But if you wanna learn/play around I’d recommend you do things the hard way (it’s not all that hard tho. ((twss!)).</p> -</span> n</span>ame</span>:</span> D</span>ownload k3s binary armhf</span></span> g</span>et_url</span>:</span></span> u</span>rl</span>:</span> h</span>ttps://github.com/rancher/k3s/releases/download/{{ k3s_version }}/k3s-armhf</span></span> d</span>est</span>:</span> /</span>usr/local/bin/k3s</span></span> o</span>wner</span>:</span> r</span>oot</span></span> g</span>roup</span>:</span> r</span>oot</span></span> m</span>ode</span>:</span> 755</span></span> w</span>hen</span>:</span> (</span> ansible_facts.architecture is search("arm") )</span></span> a</span>nd</span></span> (</span> ansible_facts.userspace_bits == "32" )</span></span></code></pre> On the worker node, the process is similar, except you have to run the K3s with agent</code> compared to server</code> argument in the control plane. Things get a bit interesting here though. You need to give the cluster server URL along with it’s token in the command. A unique token is generated by the server at (/var/lib/rancher/k3s/server/node-token</code>) which is used to join the worker nodes.</p> I did a bit of google-fu and got to know about this neat little Ansible module set_fact</code></a> which lets you “store” a variable from one host and use it in a second host. Every Ansible host maintains a Python dict of “host facts”. In the second node, I access the cluster’s host fact dict</code>, fetch the variable and use it in its systemd service template. Neat, ain’t it? Ansible has so many modules, it is mind-boggling.</p> Reading and storing the variable as a “host” fact</em>:</p> #</span> on the cluster</span></span> -</span> n</span>ame</span>:</span> R</span>ead node-token from control node</span></span> s</span>lurp</span>:</span></span> s</span>rc</span>:</span> /</span>var/lib/rancher/k3s/server/node-token</span></span> r</span>egister</span>:</span> n</span>ode_token</span></span> </span> -</span> n</span>ame</span>:</span> S</span>tore control node-token</span></span> s</span>et_fact</span>:</span></span> k</span>3s_cluster_token</span>:</span> "</span>{{ node_token.content \| b64decode \| regex_replace('</span>\n</span>', '') }}</span>"</span></span></code></pre> Using the variable from the server host, in a template on the agent host</em>:</p> #</span> on the agent (vars.yml)</span></span> k</span>3s_server_address</span>:</span> "</span>{{ hostvars[groups['control'][0]].k3s_server_address }}</span>"</span></span> k</span>3s_cluster_token</span>:</span> "</span>{{ hostvars[groups['control'][0]].k3s_cluster_token }}</span>"</span></span> #</span> use the value in a template</span></span> ...</span></span> [</span>S</span>ervice</span>]</span></span> E</span>xecStart=/usr/local/bin/k3s agent --server {{ k3s_server_address }} --token {{ k3s_cluster_token }}</span></span> ...</span></span></code></pre> P.S. Shoutout to Ansible tho. It is one of my fav infra tooling available out there. It has some gotchas that you need to be aware of but by and large, the experience has been quite pleasant.</p> On the cluster, you should be able to see the nodes.</p> </p> </p> Team #SelfHost#</a></h3> I am planning to host Bitwarden, Gitea and a Nextcloud instance on this cluster. Also will be using this as a testbed to play around with K8s internals. Stay tuned as I explore more of this!</p> Cheers! :)</p> My Personal Networking Setup 2019-06-29T18:10:55+05:30 If I have nothing to hide then I have nothing to show either</p> </blockquote> When you visit a site from your browser, there are several entities which have access to this information in one request-response cycle. Starting from the router you connect to, to the DNS resolver you use to resolve the IP Address of the site, and finally, the ISP which has access to the raw network packets (unencrypted or encrypted). They are all insecure by default and most users on the internet do not care about their privacy. ISPs can very easily profile you by the queries you do and sell that data to 3rd party Ad companies. There are still a huge number of sites on the world wide web which don’t have a basic SSL encryption turned on. Even with the HTTPS turned on your ISPs can simply see the unencrypted DNS queries and figure out the site you’re visiting. Browser fingerprinting is another well-known technique where you can be easily profiled based on the different parameters exposed by your browser and generate patterns in your browsing habits.</p> If you’re a human you already care about your privacy. You won’t give out your Whatsapp chats to a random stranger on the streets to read, you won’t leave the door open while bathing, you won’t leave your credit card details out in the open, you will have a closed private discussion with your second half. Everyone does care about privacy in real life and everyone should care about the same online too.</p> “Some” social media companies exist solely on the data you provide. They are in the business of selling data and they don’t care a bit about your privacy. Every ad company out there wants an online persona about you, to feed you “personalized” recommendations. You may argue that they are “useful” but you cannot deny the fact that they are invading your privacy. You won’t like unwanted relationship advice from your neighbour when he hears you both fighting in the middle of the night. No one likes invaders, period.</p> Correlation of data points is easy to do in this age of ML/AI (Malai</em>). Re-iterating an example I read somewhere on the interwebz, say you search for a very generic term “cancer”, this alone doesn’t indicate that you have cancer but merely is suggestive of the fact that you are interested in knowing more about the disease. Now maybe a few days later you search for the phrase “buy a wig online”. These 2 search queries put together can suggest</em> that you’ve had chemotherapy done recently. This is a piece of information you might</em> not want to hide, but you do want to have an option</strong> right?</p> That’s the whole fight against companies which invade your privacy is. You have absolutely no choice. You’re devoid of any choice.</p> Google is your friend</p> </blockquote> You may not even realise but we’ve all been guilty of Googling</em> some random shit for fun, googling about anxiety or depression when feeling low, how to lose weight and all such seemingly embarrassing questions and we simply confront these things to Google thinking that no one is spying on us or being judgemental of us and we’re in a safe zone. What if this “friend” is the one who’s stabbing your back. No surprises there! You don’t even know but these search engines might have already shared all such queries with several 3rd party providers and business who are on the lookout for such data.</p> I am scared</p> </blockquote> I hope by now you’re convinced that you want some basic privacy online. If not, stop reading here because if this didn’t persuade you enough, nothing will and the rest of the blog is pretty useless for you anyway, so thanks for stopping by!</p> For the rest of us folks, there are a few practical things we can immediately start which. These are the things which have very less involvement of effort but are a good kickstart.</p> Use DNS over HTTPS (DoH) based resolver (personal choice: 1.1.1.1)</li> Use a browser which doesn’t ship with its own god damn resolver</a> (personal choice: Firefox)</li> Use an Ad-Blocker (personal choice: uBlock Origin)</li> </ul> Only with the above 3 steps you’ve achieved:</p> ISPs cannot see your DNS queries in plain text</li> Websites cannot serve you random js code in the name of ads</li> A web browser which is more focussed towards privacy in general</li> </ul> If you’re someone who wants to spend a bit more time (trust me it won’t take even one hour) and a bit of money (one movie ticket a month) you should host your personal VPN. VPNs are awesome because you mitigate a lot of privacy invaders and you are on an anonymous browsing mode generally</em>. Theoretically, we have shifted the trust from the ISP to the cloud provider where VPN is hosted, yes. But as long as you’re not doing anything illegal in the eyes of the state, you should be fine.</p> Protip</strong>: Don’t use VPN to do something stupid. You are always trackable. The whole exercise is to not allow bad actors to access your personally identifiable information online and have a non-intrusive browsing experience. If you think by using VPNs you can get away with doing something which is illegal by your state then you’re wrong.</p> Tools I use#</a></h2> VPN: Wireguard</a></li> Adblocking: Pi-hole</a></li> DNS Resolver: Unbound</a></li> </ul> The setup is pretty straightforward. I use Wireguard client on my MBP and an Android device to connect to the Wireguard VPN server hosted on a $5 DO</a> droplet in Bengaluru, India. For ad-blocking, I use Pi-hole, which does DNS based ad-blocking. And finally to resolve the DNS queries I use vanilla unbound without any forwarders.</p> Read the following sections to know more on each of the above pieces.</p> Setting up wireguard VPN#</a></h2> Wireguard gets all the love from me (thanks to @sarat</a> for telling me about this). It is probably the best option out there if you’re looking to self-host a VPN. Wireguard gets a lot of things right, first being the ease of setup. Literally takes a few commands and you’re all set. Wireguard is very lightweight and consumes minimal resources. Wireguard has better encryption and is a lot faster</a> than IPvsec or open VPN. Cloudflare’s new app Warp is also based on Wireguard. Oh, and did I mention that Wireguard seamlessly transitions when you switch networks (which happens all the time when you’re on shitty 4G network/public WiFis).</p> Wireguard basically has 2 parts. One is the server and the other is the client. Wireguard sits in its own separate network namespace and uses this namespace as an init namespace from where the traffic is received or sent. This namespace is now responsible for flowing your traffic to the other actual interface your network card on the device (which is probably wlan or eth).</p> Installing Wireguard#</a></h3> $</span> sudo</span> add-apt-repository</span> ppa:wireguard/wireguard</span></span> $</span> sudo</span> apt-get</span> update</span></span> $</span> sudo</span> apt-get</span> install</span> wireguard-dkms</span> wireguard-tools</span> linux-headers-</span>$(</span>uname</span> -</span>r</span>)</span></span></code></pre>Generating Public/Private key pair#</a></h3> $</span> umask</span> 077</span></span> $</span> wg</span> genkey</span> \|</span> tee</span> server_private_key</span> \|</span> wg</span> pubkey</span> ></span> server_public_key</span></span> $</span> ls</span></span> server_private_key</span> server_public_key</span></span></code></pre>Configuring Wireguard (server)#</a></h3> $</span> touch</span> /etc/wireguard/wg0.cong</span></span> $</span> vim</span> /etc/wireguard/wg0.conf</span></span> #</span> Add the following lines and modify the values</span></span> [</span>Interface</span>]</span> #</span> Configuration settings for a separate network interface</span></span> Address</span> =</span> 10.200.200.1/24</span> #</span> You can choose any private subnet</span></span> SaveConfig</span> =</span> false</span> #</span> Wireguard can configure additional peers automatically without reloading wireguard, for some reason this didn't work well for me</span></span> PrivateKey</span> =</span> <</span>redacte</span>d</span>></span> #</span> Output of the `server_private_key` generated in the above setup</span></span> ListenPort</span> =</span> 51820</span> #</span> Default port</span></span> </span> #</span> Add the peers (clients which connect to the wireguard server)</span></span> [</span>Peer</span>]</span></span> #</span> MBP</span></span> PublicKey</span> =</span> <</span>laptop_public_ke</span>y</span>></span></span> AllowedIPs</span> =</span> 10.200.200.2/32</span></span> [</span>Peer</span>]</span></span> #</span> Android</span></span> PublicKey</span> =</span> <</span>phone_public_ke</span>y</span>></span></span> AllowedIPs</span> =</span> 10.200.200.3/32</span></span></code></pre>Configuring Wireguard (client)#</a></h3> Repeat the step of generating a Public/Private key pair. You can also take a look at Subspace</a> which is a nice GUI tool which helps you create additional profiles for your devices where generating key/pair is not convenient like mobile phones. I didn’t get the time to set it up personally, so I generated public/private keys from my laptop itself and then configured it manually using an Android app Viscerion</a> which is a wireguard client app.</p> </li> For the client, your config file should look like:</p> </li> </ul> [Interface]</span></span> PrivateKey = <client_private_key></span></span> Address = 10.200.200.2/32</span></span> </span> [Peer]</span></span> PublicKey = <server_public_key></span></span> AllowedIPs = 0.0.0.0/0</span></span> Endpoint = <public_ip_droplet>:51820</span></span> PersistentKeepalive = 25</span></span></code></pre>Recap#</a></h3> If you got overwhelmed at this point, let’s recap what just happened.</p> We have 2 config files, one for the server and one for the client which happens to be my laptop. On each of the devices, generate a public/private key pair.</p> On the server side, while configuring peer, give the public key of the client. On the client side, while configuring peer,give the public key of the server. This is similar to how ssh</code> works. Wireguard uses Curve25519</code> crypto technique to generate a public/private key pair, which honestly looks</em> so better than lengthy ECDSA/RSA ones :P</p> Now let us start the wireguard service. wireguard provides a nice wrapper wg-quick</code> which does the following things when you start:</p> sudo wg-quick up wg0</span></span></code></pre>[#] ip link add wg0 type wireguard</span></span> [#] wg setconf wg0 /dev/fd/63</span></span> [#] ip -4 address add 10.200.200.1/24 dev wg0</span></span> [#] ip link set mtu 1420 up dev wg0</span></span></code></pre> You can verify a new network interface now by:</p> $ ip addr \| grep wg0</span></span> 5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000</span></span> inet 10.200.200.1/24 scope global wg0</span></span></code></pre> You can also view the wireguard connection status by:</p> $ sudo wg show</span></span> interface: wg0</span></span> public key: <REDACTED></span></span> private key: (hidden)</span></span> listening port: 51820</span></span> </span> peer: <REDACTED></span></span> allowed ips: 10.200.200.2/32</span></span></code></pre>Almost There, But Not Quite#</a></h3> Turn on your wireguard client and you will notice a strange thing. At this point, you’re not able to browse the internet but you are able to connect to the wireguard server and even wireguard is acknowledging that (notice the last two lines in the following snippet):</p> $ sudo wg show</span></span> interface: wg0</span></span> public key: <REDACTED></span></span> private key: (hidden)</span></span> listening port: 51820</span></span> </span> peer: <client_public_key></span></span> endpoint: <client_public_ip>:64882</span></span> allowed ips: 10.200.200.2/32</span></span> latest handshake: 8 seconds ago</span></span> transfer: 754.60 KiB received, 5.59 MiB sent</span></span></code></pre> Wireguard status shows that our client can reach the wireguard server. But still, we’re unable to browse the internet on our client.</p> To debug this further, let’s use tcpdump</code> and monitor the packets coming in wg0</code> interface. Since tcpdump’s output can be overwhelming and contains a lot of noise, let’s filter to monitor only the ICMP packets. So we’ll be using ping</code> from a client which is the easiest way to send ICMP packets from point A to point B.</p> #</span> on wireguard client</span></span> ➜</span> ~</span> ping</span> $(</span>curl</span> icanhazip.com</span>)</span></span> %</span> Total</span> %</span> Received</span> %</span> Xferd</span> Average</span> Speed</span> Time</span> Time</span> Time</span> Current</span></span> Dload</span> Upload</span> Total</span> Spent</span> Left</span> Speed</span></span> 100</span> 14</span> 100</span> 14</span> 0</span> 0</span> 16</span> 0</span> -</span>-:--:--</span> -</span>-:--:--</span> -</span>-:--:--</span> 16</span></span> PING</span> <</span>server_ip_redacte</span>d</span>></span> (<server_ip_redacted></span>): 56 data bytes</span></span> 64</span> bytes</span> from</span> <</span>server_ip_redacte</span>d</span>></span>:</span> icmp_seq=</span>0</span> ttl=</span>64</span> time=</span>112.126</span> ms</span></span> 64</span> bytes</span> from</span> <</span>server_ip_redacte</span>d</span>></span>:</span> icmp_seq=</span>1</span> ttl=</span>64</span> time=</span>142.980</span> ms</span></span> ^C</span></span> ---</span> <</span>server_ip_redacte</span>d</span>></span> ping</span> statistics</span> -</span>--</span></span> 2</span> packets</span> transmitted,</span> 2</span> packets</span> received,</span> 0.0%</span> packet</span> loss</span></span></code></pre>#</span> on wireguard server</span></span> $</span> sudo</span> tcpdump</span> -</span>nni</span> wg0</span> -</span>Q</span> in</span> icmp</span></span> tcpdump:</span> verbose</span> output</span> suppressed,</span> use</span> -</span>v</span> or</span> -</span>vv</span> for</span> full</span> protocol</span> decode</span></span> listening</span> on</span> wg0,</span> link-type</span> RAW</span> (Raw</span> IP</span>), capture size 262144 bytes</span></span> 14:26:31.991498</span> IP</span> 10.200.200.2</span> ></span> <</span>server_ip_redacte</span>d</span>></span>:</span> ICMP</span> echo</span> request,</span> id</span> 52630,</span> seq</span> 0,</span> length</span> 64</span></span> 14:26:33.027288</span> IP</span> 10.200.200.2</span> ></span> <</span>server_ip_redacte</span>d</span>></span>:</span> ICMP</span> echo</span> request,</span> id</span> 52630,</span> seq</span> 1,</span> length</span> 64</span></span></code></pre> Okay, this is getting interesting. wg0 is definitely receiving packets from our client. Arghhh. Now</em> does it strike to you? Our actual network routing is through eth0</code> (on DO usually) and wg0</code> is just some interface created by Wireguard. They have no “connection” (no pun intended) between them. Let us confirm if this actually is the problem by detecting ICMP traffic on this interface(eth0</code>) using the same command:</p> #</span> on wireguard server</span></span> $</span> sudo</span> tcpdump</span> -</span>nni</span> eth0</span> -</span>Q</span> in</span> icmp</span></span> tcpdump:</span> verbose</span> output</span> suppressed,</span> use</span> -</span>v</span> or</span> -</span>vv</span> for</span> full</span> protocol</span> decode</span></span> listening</span> on</span> eth0,</span> link-type</span> RAW</span> (Raw</span> IP</span>), capture size 262144 bytes</span></span> .</span>.</span>.</span> #</span> nothing happens even if we are sending PING from client</span></span></code></pre> Wow, see? No ICMP packets received on eth0</code>. So that indeed is the problem and we have zero’ed it down using tcpdump</code>. (Note to self: Learn more Linux debugging utils, these things are a godsend!)</p> In order to fix this, we need to do 2 things:</p> IP Forwarding</li> </ul> $ vim /etc/sysctl.conf</span></span> # check for this line and replace the value from 0 to 1</span></span> net.ipv4.ip_forward = 1</span></span> $ sudo sysctl -p</span></span></code></pre> IP Tables Rules</li> </ul> We need to set up NAT between eth0</code>(could be different for you) and wg0</code>. This can be done using iptables</code> and wireguard actually has a nice mechanism to run custom commands using PostUp</code>/PostDown</code> signals.</p> #</span> on wireguard server</span></span> $</span> vim</span> /etc/wireguard/wg0.conf</span></span> #</span> add these lines in [Interface] section</span></span> PostUp</span> =</span> iptables</span> -</span>A</span> FORWARD</span> -</span>i</span> wg0</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>A</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span>;</span> ip6tables</span> -</span>A</span> FORWARD</span> -</span>i</span> wg0</span> -</span>j</span> ACCEPT</span>;</span> ip6tables</span> -</span>t</span> nat</span> -</span>A</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span> #</span> Configure iptables to setup a NAT on eth0 and forward the packets (ipv4 and ipv6) on interface wg0 to eth0</span></span> PostDown</span> =</span> iptables</span> -</span>D</span> FORWARD</span> -</span>i</span> wg0</span> -</span>j</span> ACCEPT</span>;</span> iptables</span> -</span>t</span> nat</span> -</span>D</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span>;</span> ip6tables</span> -</span>D</span> FORWARD</span> -</span>i</span> wg0</span> -</span>j</span> ACCEPT</span>;</span> ip6tables</span> -</span>t</span> nat</span> -</span>D</span> POSTROUTING</span> -</span>o</span> eth0</span> -</span>j</span> MASQUERADE</span> #</span> Delete the rule since when wireguard is down, wg0 doesn't exist</span></span></code></pre> Now everything’s set up and we are browsing the internet privately using a VPN. Ensure</a> that your public IP is of the VPN server when you’re browsing and it’s not leaking.</p> Setting up Pi-hole#</a></h3> Installing Pi-hole is as simple as</p> curl</span> -</span>sSL</span> https://install.pi-hole.net</span> \|</span> bash</span></span></code></pre> You can read more about installation</a> in the official docs.</p> A nice GUI is always a plus, so make sure you enable that option while installing Pi-hole.</p> </p> Now, we need to configure our Wireguard client to use Pi-hole as a nameserver for DNS resolution.</p> #</span> on the client add the following line</span></span> sudo</span> vim</span> /etc/wireguard/wg0.conf</span></span> [</span>Interface</span>]</span></span> .</span>.</span>.</span></span> DNS</span> =</span> <</span>vpn_server_public_i</span>p</span>></span></span> .</span>.</span>.</span></span></code></pre> Pi-hole runs on port 53 and accepts DNS queries over UDP. Any query is first checked by Pi-hole in the blacklist. If it’s present in the blacklist, it’s immediately dropped. If not, Pi-hole will forward our DNS query to one of the forwarder configured (1.1.1.1 for eg or our custom server, explained in the next step).</p> Setting up Unbound DNS#</a></h3> I don’t mind trusting Cloudflare. But I simply don’t have to. :) And once you make peace with the fact that you don’t need 3rd party companies controlling your networking stack, you’ll sleep better.</p> I have setup Unbound without any forwarders. Unbound is a recursive resolver which supports DNSSec and caching mainly. Unbound first checks if the query exists in cache and if it does, it directly returns the “answer”. Otherwise it talks to the root nameserver and then the whole DNS dance happens. Since our DNS query is now split into multiple parts, where each nameserver is only being queried for a part of the FQDN (also known as QNAME minimialistion), it becomes a lot harder for anyone to intercept or reconstruct your DNS queries.</p> You can install unbound using</p> $ sudo apt-get install unbound</span></span></code></pre> To start using Unbound, we need a file root.hints</code> which contains information about root nameservers. You can cron this to fetch a new copy every once in 3-4 months, it hardly changes.</p> $</span> wget</span> -</span>O</span> root.hints</span> https://www.internic.net/domain/named.root</span></span> $</span> sudo</span> mv</span> root.hints</span> /var/lib/unbound/</span></span> $</span> sudo</span> service</span> unbound</span> restart</span></span></code></pre> You can verify if the DNS queries are being resolved by:</p> #</span> 6363 is where I have configured my Unbound server to listen</span></span> dig</span> mrkaran.dev</span> @127.0.0.1</span> -</span>p</span> 6363</span></span></code></pre> Pi-hole official docs have a great</a> explainer on how to configure Unbound with Pi-hole so I won’t be repeating the steps here again.</p> You can configure Pi-hole to forward accepted DNS queries from port 53</code> (standard) to 127.0.0.1#6363</code> (unbound).</p> </p> The Endgame#</a></h2> I plan to self-host a couple of more things. DNS is something I am really interested in and in future I plan to host my own DNScrypt server soon-ish</em>.</p> I believe if you own</em> your data you’re in better control of your digital identity. Watching too much of Black Mirror added to the paranoia to an extent, I suppose! I grew up in the late nineties and I’ve seen the internet primarily as a set of the decentralized toolchain. There’s no reason we should let go any of that to the hands of a few corp giants and make it centralized.</p> Ending this long-ish post by a beautiful quote:</p> Study after study has shown that human behaviour changes when we know we’re being watched. Under observation, we act less free, which means we effectively are</em> less free. ― Edward Snowden</p> </blockquote> Cheers! :)</p> Common Docker Mistakes - Episode 1 2019-04-15T10:57:55+05:30 So, off late, I’ve been dabbling a lot with Docker to explore the world of containerization (too late to the party, eh?). I plan to write about some common docker gotchas. The plan is to document these learnings so they might help someone getting started with docker and also serve as a reference for myself in the future. Let me be clear, if you read the docs you will find the exact same information and there’s nothing new that I have discovered. It took me some time to get around the following issues and I believe some of you might be struggling with the same. I just feel when you’re starting out with completely new technology, things quickly can become overwhelming and it’s A-OK to feel so. The important part is to not get intimidated by it and focus on learning the basics. Different pieces start coming together and there you have a solved puzzle :)</p> The mysterious case of bind mounts and volumes#</a></h2> Ah! Storage. It’s always not a rosy scenario whenever someone mentions storage and containers in the same sentence. Anyway, so I had this requirement where I needed 2 containers to share data. Either of the containers could modify this data, so it made for a strong use case for volumes. But for some strange reason, I decided to use bind mounts. My thought process was that I’ll bind the mount path of the host to container and both of them could share the data. Now, I know, I know all the docker veterans already facepalming so hard, but in case anyone new to docker is reading it, it works the exact way I described. The host path will be mounted ON the container, so if your host path is empty, so will your container be. It took me quite some time to figure this out because of the side effect of it. I had this line in my docker-compose:</p> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> b</span>ind</span></span> s</span>ource</span>:</span> /</span>etc/custom/data</span></span> t</span>arget</span>:</span> /</span></span></code></pre> As you can guess, I am mounting an empty folder etc/custom/data</code> on the root directory of the container /</code>. This was an nginx</code> container and I got the weird error that nginx</code> executable isn’t found. It became clear that I have obviously done something wrong. After reading the documentation, it became clear that I had to use something like Named Volumes</a> and use the same volume label for both the containers. Here’s the correct docker compose example (I have removed the unnecessary fluff and only included the volumes part):</p> n</span>ginx</span>:</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> a</span>ssets-vol</span></span> t</span>arget</span>:</span> /</span>usr/share/nginx/frontend</span></span> f</span>rontend</span>:</span></span> v</span>olumes</span>:</span></span> -</span> t</span>ype</span>:</span> v</span>olume</span></span> s</span>ource</span>:</span> a</span>ssets-vol</span></span> t</span>arget</span>:</span> /</span>frontend/dist</span></span> </span> v</span>olumes</span>:</span></span> a</span>ssets-vol</span>:</span></span></code></pre> assets-vol</code> is a named volume and can be managed using docker API.</p> CMD vs RUN#</a></h2> Now, this is particularly interesting. So I have a volume mount as shown in the previous example, and quite naively I am copying some files from container 1 to container 2 at build stage. I get the error that this path doesn’t exist. I am seriously reconsidering my life decisions right now.</p> RUN cp /frontend/dist /usr/share/nginx/frontend</code></p> And then it became apparent that the volume is only mounted while the container is running, not when it is building. So I had to use CMD.</p> CMD cp /frontend/dist /usr/share/nginx/frontend</code></p> Why RUN</code> doesn’t work and CMD</code> works, you might ask? This is because that’s how docker volumes are in nature. They bypass the unionFS (which is used to build docker images). UnionFS chain together a layer of images and build new images on top. Whenever you run a container from an image, a new layer is created for the container process. If you specify a path as a volume, this path doesn’t get committed</code> to the container data layer and is bypassed. So, TL;DR, volumes are really only accessible when the container is running and you can’t access them while building.</p> Epilogue#</a></h3> I plan to share more such silly mistakes of mine while exploring more of Docker (and hopefully running production workloads on it soon!). It really is fun though, believe me :)</p> Setting up Kong API Gateway - Part 2/2 2019-03-23T11:57:55+05:30 Managing Kong using UI#</a></h1> Konga</a> is an unofficial</em> project which is basically an UI for Admin API of Kong. This post is about how to setup Konga and configuring an upstream service. We will also add an authentication layer to our upstream.</p> Docker to the rescue#</a></h2> We will use Docker to quickly setup Konga as it a Node.js project and it is a lot of trouble building node projects on server, so let’s just use Docker.</p> Our docker-compose.yml</code> for reference:</p> v</span>ersion</span>:</span> "</span>3.1</span>"</span></span> </span> s</span>ervices</span>:</span></span> d</span>b</span>:</span></span> i</span>mage</span>:</span> p</span>ostgres</span></span> r</span>estart</span>:</span> a</span>lways</span></span> p</span>orts</span>:</span></span> -</span> 5</span>432:5432</span></span> e</span>nvironment</span>:</span></span> P</span>OSTGRES_PASSWORD</span>:</span> "</span><redacted></span>"</span></span> v</span>olumes</span>:</span></span> -</span> /</span>home/ubuntu/docker/volumes/postgresql/:/var/lib/postgresql/data</span></span> n</span>etworks</span>:</span></span> -</span> d</span>ockergalaxy</span></span> </span> a</span>pp</span>:</span></span> i</span>mage</span>:</span> p</span>antsel/konga:latest</span></span> r</span>estart</span>:</span> a</span>lways</span></span> p</span>orts</span>:</span></span> -</span> 1</span>337:1337</span></span> e</span>nvironment</span>:</span></span> D</span>B_URI</span>:</span> "</span>postgresql://user_redacted:pass_redacted@db:5432/db_redacted</span>"</span></span> D</span>B_ADAPTER</span>:</span> "</span>postgres</span>"</span></span> n</span>etworks</span>:</span></span> -</span> d</span>ockergalaxy</span></span> </span> n</span>ginx</span>:</span></span> i</span>mage</span>:</span> n</span>ginx:latest</span></span> p</span>orts</span>:</span></span> -</span> 8</span>0:80</span></span> -</span> 4</span>43:443</span></span> v</span>olumes</span>:</span></span> -</span> /</span>home/ubuntu/docker/volumes/nginx/conf/:/etc/nginx/conf.d</span></span> -</span> /</span>home/ubuntu/docker/volumes/nginx/ssl/:/etc/ssl/certs/konga/</span></span> n</span>etworks</span>:</span></span> -</span> d</span>ockergalaxy</span></span> </span> n</span>etworks</span>:</span></span> d</span>ockergalaxy</span>:</span></span></code></pre>docker-compose</span> pull</span></span> docker-compose</span> up</span> -</span>d</span></span></code></pre> Verify, if all 3 containers are up.</p> $</span> docker</span> ps</span></span> CONTAINER</span> ID</span> IMAGE</span> COMMAND</span> CREATED</span> STATUS</span> PORTS</span> NAMES</span></span> 5f20abcd81fc</span> nginx:latest</span> "</span>nginx -g 'daemon of…</span>"</span> 29</span> hours</span> ago</span> Up</span> 28</span> hours</span> 0.0.0.0:80</span>-</span>></span>80/tcp,</span> 0.0.0.0:443</span>-</span>></span>443/tcp</span> deployment_nginx_1</span></span> ff20abcd24a66</span> pantsel/konga:latest</span> "</span>/app/start.sh</span>"</span> 30</span> hours</span> ago</span> Up</span> 29</span> hours</span> 0.0.0.0:1337</span>-</span>></span>1337/tcp</span> deployment_app_1</span></span> 5848abcd27d2a</span> postgres</span> "</span>docker-entrypoint.s…</span>"</span> 30</span> hours</span> ago</span> Up</span> 29</span> hours</span> 0.0.0.0:5432</span>-</span>></span>5432/tcp</span> deployment_db_1</span></span> </span></code></pre> Konga is running on port 1337</code> and you can verify the same by doing a curl</code>:</p> curl</span> http://localhost:1337</span> #</span> should return HTTP 200</span></span></code></pre>Adding an Upstream to Kong#</a></h2> Visit Konga Admin Dashboard to login. If this is a first time login, you need to activate the connection to Kong’s Admin API by visiting Connections tab. This step needs to be done by each admin user, individually.</p> </p> Some common terminology before we begin setting up our APIs: (Source</a>)</p> Term</th> Description</th></tr></thead> client</code></td> Refers to the downstream client making requests to Kong’s proxy port.</td></tr> upstream service</code></td> Refers to your own API/service sitting behind Kong, to which client requests are forwarded.</td></tr> Service</code></td> Service are abstraction of each of your own upstream services.</td></tr> Route</code></td> Routes are entrypoints into Kong, and defining rules for a request to be matched, and routed to a given Service.</td></tr> Plugin</code></td> This refers to Kong “plugins”, which are pieces of business logic that run in the proxying lifecycle. Plugins can be configured for an individual route, or a service or globally. An example of plugin which we use is Key Auth</code> plugin for authentication.</td></tr> </tbody></table> Adding New Service#</a></h2> Visit the Services section and click on Add new Service</strong>. Enter the following details for your upstream service here. You can refer to the below screenshot for reference:</p> </p> Field</th> Value</th></tr></thead> Name</code></td> Add a unique name for your upstream service.</td></tr> Description</code></td> Service description.</td></tr> Tags</code></td> List of tags to identify a group of services together. Press ENTER</strong> for any kind of array values in Konga UI.</td></tr> URL</code></td> Shorthand for setting Host</code>, Path</code>, Protocol</code> with just one value. Note that this is only a feature in Konga, Kong doesn’t have it, when using it with the Admin API calls directly.</td></tr> </tbody></table> Verify the details once and click on Save. Next, we’ll see how to add routes.</p> Adding Routes to Services#</a></h2> Visit the Services section and click on the service entity you just created.</p> </p> Go to the routes section and add the a new route entry for the service. You can refer to the below screenshot for reference:</p> </p> Field</th> Value</th></tr></thead> Name</code></td> Add a unique name for your route.</td></tr> Hosts</code></td> Kong checks for the hostname present in the incoming request’s header. If you specify this value then the Hostname must</strong> be present for Kong to match the request to this route. This is suitable only if you want to block any request made outside this hostname. You can leave it null if not needed.</td></tr> Path</code></td> List of paths present in incoming request. This is required to namespace the upstream endpoints. The client must send this prefix in the request, Kong will try to match any request’s path in this list of paths and based on the settint of strip_path</code> the request will be proxied.</td></tr> Strip Path</code></td> Boolean value, which configures Kong to strip the matching path from the incoming request to the upstream URL.</td></tr> </tbody></table> How routing actually works#</a></h3> Kong has the ability to configure really complex routing endpoints based on your usecases. For the simplicity and keeping this guide as generic, the basic (but most common) use case is discussed below by taking an example.</p> Let’s say your upstream URL(service</code>) is http://jsonplaceholder.typicode.com/</code>. To setup Kong for this service, we will simply add a route, with the path as /fake</code>. Here the path</code> acts as a namespace, to differentiate between different services. This can be helpful to avoid route collision if there are a lot of upstream services configured.</p> Consider the upstream API endpoint is https://jsonplaceholder.typicode.com/todos/</code>. If we specify Kong to have the path</code> as /fake</code> and set the strip_path</code> as True</code> then our incoming request should look like https://kongapigateway.com/fake/todos</code>.</p> Kong will try to match the path /fake</code> in this incoming request and look for the routes where the path is /fake</code>. Since it found the correct route, and we have set strip_path</code> to True</code>, Kong will just remove this particular path</code> prefix while reverse proxying to the upstream URL. In this way, our upstream doesn’t need to be concerned about the path</code> prefix as well.</p> Adding Consumers#</a></h2> Step 1: Visit the Consumers section to add consumers for your API. Here the consumers doesn’t really mean 1:1 users, it could be a particular production service wanting to consume</em> another service’s API.</p> </li> Step 2: Click on Add new consumer</strong> and enter the following details:</p> </li> </ul> </p> Step 3: Add the remaining details in the groups section. Visit the Credentials sections, click on API Keys</code> (since we are using Key Auth plugin for authentication) and simply click on Submit</strong> button, since Kong will auto generate the API key for you (which most likely will be more secure than any random key you will enter).</li> </ul> </p> </p> Step 4: Leave the other details as it is. We will add this consumer to our Services page. Visit the services tab, click on the service entity you created and go to Plugins section. Click on Add New Plugin</strong> and the select Key Auth from the list. </li> </ul> In the modal that opens up, you need to specify the consumer UUID which was created (or leave it as blank for all consumers to access). More on controlling the access, is present later in the next section. </p> Step 5: This feels a bit out of the flow, but after checking out Kong docs, there’s no alternate for this. In the previous step you would have been wondering, what if you need multiple consumers for the service, and want to identify them individually. Since Kong requires you to give either the Consumer UUID or leave it blank so all consumers are tied, the way around this is to use another plugin, called ACL Plugin</strong>. With this plugin, you can whitelist a group of consumers, so only the whitelisted consumers can access the API even though you specified All Consumers</em> in the Key Auth plugin. ACL Plugin</em> must be used in conjuction with another authentication plugin.</li> </ul> You can refer to the below screenshot for reference: </p> </p> NOTE</em>: This is the same group, which you created at the time of Consumer creation. You can group different consumers based on the service they consume, hence the naming convention one can</em> follow is -consumers</em>.</p> You can see all the eligible consumers for the service, in Eligible Consumers</em> tab.</p> Using Kong as an end user#</a></h2> You need to replace your upstream API endpoints and all other custom authentication with just the Kong’s URL, the namespace for the service, and add the authentication keys in header while sending the request.</p> For example:</p> If your original request is:</p> curl</span> -</span>i</span> -</span>X</span> GET</span> -</span>-url</span> http://jsonplaceholder.typicode.com/todos/</span></span></code></pre> The modified request becomes:</p> curl</span> -</span>i</span> -</span>X</span> GET</span> -</span>-url</span> https://kongapigateway.com/fake/todos/</span> -</span>-header</span> "</span>X-ACCESS: ENTER_KEY_HERE</span>"</span></span></code></pre>Accessing user authentication details in your API#</a></h3> You can remove all custom authentication method in your API services as Kong forwards the user information in the headers. You can use the following headers while processing the incoming request to identify the users and implement custom business logic:</p> Header Key</th> Value</th></tr></thead> X-Consumer-ID</code></td> ID of the Consumer on Kong</td></tr> X-Consumer-Custom-ID</code></td> custom_id of the Consumer (if set)</td></tr> X-Consumer-Username</code></td> username of the Consumer (if set)</td></tr> X-Anonymous-Consumer</code></td> will be set to true when authentication failed, and the ‘anonymous’ consumer was set instead.</td></tr> X-Credential-Username</code></td> the username of the Credential (only if the consumer is not the ‘anonymous’ consumer)</td></tr> </tbody></table> The End#</a></h2> Hope you liked the two part series of setting up and managing your Kong cluster. Do reach out to me @mrkaran_</a> in case of any feedback for the post. Thanks!</p> Setting up Kong API Gateway - Part 1/2 2019-03-23T10:57:55+05:30 Kong#</a></h1> Kong</a> is an API Gateway, which basically reverse proxies every incoming request to the upstream URL. It is pretty useful if you have a lot of internal services which need to talk to each other (through HTTP) but you don’t want to keep managing the authentication layer, rate limiting, hosts whitelisting and other such things in every service. Kong acts as a central entrypoint to all other services’ API endpoints and all the common fluff is heavylifted by Kong’s API layer.</p> Kong follows a plugin</a> approach, which makes it extensible and you can even make your own plugins</a>. Using plugins, it is possible to modify the request, add authentication layer at the Kong layer, forward user meta information headers to the upstream.</p> Amongst other API Gateway solutions, Kong is pretty straightforward to get started with and has a nice community support as well. All of the actions to configure your API endpoints and manage them can be done through a RESTful Admin API.</p> The following setup guide describes how the infra is setup for High Availability</strong> of a cluster of Kong nodes.</p> Infra Setup#</a></h2> Each individual Kong node is stateless, since it is always connected to an external datastore. In this tutorial, we will provision 2 nodes for Kong. The instances are frontended by Amazon’s ELB which routes the traffic internally to either of a Kong node using internal DNS. Kong requires a datastore to fetch the information about upstream APIs, consumers, routing mechanisms, plugins, so each of the Kong node must be in sync with the other Kong node. We will achieve the same by using Cassandra as our database for Kong, which is being run in a clustering mode. Cassandra uses it’s gossip mechanism to ensure the other Cassandra node is upto-date with any new changes to the data.</p> Scaling Kong#</a></h3> To scale Kong in future, we can keep on adding multiple Kong nodes horizontally and attaching them to one of the Cassandra node. This way we can have multiple Kong nodes in one cluster each pointing to one central Cassandra datastore.</p> Setting up the cluster#</a></h2> We will use Ansible to automate the task of setting up Kong+Cassandra in each of 2 nodes. You can refer to the playbook</a> which will do the job.</p> ---</span></span> #</span> https://github.com/mr-karan/kong-ansible</span></span> #</span> Playbook to install the Cassandra and Kong</span></span> </span> -</span> h</span>osts</span>:</span> "</span>{{control_host}}</span>"</span></span> r</span>emote_user</span>:</span> "</span>{{control_user}}</span>"</span></span> b</span>ecome</span>:</span> yes</span></span> r</span>oles</span>:</span></span> -</span> r</span>ole</span>:</span> j</span>ava</span></span> -</span> r</span>ole</span>:</span> c</span>assandra</span></span> -</span> r</span>ole</span>:</span> k</span>ong</span></span></code></pre> After you run the playbook, there are a couple of important things which needs to be configured in order to have an HA setup. This setup guide assumes the playbook is run individually on 2 servers: srvr A</code> and srvr B</code>.</p> Important Directory Paths#</a></h3> Local location</th> Description</th></tr></thead> /usr/local/bin/kong</code></td> Kong executable binary</td></tr> /usr/local/kong</code></td> All the settings and logs are available under a namespaced directory, which will be called as PREFIX</code> in further sections.</td></tr> /etc/systemd/service/kong.service</code></td> Managing Kong as systemd service</td></tr> /etc/systemd/service/cassandra.service</code></td> Managing Cassandra db as systemd service</td></tr> /etc/cassandra/cassandra.yaml</code></td> Config for Cassandra</td></tr> /etc/kong/kong.conf</code></td> Config for Kong</td></tr> </tbody></table> First Steps#</a></h3> Setting up Cassandra#</a></h4> Let’s setup Cassandra first and run in clustering mode. Do these steps in both of the servers.</p> Stop any running cassandra node:</p> sudo</span> service</span> cassandra</span> stop</span></span></code></pre></li> Edit the cassandra config file and update the following values:</p> cluster_name: 'KongAPICluster'</span></span> seed_provider:</span></span> - class_name: org.apache.cassandra.locator.SimpleSeedProvider</span></span> parameters:</span></span> - seeds: "<private_ip_srvrA>,<private_ip_srvrB>"</span></span> listen_address: <private_ip_srvr></span></span> start_rpc: true</span></span></code></pre></li> Start cassandra on both the servers and check the status:</p> sudo</span> serice</span> cassandra</span> start</span></span></code></pre></li> Verify Cassandra clustering:</p> sudo</span> nodetool</span> status</span> #</span> Give it a couple of seconds (30-45) for both nodes to warm up and discover each other.</span></span></code></pre> The output of above command should look like:</p> $</span> sudo</span> nodetool</span> status</span></span> Datacenter:</span> datacenter1</span></span> =</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span></span> Status</span>=</span>U</span>p</span>/</span>D</span>o</span>w</span>n</span></span> \|</span>/</span> State=Normal/Leaving/Joining/Moving</span></span> --</span> Address</span> Load</span> Tokens</span> Owns</span> (effective</span>) Host ID Rack</span></span> UN</span> <</span>REDACTE</span>D</span>></span> 467.26</span> KiB</span> 256</span> 100.0%</span> <</span>REDACTE</span>D</span>></span> rack1</span></span> UN</span> <</span>REDACTE</span>D</span>></span> 496.36</span> KiB</span> 256</span> 100.0%</span> <</span>REDACTE</span>D</span>></span> rack1</span></span></code></pre></li> Troubleshooting Cassandra:</p> cqlsh unable to connect to cassandra</code> server:</strong> cqlsh</code> has a known bug in some versions with Python2.7 where it cannot connect to the cassandra server. Do the following steps to fix:</p> sudo</span> pip</span> install</span> cassandra-driver</span></span> export</span> CQLSH_NO_BUNDLED</span>=</span>TRUE</span></span></code></pre></li> Unable to discover the other cassandra cluster:</strong> This usally happens because of network connectivity issues. Verify both nodes are able to talk to each other by running Cassandra in single cluster mode and then issue the following commands:</p> #</span> in srvrA</span></span> netstat</span> -</span>lntvp</span> \|</span> grep</span> cassandra</span> #</span> should be present (port 9042 usually)</span></span> #</span> in srvrB, check similarly...</span></span> #</span> in srvrA</span></span> telnet</span> private_ip_srvrB</span> 9042</span> #</span> should connect</span></span> #</span> in srvrB, check similarly...</span></span></code></pre></li> </ul> </li> </ol> Setting up Kong#</a></h4> Let’s setup each Kong node in the cluster as following.</p> Stop any running kong instance:</p> sudo</span> service</span> kong</span> stop</span></span></code></pre></li> Edit the kong config file and update the following values:</p> .</span>.</span>.</span></span> admin_listen</span> =</span> <</span>private_ip_srvr</span>A</span>></span>:8001,</span> <</span>private_ip_srvr</span>A</span>></span>:8444</span> ssl</span></span> database</span> =</span> cassandra</span></span> db_update_propagation</span> =</span> 10</span> #</span>seconds</span></span> .</span>.</span>.</span></span></code></pre></li> Start kong on both the servers and check the status:</p> sudo</span> serice</span> kong</span> start</span></span></code></pre></li> Verify if Kong is running:</p> sudo</span> service</span> kong</span> status</span></span></code></pre></li> Run Kong Migrations:</p> Run the migrations on only</em> one cluster. Since the datastores will be in sync(eventual consistency</em> thanks to Cassandra), we don’t have to run migrations on the second cluster.</p> kong</span> migrations</span> -</span>c</span> /etc/path/to/config</span></span></code></pre></li> Troubleshooting Kong:</p> Check if Kong is actually running by sudo service kong status</code>. You can check for logs in $PREFIX/logs/error.log</code>.</li> If Kong is not running, you can try running kong check</code> for checking if the config file is correct. Kong additionally provides a health check command, which can be executed using kong health</code>.</li> </ul> </li> </ol> Managing Kong#</a></h2> Kong comes with Admin API</a> to manage all aspects of Kong. There is also an unofficial project which is aN UI on top of Kong’s API and it comes pretty handy to configure upstream endpoints, adding plugins etc.</p> To read more about it, you can continue reading the second part of the series here</a></p> Adding Prometheus configuration to your CI/CD workflow 2019-03-17T10:57:55+05:30 Prometheus</a> configurations can turn into a mess in no time if you have a lot of different jobs scraping different targets. Certainly you can use tools like jsonnet</a> to keep your YAML</code> files DRY but this post is not about that. I initially started off by writing one job to scrape a set of EC2 instances in a particular AWS VPC. Over the time, I had a requirement to do it over 3-4 different targets, each of them had their own rules and different type of exporters as well. The whole practice of SSH</code>ing into the server, opening the config file in vim</code> and editing in the server didn’t just feel right</em>. No surpises for guessing that I fcked up a few times with config errors (YAML sigh</em>) and finally decided that I need a better solution for this.</p> CI/CD for config files#</a></h2> “Wait, so now you’re telling me that I need to setup a full fledged pipeline just for a few config files?”, must have been the thought that echo’ed in your head. Before you jump to the conclusion that it’s overdoing things, IMHO it’s not. And it’s not that difficult either to set it up, so why not? After integrating CI in your workflow, you can be confident that no bad syntax in your PromQL queries or in general the YAML</code> syntax would break your monitoring system.</p> “That sounds amazing, show me teh code already”. Our Pipeline is fairly fairly simple:</p> Lint the code using promtool</a>.</li> Push to S3</li> Write a shell script to pull from S3, put the config in right places and restarts Prometheus systemd service</li> </ul> </p> You can automate the last step too, but I just wanted to keep atleast one manual check in this system so I decided against it. And since I don’t have a distributed Prometheus setup yet, it’s simple to keep things the old school way here </i>.</p> You can do pretty much all of the steps in the above pipeline of any CI of your choice, but I am using Gitlab</a>, so here’s a sample .gitlab-ci.yml</code> file:</p> s</span>tages</span>:</span></span> -</span> l</span>int</span></span> -</span> d</span>eploy</span></span> </span> p</span>rometheus-lint</span>:</span></span> s</span>tage</span>:</span> l</span>int</span> #</span> run this job on stage lint</span></span> i</span>mage</span>:</span> g</span>olang:1.11-alpine</span> #</span> pull go1.11 image from official docker repo</span></span> b</span>efore_script</span>:</span></span> -</span> a</span>pk update && apk add git</span> #</span> install git</span></span> s</span>cript</span>:</span></span> -</span> G</span>OOS=linux GOARCH=amd64 CGO_ENABLED=0 go get -u github.com/prometheus/prometheus/cmd/promtool</span> #</span> fetches promtool package</span></span> -</span> $</span>GOPATH/bin/promtool check rules rules/alerts/</span> #</span> run the `check rules` command for my prometheus rule files</span></span> </span> p</span>ush-to-s3</span>:</span></span> s</span>tage</span>:</span> d</span>eploy</span> #</span> run this job on stage deploy</span></span> e</span>nvironment</span>:</span></span> n</span>ame</span>:</span> p</span>roduction</span> #</span> tag the job metadata in this environment. useful to quickly revert deploys when shit hits the fan</span></span> i</span>mage</span>:</span> p</span>ython:3.7-alpine</span></span> s</span>cript</span>:</span></span> -</span> p</span>ip install awscli</span> #</span> adds aws cli tools</span></span> -</span> a</span>ws s3 sync prometheus/rules s3://mybucket/rules/</span> #</span> pushes our rule config files to s3</span></span> o</span>nly</span>:</span></span> r</span>efs</span>:</span> #</span> only allow deploy if branch is master.</span></span> -</span> m</span>aster</span></span> w</span>hen</span>:</span> m</span>anual</span> #</span> trigger this job manually</span></span></code></pre> A brief explanation for the above file:</p> stages: Used to define multiple stages in our pipeline and they are executed in order</strong>.</li> jobs: prometheus-lint</code> is one job which will be run in the stage lint</code>. I am using Docker executor with Gitlab runner, so the runner agent talks to the docker executor. Gitlab-CI file is basically an API abstracted to hide away these details from the end user so all the CI/CD files look pretty much the same but behind the scenes the way they are executed totally depends on the executor you choose. Since we are using Docker executor, the image</code> tag is picked up and the same docker image is pulled from a public docker repo for this job.</li> before_script</code> is basically an event which is called before we begin running our actual CI stuff. You can add your project dependencies here.</li> script</code> is a list of commands to be executed inside your environment (container for us).</li> </ul> Now that we have our basics about the CI/CD pipeline in place, let us see what each of the jobs is actually doing:</p> prometheus-lint</code>: Installs promtool</code> binary and runs it against our rules/</code> folder. The default working directory is our repository itself, so we didn’t have to give the absolute path.</p> </li> push-to-s3</code>: Installs aws</code> cli tools, so we can push the files to s3. In case you are wondering where are the access key and secret key present, I have added them as protected variables in my project settings, so only the protected branches (eg master</code>) can access them.</p> </li> </ul> </p> Automate everything? Nah mate#</a></h3> This is how the deploy.sh</code> script looks like:</p> #!</span>/bin/sh</span></span> aws</span> s3</span> sync</span> s3://mybucket/rules/</span> /etc/prometheus/config/rules/</span></span> promtool</span> check</span> config</span> /etc/prometheus/prometheus.yml</span></span> sudo</span> service</span> prometheus</span> restart</span></span></code></pre> As much as I’d like to be a cool hipster and run /deploy</code> from my slack bot (last I checked, there’s a legit term for this: ChatOps</em>), I simply don’t prefer that. Having a human intervention before doing critical deployments like this is OK, IMHO. I don’t update the Prometheus config often, so I don’t mind actually SSH-ing into the single instance and triggering a shell script which does the job for me. I also don’t have a distributed Prometheus setup as of yet. Things will definitely change based on your requirements and there’s no one size that fits all.</p> I’m much more confident now with my config changes and don’t have to pray to the server overlords everytime I restart Prometheus.</p> It just works</em> </i>.</p> Makefile for Golang projects 2018-10-11T18:10:55+05:30 Makefile is an awesome tool to group together a bunch of different rules and automate your build process. Makefile is used by make</code> which is essentially a file generator tool. It is generally used to compile and build programs from source by following rules</code> listed in the Makefile. People use Makefile for a lot of different purposes as well, for example converting md</code> to html</code> and publish these files to the web server.</p> Every makefile you see, is composed of rules. A rule</code> is declaration of a target</code> and the commands to be executed to generate the target. A target can be a file or an action to be performed (more on that later).</p> This is how a rule looks like in a Makefile</p> target:</span> dependencies</span></span> recipe</span></span></code></pre> When you run make target</code>, make</code> searches for the rule which begins with this target</code> and executes the dependecies (if required). It then runs a bunch of commands which are listed in the recipe. An important thing to understand here is that make</code> tracks the dependencies by their last modified time. So if the dependencies haven’t changed, then make will complain with make: 'target' is up to date.</code></p> Enough of theory, let’s get our feet wet by writing our first Makefile. One important thing about Makefile</code> is that you need to use tabs</code> and not spaces</code>. It is one of the rare nix</code> programs which is whitespace aware and this has been mentioned in The Unix-Haters Handbook</a> as well.</p> To begin with, let’s write a simple rule which removes any tempory object files using go clean</code> and previous binary file using good ol rm</code>:</p> clean:</span></span> go</span> clean</span></span> rm</span> -</span>f</span> sample.bin</span></span></code></pre> The target</code> here is clean</code>. There is something special going on here though. Imagine we have a file called clean</code> in our source directory? Let us try to run make clean</code> now Our directory structure:.</p> ├── Makefile</span></span> ├── clean</span></span> └── sample.bin</span></span></code></pre> On running make clean</code>:</p> make: `clean' is up to date.</span></span></code></pre> Every target in Makefile by default is a file target</code>. In our case clean</code> is a file target and make</code> tries to build this file clean</code> but since we already have a file with the same name clean</code> in our directory, make</code> is complaining there’s nothing to do.</p> Moreover, in this case our rule is more of an action</code> rather than building files. So for all such scenarios, make</code> provides an easy way where we can instruct it to just run the rule and ignore any filename in our directory. This is called a PHONY</code> target which is a special kind of target. PHONY</code> is just a way in make</code> to forcefully run a target and not care about generating files. Our aim with this rule is to run 2 commands and that’s about it. So this is the perfect example for using PHONY</code>. We can add .PHONY</code> target to our Makefile simply by this line:</p> .PHONY</span> :</span> clean</span></span> </span> clean:</span></span> go</span> clean</span></span> rm</span> -</span>f</span> sample.bin</span></span> </span></code></pre> Now when we run make clean</code> we get our expected output</p> go clean</span></span> rm -f sample.bin</span></span></code></pre> Let us extend our Makefile to do some common tasks:</p> .PHONY</span> :</span> build</span> run</span> fresh</span> test</span> clean</span></span> </span> test</span>:</span></span> go</span> test</span></span> </span> build:</span></span> go</span> build</span></span> </span> run:</span></span> ./sample.bin</span></span> </span> clean:</span></span> go</span> clean</span></span> rm</span> -</span>f</span> sample.bin</span></span></code></pre> If you have worked on any Golang project, these are very trivial actions on any Golang project. You will soon realise the power of Makefile when you have to do these steps repeatedly. Some people might argue then you can use aliases or simple shell scripts for the same. I vehemently disagree with that. Reason being, make</code> is a much more powerful tool than just running commands. make</code> has support for dependency tracking and it will only rebuild whatever is required. If you are working on a huge project where the build times are to the tune of hours, you will soon realise why shell scripts are inferior. Ofcourse, someone can point that they can write a shell script to do even that, by fetching the last modified time but why do the extra work when there’s an already existing tried and tested tool? make</code> also has support for parallel task execution, so you can just pass the flag -j {num}</code> to make</code> and it will run these {num} jobs parallely. All these benefits will be apparent for larger projects, but it is a good habit to write Makefile even for smaller projects.</p> </p> We will now make our Makefile a bit more sophisticated and introduce variables. If you want to custom name your binary, or inject variables at compile time</a>, you can declare these variables, for example:</p> BIN</span> :=</span> my-awesome-pro.bin</span></span> HASH</span> :=</span> $(</span>shell</span> git</span> rev-parse</span> -</span>-short</span> HEAD</span>)</span></span> COMMIT_DATE</span> :=</span> $(</span>shell</span> git</span> show</span> -</span>s</span> -</span>-format=%ci</span> $</span>{</span>HASH</span>}</span>)</span></span> BUILD_DATE</span> :=</span> $(</span>shell</span> date</span> '</span>+%Y-%m-%d %H:%M:%S</span>'</span>)</span></span> VERSION</span> :=</span> $</span>{</span>HASH</span>}</span> ($</span>{</span>COMMIT_DATE}</span>)</span></span></code></pre> We can modify our Makefile to use these variables:</p> build:</span></span> go</span> build</span> -</span>o</span> $</span>{</span>BIN</span>}</span> -</span>ldflags=</span>"</span>-X 'main.buildVersion=</span>$</span>{</span>VERSION</span>}</span>' -X 'main.buildDate=</span>$</span>{</span>BUILD_DATE</span>}</span>'</span>"</span></span> </span> run:</span></span> ./$</span>{</span>B</span>I</span>N</span>}</span></span> </span> test</span>:</span></span> go</span> test</span></span> </span> clean:</span></span> go</span> clean</span></span> rm</span> -</span>f</span> $</span>{</span>BIN</span>}</span></span></code></pre> We can auto version our builds and pass variables during the build time with go linker tool, on passing the -X</code> flag. That’s really neat, now whenever we do a make build</code> we get new version of the build automagically.</p> So now we have a working Makefile which helps us with trivial things, but everytime if we need to change something in our program and check, we still need to do these steps manually: make clean</code>, make build</code> and make run</code>. Won’t it be awesome if we could tell Makefile to do all this with just one command? Programmers are lazy creatures after all </i>.</p> In the beginning we saw a target is composed of recipe and dependencies. So we can just create a new PHONY</code> target with all these dependencies and any recipe if we want optionally.</p> fresh:</span> clean</span> build</span> run</span></span></code></pre> We created a new target</code> which depends on clean</code> to run first, then build</code> and finally run</code>. So everytime if we make some change in our Go program, all we need to run is make fresh</code>. Awesome, isn’t it?</p> We will finally add our last target which is a highly opinionated way of generating binaries for different OS and architectures.</p> prod:</span></span> goreleaser</span> -</span>-rm-dist</span> -</span>-snapshot</span></span> cp</span> dist/linux_amd64/</span>$</span>{</span>BIN</span>}</span>-linux.bin</span> .</span></span> rm</span> -</span>rf</span> dist</span></span></code></pre> This target runs goreleaser</code> which is a build automation tool. It then copies the required linux binary to the source directory and removes all the other junk.</p> You can even extend your Makefile to commit files to a repo, and rsync these binaries to the production server or initiate your CI/CD build process. The reason I like Makefile is because it serves as a living documentation for your project on how to build/deploy the project making it easier for new contributors to get started.</p> Some additional information#</a></h5> If you run make</code> without passing any target name, make</code> will run the first target present in the Makefile. To override this, you should set .DEFAULT_GOAL</code> setting and override the target</a> which you want to make as default.</p> </li> .PHONY</code> is just one way to tell make</code> that it is a special kind of target, you can also do the same by creating a target without any recipe. Read this</a> to know more.</p> </li> </ul> To know more about Makefile</code>, you can read the manual here</a></p> I hope you now appreciate Makefile</code> and try it out in your next project. I’d love feedback on this blog post, do reach me out at twitter</a> or email</a></p> Fin!</p> Bi-Weekly Music Post Part 4 2018-07-28T21:10:55+05:30 I am writing this post exactly after 30 days of the last one, caught up with some work and I couldn’t write a new post, apologies! To make up for the delay, I have some absolutely gold reccos this time which I believe you will like it.</p> Isaac Gracie - reverie#</a></h3> </a></p> Tash Sultana - Jungle#</a></h3> </a></p> 18th Dec - Tienas#</a></h3> </a></p> Pareek - Ariana and Amrina, Coke Studio Explorer 2018#</a></h3> </a></p> When The Curtain Falls - Greta Van Fleet#</a></h3> </a></p> Poppy Ackroyd - Paper#</a></h3> </a></p> Nothing Personal - Night Riots#</a></h3> </a></p> Do reach out to me @mrkaran_</a> with your dose of music reccos.</p> Fin!</p> Bi-Weekly Music Post Part 3 2018-06-28T21:30:55+05:30 Since the past one week, I’m having troubles in connecting to Spotify from India and my #1 source for discovering music is gone.</i> I’ve started using Saavn</code> but it has crappy recommendations and too much of “Bollywoody”/crap shit thrown all over. Also, it requires location access to suggest new music. LOL is this a joke or what? Like seriously, can’t think of a single reason why a ML algo would require my location to suggest me music.</p> Anyway, this week’s recommendation might be a bit more “Desi” as I’ve discovered some real good bands from our country and our beloved neighbours(ofcourse!)</p> Mist of Capricorn - Agam#</a></h3> </a></p> Rangapura Vihaara - Agam#</a></h3> </a></p> Toh Phir Aao - Levi’s Live Session 7 by Mustafa Zahid & ROXEN#</a></h3> </a></p> Aankhon Kay Sagar - Shafqat Amanat Ali, Coke Studio Pakistan, Season 2#</a></h3> </a></p> Hit Me Up - The PropheC#</a></h3> </a></p> Tareefan Reprise - Lisa Mishra#</a></h3> </a></p> Message In A Bottle - The Police#</a></h3> </a></p> Agnes - Glass Animals#</a></h3> </a></p> Psycho Killer - Talking Heads#</a></h3> </a></p> Do reach out to me @mrkaran_</a> with your dose of music reccos.</p> Fin!</p> Bi-Weekly Music Post Part 2 2018-06-14T12:30:55+05:30 Ive looped through Nescafe Basement like 100 times the past two weeks and still cant get enough of it. This is the kind of music I like, the kind that just grows on you until you loop endlessly and then hate it </i></p> Anyway, the following is a list of songs that I have discovered and liked the most in the past two weeks.</p> Tattva - aswekeepsearching#</a></h3> </a></p> Koothu Over Coffee - Agam#</a></h3> </a></p> Tere Ishq Mein Jo Bhi - Nescafe Basement#</a></h3> </a></p> Houndmouth - Sedona#</a></h3> </a></p> Heavy - Collective Soul#</a></h3> </a></p> The Youth - George Taylor#</a></h3> </a></p> Porcupine - Pigeons Playing Ping Pong#</a></h3> </a></p> Misirlou - Ashar Kazi & Shruti Naik (cover)#</a></h3> </a></p> Redbone - Childish Gambino (ukulele cover)#</a></h3> </a></p> Do reach out to me @mrkaran_</a> with your dose of music reccos.</p> Fin!</p> Receiving notifications from Supervisor 2018-06-03T10:57:55+05:30 </p> Supervisor Events#</a></h3> I had a seemingly simple task which was to receive notifications any time a process managed by Supervisor restarts. I wanted a generic solution where I could get notifications for any change in the process state. Supervisor Events</code> saved my day, although I would admit it wasn’t straightforward to set up.</p> Supervisor uses STDIN/STDOUT</code> mechanism to communicate with the event listener. You need to configure your event listener in such a way that it can understand the STDIN</code> sent by Supervisor and also communicate back using STDOUT</code>. You can write this event handler in any language you like as long as you conform to the specially formatted messages that Supervisor sends and expects. I had struggled the most at this step and my google-fu</code> didn’t help much in this case.</p> Supervisor by default will send these events even if no listener is configured. Once you have your own listener setup, you can execute any task you want, eg: send email/telegram/slack messages etc.</p> In order to configure your event listener, you need to add it to your supervisor.conf</code>. Here’s an example configuration:</p> [</span>eventlistener:wowevent</span>]</span></span> command</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>w</span>o</span>r</span>k</span>/</span>t</span>e</span>s</span>t</span>e</span>v</span>e</span>n</span>t</span>/</span>t</span>e</span>s</span>t</span>.</span>p</span>y</span></span> events</span>=</span>P</span>R</span>O</span>C</span>E</span>S</span>S</span>_</span>S</span>T</span>A</span>T</span>E</span>_</span>S</span>T</span>A</span>R</span>T</span>I</span>N</span>G</span></span> process_name</span>=</span>%</span>(</span>program_name</span>)</span>s</span>_</span>%</span>(</span>process_num</span>)</span>s</span></span> numprocs</span>=</span>1</span></span> autorestart</span>=</span>t</span>r</span>u</span>e</span></span> stderr_logfile</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>w</span>o</span>r</span>k</span>/</span>t</span>e</span>s</span>t</span>e</span>v</span>e</span>n</span>t</span>/</span>l</span>o</span>g</span>s</span>/</span>e</span>v</span>e</span>n</span>t</span>_</span>e</span>r</span>r</span>.</span>l</span>o</span>g</span></span> stdout_logfile</span>=</span>/</span>h</span>o</span>m</span>e</span>/</span>w</span>o</span>r</span>k</span>/</span>t</span>e</span>s</span>t</span>e</span>v</span>e</span>n</span>t</span>/</span>l</span>o</span>g</span>s</span>/</span>e</span>v</span>e</span>n</span>t</span>.</span>l</span>o</span>g</span></span></code></pre> For the program to know that it has to send a notification at wowevent</code> pool, you need to add the events</code> key to the program</code> section of supervisor.conf</code>.</p> [</span>program:myprog</span>]</span></span> .</span>.</span>.</span></span> events</span>=</span>P</span>R</span>O</span>C</span>E</span>S</span>S</span>_</span>S</span>T</span>A</span>T</span>E</span>_</span>S</span>T</span>A</span>R</span>T</span>I</span>N</span>G</span></span> .</span>.</span>.</span></span></code></pre> Now everytime myprog</code> is about to start, it will send an event to wowevent</code> event pool. Your event listener which is configured at /home/work/testevent/test.py</code> will handle the notification and execute tasks which you want to perform.</p> There are a bunch of event states that Supervisor captures, I was interested in knowing when my process has started, so I used PROCESS_STATE_RUNNING</code>. You can take a look at all different event types here</a>.</p> During all this experimentation I came across a bug (which I later found it is a known issue</a>, and an open bug since 4 years now). If you’ve been using Supervisor I am sure at least once you’ve been bitten by not rereading</code> the config file and wondering why Supervisor isn’t picking up changes in config file when you restart the Supervisor. So reread</code> and update</code> becomes a muscle memory after this event </i>. The bug with events is that if you make any changes to the event group, reread</code> doesn’t pick up this change. I initially thought this must be the standard way Supervisor is behaving because I didn’t change any program</code> group. I randomly decided to change the event listener name and BAM! Supervisor read the new configuration and everything suddenly works! ARGH. Why do complex problems have such simple solutions </i> (not a solution, rather a workaround, but you get the drift right?)</p> Supervisor Event Listener Protocol#</a></h3> Supervisor sends a header which is a key value pair of meta-attributes about the process and event. This header looks something like</p> ver:3.0</span> server:supervisor</span> serial:208</span> pool:mylistener</span> poolserial:0</span> eventname:PROCESS_STATE_RUNNING</span> len:69</span></span></code></pre> The event listener mylistener</code> will be in ACKNOWLEDGED</code> state when Supervisor sends the event PROCESS_STATE_RUNNING</code>. Now that the myslistener</code> state has received this ACKNOWLEDGED</code> state, the event listener will send READY</code> back to Supervisor. This is to let Supervisor know that the listener has received the notification. Supervisor puts the listener to BUSY</code> state now and here you can do write your custom task. Supervisor waits for the task to get executed and when it does, the listener needs to communicate back with the result. This process is one full request-response cycle.</p> Let us write a simple Python script which will listen to Supervisor event notification and communicate back, all in a protocol Supervisor understands.</p> import</span> sys</span></span> import</span> requests</span></span> </span> import</span> logging</span></span> logger</span> =</span> logging</span>.</span>getLogger</span>(</span>'</span>event_listener</span>'</span>)</span></span> handler</span> =</span> logging</span>.</span>FileHandler</span>(</span>'</span>/home/user/prod/events/logs/response.log</span>'</span>)</span></span> formatter</span> =</span> logging</span>.</span>Formatter</span>(</span>'</span>%(asctime)s</span> %(levelname)s</span> %(message)s</span>'</span>)</span></span> handler</span>.</span>setFormatter</span>(</span>formatter</span>)</span></span> logger</span>.</span>addHandler</span>(</span>handler</span>)</span></span> logger</span>.</span>setLevel</span>(</span>logging</span>.</span>DEBUG</span>)</span></span></code></pre> Here I am just setting up basic logging structure since I want to keep the stdout</code> messages separate from what Supervisor uses.</p> def</span> write_stdout</span>(</span>s</span>)</span>:</span></span> #</span> only eventlistener protocol messages may be sent to stdout</span></span> sys</span>.</span>stdout</span>.</span>write</span>(</span>s</span>)</span></span> sys</span>.</span>stdout</span>.</span>flush</span>(</span>)</span></span> </span> def</span> write_stderr</span>(</span>s</span>)</span>:</span></span> sys</span>.</span>stderr</span>.</span>write</span>(</span>s</span>)</span></span> sys</span>.</span>stderr</span>.</span>flush</span>(</span>)</span></span></code></pre> We have written helper functions which will be used to communicate with Supervisor. Now comes the main</code> part (Sorry for bad pun! </i>)</p> def</span> main</span>(</span>)</span>:</span></span> while</span> 1</span>:</span></span> #</span> Hey Supervisor, I'm ready for some action</span></span> write_stdout</span>(</span>'</span>READY</span>\n</span>'</span>)</span></span> </span> #</span> Reading the header from STDIN</span></span> line</span> =</span> sys</span>.</span>stdin</span>.</span>readline</span>(</span>)</span></span> write_stderr</span>(</span>line</span>)</span></span> </span> #</span> read event payload and print it to stderr</span></span> headers</span> =</span> dict</span>(</span>[</span> x</span>.</span>split</span>(</span>'</span>:</span>'</span>)</span> for</span> x</span> in</span> line</span>.</span>split</span>(</span>)</span> ]</span>)</span></span> data</span> =</span> sys</span>.</span>stdin</span>.</span>read</span>(</span>int</span>(</span>headers</span>[</span>'</span>len</span>'</span>]</span>)</span>)</span></span> write_stderr</span>(</span>data</span>)</span></span> </span> #</span> add your events here</span></span> notify_user</span>(</span>)</span></span> </span> #</span> transition from READY to ACKNOWLEDGED</span></span> write_stdout</span>(</span>'</span>RESULT 2</span>\n</span>OK</span>'</span>)</span></span> logger</span>.</span>debug</span>(</span>"</span>It's all fine and dandy</span>"</span>)</span></span> </span> if</span> __name__</span> ==</span> '</span>__main__</span>'</span>:</span></span> main</span>(</span>)</span></span></code></pre> Let us break this into pieces.</p> write_stdout(</span>'</span>READY\n</span>'</span>)</span></span></code></pre> We flush READY</code> with a linefeed character (\n</code>) to STDOUT</code>. Supervisor has put mylistener to BUSY</code> state now.</p> line</span> =</span> sys.stdin.readline</span>(</span>)</span></span> write_stderr(line</span>)</span></span></code></pre> line</code> would be the header which we discussed previously.</p> data</span> =</span> sys.stdin.read</span>(</span>int(headers[</span>'</span>len</span>'</span>]</span>)</span>)</span></span></code></pre> This part is interesting. Here, we capture the len</code> key from the header and read the next STDIN</code> line up to this many chars. The data</code> would consist of our event payload. Event payload looks something like:</p> processname:prog-restartv3_0</span> groupname:prog-restartv3</span> from_state:STOPPED</span> tries:0ver:3.0</span> server:supervisor</span> serial:25</span> pool:prog-restartv3</span> poolserial:3</span> eventname:PROCESS_STATE_STARTING</span> len:76</span>`</span></span></code></pre>notify_user</span>(</span>)</span></span></code></pre> This is the handler where you can send an email, send a request to API, log it to file etc.</p> write_stdout</span>(</span>'</span>RESULT 2</span>\n</span>OK</span>'</span>)</span></span></code></pre> Finally, we tell Supervisor to put the listener from BUSY</code> to ACKNOWLEDGED</code> state, by sending a result structure. The result could be FAIL</code> or OK</code>, so you need to send RESULT</code> followed by the length of the state variable. For example for OK</code> you will send RESULT 2\nOK</code> but for FAIL</code> you have to send RESULT 4\nFAIL</code>.</p> That’s pretty much all you need to start receiving notifications from Supervisor every time your program changes its state. If you found this article useful, I’d love if you share this on Twitter</a> or Facebook</a> and let your friends know about it too.</p>

Karan Sharma - Posts

Building an LLM-Assisted Incident Investigation Tool

2026-03-12T00:00:00+00:00

When you are debugging a production incident, you are forming hypotheses, testing them against evidence, ruling things out, trying not to confuse correlation with causation, and making triage calls under time pressure. The hard part is the reasoning – figuring out why the system is misbehaving, deciding what to stabilize first, knowing when to stop chasing a red herring.

But a surprising amount of the time is not spent reasoning. It is spent gathering evidence.

You are flipping between your metrics dashboard, log viewer, scheduler, cloud console, SSH sessions, deployment history, and a half-written scratchpad. You are trying to answer questions that are conceptually simple but operationally expensive:

What changed in the last hour?</li>
Did the error spike start before or after the deploy?</li>
Is this one sick node or a fleet-wide regression?</li>
Did the autoscaling event line up with the HAProxy alert?</li>
Are we looking at cause, effect, or a noisy side symptom?</li> </ul>
Each question requires pulling data from a different tool, in a different format, often with timestamps in different timezones. Then you stitch them into a timeline before you can even begin hypothesizing.
That evidence-gathering phase is where LLMs turned out to be useful for me. Not because they “solve incidents” – the hypothesis formation, the triage, the mitigation calls stay human. But an LLM with access to the right CLIs can chew through the data-gathering faster than I can by hand.
I took the system I’d been building internally and rebuilt it as a package for Pi</a> – a minimal, extensible terminal coding agent – called pi-sre-mode</code></a>. This post is about the journey from a custom agent platform to a thin extension, and why the value turned out to live in content, not code.
The Predecessor: llmduck#</a></h2> Before pi-sre-mode</code>, I had a system called llmduck</code>. It went through several rewrites – first in Go, then partially in Rust, and finally as a full TypeScript web application with a server (LLM orchestration, session persistence, WebSocket dispatch), a remote agent (command execution on production hosts with policy enforcement), and a React frontend with a split-panel investigation UI. The agent had one tool: bash</code>. All infrastructure interaction happened through CLIs – metrics, logs, scheduler state, cloud APIs, SSH. The LLM decided what to run, the agent executed it, and the server tracked the conversation. An SRE methodology skill guided the model through a structured investigation loop: triage, scope, gather evidence across metrics/logs/scheduling, correlate, conclude. It worked. Give the model good operational guidance and read-only access to the right CLIs, and it can do a lot of the mechanical data gathering that normally burns 20-40 minutes during an incident. It can compare scheduler restarts with metric trends. It can correlate a failing backend with a cloud autoscaling event. It can notice that one host is missing metrics entirely and that the “healthy” cluster is only healthy because the remaining nodes are absorbing the load. But the runtime was increasingly the wrong abstraction. I had a custom server, a custom agent, a custom frontend, custom registries for skills and templates, custom session persistence, and a CI pipeline to build and deploy it all. The upstream/wrapper two-repo design – where the generic product lived on GitHub and the org-specific overlay was rsynced into it during build – still meant I was maintaining a bespoke investigation platform. The more I worked on it, the more obvious it became: the content was the product. The runtime was plumbing. The skills, the prompts, the methodology, the failure pattern library, the runbooks, the templates – that was where the value lived. Everything else was infrastructure I was maintaining just to get those skills into an LLM session. Why Pi#</a></h2> I did not want another chat app. I wanted the investigation to live where I already work: in the terminal, next to the real tools. Pi gave me that, and the specific SDK primitives it exposes turned out to map almost 1:1 to what I had been building by hand in llmduck. A session looks like this: /check-connectors /incident investigate elevated p99 latency for payments-api, start with a timeline /report</code></pre> /incident</code> is an interactive wizard – pick a template, name the service, set a time window. /check-connectors</code> – verifies your infrastructure CLIs are reachable</li> /report</code> – generates a structured postmortem from the investigation</li> /sudo</code> – temporarily bypasses the read-only guardrails</li> </ul> How It Works#</a></h2> Pi extensions are TypeScript modules that receive an ExtensionAPI</code> object – hooks into the agent lifecycle, slash commands, session state, shell execution, UI primitives, and an inter-extension event bus. The whole extension is one file and the rest is markdown. The SDK has a few primitives that matter here: Carrying context across the conversation#</a></h3> The before_agent_start</code> hook fires before every agent turn. The extension builds a context block from the current incident state and appends it to the system prompt: pi.on("before_agent_start", (event) => { return { systemPrompt: `${event.systemPrompt}\n\n${incidentContext}` }; });</code></pre> Template, service name, time window, guardrail posture, overlay guidance, preferred skills – all injected automatically. You do not need to keep re-explaining “payments API, last 2 hours, high latency, read-only.” It follows you. Blocking mutations before they execute#</a></h3> The tool_call</code> hook intercepts every tool invocation before execution: pi.on("tool_call", async (event) => { const reason = getGuardrailBlockReason(event.toolName, event.input); if (reason) return { block: true, reason }; });</code></pre> The guardrail implementation is regex-based with token boundary detection: const BLOCKED_PATTERNS = [ { pattern: /sudo/, reason: "sudo blocked" }, { pattern: /rm/, reason: "file deletion blocked" }, { pattern: /kill|pkill|killall/, reason: "process termination blocked" }, { pattern: /bash\s+-c|sh\s+-c|zsh\s+-c/, reason: "shell trampoline blocked" }, { pattern: /eval/, reason: "eval blocked" }, { pattern: /\$\(/, reason: "subshell blocked" }, { pattern: /systemctl\s+(restart|stop|start)/, reason: "systemctl mutation blocked" }, { pattern: /aws\s+.*\s+(create|delete|update|put|run|start|stop|terminate|reboot)/, reason: "AWS mutation blocked" }, ];</code></pre> The token boundary prevents false positives – remove</code> does not match the rm</code> rule. The write</code> and edit</code> tools are unconditionally blocked. /sudo</code> disables this when needed. Persisting incident state#</a></h3> Pi sessions are tree-structured – you can fork, navigate branches, switch contexts. Incident state is persisted as append-only entries: pi.appendEntry<IncidentModeState>("incident-mode-state", state);</code></pre> The extension hooks into session_tree</code> and session_fork</code> events to carry incident state across branches automatically. If you fork to explore a side hypothesis, the incident context comes with you. Commands and UI#</a></h3> pi.registerCommand()</code> handles slash commands. The /incident</code> wizard uses ctx.ui.select()</code> and ctx.ui.input()</code> for interactive setup – template selection, service name, time window. pi.exec()</code> runs shell commands for connector checks. Inter-extension events#</a></h3> Pi has an in-process event bus (pi.events</code>) for cross-package communication – same JS runtime, no network, just pub/sub between extensions loaded from different npm packages. The base package subscribes to a channel, the overlay emits to it at session start. The decoupling is at the import level, not the process level. More on this below. Templates and Skills#</a></h2> Most incidents fall into a handful of patterns – 5xx spike, high latency, OOM crash loop, deploy regression, service down. Each pattern has a different starting point: for latency you want percentile breakdowns and dependency response times first; for OOMs you want restart history and memory growth; for a deploy regression you want a before/after comparison. Templates encode that. When you pick “High Latency” in the /incident</code> wizard, the system prompt gets a focused directive: check p50/p95/p99, look for saturation signals, check upstream dependencies, identify whether queueing is involved. The model starts in the right place instead of flailing. You can also skip the wizard entirely – /incident-5xx payments-api last 2h</code> sets up the investigation directly. The investigation methodology itself lives in skills – markdown files that get loaded into context: --- name: sre-methodology allowed-tools: [Read, Bash, Grep] --- ## Core Loop 1. **Observe**: Gather data. Metrics, logs, scheduler state. 2. **Hypothesize**: Form a theory from the evidence. 3. **Test**: Run a specific query that would confirm or disprove. 4. **Evaluate**: Did the evidence match? Adjust and repeat.</code></pre> The base package includes two skills: an observe-hypothesize-test-evaluate loop with first principles (evidence over speculation, build timeline before concluding, the loudest symptom is often downstream), and a concrete investigation playbook (scope the blast radius, check recent changes, follow the service path from edge to dependency, narrow using the template focus). Without the methodology, the model tends to jump to conclusions or fixate on the noisiest symptom. With it, investigations are more systematic. The Overlay System#</a></h2> The public package is intentionally generic. The real leverage comes from overlays – private Pi packages that layer org-specific knowledge on top. An overlay is a separate Pi package that emits an event at session start. The base package picks it up and merges it into the active configuration – templates, connector checks, skills, timezone hints, report output paths: export default function myOrgOverlay(pi: ExtensionAPI) { const overlay: IncidentOverlay = { id: "my-org", priority: 100, timezoneHint: "IST (UTC+5:30); metrics are UTC, logs are IST.", reportPathPattern: "rca/{{date}}-{{slug}}.md", defaultSkills: ["org-sre-methodology"], promptPreamble: "Use org service topology, strict timezone handling...", connectorChecks: [ /* ... */ ], templates: [ /* ... */ ], }; pi.on("session_start", () => { pi.emit("incident-mode:register-overlay", overlay); }); }</code></pre> Templates merge by ID (overlay wins). Skills are deduplicated. Prompt preambles stack across overlays. What an Overlay Adds#</a></h3> At work, my private overlay adds investigation templates tied to specific services and a few thousand lines of operational knowledge as skills – how to query our metrics stack, how to read our logs, how to interpret scheduler state, and how to write the postmortem. The interesting part is what those skills encode. Not just CLI syntax, but the gotchas you learn the hard way: timezone mismatches between tools, CPU metrics that mean something different depending on allocation config, log columns that don’t exist where you’d expect them, query patterns that accidentally match unrelated data. A failure pattern library teaches the model to recognize cascading failures, throughput cliffs (silence often means blocked, not recovered), and the difference between a root cause and a downstream symptom. This is the kind of institutional knowledge that normally lives in someone’s head or a stale wiki page. Encoding it as skills means it gets applied consistently during every investigation. Every team has different tools, naming conventions, auth assumptions, and definitions of “normal.” Mix that into the base package and you either leak private knowledge or make the tool unusably abstract. The overlay keeps them cleanly separated. What the LLM Is Actually Good At#</a></h2> This is not an “AI will run your ops team” post. The model does not do triage or make mitigation calls. What it does well is the stuff that eats wall-clock time: Gathering and cross-referencing evidence#</a></h3> The model does not stop at “good enough.” If you ask it to investigate a latency spike, it will check p50/p95/p99, break it down per node, compare against the pre-deploy baseline, look at upstream dependency latency, check if the error rate moved with it, and then go look at logs for the time window where the percentiles diverged. Building timelines from messy evidence#</a></h3> The model is good at stitching together outputs from different tools – logs in IST, metrics in UTC, a deploy at some offset, a scheduler restart in the middle, a cloud instance termination, a queue backup two minutes later – into a coherent narrative. It can produce a first-pass timeline like “memory growth started after deploy X, restart happened 14 minutes later, error rate rose only on one node, autoscaling replaced the sick instance, and downstream proxy alerts were a secondary symptom” faster than I can by hand. Producing a first draft of the postmortem#</a></h3> The postmortem is hardest to write right after the incident, precisely when it is most valuable. If the investigation happened inside Pi, /report</code> turns the conversation into a structured markdown RCA draft immediately – timeline, 5 Whys, impact, action items. That alone saves a lot of the “I’ll write the postmortem tomorrow” drift. Keeping the investigation disciplined#</a></h3> The methodology skills help the model stay in evidence-first mode: observe before hypothesizing, build a timeline before concluding, distinguish cause from effect, state uncertainty explicitly, say when a connector is missing instead of guessing. Without this, the model tends to fixate on the noisiest symptom. With it, investigations follow an observe-hypothesize-test-evaluate loop. Where It Still Fails#</a></h2> Guardrails are not a sandbox#</a></h3> The read-only protections are operational guardrails, not security boundaries. They catch accidental mutations during investigation, but they are regex patterns – not a sandbox. The real defense is least-privilege at the infrastructure level: an AWS IAM role scoped to read-only, a metrics API token without write access, an SSH key that can only reach jump hosts. If the credentials the agent has cannot mutate anything, it does not matter if a guardrail regex gets bypassed. The model can still hallucinate#</a></h3> It will sometimes invent a metric name, over-index on a noisy symptom, or keep running commands when the evidence is already sufficient. Good skills help but do not make it reliable enough to trust unattended. The Bigger Lesson#</a></h2> I spent weeks building and rebuilding a custom agent platform. Rewrote it twice in different languages. All plumbing. The things that actually made the investigations good were: the SRE methodology skill that encoded how to think about incidents</li> the failure pattern library that taught the model to recognize cascade failures and throughput cliffs</li> the per-tool skills that encoded gotchas like “that CPU metric means something different in a cgroup” and “silence in the logs means blocked, not recovered”</li> the templates that seeded the investigation direction</li> the guardrails that kept the model from accidentally mutating production</li> </ul> All of those are markdown files and a few dozen regex patterns. I kept building runtime to deliver content. Once I had a platform that could host markdown skills and intercept tool calls, the custom stack had no reason to exist. The code is at github.com/mr-karan/pi-sre-mode</a>. Fin!

A Web Terminal for My Homelab with ttyd + tmux 2026-03-05T00:00:00+00:00 I wanted a browser terminal at terminal.mrkaran.dev</code> that works from laptop, tablet, and phone without special client setup. The stack that works cleanly for this is ttyd</a> + tmux. Architecture#</a></h2> Browser -> Caddy -> ttyd -> nsenter -> su - karan -> tmux(main)</code></pre> Two decisions matter most: ttyd</code> handles terminal-over-websocket behavior well.</li> -m 1</code> enforces a single active client, which avoids cross-tab resize contention.</li> </ol> Docker Compose (current)#</a></h2> services: webterm: image: tsl0922/ttyd container_name: webterm restart: unless-stopped command: > ttyd -W -p 8080 -m 1 nsenter -t 1 -m -u -i -p -- su - karan -c "tmux new-session -A -s main" privileged: true pid: "host" networks: - public_proxy</code></pre> Why each flag matters: -W</code>: writable shell</li> -p 8080</code>: matches my existing Caddy upstream (webterm:8080</code>)</li> -m 1</code>: one active client only (no resize fight club)</li> nsenter ...</code>: real host shell from inside the container</li> su - karan</code>: correct login environment and tmux config loading</li> tmux new-session -A -s main</code>: persistent attach/re-attach</li> </ul> Caddy#</a></h2> terminal.mrkaran.dev</code> reverse proxies to webterm:8080</code> with TLS via Cloudflare DNS challenge. Because ttyd uses WebSockets heavily, reverse proxy support for upgrades is essential. tmux profile for agentic workflows#</a></h2> I tuned tmux for long-running agent sessions, not just manual shell use. Long-run defaults#</a></h3> history-limit 200000</code></li> remain-on-exit on</code></li> window-size latest</code></li> mode-keys vi</code> / status-keys vi</code></li> </ul> Better operational visibility#</a></h3> status line shows host + session + path + time</li> pane border shows pane number + current command</li> active pane is clearly highlighted</li> </ul> Keybinds I actually use#</a></h3> Prefix: Ctrl-b</code> S</code>: create/attach named session</li> N</code>: create named window</li> R</code>: rename window</li> s</code>: session/window picker</li> y</code>: toggle synchronize-panes</code></li> h/j/k/l</code>: pane movement</li> H/J/K/L</code>: pane resize</li> </ul> Copy/paste that is not annoying#</a></h3> This was a big pain point, so I added both workflows: Browser-native copy Ctrl-b m</code> to turn tmux mouse off</li> drag-select + browser copy shortcut</li> Ctrl-b m</code> to turn tmux mouse back on</li> </ul> </li> tmux copy mode Ctrl-b [</code> enters copy mode and shows COPY MODE ON</code></li> v</code> select, y</code> copy (shows Copied selection</code>)</li> q</code> or Esc</code> exits (shows COPY MODE OFF</code>)</li> </ul> </li> </ol> On mobile, ttyd’s top-left menu (special keys) makes prefix navigation workable. Security model#</a></h2> This is tailnet-only behind Tailscale. No public exposure. Still, the container has privileged: true</code> and pid: host</code>, which is a strong trust boundary. If you expose anything like this publicly, add auth in front and treat it as high-risk infrastructure. Result#</a></h2> The terminal is now boring in the best way: stable, predictable, and fast to reach from any device. Setting Up LiteLLM with AWS Bedrock 2026-02-16T00:00:00+00:00 I recently set up LiteLLM</a> with AWS Bedrock as the LLM provider. The docs cover the happy path, but there are a few gotchas that cost me some debugging time. This post covers what I learned, from basic setup to per-project cost tracking with Application Inference Profiles. Model Format#</a></h2> Bedrock models use the bedrock/</code> prefix followed by AWS’s model identifiers: bedrock/anthropic.claude-opus-4-6-v1 bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0</code></pre> Nothing surprising here. LiteLLM uses the prefix to route to the right provider. Authentication#</a></h2> This is where the first gotcha lives. Bedrock doesn’t use API keys. It authenticates via standard AWS credentials: Environment variables: AWS_ACCESS_KEY_ID</code> + AWS_SECRET_ACCESS_KEY</code></li> AWS profile: AWS_PROFILE</code></li> IAM role (for EC2/ECS/Lambda)</li> </ul> You also need AWS_REGION_NAME</code> set to the region where Bedrock is enabled (e.g., ap-south-1</code>). The gotcha: don’t pass api_key</code> to LiteLLM when using Bedrock. If you include an api_key</code> parameter in your config, LiteLLM tries to use it instead of the AWS credential chain and auth fails silently. You need to either return None</code> for the API key or omit it from the config entirely. # Wrong - breaks AWS credential chain llm_config = {"model": "bedrock/anthropic.claude-opus-4-6-v1", "api_key": some_key} # Correct - let LiteLLM use AWS credentials llm_config = {"model": "bedrock/anthropic.claude-opus-4-6-v1"}</code></pre> This one took a while to figure out because the error messages don’t point you in the right direction. The top_p</code> and Temperature Conflict#</a></h2> Bedrock’s Anthropic models reject requests that include both temperature</code> and top_p</code>. If your SDK or framework defaults top_p=1.0</code>, you need to explicitly clear it: if model.startswith("bedrock/"): llm_config["top_p"] = None</code></pre> Without this, you’ll get a validation error from the Bedrock API. The fix is simple, but the error message isn’t immediately obvious about what’s conflicting. Inference Profiles#</a></h2> Bedrock has two types of inference profiles, and the second one is where things get interesting for cost management. Cross-Region (System-Defined)#</a></h3> AWS provides these out of the box. They route requests across regions for higher throughput and availability: global.anthropic.claude-opus-4-6-v1</code></pre> You can list them with: aws bedrock list-inference-profiles \ --type-equals SYSTEM_DEFINED \ --region ap-south-1</code></pre>Application Inference Profiles for Cost Tracking#</a></h3> Application Inference Profiles (AIPs) are tagged wrappers around a model. The killer use case is granular cost attribution via cost allocation tags. Instead of seeing one blob of “Bedrock spend” in your AWS bill, you can break it down by project, team, or service. Create one with: aws bedrock create-inference-profile \ --inference-profile-name my-project-opus-4-6 \ --model-source "copyFrom=arn:aws:bedrock:ap-south-1::foundation-model/anthropic.claude-opus-4-6-v1:0" \ --tags key=project,value=my-project \ --region ap-south-1</code></pre> This gives you an ARN like: arn:aws:bedrock:ap-south-1:123456789012:application-inference-profile/abcdef123456</code></pre>Using AIPs with LiteLLM#</a></h3> The standard bedrock/</code> route can’t parse ARNs. Use the bedrock/converse/</code> route instead: bedrock/converse/arn:aws:bedrock:ap-south-1:123456789012:application-inference-profile/abcdef123456</code></pre> The bedrock/</code> prefix still matches for provider detection and the top_p=None</code> fix, so no code changes needed on your end. Querying Costs via CLI#</a></h2> Once your AIPs are tagged, you can query costs using the Cost Explorer API. Total Bedrock spend for a month: aws ce get-cost-and-usage \ --time-period Start=2026-02-01,End=2026-03-01 \ --granularity MONTHLY \ --metrics "UnblendedCost" \ --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \ --region us-east-1</code></pre> Bedrock spend grouped by your cost allocation tag (per-project breakdown): aws ce get-cost-and-usage \ --time-period Start=2026-02-01,End=2026-03-01 \ --granularity DAILY \ --metrics "UnblendedCost" \ --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Bedrock"]}}' \ --group-by Type=TAG,Key=project \ --region us-east-1</code></pre> Note: The Cost Explorer API always runs against us-east-1</code> regardless of where your resources are deployed. Cost Explorer Setup Checklist#</a></h3> A few things to get right before cost data starts flowing: The tag must be an active cost allocation tag. Enable it under Billing → Cost Allocation Tags.</li> Use the AIP ARN as the model string in LiteLLM. All invocations through it get tagged automatically.</li> In Cost Explorer, group by tag and select your tag key to see the per-project breakdown.</li> Cost data takes ~24 hours to populate after first usage, so don’t panic if it shows up empty initially.</li> </ol> Dependencies#</a></h2> LiteLLM needs boto3</code> to talk to Bedrock: boto3>=1.28.57</code></pre> Make sure it’s installed in your environment, otherwise LiteLLM will fail with an import error when you try to use the bedrock/</code> provider. Fin! Why Plain-Text Ledger is Powerful for Gullak 2026-02-02T00:00:00+00:00 In my previous post</a>, I introduced Gullak, an expense tracker I built to categorize transactions using LLMs. While the initial prototype used SQLite, I’ve since made a fundamental shift in how Gullak stores data. It now uses the ledger-cli format—a plain-text accounting standard that has been around for over 20 years. The Core Insight#</a></h2> Ledger-cli uses a format that is, at its core, just text files. There are no database migrations to manage, no proprietary binary formats, and absolutely no vendor lock-in. The promise is simple: your financial data should outlive the application you use to track it. Hackability in Gullak’s Context#</a></h2> Moving to a plain-text format unlocked several advantages that align perfectly with modern AI capabilities and the Unix philosophy. 1. AI-Native Format#</a></h3> Consider a typical transaction entry in a ledger file: 2026/01/21 Swiggy Expenses:Food:Delivery 584.23 INR Liabilities:CreditCard:ICICI -584.23 INR</code></pre> This structure is trivially parseable by Large Language Models (LLMs). An AI agent can read, write, and reason about these transactions without needing complex serialization logic. Compare this to the friction of extracting data from a SQLite blob or interfacing with a proprietary API like Splitwise’s. The text is the interface. 2. Git-Friendly#</a></h3> Because every transaction is just a few lines of text, your financial history becomes a git repository. Every change is a diff. This gives you: Full audit trail: git log -p -- main.ledger</code> shows exactly what changed and when.</li> Easy rollback: Made a mistake? git revert</code>.</li> “What-if” scenarios: Branch off to model a major purchase or a different budget strategy.</li> Collaboration: Family budgeting can be handled via Pull Requests.</li> </ul> 3. Unix Philosophy#</a></h3> The file serves as the API. You don’t need export buttons or data liberation requests. You can use standard Unix tools to query your finances. Find all Swiggy orders over 500 INR: grep -A2 "Swiggy" main.ledger | grep -E "[5-9][0-9]{2,}|[0-9]{4,}"</code></pre> Check your monthly food spending: ledger -f main.ledger bal Expenses:Food -p "this month"</code></pre> Export to CSV for Google Sheets: ledger -f main.ledger csv Expenses</code></pre>4. Extensibility via Comments#</a></h3> Gullak adds its own metadata using standard ledger comments, which are ignored by the accounting tools but used by the app: 2026/01/21 Zomato ; gullak:id 7559a51f ; gullak:source whatsapp ; gullak:user 919876543210 Expenses:Food:Delivery 584.23 INR</code></pre> gullak:id</code>: A unique ID for CRUD operations.</li> gullak:source</code>: Provenance tracking (e.g., entered via WhatsApp, web, or CSV).</li> gullak:user</code>: Multi-user support.</li> </ul> Custom tags for your own organization—like ; Recurring: Netflix</code>—just work out of the box. 5. Ecosystem Interoperability#</a></h3> Because the format is standard, Gullak plays nice with others: Paisa: Reads the same file for beautiful visualizations.</li> hledger: A Haskell alternative that is drop-in compatible.</li> Beancount: Can import ledger files.</li> Text Editors: Any editor (VS Code, Vim, Sublime) is a valid client.</li> </ul> What This Enables#</a></h2> Feature</th> How Plain Text Helps</th></tr></thead> WhatsApp logging</td> AI parses “swiggy 500” → appends text to file</td></tr> Receipt OCR</td> Extract data → format as ledger → append</td></tr> Bank CSV import</td> Transform CSV → ledger format → append</td></tr> Transaction editing</td> Find by gullak:id</code> → text replacement</td></tr> Undo/history</td> Git handles it for free</td></tr> Backup</td> cp main.ledger backup.ledger</code></td></tr> Migration</td> It’s text. There’s nothing to migrate.</td></tr> </tbody></table> The Trade-off#</a></h2> Of course, you lose ACID transactions, database indexes, and complex SQL queries. But for personal finance, the scale makes these unnecessary trade-offs. You likely have 10-50 transactions a month.</li> Running ledger bal</code> on 10 years of data takes less than 100ms.</li> The simplicity is the feature.</li> </ul> Conclusion#</a></h2> Adopting the ledger format turns Gullak from “yet another expense app” into a thin AI layer over your permanent financial record. By decoupling the data from the application logic, we ensure that the data remains accessible, hackable, and enduring. CLIs are the New AI Interfaces 2026-01-31T00:00:00+00:00 The industry is currently obsessed with defining standards for how Large Language Models (LLMs) should interact with software. We see a proliferation of SDKs, function calling schemas, and protocols like MCP (Model Context Protocol). They all aim to solve the same problem: bridging the gap between natural language intent and deterministic code execution. But we might be reinventing the wheel. The most effective tools for AI agents aren’t those wrapped in heavy “AI-native” integration layers. They are the tools that adhere to a philosophy established forty years ago: the command-line interface. The Unix Philosophy as an AI Protocol#</a></h2> An LLM’s native tongue is text. It reasons in tokens, generates strings, and parses patterns. The Unix philosophy, which emphasizes small tools, plain text interfaces, and standard streams, is accidentally the perfect protocol for AI interaction. Consider the anatomy of a well-behaved CLI: Discovery: tool --help</code> explains capabilities without hallucination.</li> Structure: tool --json</code> provides deterministic output for parsing.</li> Composition: Pipes (|</code>) allow complex workflows to be assembled on the fly.</li> </ul> When you give an agent access to a robust CLI, you don’t need to define 50 separate function schemas. You give it a shell and a single instruction: “Figure it out using --help</code>.” Context Economy: Lazy vs. Eager Loading#</a></h2> The current approach to agent tooling often involves dumping massive JSON schemas into the context window. Connecting to a standard MCP server might load dozens of tool definitions, involving thousands of tokens describing every possible parameter, before the user has even asked a question. This is “eager loading,” and it is expensive in terms of both latency and context window utilization. A CLI-driven approach is “lazy loaded.” The agent starts with zero knowledge of the tool’s internals. It burns zero tokens on schema definitions. Only when tasked with a specific goal does it invoke man</code> or --help</code>. It retrieves exactly the information needed to construct the command, executes it, and parses the result. This reflects the professional intuition of a senior engineer. We rarely memorize documentation. Instead, we prioritize the ability to quickly discover and apply the specific flags required for the task at hand. Leveraging the Skills Pattern#</a></h2> To bridge the gap between a raw CLI and an agent’s reasoning, we can leverage the Skills</a> pattern. This is an emerging standard for agent-based systems where capabilities are documented as self-contained units of knowledge. Instead of writing a Python wrapper that maps an API to a function call, you provide a Markdown file that explains when and why to use a specific CLI command. The agent uses this as a semantic index. Here is a snippet from a logchef.md</code> skill: --- name: logchef description: Query application logs via LogChef CLI. Use for investigating production incidents and analyzing traffic patterns. --- ## Common Workflows | Goal | Command Pattern | | -------------- | --------------------- | | Error Analysis | `logchef sql "..."` | | Live Tail | `logchef query '...'` | ## Example: Error Rates by Minute To visualize error spikes, use aggregation: ```sql logchef sql "SELECT toStartOfMinute(_timestamp) as ts, count() as errors FROM logs.app_logs WHERE service='api-gateway' AND level='ERROR' GROUP BY ts ORDER BY ts DESC LIMIT 60" --output json ```</code></pre> When I ask an agent to “check for error spikes in the API gateway,” Claude identifies that this skill is relevant to the request and loads it on-demand. It sees the example, adapts the SQL query to the current context, and executes the CLI command. The Markdown file serves as a few-shot prompt, teaching the model how to use the tool effectively without rigid code constraints. I maintain similar skill sets for AWS, Kubernetes, and Nomad. The AWS skill doesn’t wrap boto3; it simply documents useful aws ec2</code> and aws cloudwatch</code> commands. The Developer Experience: uv</code> and Single-File CLIs#</a></h2> When a CLI doesn’t exist, the barrier to creating one has never been lower. Modern Python tooling, specifically uv</code> with its inline script metadata, allows us to treat CLIs as disposable, single-file artifacts. I recently needed an agent to manage my Trello board. Rather than fighting with the Trello API documentation or looking for an abandoned library, I had the agent generate a CLI wrapper: #!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.11" # dependencies = ["typer", "httpx", "rich"] # /// import typer import httpx import json app = typer.Typer() @app.command() def list_cards(list_id: str, format: str = "table"): """Fetch all cards from a specific list.""" # Implementation details...</code></pre> This script is self-contained. It defines its own dependencies. It implements --help</code> and --json</code> automatically via typer</code>. It took minutes to generate and immediately unlocked Trello capabilities for the agent. The SaaS Imperative#</a></h2> The strategic takeaway for SaaS founders and platform engineers is significant. Your CLI is no longer just a developer convenience; it is your primary AI API. We are moving past the era where a REST API and a web dashboard are sufficient. If your product lacks a terminal interface, you are locking out the growing workforce of AI agents. Browser Automation is brittle, slow, and breaks with every UI update.</li> Direct API Integration puts the burden of schema management on the user.</li> CLIs offer a stable, discoverable, and composable interface that agents can learn and use autonomously.</li> </ul> The “hobby” CLI wrappers built by enthusiasts, such as those for Notion, Jira, or Spotify, are no longer just developer conveniences. They are becoming critical infrastructure. They provide the stable, text-based interface required for agents to interact with these platforms reliably. If you want your platform to be AI-ready, don’t just build an MCP server. Build a great CLI. Make sure it supports --json</code>. Write good man pages. The agents will figure out the rest. Logchef v1.0: The Journey to a Real Log Viewer 2025-12-22T00:00:00+00:00 About eight months ago I wrote about Logchef</a> – a log viewer I’d been building to scratch my own itch with log exploration at work. Back then it was basically a nicer way to query ClickHouse without writing raw SQL every time. Today I’m shipping v1.0, and it’s evolved into something I didn’t quite expect. </a> </div> Let me walk through the major features that made it to 1.0 and some of the engineering decisions behind them. Alerting with Alertmanager Integration#</a></h2> In that first post, I mentioned alerting as a “roadmap” item. It always felt like the obvious next step – you find a pattern in your logs, you want to know when it happens again. </a> </div> But building it took longer than expected. My first attempt was a “rooms” system – a home-grown notification router with its own email, Slack, and webhook channels. I got it working, then stared at the code for notification deduplication, grouping, silencing, and escalation. All problems that Alertmanager</a> has already solved and battle-tested in production for years. So I ripped out rooms and integrated Alertmanager instead. Now Logchef just fires alerts to Alertmanager, and you get all the routing logic – Slack, PagerDuty, email, webhooks, silencing, grouping, inhibition – without me reinventing it poorly. The workflow is simple: write a LogchefQL or SQL query, set a threshold (e.g., “fire if count > 100”), pick a frequency, configure severity and labels. Logchef runs your query on schedule, evaluates the threshold, and if it triggers, fires an alert. Alert history is stored with execution logs so you can debug why something fired (or didn’t). LogchefQL: From Toy Parser to Production Backend#</a></h2> The query language I wrote about originally was pretty basic – just filters that compiled to SQL on the frontend. Over the months it grew into something more capable, but more importantly, I rewrote the entire parser in Go and moved it to the backend. This also opens the door for a CLI tool later – same parser, same query language, different interface. Here’s what LogchefQL looks like now: namespace="prod" AND level="error" | message, trace_id</code></pre> The pipe operator (|</code>) selects specific columns instead of SELECT *</code>: msg.level="ERROR" | timestamp, msg.request.method</code></pre> Dot notation handles nested JSON fields. If your logs have a log_attributes</code> Map column with nested data: log_attributes.user.name = "john"</code></pre> For keys that contain dots (common in OTEL-style logs), use quoted field syntax: log_attributes."http.status_code" >= 500</code></pre>Why Move Parsing to the Backend?#</a></h3> The original frontend parser was TypeScript. It worked, but had problems: Inconsistency: The frontend generated SQL, but the backend had no idea what that SQL meant. Validation happened in two places. </li> Type-awareness: ClickHouse has Map</code>, JSON</code>, LowCardinality</code>, and various string types. The frontend didn’t know the schema, so it couldn’t generate optimal SQL for each column type. For a Map(String, String)</code> column, you want mapContains()</code> or ['key']</code> access. For JSON</code>, you want JSONExtractString()</code>. For regular String</code>, it’s a simple comparison. </li> Debugging hell: When a query failed, was it the parser? The SQL generator? ClickHouse syntax? Everything happened client-side, invisible to server logs. </li> </ol> The new architecture is cleaner: Frontend Backend | | | --- LogchefQL query --> | | | --> Parse (Go) | | --> Validate against schema | | --> Generate type-aware SQL | <-- SQL + results ----- | --> Execute on ClickHouse</code></pre> The backend exposes three endpoints: /logchefql/translate</code> (returns the SQL for “View as SQL”), /logchefql/validate</code> (real-time validation with debouncing), and /logchefql/query</code> (parse, validate, execute, return results). Moving parsing to the backend also made the field sidebar implementation cleaner – the same schema-aware code that generates WHERE clauses can filter field values based on your current query. The Field Sidebar#</a></h2> If you’ve used Kibana, you know the interaction: click a field, see its top values, click a value to add it as a filter. It’s the fastest way to explore logs when you don’t know exactly what you’re looking for. </a> </div> Building this for ClickHouse required solving a few problems: High-Cardinality Fields#</a></h3> You can’t just run SELECT DISTINCT field FROM logs</code> on a table with billions of rows. String fields like trace_id</code> would take forever and return millions of values. The solution is a hybrid loading strategy based on column types: LowCardinality and Enum fields: Auto-load values when the sidebar opens. These are designed for fields with limited distinct values.</li> String fields: Require an explicit click. A badge shows the count is unknown until you ask.</li> Complex types (Map, Array, Tuple, JSON): Excluded. You can’t have meaningful “distinct values” for a JSON blob.</li> </ul> Progressive Loading#</a></h3> Each field loads in parallel (max 4 concurrent) with a 15-second timeout. One slow or failed field doesn’t block others – you get a retry button for that specific field. Query Context#</a></h3> The sidebar respects your current query. If you’ve filtered to level="error"</code>, the field values update to show only values from error logs. This happens through the backend – the field values endpoint accepts the current LogchefQL query and applies it as a WHERE clause filter. Same parser, same SQL generator, consistent results. Query Cancellation#</a></h2> Hit Esc and it cancels the query in ClickHouse. Without this, pressing “Cancel” would just hide the spinner – the query kept running on the server, burning resources. The implementation uses ClickHouse’s query ID feature: SELECT * FROM logs WHERE ... SETTINGS query_id = 'logchef-abc123'</code></pre> When you hit Esc, the frontend calls a cancellation endpoint that runs: KILL QUERY WHERE query_id = 'logchef-abc123'</code></pre> The original query returns an error, the UI clears, ClickHouse frees resources. Simple, but requires plumbing the query ID through every execution path. AI Query Assistant#</a></h2> “Write a query that finds slowest endpoints by p99” actually works. The AI generates LogchefQL or SQL based on natural language and your table schema. </a> </div> Under the hood it uses go-openai</a>, so any OpenAI-compatible endpoint works – OpenAI, Ollama, vLLM, whatever you prefer. The system prompt includes your table schema so the model knows what fields exist. There’s also an MCP server</a> that exposes Logchef to AI assistants like Claude Desktop, Cursor, or any MCP-compatible client. Instead of context-switching between your AI chat and the log viewer, you can ask directly: “What log sources do I have access to?”</li> “Find all 500 errors in the last hour from the web service”</li> “Show me a histogram of log volume over the past day”</li> “What are the most common error messages in the database logs?”</li> </ul> The MCP server handles discovery (teams, sources, schemas), querying (full ClickHouse SQL), analysis (histograms, saved queries), and even admin operations. It’s a separate binary that runs alongside Logchef – configure it once, and your AI assistant can query your logs through natural conversation. Compact View for Terminal Lovers#</a></h2> </a> </div> Not everyone wants a table. The compact view is a terminal-style display that shows logs as formatted text with syntax highlighting. Denser and faster to scan for certain debugging workflows. Query Variables#</a></h2> Use {{namespace}}</code> in your query, and an input field appears automatically. Great for saved queries that teams want to reuse with different parameters. This was a community contribution from @songxuanqing</a>. The implementation detects {{variable}}</code> patterns in the query text and renders input fields dynamically. Team Management and RBAC#</a></h2> </a> </div> Logchef supports multi-tenancy with role-based access. Teams can have multiple data sources, and users can be members of multiple teams with different roles: Admin: Full access, can manage team members and sources</li> Editor: Can create/edit saved queries and collections</li> Viewer: Read-only access to query and explore logs</li> </ul> This integrates with OIDC for SSO, so you can use your existing identity provider. Admin UI for Runtime Config#</a></h2> Configure stuff without touching config files. The admin settings panel lets you change AI configuration, Alertmanager connection, authentication settings, and query timeouts. </a> </div> This was a migration from config files to database-backed settings. On first boot, Logchef seeds the database from config.toml</code>. After that, the UI takes over and changes are stored in SQLite. Backward compatible – existing config files still work, the UI just overrides them at runtime. No more SSH-ing into production to bump a timeout. Prometheus Metrics#</a></h2> A /metrics</code> endpoint exposes query execution times, error rates, active queries, and other operational data. There’s a pre-built Grafana dashboard</a> for monitoring Logchef itself. What’s Not in 1.0#</a></h2> Some things didn’t make the cut: Live tail: Streaming logs in real-time. Still on the roadmap.</li> Dashboarding: Multiple visualizations on one page. Logchef is query-focused; for dashboards, you probably want Grafana with ClickHouse as a datasource.</li> </ul> Calling It 1.0#</a></h2> Calling something “1.0” is weird. There’s no clear line where software becomes “ready.” But I’ve been using Logchef daily at work for months now, and it’s at the point where I trust it. The rough edges are mostly smoothed out. The architecture feels right. Building tools you use yourself is different. You’re the first to hit the rough edges, so you fix them. Slower than building for imaginary users, but the result is something you actually want to use. Thanks again to Kailash</a> for the early direction (schema-agnostic was his idea), and to everyone at Zerodha who’s been using this and giving feedback. Thanks to @songxuanqing</a> for query variables and other contributors for docs and bug fixes. Demo</a> | Docs</a> | GitHub</a> | v1.0.0 Release</a> Fin! Fixing a CIBIL Score Disaster with AI 2025-12-04T00:00:00+00:00 About a month ago, I downloaded my CIBIL report expecting a routine check. Instead, I found loans from lenders I had never interacted with, written-off accounts, overdues from fintechs I had never installed, and even two-wheeler loan enquiries. I don’t even ride a bike. My credit score had collapsed to under 680. I stared at the report trying to understand how this could happen. The Root Cause: A Wrong Date of Birth#</a></h2> Buried in my profile section was the problem: my date of birth was wrong. Not a typo, but a completely different year. Because of this mismatch, CIBIL’s system had paired my PAN and mobile number with someone else’s DOB, effectively merging two individuals’ credit histories into one report. The accounts mapped to me included: Aditya Birla Capital: Short-term personal loan marked as doubtful/substandard</li> Clix Capital: A loan marked written-off (₹50,000+)</li> Poonawalla Fincorp: Personal loans with delayed payments</li> Ring / Kissht: Unsecured digital loans</li> InCred: Personal loan I never took</li> Dhani Loans: BNPL-style loan with unrecognized activity</li> Axio (Capital Float): Old consumer loan</li> KrazyBee: Various short-term loans</li> Transactree: Small-ticket personal loan</li> Multiple enquiries from HDFC, ICICI, IDFC First, Shriram Finance, and others</li> </ul> Some were written-off, others 90+ days overdue, others still active. On paper, I looked like a serial defaulter. Using AI to Understand the Problem#</a></h2> I opened ChatGPT and uploaded the entire PDF with a simple prompt: identify everything wrong in this report. Within minutes, it had mapped every suspicious account, flagged which ones didn’t match my history, highlighted the incorrect DOB, and explained why CIBIL systems mis-map accounts when demographic data is inconsistent. More usefully, it drafted formal dispute letters citing relevant RBI regulations and prepared lender-specific escalations with the right legal language. It felt like having a credit compliance team on demand. The Dispute Process#</a></h2> With the AI-drafted communications as a starting point, I sent disputes to CIBIL and direct emails to each lender. The key was being specific: every email included the CIBIL report control number, the exact account identifiers from the report, and references to specific RBI regulations. For example, when writing to Poonawalla Fincorp about a co-lending arrangement with Kissht, the email included: CIBIL Report details: Control Number: [REDACTED]</li> Downloaded on: [DATE]</li> Where your name appears: “POONAFIN – Personal Loan – Account No. [REDACTED]”</li> Delinquency trail in history: DPD values 35 / 62 / 93 / 124 during [MONTHS]</li> </ul> I reiterate that I have never applied for, signed, or availed any facility from Poonawalla/Kissht. This appears to be erroneous mapping / data contamination. </blockquote> The emails also cited the relevant regulations explicitly: Under Section 45-A(2) of the Credit Information Companies (Regulation) Act 2005 and Para 7.2.2 & 8.1.3 of the RBI Master Directions on Credit Information Companies (2021), please verify this record against your origination/KYC systems. If the record is not verifiable or was created with misused/incorrect KYC, immediately instruct TransUnion CIBIL to delete/correct the entry. </blockquote> This kind of precise, regulation-backed language gets results. Vague complaints are likely to get ignored or deprioritized. Specific complaints with control numbers, account IDs, and regulatory citations get escalated to teams that can actually fix things. For co-lending cases (common with fintechs like Kissht, Ring, etc.), I learned to CC both parties and explicitly request a “consolidated correction” so the entry gets fully removed rather than bouncing between two institutions. When initial responses were slow, I sent reminders that referenced the original complaint number and the 30-day statutory deadline: This is a reminder regarding my complaint Ref No. [REDACTED]. The acknowledgement stated that the issue would be resolved by [DATE], yet I have not received any confirmation. Failure to resolve within the statutory period will leave me with no option but to escalate to the RBI Integrated Ombudsman. </blockquote> CIBIL started closing disputes. One by one, accounts were removed. Eight fraudulent accounts were purged in the first wave. But there was a catch: even after the fraudulent accounts were removed, my DOB was still wrong. CIBIL kept closing my DOB correction disputes without actually fixing the underlying data. Their responses were templated and generic, treating it like a lender issue when DOB is actually a CIBIL demographic field that they control directly. This required escalating to the Nodal Officer with a sharper tone: My Date of Birth correction dispute has been closed twice, yet my DOB remains incorrect in every new CIBIL report. This is a CIBIL demographic field — it is not lender-controlled and should have been corrected immediately once KYC was submitted. Because of this incorrect DOB, my profile was wrongly merged with another individual’s data. Although many wrong accounts have been removed, the root cause remains uncorrected — the wrong DOB is still mapped, and therefore the risk of future wrongful linkages still exists. </blockquote> Only after escalating to the Nodal Officer did the DOB finally get corrected. Once that happened, the system stopped associating the other person’s accounts with my profile. It was an algorithmic identity collision, and fixing the DOB resolved it. The Outcome#</a></h2> My latest CIBIL report shows the correct date of birth, zero fraudulent loans, no written-off or overdue accounts, and a score back in a healthy range. Only my actual accounts remain. What I Learned#</a></h2> Credit bureaus are not infallible. A single incorrect demographic detail (in my case, a mismatched DOB) can cause wrong loan mappings, score drops, false delinquencies, and a complete distortion of your financial identity. The resolution required documentation, persistence with escalations, and an understanding of RBI regulations. AI made the last part significantly easier. Instead of spending hours researching dispute procedures and drafting formal letters, I could focus on gathering the right documents and following up with the right people. If you haven’t checked your CIBIL report recently, it’s worth verifying that your basic details are correct: DOB, PAN, address, mobile, email. One wrong field can create problems that take weeks to untangle. Fin! AI and Home-Cooked Software 2025-10-05T00:00:00+00:00 Everyone is worried that AI will replace programmers. They’re missing the real revolution: AI is turning everyone into one. I’ve been noticing a new pattern: people with deep domain knowledge but no coding experience are now building their own tools. Armed with AI assistants, they can create custom workflows in a matter of days, bypassing traditional development cycles. Are these solutions production-ready? Not even close. But they solve urgent, specific problems, and that’s what matters. Tasks that once required weeks of specialized training are quickly becoming weekend projects. This trend is happening even within the AI companies themselves. Anthropic, for example, shared how their own teams use Claude to accelerate their work. Crucially, this isn’t limited to developers. Their post details how non-technical staff now build their own solutions</a> and create custom automations, providing a powerful real-world example of this new paradigm. Home-Cooked Software#</a></h2> Why search for a generic tool when you can build exactly what you need? This question leads to what I call ‘home-cooked software’: small, personal applications we build for ourselves, tailored to our specific needs. Robin Sloan</a> beautifully describes building an app as making “a home-cooked meal,” while Maggie Appleton</a> writes about “barefoot developers” creating software outside traditional industry structures. What’s new isn’t the concept but the speed and accessibility. With AI, a custom export format, a specific workflow, or the perfect integration is now an afternoon’s work. We’re entering an unprecedented era where the barrier between wanting a tool and having it has nearly vanished. But let’s be clear: the journey from a prototype to a production-ready application is as challenging as ever. In my experience, an AI can churn out a first draft in a few hours, which gets you surprisingly far. But the devil is in the details, and the last stretch of the journey – handling edge cases, ensuring security, and debugging subtle issues – can stretch into weeks. This distinction is crucial. AI isn’t replacing programmers; it’s creating millions of people who can build simple tools. There’s a significant difference. The New Economics#</a></h2> AI is fundamentally reshaping the economics of building software. Before AI, even a simple tool required a significant time investment in learning programming basics, understanding frameworks, and debugging. Only tools with broad appeal or critical importance justified the effort. Now, that effort is measured in hours, not months, and the primary barrier is no longer technical knowledge, but imagination and a clear understanding of one’s own needs. This doesn’t apply to complex or security-critical systems, where deep expertise remains essential. But for the long tail of personal utilities, automation scripts, and custom workflows, the math has changed completely. I’m talking about solving all those minor irritations that pile up: the script to reformat a specific CSV export, the dashboard showing exactly the three metrics you care about, or a script that pulls data from a personal project management tool to sync with an obscure time-tracking app. These tools might be held together with digital duct tape, but they solve real problems for real people. And increasingly, that’s all that matters. The Hidden Costs#</a></h2> But this newfound capability isn’t free. It comes with what I call the “AI Tax”: a set of hidden costs that are rarely discussed. First, prompt engineering can be surprisingly time-consuming, especially for tasks of moderate complexity. While simple requests are often straightforward, anything more nuanced can become an iterative dialogue. You prompt, the AI generates a flawed output, you clarify the requirements, and it returns a new version that misses a different detail. It’s a classic 80/20 scenario: you get 80% of the way there with a simple prompt, but achieving the final 20% of correctness requires a disproportionate amount of effort in refining, correcting, and clarifying your intent to the model. Second, there’s the verification burden. Every line of AI-generated code is a plausible-looking liability. It may pass basic tests, only to fail spectacularly in production with an edge case you never considered. AI learned from the public internet, which means it absorbed all the bad code along with the good. SQL injection vulnerabilities, hardcoded secrets, race conditions—an AI will happily generate them all with complete confidence. Perhaps the most frustrating aspect is “hallucination debugging”: the uniquely modern challenge of troubleshooting plausible-looking code that relies on APIs or methods that simply don’t exist. Your codebase becomes a patchwork of different AI-generated styles and patterns. Six months later, it’s an archaeological exercise to determine which parts you wrote and which parts an AI contributed. But the most significant danger is that AI enables you to build systems you don’t fundamentally understand. When that system inevitably breaks, you lack the foundational knowledge to debug it effectively. Building for One#</a></h2> Despite these challenges, there’s something profoundly liberating about building software just for yourself. Instead of just sketching out ideas, I’ve started building these small, specific tools. For this blog, I wanted a simple lightbox for images; instead of pulling in a heavy external library, I had Claude write a 50-line JavaScript snippet that did exactly what I needed. I built a simple, single-page compound interest calculator</a> tailored for my own financial planning. To save myself from boilerplate at work, I created prom2grafana</a>, a tool that uses an LLM to convert Prometheus metrics into Grafana dashboards. Ten years ago, I might have thought about generalizing these tools, making them useful for others, perhaps even starting an open source project. Today? I just want a tool that works exactly how I think. I don’t need to handle anyone else’s edge cases or preferences. Home-cooked software doesn’t need product-market fit—it just needs to fit you. We’re witnessing the emergence of a new software layer. At the base are the professionally-built, robust systems that power our world: databases, operating systems, and rock-solid frameworks. In the middle are commercial applications built for broad audiences. And at the top, a new layer is forming: millions of tiny, personal tools that solve individual problems in highly specific ways. This top layer is messy, fragile, and often incomprehensible to anyone but its creator. It’s also incredibly empowering. Creating simple software is becoming as accessible as writing. And just as most writing isn’t professional literature, most of this new software won’t be professional-grade. That’s not just okay; it’s the point. The implications are profound. Subject-matter experts can now solve their own problems without waiting for engineering resources, and tools can be hyper-personalized to a degree that is impossible for commercial software. This unlocks a wave of creativity, completely unconstrained by the need to generalize or find a market. Yes, there are legitimate concerns. Security is a real risk, though the profile changes when a tool runs locally on personal data with no external access. We’re creating personal technical debt, but when a personal tool breaks, the owner is the only one affected. They can choose to fix it, rebuild it, or abandon it without impacting anyone else. Organizations, on the other hand, will soon have to grapple with the proliferation of incompatible personal tools and establish new patterns for managing them. But these challenges pale in comparison to the opportunities. The barrier between user and creator is dissolving. We’re entering the age of home-cooked software, where building your own tool is becoming as natural as cooking your own meal. The kitchen is open. What will you cook? State of My Homelab 2025 2025-10-04T00:00:00+00:00 Introduction#</a></h2> For the past five years, I have maintained a homelab in various configurations. This journey has served as a practical exploration of different technologies, from Raspberry Pi clusters running K3s</a> to a hybrid cloud setup</a> and eventually a cloud-based Nomad setup</a>. Each iteration provided valuable lessons, consistently highlighting the operational benefits of simplicity. This article details the current state of my homelab. A primary motivation for this build was to dip my toes into “actual” homelabbing—that is, maintaining a physical server at home. The main design goal was to build a dedicated, reliable, and performant server that is easy to maintain. This led me to move away from complex container orchestrators like Kubernetes in favor of a more straightforward Docker Compose workflow. I will cover the hardware build, software architecture, and the rationale behind the key decisions. Hardware Configuration#</a></h2> After considerable research, I selected components to balance performance, power efficiency, and cost. The server is designed for 24/7 operation in a home environment, making noise and power consumption important considerations. The Build#</a></h3> Component</th> Choice</th> Price</th></tr></thead> CPU</td> AMD Ryzen 5 7600X (6-core, 4.7 GHz)</td> $167.58</td></tr> CPU Cooler</td> ARCTIC Liquid Freezer III Pro 360</td> $89.99</td></tr> Motherboard</td> MSI B650M Gaming Plus WiFi</td> $225.83</td></tr> RAM</td> Kingston FURY Beast 32GB DDR5-6000</td> $136.99</td></tr> Boot Drive</td> WD Blue SN580 500GB NVMe</td> $88.76</td></tr> Storage 1</td> WD Red Plus 4TB (5400 RPM)</td> $99.99</td></tr> Storage 2</td> Seagate IronWolf Pro 4TB (7200 RPM)</td> $150.00</td></tr> Case</td> ASUS Prime AP201 MicroATX</td> $89.99</td></tr> PSU</td> Corsair SF750 (80+ Platinum)</td> $169.99</td></tr> Total</td> </td> $1,219.12</td></tr> </tbody></table> </a> </a> </a> </div> Component Rationale#</a></h3> CPU: The Ryzen 5 7600X provides a strong price-to-performance ratio. Its 6 cores offer ample headroom for concurrent containerized workloads and future experimentation.</li> Storage: The boot drive is a 500GB NVMe for fast OS and application performance. The primary storage consists of two 4TB HDDs in a BTRFS RAID 1 configuration. To mitigate the risk of correlated failures, I chose drives from different manufacturers (WD and Seagate) purchased at different times.</li> RAM: 32GB of DDR5-6000 provides sufficient memory for a growing number of services without risking contention.</li> Case & PSU: The ASUS Prime AP201 is a compact MicroATX case with a clean aesthetic suitable for a home office. The Corsair SF750 (80+ Platinum) PSU was chosen for its efficiency and to provide capacity for a future GPU for local LLM or transcoding workloads.</li> </ul> System Architecture & Deployment#</a></h2> My previous setups involved Kubernetes and Nomad, but the operational overhead proved unnecessary for my use case. I have since standardized on a Git-based, Docker Compose workflow that prioritizes simplicity and transparency. Directory Structure and “Stacks”#</a></h3> The core of the system is a Git repository that holds all configurations. Each service is defined as a self-contained “stack” in its own directory. The structure is organized by machine, making it easy to manage multiple environments: homelab/ ├── deploy.sh # Main deployment script ├── justfile # Task runner for common commands └── machines/ ├── floyd-homelab-1/ # Primary home server │ ├── config.sh # SSH and deployment settings │ └── stacks/ │ ├── immich/ │ │ └── docker-compose.yml │ └── paperless/ │ └── docker-compose.yml └── floyd-pub-1/ # Public-facing VPS ├── config.sh └── stacks/ ├── caddy/ └── ntfy/</code></pre> This modular approach allows me to manage each application’s configuration, including its docker-compose.yml</code> and any related files, as an independent unit. Deployment Workflow#</a></h3> Deployments are handled by a custom deploy.sh</code> script, with a justfile</code> providing a convenient command-runner interface. The process is fundamentally simple: Sync: rsync</code> copies the specified stack’s directory from the local Git repository to a REMOTE_BASE_PATH</code> (e.g., /opt/homelab</code>) on the target machine.</li> Execute: ssh</code> runs the appropriate docker compose</code> command on the remote machine.</li> </ol> Each machine’s connection settings (SSH_HOST</code>, SSH_USER</code>, REMOTE_BASE_PATH</code>) are defined in its machines/<name>/config.sh</code> file. This file can also contain pre_deploy</code> and post_deploy</code> hooks for custom actions. The justfile</code> makes daily operations trivial: # Deploy a single stack to a machine just deploy-stack floyd-homelab-1 immich # View the logs for a stack just logs floyd-homelab-1 immich # Test a deployment without making changes just dry-run floyd-homelab-1</code></pre> This system provides fine-grained control over deployments, with support for actions like up</code>, down</code>, restart</code>, pull</code>, and recreate</code> (which also removes persistent volumes). Container & Configuration Patterns#</a></h3> To keep the system consistent, I follow a few key patterns: Data Persistence: Instead of using Docker named volumes, I use host bind mounts. All persistent data for a service is stored in a dedicated directory on the host, typically /data/<service-name></code>. This makes backups and data management more transparent.</li> Reverse Proxy Network: The Caddy stack defines a shared Docker network called public_proxy</code>. Other stacks that need to be exposed to the internet are configured to join this network. This allows Caddy to discover and proxy them without exposing their ports on the host machine. I have written about this pattern in detail in a previous post</a>.</li> Port Exposure: Services behind the reverse proxy use the expose</code> directive in their docker-compose.yml</code> to make ports available to Caddy within the Docker network. I avoid binding ports directly with ports</code> unless absolutely necessary.</li> </ul> Multi-Machine Topology#</a></h2> The homelab comprises three distinct machines to provide isolation and redundancy. floyd-homelab-1 (Primary Server): The core of the homelab, running on the AMD hardware detailed above. It runs data-intensive personal services (e.g., Immich, Paperless-ngx) and is accessible only via the Tailscale network.</li> floyd-pub-1 (Public VPS): A small cloud VPS that hosts public-facing services requiring high availability, such as DNS utilities, analytics, and notification relays.</li> floyd-monitor-public (Monitoring VPS): A small Hetzner VM running Gatus for health checks. Its independence ensures that I am alerted if the primary homelab or home network goes offline.</li> </ul> This distributed setup isolates my home network from the public internet and ensures that critical public services remain online even if the home server is down for maintenance. Hosted Services#</a></h2> The following is a breakdown of the services, or “stacks,” running on each machine. A few key services that are central to the homelab are detailed further in the next section. floyd-homelab-1 (Primary Server)#</a></h3> Actual</a>: A local-first personal finance and budgeting tool.</li> Caddy</a>: A powerful, enterprise-ready, open source web server with automatic HTTPS.</li> Gitea</a>: A Git service for personal projects.</li> Glance</a>: A dashboard for viewing all my feeds and data in one place.</li> Immich</a>: A photo and video backup solution, directly from my mobile phone.</li> Karakeep</a>: An app for bookmarking everything, with AI-based tagging and full-text search.</li> Owntracks</a>: A private location tracker for recording my own location data.</li> Paperless-ngx</a>: A document management system that transforms physical documents into a searchable online archive.</li> Silverbullet</a>: A Markdown-based knowledge management and note-taking tool.</li> </ul> floyd-monitor-public (Monitoring VPS)#</a></h3> Caddy</a>: Reverse proxy for the services on this node.</li> </ul> floyd-pub-1 (Public VPS)#</a></h3> Beszel-agent</a>: The agent for the Beszel monitoring platform.</li> Caddy</a>: Reverse proxy for the services on this node.</li> Cloak</a>: A service to securely share sensitive text with others.</li> Doggo</a>: A command-line DNS Client for Humans, written in Golang.</li> Ntfy</a>: A self-hosted push notification service.</li> prom2grafana</a>: A tool to convert Prometheus metrics to Grafana dashboards and alert rules using AI.</li> Umami</a>: A simple, fast, privacy-focused alternative to Google Analytics.</li> </ul> Service Highlights#</a></h2> Technitium: A Powerful DNS Server#</a></h3> I came across Technitium DNS after seeing a recommendation from @oddtazz</a>, and it has been a revelation. For anyone who wants more than just basic ad blocking from their DNS server, it’s a game-changer. It serves as both a recursive and authoritative server, meaning I don’t need a separate tool like unbound</code> to resolve from root hints. The level of configuration is incredible—from DNSSEC, custom zones, and SOA records to fine-grained caching control. The UI is a bit dated, but that’s a minor point for me given the raw power it provides. It is a vastly underrated tool for any homelabber who wants to go beyond Pi-hole or AdGuard Home. </a> </div> Beszel: Lightweight Monitoring#</a></h3> For a long time, I felt that monitoring a homelab meant spinning up a full Prometheus and Grafana stack. Beszel is the perfect antidote to that complexity. It provides exactly what I need for basic node monitoring—CPU, memory, disk, and network usage—in a simple, lightweight package. It’s incredibly easy to set up and provides a clean, real-time view of my servers without the overhead of a more complex system. For a simple homelab monitoring setup, it’s hard to beat. </a> </div> Gatus: External Health Checks#</a></h3> While Beszel monitors the servers from the inside, Gatus watches them from the outside. Running on an independent Hetzner VM, its job is to ensure my services are reachable from the public internet. It validates HTTP status codes, response times, and more. This separation is crucial; if my entire home network goes down, Gatus is still online to send an alert to my phone. It’s the final piece of the puzzle for robust monitoring, ensuring I know when things are broken even if the monitoring service itself is part of the outage. </a> </div> Storage and Backup Strategy#</a></h2> Data integrity and recoverability are critical. My strategy is built on layers of redundancy and encryption. Storage: BTRFS RAID 1 + LUKS Encryption#</a></h3> I chose BTRFS for its modern features: Checksumming: Protects against silent data corruption.</li> Copy-on-Write: Enables instantaneous, low-cost snapshots.</li> Transparent Compression: zstd</code> compression saves space without significant performance overhead.</li> </ul> The two 4TB drives are mirrored in a RAID 1 array, providing redundancy against a single drive failure. The entire array is encrypted using LUKS2, with the key stored on the boot SSD for automatic mounting. This protects data at rest in case of physical theft or drive disposal. Mount options in /etc/fstab</code>: /dev/mapper/crypt-sda /mnt/storage btrfs defaults,noatime,compress=zstd 0 2</code></pre>Backup: Restic + Cloudflare R2#</a></h3> RAID does not protect against accidental deletion, file corruption, or catastrophic failure. My backup strategy follows the 3-2-1 rule. Daily, automated backups are managed by systemd timers running restic</code>. Backups are encrypted and sent to Cloudflare R2, providing an off-site copy. R2 was chosen for its zero-cost egress, which is a significant advantage for restores. The backup script covers critical application data and the Docker Compose configurations: BACKUP_PATHS=( "/mnt/storage" # All application data "/home/karan/stacks" # Docker Compose configs )</code></pre> Each backup run reports its status to a healthchecks.io</a> endpoint, which sends a push notification on failure. I must appreciate its generous free tier, which is more than sufficient for my needs. </a> </div> Conclusion#</a></h2> This homelab represents a shift in philosophy from exploring complexity to valuing simplicity and reliability. The upfront hardware investment of ~$1,200 is offset by eliminating recurring cloud hosting costs and providing complete control over my data and services. For those considering a homelab, my primary recommendation is to start with a simple, well-understood foundation. A reliable machine with a solid backup strategy is more valuable than a complex, hard-to-maintain cluster. The goal is to build a system that serves your needs, not one that you serve. TIL: WireGuard's Misleading "No Route to Host" Error 2025-07-30T00:00:00+00:00 I recently spent some time debugging a WireGuard tunnel that was acting weird. The handshake was successful, pings worked perfectly, but any TCP connection failed with connect: no route to host</code>. Classic misleading error message. The routing was fine. The Setup#</a></h2> Server with a public IP running WireGuard (wg0</code>) with IP 10.100.0.1/24</code>. Client connects and gets assigned 10.100.0.2/32</code>. I wanted to proxy TCP traffic from the server to a service running on the client at 10.100.0.2:7778</code>. The Investigation#</a></h2> Diagnostics showed contradictory results: Routing worked fine: Server routing table correctly directed 10.100.0.0/24</code> traffic to wg0</code>. Pings were successful: # On the server $ ping -c 3 10.100.0.2 PING 10.100.0.2 (10.100.0.2) 56(84) bytes of data. 64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=150 ms ...</code></pre> TCP failed immediately: # On the server $ curl -v http://10.100.0.2:7778 * Trying 10.100.0.2:7778... * connect to 10.100.0.2 port 7778 from 10.100.0.1 port 59812 failed: No route to host</code></pre> The key insight: ICMP</code> was being treated differently than TCP</code>. This pointed to a firewall issue, not routing. The “no route to host” error was the kernel interpreting an ICMP “Destination Unreachable” message from the remote peer. But when I ran tcpdump</code> on the client, things got stranger: # On the client $ sudo tcpdump -i any -n 'host 10.100.0.1' # Output when the server tries to connect 17:36:03.043147 wg0 In IP 10.100.0.1.14808 > 10.100.0.2.7778: Flags [S], seq 324784341, win 42780, ...</code></pre> The TCP SYN</code> packet arrived successfully through wg0</code>. But no response. No SYN-ACK</code> (success), no ICMP</code> error (rejection). The packet was being silently dropped. The Culprit: firewalld</code>#</a></h2> The client was running Arch Linux with firewalld</code>. My mistake was trying to manage firewall rules with iptables</code> commands in the WireGuard PostUp</code> script. While iptables</code> was installed, firewalld</code> was the active manager, using nftables</code> as its backend. When a new interface like wg0</code> comes up, firewalld</code> needs to know which “zone” it belongs to. If unassigned, it gets handled by a restrictive default policy that silently DROP</code>s unsolicited TCP packets while allowing ICMP (pings). The Fix#</a></h2> Don’t add iptables</code> rules. Just assign the WireGuard interface to the right firewalld</code> zone. For internal tunnels, trusted</code> works well. On the client: sudo firewall-cmd --permanent --zone=trusted --add-interface=wg0 sudo firewall-cmd --reload</code></pre> TCP connections worked instantly after this. TL;DR: If WireGuard pings work but TCP fails with “no route to host”, it’s probably a client firewall issue. On firewalld</code> systems, assign the WireGuard interface to the right zone instead of messing with iptables</code>. Fin! Announcing Logchef 2025-04-27T00:00:00+00:00 So, for the last 3-4 months, I’ve been busy building Logchef</a>. This tool basically grew straight out of my day job managing logs at Zerodha</a>, where I’ve been managing logs for almost half a decade. I wanted to share a bit about how Logchef came to be. Like many, we journeyed through the complexities of ELK (a management nightmare) and found its OSS fork, OpenSearch, didn’t quite hit the mark for us either. We eventually found solid ground with Clickhouse, as detailed on our tech blog: Logging at Zerodha</a>. Challenges Faced with Metabase#</a></h2> However, as I noted in that post, while Metabase served us well for analytics, it wasn’t the ideal UI specifically tailored for log analysis against Clickhouse: “While Metabase has served us well so far, there is certainly room for improvement, especially regarding a more tailored UI for Clickhouse… we plan to continue exploring potential solutions.” </blockquote> Here’s a distilled version of the common pain points we experienced: Ad-hoc Querying Was Painful: Writing raw Clickhouse SQL in Metabase for quick log searches felt cumbersome and slow. Even modifying existing complex query templates was error-prone – a tiny syntax mistake could lead to minutes spent debugging the query itself, especially stressful during production incidents.</li> Disconnect Between Visualization and Raw Logs: A common workflow is to visualize trends (e.g., errors over time) and then drill down into the specific logs causing those trends. In Metabase, this often meant writing two separate queries – one for aggregation/visualization and another (often rewritten from scratch) just to see the raw log lines. Metabase’s row limits (around 2k) further complicated viewing the full context of raw logs after filtering. The intuitive “slice and drill-down” experience many log tools offer was missing.</li> UI/UX Annoyances: Several smaller but cumulative issues added friction: difficulty selecting precise time ranges like “last 6 hours,” viewing logs immediately surrounding a relevant event, columns getting truncated (...</code>), and limited timestamp precision display in results. Though there are some workarounds, they often felt like band-aids rather than solutions.</li> </ul> TL;DR: Metabase interface wasn’t optimized for the specific task of log exploration. Debugging sessions that should have taken minutes were stretching significantly longer. Querying and exploring logs felt clunkier than it needed to be. And one fine day, I decided to stop just wishing for a better tool and start building one: Logchef#</a></h2> When I first started prototyping, I kept the scope pretty tight: just build a viewer for the standard OTEL schema</a>. OTEL’s flexible enough, but a quick chat with Kailash</a> sparked what turned out to be a game-changing idea: make Logchef schema-agnostic. And that really became the core concept. Basically, Logchef lets you connect it straight to your existing Clickhouse log tables, no matter their structure. All it really needs is a timestamp field (DateTime</code> or DateTime64</code>). Bring your own custom schemas, stick with the OTEL standard, or even adapt it to your own needs. Logchef doesn’t force you into a specific format. From what I’ve seen, not many tools offer this kind of plug-and-play flexibility with existing tables today. Logchef is designed as a specialized query and visualization layer sitting on top of Clickhouse. Logchef intentionally excludes log collection and ingestion. Why reinvent the wheel when excellent tools like Vector, Fluentbit, Filebeat, etc., already handle this reliably? Logchef focuses purely on exploring the logs once it’s in Clickhouse. Stack#</a></h2> Backend: Written in Go for performance and concurrency.</li> Metadata Storage: Uses SQLite for lightweight management of users, teams, Clickhouse source connections, and query collections. It’s simple and perfectly suited for this kind of a metadata store.</li> Frontend: An interactive log viewer with Vue.js and styled with shadcn/ui and Tailwind CSS. I also implemented a simple search syntax for common filtering tasks (e.g., status=200 and path~"/api/"</code>). This involved writing a tokenizer and parser that translates this syntax into efficient ClickHouse SQL conditions optimised for querying logs. Building this parser, validator, and integrating it smoothly with the Monaco editor for syntax highlighting was a significant effort but quite happy with the end result.</li> </ul> Setting Up the Public Demo (demo.logchef.app</code>)#</a></h2> I wanted a public demo instance</a> so people could try Logchef easily. Setting this up involved a few specific tweaks compared to a standard deployment, all managed within the Docker Compose setup: Generating Dummy Data: A log viewer isn’t much use without logs! Instead of ingesting real data, I configured vector</code> using its demo_logs</code> source type. This continuously generates realistic-looking syslog and HTTP access logs and pushes them into the demo Clickhouse instance (syslogs</code> and http_logs</code> tables). It gives users immediate data to query without any setup on their part. # vector.toml snippet [sources.generate_syslog] type = "demo_logs" format = "syslog" interval = 0.3 # Generate logs frequently [sinks.clickhouse_syslog] # ... config to send to Clickhouse ... table = "syslogs"</code></pre></li> Securing Admin Endpoints (Demo Mode): Since this is a public, shared instance, I wanted to prevent users from making potentially disruptive changes via the API (like deleting sources or teams). I used Caddy</code> as the reverse proxy and configured it to intercept requests to admin-specific API routes (like /api/v1/admin/*</code>) and block any method other than GET</code>. If someone tries a POST</code>, PUT</code>, or DELETE</code> to these endpoints, Caddy returns a 403 Forbidden</code> directly. This keeps the demo environment stable. Caddyfile snippet (conceptual) handle /api/v1/admin/* { @block_methods method POST PUT DELETE PATCH respond @block_methods `{"error":"Operation not permitted in demo mode"}` 403 reverse_proxy logchef:8125 # Forward GET requests }</code></pre></li> Improving Demo Login UX: Logchef uses OIDC for authentication. For the demo, I’m running Dex</code> as the OIDC provider. To make it completely frictionless for users, I didn’t want them needing to sign up or guess credentials. I simply customized Dex’s theme template for the login page to explicitly display the static demo username (demo@logchef.app</code>) and password (password</code>) right there. It’s a small UX tweak (again, thanks to Kailash for the idea!), but it means anyone landing on the demo can log in instantly.  <div class="dex-info-box"> <strong>Demo Credentials:</strong><br> Username: <code>demo@logchef.app</code><br> Password: <code>password</code> </div> <input ... value="demo@logchef.app" .../> <input ... type="password" value="password" .../></code></pre></li> </ol> What’s Next?#</a></h2> Logchef is already being used internally, but the journey towards a full v1.0 release continues this year. The roadmap</a> includes exciting additions like: Alerting: Trigger notifications based on query results.</li> Live Tail Logs: Stream logs in real-time.</li> Enhanced Dashboarding: More powerful visualization capabilities.</li> </ul> Logchef is open source (AGPLv3), and community involvement is welcomed. You can check out the Demo</a> or view the code on GitHub</a>. If you have more ideas or features you’d like to see, please reach out on GitHub</a> or email me</a>! I’m always open to suggestions and feedback. Honestly, building Logchef has been incredibly rewarding. It started as a way to fix something that bugged me (and others!), and seeing it turn into a tool I’m genuinely excited about feels great. I couldn’t have done it alone, though. I’m really grateful to my friends and colleagues who jumped in with feedback along the way. Huge thanks to Kailash</a> for the constant support and encouragement, and to Vivek</a>, Sarat</a>, and Rohan</a> for testing the early builds and offering great suggestions. Finally, a big thank you to my wife, who patiently endured my late-night coding sessions. Her support means the world to me <3 Fin! Trying out NixOS 2025-03-23T10:15:00+05:30 I’ve been introduced to Nix by my colleagues at work. Being a Linux user for over a decade and a serial distro hopper, I was curious to learn more about it. I’d seen Nix mentioned before, but the comments about its steep learning curve made me wonder if the effort was worth it. I decided to give it a try by reading this excellent beginner’s guide</a> however got bored very quickly and decided to “learn on the fly”. I spun up a VM in my homelab to install NixOS using their official GUI installer image</a>. Installation & First Impressions#</a></h2> The installation was as straightforward as any other Linux distro. NixOS is a declarative operating system that leverages the Nix functional package manager and a rich ecosystem of Nix packages. The flexibility is mind-blowing: you can configure everything—from user accounts and SSH keys to $SHELL</code> config and plugins entirely through code. Once installed, the first place you’d want to poke around is the /etc/nixos</code> directory, which contains two essential configuration files: hardware-configuration.nix</code>: Generated during installation (or regenerated with commands like nixos-generate-config</code>), it has hardware-specific details such as filesystem mount points, disk configurations, kernel modules etc. See an example file here</a>. </li> configuration.nix</code>: This is the most important file you want to start editing with. Here you define system-wide settings like timezone, locale, user accounts, and networking. Everything is declared in one place, making your system’s state reproducible. </li> </ul> First Configuration Changes#</a></h3> When I opened the terminal, I immediately noticed that vim</code> wasn’t installed. So, I updated my configuration.nix</code> to include the packages I needed: environment.systemPackages = with pkgs; [ git vim ];</code></pre> After saving, I ran: sudo nixos-rebuild switch</code></pre> This rebuilds the system using the new declarative configuration. Version Control & Flakes#</a></h2> Next, I wanted to set up version control for my Nix configurations. The key takeaway is that while the system’s state is revertable in NixOS, your personal data (which includes configuration.nix</code>) isn’t automatically backed up. You must manage your own version history for your Nix configs. Since I was tweaking with no knowledge of Nix, having a version history was crucial. I moved my /etc/nixos</code> configs to ~/Code/nixos-configs</code> and initialized a Git repository: # Create repo in home directory (better than root-owned /etc/nixos) mkdir ~/nixos-config cp -r /etc/nixos/* ~/nixos-config/ cd ~/nixos-config # Initialize Git git init git add . git commit -m "Initial NixOS configuration" # Add GitHub remote git remote add origin https://github.com/username/nixos-config.git git push -u origin main</code></pre> Here’s how flake.nix</code> looks: { description = "NixOS configuration for Karan's homelab, servers, and personal dev machines"; inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; # Add agenix as an input agenix = { url = "github:ryantm/agenix"; inputs.nixpkgs.follows = "nixpkgs"; }; # Optionally add other inputs like home-manager # home-manager = { # url = "github:nix-community/home-manager/release-24.11"; # inputs.nixpkgs.follows = "nixpkgs"; # }; }; outputs = { self, agenix, nixpkgs, ... }@inputs: { # Make agenix available as a package packages.x86_64-linux.agenix = agenix.packages.x86_64-linux.default; nixosConfigurations.work = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ ./configuration.nix agenix.nixosModules.default # Add agenix module ]; }; }; }</code></pre>A Note on Flakes#</a></h3> Flakes are an experimental (although widely adopted in the community) feature in Nix that bring reproducibility, composability, and a standardized structure to your configurations and package definitions. They allow you to declare all inputs (like nixpkgs, home-manager, or other repositories) and outputs (such as system configurations, packages, or development shells) in a single file. Flakes also create a lock file (flake.lock</code>) that pins your dependencies to specific revisions, ensuring that your builds remain reproducible over time. I learned the hard way that—even for local configurations you must commit your files. Otherwise, you may see errors like: path '/nix/store/...source/flake.nix' does not exist</code></pre> Even if you’re using local paths and have no intention to push to git</code>, you still need git init</code> && git add</code> for flakes to work. From whatever google-fu I did, it seems this requirement is to ensure that flakes can reliably reference the exact content in your configuration. I am sure there might be good reasons for it (as I said before, I’ve skipped RTFMing altogether ^_^), but atleast the errors can be more verbose/helpful. And why I skipped docs: Remember, we’re on a mission to get things up and running with Nix and then later spend time about reading their internals if it actually proves to be a valuable experiment. Switching Channels#</a></h2> While installing packages, I noticed some packages were quite outdated. That’s when I learned about NixOS channels. Think of channels as analogous to LTS releases. For faster updates, you can switch to the unstable</code> channel. Although the name sounds intimidating, it simply means you’ll receive more frequent package updates. To do this, you can edit your flake.nix</code> and switch the URL to an unstable channel: inputs = { - nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11"; # Stable channel + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; # Unstable channel };</code></pre>Firmware Updates#</a></h2> After setting up packages, it was time to configure firmware updates using fwupd—essential</code> for keeping your hardware up to date. I asked Claude to help me for a quick setup. Here’s what I did: { config, pkgs, ... }: { services.fwupd.enable = true; }</code></pre> Then run a rebuild: sudo nixos-rebuild switch</code></pre> Once enabled, you can use the fwupdmgr</code> command-line tool to manage firmware updates: # Refresh metadata and check for available updates fwupdmgr refresh fwupdmgr get-updates # Install available firmware updates fwupdmgr update</code></pre>Fine Tuning#</a></h2> I also tweaked some settings for the Nix package manager to optimize builds, caching, and overall performance. Here’s a snippet from my configuration: # Nix package manager optimizations nix = { settings = { # Optimize store to remove duplicate files auto-optimise-store = true; # Allow building multiple derivations in parallel max-jobs = "auto"; # Number of parallel build tasks per job cores = 0; # 0 means use all available cores # Use the binary cache aggressively substituters = [ "<https://cache.nixos.org>" "<https://nix-community.cachix.org>" "<https://nixpkgs-wayland.cachix.org>" ]; # Optimize fetching from GitHub connect-timeout = 5; # Prevent unneeded rebuilds commit-lockfile-summary = "Update flake.lock"; }; # Garbage collection settings gc = { automatic = true; dates = "weekly"; options = "--delete-older-than 30d"; }; # Optimize builds using different build cores buildCores = 0; # 0 means use all available cores # Enable flakes and modern Nix command features extraOptions = '' experimental-features = nix-command flakes warn-dirty = false keep-going = true log-lines = 20 ''; };</code></pre>Escape Hatches#</a></h2> So far things seems all rosy. Within just spending a couple of minutes - I had a perfectly working machine for myself - and the best part - all reproducible with a single command. I was starting to see why people who use NixOS preach about it so much. However, not everything is smooth when you deviate from the happy path. For instance, I use Aider</a> for LLM assisted programming, but the version on Nixpkgs was about three minor versions</a> behind. Typically for any other software, I wouldn’t have cared so much - however with these LLM tools, a lot changes rapidly and I didn’t want to stay behind. Besides, it seemed like a fun exercise in getting my hands dirty by installing a Python package on NixOS which turned out to be quite tricky because Nix is absurdly obsessive about fully isolated builds. Here’s an example flake that I used for attempting to install Aider with uv</code> in a dev shell (which didn’t work btw): { description = "Aider development environment"; inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; flake-utils.url = "github:numtide/flake-utils"; }; outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system: let pkgs = nixpkgs.legacyPackages.${system}; in { devShell = pkgs.mkShell { buildInputs = with pkgs; [ python312 uv ]; shellHook = '' export PATH="$HOME/.local/bin:$PATH" ''; }; } ); }</code></pre> Entering the dev shell with nix develop</code> and installing Aider with uv</code>: uv tool install --force --python python3.12 aider-chat@latest</code></pre> However, I ran into this error: "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/litellm_core_uti ls/llm_cost_calc/utils.py", line 9, in <module> from litellm.utils import get_model_info File "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/litellm/utils.py", line 53, in <module> from tokenizers import Tokenizer File "/home/karan/.local/share/uv/tools/aider-chat/lib/python3.12/site-packages/tokenizers/__init__.py", line 78, in <module> from .tokenizers import ( ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory</code></pre> The error indicated that Aider was missing a required dependency libstdc++.so.6</code> which is a part of the C++ standard library needed by the tokenizers package. To fix this, I added stdenv.cc.cc.lib</code> (and even gcc</code> to be on the safer side) to my buildInputs</code>. This is because while uv</code> installs Python packages, it doesn’t handle system-level dependencies. In a Nix environment, every dependency, including system libraries, must be explicitly specified. Frankly, Python’s packaging ecosystem is still a mess. Although tools like uv</code> help, achieving a completely isolated build, especially when shared libraries are involved is challenging. I wish the Python community would put more effort into resolving these issues. While I was able to make aider</code> work by explicitly adding all the dependencies, I faced another outdate package: code-cursor</code>. Since this is a full blown electron app, I didn’t wish to package this myself. After some frustration, I tried using Distrobox</a> as recommended by a colleague</a>. Distrobox lets you run containers that feel almost like a native OS by managing user IDs, host mounts, network interfaces, and more. I used an Arch Linux image, installed cursor-bin</code> from the AUR, and everything worked fine. Well mostly: Fonts were missing. So, if I want to use custom fonts in my IDE - I need to have them installed in the container as well.</li> Since my fish</code> shell config had export EDITOR=nvim</code>, I had to install neovim</code> in the container as well, otherwise, I’d get an error when trying to git commit</code> etc. There’s an option to customise the shell</a> in distrobox, but for whatever reason (that I didn’t want to debug), it didn’t work for me.</li> </ul> Yet, something still felt off. The whole point of using NixOS is to achieve a fully declarative and reproducible setup. Resorting to an escape hatch like Distrobox undermines that goal. So I was very conflicted about this. I’m sure there’s a better way to handle these situations, and I should probably read the docs to find out. Final Thoughts#</a></h2> I’m definitely sold on running NixOS, especially when managing multiple systems. With a single declarative file (configuration.nix</code>), duplicating your setup across machines becomes effortless. No more “documenting” (or rather forgetting to document and keeping it updated)- as the config is the single source of truth. Fun fact: I even messed up my NixOS build by misconfiguring the hardware-configuration.nix</code>, and my system became unusable even after a reboot, it couldn’t mount the filesystem on the correct device. In other distros, that would have sent me into panic mode, but with NixOS, all I had to do was revert to the previous working state, and everything was fine. That was so cool! I’m definitely considering moving my homelab to NixOS in the coming few days because I honestly see the value for a server setup. I often set up my personal server and then forget everything I’ve done and I’m always scared of touching or creating a new server from scratch. I even created a small shell script installer</a> to help me for getting a base system ready. But like this shell script or even tools such as Ansible - they are all idempotent in nature. However in Nix, if I remove a certain piece from the configuration, there isn’t a trace of it left on the system. That makes it truly declarative and reproducible - unlike Ansible where you can still have some parts of the old setup. However, for my primary machine at work, I’ll wait on the sidelines until the packages I depend on resolve their dependency issues and I get a chance to read up more on the escape hatches I tried to see if there’s a more streamlined way of doing things. I might be missing a lot of fundamental details since I skipped the docs entirely to get my hands dirty. But now that I see the value of a declarative system and especially how easy it is to roll back the machine to a previously known good state, I’m motivated to read up more on this and might post an update to this blog. Fin! Automating Badminton Game Alerts 2025-03-04T00:10:55+05:30 I’ve been playing badminton more regularly since the start of 2025 - almost 4-5 days a week. I recently moved to a new part of the city, which meant I couldn’t play with my old friends anymore. PlayO</a> has been super helpful for finding games with new people. On PlayO, a host creates a game and up to 6 people can join one court for a one-hour badminton doubles session. However, on hectic days I would often forget to check for badminton games, only to find them fully booked later. I wanted to automate this process by creating a small script that would send me scheduled alerts about today’s game availability, allowing me to book slots before they filled up. I drew inspiration from Matt’s</a> post where he did something similar. Thankfully, PlayO has a public API endpoint to retrieve a list of available games: https://api.playo.io/activity-public/list/location</code>. You can send a POST</code> request to this URL with these parameters for filtering: { "lat": 12.9783692, "lng": 77.6408356, "cityRadius": 5, "gameTimeActivities": false, "page": 0, "lastId": "", "sportId": ["SP5"], "booking": false, "date": ["2025-03-04T11:03:17.260Z"] }</code></pre> It returns a list of activities matching these filters. One such activity looks like: { "userInfo": [ { "profilePicUrl": "https://playov2.gumlet.io/profiles/redacted.511716.jpg", "fName": "Redacted", "lName": "", "karma": 2800 }, { "profilePicUrl": "https://playov2.gumlet.io/profiles/redacted-redacted.png", "fName": "redacted", "lName": "N", "karma": 499 } ], "isPlayoGame": false, "skill": "Intermediate & above", "sportName": "Badminton", "shortListed": false, "joineeList": [ "7f3cf298-3324-4fc2-96ad-b0f00093cd8f", "250572a2-555d-4a77-94f0-452142c08f81", "cc3b9eb6-a3b5-4c26-8605-0486fa000a4b", "8d5d4299-950b-4011-a7ac-b466b1c00e84", "235ae56d-6f4f-4106-9304-fb38e7d4add8" ], "isPlaypalPlaying": false, "lat": 12.976394040119704, "lng": 77.63644146986815, "location": "Game Theory - Double Road Indiranagar, Indiranagar", "joineeCount": 6, "status": -1, "sportsPlayingMode": { "name": "", "icon": "" }, "maxPlayers": 7, "full": false, "price": 0, "startTime": "2025-03-04T13:30:00.000Z", "endTime": "2025-03-04T15:30:00.000Z", "minSkill": 3, "maxSkill": 5, "skillSet": true, "booking": false, "bookingId": "", "type": 0, "venueId": "82af038f-058c-4b2f-bc3d-3a47910d4f97", "venueName": "Game Theory - Double Road Indiranagar, Indiranagar", "activityType": "regular", "isOnline": false, "groupId": "", "groupName": "", "currencyTxt": "INR", "strictSkill": true, "date": "2025-03-04T00:00:00.000Z", "hostId": "redacted", "sportId": "SP5", "timing": 2, "id": "e2ee9f62-c9b6-472b-aea2-b0c52dd7c525", "distance": 0.5249236963063415, "courtInfo": "", "sponsored": false, "groups": [] }</code></pre> Using the above response, I filtered for games where: full</code> is false</code> (This indicates that joineeCount == maxPlayer</code> is not true, meaning spots are still available to join)</li> startTime</code> and endTime</code> fall within 7-8PM IST</li> </ul> I also wanted to add a feature to send these details to Telegram for convenient notifications. I then vibe coded</a> with Claude 3.7 to create a Python script to automate this whole process. Impressively, it produced a working script pretty much in a one-shot prompt, though I had to make a few minor tweaks. I quite like Simon Willison’s approach</a> of using uv</code> to build one-shot tools. Managing dependencies, virtual environments, etc. is still a pain point in Python, but using uv</code> feels like magic by comparison. # /// script # requires-python = ">=3.12" # dependencies = [ # "click", # "requests", # "pytz", # "rich", # "python-dateutil", # "python-telegram-bot", # ] # /// import click import requests import json import datetime import pytz import os import sys from rich.console import Console from rich.table import Table from dateutil import parser from telegram import Bot, InputMediaPhoto from telegram.constants import ParseMode from io import BytesIO import asyncio console = Console() @click.command() @click.option("--lat", default=12.9783692, help="Latitude for search") @click.option("--lng", default=77.6408356, help="Longitude for search") @click.option("--radius", default=50, help="City radius in km") @click.option("--sport", default="SP5", help="Sport ID (default: SP5 for Badminton)") @click.option("--start-time", default="19:00", help="Desired start time (24-hour format HH:MM)") @click.option("--end-time", default="20:00", help="Desired end time (24-hour format HH:MM)") @click.option("--timezone", default="Asia/Kolkata", help="Your timezone") @click.option("--verbose", is_flag=True, help="Show detailed information including exact UTC/IST times") @click.option("--include-full", is_flag=True, help="Include games that are full") @click.option("--telegram", is_flag=True, help="Send results to Telegram") @click.option("--telegram-token", envvar="TELEGRAM_BOT_TOKEN", help="Telegram Bot Token (or set TELEGRAM_BOT_TOKEN env var)") @click.option("--telegram-chat-id", envvar="TELEGRAM_CHAT_ID", help="Telegram Chat ID (or set TELEGRAM_CHAT_ID env var)") def find_games(lat, lng, radius, sport, start_time, end_time, timezone, verbose, include_full, telegram, telegram_token, telegram_chat_id): """Find available badminton games on Playo matching your criteria.""" # Get today's date in the specified timezone local_tz = pytz.timezone(timezone) now = datetime.datetime.now(local_tz) today_date = now.strftime("%Y-%m-%dT%H:%M:%S.%fZ") # Parse desired time window try: desired_start = datetime.datetime.strptime(start_time, "%H:%M").time() desired_end = datetime.datetime.strptime(end_time, "%H:%M").time() except ValueError: console.print("[bold red]Error:[/bold red] Invalid time format. Please use HH:MM (24-hour format).") return console.print(f"[bold green]Searching for badminton games around your location...[/bold green]") console.print(f"Looking for games between [bold]{start_time}[/bold] and [bold]{end_time}[/bold] IST today") if verbose: console.print(f"[dim]Search parameters: lat={lat}, lng={lng}, radius={radius}km[/dim]") console.print(f"[dim]Current time in {timezone}: {now.strftime('%Y-%m-%d %H:%M:%S')}[/dim]") # Prepare API request url = "https://api.playo.io/activity-public/list/location" payload = { "lat": lat, "lng": lng, "cityRadius": radius, "gameTimeActivities": False, "page": 0, "lastId": "", "sportId": [sport], "booking": False, "date": [today_date] } headers = { "Content-Type": "application/json", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" } try: response = requests.post(url, headers=headers, json=payload) response.raise_for_status() data = response.json() if data.get("requestStatus") != 1 or "data" not in data: console.print("[bold red]Error:[/bold red] Failed to get valid response from Playo API") return # Process activities activities = data["data"].get("activities", []) if not activities: console.print("[yellow]No badminton activities found for today[/yellow]") return # Filter activities based on criteria matching_games = [] for activity in activities: # Convert UTC times to local timezone start_time_utc = parser.parse(activity["startTime"]) end_time_utc = parser.parse(activity["endTime"]) start_time_local = start_time_utc.astimezone(local_tz) end_time_local = end_time_utc.astimezone(local_tz) # Print all times in debug mode # console.print(f"DEBUG: {activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST (UTC: {start_time_utc.strftime('%H:%M')})") # Convert time objects correctly for comparison start_hour = start_time_local.hour start_minute = start_time_local.minute # Convert desired times to hours and minutes for easier comparison desired_start_hour = desired_start.hour desired_start_minute = desired_start.minute desired_end_hour = desired_end.hour desired_end_minute = desired_end.minute # Check if this game starts at 7PM (19:00) and ends at 8PM (20:00) is_time_match = False # Get duration in minutes duration_minutes = ((end_time_local.hour * 60 + end_time_local.minute) - (start_time_local.hour * 60 + start_time_local.minute)) # Check if start time is 7PM (with small tolerance) if (start_hour == desired_start_hour and start_minute >= desired_start_minute and start_minute < desired_start_minute + 10): # Allow a small window of 10 minutes # Check if duration is approximately 1 hour (between 50-70 minutes) if 50 <= duration_minutes <= 70: is_time_match = True # Check if there are available slots is_available = ( not activity.get("full", True) and (activity.get("maxPlayers", 0) == -1 or activity.get("joineeCount", 0) < activity.get("maxPlayers", 0)) ) # When verbose, print time details for each game to help debug if verbose: time_info = f"[dim]{activity.get('location', 'Unknown')} - Start: {start_time_local.strftime('%H:%M')} IST ({start_time_utc.strftime('%H:%M')} UTC), " + \ f"End: {end_time_local.strftime('%H:%M')} IST, Duration: {duration_minutes} min, " + \ f"Time match: {'Yes' if is_time_match else 'No'}, Available: {'Yes' if is_available else 'No'}[/dim]" console.print(time_info) # Both conditions must be true if is_time_match and is_available: matching_games.append({ "id": activity["id"], "location": activity["location"], "venue_name": activity.get("venueName", "N/A"), "start": start_time_local.strftime("%I:%M %p"), "end": end_time_local.strftime("%I:%M %p"), "players": f"{activity.get('joineeCount', 0)}/{activity.get('maxPlayers', 'unlimited')}", "host": activity.get("userInfo", [{}])[0].get("fName", "Unknown"), "skill": activity.get("skill", "Any"), "price": f"{activity.get('price', 0)} {activity.get('currencyTxt', 'INR')}" }) # Display results if matching_games: table = Table(title=f"Available Badminton Games ({len(matching_games)} matches found)") table.add_column("Location", style="cyan") table.add_column("Time", style="green") table.add_column("Players", style="yellow") table.add_column("Host", style="magenta") table.add_column("Skill Level", style="blue") table.add_column("Link", style="bright_blue") for game in matching_games: table.add_row( f"{game['venue_name']}", f"{game['start']} - {game['end']}", game["players"], game["host"], game["skill"], f"https://playo.co/match/{game['id']}" ) console.print(table) # Send to Telegram if requested if telegram: if not telegram_token or not telegram_chat_id: console.print("[bold red]Error:[/bold red] Telegram token and chat ID are required for Telegram notifications") console.print("[dim]Set them with --telegram-token and --telegram-chat-id or via environment variables[/dim]") else: try: send_to_telegram(matching_games, telegram_token, telegram_chat_id) console.print("[green]Results sent to Telegram successfully![/green]") except Exception as e: console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}") else: console.print("[yellow]No games found matching your criteria[/yellow]") if telegram and telegram_token and telegram_chat_id: try: asyncio.run(send_telegram_message( "No badminton games found matching your criteria for today.", telegram_token, telegram_chat_id )) console.print("[green]Empty results notification sent to Telegram[/green]") except Exception as e: console.print(f"[bold red]Error sending to Telegram:[/bold red] {e}") except requests.RequestException as e: console.print(f"[bold red]Error:[/bold red] Failed to connect to Playo API: {e}") except json.JSONDecodeError: console.print("[bold red]Error:[/bold red] Failed to parse API response") except Exception as e: console.print(f"[bold red]Error:[/bold red] An unexpected error occurred: {e}") def send_to_telegram(games, token, chat_id): """Send game information to Telegram as a nicely formatted message.""" if not games: return # Create a formatted message for Telegram message = "🏸 *Available Badminton Games* 🏸\n\n" for i, game in enumerate(games, 1): message += f"*{i}. {game['venue_name']}*\n" message += f"⏰ {game['start']} - {game['end']}\n" message += f"👥 Players: {game['players']}\n" message += f"👤 Host: {game['host']}\n" message += f"🎯 Skill: {game['skill']}\n" message += f"🔗 [Join Game](https://playo.co/match/{game['id']})\n\n" # Send the message asyncio.run(send_telegram_message(message, token, chat_id)) async def send_telegram_message(message, token, chat_id): """Send a message to Telegram using the Bot API.""" bot = Bot(token=token) await bot.send_message( chat_id=chat_id, text=message, parse_mode=ParseMode.MARKDOWN, disable_web_page_preview=False ) if __name__ == "__main__": find_games()</code></pre> The script outputs a beautiful output: Telegram: Scheduling#</a></h2> I wanted this script to run reliably every day and used GitHub Actions for that. GitHub Actions felt like the path of least resistance as I didn’t have to worry about keeping a server running or getting alerts if something crashed. For a small personal script like this, it was the perfect “set it and forget it” solution. name: Badminton Game Checker Base on: schedule: # Run Monday to Friday at 12:00 PM IST (6:30 AM UTC) - cron: "30 6 * * 1-5" jobs: check-games: runs-on: ubuntu-latest env: TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }} LATITUDE: ${{ inputs.latitude }} LONGITUDE: ${{ inputs.longitude }} RADIUS: ${{ inputs.radius }} SPORT_ID: ${{ inputs.sport_id }} TIMEZONE: ${{ inputs.timezone }} START_TIME: ${{ inputs.start_time }} END_TIME: ${{ inputs.end_time }} steps: - name: Checkout code uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" cache: "pip" - name: Install dependencies run: | python -m pip install --upgrade pip pip install uv - name: Run game check run: | echo "Checking for games from $START_TIME to $END_TIME" uv run finder.py \ --lat "$LATITUDE" \ --lng "$LONGITUDE" \ --radius "$RADIUS" \ --sport "$SPORT_ID" \ --timezone "$TIMEZONE" \ --start-time "$START_TIME" \ --end-time "$END_TIME" \ --telegram env: TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}</code></pre> I used GitHub Actions inputs to configure the variables for my script. Found this feature to be quite neat for scheduling different crons for weekday/weekends. Summary#</a></h2> For small quality-of-life improvements - solving your own specific problems with custom scripts tailored exactly to your needs - gotta love the LLMs man. We’re gonna see more and more of such “personal tooling” in future as the entry to barrier for coding is lowered with LLMs. The democratization of coding through LLMs means people (even non-technical ones) can focus on “describing” the problem well, rather than struggling with implementation details. Being able to articulate what you want clearly becomes the primary skill - yes, it’s a skill issue if you can’t prompt well, but it’s far more accessible than learning programming from scratch. Fin! Cleaning up Notes with LLM 2025-01-03T10:15:00+05:30 My Obsidian vault has gotten quite messy over time. I’ve been dumping notes without proper frontmatter, tags were all over the place, and some notes didn’t even have proper titles! I needed a way to clean this up without spending hours manually organizing everything. I’d been playing around with Claude’s API lately, and thought – hey, why not use an LLM to analyze my notes and add proper frontmatter? After all, that’s what these AI models are good at – understanding context and categorizing stuff. I wrote a small Python script using the llm</a> library (which is pretty neat btw) to do just this. Here’s what it looks like: import llm import os import yaml import datetime from pathlib import Path import re class ObsidianNoteProcessor: def __init__(self, notes_dir, model_name="claude-3.5-sonnet"): self.notes_dir = Path(notes_dir) self.model = llm.get_model(model_name) def extract_existing_frontmatter(self, content): """Extract existing frontmatter if present.""" frontmatter_pattern = r'^---\n(.*?)\n---\n' match = re.match(frontmatter_pattern, content, re.DOTALL) if match: try: return yaml.safe_load(match.group(1)), content[match.end():] except yaml.YAMLError: return {}, content return {}, content def generate_prompt(self, content): """Generate a prompt for the LLM to analyze the note content.""" return f"""Analyze the following note content and extract/infer the following properties: 1. A clear title (if not present, generate from content) 2. Relevant categories based on the content 3. Appropriate tags (include 'inbox' if content seems draft-like) 4. Status (Draft/In Progress/Complete) based on content completeness 5. Priority (Low/Medium/High) based on content importance 6. A brief description summarizing the content Note content: {content} Return ONLY the YAML frontmatter without any code block markers. Use this exact format (omit fields if not applicable): title: <title> category: <category> tags: - tag1 - tag2 status: <status> priority: <priority> description: <description>""" def clean_llm_response(self, response_text): """Clean up the LLM response to ensure proper YAML.""" # Remove yaml code block markers if present response_text = response_text.strip() if response_text.startswith('```yaml'): response_text = response_text.split('\n', 1)[1] if response_text.endswith('```'): response_text = response_text.rsplit('\n', 1)[0] return response_text.strip() def process_note(self, file_path): """Process a single note file.""" try: with open(file_path, 'r', encoding='utf-8') as f: content = f.read() # Extract existing frontmatter and content existing_frontmatter, main_content = self.extract_existing_frontmatter(content) # Generate and execute prompt response = self.model.prompt(self.generate_prompt(main_content)) response_text = self.clean_llm_response(response.text()) try: new_frontmatter = yaml.safe_load(response_text) if not isinstance(new_frontmatter, dict): print(f"Warning: Invalid response format for {file_path.name}") new_frontmatter = {} except yaml.YAMLError as e: print(f"YAML parsing error for {file_path.name}") print(f"Response text was:\n{response_text}") raise e # Merge with existing frontmatter, preferring existing values merged_frontmatter = {**new_frontmatter, **existing_frontmatter} # Add date if not present if 'date' not in merged_frontmatter: merged_frontmatter['date'] = datetime.date.today().isoformat() # Generate new note content new_content = "---\n" new_content += yaml.dump(merged_frontmatter, sort_keys=False, allow_unicode=True) new_content += "---\n\n" new_content += main_content.strip() # Write back to file with open(file_path, 'w', encoding='utf-8') as f: f.write(new_content) print(f"✓ Processed: {file_path.name}") except Exception as e: print(f"✗ Error processing {file_path.name}: {str(e)}") def process_vault(self): """Process all markdown files in the vault.""" print("Starting Obsidian vault cleanup...") for file_path in self.notes_dir.glob('**/*.md'): self.process_note(file_path) print("\nVault cleanup completed!") def main(): # Set up the model key if not already configured model = llm.get_model("claude-3.5-sonnet") if not hasattr(model, 'key'): api_key = os.getenv('ANTHROPIC_API_KEY') if not api_key: raise ValueError("Please set ANTHROPIC_API_KEY environment variable") model.key = api_key # Initialize and run the processor notes_dir = "/Users/karan/Notes/Obsidian/The Wall/Notes" processor = ObsidianNoteProcessor(notes_dir) processor.process_vault() if __name__ == "__main__": main()</code></pre> The script is pretty straightforward – it reads each markdown file, extracts any existing frontmatter (because I don’t want to lose that!), and then asks Claude to analyze the content and generate appropriate frontmatter. It adds stuff like title, category, tags, status, priority. What I love about this approach is that it’s contextual. Unlike regex-based approaches or keyword matching, the LLM actually understands what the note is about and can categorize it properly. A note about “Setting up BTRFS on Arch” automatically gets tagged with “linux”, “filesystem”, “arch” without me having to maintain a predefined list of tags. The categorization is probably better than what I’d have done manually at 2 AM while organizing my notes! Fin! 2024: A Year In Review 2024-12-23T12:00:00+05:30 2024 was indeed an important year for me as it marked several significant milestones. Quite happy with how this year was! Here’s my reflection on this memorable year. Life#</a></h2> Got married to the prettiest and dearest Saumya 💗</li> Did my first international trip, exploring Europe</li> Bought a fun toy - Maruti Jimny 4x4</li> Relocated to Bangalore after working from home for 4+ years since Covid</li> Attended several amazing concerts: Indian Ocean</li> Parvaaz</li> Blackstratblues</li> Anand Bhaskar Collective</li> Bandland</li> </ul> </li> </ul> Travel#</a></h2> Switzerland</li> Italy</li> Austria</li> Binsar</li> Cochin</li> Ajmer</li> Ranthambore</li> Pondicherry</li> Chennai</li> </ul> Projects#</a></h2> Released v1.0.0 of Doggo</a> - It hit frontpage of HN</a> as well!</li> Built an expense tracker app - Gullak</a></li> Made a lot of small utility apps: lil</a> - URL shortener</li> silencer</a> - Prometheus alerts <> Mattermost bridge</li> toru</a> - Go modules proxy with caching</li> junbi</a> - Server Setup and Hardening Tool</li> Ovenly Delights</a> - Small bakery shop website</li> nomcfg</a> - Nomad config generator</li> clx</a> - Generate CLI commands using AI for common ops</li> </ul> </li> Started working on a log analytics app - full focus on that in 2025. Read more</a></li> </ul> This year has been truly transformative, bringing together personal joy, professional growth, and exciting adventures. Looking forward to what 2025 has in store! How I use LLMs 2024-10-30T00:00:00+00:00 Just yesterday, GitHub announced</a> integrating Claude 3.5 Sonnet with Copilot. Interesting times ahead. In my experience, Claude has been remarkably better than the GPT-4 family of models for programming tasks. I’ve tried a bunch of tools like Cursor, Continue.dev but finally settled with Aider</a> for most of my tasks. In this post, I want to write about my workflow of using Aider when working on small coding tasks. Aider is an open source Python CLI which supports multiple models, including Claude 3.5 Sonnet. Aider describes itself as “AI pair programming in your terminal”. The tool integrates git</code> quite well in its workflow so it can edit files, create new files, and track all changes via git. In case you want to revert, simply reverting the commit or using the /undo</code> shortcut would do the same. The tool has multiple modes that serve different purposes: /ask</code>: Use it when you simply want to chat with the model about the codebase or explain some pieces of it. This mode won’t touch your files. It’s great for understanding existing code or getting explanations.</li> /architect</code>: Use it to discuss a broad overall idea. The model will propose some changes to your files. You can further chat and tune it to your preferences.</li> /code</code>: This will directly edit your files and commit them.</li> </ul> My typical workflow involves running Aider in a terminal while keeping VSCode open for manual code review. I often use the --no-auto-commits</code> flag to view the diffs before committing. Despite advances in LLM technology, I believe they haven’t yet reached the stage where they can fully understand your team’s coding style guides, and I prefer not to have a certain style forced upon me. Manually tweaking portions of AI-generated functions still proves helpful and saves considerable time. To begin, aider --sonnet</code> would open the interactive window where you can begin writing prompts. To add context, you need to add files using commands like /add main.py</code>. What makes Aider powerful is its control over the LLM context - you can /add</code> or /drop</code> source code, or even /reset</code> to drop all files and start with a fresh context. This granular control helps manage the context window effectively. A really cool thing about it is that it gives an approximate idea of the number of tokens (cost) associated with each prompt. I find it useful to remove unnecessary files from the context window, which not only helps in getting sharper, more accurate responses but also helps with the costs. There’s a nice /tokens</code> command which will show the cost of sending each file added in context with the prompt. I find the Aider + Claude 3.5 combo works really well when you have a narrow-scoped, well-defined task. For example, this is the prompt I used on a codebase I was working on: Theme preference is not preserved when reloading pages or navigating to new pages. We should store this setting in localStorage. Please implement using standard best practices. </blockquote> Under the hood, Aider uses tree-sitter</a> to improve code generation and provide rich context about your codebase. Tree-sitter parses your code into an Abstract Syntax Tree (AST), which helps Aider understand the structure and relationships in your code. Unlike simpler tools that might just grep through your codebase, tree-sitter understands the actual syntax of your programming language. It can identify function definitions, class declarations, variable scopes, and their relationships</li> It extracts full function signatures and type information</li> It builds a dependency graph showing how different parts of your code relate to each other</li> It helps rank the importance of different code sections based on how often they’re referenced</li> </ul> This means when you’re working on a task, Aider isn’t just blindly sending your entire codebase to the LLM. Instead, it creates an optimized “repository map” that fits within your token budget (default is 1k tokens, adjustable via --map-tokens</code>). This map focuses on the most relevant pieces of your code, making sure the LLM understands the context without wasting tokens. Aider’s approach to AI pair programming feels natural and productive. Here are some example prompts where it helped me build stuff in less than a minute: Modify fetch method in store/store.go to filter out expired entries </blockquote> Write a k6 load test script to benchmark the POST /submit</code> endpoint and simulate real-world traffic patterns </blockquote> Create a Makefile, Dockerfile, goreleaser.yml for my Go binary. Target platforms: arm64 and amd64 </blockquote> I prefer to invoke aider</code> with a few extra flags: aider --no-auto-commits --cache-prompts --cache-keepalive-pings 12 --no-suggest-shell-commands</code></pre> Make sure to go through the Tips</a> page to effectively try out Aider on your existing projects. Fin! Self Hosting Outline Wiki 2024-09-20T00:00:00+00:00 I recently discovered Outline</a> a collaborative knowledge base. I wanted to self-host it on my server, but the mandatory auth provider requirement was off-putting. My server is on a private encrypted network (Tailscale) that only my approved devices in the tailnet can access, so I don’t really need authentication for my personal single-use apps. I found a few guides using Authelia/Keycloak, but these are heavy-duty applications that would consume a lot of resources (DBs, caches, proxies, and whatnot) just to have an OIDC provider for Outline. There had to be a simpler way, right? Enter Dex</a>. As recommended by my friend and colleague Chinmay</a>, it turned out to be quite easy. Here’s the full docker-compose.yml</code> setup you need to get Outline up and running on your local instance! services: outline: image: docker.getoutline.com/outlinewiki/outline:latest env_file: ./docker.env ports: - "3000:3000" volumes: - storage-data:/var/lib/outline/data depends_on: - postgres - redis environment: PGSSLMODE: disable redis: image: redis env_file: ./docker.env ports: - "6379:6379" healthcheck: test: [ "CMD", "redis-cli", "ping" ] interval: 10s timeout: 30s retries: 3 postgres: image: postgres env_file: ./docker.env ports: - "5432:5432" volumes: - database-data:/var/lib/postgresql/data healthcheck: test: [ "CMD", "pg_isready", "-d", "outline", "-U", "user" ] interval: 30s timeout: 20s retries: 3 environment: POSTGRES_USER: 'user' POSTGRES_PASSWORD: 'pass' POSTGRES_DB: 'outline' dex: image: dexidp/dex:v2.35.3 ports: - "5556:5556" volumes: - ./dex:/etc/dex command: [ "dex", "serve", "/etc/dex/config.yaml" ] volumes: storage-data: database-data: </code></pre> You’ll need to add the following env variables as well NODE_ENV=production SECRET_KEY=your-key UTILS_SECRET=your-key DATABASE_URL=postgres://user:pass@postgres:5432/outline PGSSLMODE=disable REDIS_URL=redis://redis:6379 URL=http://localhost:3000 PORT=3000 FILE_STORAGE=local FILE_STORAGE_LOCAL_ROOT_DIR=/var/lib/outline/data FILE_STORAGE_UPLOAD_MAX_SIZE=262144000 OIDC_CLIENT_ID=outline OIDC_CLIENT_SECRET=outline-secret OIDC_AUTH_URI=http://localhost:5556/dex/auth OIDC_TOKEN_URI=http://dex:5556/dex/token OIDC_USERINFO_URI=http://dex:5556/dex/userinfo OIDC_USERNAME_CLAIM=preferred_username OIDC_DISPLAY_NAME=Dex OIDC_SCOPES=openid profile email FORCE_HTTPS=false ENABLE_UPDATES=true WEB_CONCURRENCY=1 DEBUG=http LOG_LEVEL=info</code></pre> And finally, to configure Dex, we need the following config: issuer: http://localhost:5556/dex storage: type: sqlite3 config: file: /var/dex/dex.db web: http: 0.0.0.0:5556 staticClients: - id: outline redirectURIs: - "http://localhost:3000/auth/oidc.callback" name: "Outline" secret: outline-secret oauth2: skipApprovalScreen: true enablePasswordDB: true staticPasswords: - email: "admin@example.com" hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W" username: "admin" userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"</code></pre> Voilà! With docker compose up</code>, you’ll have an Outline server ready to go. You can log in using the admin</code> user. Building an expense tracker app 2024-06-14T00:00:00+00:00 A couple of weeks ago, I decided to start logging and tracking my expenses. The goal was not to record every minor purchase but to gain a general insight into where my money was going. In this post, I’ll dive deep into the behind-the-scenes of building Gullak</a>—an expense tracker app with a dash of AI (yes). </iframe> </div> Why#</a></h2> My wife and I have a simple system for tracking our expenses during trips: we use Apple Notes to maintain a day-wise record, jotting down a one-liner for each expense under the date. This straightforward method has proven effective in keeping tabs on our spending habits while traveling. For instance, during our last Europe trip, we recorded our daily expenses. After returning home, I was eager to analyze our spending patterns. I copied all these items into Google Sheets to analyse the top categories that I spent on during the trip. I decided to develop a simple expense tracker app that automatically categorizes expenses into various groups like food, travel, shopping, etc. I believed this was a practical use case for leveraging an LLM paired with Function calling</a> to parse and categorize expenses. Initial Prototype#</a></h2> The first step involved designing a prompt to capture user input about their spending. I picked up go-openai</a> library and experimented with it. Almost a year ago, I had developed a small bot for personal use, which provided a JSON output detailing the macronutrients and calories in specific food items, storing this information in Metabase. However, this was during the early days of API access provided by OpenAI. Due to occasionally unsatisfactory and inconsistent responses (despite instructions like “MUST RETURN JSON OR 1000 CATS WILL D*E SOMEWHERE”), it wasn’t entirely reliable. Function calling addresses two main limitations of traditional language model responses: Inconsistent response format: Without function calling, responses from language models can be unstructured and inconsistent, requiring complex validation and parsing logic on the application side.</li> Lack of external data integration: Language models are typically limited to the knowledge they were trained on, making it challenging to provide answers based on real-time or external data.</li> </ul> It’s important to note that the LLM does not actually execute any functions. Rather, we create a structure for the LLM to follow in its responses. The LLM would then generate a response with the content as a stringified JSON object following the schema provided in the function definiton. I created a function called categorize_expense</code>. This function takes a list of transactions as parameters, with each transaction having properties like transaction_date</code>, amount</code>, category</code>, and description</code>. Here’s what this looks like: fnCategorizeExpenses := openai.FunctionDefinition{ Name: "categorize_expense", Description: "Categorize expenses from the given input.", Parameters: jsonschema.Definition{ Type: jsonschema.Object, Properties: map[string]jsonschema.Definition{ "transactions": { Type: jsonschema.Array, Description: "List of items purchased", Items: &jsonschema.Definition{ Type: jsonschema.Object, Properties: map[string]jsonschema.Definition{ "transaction_date": { Type: jsonschema.String, Description: "Date of transaction in ISO 8601 format (e.g., 2021-09-01) if specified else today's date.", }, "amount": { Type: jsonschema.Number, Description: "Amount of the item", }, "category": { Type: jsonschema.String, Description: "One word category of the expense (e.g., food, travel, entertainment)", }, "description": { Type: jsonschema.String, Description: "Concise and short description of the item", }, }, Required: []string{"transaction_date", "amount", "category", "description"}, }, }, }, Required: []string{"transactions"}, }, }</code></pre> The response from this API call can then be unmarshalled into a struct. var transactions models.Transactions if err := json.Unmarshal([]byte(toolCall.Function.Arguments), &transactions); err != nil { return err }</code></pre> The next step was to determine exactly how users would provide input. I considered various methods that would make entering expenses as straightforward as my approach with Apple Notes and decided to create a Telegram bot. I developed a Telegram bot that would parse the expenses and save them to a SQLite database. I explored tools like evidence.dev</a>, a nice platform for creating frontends using the database as the sole source of truth. However, I encountered an issue where it could not correctly parse date values (see GitHub issue</a>). Ultimately, I returned to my reliable old friend—Metabase. However, I faced two main challenges with this approach: Privacy Concerns: Telegram does not offer the option to create a private bot; all bots generated through BotFather are public. To restrict access, I considered adding session tokens, but this approach was unsatisfactory. If I planned to distribute this bot, implementing a token-based, DIY authentication system on Telegram did not seem appropriate. </li> Fixing Bad Entries: To correct erroneous entries, I had to manually update the SQLite table. As I intended to share this bot with my wife, I needed a more user-friendly workflow. Manually raw dogging UPDATE</code> SQL queries was not the most user-friendly solution. </li> </ul> After a day or two of experimenting, I decided to build a small frontend for now. Building Frontend#</a></h2> As a backend developer, my core expertise is NOT JavaScript, and I strongly dislike the JS ecosystem. Obviously there’s no dearth of choices when it comes to frameworks, however for this project I wanted to stay away from the hype and choose a stack that is simple to use and productive (for me) out of the box. Having used Vue.js in production in the past, I feel it ticks those boxes for me as it comes bundled with a router, store, and all the niceties, and it has excellent documentation. After reading a refresher on the new Vue3 composition API syntax, I hit the ground running. I find Tailwind CSS ideal for someone like me who prefers not to write CSS or invent class names. It’s a heavily debated topic online, but it’s important to pick our battles. An issue I encountered while researching UI frameworks was that Vue.js seems to have fewer options compared to React, likely due to its lower popularity. After some google-fu, I discovered a promising project called shadcn-vue</a>, an unofficial community led port of the shadcn/ui</a> React library. The cool thing about this library is that it doesn’t come bundled as a package, meaning there’s no way to install it as a dependency. Instead, it gets added directly to your source code, encouraging you to tweak it the way you like. I believe it’s an excellent starting point for anyone looking to build their own design system from scratch, as it allows for customization of both appearance and behavior. It might have been overkill for my simple UI, but I thought, what the heck, if side projects aren’t for exploring new things, what’s the point of it all? 😄 Database#</a></h2> For the database, I opted for SQLite. It’s perfect for a small project like this since the database is just a single file, making it easier to manage. Initially, I used the popular driver mattn/go-sqlite3</a>, but I found that the CGO-free alternative modernc/sqlite</a> works just as well. I also experimented with sqlc</a> for the first time. For those unfamiliar, sqlc</code> generates type-safe Go code from your raw SQL queries. It handles all the boilerplate database code needed to retrieve results, scan them into a model, manage transactions, and more. sqlc makes it seem like you’re getting the best of both worlds (ORM + raw SQL). Here’s an example query: -- name: CreateTransaction :many -- Inserts a new transaction into the database. INSERT INTO transactions (created_at, transaction_date, amount, currency, category, description, confirm) VALUES (?, ?, ?, ?, ?, ?, ?) RETURNING *;</code></pre> Using sqlc generate</code>, it generates the following code: // Code generated by sqlc. DO NOT EDIT. // versions: // sqlc v1.26.0 // source: queries.sql package db import ( "context" "database/sql" "time" ) const createTransaction = `-- name: CreateTransaction :many INSERT INTO transactions (created_at, transaction_date, amount, currency, category, description, confirm) VALUES (?, ?, ?, ?, ?, ?, ?) RETURNING id, created_at, transaction_date, currency, amount, category, description, confirm ` type CreateTransactionParams struct { CreatedAt time.Time `json:"created_at"` TransactionDate time.Time `json:"transaction_date"` Amount float64 `json:"amount"` Currency string `json:"currency"` Category string `json:"category"` Description string `json:"description"` Confirm bool `json:"confirm"` } // Inserts a new transaction into the database. func (q *Queries) CreateTransaction(ctx context.Context, arg CreateTransactionParams) ([]Transaction, error) { rows, err := q.query(ctx, q.createTransactionStmt, createTransaction, arg.CreatedAt, arg.TransactionDate, arg.Amount, arg.Currency, arg.Category, arg.Description, arg.Confirm, ) if err != nil { return nil, err } defer rows.Close() items := []Transaction{} for rows.Next() { var i Transaction if err := rows.Scan( &i.ID, &i.CreatedAt, &i.TransactionDate, &i.Currency, &i.Amount, &i.Category, &i.Description, &i.Confirm, ); err != nil { return nil, err } items = append(items, i) } if err := rows.Close(); err != nil { return nil, err } if err := rows.Err(); err != nil { return nil, err } return items, nil }</code></pre>Apple Shortcuts#</a></h2> Similar to my Apple Notes approach, I wanted to create a shortcut that would allow me to log expenses quickly. I created a simple shortcut that would prompt me to enter the expenses and send an HTTP POST request to Gullak’s API server. I then open the dashboard once in a while to confirm/edit these unconfirmed transactions. You can read more about setting up the Shortcut in your Apple devices here</a>. Proudly, Not a Weekend Project#</a></h2> For every “I could do this in a weekend” comment, yes, this project is straightforward—a “CRUD GPT” wrapper that isn’t complicated to build. Yet, it took me over a month to develop. I spent less than an hour most days on this project, instead of cramming it into an all-nighter weekend project - an approach I want to move away from. Slow and steady efforts compound, outlasting quick, sporadic bursts. I’m pleased to balance this with my full-time job without burning out. Ideas for the Future#</a></h2> Initially, I didn’t set out to build a comprehensive budgeting app, just an expense logger, as that was my primary need. However, if usage increases and the tool proves helpful in reducing unnecessary spending, I’m open to adding more features. Some possibilities include a subscription tracker, integration with budgeting tools like YNAB or Actual through their APIs, and monthly reports sent via email. The best part is that you own complete data, as the data is stored locally on your device so you can also export it anytime and build other integrations on top of it. Feel free to open a GitHub issue or reach out</a> if you have any suggestions or feedback. I’m excited to see where this project goes! Update (Feb 2026)#</a></h2> I’ve rewritten Gullak to use plain-text ledger files. Read more about why in the follow-up post: Why Plain-Text Ledger is Powerful for Gullak</a>. A Random Act of Kindness 2024-04-21T00:00:00+00:00 Last month, I did a wonderful trip travelling through the scenic landscapes of Switzerland. My wife and I were in Lucerne and had scheduled a day trip to Mt. Titlis for the next day but were wondering what to do that evening. After strolling along the Chapel Bridge and enjoying an amazing lunch by the waterfront, my wife and I collectively decided to book a Lake Lucerne cruise for the evening. It seemed like the perfect setup for a romantic date night, or so I thought! We got the tickets from the information booth for the evening and went back to our hotel to freshen up and relax for a bit. We arrived exactly at 6:45 PM, as mentioned on our tickets, and started to wait. Except, there were only the two of us waiting. We waited for probably half an hour, until 7:15 PM, which was the departure time. Knowing how precise the Swiss transport system usually is, I sensed something fishy. Luckily, I spotted Martin, a staff member of the cruise company, and asked him about it. He looked puzzled by my question and informed me that there was no cruise scheduled for today. Yep, not today, not tomorrow, and not for the rest of the weekend. It was Easter time, and all the cruise trips were cancelled. In fact, he was quite as puzzled as I was as to how the lady at the ticket counter even gave us the tickets for today. However, he told me that he couldn’t do much and suggested writing an email for a refund. I was a bit sad as this dashed our evening plans, but I thought, fine… Shit happens. It’s not the end of the world! And then, out of nowhere, by pure serendipity, my wife spotted another member of the cruise—this time, in fact, the captain! She began to tell the captain about our ordeal. The captain, a very warm and kind lady, listened to my wife patiently and understood our plight! She apologized on behalf of her company and immediately offered a refund in cash for the tickets that we had purchased. We were quite happy, as following up on emails, etc., was something that I was not particularly excited about looking forward to during the other half of my trip. So we said yes, except she needed to go to the ATM to draw cash. We waited for her and chatted with Martin about random stuff! He told us some really fun stories about how Toblerone’s iconic packaging no longer features the Matterhorn mountain</a>. He also shared practical advice on how to safeguard oneself against pickpockets in Italy(we were gonna visit it soon), and reminisced about his life in Zermatt before relocating to Lucerne. Anyway, the wait stretched longer than usual, and I found myself wondering about her whereabouts. And then finally, we saw her approaching us. She told us that she hadn’t found a working ATM nearby, and had to go a bit far. But she didn’t come back empty-handed; she also brought us macaroons as a token of her apologies, a gesture that was incredibly thoughtful. She handed us 200 CHF in cash, covering more than the 180 CHF we had paid for our tickets. When we attempted to return the excess 20 CHF, she firmly refused to take it back. Despite our insistence on returning the excess 20 CHF, she remained steadfast, refusing to accept it. She encouraged us to use the extra money to enjoy a few drinks, suggesting it as a consolation for our evening plans being spoiled by the canceled cruise. I am glad the cruise didn’t happen. Life has its own ways of revealing that kindness exists in every corner of the world, and serendipity can lead to the most memorable encounters! For most, it may not be a huge thing, but for me, it was a profoundly touching and generous act from a stranger who simply chose to be kind without any ulterior motives. The lesson: Be kind to others, do no harm, and always pay it forward. :) Fin! Travelling with Tailscale 2024-03-27T00:00:00+00:00 I have an upcoming trip to Europe, which I am quite excited about. I wanted to set up a Tailscale exit node to ensure that critical apps I depend on, such as banking portals continue working from outside the country. Tailscale provides a feature called “Exit nodes”. These nodes can be setup to route all traffic (0.0.0.0/0, ::/0) through them. I deployed a tiny DigitalOcean droplet in BLR</code> region and setup Tailscale as an exit node. The steps are quite simple and can be found here</a>. $ echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf $ echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf $ sudo sysctl -p /etc/sysctl.d/99-tailscale.conf $ sudo tailscale up --advertise-exit-node</code></pre> The node is now advertised as an exit node, and we can confirm that from the output of tailscale status</code>: $ sudo tailscale status 100.78.212.33 pop-os mr-karan@ linux - 100.75.180.88 homelab mr-karan@ linux - 100.100.191.57 iphone mr-karan@ iOS offline 100.123.189.14 karans-macbook-pro mr-karan@ macOS offline 100.104.67.7 lab mr-karan@ linux offline 100.108.220.87 tailscale-exit mr-karan@ linux active; exit node; direct 167.71.236.222:41641, tx 21540 rx 17356</code></pre> On the client side, I was able to start Tailscale and configure it to send all the traffic to the exit node with: sudo tailscale up --exit-node=100.108.220.87</code></pre> We can confirm that the traffic is going via the exit node by checking our public IP from this device: ➜ curl https://ipinfo.io { "ip": "167.x.x.222", "city": "Doddaballapura", "region": "Karnataka", "country": "IN", "loc": "13.2257,77.5750", "org": "AS14061 DigitalOcean, LLC", "postal": "560100", "timezone": "Asia/Kolkata", "readme": "https://ipinfo.io/missingauth" } </code></pre> However, I encountered a minor issue since I needed to bring my work laptop for on-call duties, in case any critical production incidents required my attention during my travels. At my organization, we use Netbird as our VPN, which, like Tailscale, creates a P2P overlay network between different devices. The problem was that all 0.0.0.0 traffic was routed to the exit node, meaning the internal traffic meant for Netbird to access internal sites on our private AWS VPC network was no longer routed via the Netbird interface. Netbird automatically propagates a bunch of IP routing rules when connected to the system. These routes are to our internal AWS VPC infrastructure. For example: 10.0.0.0/16 via 100.107.12.215 dev wt0</code></pre> Here, wt0</code> is the Netbird interface. So, for example, any IP like 10.0.1.100</code> will go via this interface. To verify this: $ ip route get 10.0.1.100 10.0.1.100 dev wt0 src 100.107.12.215 uid 1000 </code></pre> However, after connecting to the Tailscale exit node, this was no longer the case. Now, even the private IP meant to be routed via Netbird was being routed through Tailscale: $ ip route get 10.0.1.100 10.0.1.100 dev tailscale0 table 52 src 100.78.212.33 uid 1000 </code></pre> Although Tailscale nodes allow for the selective whitelisting of CIDRs to route only the designated network packets through them, my scenario was different. I needed to selectively bypass certain CIDRs and route all other traffic through the exit nodes. I came across a relevant GitHub issue</a>, but unfortunately, it was closed due to limited demand. This led me to dig deeper into understanding how Tailscale propagates IP routes, to see if there was a way for me to add custom routes with a higher priority. Initially, I examined the IP routes for Tailscale. Typically, one can view the route table list using ip route</code>, which displays the routes in the default</code> and main</code> tables. However, Tailscale uses routing table 52 for its routes, instead of the default or main table. $ ip route show table 52 default dev tailscale0 100.75.180.88 dev tailscale0 ... others ... throw 127.0.0.0/8 192.168.29.0/24 dev tailscale0 </code></pre> A few notes on the route table: default dev tailscale0</code> is the default route for this table. Traffic that doesn’t match any other route in this table will be sent through the tailscale0</code> interface. This ensures that any traffic not destined for a more specific route will go through the Tailscale network. </li> throw 127.0.0.0/8</code>: This is a special route that tells the system to “throw” away traffic destined for 127.0.0.0/8 (local host addresses) if it arrives at this table, effectively discarding it before it reaches the local routing table. </li> </ul> We can see the priority of these IP rules are evaluated using ip rule show</code>: ➜ ip rule show 0: from all lookup local 5210: from all fwmark 0x80000/0xff0000 lookup main 5230: from all fwmark 0x80000/0xff0000 lookup default 5250: from all fwmark 0x80000/0xff0000 unreachable 5270: from all lookup 52 32766: from all lookup main 32767: from all lookup default</code></pre> This command lists all the current policy routing rules, including their priority (look for the pref or priority value). Each rule is associated with a priority, with lower numbers having higher priority. By default, Linux uses three main routing tables: Local (priority 0)</li> Main (priority 32766)</li> Default (priority 32767)</li> </ul> Since Netbird already propagates the IP routes in the main routing table, we only need to add a higher priority rule to lookup in the main</code> table before Tailscale takes over. $ sudo ip rule add to 10.0.0.0/16 pref 5000 lookup main</code></pre> Now, our ip rule</code> looks like: $ ip rule show 0: from all lookup local 5000: from all to 10.0.0.0/16 lookup main 5210: from all fwmark 0x80000/0xff0000 lookup main 5230: from all fwmark 0x80000/0xff0000 lookup default 5250: from all fwmark 0x80000/0xff0000 unreachable 5270: from all lookup 52 32766: from all lookup main 32767: from all lookup default</code></pre> To confirm whether the packets for destination 10.0.0.0/16</code> get routed via wt0</code> instead of tailscale0</code>, we can use the good ol’ ip route get</code>: $ ip route get 10.0.1.100 10.0.1.100 dev wt0 src 100.107.12.215 uid 1000</code></pre> Perfect! This setup allows us to route all our public traffic via exit node and only the internal traffic meant for internal AWS VPCs get routed via Netbird VPN. Since, these rules are ephemeral and I wanted to add a bunch of similar network routes, I created a small shell script to automate the process of adding/deleting rules: #!/bin/bash # Function to add IP rules for specified CIDRs add() { echo "Adding IP rules..." sudo ip rule add to 10.0.0.0/16 pref 5000 lookup main # ... others ... } # Function to remove IP rules based on preference numbers remove() { echo "Removing IP rules..." sudo ip rule del pref 5000 # ... others .... } # Check the first argument to determine which function to call case $1 in add) add ;; remove) remove ;; *) echo "Invalid argument: $1" echo "Usage: $0 add|remove" exit 1 ;; esac</code></pre> Fin! One Billion Row Challenge in Go 2024-01-10T00:00:00+00:00 Earlier this week, I had stumbled upon 1brc</a>, which presents a fun task: loading a huge text file (1 billion lines) in Java as quickly as possible. The One Billion Row Challenge (1BRC) is a fun exploration of how far modern Java can be pushed for aggregating one billion rows from a text file. Utilize all your virtual threads, leverage SIMD, optimize your GC, or employ any other technique to create the fastest implementation for this task! </blockquote> The challenge is mainly about Java, but I thought to do the same in my preferred language: Go. This post is about how I did several iterations to my Go program to reduce the time and discuss the main techniques used in each iteration to make it faster. I was able to create a solution which takes ~20s to read, parse and calculate stats for 1bn lines on my Apple M2 (10 vCPU, 32GB RAM). There are some insane</a> solutions that people have come up with, be sure to check out GitHub Discussions</a> to go through them! Prerequisites#</a></h2> To generate the text file for these measurements, follow the steps outlined here</a>. After running the commands, I have a measurements.txt</code> on my file system: Example output after running the commands: ➜ 1brc-go git:(main) du -sh measurements.txt 13G measurements.txt ➜ 1brc-go git:(main) tail measurements.txt Mek'ele;13.3 Kampala;50.8 Dikson;-3.7 Dodoma;20.3 San Diego;7.1 Chihuahua;20.3 Ngaoundéré;24.2 Toronto;12.7 Wrocław;12.6 Singapore;14.4</code></pre>Ultra minimalistic example of reading a file#</a></h2> Let’s take a look at a basic Go code to read and parse the above file. We’ll also calculate stats on the fly. package main import ( "bufio" "fmt" "os" "sort" "strconv" "strings" ) type Measurement struct { Station string Temp float64 } type Stats struct { Min, Mean, Max float64 } func main() { // Open the file. file, err := os.Open("measurements.txt") if err != nil { panic(err) } defer file.Close() // Map to hold the temperatures for each station. stationTemps := make(map[string][]float64) scanner := bufio.NewScanner(file) for scanner.Scan() { // Parse each line into a Measurement struct. parts := strings.Split(scanner.Text(), ";") temp, _ := strconv.ParseFloat(parts[1], 64) stationTemps[parts[0]] = append(stationTemps[parts[0]], temp) } // Calculate min, mean, and max for each station. results := make(map[string]Stats) for station, temps := range stationTemps { min, max, sum := temps[0], temps[0], 0.0 for _, t := range temps { if t < min { min = t } if t > max { max = t } sum += t } mean := sum / float64(len(temps)) results[station] = Stats{Min: min, Mean: mean, Max: max} } // Sort the stations and format the output. var stations []string for station := range results { stations = append(stations, station) } sort.Strings(stations) fmt.Print("{") for i, station := range stations { r := results[station] fmt.Printf("%s=%.1f/%.1f/%.1f", station, r.Min, r.Mean, r.Max) if i < len(stations)-1 { fmt.Print(", ") } } fmt.Println("}") }</code></pre> On running the above program, we get the following output: {Chihuahua=20.3/20.3/20.3, Dikson=-3.7/-3.7/-3.7, Dodoma=20.3/20.3/20.3, Kampala=50.8/50.8/50.8, Mek'ele=13.3/13.3/13.3, Ngaoundéré=24.2/24.2/24.2, San Diego=7.1/7.1/7.1, Singapore=14.4/14.4/14.4, Toronto=12.7/12.7/12.7, Wrocław=12.6/12.6/12.6}</code></pre> This approach works well for small, simple files. However, there are certain restrictions: It reads the file line by line using a scanner. Reading and processing a billion rows is time-consuming.</li> Each operation, even if small, adds up when repeated a billion times. This includes string splitting, type conversion, error checking, and appending to a slice.</li> Additionally, we need to consider the potential of hitting the max Disk IOPS limit if we perform too many file operations per second.</li> </ul> Before we proceed to optimize this further, let’s establish a baseline performance of 100 million lines first: $ wc -l measurements.txt 100000000 measurements.txt $ time go run main.go go run main.go 18.44s user 0.83s system 100% cpu 19.135 total</code></pre> Baseline: It takes approximately 19s to read and calculate stats from 100 mn lines. There’s a lot of room to optimize it further, let’s go through them one by one. Iteration 1: Producer-Consumer Pattern#</a></h2> The concept involves reading multiple lines simultaneously in the producer Goroutine and then dispatching these batches to worker Goroutines. We can establish a worker pool to implement a producer-consumer pattern. Producers read lines from the file and send them to a channel. Consumers retrieve lines from the channel, parse the data, and calculate the minimum, mean, and maximum temperatures for each station. func main() { numWorkers := runtime.NumCPU() runtime.GOMAXPROCS(numWorkers) linesChan := make(chan string, 1000000) resultsChan := make(chan map[string]Stats, numWorkers) // Start worker goroutines var wg sync.WaitGroup for i := 0; i < numWorkers; i++ { wg.Add(1) go worker(linesChan, resultsChan, &wg) } // Read the file and send lines to the workers go func() { file, err := os.Open(measurementsFile) if err != nil { panic(err) } defer file.Close() scanner := bufio.NewScanner(file) for scanner.Scan() { linesChan <- scanner.Text() } close(linesChan) }() // Collect results from workers wg.Wait() close(resultsChan) // Aggregate results finalResults := make(map[string]Stats) for workerResult := range resultsChan { for station, stats := range workerResult { finalStats := finalResults[station] finalStats.Min = min(finalStats.Min, stats.Min) finalStats.Max = max(finalStats.Max, stats.Max) finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(finalStats.Count+stats.Count) finalStats.Count += stats.Count finalResults[station] = finalStats } } // Print results printStats(finalResults) } func worker(linesChan <-chan string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) for line := range linesChan { parts := strings.Split(line, ";") temp, err := strconv.ParseFloat(parts[1], 64) if err != nil { continue } stats := stationStats[parts[0]] stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[parts[0]] = stats } resultsChan <- stationStats } func min(a, b float64) float64 { if a == 0 || a > b { return b } return a } func max(a, b float64) float64 { if a < b { return b } return a } func printStats(statsMap map[string]Stats) { var stations []string for station := range statsMap { stations = append(stations, station) } sort.Strings(stations) fmt.Print("{") for i, station := range stations { stats := statsMap[station] fmt.Printf("%s=%.1f/%.1f/%.1f", station, stats.Min, stats.Mean, stats.Max) if i < len(stations)-1 { fmt.Print(", ") } } fmt.Println("}") }</code></pre>Results#</a></h3> The concurrent version, unexpectedly, resulted in almost a 3x decrease in performance. go run main.go 84.15s user 101.34s system 342% cpu 54.225 total</code></pre> Where did we go wrong? This is a classic case where the overhead of concurrency mechanisms outweighs their benefits. In our current implementation, each line is sent to the channel individually, which is likely less efficient than batching lines for processing. This means that for a file with a large number of lines, there will be an equally large number of channel send operations. Each channel operation involves locking and unlocking, which can be costly, especially in a high-frequency context. Iteration 2: Batch processing of lines#</a></h2> In this version we are Batching the lines before sending to the worker which will significantly reduce the overhead of channel communication. Batch Processing: Each batch contains batchSize</code> lines. This reduces the frequency of channel operations (both sending and receiving), as well as the overhead associated with these operations. </li> Efficient Worker Utilization: With batch processing, each worker goroutine spends more time processing data and less time interacting with channels. This reduces the overhead of context switching and synchronization, making the processing more efficient. </li> </ol> const ( batchSize = 1000000 // Number of lines per batch ) // ... scanner := bufio.NewScanner(file) var batch []string for scanner.Scan() { batch = append(batch, scanner.Text()) if len(batch) >= batchSize { batchesChan <- batch batch = nil // Start a new batch } } // Send any remaining lines in the last batch if len(batch) > 0 { batchesChan <- batch } close(batchesChan) // ... func worker(batchesChan <-chan []string, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) for batch := range batchesChan { for _, line := range batch { // Process the line ... } } resultsChan <- stationStats } // ...</code></pre>Results#</a></h3> The improvement from iteration 2 to iteration 3 is quite remarkable, thanks to efficiently batching the lines together and reducing the number of channel ops. go run main.go 30.02s user 0.67s system 476% cpu 6.442 total</code></pre> So far, we’ve reduced the time to about 6.5s which is a great start and improvement of our baseline version of 19s. However, we’re making quite a few extra memory allocations and the focus of next iteration should be to reduce that. Iteration 3: Reducing memory allocations#</a></h2> A batch slice is pre-allocated with a capacity of batchSize</code> and reused for each batch of lines.</li> After sending a batch to the channel, the slice is reset to zero length (batch = batch[:0]</code>), but the underlying array is retained and reused.</li> </ul> // Read the file and send batches of lines to the workers go func() { file, err := os.Open(measurementsFile) if err != nil { panic(err) } defer file.Close() scanner := bufio.NewScanner(file) batch := make([]string, 0, batchSize) // Pre-allocate with capacity for scanner.Scan() { line := scanner.Text() // Reuse the batch slice by appending to it until it reaches the batch size batch = append(batch, line) if len(batch) >= batchSize { batchesChan <- batch batch = batch[:0] // Reset the slice without allocating new memory } } // Send any remaining lines in the last batch if len(batch) > 0 { batchesChan <- batch } close(batchesChan) }()</code></pre>Results#</a></h3> Down to 5.3s! go run main.go 25.43s user 0.53s system 485% cpu 5.346 total</code></pre>Iteration 3 (cont): Further reducing memory allocations#</a></h2> Avoiding strings.Split</code>: Instead of using strings.Split</code>, which allocates a new slice for each line, we can use strings.Index</code> to find the delimiter and manually slice the string. strings.Split</code> typically creates a new slice for each split part, leading to more memory usage and subsequent GC overhead.</li> </ul> for batch := range batchesChan { for _, line := range batch { delimiterIndex := strings.Index(line, ";") if delimiterIndex == -1 { continue // Delimiter not found, skip this line } station := line[:delimiterIndex] tempStr := line[delimiterIndex+1:] temp, err := strconv.ParseFloat(tempStr, 64) if err != nil { continue // Invalid temperature value, skip this line } stats := stationStats[station] stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[station] = stats } }</code></pre>Results#</a></h3> The time has further decreased from 5.3s to 4.8s with these changes. go run main.go 15.69s user 0.44s system 332% cpu 4.853 total</code></pre>Iteration 4: Read file in chunks#</a></h2> In this version, the file is read in chunks, and each chunk is processed to ensure it contains complete lines. The processChunk</code> function is used to separate valid data from leftover data in each chunk. Chunk size can be controlled with command line args as well. func main() { // .... const chunkSize = 256 * 1024 // 256 KB buf := make([]byte, chunkSize) leftover := make([]byte, 0, chunkSize) go func() { for { bytesRead, err := file.Read(buf) if bytesRead > 0 { // Copy the chunk to a new slice, because the // buffer will be reused in the next iteration. chunk := make([]byte, bytesRead) copy(chunk, buf[:bytesRead]) // Process the chunk. The returned leftover will be processed in the next iteration. validChunk, newLeftover := processChunk(chunk, leftover) leftover = newLeftover // Send the valid chunk to the processing goroutine. if len(validChunk) > 0 { wg.Add(1) go processChunkData(validChunk, resultsChan, &wg) } } if err != nil { break } } wg.Wait() close(resultsChan) }() // ... } func processChunk(chunk, leftover []byte) (validChunk, newLeftover []byte) { firstNewline := -1 lastNewline := -1 // Find the first and last newline in the chunk. for i, b := range chunk { if b == '\n' { if firstNewline == -1 { firstNewline = i } lastNewline = i } } if firstNewline != -1 { validChunk = append(leftover, chunk[:lastNewline+1]...) newLeftover = make([]byte, len(chunk[lastNewline+1:])) copy(newLeftover, chunk[lastNewline+1:]) } else { newLeftover = append(leftover, chunk...) } return validChunk, newLeftover } func processChunkData(chunk []byte, resultsChan chan<- map[string]Stats, wg *sync.WaitGroup) { defer wg.Done() stationStats := make(map[string]Stats) scanner := bufio.NewScanner(strings.NewReader(string(chunk))) for scanner.Scan() { line := scanner.Text() // Find the index of the delimiter delimiterIndex := strings.Index(line, ";") if delimiterIndex == -1 { continue // Delimiter not found, skip this line } // Extract the station name and temperature string station := line[:delimiterIndex] tempStr := line[delimiterIndex+1:] // Convert the temperature string to a float temp, err := strconv.ParseFloat(tempStr, 64) if err != nil { continue // Invalid temperature value, skip this line } // Update the statistics for the station stats, exists := stationStats[station] if !exists { stats = Stats{Min: temp, Max: temp} } stats.Count++ stats.Min = min(stats.Min, temp) stats.Max = max(stats.Max, temp) stats.Mean += (temp - stats.Mean) / float64(stats.Count) stationStats[station] = stats } // Send the computed stats to resultsChan resultsChan <- stationStats }</code></pre> In addition to this, I moved the aggregateStats</code> to a separate Goroutine as well: aggWg.Add(1) finalResults := make(map[string]Stats) // Start a separate goroutine for aggregation go func() { defer aggWg.Done() for workerResult := range resultsChan { for station, stats := range workerResult { finalStats, exists := finalResults[station] if !exists { finalResults[station] = stats continue } finalStats.Min = min(finalStats.Min, stats.Min) finalStats.Max = max(finalStats.Max, stats.Max) totalCount := finalStats.Count + stats.Count finalStats.Mean = (finalStats.Mean*float64(finalStats.Count) + stats.Mean*float64(stats.Count)) / float64(totalCount) finalStats.Count = totalCount finalResults[station] = finalStats } } }()</code></pre>Results#</a></h3> We’re down from 4.8s to just 2.1s to read/parse/process 100mn lines! ./bin/1brc.bin --file=input.txt --chunksize=1048576 17.58s user 0.77s system 837% cpu 2.190 total</code></pre>Summary#</a></h2> Basic File Reading and Parsing (Baseline): Time: 19s (baseline).</li> Key Change: Sequentially reading and processing each line.</li> Speedup: N/A (baseline).</li> </ul> </li> Producer-Consumer Pattern: Time: 54.225s.</li> Key Change: Implemented concurrent line processing with producer-consumer pattern.</li> Speedup: -185% (slower than baseline).</li> </ul> </li> Batch Processing of Lines: Time: 6.442s.</li> Key Change: Batched lines before processing, reducing channel communication.</li> Speedup: +66% (compared to baseline).</li> </ul> </li> Reducing Memory Allocations - Iteration 1: Time: 5.346s.</li> Key Change: Reused batch slices and reduced memory allocations.</li> Speedup: +72% (compared to baseline).</li> </ul> </li> Reducing Memory Allocations - Iteration 2 (Avoiding strings.Split</code>): Time: 4.853s.</li> Key Change: Replaced strings.Split</code> with manual slicing for efficiency.</li> Speedup: +75% (compared to baseline).</li> </ul> </li> Read File in Chunks: Time: 2.190s.</li> Key Change: Processed file in chunks and optimized aggregation.</li> Speedup: +87% (compared to baseline).</li> </ul> </li> </ol> Final Run#</a></h2> I’m quite satisfied with the final version for now. We can now proceed to test it with 1 billion lines. However it’s evidently CPU-bound, as we spawn N workers for N CPUs. I experimented with different chunk sizes, and here are the results from each run: Chunk Size</th> Time</th></tr></thead> 512.00 KB</td> 23.756s</td></tr> 1.00 MB</td> 21.798s</td></tr> 16.00 MB</td> 20.693s</td></tr> 32.00 MB</td> 19.501s</td></tr> </tbody></table> Tweaking the chunk size doesn’t significantly impact performance, as processing larger chunks takes longer. TL;DR: On an average and with multiple runs it takes approx 20s with the final iteration for 1bn lines. Checkout the full code on my GitHub</a>. Potential Improvements#</a></h2> This project was not only fun but also a great opportunity to revisit and refine many Go concepts. There are several ideas to contemplate for further improving this version’s timings: I haven’t yet considered using mmap</code>, but I believe it could substantially speed things up.</li> To delve even deeper, custom line parsing functions, especially for converting string</code> to float64</code>, could offer improvements.</li> Employing custom hashing functions (perhaps FnV) might aid in faster map lookups.</li> </ul> Fin! Making sad servers happy 2023-11-06T00:00:00+00:00 Introduction to SadServers#</a></h2> Recently, I stumbled upon sadservers</a>, a platform described as “Like LeetCode for Linux”. The premise is: you are given access to a full remote Linux server with a pre-configured problem. Your mission is to diagnose and fix the issues in a fixed time window. With the goal of documenting my journey through these challenges and sharing the knowledge gained, I decided to not only tackle these puzzles but also to record my solutions in a video format. The format is twofold in its purpose: it allows me to reflect on my problem-solving approach and provides a resource for others who may encounter similar problems, whether in real-world scenarios or in preparation for an SRE/DevOps interview. The Learning Curve#</a></h2> Each server presented a different issue, from misconfigured network settings to services failing to start, from permission issues to resource overutilization. One server, for instance, had a failing database service because of a disk full partition. The cause? Stale backup files. Another had a web server throwing errors because of incorrect file permissions. Recording the Solutions#</a></h2> The video recordings start with an introduction to the problem and my initial thoughts. Viewers can see my screen as I work through the issue, making the troubleshooting process transparent and educational. The commentary explains my thought process, the tools/CLI utilities used, and the solutions applied. Part 1#</a></h3> </iframe> Part 2#</a></h3> </iframe> Part 3#</a></h3> </iframe> Conclusion#</a></h2> For those looking to enhance their Linux troubleshooting skills, sadservers.com is a gold mine. It’s an excellent preparation ground for anyone aiming to step into the SRE/DevOps field or wanting to keep their skills sharp. As I continue to record and share these troubleshooting escapades, I invite you to subscribe, comment with your insights, or even suggest what types of challenges you’d like to see addressed next. Nomad can do everything that K8s can 2023-07-23T05:34:47+00:00 This blog post is ignited by the following Twitter exchange</a>: I don’t take the accusation of unsubstantiated argument, especially on a technical topic lightly. I firmly believe in substantiated arguments and hence, here I am, elaborating on my stance. If found mistaken, I am open to corrections and revise my stance. Some Historical Context#</a></h3> In my professional capacity, I have run and managed several K8s clusters (using AWS EKS) for our entire team of devs (been there done that). The most complex piece of our otherwise simple and clean stack was K8s and we’d been longing to find a better replacement. None of us knew whether that would be Nomad or anything else. But we took the chance and we have reached a stage where we can objectively argue that, for our specific workloads, Nomad has proven to be a superior tool compared to K8s. Building with Nomad#</a></h2> Nomad presents a fundamental building block approach to designing your own services. It used to be true that Nomad was primarily a scheduler, and for serious production workloads, you had to rely on Consul for service discovery and Vault for secret management. However, this scenario has changed as Nomad now seamlessly integrates these features, making them first-class citizens in its environment. Our team replaced our HashiCorp stack with just Nomad, and we never felt constrained in terms of what we could accomplish with Consul/Vault. While these tools still hold relevance for larger clusters managed by numerous teams, they are not necessary for our use case. Deconstructing Infrastructure#</a></h2> Kubernetes employs a declarative state for every operation in the cluster, essentially operating as a reconciliation mechanism to keep everything in check. In contrast, Nomad requires dealing with fewer components, making it appear lacking compared to K8s’s concept of everything being a “resource.” However, that is far from the truth. Ingress: We run a set of HAProxy on a few nodes which act as “L7 LBs”. Configured with Nomad services, they can do the routing based on Host headers.</li> DNS: To provide external access to a service without using a proxy, we developed a tool</a> that scans all services registered in the cluster and creates a corresponding DNS record on AWS Route53.</li> Monitoring: Ah my fav. You wanna monitor your K8s cluster. Sure, here’s kube-prometheus</a>, prometheus-operator</a>, kube-state-metrics</a>. Choices, choices. Enough to confuse you for days. Anyone who’s ever deployed any of these, tell me why this thing needs such a monstrosity setup of CRDs and operators. Monitoring Nomad is such a breeze, 3 lines of HCL</a> config and done.</li> Statefulsets: It’s 2023 and the irony is rich - the recommended way to run a database inside K8s is… not to run it inside K8s at all. In Nomad, we run a bunch of EC2 instances and tag them as db</code> nodes. The DBs don’t float around as containers to random nodes. And there’s no CSI plugin reaching for a storage disk in AZ-1 when the node is basking in AZ-2. Running a DB on Nomad feels refreshingly like running it on an unadorned EC2 instance.</li> Autoscale: All our client nodes (except for the db</code> nodes) are ephemeral and part of AWS’s Auto Scaling Groups (ASGs). We use ASG rules for the horizontal scaling of the cluster. While Nomad does have its own autoscale, our preference is to run large instances dedicated to specific workloads, avoiding a mix of different workloads on the same machine.</li> </ul> Over abstraction of Kubernetes#</a></h2> One of my primary critiques of K8s is its hidden complexities. While these abstractions might simplify things on the surface, debugging becomes a nightmare when issues arise. Even after three years of managing K8s clusters, I’ve never felt confident dealing with databases or handling complex networking problems involving dropped packets. You might argue that it’s about technical chops, which I won’t disagree with - but then do you want to add value to the business by getting shit done or do you want to be the resident K8s whiz at your organization? Consider this: How many people do you know who run their own K8s clusters? Even the K8s experts themselves preach about running prod clusters on EKS/GKE etc. How many fully leverage all that K8s has to offer? How many are even aware of all the network routing intricacies managed by kube-proxy? If these queries stir up clouds of uncertainty, it’s possible you’re sipping the Kubernetes Kool-Aid without truly comprehending the recipe, much like I found myself doing at one point Nomad: Not Perfect, But Simpler#</a></h2> Now, if you’re under the impression that I’m singing unabashed praises for Nomad, let me clarify - Nomad has its share of challenges. I’ve personally encountered and reported several. However, the crucial difference lies in Nomad’s lesser degree of abstraction, allowing for a comprehensive understanding of its internals. For instance, we encountered service reconciliation issues</a> with a particular Nomad version. However, we could query the APIs, identify the problem, and write a bash script to resolve and reconcile it. It wouldn’t have been possible when there are too many moving parts in the system and we don’t know where to even begin debugging. The YAML hell</a> is all too well known to all of us. In K8s, writing job manifests required a lot of effort (by the developers who don’t work with K8s all day) and were very complex to understand. It felt “too verbose” and involved copy pasting large blocks from the docs and trying to make things work. Compare that to HCL, it feels much nicer to read and shorter. Things are more straightforward to understand. I’ve not even touched upon the nice-ities on Nomad yet. Like better humanly understandable ACLs? Cleaner and simpler job spec, which defines the entire job in one file? A UI which actually shows everything about your cluster, nodes, and jobs? Not restricting your workloads to be run as Docker containers? A single binary which powers all of this? The central question this post aims to raise is: What can K8s do that Nomads can’t, especially considering the features people truly need? My perspectives are informed not only by my organization but also through interactions with several other organizations at various meetups and conferences. Yet, I have rarely encountered a use case that could only be managed by K8s. While Nomad isn’t a panacea for all issues, it’s certainly worth a try. Reducing the complexity of your tech stack can prove beneficial for your applications and, most importantly, your developers. At this point, K8s enjoys immense industry-wide support, while Nomad remains the unassuming newcomer. This contrast is not a negative aspect, per se. Large organizations often gravitate towards complexity and the opportunity to engage more engineers. However, if simplicity were the primary goal, the prevailing sense of overwhelming complexity in the infrastructure and operations domain wouldn’t be as pervasive. Conclusion#</a></h2> I hope my arguments provide a more comprehensive perspective and address the earlier critique of being unsubstantiated. Update#</a></h2> INFO Darren has responded to this blog post. You can read the response on Twitter</a>. </blockquote> Fin! Storing AWS Pinpoint Logs 2023-07-21T05:24:06+00:00 At $dayjob, we use AWS Pinpoint to send out SMS to our customers. We’ve also written a detailed blog post</a> on how we use Clickhouse</a> + vector</a> stack for our logging needs. We additionally wanted to store the delivery logs generated by the Pinpoint service. But like with anything else in AWS, even simpler tasks like these usually tend to piggyback on other counterparts of AWS - in this case, it happens to be AWS Kinesis. All the delivery logs which contain metadata about SMS delivery are streamed to Kinesis. Our setup involves configuring Pinpoint with Amazon Kinesis Data Firehose stream. Firehose is an ETL service that helps stream events to other persistent stores. Firehose supports multiple such output sinks and in our case we use HTTP</code> sink. This is what the flow looks like: Pinpoint -> Kinesis Firehose -> Vector HTTP -> Clickhouse</code> Ingesting Data#</a></h2> On the HTTP server side, we used vector</code>’s aws_kinesis_firehose</a> source. Compared to just using http</a> source, here are the differences I found: Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value X-Amz-Firehose-Access-Key</code> header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the http</code> source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether X-Amz-Firehose-Access-Key</code> is present in the headers and do a value comparison with our key. </li> Has native support for base64</code> decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the http</code> source. So, basically, this is how the server receives the payload: { "requestId": "6a14a06b-6eae-4218-...", "timestamp": 1689766125971, "records": [ { "data": "eyJld..." }, { "data": "eyJldmVudF9..." } ] }</code></pre> The value of the payload is a base64 encoded value of the JSON Object</a> of an SMS event. However, the aws_kinesis_firehose</code> source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using aws_kinesis_firehose</code> source: { "message": "{\"event_type\":\"_SMS.SUCCESS\",\"event_timestamp\":1689827914426,\"arrival_timestamp\":1689827917659,\"event_version\":\"3.1\",\"application\":{\"app_id\":\"redacted\",\"sdk\":{}},\"client\":{\"client_id\":\"redacted\"},\"device\":{\"platform\":{}},\"session\":{},\"attributes\":{\"sender_request_id\":\"redacted\",\"destination_phone_number\":\"+91xxx\",\"record_status\":\"DELIVERED\",\"iso_country_code\":\"IN\",\"mcc_mnc\":\"xxx\",\"number_of_message_parts\":\"1\",\"message_id\":\"redacted\",\"message_type\":\"Transactional\",\"origination_phone_number\":\"redactedORG\"},\"metrics\":{\"price_in_millicents_usd\":xx.0},\"awsAccountId\":\"redacted\"}\n", "request_id": "6dd45388-xxx", "source_arn": "arn:aws:firehose:ap-south-1:redacted:deliverystream/redacted", "source_type": "aws_kinesis_firehose", "timestamp": "2023-07-20T04:39:38.772Z" }</code></pre> This makes it straightforward because now we just have to parse the JSON inside the message</code> key and do transformations on that object. If it was http</code> source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least. </li> </ul> Here’s the vector config so far: [sources.firehose] # General type = "aws_kinesis_firehose" address = "127.0.0.1:9000" store_access_key = false access_keys = ["superdupersecret"] # Use it for debugging [sinks.console] type = "firehose" inputs = ["format_pinpoint_logs"] encoding.codec = "json" </code></pre>Formatting the data#</a></h2> Now that we have a pipeline which sends and receives data, we can process the events and transform them into a schema that is more desirable. Since we require the events to be queryable in a Clickhouse DB, this is the schema we have: CREATE TABLE default.pinpoint_logs ( `_timestamp` DateTime('Asia/Kolkata'), `app_id` LowCardinality(String), `event_type` LowCardinality(String), `record_status` LowCardinality(String), `origination_phone_number` String, `message_id` String, `destination_phone_number` String, `arrival_timestamp` DateTime('Asia/Kolkata'), `event_timestamp` DateTime('Asia/Kolkata'), `meta` Nullable(String) ) ENGINE = MergeTree PARTITION BY toYYYYMM(_timestamp) ORDER BY _timestamp SETTINGS index_granularity = 8192;</code></pre> To achieve the above format, we can use VRL to parse and format our SMS events: [transforms.format_pinpoint_logs] type = "remap" inputs = ["firehose"] source = ''' # Decode the JSON message and set ingestion timestamp .message = parse_json!(.message) .ingestion_timestamp = .timestamp # Convert timestamps from Unix to DateTime .event_timestamp = from_unix_timestamp!(.message.event_timestamp, unit:"milliseconds") .arrival_timestamp = from_unix_timestamp!(.message.arrival_timestamp, unit:"milliseconds") # Extract keys to top level and remove from attributes .record_status = del(.message.attributes.record_status) .origination_phone_number = del(.message.attributes.origination_phone_number) .destination_phone_number = del(.message.attributes.destination_phone_number) .message_id = del(.message.attributes.message_id) # Encode the remaining attributes as JSON string .attr = encode_json(.message.attributes) # Format Payload for Clickhouse . = { "_timestamp": .ingestion_timestamp, "arrival_timestamp": .arrival_timestamp, "event_timestamp": .event_timestamp, "app_id": .message.application.app_id, "event_type": .message.event_type, "record_status": .record_status, "message_id": .message_id, "origination_phone_number": .origination_phone_number, "destination_phone_number": .destination_phone_number, "meta": .attr } ''' </code></pre> Plugging this, we have a clean JSON object for each SMS event. The only thing now we need to add is an output sink to Clickhouse: [sinks.clickhouse] type = "clickhouse" inputs = ["format_pinpoint_logs"] skip_unknown_fields = true compression = "gzip" database = "default" endpoint = "http://127.0.0.1:8123" table = "pinpoint_logs" encoding.timestamp_format = "unix" batch.max_bytes = 1049000 # 1 MB batch.timeout_secs = 5 buffer.max_size = 268435488 buffer.type = "disk" buffer.when_full = "block"</code></pre> Perfect! On running this pipeline with vector -c config.toml</code> we can see the consumption the records Hope this short post was useful if you’ve to do anything similar! Fin! Bridge Networking in Nomad 2023-07-17T05:45:04+00:00 To set the stage, it’s crucial to understand what we mean by “bridge networking”. In a nutshell, it is a type of network connection in Linux that allows virtual interfaces, like the ones used by virtual machines and containers, to share a physical network interface. With Nomad, when a task is allocated, it creates a network namespace with its own network stack. Within this, a virtual ethernet (veth) pair is established, one end of which is assigned to the network namespace of the allocation, and the other remains in the host namespace. The Network Journey#</a></h2> To illustrate this practically, let’s assume a packet is sent from a task within an allocation. The packet would first be received by the local end of the veth pair, it would then traverse to the other end residing in the host’s namespace. From there, it is sent to the bridge on the host (in this case, the “nomad” bridge), which finally sends the packet out to the world via the host’s physical network interface (typically “eth0” or equivalent in your machine). The journey of a packet from the outside world to a task inside an allocation is the exact mirror image. The packet reaches “eth0” first, then the nomad bridge, it is then forwarded to the appropriate veth interface in the host’s namespace. From there, it crosses over to the other end of the veth pair in the allocation’s network namespace and finally gets routed to the destination task. To bridge or not to#</a></h2> Let’s take a look at the following jobspec which is for deploying my tiny side project - Cloak</a> on Nomad job "cloak" { datacenters = ["dc1"] type = "service" group "redis" { network { mode = "host" port "redis" { to = 6379 } } service { name = "cloak-redis" port = "redis" provider = "nomad" } task "redis" { driver = "docker" config { image = "redis:7" advertise_ipv6_address = false ports = [ "redis", ] volumes = [ "/data/cloak/redis:/data", ] } resources { cpu = 500 # MHz memory = 256 # MB } } } group "cloak" { network { mode = "host" port "cloak" { static = 7000 to = 7000 } } task "cloak" { driver = "docker" config { image = "ghcr.io/mr-karan/cloak:v0.2.0" command = "--config=config.toml" ports = [ "cloak", ] } template { data = <<EOH # Configuration for 1 redis instances, as assigned via rendezvous hashing. {{$allocID := env "NOMAD_ALLOC_ID" -}} {{range nomadService 1 $allocID "cloak-redis"}} CLOAK_REDIS__address={{ .Address }}:{{ .Port }} {{- end}} EOH destination = "secrets/file.env" env = true } resources { cpu = 500 # MHz memory = 700 # MB } } } }</code></pre> Our focus should be on the network.mode</code> stanza. To illustrate what happens behind the scenes when an alloc runs in network.mode=host</code> (host network), we can run the above job. On the machine, we can see that port 7000 (static) and port 27042 (dynamic) are allocated on the host network interface (eth0): We can also see the port and process details using ss</code>: sudo ss -ltpn 'sport = :7000' State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 4096 95.216.165.210:7000 0.0.0.0:* users:(("docker-proxy",pid=67068,fd=4))</code></pre> This config is more suitable for specific workloads - like load balancers or similar deployments where you want to expose the network interface on the host. It’s also helpful for applications running outside of Nomad on that host to connect via the host network interface. However, typically in a job where you want to connect to multiple different allocs - you’d want to set up a bridge network. This generally avoids exposing the workload on the host network directly. It’s a typical setup where you want to put applications behind a reverse proxy (NGINX/Caddy). Let’s change network.mode=bridge</code> in the above job spec and see the changes. $ nomad job plan cloak.nomad +/- Job: "cloak" +/- Task Group: "cloak" (1 create/destroy update) + Network { Hostname: "" + MBits: "0" + Mode: "bridge" + Static Port { + HostNetwork: "default" + Label: "cloak" + To: "7000" + Value: "7000" } } - Network { Hostname: "" - MBits: "0" - Mode: "host" - Static Port { - HostNetwork: "default" - Label: "cloak" - To: "7000" - Value: "7000" } } Task: "cloak" +/- Task Group: "redis" (1 create/destroy update) + Network { Hostname: "" + MBits: "0" + Mode: "bridge" + Dynamic Port { + HostNetwork: "default" + Label: "redis" + To: "6379" } } - Network { Hostname: "" - MBits: "0" - Mode: "host" - Dynamic Port { - HostNetwork: "default" - Label: "redis" - To: "6379" } } Task: "redis"</code></pre> Now we don’t see the ports forwarded on the host network: Similarly, ss</code> also shows no process listening on the host network IPTables and Routing#</a></h2> To understand what happened when we switched the networking mode to bridge</code>, we need to take a look at the Nomad iptables</code> magic which comes into play when using bridge</code> network. I pulled up the iptables</code> and saw specific rules under the chains CNI-FORWARD</code> and NOMAD-ADMIN</code>. These rules, in essence, allow all traffic to and from the allocation’s network namespace. $ sudo iptables -L CNI-FORWARD Chain CNI-FORWARD (1 references) target prot opt source destination NOMAD-ADMIN all -- anywhere anywhere /* CNI firewall plugin admin overrides */ ACCEPT all -- anywhere 172.26.64.5 ctstate RELATED,ESTABLISHED ACCEPT all -- 172.26.64.5 anywhere ACCEPT all -- anywhere 172.26.64.6 ctstate RELATED,ESTABLISHED ACCEPT all -- 172.26.64.6 anywhere sudo iptables -L NOMAD-ADMIN Chain NOMAD-ADMIN (1 references) target prot opt source destination ACCEPT all -- anywhere 172.26.64.0/20</code></pre> Nomad uses 172.26.64.0/20</code> as the default subnet</a> for the bridge network. The IPs 172.26.64.5</code> and 172.26.64.6</code> are assigned to 2 different allocs in this CIDR. The iptables</code> rules allow complete traffic to flow on this subnet. To check the routing,ip route</code> command can be used. $ ip route show 172.26.64.0/20 172.26.64.0/20 dev nomad proto kernel scope link src 172.26.64.1</code></pre> It uses the nomad</code> network interface for routing packets related to the default bridge network. Using nsenter</code> we can find more details about the network namespace created for an alloc. Let’s find details about the redis</code> alloc: sudo nsenter -t $(pgrep redis) --net ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0@if113: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 76:47:6d:49:00:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.26.64.5/20 brd 172.26.79.255 scope global eth0 valid_lft forever preferred_lft forever </code></pre> We can see that one end of the pair is eth0</code> (container’s default gateway) which is connected to a network interface with an index 113</code>. For the tunnel to actually work, the veth pair should also exist on the host: $ ip a 113: veth3402deda@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master nomad state UP group default link/ether 3a:85:1b:37:75:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::3885:1bff:fe37:7517/64 scope link valid_lft forever preferred_lft forever</code></pre> So, when we see veth3402deda@if2</code> in the host’s network namespace (with the index 113</code>), and then we see eth0@if113</code> inside the Redis container, we can infer that these two interfaces form a veth</code> pair: veth3402deda@if2</code> on the host side and eth0</code> inside the container. This connection enables the container to communicate with the external network through the host’s network stack. Capturing packets#</a></h2> We can capture TCP packets on the veth</code> interface to see the routing work: sudo tcpdump -i veth971858d5 -n tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on veth971858d5, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:51:27.801319 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [S], seq 1331933249, win 65495, options [mss 65495,sackOK,TS val 248300785 ecr 0,nop,wscale 7], length 0 10:51:27.801549 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [S.], seq 107697826, ack 1331933250, win 65160, options [mss 1460,sackOK,TS val 3965422857 ecr 248300785,nop,wscale 7], length 0 10:51:27.801616 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 1, win 512, options [nop,nop,TS val 248300785 ecr 3965422857], length 0 10:51:27.801737 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [P.], seq 1:79, ack 1, win 512, options [nop,nop,TS val 248300786 ecr 3965422857], length 78 10:51:27.801751 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [.], ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 0 10:51:27.802022 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 1:4097, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 4096 10:51:27.802059 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 4097, win 491, options [nop,nop,TS val 248300786 ecr 3965422858], length 0 10:51:27.802120 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [P.], seq 4097:5396, ack 79, win 509, options [nop,nop,TS val 3965422858 ecr 248300786], length 1299 10:51:27.802135 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5396, win 502, options [nop,nop,TS val 248300786 ecr 3965422858], length 0 10:51:27.803484 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [F.], seq 79, ack 5396, win 512, options [nop,nop,TS val 248300787 ecr 3965422858], length 0 10:51:27.803567 IP 172.26.64.6.7000 > 172.26.64.1.35826: Flags [F.], seq 5396, ack 80, win 509, options [nop,nop,TS val 3965422859 ecr 248300787], length 0 10:51:27.803597 IP 172.26.64.1.35826 > 172.26.64.6.7000: Flags [.], ack 5397, win 512, options [nop,nop,TS val 248300787 ecr 3965422859], length 0 10:53:08.523431 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 2169295212, win 501, options [nop,nop,TS val 735542538 ecr 4133067854], length 0 10:53:08.523551 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0 10:53:08.523554 IP 95.216.165.210.27372 > 172.26.64.6.53042: Flags [.], ack 1, win 509, options [nop,nop,TS val 4133379150 ecr 735231242], length 0 10:53:08.523562 IP 172.26.64.6.53042 > 95.216.165.210.27372: Flags [.], ack 1, win 501, options [nop,nop,TS val 735542538 ecr 4133379150], length 0 </code></pre> To summarize the output, we can see that the log is showing a TCP connection between 172.26.64.1 (source) and 172.26.64.6 (destination), specifically on port 7000. 172.26.64.1</code> happens to be the gateway for nomad</code> subnet. Summary#</a></h2> Hope this post clarified some networking internals and behind the scenes magic when using Nomad bridge networking. Refer to my other post - Nomad networking explained</a> for a practical breakdown of all the different ways to expose and connect applications in a Nomad cluster. Fin! Analyzing credit card transactions with GPT and Python 2023-04-30T07:08:02+00:00 You know those budget freaks? People who log and categorise every Rupee they’ve spent over the month? The financially sane people? I am definitely not one and I suck at it. I moved cities a couple of months back and had some big ticket spends off late, mostly financed by credit card. I wanted an easy way to list down where all I’ve spent most money and spot some recurring expenses so I can be better prepared for them from next month. I’ve found that broadly keeping an idea of things where you spend money works for me (v/s the two extremes - completely blind or logging every small transactions). Of course I know people who make budgeting a habit but I only wish I was consistent enough to do that. Anyway, I downloaded the statement in CSV format from my bank. Initially I thought I’d use some simple Excel to make sense of this but I realised how bad my Excel skills really are. I got an idea to dump the CSV file to ChatGPT (yay privacy) and ask questions. It kinda sucked at it and gave wrong answers for a lot of questions and also started to hallucinate data which wasn’t even present in the CSV. The next most obvious step would be to write a simple script and parse it. I wanted to experiment if ChatGPT could do this entire exercise of writing the script and the relevant code for the analysis I wanted to perform. Here’s the initial prompt I gave: I am pasting a CSV file which also includes the first row as header. Write Python code to parse this: ,,Transaction Date,Details,,,Amount (INR),,,Reference Number,, ,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,7477468633,, ... other data ...</code></pre> It returned the following code: import pandas as pd from io import StringIO csv_data = """\ ,,Transaction Date,Details,,,Amount (INR),,,Reference Number,, ,,27/04/2023,"Blinkit, BANGALORE, IN",,,290.00 Dr.,,,747746xxxx,, # ... (the rest of your CSV data) ,,03/04/2023,"DIGITAL LOUNGE, GURGAON, IN",,,799.00 Dr.,,,738201xxx,,""" # Read the CSV data into a pandas DataFrame data = StringIO(csv_data) df = pd.read_csv(data, header=1) # Clean up the DataFrame (remove unnecessary columns) df = df.dropna(axis=1, how="all") # Rename columns df.columns = [ "Transaction Date", "Details", "Amount (INR)", "Reference Number" ] print(df)</code></pre> Looking at this, I was a bit impressed as it figured the CSV contains some useless empty columns and it removed (without me giving any information about it). I also asked it to modify the code to read the file locally from disk and it swapped StringIO</code> with the path to CSV file: # Provide the file path to your local CSV file file_path = "your_file.csv" # Read the CSV data into a pandas DataFrame df = pd.read_csv(file_path, header=1)</code></pre> Next, I prompted to do some analysis on it: Write some more code to do the analysis to answer these questions: - How many total debit and credit entries - What is the total amount of debits and total amount of credits - Which vendor have I spent the most? - How much have I spent on `SWIGGY` ?</code></pre> It returned some one-liners to answer each question: # Filter debit and credit entries debit_entries = df[df["Amount (INR)"] > 0] credit_entries = df[df["Amount (INR)"] < 0] # Count the total number of debit and credit entries total_debit_entries = len(debit_entries) total_credit_entries = len(credit_entries) # Calculate the total amount of debits and credits total_debit_amount = debit_entries["Amount (INR)"].sum() total_credit_amount = credit_entries["Amount (INR)"].sum() # Find the vendor with the most spending most_spent_vendor = debit_entries.groupby("Details")["Amount (INR)"].sum().idxmin() # Calculate the total amount spent on SWIGGY swiggy_spending = debit_entries[debit_entries["Details"].str.contains("SWIGGY", case=False)]["Amount (INR)"].sum() # Calculate the total amount spent on AMAZON RECHARGES amazon_recharges_spending = debit_entries[debit_entries["Details"].str.contains("AMAZON RECHARGES", case=False)]["Amount (INR)"].sum() # Print the results print(f"Total debit entries: {total_debit_entries}") print(f"Total credit entries: {total_credit_entries}") print(f"Total debit amount: {total_debit_amount}") print(f"Total credit amount: {total_credit_amount}") print(f"Vendor with the most spending: {most_spent_vendor}") print(f"Total amount spent on SWIGGY: {swiggy_spending}")</code></pre> At this point, I know that this will fail because we’ve not cleaned up the data. The column Amount (INR)</code> needs cleaning up. I prompted ChatGPT to write a function to clean this column: The amount column as `21,620.54 Dr.` to represent that Rs 21,620.54 is debited. And `73,500.00 Cr.` to represent that Rs 73,500.00 is credited. Can you write a function to apply this transformation on the `Amount (INR)` column?</code></pre> It responded with: # Clean up the data def clean_amount(value): if ' Cr.' in value: value = value.replace(' Cr.', '') return -float(value.replace(',', '')) elif ' Dr.' in value: value = value.replace(' Dr.', '') return float(value.replace(',', '')) df["Amount (INR)"] = df["Amount (INR)"].apply(clean_amount) # Now proceed with the analysis code </code></pre> Perfect! After transforming the amounts, I ran the analysis code: The next prompt I gave was to analyse the spending in various categories. In the above CSV file can you guess the categories of each merchant based on the `Details`? And write the code for grouping the spends on various categories. If you're unable to figure out the category for the transaction, you can use a "Misc" column to bucket them.</code></pre> It actually did an okayish job at this and ignored a lot of vendors which I think it could have guessed easily: def categorize(details): details = details.lower() if "swiggy" in details: return "Food Delivery" elif "amazon recharges" in details: return "Recharge" elif "uber" in details or "ola" in details: return "Cab" elif "zomato" in details: return "Food Delivery" elif "netflix" in details or "prime video" in details: return "Streaming" else: return "Misc" # Create a new column called "Category" by applying the categorize function to the "Details" column df["Category"] = df["Details"].apply(categorize) # Group the transactions by category and calculate the total amount spent in each category grouped = df[df["Amount (INR)"] > 0].groupby("Category")["Amount (INR)"].sum() print(grouped)</code></pre> I decided to give some manual inputs to it to refine the function. I can help you with categorising it further. `Blink` - Grocery `Indigo` - Travel `PEGS` - Alcohol `Nike` - Shopping Also note that the cateorgy name matching should be case-insensitive.</code></pre> And it modified the Python snippet to add these rules. Mixed reactions looking at this. Happy that I could practically get exactly the result I had in mind in just 10 minutes without writing any code. Sad because damn I need to limit those empty calories from next month (famous last words). Next, I wanted to see if my spends on weekends are higher or not. (I don’t expect them to be, but you never know). Prompt: Can you also write the code to figure how much spends I've done grouped by the day of week?</code></pre> Response: # Convert the 'Transaction Date' column to a datetime format df['Transaction Date'] = pd.to_datetime(df['Transaction Date']) # Create a new column called "Day of Week" containing the day of the week for each transaction df['Day of Week'] = df['Transaction Date'].dt.day_name() # Group the transactions by the "Day of Week" and calculate the total amount spent on each day of the week grouped_by_day = df[df["Amount (INR)"] > 0].groupby('Day of Week')['Amount (INR)'].sum().abs() # Sort the grouped data by the total amount spent in descending order sorted_grouped_by_day = grouped_by_day.sort_values(ascending=False) print(sorted_grouped_by_day)</code></pre> It was a fun 10-15 min exercise to figure out my spending habits based on the last month’s statement. I intend to do this for the next couple of months and then it would make sense to write more queries which would show trend-lines of spends in various categories over time. Honestly, I just loved how ChatGPT made this task so seemingly simple. It’s not that I can’t myself write the code for these kind of simple analysis. It’s the sheer power at your hand to go from ideation phase to answer within seconds. And I think that’s why I love it so much. I didn’t have to go through Pandas docs (because I don’t use that in my day job so it’s quite normal to not know various syntax/functions that I could use) and I’d grok through different StackOverflow questions to achieve what I wanted to. And maybe imagining all this resistance on a Sunday morning would have meant that I never got to write the script in the first place. Fin! The curious case of missing and duplicate logs 2023-02-16T08:27:13+00:00 At work, we use a Vector</a> pipeline for processing and shipping logs to Clickhouse</a>. We also self-host our SMTP servers and recently started using Haraka SMTP</a>. While Haraka is excellent in raw performance and throughput, it needed an external logging plugin for audit and compliance purposes. I wrote haraka-plugin-outbound-logger</a> to log basic metadata like timestamps/subject/SMTP response in a JSON file. The plan was to dump these logs into a file and use Vector’s file source for reading them and doing the further transformation. However, things went differently than I had planned. There were mainly two issues propped up due to bad Vector configuration. Missing logs#</a></h3> The vector configuration to read the file looked like this: [sources.outbound_logger] type = "file" include = ["/var/log/haraka/outbound/*.log"] read_from = "beginning" # Remove the files after 24 hours. Vector must have permission to delete these files. remove_after_secs = 86400 fingerprint.strategy = "device_and_inode" </code></pre> Vector has a handy configuration of automagically deleting the file if the file hasn’t received any new write in the configured time interval. So remove_after_secs=86400</code> specifies if the file hasn’t had any new writes since 24h, it can delete it. It made sense to configure because our workload was a shorter process. It was a batch job done once every N days. When the file didn’t receive any new writes after 24h, vector</code> deleted the file as expected. However, the plugin continued logging into the same file handler, even for newer batch jobs. As a result, the file didn’t receive any new logs and was empty. I created a minimal POC to reproduce this seemingly strange issue: var pino = require('pino'); // Initialise pino js logger and inject in the plugin context. var opts = { name: 'outbound_logger', level: 'debug', // uses the ISO time format. timestamp: pino.stdTimeFunctions.isoTime, formatters: { level: (label) => { return { level: label }; }, }, } pilog = pino(opts, pino.destination(`${__dirname}/app.log`)) pilog.info('this is a first message') setTimeout(() => pilog.info('this message should get logged'), 10000) pilog.info('this message will be recorded as well')</code></pre> This snippet logs to app.log</code>. {"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this is a first message"} {"level":"info","time":"2023-02-16T06:45:04.031Z","pid":206573,"hostname":"pop-os","name":"outbound_logger","msg":"this message will be recorded as well"} </code></pre> During the 10s time interval, I deleted the file from the disk rm app.log</code> to mimic the behaviour of remove_after_secs</code>. I expected the file to get re-created and this message should get logged</code> logged by the above script. However, that didn’t happen. The script didn’t complain about a missing file, either. I was perplexed and did some google-fu and found the following via Stackoverflow</a>: The writes actually do not fail.When you delete a file that is open in another program you are deleting a named link to that file’s inode. The program that has it open still points to that inode. It will happily keep writing to it, actually writing to disk. Only now you don’t have a way to look it at, because you deleted the named reference to it. (If there were other references, e.g. hard links, you would still be able to!). </blockquote> This is exactly what was happening in production. When vector</code> deleted the file (as configured via remove_after_secs</code> ), the plugin didn’t know about it and kept writing to the same inode. This was a major TIL moment for me. Fix: The fix was simple enough; I removed remove_after_secs</code> from Vector’s config. To address the problem of the file not growing unbounded forever, I created a logrotate</code> config: /opt/app/logs/app.log { daily rotate 15 dateext dateformat -%Y-%m-%d.log delaycompress compress notifempty missingok copytruncate }</code></pre> Some notes: copytruncate</code> is useful in this context. It copies the existing file to a new one, which now becomes a stale one. The current (active) file will be truncated to zero bytes. E.g., if app.log</code> it is rotated, logrotate will copy the file to a new file app-2023-02-05.log</code> and then truncate the existing one to zero bytes.</li> delaycompress</code> will not compress the logs until the next rotation happens. This is useful if vector</code> hasn’t finished processing the log and can continue to do that.</li> </ul> Duplicate Logs#</a></h3> Now after fixing the case of missing logs, I found myself in the opposite problem - now the logs were duplicated on Clickhouse. My mental state at that moment couldn’t be more accurately described than this meme: To add more context, before developing my plugin for logging email delivery in Haraka, we used another plugin (acharkizakaria/haraka-plugin-accounting-files</a>) to get these logs. This plugin records the metadata to CSV files. Still, there were some issues in properly escaping the subject lines (if the subject had a comma, that was incorrectly parsed); hence, the log file had inconsistent output. To address these issues, I found writing another plugin from scratch that outputs to a fixed JSON schema is better. As seen above, vector</code>’s file source was configured like the following for reading CSV files. The only change here is that remove_after_secs</code> is gone after fixing issue #1. [sources.outbound_logger] type = "file" include = ["/var/log/haraka/outbound/*.log"] read_from = "beginning" fingerprint.strategy = "device_and_inode"</code></pre> Vector “fingerprints</a>” the file source, as it keeps the checkpoint of how many bytes it has read for each file in its own “disk buffer”. This buffer is helpful if Vector crashes so it can restart reading the file from where it last stopped. There are two strategies for fingerprinting that vector uses: The checksum</code> strategy uses CRC check on the first N lines of the file.</li> The device_and_inode strategy</code> uses the disk’s actual inode location to identify the file uniquely.</li> </ul> As I was using a different plugin which logged to a CSV file, the checksum strategy did not work in my context. Since vector fingerprints, the first few bytes (usually just enough for a header of a CSV), all the CSV files in that disk would have the same title, and Vector would not read all of them. To work around this, I changed the fingerprint.starategy = "device_and_inode"</code> so Vector uniquely identifies all CSV files by their inode path. (In hindsight, I should have just used checksum</code> with a higher value of fingerprint.lines</a> value.) The mistake this time was when I switched to a JSON log file, I continued with the device_and_inode</code> strategy. This isn’t a problem if there is no log rotation setup. Since I did configure logrotate</code> to fix issue #1, as you would have guessed, copytruncate</code> created another log file, and because I was using device_and_inode</code> strategy, vector thought this was a “new” file to be watched and processed. So now I had duplicate entries from this new file, which is technically just an older log-rotated file. The fix: [sources.outbound_logger.fingerprint] lines = 14 strategy = "checksum" ignored_header_bytes = 2048</code></pre> I switched back to the default checksum</code> strategy and adjusted the thresholds for lines/header bytes to account for JSON logs. The same is also documented</a> very clearly in Vector and it was my RTFM moment. This strategy avoids the common pitfalls associated with using device and inode names since inode names can be reused across files. This enables Vector to properly tail files across various rotation strategies. </blockquote> Phew! I am glad after these fixes; vector</code> is durably and reliably processing all the logs and logrotate</code> is happily working in conjunction as well. I hope documenting my learnings about these production issues would help someone with the same problems. Fin! Structured logging in Go with slog 2023-02-15T07:29:53+00:00 A few months ago, a proposal</a> for adding a structured logging library in Go was introduced by Jonathan Amsterdam</a>. At present, Go has a minimal and bare-bones log package which works all right for basic use cases. However, the current library has a few shortcomings that this proposal aims to solve: Emitting logs with different severity/levels</li> Structured output: Makes parsing of logs harder</li> Logging a set of common fields/attributes</li> Difficult to have a log object inside libraries as each service could have its log implementation.</li> </ul> As a result, many code bases have their wrappers around the log package. Additionally, there are plenty of 3rd party libraries to choose from - including logf</a> (which my work colleagues and I built at Zerodha). This article is about how to get started with slog</a> for logging in Go applications. NOTE Since slog is currently in the proposal state and hasn’t yet merged in the official library, the API could change in future. </blockquote> Architecture of slog#</a></h2> At a higher level, slog contains three main entities: Logger: The user-facing API for interacting with slog. All the public methods are defined on the Logger object.</li> Record: Contains information about the log event itself. A standard record will have timestamp, level and message fields as default. Additional attributes and metadata like caller info can be added to the Record.</li> Handlers: A handler is an interface implementation. The Logger object passes the Record to a handler, and the handler can choose whatever it wants to do with the Record. This is a common approach in Go libraries, where a “provider” can be abstracted in handling that task. Currently, slog ships with two handlers: JSON and logfmt</a>. Some projects have also created handlers for zap/logrus (popular 3rd party libraries).</li> </ul> Initialization#</a></h3> This snippet initializes a Text Handler, which produces logfmt</code> format messages on os.Stdout</code>. package main import ( "os" "golang.org/x/exp/slog" ) func main() { log := slog.New(slog.NewTextHandler(os.Stdout)) log.Info("Hello world") fakeErr := os.ErrNotExist log.Error("something went wrong", fakeErr, "file", "/tmp/abc.txt") } </code></pre> Log output: time=2023-02-15T19:58:10.615+05:30 level=INFO msg="Hello world" time=2023-02-15T19:58:10.615+05:30 level=ERROR msg="something went wrong" file=/tmp/abc.txt err="file does not exist"</code></pre>Customizing#</a></h3> You’ll notice that the caller information isn’t exposed by default. The reason could be that finding the stack trace of the calling line is a bit expensive operation. However, for libraries/apps which need it can do that by customizing the handler: func main() { handler := slog.HandlerOptions{AddSource: true} log := slog.New(handler.NewTextHandler(os.Stdout)) log.Info("Hello world") } </code></pre> Log Output: time=2023-02-15T12:17:53.742+05:30 level=INFO source=/home/karan/Code/Personal/slog-examples/main.go:14 msg="Hello world"</code></pre>Attributes#</a></h3> Sometimes, it’s helpful to append specific metadata to each log line which will help in aggregating/filtering with a central log-collecting agent. E.g., you can export a component key for each sub-service of your primary application. func main() { log := slog.New(slog.NewTextHandler(os.Stdout)).With("component", "demo") log.Info("Hello world") } </code></pre> Log Output: time=2023-02-15T12:21:50.231+05:30 level=INFO msg="Hello world" component=demo</code></pre>Nested Keys#</a></h3> So far, we’ve seen flat keys in the log message. It may be helpful to group together specific keys together and form a nested object. In JSON, that would mean a top-level object with different fields inside. However, in logfmt</code>, it would-be parent.child</code> format.To use nested keys, slog.Group</code> can be used. This example uses http</code> as the top-level key, and all its associated fields will be nested inside. log.Info("Hello world", slog.Group("http", slog.String("method", "GET"), slog.Int("status", 200), slog.Duration("duration", 250), slog.String("method", "GET"), slog.String("path", "/api/health"))) </code></pre>time=2023-02-15T12:30:43.130+05:30 level=INFO msg="Hello world" component=demo http.method=GET http.status=200 http.duration=250ns http.method=GET http.path=/api/health</code></pre>Configurable Handlers#</a></h3> JSON logs are daunting and tedious to read when locally developing applications. However, it’s a great fit for machine parsing of the logs. logfmt</code> hits the sweet spot for being machine parseable and human-readable.However, thanks to the powerful “interface” implementation approach, it’s easy to switch to any handler via user-configurable methods (like config files/env variables): package main import ( "os" "golang.org/x/exp/slog" ) func main() { var ( env = os.Getenv("APP_ENV") handler slog.Handler ) switch env { case "production": handler = slog.NewJSONHandler(os.Stdout) default: handler = slog.NewTextHandler(os.Stdout) } log := slog.New(handler) log.Info("Hello world") } </code></pre>$ go run main.go time=2023-02-15T12:39:45.543+05:30 level=INFO msg="Hello world" $ APP_ENV=production go run main.go {"time":"2023-02-15T12:39:53.523477544+05:30","level":"INFO","msg":"Hello world"} </code></pre>Summary#</a></h2> slog</code> is an excellent proposal, and it’s high time Go gets its official structured logging library. The API is designed to be easy to use, and a clear path is given for users wanting high-performance/zero-allocs by creating their handlers and making these performance improvements. Fin Building a CoreDNS plugin 2023-01-05T00:00:00+05:30 CoreDNS is an extensible DNS server (which is actually a fork of Caddy v1) that can be used to serve DNS records for a domain. It is written in Go and is very easy to extend. It has a plugin system that allows you to write your own plugins to extend its functionality. In this post, I will be writing a plugin for CoreDNS that will allow it to serve DNS records for Nomad services. I recently came across a niche use case</a> which required me to use a resolver address for querying Nomad services. Currently Nomad native service discovery is only possible via consul-template</code> (which renders static block of address/port of services) and HTTP API. I felt adding a DNS interface would be a nice add-on. Rather than writing and implementing all the boring crux of a DNS server, it’s better to extend on an existing server. CoreDNS fits in well here, it’s also used by K8s for service discovery. CoreDNS has an extensible plugin system which allows you to chain multiple different plugins for handling a request. Stuff like logs/metrics/caching comes for free with CoreDNS in form of a plugin. CoreDNS docs have a handy guide</a> on how to write a plugin from scratch, so I won’t cover that again here. These are just my short notes on how I locally developed the plugin, tested and some problems I encountered during the process. Developing the plugin#</a></h2> Firstly, you need to clone CoreDNS repo. Then, using the example</a> plugin provided, you can create a new repository for your own plugin. To make CoreDNS aware about this plugin, you need to add it to the plugin.cfg</code> file. This file is used by the build script to generate the plugin list. The order of plugins matter here as they define how the request is handled. For example, if you want to log all the requests, you need to add the log</code> plugin before your plugin. To add an external plugin this is the format used: nomad:github.com/mr-karan/coredns-nomad</code></pre> However, since we are developing the plugin locally, we need to add a replace</code> directive in go.mod</code> file to point to the local plugin directory. replace github.com/mr-karan/coredns-nomad => ../../coredns-nomad</code></pre> Next, you can run make</code> in coredns</code> repository. It’ll build the binary and place it in coredns</code> directory. You can run this binary to test your plugin. To check if the plugin indeed exists in the binary, you can use the following command ./coredns -plugins | grep nomad</code></pre>Handling requests#</a></h2> The ServeDNS</code> function is used to handle the DNS request by the plugin. It takes a context.Context</code> and a dns.ResponseWriter</code> as arguments. The dns.ResponseWriter</code> is used to write the response back to the client. The ServeDNS</code> function returns an int</code> which is the status code of the response. The status code is used by the next plugin in the chain to determine if it should handle the request or not. Since the nomad</code> plugin expects a query in format of service.namespace.nomad</code>, it validates the query and extracts the service name and namespace from it. If the query is invalid, it returns dns.RcodeServerFailure</code> status code. If the query is valid, it queries the Nomad API for the service and returns the response. func (n Nomad) ServeDNS(ctx context.Context, w dns.ResponseWriter, r *dns.Msg) (int, error) { state := request.Request{W: w, Req: r} qname := state.Name() qtype := state.QType() // Split the query name with a `.` as the delimiter and extract namespace and service name. // If the query is not for a Nomad service, return. qnameSplit := dns.SplitDomainName(qname) if len(qnameSplit) < 3 || qnameSplit[2] != "nomad" { return plugin.NextOrFailure(n.Name(), n.Next, ctx, w, r) } namespace := qnameSplit[1] serviceName := qnameSplit[0] ... </code></pre> The plugin handles A,AAAA and SRV record requests currently. Since A/AAAA records can only contain an IP address, SRV records can be used to advertise the port number. // Check the query type to format the appriopriate response. switch qtype { case dns.TypeA: m.Answer = append(m.Answer, &dns.A{ Hdr: header, A: addr, }) case dns.TypeAAAA: m.Answer = append(m.Answer, &dns.AAAA{ Hdr: header, AAAA: addr, }) ... </code></pre>Caching#</a></h3> While some coredns plugins have an in-built support for caching the records to avoid a lookup to Nomad server everytime (which can get expensive), I decided to skip the caching implementation. This is because coredns</code> itself has a cache</code> plugins which supports a lot of various options for controlling the cache. In my testing, just using this cache</code> plugin was sufficient to avoid Nomad lookups each time a query came in. Testing the plugin#</a></h2> I created a fake HTTP test server and added the URI paths which the Nomad Go client uses to query the Nomad API. This way I could test the plugin without having to run a Nomad cluster locally. // Setup a fake Nomad server. nomadServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { default: t.Errorf("Not implemented: %v", r.URL.Path) return case "/v1/service/example": w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":23202,"ServiceName":"example"}]`)) case "/v1/service/fakeipv6": w.Write([]byte(`[{"Address":"1:2:3::4","Namespace":"default","Port":8000,"ServiceName":"fakeipv6"}]`)) case "/v1/service/multi": w.Write([]byte(`[{"Address":"1.2.3.4","Namespace":"default","Port":25395,"ServiceName":"multi"},{"Address":"1.2.3.5","Namespace":"default","Port":20888,"ServiceName":"multi"},{"Address":"1.2.3.6","Namespace":"default","Port":26292,"ServiceName":"multi"}]`)) case "/v1/service/nonexistent": w.Write([]byte(`[]`)) } }))</code></pre>Usage Example#</a></h2> Here are some examples of how this plugin works. The Corefile I’ve used is: nomad:1053 { errors debug health log nomad { address http://127.0.0.1:4646 ttl 10 } prometheus :9153 cache 30 }</code></pre> On running coredns</code>, it connects to a local Nomad agent which is running at http://127.0.0.1:4646</code>. I’m running a redis</code> job in Nomad, so I can query the service using the following command: nomad service info -namespace=default redis Job ID Address Tags Node ID Alloc ID redis 192.168.29.76:25395 [] 9e02c85b 95170495 redis 192.168.29.76:20888 [] 9e02c85b a1cf923c redis 192.168.29.76:26292 [] 9e02c85b a9d1181a</code></pre> Now, the same query can also be handled using the DNS server running by coredns</code>: doggo redis.default.nomad @tcp://127.0.0.1:1053 NAME TYPE CLASS TTL ADDRESS NAMESERVER redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053 redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053 redis.default.nomad. A IN 10s 192.168.29.76 127.0.0.1:1053</code></pre> Quering an SRV record is also possible: dig +noall +answer +additional redis.default.nomad @127.0.0.1 -p 1053 SRV redis.default.nomad. 10 IN SRV 10 10 25395 redis.default.nomad. redis.default.nomad. 10 IN SRV 10 10 20888 redis.default.nomad. redis.default.nomad. 10 IN SRV 10 10 26292 redis.default.nomad. redis.default.nomad. 10 IN A 192.168.29.76 redis.default.nomad. 10 IN A 192.168.29.76 redis.default.nomad. 10 IN A 192.168.29.76</code></pre>Code#</a></h2> You can checkout the source code here</a>. Fin! Writing a disk-based key-value store in Golang 2022-12-18T00:00:00+05:30 I’d been mulling around reading a computer science paper and implementing a project based on it. Distributed systems, Networking and Databases are some of the things that fascinate me a lot. However, I had been looking to implement a more approachable project to avoid getting inundated initially. And I happened to chance upon the Bitcask paper</a> through Avinash’s project: CaskDB</a>. After giving a quick read of this reasonably short paper, I decided to write a Golang implementation of the same, as it looked like an exciting project. If you’re interested in checking out the complete project, checkout BarrelDB</a>. Bitcask is a disk-based key-value storage engine designed for fast read and write operations. It is mainly in production use by Riak (which is a distributed database) as one of the storage engines</a>. Bitcask under the hood has a straightforward yet clever design. It writes to the file in an append-only mode. This means that writes are performed only by appending to the end of the file, thus avoiding the need to perform any random disk I/O seek. Let’s look at various components of Bitcask: Format of the record#</a></h2> CRC: Stores the checksum of the value to ensure data consistency</li> Timestamp: Timestamp in UNIX format, stored as int32.</li> Expiry: If the record has an expiry defined, then the timestamp, in UNIX format, is stored as int32.</li> Key Size: Size of the key in bytes</li> Value Size: Size of the value in bytes</li> Key</li> Value</li> </ul> This additional metadata stored alongside the key/value is represented with a fixed-width header. Each field is represented as int32</code>, so the total size of the header is 4*5 = 20 bytes. Here’s the code which encodes and decodes this record: type Record struct { Header Header Key string Value []byte } // Header represents the fixed width fields present at the start of every record. type Header struct { Checksum uint32 Timestamp uint32 Expiry uint32 KeySize uint32 ValSize uint32 } // Encode takes a byte buffer, encodes the value of header and writes to the buffer. func (h *Header) encode(buf *bytes.Buffer) error { return binary.Write(buf, binary.LittleEndian, h) } // Decode takes a record object decodes the binary value the buffer. func (h *Header) decode(record []byte) error { return binary.Read(bytes.NewReader(record), binary.LittleEndian, h) }</code></pre> The record is encoded in the binary format before storing it on the disk. Datafile#</a></h2> A “datafile” (term used for the DB file on disk) is an append-only record of all the write operations. An instance of Bitcask can have several datafiles. However, there’s only one “active” datafile. In BarrelDB, a goroutine runs in the background at regular intervals to check if the size of the active DB file has crossed the threshold and then rotates the active file. It appends this DB file to the list of “stale” data files. All the new writes only happen to the “active” data file, and the stale files are merged as a part of the “compaction” process (described later in the post). Here’s how a datafile</code> is represented: type DataFile struct { sync.RWMutex writer *os.File reader *os.File id int offset int }</code></pre> It contains different handlers for writing and reading the file. The reason we have 2 file handlers instead of re-using the same one is that the writer</code> is only opened in an “append-only” mode. In addition, since the active file can be rotated, the writer can be set to nil</code>, ensuring no new writes ever happen on that file. writer, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) if err != nil { return nil, fmt.Errorf("error opening file for writing db: %w", err) } // Create a reader for reading the db file. reader, err := os.Open(path) if err != nil { return nil, fmt.Errorf("error opening file for reading db: %w", err) }</code></pre>KeyDir#</a></h2> In addition to storing the file on disk, Bitcask also stores additional metadata, which defines how to retrieve the record. This hashtable is a map of keys with this metadata and is referred to as KeyDir</code>. An important point to note here is that the value</code> is never stored in this map. This makes it possible for Bitcask to handle datasets more significant than what the RAM can hold. // KeyDir represents an in-memory hash for faster lookups of the key. // Once the key is found in the map, the additional metadata, like the offset record // and the file ID is used to extract the underlying record from the disk. // Advantage is that this approach only requires a single disk seek of the db file // since the position offset (in bytes) is already stored. type KeyDir map[string]Meta // Meta represents some additional properties for the given key. // The actual value of the key is not stored in the in-memory hashtable. type Meta struct { Timestamp int RecordSize int RecordPos int FileID int }</code></pre> Here, RecordPos</code> tells the record’s position offset (in bytes) in the entire file. Since the position of the record is stored in memory along with the key, the retrieval of the key doesn’t require more than a single disk seek. Bitcask achieves really low latency even with many keys in the database. A file system read-ahead cache also helps boost the performance and comes for free - no need to design a separate caching mechanism. Compaction#</a></h2> As we looked at previously, a datafile is simply an append-only sequence of writes. Any modification of the key is merely a new record appended to the datafile. KeyDir overwrites the entry of the key with the new metadata, which contains the new location of the record. Thus all reads will automatically return the updated value. Deletes are handled similarly by writing a “tombstone” record for the key. When the user requests the key after it’s been deleted, BarrelDB can check whether that value equals the tombstone value and return an appropriate error. As you would have guessed, our database will grow unbounded if we don’t perform any garbage cleanup. The datafiles need to be pruned for deleting expired/deleted records and merging all stale files in a single active file - to keep the number of opened files in check. All of these processes are together called “Compaction”. Let’s take a look at how each of these compaction routine works under the hood: Merge#</a></h3> The merge process iterates over all the keys inside KeyDir and fetches their value. The value could come from a stale file as well. Once the new keys/values are updated, it writes them to a new active file. All the old file handlers are closed, and the stale files are deleted from the disk. The KeyDir is updated similarly since the new records live in a different position/file. Hints File#</a></h3> Bitcask paper describes a way of creating a “hints” file initially loaded in the database for faster startup time. This file is essential to bootstrap KeyDir after a cold startup. This avoids iterating over all data files and reading their values sequentially. In BarrelDB, gob</code> encoding is used to dump the KeyDir</code> map as a gob</code> dump. // generateHints encodes the contents of the in-memory hashtable // as `gob` and writes the data to a hints file. func (b *Barrel) generateHints() error { path := filepath.Join(b.opts.dir, HINTS_FILE) if err := b.keydir.Encode(path); err != nil { return err } return nil }</code></pre> During the startup, BarrelDB checks the presence of a .hints</code> file, decodes this gob dump, and loads the data in KeyDir</code>. Removing expired keys#</a></h3> A goroutine runs at a configurable interval to check if the value of the key has expired. If it has, it deletes the entry from KeyDir. During the following merge process, since this entry won’t be present in KeyDir, it’ll automatically be removed when the new datafile is created. To check if the key has expired, a simple check, like comparing their timestamps in UNIX epoch format, is enough: time.Now().Unix() > int64(r.Header.Expiry)</code>. Redis Server#</a></h2> In addition to using BarrelDB as a Go library, I also implemented a redis-compatible server. I found tidwall/redcon</a> as an easy-to-use library to create a Redis-compatible server for Go applications. All I’d do was wrap BarrelDB API methods and define handlers for SET</code> / GET</code>. I was able to use redis-cli</code> and connect to the BarrelDB server: 127.0.0.1:6379> set hello world OK 127.0.0.1:6379> get hello "world"</code></pre>Benchmarks#</a></h2> You can check the repo</a> for the actual benchmarks. However, I’d like to point out some inferences of the results from redis-benchmark</code>. First, let’s send 100000 requests to the server using 50 parallel clients. This command creates a unique key for each SET</code> operation. redis-benchmark -p 6379 -c 50 -t set -n 100000 -r 100000000 Summary: throughput summary: 145985.41 requests per second latency summary (msec): avg min p50 p95 p99 max 0.179 0.016 0.183 0.207 0.399 1.727</code></pre> So, 140k requests per second is not bad at all for a disk-based KV. But the exciting thing to note here is that the performance is predictable even if you increase the load by increasing clients: redis-benchmark -p 6379 -c 200 -t set -n 100000 -r 100000000 Summary: throughput summary: 140845.08 requests per second latency summary (msec): avg min p50 p95 p99 max 0.718 0.224 0.711 0.927 1.183 5.775</code></pre> If we increase the number of requests (by 5x) as well, the throughput looks almost the same: redis-benchmark -p 6379 -c 200 -t set -n 500000 -r 100000000 Summary: throughput summary: 138350.86 requests per second latency summary (msec): avg min p50 p95 p99 max 0.748 0.056 0.711 0.879 1.135 63.135</code></pre> This magic is all because of the way Bitcask uses Log structured hash table (just append-only records for writing data). Even with a lot of records, all it has to do is to write to the end of the file, which avoids any expensive I/O operations. Summary#</a></h2> Overall, I am happy with BarrelDB</code> implementation as I covered everything described in the paper. This project had excellent learning outcomes for me. I spent a lot of time coming up with a design for structuring different components and their API methods and handling all the edge scenarios during the compaction process. Although, full credit to Bitcask as it kept its design so elegant and minimal yet achieved some significant numbers in the benchmarks. This is also a reminder that simple need not necessarily mean less powerful. I look forward to implementing a distributed KV store by adding support for multiple BarrelDB nodes connected via Raft. For now, gonna enjoy some chai and release this project to the WWW :) Fin! Logging on Nomad with Vector 2022-09-04T00:10:55+05:30 Application orchestrators like Nomad</a>, Kubernetes etc., allow you to deploy multiple applications on the same host. Logs are stored on the underlying node wherever the applications are run. However, it’s pretty common to treat these instances as ephemeral if they’re a part of an autoscaling group. Therefore depending on the node’s availability to search these logs is not a practical idea as the node can go down anytime. Moreover, in most cases, access to these nodes is limited to cluster administrators, not the maintainers (developers/application owners). In such cases, a log shipping agent must ship and aggregate logs from all cluster nodes and store them centrally (like Elasticsearch, Clickhouse, Loki). I’d recommend reading this excellent post</a> by Adrian, who has explained how to set up a Vector</a> logging pipeline for applications running with docker</code> task driver and ship the logs to Loki. For the applications running using docker</code> task driver, Nomad piggybacks</a> to docker daemon for configuring logging options. Docker daemon supports many logging options; in my experience, the journald</code> log driver works reliably well. However, this post is about tasks not using docker</code> but any other driver (e.g. raw_exec</code> and exec</code>). Nomad doesn’t provide many configuration options for logging for these drivers. The biggest issue is that Nomad logs the application’s stdio/stderr</code> stream to the log directory as-is without annotating any metadata about the task. This means that if you’ve multiple applications running on one host, the log shipping agent will not be able to identify which application’s logs are being ingested. Consider this as an example. We’re running a simple web server using exec</code> driver: job "http" { datacenters = ["dc1"] type = "service" group "app" { count = 1 network { mode = "bridge" port "python-http" { to = "8888" } } task "server" { driver = "exec" config { command = "/usr/bin/python3" args = ["-m", "http.server", "8888"] } } } }</code></pre> Once the alloc is running, we can find its IP address using: nomad alloc status 1d05d64b | grep -A 3 'Allocation Addresses' Allocation Addresses (mode = "bridge") Label Dynamic Address *python-http yes 192.168.29.76:31775 -> 8888</code></pre> On sending an HTTP request using cURL</code> we can see the logs that this webserver generated: curl -i 192.168.29.76:31775 HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.10.4 Date: Sun, 04 Sep 2022 06:18:45 GMT Content-type: text/html; charset=utf-8 Content-Length: 869 ...</code></pre> Nomad stores the logs inside the applications’ allocation directory, inside /opt/nomad/data/alloc</code>. To see the logs for the above allocation ID, we can use: tail -f /opt/nomad/data/alloc/1d05d64b-4b59-3c65-8261-760499d9e4f6/alloc/logs/server.stderr.0tail -f server.stderr.0 192.168.29.76 - - [04/Sep/2022 11:48:26] "GET / HTTP/1.1" 200 - 192.168.29.76 - - [04/Sep/2022 11:48:45] "GET / HTTP/1.1" 200 -</code></pre>Enriching logs with metadata#</a></h2> As you can see, these logs are precisely what the python3 -m http.server</code> command generates. Ideally, Nomad should have enriched these logs with metadata about the allocation ID, job name, namespace, the node it’s running on, etc., as noted in this GitHub issue</a>. However, since that’s not yet available, I brainstormed a few different options: Approach 1: Sidecar#</a></h3> The first approach was to run vector</code> as a sidecar next to the main task. This is the simplest option, to begin with. Vector can be independently configured to add metadata for the logs collected from the allocation directory of the group. However, as with every sidecar deployment, there’s a lot of extra resource usage. For every 10 different groups, reserving resources for 10 vector agents quickly eats up your available CPU/Memory of the underlying node. A more critical downside, though, was asking every developer to also configure a Vector sidecar job. And to keep all these configs in sync to ensure they’re unified across namespaces is also another headache. Due to these reasons, I discarded this option early on. However, suppose your deployment scale (managing applications) is relatively smaller. In that case, this is actually not a bad idea. Approach 2: Events Stream#</a></h3> My next option was to listen to the Nomad events stream</a> and generate a “vector” configuration template to collect logs and enrich them with metadata from the Events Stream. I developed v0.1 of nomad-vector-logger</a> based on this concept. Since I’ve written a wrapper to collect events from Nomad using nomad-events-sink</a> it was relatively straightforward to extend it to generate a vector.toml</code> config. However, after testing in prod for a few days, I noticed that relying on the events</code> stream is unreliable. Nomad events are not WebSocket based (as of yet). It’s a simple long polling mechanism which sends events to a Go channel as and when they happen. What happens when you miss an event? What happens when you run nomad system gc</code>, which clears the events index? These were some of the challenges I faced with this v0.1 approach. There needs to be some sort of “reconcile” mechanism that periodically runs. A reconciliation loop that lists all allocations using the HTTP API can help whenever there are missing events. Approach 3: Enrichment Tables#</a></h3> I also posted about the above program in Vector’s discord group (they’re super active+helpful folks) and discussed this daemon with them. They also suggested a simpler alternative: generating a CSV of running allocations instead of a .toml</code> config. Vector has support for Enrichment Tables</a> which means that it can “lookup” a CSV file to find a particular row and enrich the log event with the information found from the CSV. This seemed a super cool idea, and I developed v0.2 using this. Super thankful to Vector maintainers for giving me this idea! However, this approach had a few “subtle” drawbacks that I found: vector</code> doesn’t support live-reloading if the CSV file changes. vector</code> has support for watching a config file for changes or sending a SIGHUP</code> to reload. However, that only works for vector’s own config files. Since the CSV file is an external file, vector cannot watch it for changes. I came up with an ugly bash script hack and compared the md5</code> hash of the file in a while</code> loop and if it changed, then send a SIGHUP</code> to vector. All I can say is it’s ugly, but it works. If you wish to see it, it’s available here</a> in all it’s glory.</li> The most significant issue was the chance of losing logs for the initial 10-20s of a new allocation. The above shell script had a sleep(10)</code> because md5sum</code> can be a bit CPU intensive to keep frequently doing. Vector sees a new allocation and starts ingesting events. It tries to look up the CSV row by the allocation ID, but it doesn’t find it yet in the CSV file, complains about it, and drops the log event. Thus, I had to drop the CSV idea in search to find another more reliable approach to this. For people interested in this approach, you can checkout the csv</code> branch here</a>.</li> </ul> Approach 4: Periodic Reconciliation Loop#</a></h3> The final v0.3.0 solution, which IMHO fixed all the above issues, was: Skip Nomad events stream. Since I have to build a reconciliation loop anyway, listening to events is just extra work without tangible benefits.</li> I used a background Goroutine channel to periodically refresh the list of allocations running on that node. Even if I fetched this list once every 30s or so, it’s OK because Vector will start ingesting logs once the config gets generated. It will start reading the file from the beginning. So logs aren’t lost even if I templated the config much later after the alloc began running.</li> I added the support to delay the removal of allocation from the file. If an allocation is stopped (e.g., a new version is deployed or the job restarted), the program doesn’t immediately removes the allocation from the config file. The user can set a delay period which works like a cooling down period. In this period, one can assume that Vector would have finished sending all logs to the upstream. In case the application generates too many logs faster than what the upstream sink can accept (e.g. if the upstream Elasticsearch gets slower). Suppose we remove the allocation _immediately whenever it stops. In that case, there’s a probability that Vector wouldn’t have read the file to the end. This cooling period helps to ensure that doesn’t happen. This is not a fool-proof situation but should cover most cases unless the upstream sink is dead for many hours.</li> </ul> How it works#</a></h3> Now that we’ve covered a few different approaches and the pros/cons of each let’s see how nomad-vector-logger</code> works. Essentially nomad-vector-logger</code> is meant to be deployed inside a Nomad cluster as a system</code></a> job. A system job in Nomad runs on each node. Whenever a new node gets added to the cluster, Nomad’s scheduler schedules a copy of this program on that new node automatically. This is the equivalent of a “Daemonset” in K8s. nomad-vector-logger</code> uses Nomad’s HTTP API</a> to query all nodes’ running allocations. Once it gets the list, it adds it to an internal map and signals to generate a config. The final config that is templated out looks like this: [sources.source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy] type = "file" include = [ "/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy*" ] line_delimiter = "\n" read_from = "beginning" [transforms.transform_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy] type = "remap" inputs = ["source_nomad_alloc_64a2f9fd-e003-0bb3-b5cd-838125283a06_proxy"] source = ''' # Store Nomad metadata. .nomad.namespace = "default" .nomad.node_name = "pop-os" .nomad.job_name = "nginx" .nomad.group_name = "nginx" .nomad.task_name = "proxy" .nomad.alloc_id = "64a2f9fd-e003-0bb3-b5cd-838125283a06" '''</code></pre> For people unfamiliar with vector, it’s essentially doing 2 things: Get logs from a “file” source. The file path comes from nomad-vector-logger</code> (where all the logs for proxy</code> task are located)</li> It adds a JSON object nomad</code> with relevant keys.</li> </ul> Vector pipeline will send this event to another “transformer” which can further process the log event (for eg parsing it as logfmt</code> or JSON etc) and then finally send it to an upstream sink like Loki/Elasticsearch etc. Here’s an example of the before/after of a log line shown above in this post: Before#</a></h4> After#</a></h4> Perfect! We’ve annotated the same log event with Nomad metadata, and Vector will be able to identify these logs. If you’re interested in a complete setup on deploying this to Nomad, take a look at dev setup</a> which contains a Nomad jobspec to deploy nomad-vector-logger</code> as a sidecar with vector</code> as the main task. Hope this post helped you start configuring a logging pipeline for applications running with non-docker task drivers. Fin! Don't use which 2022-08-08T00:00:00+05:30 which</code> is a non-standard/non-POSIX compliant program. I faced many issues getting which</code> to work in a chroot environment (Nomad). So basically, which</code> is a simple shell script program to find out the dependency by searching the $PATH</code> (which is what makes it less deterministic). It’s also somehow symlinked 3 levels deep in Debian: $ ls -laht /usr/bin/which lrwxrwxrwx 1 root root 23 Apr 26 22:25 /usr/bin/which -> /etc/alternatives/which $ ls -laht /etc/alternatives/which lrwxrwxrwx 1 root root 26 Apr 26 22:25 /etc/alternatives/which -> /usr/bin/which.debianutils</code></pre> Now, notice this: $ which ls ls: aliased to ls --color=tty $ /usr/bin/which.debianutils ls /usr/bin/ls</code></pre> Both are the same programs. However, why is the output different? This is because which</code> is apparently a shell built-in zsh</code>, that is why: # zsh which which which: shell built-in command # bash which which /usr/bin/which</code></pre> The inconsistency happens because zsh</code> treats which</code> as a shell built-in when it’s apparently not one. This</a> article has some more details on why which</code> is bad and how the Debian team is slowly deprecating it from being a part of debianutils anymore. When I invoked which</code> from inside a Nomad chroot, it complained that I didn’t have /bin/sh</code> (because I was using a custom chroot mount). I started looking hard for alternatives because this silly utility already wasted too much of my time. What to use#</a></h2> Use command -v</code>. It’s a shell built-in, so it avoids a dependency on an external binary (unlike which</code>). Usage example:#</a></h3> if ! command -v aws > /dev/null; then echo "Can't find 'aws' executable. Aborted." exit 1 fi</code></pre>References#</a></h2> I found the following posts while digging which</code> and its alternatives. which-not-posix</a></li> why-not-use-which-what-to-use-then</a></li> </ul> Fin Attending IndiaFOSS 2022 2022-08-07T00:00:00+05:30 On June 28, I received a text from Anand</a> nudging me to submit a CFP for IndiaFOSS</a>. Until then, I’d not planned to attend IndiaFOSS because travelling to participate in a conference didn’t seem enticing. However, the thought of visiting Bangalore (a city where I spent five years before Covid took the joy from our lives) stayed in my mind for the rest of the day. Call it serendipity or just that little external motivation by Anand; I decided to submit a proposal</a> on Self Hosting with Nomad. IndiaFOSS would be the first physical conference I would attend after the pandemic. I love attending conferences because you get to meet and interact with many people (I somehow dislike the term “networking with people” because it’s heavily connoted with marketing/sales pitches). IndiaFOSS is a v2 of a small pilot conference IndiaOS</a> that happened in 2020. Around 650 people attended IndiaFOSS, and the execution scale was much bigger this time. IndiaFOSS is a 100% volunteer-driven conference, yet everything was quite professionally managed. My friend Dhiraj</a> travelled from Chennai to attend the conference, and I hung out with him for most of the next two days. Filling our tummies with a scrumptious breakfast of Thatte Idlis, we headed to attend the keynote by Rushabh</a>, who gave an excellent overview of FOSS United and its journey so far. There were a series of exciting talks right after - especially by Kovid</a>, a prolific FOSS contributor from India. Kovid’s software is widely used across the globe. He shared the story of how calibre</a> evolved from a bunch of scripts written to get around Sony’s DRM to become the most sought-after ebook management tool that it is now. I’ve myself used calibre in one of my initial projects</a>(back when I was still learning Python, please don’t judge that codebase). Next was the much-awaited interview of Rudra Saraswat</a> - the 12-year-old whiz kid who’s a primary contributor to Ubuntu Unity Remix and many other projects. The entire auditorium was stunned to witness how this bright young kid manages his studies, extra-curricular and FOSS projects with a laser-sharp focus. I spent some time reviewing my slides which were scheduled right after an excellent talk on hpw the Kubernetes release team works - by Nabarun</a>. I was pleasantly surprised to see many people interested in Self Hosting/Nomad. You can check out the slides here</a>. It was time for lunch, and I finally got to meet Vinayak</a>, Bibhas</a>, Gaurav</a> and Raghav</a> with whom I’ve been interacting in a small Discord group since so many months! This is what pandemic years took away from us, sigh! After lunch, Kailash</a> presented his fantastic talk on languages, dictionaries, and his FOSS project - dictpress</a>. Kailash was anticipating that people would be drowsy after lunch, but I doubt anyone would want to miss this talk. He gave a glimpse of the monumental work done by Datuk KJ Joseph and V. Krishna in the field of dictionaries and how dictpress makes it easier for anyone to host a dictionary website for any language. The rest of the day was spent hanging out in the lobby, meeting fellow hackers, and chit-chatting. I consider myself an introvert, but I guess with developers, I tend to speak up openly because we tend to ignore any small-talks :P. Rainy evenings and a weekend meant all the Peak Bengaluru memes about not finding cabs were experienced first-hand. Day 2 had quite a good lineup of talks as well. Krutika</a> shared the designer’s dilemma of contributing to FOSS Projects. Nemo</a> talked about how we could get a FOSS UPI app and touched upon the openness of UPI. We even did an open space around Self Hosting where many of us self-hosted enthusiasts came together and discussed different topics/problems we encountered while self-hosting. It was my first time interacting with Abhas</a>, who runs Deeproot Linux</a>. He shared his wisdom and experiences from self-hosting email servers, media servers and just about anything. I was pretty motivated by his dedication. The best part about IndiaFOSS was that this wasn’t a conference meant just for techies. The talks highlighted every domain of FOSS- policy, design, languages, and hardware rather than just keeping it limited to software. I was happy to see a conference not peddled with any “corporate” product talks. This is what all FOSS conferences should look like, I guess? I look forward to attending (and hopefully volunteering) next year’s conference! Fin Debugging DNS Issues in Nomad 2022-07-14T00:00:00+05:30 At work, my colleagues and I stumbled upon a hair-pulling networking issue involving a specific problem when connecting to a Kafka cluster. We use the franz-go</a> library in our Golang applications to interact with an external Kafka cluster. These Go apps are hosted inside a Nomad</a> cluster and running with the exec</a> driver. The Issue#</a></h2> Solving issues where a specific condition happens only sometimes is terribly difficult to debug because you need to reproduce the bug in a controlled environment. Our issue was that writing the first message to a Kafka topic took unusually high time (>5 seconds) while the writes of subsequent messages were instant. The following messages were instant for the next 30s, and then the write again took >5s. The request flow looks something like this: Go app in Nomad cluster (bridge mode) -> Kafka node in a Kafka cluster Both apps are inside the same VPC running on AWS EC2 instances. The Nomad task is running with network.mode=bridge</code>, which means that there are some iptables</code> rules configured to do SNAT/DNAT translation to forward packets from the default bridge network (nomad</code>) to the default ethernet interface (ens5</code>). We have a couple of other Nomad clusters in our environment and regularly use external EC2 instances to communicate, and we’ve never observed any slowdowns in our existing applications. This behaviour seemed something specific to Kafka. However, we discovered that we could not reproduce the high wait time issue in Kafka when the message was sent from a task running as host mode. So now, we had two conflicting things which made this issue strange: Issue happens in bridge</code> mode, not in host</code> mode.</li> Issue happens only with Kafka nodes, not with any external services - even in bridge mode.</li> </ul> We spent a lot of time dissecting our apps, turning on TRACE</code> level logging for Kafka clusters and enabling debug mode in franz-go</code>. One of the significant challenges for us was that even an idle Kafka cluster could be super chatty, producing ~1Mn records for a few minutes when run with TRACE</code>. Cutting through the noise and finding the exact point where the slowdown happened turned out to be more difficult than expected. However, with the debug logs in franz-go</code>, we’d arrived at a breakthrough which helped us narrow down the issue. We saw the write_wait</code> time to be ~5s in the logs emitted</a> by franz-go</code>. The subsequent messages had write_wait</code> as low as a few microseconds. What was puzzling was why Kafka took so much time to wait before writing the bytes to the underlying socket. We forked the franz-go library and added a bunch of our custom logs to figure out where and why exactly the slowdown happens. One issue in the logs emitted by franz-go was that no timestamps were attached to the logger. We added that and deployed the binaries with the patched version. This time we immediately found the logs, which pointed that it took ~5s from the time it initiated the connection</a> to the Kafka broker node and the time it was able to connect to the node</a>. The Fix#</a></h2> The node’s address was an internal hostname kafka-abc.private-zone.internal</code>. We postulated that it could be a DNS resolver issue. We did a dig kafka-abc.private-zone.internal</code> and instantly got the record. Maybe it’s cached? We decided to verify the /etc/resolv.conf</code> till we waited for the TTL of the record to expire. Opening this seemingly innocent /etc/resolv.conf</code> revealed what the issue was DNS indeed. nameserver bad nameserver good</code></pre> We had an unreachable nameserver address in the nameserver</code> list. The first message has ~5s write timeout because /etc/resolv.conf</code> has a 5s default timeout in case the nameserver is unreachable. Go’s DNS resolver picked up the second resolver, which cached the DNS records until the record’s TTL (30 seconds). Subsequent Kafka write messages on the topic worked without any issues in the TTL window. When the DNS record expires, rinse, and repeat. We later found the source of the bad nameserver came from our Nomad client initialising script. Takeaways#</a></h2> We use a custom Nomad client initialising script in our Nomad clusters to populate the chroot_env</code>. Since, by default, the chroot</a> provided by Nomad copies the/usr</code>/ directory, we found that it increased the initial startup time of the alloc. It made sense to customise the chroot_env</code> with the list of binaries and config files we would need. One of the configs happens to be /etc/resolv.conf</code>, which DNS resolvers use to resolve queries. On the host, we have systemd-resolve</code> running and /etc/resolv.conf</code> is configured with the stub resolver address. However, since that address (127.0.0.53</code>) is unreachable by the bridge network in Nomad, we mount our custom config, which looks like: nameserver 10.100.0.2 options edns0 trust-ad search ap-south-1.compute.internal</code></pre> The nameserver</code> represents the AWS R53 resolver running in all VPCs if configured with enableDnsSupport</a> in VPC settings. The above config ensures that all DNS queries for the tasks inside the bridge network go directly to the AWS R53 resolver. The resolver can do DNS Lookups for the private zones associated with the R53 in that VPC and forward all other hostnames to their own upstream DNS resolver. Fun Fact#</a></h3> If you don’t mount a custom /etc/resolv.conf</code>, then DNS resolution is broken by default in any Nomad exec task. You can quickly reproduce it with this task definition: job "sleep" { datacenters = ["dc1"] type = "service" group "sleep" { count = 1 task "sleep" { driver = "exec" config { command = "bash" args = ["-c", "sleep infinity"] } } } }</code></pre> Run a Nomad agent in -dev</code> mode: nomad agent -dev nomad run sleep.nomad</code></pre> When you exec inside the alloc: $ nomad alloc exec -i -t -task sleep 8b4a0a82 /bin/sh $ cat /etc/resolv.conf nameserver 127.0.0.1 $ dig mrkaran.dev ;; communications error to 127.0.0.1#53: connection refused</code></pre> I believe Nomad should bootstrap some DNS resolver or a relevant iptables</code> rule for the exec tasks so that DNS can be resolved by default without the need to mount a custom config. For comparison, a docker</code> also bootstraps the container with a customisable /etc/resolv.conf</code>, and the settings can be specified either at runtime and fallback to the global settings in /etc/docker/daemon.json</code>. I hope this post helps someone solve these weird DNS issues when running tasks with the bridge network and exec task driver in the Nomad cluster. Like they say: Fin! Understanding Networking in Nomad 2022-05-14T00:10:55+05:30 Nomad</a> is a general-purpose cluster orchestrator and scheduler. Up until Nomad 1.3</a> was released, it had no native support for discovering other applications running in the cluster. This is sort of a very elementary requirement when scheduling tasks in a cluster. Nomad relies on Consul to discover other “services” and has first class support for registering and fetching service records which makes things easier. Consul provides the records via various mechanisms such as a REST API, DNS and Consul Templates which render the exact IP/Port of the service in a Go template that can be injected into your application. I’ve been using Nomad since quite some time (both at work and for my self hosted instance) however I’ve often tripped when it comes to Networking. Nomad has a lot of simple concepts and it all “clicks” once you understand and recognise various patterns that can be used to connect the applications. A major learning curve for someone new to Nomad and trying to integrate Consul is that the person now has to first understand how Consul works, deploy a Consul cluster and this creates a lot of friction amongst newcomers to Nomad. Nomad 1.3 solves a part of this issue (i.e. no need to run Consul for basic service discovery) and is a great fit for just getting started with Nomad based networking. However, in this post I’d like to go through all the different networking patterns I’ve known or used in production and make an attempt at simplifying these concepts for Nomad beginners. I'll be running a single node Nomad on my dev machine. The instructions to do that can be seen here</a>. Scenario 1: Expose an application on the host#</a></h2> We’ll start off with the simplest usecase: You have a redis</code> container and you want to expose that to the host. The docker run</code> equivalent to what we wanna do is: docker run --rm -p=6379 redis:7</code></pre> This command exposes a dynamic port on your host. To see what exactly is the port number, you can do docker ps</code> and find out an output similar to 0.0.0.0:49153->6379/tcp</code> under PORTS</code>. To verify, I can use redis-cli</code> to connect to this host: $ redis-cli -p 49153 127.0.0.1:49153> ping PONG</code></pre> Fantastic! Now, let’s stop this container and see how we can do the same in Nomad. job "redis" { datacenters = ["dc1"] type = "service" group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 } } task "redis" { driver = "docker" config { image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } } }</code></pre> This is a barebones definition of how to run Redis on Nomad. We’re not dealing with any volume mounts, custom config etc here - the idea is to only learn networking concepts in Nomad! Save the above file as job.nomad</code> and deploy to the cluster with: nomad run job.nomad</code></pre> Within a few lines of config, we have a Docker container running, which exposes a dynamic port 23522</code>. We can connect to it via redis-cli</code> on our host: $ redis-cli -p 23522 127.0.0.1:23522> ping PONG</code></pre> NOTE: It’s important to have ports</code> in your task.config</code> section. Nomad passes this information to the docker</code> daemon running on the host. So unless you specify which ports to advertise in the container, it won’t know whether to expose 6379 or not. This can be easily verified with docker inspect</code>: # without `ports` $ docker inspect fc32a4ffd148 -f "{{json .NetworkSettings.Ports }}" | jq '.' { "6379/tcp": null } # with `ports` $ docker inspect 0421101d0228 -f "{{json .NetworkSettings.Ports }}" | jq '.' { "6379/tcp": [ { "HostIp": "127.0.0.1", "HostPort": "31930" } ], "6379/udp": [ { "HostIp": "127.0.0.1", "HostPort": "31930" } ] }</code></pre>Exposing Static ports#</a></h3> A less common scenario is to bind an application to a static port on the host. We’ll cover an example of when you want to do that a bit later, but this is generally not widely used because in a cluster your application can “float” around and the idea of sticking to a port is not useful. However, there’s a way for us to do that by simply adding a static</code> line in our port</code> block: network { port "redis" { to = 6379 static = 6379 } }</code></pre> When we deploy the same file again, we can see the port allocation has changed from dynamic to the static port we assigned. It’s your job to ensure no other applications are listening on that same interface and port because that’s bound to cause conflicts. Scenario 2: Communicate to Redis within the same group#</a></h2> For this scenario, we are going to assume there’s a Go application that needs to talk to Redis. However, in this scenario, the Redis is sort of like an ephemeral cache, so it’s okay to deploy both of them in the same Task Group. If you don't know the difference between a Task and Group, here's a very primitive explanation but please read the docs for more clarity. A Group can contain multiple tasks. What's important to know here is that a group will always have its own shared network namespace. This means, that if you have 2 tasks in the group, they both will have access to the same network namespace. This allows both tasks to talk to each other on the same network interface. job "hello" { datacenters = ["dc1"] type = "service" group "app" { count = 1 network { mode = "host" port "app" { to = 8080 static = 8080 } port "redis" { to = 6379 static = 6379 } } task "redis" { driver = "docker" config { network_mode = "host" image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } task "app" { driver = "docker" env { DEMO_REDIS_ADDR = "${NOMAD_ADDR_redis}" } config { network_mode = "host" image = "mrkaran/hello-app:1.0.0" ports = ["app"] } resources { cores = 1 memory = 512 } } } }</code></pre> Key Points: You can see we have defined task</code> app and task redis</code> under the same group, app</code>. This means that Nomad will co-locate both of these tasks on the same client (because they tend to share not just the same network namespace but a common allocation directory as well - which makes it super easy to share files across tasks).</li> We are using NOMAD_ADDR_redis</code> to get the IP:Port combination for the redis</code> task. This gets injected at runtime by Nomad. You can find a list of runtime variables here</a>.</li> This is ideal for quick tests/dev setup where you don’t want the overhang of Service Discovery etc and want to connect to your applications in the least friction possible.</li> </ul> The above config is suitable if you’re migrating from docker-compose</code> based environments, you can use this template for your services. The biggest limitation of this approach is that it’s using a host network so it’s not possible to set up any kind of Access Controls on it. This effectively means that nothing prevents any other application on the cluster to talk to these ports. Scenario 3: Communicate across different groups#</a></h2> Task Groups are useful if you have related tasks (like the init task where you wanna fetch the files before the task starts). But a drawback of using group</code> is that you can’t scale the tasks independently. In the above example, we placed Redis and App in the same group, but that means if you increase count</code> of same group to scale the app, you end up scaling Redis containers too. This is undesirable as Redis may not to scale proportionally to app. The way to do create multiple groups is to split the tasks into their own individual groups: job "hello" { datacenters = ["dc1"] type = "service" group "app" { count = 1 network { mode = "host" port "app" { to = 8080 static = 8080 } } task "app" { driver = "docker" env { DEMO_REDIS_ADDR = "localhost:6379" } config { image = "mrkaran/hello-app:1.0.0" ports = ["app"] } resources { cores = 1 memory = 512 } } } group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 static = 6379 } } task "redis" { driver = "docker" config { image = "redis:7" ports = ["redis"] } resources { cores = 1 memory = 256 } } } }</code></pre> When you submit this job, you get 2 allocation IDs (each group creates one alloc</code>). The key point here is that both of these groups have their own network namespace. So, we don’t really have any way to reach the other application (we can’t really rely on the host network, because there’s no guarantee that both of these groups will be deployed on the same node). In the previous example, we saw how Nomad exposes runtime variables that contained information about all tasks in other groups. But now since the groups are separate, the app</code> container has no idea about redis</code> (or vice-versa): env | grep NOMAD NOMAD_REGION=global NOMAD_CPU_LIMIT=4700 NOMAD_IP_app=127.0.0.1 NOMAD_JOB_ID=hello NOMAD_TASK_NAME=app NOMAD_SECRETS_DIR=/secrets NOMAD_CPU_CORES=1 NOMAD_NAMESPACE=default NOMAD_ALLOC_INDEX=0 NOMAD_ALLOC_DIR=/alloc NOMAD_JOB_NAME=hello NOMAD_HOST_IP_app=127.0.0.1 NOMAD_SHORT_ALLOC_ID=a9da72dc NOMAD_DC=dc1 NOMAD_ALLOC_NAME=hello.app[0] NOMAD_PORT_app=8080 NOMAD_GROUP_NAME=app NOMAD_PARENT_CGROUP=nomad.slice NOMAD_TASK_DIR=/local NOMAD_HOST_PORT_app=8080 NOMAD_MEMORY_LIMIT=512 NOMAD_ADDR_app=127.0.0.1:8080 NOMAD_ALLOC_PORT_app=8080 NOMAD_ALLOC_ID=a9da72dc-94fc-6315-bb37-63cbeef153b9 NOMAD_HOST_ADDR_app=127.0.0.1:8080</code></pre>Service Discovery#</a></h3> This is where things get interesting. The app</code> group needs to discover redis</code> before connecting to it. There are multiple ways to do that, but we’ll cover 2 standard ways which are more common. Using Nomad native service discovery#</a></h3> This is a feature launched in Nomad 1.3. Up until this release, Nomad had to rely on Consul for this. But with native service discovery built in Nomad, things are much simpler. Let’s make the following changes to our job file. In each group, we’ll add a service</code> definition: group "app" { count = 1 network { mode = "host" port "app" { to = 8080 } } service { name = "app" provider = "nomad" port = "app" } // task is the same } group "redis" { count = 1 network { mode = "host" port "redis" { to = 6379 } } service { name = "redis" provider = "nomad" port = "redis" } // task is the same }</code></pre> So, we added a new service</code> block and got rid of static</code> ports. Well, there’s no need to bind to static ports when we’re using service discovery. After submitting the job, we can use the nomad service list</code> command to ensure the services are registered with Nomad. nomad service list Service Name Tags app [] redis []</code></pre> To find out details about a particular service, we can use nomad service info</code>: $ nomad service info app Job ID Address Tags Node ID Alloc ID hello 127.0.0.1:29948 [] d92224a5 5f2ac51f $ nomad service info redis Job ID Address Tags Node ID Alloc ID hello 127.0.0.1:22300 [] d92224a5 8078c9a6</code></pre> Perfect! We can see the dynamic port assignment in each of the services. To use this config in our app, we will template it: task "app" { driver = "docker" template { data = <<EOH {{ range nomadService "redis" }} DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }} {{ end }} EOH destination = "secrets/config.env" env = true } config { image = "mrkaran/hello-app:1.0.0" ports = ["app"] } resources { cores = 1 memory = 512 } }</code></pre> We added the template</code> stanza which will interpolate the env variables in the container. We loop over nomadService</code> and get the address and port of the redis</code> service. This makes it convenient for tasks on other nodes to discover each other. Using Consul Service Discovery#</a></h3> Just by tweaking provider</code> in our service</code> block, we can use the Consul agent for service discovery. service { name = "app" provider = "consul" port = "app" } task "app" { driver = "docker" template { data = <<EOH {{ range service "redis" }} DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }} {{ end }} EOH</code></pre> Ensure that you're running Consul and have connected Nomad to it. Please refer to docs</a> for the same. Since now we are using consul</code> for registering services, we have to loop over service</code> instead of nomadService</code>. The rest of the things remain pretty much the same. I really like how with just 2 lines of code you can switch between Nomad/Consul for discovering services. Now, of course, there are certain advantages to using Consul: You can query the address of the service with DNS:</li> </ul> doggo redis.service.consul @tcp://127.0.0.1:8600 NAME TYPE CLASS TTL ADDRESS NAMESERVER redis.service.consul. A IN 0s 172.20.10.3 127.0.0.1:8600 </code></pre> Define health checks. Since it’s a new feature, health checks on Nomad service aren’t there but there’s a GitHub issue</a> open for the same.</li> </ul> Update (2024): Native health checks are now available in Nomad since version 1.4 (released October 2022). This means you can define health checks directly in your Nomad service definitions without requiring Consul. You can read more about this in the Nomad documentation</a>. </blockquote> Accessible by applications outside Nomad. In case consul</code> is used by other applications outside of the Nomad cluster, they can still get their address (using DNS or REST APIs)</li> </ul> However, Nomad native service discovery is perfect for local setups and even smaller use-cases in production because it eliminates the need of running Consul in your stack which is a big thing! Scenario 4: Restricting access to certain namespaces#</a></h2> In all the above scenarios, we found that the service gets exposed to the local Nomad client. In case you’re running multiple namespaces on your cluster, you’d like to not expose them at all. In addition, you may want to express fine-grained controls on which application can access a particular service. All of this is possible via a Service Mesh. Nomad provides a way to have a “service mesh” via Consul Connect. Consul Connect can do mTLS and service authorization. Under the hood, it’s an Envoy proxy that runs alongside your app (sidecar is a fancy way to say that). The consul</code> agent configures an Envoy configuration for you so it’s all pretty seamless. For this to work, the first thing we need is a bridge</code> network mode. This network model is actually a CNI plugin and needs to be installed separately in /opt/cni/bin</code>. Follow the steps mentioned here</a>. network { mode = "bridge" port "redis" { to = 6379 } }</code></pre> The service in redis</code> is called as a Consul Connect Ingress: service { name = "redis" provider = "consul" port = "6379" connect { sidecar_service {} } }</code></pre> It’s an empty block because we don’t need to define any upstream here. The rest of the values will be default values. Next, we create a service for our app</code> and that is a Consul Connect Egress: service { name = "app" provider = "consul" port = "app" connect { sidecar_service { proxy { upstreams { destination_name = "redis" local_bind_port = 6379 } } } } }</code></pre> Here we define an upstream for redis</code>. If you notice closely, we are using a port number in Consul Connect Ingress. For some reason, if you use a named port instead of 6379 it doesn’t work. I am not entirely sure if it’s a bug or it’s intended to work like this. So here, when the app</code> wants to talk to redis, it talks to localhost:6379</code> which is the local port that the Envoy sidecar is listening to. We can verify that using netstat</code>: $ netstat -tulpvn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.2:19001 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:23237 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN - tcp6 0 0 :::8080 :::* LISTEN 1/./hello.bin</code></pre> The traffic is sent from this port to the other Envoy proxy on a port that it advertises (and Consul automagically configured). That Envoy proxy further sends the traffic to the redis</code> container on port 6379. The proxied traffic is securley encrypted via mTLS and authorized (via Consul Intentions - not covered in this post). Scenario 5: Exposing services to end-user#</a></h2> In the first scenario, we discussed using static ports. Well, it turns out it’s super helpful if you want to define a Traffic Ingress service. Unlike K8s, Nomad doesn’t have any Ingress Controllers, so the best way is to deploy these web proxies on each node as a system job</a> (which means it’s ensured to run on every client node) and bind them to a static port (say 443/80). Then, configure your Load Balancers and register all the Nomad nodes as target IPs and their ports would be the static port you define. These Ingress proxies (like HAProxy/Nginx) can then be used to talk to your application via any of the patterns we’ve mentioned above. Typically, you’d want to use a “Host-Based” routing pattern for your ingress proxy to make a routing decision. For eg, in case you have an a.example.org</code> DNS record pointing to an ALB. Now, when the request comes to the ALB, it forwards to any one of the NGINX/HAProxy. For HAProxy to correctly route the traffic to `a service, you can use a “Host” header. Summary#</a></h2> These were some of the common networking patterns that I’m aware of. Since some of these concepts are not really straightforward I hope the explanation helped in bringing some clarity. There’s much more to this topic like Consul Gateways and multiple kind of CNIs which tweak how networking happens in the cluster but those are some really advanced topics that are out of the scope for this post. Fin! How I expose services while self hosting 2022-02-10T00:00:00+00:00 I’ve often been asked how to expose public and private services running on DigitalOcean droplets/RPis when self hosting apps. Most people don’t have access to a static IP for their RPis. I felt I’d summarize my approach to this in this blog post and hope it’ll be useful for others trying to do the same. Tools I use#</a></h3> Tailscale</a>: I love this and I’ve written about it in the past</a>. I am one of the early adopters, using this for 2 years now and while it can be replaced with other similar Wireguard based mesh services that have popped up recently, I still like the UX of this application a lot. In simpler words, it just works!</li> Caddy</a>: I use Caddy as a webserver to host some static websites and as a reverse proxy for all the applications I’ve deployed. It makes the SSL setup a breeze, the config is nice to read as well. There’s nothing about Caddy to not like it if you’re not in the business of dealing with super high concurrent traffic on your website (in which case NGINX/HAProxy/Envoy et al might be worth looking at). I’ve never used Traefik myself, and while I know it makes the routing simple with Docker labels, I think my current approach with Caddy is also similar and easier for me to use without having to learn a new config.</li> Docker</a>: All the services are containerized and I make special use of Docker networks, which I’ll show in the post.</li> </ul> Setup Overview#</a></h2> I use 2 instances of Caddy for my setup: Public: This is to reverse proxy all the public-facing websites.</li> Private: This is to reverse proxy all the internal websites.</li> </ul> The docker-compose</code> looks like this: version: "3.7" services: caddy_public: ... ports: - "<do_floating_ip>:80:80" - "<do_floating_ip>:443:443" networks: - public ... caddy_internal: ... ports: - "100.111.91.100:80:80" - "100.111.91.100:443:443" networks: - internal ... networks: public: name: caddy_public internal: name: caddy_internal</code></pre> This is where most of the magic lies. I use 2 Docker networks caddy_public</code> and caddy_internal</code>. Both these networks are configured as Bridge Networks. The containers connected to the same bridge network can even reach other containers using internal DNS. The published port section is the one of importance here. In the internal</code> caddy instance, the TCP port 80 in the container is mapped to port 80 on the Docker host for connections to host IP 100.111.91.100</code> (this is a private IP and belongs to the CGNAT space</a>). The same is done for caddy_public</code> where instead of Tailscale IP, the Floating IP of the DigitalOcean droplet is used. Next comes the part where we’ll attach these networks to our applications. docker-compose</code> by default creates a user-defined bridge network if you leave networks</code> unspecified. However, if you want more granular control, you can specify the networks in the Compose spec itself. Here’s an example of Plausible</code> compose spec which is exposed publicly: plausible_events_db: image: yandex/clickhouse-server:21.3.2.5 networks: - plausible plausible: image: plausible/analytics:latest networks: - web - plausible networks: web: name: caddy_public external: true plausible: name: plausible</code></pre> Here you can see that the ClickHouse container is only attached to the plausible</code> network. This plausible</code> network is scoped only to these services defined in this file. We can exec</code> inside the caddy_public</code> container and find out the IP of plausible</code> and verify if the network is correctly configured and reachable: $ host plausible plausible has address 172.20.0.3 $ curl plausible:8000 <html><body>You are being <a href="/login">redirected</a>.</body></html>/srv # </code></pre> Some notes on this setup: You’ll also note that I haven’t published the port 8000 anywhere. Publishing is only required if you want to forward the traffic from your host network to the docker network (via the bridge). But here, since both are attached to the same network (caddy_public</code>), that is not required anymore.</li> This also means that the only way someone can reach port 8000 on Plausible is only via the Caddy container (which is firewall restricted to Cloudflare IPs.)</li> The DB container doesn’t need to be accessed at all from Caddy, so we’ve not attached the web</code> network there</li> </ul> Here’s another example of exposing an internal service, which works on the same principles: grafana: image: grafana/grafana:8.3.4 networks: - monitoring - internal networks: internal: name: caddy_internal external: true monitoring: name: monitoring</code></pre> Here, the Grafana container is attached to caddy_internal</code> network. Since the caddy_internal</code> container only publishes ports on the Tailscale IP, anyone who is not inside the Tailscale network will not be able to access this. Tailscale can do much more by setting up ACL rules per device for each user, but since I am the only user, I’ve not configured ACL rules on it yet. Hope this approach was simplistic enough. I follow this pattern across all the applications I self-host and honestly pretty happy with it Fin! Using ClickHouse Keeper for Replication 2021-12-17T08:10:55+05:30 ClickHouse is an extremely performant columnar DB used for fast analytical processing. ClickHouse supports data replication</a> using Apache Zookeeper which needs to be deployed externally. While Zookeeper works well after you’ve tuned it properly, it was still an additional maintenance overhead. The good news is that you don’t have to worry about this anymore. ClickHouse 21.12 release annoucement</a> mentions ClickHouse Keeper</code> as feature complete. Keeper</a> is a replacement for Zookeeper, written in C++ and uses the RAFT algorithm for consensus across different nodes in the cluster. It also has some nice improvements over Zookeeper, such as compressed snapshots/logs of the state changes in the cluster and the ability to run it inside the server binary itself. I decided to spin up a local Clickhouse cluster and test out the new clickhouse-keeper</code> feature. For our local setup, we’ll set up 4 nodes. We’ll create 2 shards to distribute our data and each shard will have 2 replicas each. The setup looks something like this: We need to run clickhouse-keeper</code> only on 3 of these nodes to ensure a quorum. Here’s a sample docker-compose.yml</code> to spin up these nodes as containers: version: "3.7" x-clickhouse-defaults: &clickhouse-defaults restart: unless-stopped image: yandex/clickhouse-server:21.12.2.17 ulimits: nproc: 65535 nofile: soft: 262144 hard: 262144 services: clickhouse-blue-1: <<: *clickhouse-defaults container_name: clickhouse-blue-1 hostname: clickhouse-blue-1 ports: - 9000:9000 - 8123:8123 - 9181:9181 volumes: - type: volume source: ch-blue-1-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-blue-1:/etc/clickhouse-server/config.d/" clickhouse-blue-2: <<: *clickhouse-defaults container_name: clickhouse-blue-2 hostname: clickhouse-blue-2 ports: - 9001:9000 - 8124:8123 - 9182:9181 volumes: - type: volume source: ch-blue-2-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-blue-2:/etc/clickhouse-server/config.d/" clickhouse-green-1: <<: *clickhouse-defaults container_name: clickhouse-green-1 hostname: clickhouse-green-1 ports: - 9002:9000 - 8125:8123 - 9183:9181 volumes: - type: volume source: ch-green-1-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-green-1:/etc/clickhouse-server/config.d/" clickhouse-green-2: <<: *clickhouse-defaults container_name: clickhouse-green-2 hostname: clickhouse-green-2 ports: - 9003:9000 - 8126:8123 - 9184:9181 volumes: - type: volume source: ch-green-2-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-green-2:/etc/clickhouse-server/config.d/" volumes: ch-blue-1-data: ch-blue-2-data: ch-green-1-data: ch-green-2-data:</code></pre> clickhouse-keeper</code> runs only if the <keeper_config></code> section is present inside the config. Here’s a sample config: <clickhouse> <keeper_server> <tcp_port>9181</tcp_port> <server_id>${SERVER_ID}</server_id> <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path> <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path> <coordination_settings> <operation_timeout_ms>10000</operation_timeout_ms> <session_timeout_ms>30000</session_timeout_ms> <raft_logs_level>trace</raft_logs_level> </coordination_settings> <raft_configuration> <server> <id>1</id> <hostname>clickhouse-blue-1</hostname> <port>9234</port> </server> <server> <id>2</id> <hostname>clickhouse-blue-2</hostname> <port>9234</port> </server> <server> <id>3</id> <hostname>clickhouse-green-1</hostname> <port>9234</port> </server> </raft_configuration> </keeper_server> </clickhouse></code></pre> There are some other configs required for Clickhouse to discover other nodes and enable replication. You can find a working example in this repo</a>. Verifying Cluster State#</a></h2> Once the containers are configured and running, we can verify if the replication is working as intended: Let’s first check if the keeper</code> daemon is running by: echo ruok | nc 127.0.0.1 9181 imok</code></pre> ruok</code> is a part of Four Letter Commands</a> that are mostly used to diagnose Keeper’s client/server. To ensure that clickhouse-server</code> is aware of the keeper</code> cluster, we can query the system.zookeeper</code> table: SELECT * FROM system.zookeeper WHERE path = '/' FORMAT Vertical Query id: 287d3c2d-b93f-4d48-b335-6df2f89a8ab3 Row 1: ────── name: clickhouse value: czxid: 3 mzxid: 3 ctime: 2021-12-17 09:11:05 mtime: 2021-12-17 09:11:05 version: 0 cversion: 1 aversion: 0 ephemeralOwner: 0 dataLength: 0 numChildren: 1 pzxid: 4 path: /</code></pre> If you don’t see any results in the system.zookeeper</code> table, then re-check if zookeeper</code> section is present inside the config. This config tells ClickHouse how to discover keeper nodes. We can also see if our cluster is configured correctly with: SELECT host_name, host_address, replica_num FROM system.clusters WHERE cluster = 'events' Query id: a4bacfa1-d3aa-482f-b8b2-30b05442a173 ┌─host_name──────────┬─host_address─┬─replica_num─┐ │ clickhouse-blue-1 │ 172.19.0.5 │ 1 │ │ clickhouse-blue-2 │ 172.19.0.3 │ 2 │ │ clickhouse-green-1 │ 172.19.0.4 │ 1 │ │ clickhouse-green-2 │ 172.19.0.2 │ 2 │ └────────────────────┴──────────────┴─────────────┘</code></pre> (Here events</code> is our cluster name specified in the remote_servers</code> section of the config.) Inserting Sample Data#</a></h2> Let’s create a DB and add some data to the DB. We need to ensure that our data is split across shards and we can query all shards using a central view. Cluster Schema#</a></h3> CREATE DATABASE app ON CLUSTER 'events'; CREATE TABLE app.events_local ON CLUSTER '{cluster}' ( time DateTime, event_id Int32, uuid UUID ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}') PARTITION BY toYYYYMM(time) ORDER BY (event_id); CREATE TABLE app.events_main ON CLUSTER '{cluster}' AS app.events_local ENGINE = Distributed('{cluster}', app, events_local, rand());</code></pre> What’s happening here: We’ll create a sample database app</code> with a single table events_local</code>.</li> We’re using ReplicatedMergeTree</code> here as that tells ClickHouse</code> to automatically replicate the data inside the table when it’s inserted. Properties like cluster</code>/shard</code>/replica</code> are automatically populated from the server’s macros. It’s a handy feature that allows you to execute this command just once on the cluster and it automatically populates the relevant config in each server.</li> </ul> </li> Finally we create a Distributed</code> table to perform our INSERT</code>/SELECT</code> operations in a central place. It’s possible to manually insert data to particular replicas, but since Clickhouse supports load balancing across shards, it’s preferred to create a table with a Distributed</code> engine and let that happen automatically. Write operations can be sharded based on a particular column name, but here we are simply using the rand()</code> function, which splits the write randomly to different shards.</li> All read operations are parallelized and Clickhouse selects one replica from each shard to query the data from.</li> </ul> </li> We use ON CLUSTER</code> keyword to indicate that this query has to be run on all servers which are part of the cluster events</code>.</li> </ol> It’s now time to insert some random data, for which you can use this command: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4());</code></pre> We can now query the table and see if our records are present: SELECT * FROM app.events_main ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └─────────────────────┴────────────┴──────────────────────────────────────┘ ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └─────────────────────┴────────────┴──────────────────────────────────────┘ ┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └─────────────────────┴───────────┴──────────────────────────────────────┘ ┌────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └─────────────────────┴────────────┴──────────────────────────────────────┘</code></pre> Now, you must be wondering how to check if the data is replicated and how the data is distributed across our shards. For that, we can use the handy remote()</code> function: SELECT * FROM ( SELECT hostName(), * FROM remote('172.20.0.2', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.3', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.4', 'app', 'events_local') UNION ALL SELECT hostName(), * FROM remote('172.20.0.5', 'app', 'events_local') ) Query id: 34a81447-a31b-4915-9bd8-7a6e17bb0860 ┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-blue-2 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-blue-1 │ 2021-12-17 09:36:17 │ 1888949839 │ 3c33305e-8ea0-4ac1-a07a-667465ec9a85 │ └───────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:16 │ 1303963476 │ 0b27d5bd-6315-4937-82a4-18199e5eebb7 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:15 │ 415899002 │ 5980299f-1594-4c17-8eb5-e512f15ecf34 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘ ┌─hostName()─────────┬────────────────time─┬───event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-1 │ 2021-12-17 09:36:16 │ -689113926 │ 2a944d71-73f3-4491-b851-4f0e6f296e44 │ └────────────────────┴─────────────────────┴────────────┴──────────────────────────────────────┘</code></pre> Perfect! We can see our data is sharded as some parts of it exists in green</code> and blue</code>. We can also see that for each record, we have 2 entries thus confirming that ReplicatedMergeTree</code> is doing its job! Additional Scenarios#</a></h2> All is well and good so far, but to ensure that our cluster setup is resilient we need to introduce our cluster to some Non-Happy scenarios. Stop a replica node#</a></h3> ❯ docker-compose stop clickhouse-green-2 Stopping clickhouse-green-2 ... done</code></pre> Now, let’s query our records: SELECT count(*) FROM app.events_main ┌─count()─┐ │ 4 │ └─────────┘</code></pre> That seems right. We’re able to access all our data with just one replica down. Let’s try inserting data and bring back the replica to see if it got automatically replicated or not: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Ok.</code></pre>SELECT * FROM app.events_main ORDER BY time DESC LIMIT 1 ┌────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │ └─────────────────────┴───────────┴──────────────────────────────────────┘</code></pre> Let’s start the replica again: ❯ docker-compose start clickhouse-green-2 Starting clickhouse-green-2 ... done</code></pre> On querying the replica to see if it has the data: SELECT hostName(), * FROM remote('172.20.0.5', 'app', 'events_local') ORDER BY time DESC LIMIT 1 Query id: 422041dc-afe8-4f28-9659-6d58726f8c90 ┌─hostName()─────────┬────────────────time─┬──event_id─┬─uuid─────────────────────────────────┐ │ clickhouse-green-2 │ 2021-12-17 10:09:10 │ 375826335 │ 8dbbc9cf-8f00-4cc3-a4fb-0f17a68340e5 │ └────────────────────┴─────────────────────┴───────────┴──────────────────────────────────────┘</code></pre> Perfect! The record 375826335</code> automatically got replicated once the replica was healthy. Stop a Keeper Node#</a></h3> We’ll stop a server instance that is running the clickhouse-keeper</code> process. By doing this, we’ll also be killing a replica, but that is okay. ❯ docker-compose stop clickhouse-blue-2 Stopping clickhouse-blue-2 ... done</code></pre> Let’s insert some data: INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Ok</code></pre> We can check the server information of the other 2 keeper</code> nodes: $ echo stat | nc 127.0.0.1 9181 | grep Mode Mode: follower $ echo stat | nc 127.0.0.1 9183 | grep Mode Mode: leader</code></pre> So, one of the keeper</code> nodes has elected itself to be the leader and we have no problems in the setup so far. However, if we stop another keeper</code> node, then there will be only keeper</code> node remaining in the setup and to avoid a Split Brain issue, it won’t be able to elect itself as the leader. What happens then? Only one way to find out: ❯ docker-compose stop clickhouse-blue-1 Stopping clickhouse-blue-1 ... done</code></pre>INSERT INTO app.events_main VALUES (now(), rand(1), generateUUIDv4()); Received exception from server (version 21.12.2): Code: 242. DB::Exception: Received from localhost:9000. DB::Exception: Table is in readonly mode (zookeeper path: /clickhouse/tables/events/green/table). (TABLE_IS_READ_ONLY)</code></pre> Ah! So, we can still query for the data (which will be incomplete since blue shard is completely down), but we cannot insert any new data at all. We can even verify this by querying for the health of the keeper</code> node: $ echo mntr | nc localhost 9183 This instance is not currently serving requests% </code></pre>Add a new shard#</a></h3> Alright, so our tests so far have been quite good and show that the cluster is resilient to failures as long as there exists a keeper</code> node running as leader</code> mode in the quorum. Now, let’s see how to add a new shard. We’ll extend our docker-compose.yml</code> to add a new orange</code> shard: clickhouse-orange-1: <<: *clickhouse-defaults container_name: clickhouse-orange-1 hostname: clickhouse-orange-1 ports: - 9004:9000 - 8127:8123 - 9185:9181 volumes: - type: volume source: ch-orange-1-data target: /var/lib/clickhouse - "./configs/gen/clickhouse-orange-1:/etc/clickhouse-server/config.d/"</code></pre> Inside our remote_servers.xml</code>, we’ll add the orange</code> shard as well: <shard> <internal_replication>true</internal_replication> <replica> <host>clickhouse-orange-1</host> <port>9000</port> </replica> </shard></code></pre> That’s pretty much it. Let’s start the new replica: docker-compose up</code></pre> Let’s insert some data and query if the shard is getting data or not: SELECT hostName(), * FROM remote('172.20.0.2', 'app', 'events_local') Received exception from server (version 21.12.2): Code: 519. DB::Exception: Received from localhost:9000. DB::Exception: All attempts to get table structure failed. Log: Code: 279. DB::NetException: All connection tries failed. Log: There is no table `app`.`events_local` on server: 172.20.0.2:9000 . (ALL_CONNECTION_TRIES_FAILED) (version 21.12.2.17 (official build)) . (NO_REMOTE_SHARD_AVAILABLE)</code></pre> Houston, we have a problem. > There is no table `app`.`events_local` on server: 172.20.0.2:9000</code></pre> This is mentioned in the ClickHouse docs on Replication</a>: CREATE, DROP, ATTACH, DETACH and RENAME queries are executed on a single server and are not replicated: </blockquote> This means that although we’d run CREATE DATABASE</code> and CREATE TABLE</code> commands using ON CLUSTER</code> which executes on all servers, but since the orange-1</code> node is introduced after we ran that command, we need to manually create the DB and Table here. We have to execute the below commands inside the orange-1</code> replica: CREATE DATABASE app; CREATE TABLE app.events_local ( time DateTime, event_id Int32, uuid UUID ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/{table}', '{replica}') PARTITION BY toYYYYMM(time) ORDER BY (event_id);</code></pre> That’s all that is required. Adding a new replica is also the same process. Summary#</a></h2> Hope this tutorial helped you figure out how to use clickhouse-keeper</code> to set up a distributed ClickHouse cluster DB. clickhouse-keeper</code> is still a relatively new feature as the docs mention already, but given that it solves operations overhead of running a Zookeeper cluster, it’s worth checking it out. References#</a></h3> https://www.youtube.com/watch?v=abhcCRW09Ac</a> Slides used in above talk: https://presentations.clickhouse.com/meetup54/keeper.pdf</a></li> </ul> </li> https://clickhouse.com/docs/en/operations/clickhouse-keeper</a></li> https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration/test_keeper_multinode_simple</a></li> https://github.com/ClickHouse/ClickHouse/issues/2161</a></li> </ul> For the full code/config samples, you can check out the repo</a>. Fin! Updates#</a></h3> If you have setup RBAC on your cluster, make sure you add <user></code> and <password></code> fields to the <remote_server></code> configuration.</li> </ul> Breaking Software and Getting Older 2021-12-09T08:10:55+05:30 Recently, I’d posted on Twitter that my feed has become messy overtime. Maybe I followed some accounts that were of no interest, maybe Twitter’s algos don’t really know what to show to me. Whatever, I wanted a fresh start. And, I’ve done this in past. I’ve removed all the accounts that I follow but they don’t follow me back. This little trick is helpful to start afresh while still not offending your friends ;) I’d written a really simple Python script to do the job! It looks like: import tweepy SCREEN_NAME = 'mrkaran_' CONSUMER_KEY = '' CONSUMER_SECRET = '' ACCESS_TOKEN = '' ACCESS_TOKEN_SECRET = '' auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET) auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET) api = tweepy.API(auth) followers = api.followers_ids(SCREEN_NAME) friends = api.friends_ids(SCREEN_NAME) for f in friends: if f not in followers: print("Unfollow {0}?".format(api.get_user(f).screen_name)) api.destroy_friendship(f)</code></pre> I’d last ran this in 2019 or so. That is dinosaur ages in the world of software. All I wanted to do was run this goddamn script again, 2 years later. Seems to be too much of an ask? Apparently tweepy</code> the library used here to interact with Twitter’s APIs had a major release</a> with lots of breaking changes. They’ve internally migrated to start using v2 Twitter API. So, when I naively ran pip install tweepy</code>, my code threw: ❯ python main.py Traceback (most recent call last): File "/home/karan/Code/Personal/twitter-unfollow/main.py", line 13, in <module> followers = api.followers_ids(SCREEN_NAME) AttributeError: 'API' object has no attribute 'followers_ids'</code></pre> So, some method name changed. But that’s not all. The whole auth process including the initialisation of the API object changed as well. I’d spent some ~15 minutes grokking the docs but frustrated because 1) I don’t give a shit about v2/v1 APIs. 2) I just want to carry on with whatever I was doing. Why is this shit taking more time than I can care to give this to? I’d have cared enough if it was a side-project I maintained or something that I used daily. A utility like this which gets used once in a couple of years, will see more such breaking changes in future. Why, then should I spend migrating to v2 APIs, when after 2 years, v3 APIs would have broken my code again? What is the damn point? Why can’t software just keep working without troubling their users? In the end, I just installed the last version that works with v1</code> APIs and ran the script. I get breaking changes, I totally do. And I’ve no qualms with Tweepy. They did what they had to, in order to be compatible with v2. I am just angry/sad at the whole ecosystem of “Move fast, break things”. Please. Slow. Down. So that the rest of us who have a life can enjoy it and not spend an entire weekend migrating across versions! Sigh Debugging issues with Packer and Ansible in Docker 2021-11-16T00:00:00+05:30 Today I faced an issue that questioned my sanity. Since I didn’t find many “related” issues on StackOverflow/Google-fu except a lone GitHub thread where a kind stranger hinted at what could be the issue, I am writing about it here in the hopes (although I don’t wish this torture on anyone) that it helps someone! To give some context, I am running a Gitlab CI</a> job that bakes an AMI</a> using Packer</a>. Packer can use different kinds of provisioners to do configure stuff on the host and then prepare the image. I am using Ansible provisioner</a> to install and configure Consul</a>. At this point you may think this post is sponsored by Hashicorp by the sheer mention of all their products, but I assure you that is not the case. Anyway, so this role works locally but it fails on the damn Gitlab CI. Classic case of Works on my machine, Ops problem now</a>. These kinds of issues although are particularly exciting for me because they give me a chance to dig down deeper in the internals and slowly peel apart layers to figure out where the “drift” between local and CI is happening. Here’s the relevant Packer snippet (Oh and this week as I updated myself with new Packer releases, it’s now possible to write Packer config with HCL and not just JSON anymore! Yayie. Again a reminder: not a sponsored post). build { sources = ["source.amazon-ebs.golden-ami"] provisioner "ansible" { playbook_file = "${var.playbook_file}" extra_arguments = ["--tags", "install", "-e", "ansible_python_interpreter=/usr/bin/python3"] ansible_env_vars = ["ANSIBLE_LOCAL_TEMP=$HOME/.ansible/tmp", "ANSIBLE_REMOTE_TEMP=$HOME/.ansible/tmp"] galaxy_file = "${var.galaxy_file}" inventory_file_template = "[consul_instances]\n{{ .HostAlias }} ansible_host={{ .Host }} ansible_user={{ .User }} ansible_port={{ .Port }}\n" } }</code></pre> Running packer build</code> in CI results in a failure of a task defined in the Ansible playbook. The task simply creates a new group: - name: Add Consul group group: name: "{{ consul_group }}" state: present when: - consul_manage_group | bool</code></pre> group</code></a> runs groupapp</code> command behind the scenes and you should be in the list of sudoers to actually create new groups. Since I already have become: true</code> and become_user: root</code> in my playbook that requirement is fulfilled. Moreover, this task runs just fine in the local as I mentioned above. While running in CI, I see the following error: amazon-ebs.golden-ami: TASK [consul : Add Consul group] *********************************************** amazon-ebs.golden-ami: fatal: [default]: FAILED! => {"changed": false, "msg": "groupadd: Permission denied.\ngroupadd: cannot lock /etc/group; try again later.\n", "name": "consul"}</code></pre> Erhm, okayyy. That looks like a permission error. But why does this not happen in my local was the question eating me up. Now was the time to start from the ground up and dissect different things going here. I will add a small hint though: The Gitlab CI runner is a Docker-based</a> runner. That means all the commands like packer build</code> etc happen inside a Docker container. I am using hashicorp/packer:light</code> image, which is an Alpine based image containing just the packer</code> executable. I tried to run the container locally with: docker run -v `pwd`:/app --rm -it --entrypoint='' hashicorp/packer:light sh</code></pre> And yes. When I ran packer build</code>, I could replicate the issue here! But wait. More questions than answers. Ansible would run this groupadd</code> command on the remote host, right? Why does Ansible care if it’s inside a container or not? So, I created a really simple playbook to reproduce this further. - name: Assemble Consul cluster hosts: localhost any_errors_fatal: true become: true become_user: root tasks: - name: Add Consul group group: name: debug_consul state: present</code></pre> I ran this ansible-playbook test.yml</code> inside the container and… it worked! Okay, now it’s becoming clear. We aren’t executing ansible-playbook</code> directly, it’s being wrapped by Packer. So this is clearly getting messed up by Packer. This time I did find a GitHub issue</a> where people were asking about similar issues and this person described it well: I opened Packer docs</a> again and that’s when I read this: We recommend against running Packer as root; if you do then you won’t be able to successfully run your Ansible playbook as root; become: yes will fail. </blockquote> WTF!!! This was right there in the docs, hiding in plain sight. Sheesh! Okay, so I can not run my playbook with become: true</code> with the packer Image (which uses root user). Time to fix that by building a custom image. That is because Gitlab CI won’t let me change the user</a> without hacking stuff. And a custom image also allows me to ditch Alpine for Ubuntu, which is what I prefer. FROM ubuntu:latest RUN apt-get update && apt-get install -y \ python3 \ git \ curl \ unzip \ python3-pip \ && rm -rf /var/lib/apt/lists/* RUN curl -o packer.zip https://releases.hashicorp.com/packer/1.7.8/packer_1.7.8_linux_amd64.zip RUN unzip packer.zip RUN mv packer /usr/local/bin RUN pip3 install ansible RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 ubuntu USER ubuntu WORKDIR /tmp ENV PATH="$HOME/.local/bin:$PATH" WORKDIR /app</code></pre> Using this custom image, the packer build worked just fine. Fun day indeed (/s). Fin! Load testing with K6 2021-10-29T00:00:00+05:30 This week I was occupied with optimising a Golang program I’d written at work. I wanted a way to reproduce the issue under heavy load on my development environment and Load Tests are a good way to do that. The service in question is a RESTful API so it’s relatively easy to use any HTTP load test tools. The endpoint had an input parameter uuid</code> which accepted a valid UUIDv4</code> as the input. To my surprise, this was not so straightforward with hey</a> (which is my tool of choice for simple tests) and ab</a>. While it was possible to write an external script to do that, I thought to look around at some “scriptable” alternatives. I found wrk</a> which allowed me to write custom Lua modules. Now, I didn’t want to lose my focus from the main task which was load testing my service to write Lua, so I didn’t use wrk</code> but it’s still a pretty decent option (and very very fast, at that). Hello k6!#</a></h2> Some more Google-fu resulted in me finding k6</a>. I’d never heard of this but after exploring the GitHub repo</a> and the docs it looks like a pretty active project. So, k6</code> basically allows you to write scriptable tests which allow you to test a variety of scenarios</a>. The scripts are written in Javascript and treated as ES6 Modules for extensibility. k6</code> has a concept of Virtual Users</a> to mimic a real-world user. Each VU runs the “script” in an isolated self-contained JS runtime using Goja</a>. Now obviously at this point, if speed is your utmost concern to generate very heavy load tests, I guess wrk</code> is your only real choice as invoking a JS runtime inside Go won’t be super fast. But for most use-cases and people, like my case, this will just be fine. Basic Usage#</a></h2> Anyway, I quickly grokked the docs and copy-pasted some examples and modified them to what I needed. I was able to run a basic load test running very quickly and admired the simplicity here. It generated some p90, p95 etc stats which were helpful to look at. Here’s a basic example of how the script looks: import http from 'k6/http'; import { check, sleep } from 'k6'; import { Rate } from 'k6/metrics'; import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.0.0/index.js"; export const errorRate = new Rate('errors'); export default function () { const url = 'https://httpbin.org/post'; const params = { headers: { 'Content-Type': 'application/x-www-form-urlencoded', }, }; const data = { custname: "hello", comments: uuidv4(), }; check(http.post(url,data, params), { 'status is 200': (r) => r.status == 200, }) || errorRate.add(1); sleep(0.5); }</code></pre> To run it: k6 run -d 10s -u 10 httpbin_load.js</code></pre> (Here -d</code> is for the duration to run the test and -u</code> is to specify Virtual User</code>) Explanation: It’s an HTTP POST</code> request with some form data to https://httpbin.org/post</a></li> We use the uuid</a> function because the JS stdlib is great at providing basic helper methods (/s)</li> We define a check for HTTP status code as 200. Later we’ll see how to add more real-world checks under heavy load.</li> We have a sleep</a> function to pause a little bit before each iteration. This is pretty important as leaving sleep</code> is akin to a user pressing F5 on a browser non-stop and you’d probably not want your load tests to be that aggressive. Read docs</a> for more info.</li> </ul> Output: ❯ k6 run -d 30s -u 10 test.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: test.js output: - scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop): * default: 10 looping VUs for 30s (gracefulStop: 30s) running (0m30.6s), 00/10 VUs, 387 complete and 0 interrupted iterations default ✓ [======================================] 10 VUs 30s ✓ status is 200 checks.........................: 100.00% ✓ 387 ✗ 0 data_received..................: 318 kB 10 kB/s data_sent......................: 78 kB 2.5 kB/s http_req_blocked...............: avg=28.65ms min=210ns med=857ns max=1.1s p(90)=1.49µs p(95)=1.68µs http_req_connecting............: avg=7.77ms min=0s med=0s max=301.24ms p(90)=0s p(95)=0s http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms { expected_response:true }...: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms http_req_failed................: 0.00% ✓ 0 ✗ 387 http_req_receiving.............: avg=177.35µs min=39.43µs med=170.18µs max=604.22µs p(90)=259.75µs p(95)=285.73µs http_req_sending...............: avg=268.41µs min=48.62µs med=216.61µs max=7.95ms p(90)=334.94µs p(95)=448.21µs http_req_tls_handshaking.......: avg=15.87ms min=0s med=0s max=615.25ms p(90)=0s p(95)=0s http_req_waiting...............: avg=253.61ms min=215.34ms med=230.77ms max=826.57ms p(90)=316.98ms p(95)=324.69ms http_reqs......................: 387 12.666038/s iteration_duration.............: avg=784ms min=717ms med=732.43ms max=1.92s p(90)=820.35ms p(95)=926.87ms iterations.....................: 387 12.666038/s vus............................: 10 min=10 max=10 vus_max........................: 10 min=10 max=10</code></pre> Things to look for: From the above output, I think these 2 metrics are the most important to look at: http_req_duration..............: avg=254.05ms min=215.63ms med=231.15ms max=826.99ms p(90)=317.33ms p(95)=325.1ms http_reqs......................: 387 12.666038/s</code></pre> We see the total requests sent in 30s were 387</code> and the p95</code> response time is 325.1ms</code>. Testing some real-world scenarios#</a></h2> This was a really simple example but we can add some more scenarios to mimic real-world checks. Let’s tweak the script to Go from 1 to 10 users in 10s.</li> Stay at 10 users for 5s.</li> Ramp down to 1 user for the next 15s.</li> Have a threshold of not exceeding 500ms as p95.</li> Have a threshold for the count of non 200 OK</code> responses.</li> </ul> The above script now becomes: import http from 'k6/http'; import { check, sleep } from 'k6'; import { Rate } from 'k6/metrics'; import { uuidv4 } from "https://jslib.k6.io/k6-utils/1.0.0/index.js"; export const errorRate = new Rate('non_200_requests'); export let options = { stages: [ // Ramp-up from 1 to 10 VUs in 10s. { duration: "10s", target: 10 }, // Stay at rest on 10 VUs for 5s. { duration: "5s", target: 10 }, // Linearly ramp down from 10 to 0 VUs over the last 15s. { duration: "15s", target: 0 } ], thresholds: { // We want the 95th percentile of all HTTP request durations to be less than 500ms "http_req_duration": ["p(95)<500"], // Thresholds based on the custom metric `non_200_requests`. "non_200_requests": [ // Global failure rate should be less than 1%. "rate<0.01", // Abort the test early if it climbs over 5%. { threshold: "rate<=0.05", abortOnFail: true }, ], }, }; export default function () { const url = 'https://httpbin.org/post'; const params = { headers: { 'Content-Type': 'application/x-www-form-urlencoded', }, }; const data = { custname: "hello", comments: uuidv4(), }; check(http.post(url,data, params), { 'status is 200': (r) => r.status == 200, }) || errorRate.add(1); sleep(Math.random() * 1 + 1); // Random sleep between 1s and 2s. }</code></pre> Run with k6 run test.js</code>: 2.21.0 on ☁️ (ap-south-1) took 24s ❯ k6 run test.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: test.js output: - scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop): * default: Up to 10 looping VUs for 30s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s) running (0m30.3s), 00/10 VUs, 105 complete and 0 interrupted iterations default ✓ [======================================] 00/10 VUs 30s ✓ status is 200 checks.........................: 100.00% ✓ 105 ✗ 0 data_received..................: 126 kB 4.2 kB/s data_sent......................: 25 kB 841 B/s http_req_blocked...............: avg=73.69ms min=292ns med=761ns max=1.02s p(90)=1.54µs p(95)=688.76ms http_req_connecting............: avg=21.67ms min=0s med=0s max=245.46ms p(90)=0s p(95)=223.77ms ✓ http_req_duration..............: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms { expected_response:true }...: avg=252.72ms min=215.88ms med=230.31ms max=560.17ms p(90)=299.12ms p(95)=406.94ms http_req_failed................: 0.00% ✓ 0 ✗ 105 http_req_receiving.............: avg=177.85µs min=103.82µs med=163.76µs max=366.5µs p(90)=235.86µs p(95)=266.39µs http_req_sending...............: avg=258.17µs min=96.92µs med=215.88µs max=958.67µs p(90)=410.05µs p(95)=487.18µs http_req_tls_handshaking.......: avg=50.22ms min=0s med=0s max=614.14ms p(90)=0s p(95)=460.3ms http_req_waiting...............: avg=252.29ms min=215.17ms med=229.69ms max=559.86ms p(90)=298.42ms p(95)=406.52ms http_reqs......................: 105 3.471037/s iteration_duration.............: avg=1.84s min=1.22s med=1.85s max=3.02s p(90)=2.22s p(95)=2.46s iterations.....................: 105 3.471037/s vus............................: 1 min=1 max=10 vus_max........................: 10 min=10 max=10</code></pre> We can see that all the checks passed without breaching any thresholds we’d set. Some important points: In my local environment, I stress tested my service with 10k VUs which is quite a high number for the service but it was good to see it hold under extreme conditions as well. An important thing to note if you are spawning many VUs is that ulimit</code> number should be high. This is described in their docs</a> as well. </li> To debug the HTTP response you can run with --http-debug="full"</code> flag and get the verbose output for debugging. </li> </ul> Summary#</a></h2> I’ve barely scratched the surface of what this tool does. You can export metrics to various data sources, add a lot more checks on the response code, use it with GRPC or WebSockets as well. Overall pretty happy with this tool and I am going to use more of it for future projects. References#</a></h3> https://k6.io/blog/comparing-best-open-source-load-testing-tools/</a></li> https://k6.io/our-beliefs/#simple-testing-is-better-than-no-testing</a></li> https://k6.io/docs/</a></li> https://github.com/grafana/k6</a></li> </ul> Fin Analyzing CoreDNS logs with Clickhouse and Vector 2021-06-05T00:00:00+05:30 I’ve been toying around Clickhouse</a> and Vector</a> at my day job and find both of these tools pretty interesting at what they do. A short summary for those unaware of these tools: Vector helps you build a pipeline for collecting, transforming and processing different kinds of observability data (logs and metrics).</li> Clickhouse is a columnar based DB used as a warehousing tool for generating reports and analytics.</li> </ul> Now, for the context, I use coredns</code> on my personal dev machine as it supports Split DNS (routing certain zones to a particular resolver) which I need for accessing internal domains at work. Yep, systemd-resolved</code> can also do this, but I find coredns</code> easier to configure and manage with OpenVPN as well. Anyway, so one random evening, I got the idea of dumping CoreDNS Logs to Clickhouse. Maaaybe I was still hungover from the Vector/Clickhouse work I was doing at work but nevertheless I was interested in it. Overview#</a></h2> Gather logs from CoreDNS.</li> Transform the logs with regex</code> and construct a payload for Clickhouse.</li> Write a schema for the logs table.</li> Dump to Clickhouse.</li> </ul> Here’s how the pipeline looks: Collecting Logs#</a></h2> First, let’s look at how the raw logs are structured by CoreDNS: [INFO] 127.0.0.1:55678 - 21963 "A IN xpui.app.spotify.com. udp 38 false 512" NXDOMAIN qr,rd,ra 121 0.061416978s [INFO] 127.0.0.1:59333 - 22742 "AAAA IN incoming.telemetry.mozilla.org. udp 48 false 512" NOERROR qr,aa,rd,ra 106 0.049235139s [INFO] 127.0.0.1:39609 - 47247 "AAAA IN ping.archlinux.org. udp 36 false 512" NOERROR qr,rd,ra 140 0.056721154s</code></pre> Vector provides a variety of sources</a> to collect these logs. Since I am running coredns</code> as a Docker container, the following config shows how to collect logs from a particular container: # Filter coredns logs from Docker logs [sources.coredns_logs] type = "docker_logs" # required docker_host = "unix:///var/run/docker.sock" include_images = ["coredns/coredns"] # optional, no default</code></pre> The above config basically specifies a Docker Host variable and an image name filter. Vector talks to the Docker API over a unix</code> socket and gathers metadata about the container (like container_created_at</code>, container_name</code>, label</code> etc). After collecting logs from Vector, it enriches with the following metadata. { "container_created_at": "2021-06-04T14:18:03.967143133Z", "container_id": "00c5c4d36ea5b4772b517d3cca7d397c92f72be2a2bf45bb8c430f717fbd331e", "container_name": "coredns_coredns_1", "host": "iris", "image": "coredns/coredns", "label": { "com.docker.compose.config-hash": "928d71143c2af6553d551dbbf14140304d53f92378746454fbfeb0382a896d5b", "com.docker.compose.container-number": "1", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "coredns", "com.docker.compose.project.config_files": "/home/karan/Code/Infra/coredns/hydra-vpn-compose.yml", "com.docker.compose.project.working_dir": "/home/karan/Code/Infra/coredns", "com.docker.compose.service": "coredns", "com.docker.compose.version": "1.29.2" }, "message": "[INFO] 127.0.0.1:38266 - 20341 \"A IN open.spotify.com. udp 34 false 512\" NOERROR qr,rd,ra 160 0.300268123s", "source_type": "docker", "stream": "stdout", "timestamp": "2021-06-04T16:13:07.454601872Z" }</code></pre> (NOTE: I am using console</a> sink to dump these logs to STDOUT</code>. It’s pretty handy for inspecting logs). # Print parsed logs to stdout [sinks.print] type = "console" inputs = ["coredns_logs"] encoding.codec = "json"</code></pre> As you can see from the above JSON object, Vector has transformed the log with its own Data Model. The log that we care about now is inside .message</code> key. It’s nice to have other metadata as well. Transforming the logs#</a></h3> Our objectives at this step: Discard unused fields. We don’t really care about container metadata for this mini-project.</li> Parse the message</code> field with regex</code> so they can be stored in individual columns in our Clickhouse table.</li> </ul> Now, CoreDNS can emit two kinds of logs (INFO</code> and ERROR</code>). The error usually happens when the upstream resolver is unreachable or there’s an issue with any of the CoreDNS plugins. We need to write regex for both cases: INFO</code> logs:</li> </ol> \[(?P<level>[^]]+)]\s(?P<server_addr>[^:]+):(?P<server_port>\S+)\s+-\s+(?P<id>\S+)\s+(?P<type>\S+)\s+(?P<class>\S+)\s+(?P<name>\S+)\s+(?P<proto>\S+)\s+(?P<size>\S+)\s+(?P<do>\S+)\s+(?P<bufsize>[^"]+)"\s+(?P<rcode>\S+)\s+(?P<rflags>\S+)\s+(?P<rsize>\S+)\s+(?P<duration>[\d\.]+).*</code></pre> ERROR</code> logs:</li> </ol> \[(?P<level>ERROR)]\s+(?P<component>plugin\/errors):\s+(?P<code>\S)+\s+(?P<name>\S+)\s+(?P<type>[^:]*):\s+(?P<error_msg>.*)</code></pre> Combining a bunch of other things to remove some fields and constructing the final payload, the config looks like this: # Parse coredns logs [transforms.parse_logs] type = "remap" inputs = ["coredns_logs"] source = ''' # parse the log event. ts = .timestamp log,err = parse_regex(.message,r'\[(?P<level>[^]]+)]\s(?P<server_addr>[^:]+):(?P<server_port>\S+)\s+-\s+(?P<id>\S+)\s+"(?P<type>\S+)\s+(?P<class>\S+)\s+(?P<name>\S+)\s+(?P<proto>\S+)\s+(?P<size>\S+)\s+(?P<do>\S+)\s+(?P<bufsize>[^"]+)"\s+(?P<rcode>\S+)\s+(?P<rflags>\S+)\s+(?P<rsize>\S+)\s+(?P<duration>[\d\.]+).*') if err !=null { # capture the error log. If the error log also fails to get parsed, the log event is dropped. log = parse_regex!(.message,r'\[(?P<level>ERROR)]\s+(?P<component>plugin/errors):\s+(?P<code>\S)+\s+(?P<name>\S+)\s+(?P<type>[^:]*):\s+(?P<error_msg>.*)') } . = log # add timestamp .timestamp = ts # remove fields we dont care about del(.do) ''' drop_on_error = true</code></pre> Apart from regex matching, we store the timestamp</code> as received from Vector (since CoreDNS logs don’t contain any timestamp information). We delete some fields that we don’t care about. Vector uses a powerful DSL (called VRL</a>) to do such kind of transformations on the fly. It has a lot of functions to do almost any kind of transformation to your original event payload. You can invoke vector vrl</code> from the terminal and get a shell to write the above transformations and debug quickly. It proved to be really useful when dealing with such a long regex pattern. Storing in Clickhouse#</a></h2> Finally we get to the part where we need to dump these logs to our Clickhouse DB. Here’s the schema for the table where we will be storing these records: CREATE DATABASE IF NOT EXISTS `coredns`; CREATE TABLE IF NOT EXISTS `coredns`.`logs` ( `timestamp` DateTime('Asia/Kolkata'), `bufsize` Int32, `class` LowCardinality(String), `duration` Float64, `id` Int32, `level` LowCardinality(String), `name` String, `proto` LowCardinality(String), `rcode` LowCardinality(String), `rflags` String, `server_addr` String, `server_port` Int32, `rsize` Int32, `size` Int32, `type` LowCardinality(String) ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY toYYYYMMDD(timestamp) TTL timestamp + INTERVAL 1 WEEK;</code></pre> Key things to note: LowCardinality</code></a> is used for columns where the data is predictable to reduce the disk space used.</li> Clickhouse uses the sort key as the primary key if unspecified. This is the default behaviour.</li> TTL for the records is set to 1 week. After 1 week all the records will be purged. Since this is my dev machine, I don’t really care about a higher TTL.</li> This also means that the partition is never really put to use since I am partitioning by month but logs are being deleted every week. At this scale, it doesn’t really make sense to even have it, but I just included it for posterity.</li> </ul> UPDATE: Clickhouse on Twitter clarified</a> that ORDER BY timestamp</code> will have better performance in this context. Usually if your queries are “last 1h”, “last 5m” based, it is better to not store the the sort key as YYYYMMDD</code> format. Now, we need to instruct Vector to send these logs to Clickhouse: [sinks.coredns_logs_clickhouse_output] type = "clickhouse" inputs = ["parse_logs"] compression = "gzip" database = "coredns" endpoint = "http://localhost:8123" table = "logs" encoding.timestamp_format = "unix" batch.timeout_secs = 10</code></pre> Clickhouse offers an HTTP API (which is running on port 8123 by default). Vector takes the input from the previous step (parse_logs</code> transformation) and sends it to Clickhouse over the HTTP interface. Clickhouse stores datetimes in UNIX, so before sending the data, Vector can encode certain fields in the payload to a different data type as well (isn’t that cool 😎) Query Examples#</a></h2> I’ve been running this pipeline for 3-4 days, so I have a decent amount of data collected to show for the blog post. Total Count of Queries SELECT count(*) FROM coredns.logs ┌─count()─┐ │ 16774 │ └─────────┘</code></pre></li> Top Query Types SELECT count(*) AS total, type FROM coredns.logs GROUP BY type ORDER BY total DESC LIMIT 5 ┌─total─┬─type─┐ │ 9931 │ A │ │ 6852 │ AAAA │ └───────┴──────┘</code></pre></li> Top Query Names SELECT count(*) AS total, name FROM coredns.logs GROUP BY name ORDER BY total DESC LIMIT 5 ┌─total─┬─name────────────────────────────┐ │ 2513 │ ping.archlinux.org. │ │ 1868 │ ws.todoist.com. │ │ 1011 │ incoming.telemetry.mozilla.org. │ │ 802 │ vortex.data.microsoft.com. │ │ 707 │ logs-01.loggly.com. │ └───────┴─────────────────────────────────┘</code></pre></li> Max/Min duration SELECT max(duration) AS max, min(duration) AS min FROM coredns.logs FORMAT Vertical Row 1: ────── max: 4.056606352 min: 0.000020837</code></pre></li> Top TLDs Queried SELECT count(name) AS total, topLevelDomain(substring(name, 1, -1)) AS tld FROM coredns.logs GROUP BY tld ORDER BY total DESC LIMIT 10 ┌─total─┬─tld──┐ │ 10666 │ com │ │ 3950 │ org │ │ 671 │ net │ │ 346 │ so │ │ 288 │ tech │ │ 279 │ io │ │ 190 │ co │ │ 167 │ dev │ │ 82 │ arpa │ │ 43 │ in │ └───────┴──────┘</code></pre></li> </ul> Well, that’s all I could think of really. If you’ve some more interesting analysis to get from this data, let me know! Summary#</a></h2> The intention behind writing this post is to give an overview of how the entire log collection and processing pipeline works. Using Vector has been an amazing experience however the sad bit is that I don’t know Rust and I cannot contribute to some of the issues I’ve opened (even though they are presumably trivial). Maybe I should pick up Rust, finally? 🤭 Thanks for reading! How I take Notes 2021-04-14T08:10:55+05:30 Over the past 2-3 years, note-taking apps have become all the rage. Note-taking is an extremely subjective topic and a lot of it depends on the individual’s workflow. There’s no one size fits all and maybe that justifies the ever-expanding landscape of such apps. I’ve tried a few popular ones (Notion, Roam Research) in the recent past but never really quite stuck to any after the initial hype phase. I even collaborated with @iamd3vil</a> to make our own version</a> of the Zettelkasten based note-taking app. I found the Zettelkasten system to be really useful on paper but then again, I didn’t use it after a few weeks. Tried the old school way of Bullet Journal (and I did end up liking it quite a lot) but it was not so practical in many cases (like documenting code snippets, URLs etc). Disgruntled with all the options, I just had a simple folder on my laptop with some markdown files in it. It was a stop-gap solution until I found something better. TL;DR: I’ve been through a pendulum phase of finding a new note-taking app, wasting time to set it up the “proper way” and then just end up not using it. Enter Joplin#</a></h2> Thanks to @shantanugoel</a> who introduced me to Joplin</a>. He’s a heavy user</a> of it as well and that gave me some confidence to try it out. I initially disliked it because of not-so-great-looking UI theme and how it essentially looked just an editor. Admittedly, I was proven wrong quickly in my initial judgement. As I gave more time to it, I noticed I kept coming back to Joplin “naturally” and stuck through it because it’s so damn simple to use. Notion, for people who have tried it would know it complicates a lot of simple tasks. You need to create databases to render a simple table, every component is a “block” (a new page) and yes it’s slow (although they are working on it to make it better</a>, just to be fair). Notion focuses a lot on team collaboration features, which I didn’t need for my “personal” note-taking system. TL;DR: Joplin is fast, it’s open-source, it’s based on Markdown, and it’s simple to use. A tool you just forget that it exists, because it becomes a natural extension to your workflow. It has a great plugin system that you can use to extend it and build your own utilities on top of it. The search is based on sqlite3</code> FTS which is pretty awesome! Workflow#</a></h2> Joplin revolves around the concept of notebooks. Notebooks are broader categories for your content and you can nest multiple subnotebooks for specific categories. I’ve the following notebooks and subnotebooks in my Joplin setup: - Bookmarks - Twitter Threads - HN Threads - Articles - Design Inspo - Youtube Videos - Inbox - Links - Adhoc Notes - Personal - Finance - Dev Setup - OSS Ideas - Self Hosted Setup - Work - Org-Stuff - Redacted</code></pre> Over the week, I primarily use the Inbox/Adhoc Notes</code> notebook as a brain dump. I don’t focus much on the structure, the aim is to get the content out and stored. I’m also someone who doesn’t like to keep more than 5 browser tabs open at any time, so I use Links</code> notebook with Joplin’s Web Clipper</a> service to store these links to read later. Every weekend, I clean up these notebooks to achieve “Inbox Zero”. The idea is to move all these ad-hoc notes to their proper notebooks, annotated with tags. All the useful links are moved in the appropriate Bookmarks/...</code> notebook as well. This helps me find stuff quicker at a later time. I heavily use Tags in all my notebooks, which allows me to have a unified view of different kind of stuff I’ve. For example “golang” tag in my Work notes and Personal notes, allows me to see all the “golang” stuff together in one place. For stuff that can be shared publicly, I basically copy-paste those notes in my public wiki</a> as well. This allows me to share snippets/commands with others, which Joplin cannot do. Synchronisation#</a></h2> Joplin provides a bunch of different sync</a> options. I’ve tried Dropbox, Nextcloud and AWS S3 targets in the past but off late there’s a new sync option, Joplin Server</a> which provides native sync for Joplin files. I found this option to be the best so far because Dropbox/OneNote etc have API limits and syncing on an initial device with lots of notes will be time-consuming. I self-host this Joplin Sync Server on my server and have configured the Android app and the Desktop app to use this server endpoint as the sync target</code>. So far so good, although it’s relatively a newer sync option so it’s pertinent to have alternate backups. Backups and Export#</a></h2> Joplin stores all the files locally on a device in a sqlite3</code> DB. It can export notes in markdown/HTML format however the file names are all named with the id</code> of the note (and not really the title of the note). I found this to be a bit of a drawback, however, one can quickly whip up a small Python script to fix this using the Joplin API. Joplin also has the option to export all of the notes and notebooks with their metadata (Geolocation, creation time etc), tags in a custom format called Joplin Export File (JEX). This option is pretty convenient to re-import in a new Joplin installation as well. Sidenote for people using Joplin Server: Once https://github.com/laurent22/joplin/issues/4836 gets resolved, it’ll be possible to do joplin sync</code> and cron it, just like other sync targets. Support#</a></h2> Joplin is 100% FOSS and is actively developed by @laurent22</code></a> and a few other regular contributors. I contribute $5/mo</code> to laurent22 via Github Sponsors. It’s more or less the same amount that most note-taking apps charge for personal use as well, so this is just me expressing gratitude for building such a lovely software for the world to use it. I’m not sure of the motivations of laurent22</code> behind building this and I don’t wanna incorrectly assume anything either, but I guess some amount of financial incentive makes the whole deal sustainable for the open-source ecosystem. Thanks for reading! It’s been around a year that I am using Joplin and I posted this blog post only after really really using it a lot. I’d love to know about your note-taking setups too, so please reach out to me on the usual channels that I’m available on and feel free to discuss! Fin! (Bonus Section) Why Not Obsidian#</a></h4> Yes, Obsidian is comparable to Joplin in a lot of ways. However there’s a term in the license</a> of Obsidian for personal use that makes it impossible to use it for your work stuff: You need to pay for Obsidian if and only if you use it for revenue-generating, work-related activities in a company that has two or more people. Get a commercial license for each user if that’s the case </blockquote> I don’t have a problem for “paying” for software but such kinda licenses are just BS. Running Nomad for home server 2021-02-14T08:10:55+05:30 It’s been a long time since I’ve written a post on Hydra (my home server). I use Hydra as a testbed to learn new tools, workflows and it just gives me joy to self-host applications while learning something in return. History#</a></h2> A brief history of how Hydra’s</a> setup evolved over time: 2019</a>: A pretty minimal K3s setup deployed on 2 RPi4 nodes. I couldn’t continue with this setup because: Some of the apps didn’t have ARM-based image (this was 2019, pre M1 hype era).</li> Didn’t want to risk deploying persistent workloads on RPi.</li> A lot of tooling to deploy workloads was missing (storing env variables for eg.).</li> It was so boring to write YAML (that I also did at work). Didn’t give me joy.</li> </ul> </li> </ul> 2020 First Half</a>: RPi 2x Nodes + K3s + DO Droplet. Tailscale for networking. This was a considerable step up from the previous setup. I deployed a DO node and added Node Labels</a> to deploy persistent workloads on DO Node only.</li> I used my own tooling Kubekutr</a> + Kustomize which helped with version control of my configs.</li> Took quite a bit of time to onboard new services. Got lazy, didn’t host much apart from initial 3-4 applications.</li> Writing long YAMLs. No joy.</li> </ul> </li> </ul> 2020 Second Half: Single node on DO. Terraform for deploying Docker containers. I believe the third iteration nailed it for me. I kept the setup super simple, used Terraform for deploying workloads as Docker containers.</li> Used Terraform extensively for setting up the node, Cloudflare records, DO firewall rules.</li> Time to onboard new services reduced from a couple of hours to a few minutes. This was a huge win for me. I deployed around 10-15 new services to try it out on the server directly.</li> Writing HCL is actually a much better experience than YAML.</li> </ul> </li> </ul> Why Nomad#</a></h2> Around a month back, Kailash</a> had asked about feedback on Nomad</a>. We at Zerodha</a> (India’s largest stock broker) are evaluating it to migrate our services to Nomad from Kubernetes (more on this later). It was almost 2 years since I last saw Nomad so it was definitely worth re-evaluating (esp since it hit 1.0 recently). I wanted to try out Nomad to answer a personal curiosity: What does it do differently than Kubernetes? No better way than actually getting hands dirty, right?! After following the brief tutorials from the official website</a> I felt confident to try it for actual workloads. In my previous setup, I was hosting quite a few applications (Pihole, Gitea, Grafana etc) and thought it’ll be a nice way to learn how Nomad works by deploying the same services in the Nomad cluster. And I came in with zero expectations, I already had a nice setup which was reliable and running for me. My experience with a local Nomad cluster was joyful, I was able to quickly go from 0->1 in less than 30 minutes. This BTW is a strong sign of how easy Nomad is to get started with as compared to K8s. The sheer amount of different concepts you’ve to register in your mind before you can even deploy a single container in a K8s cluster is bizarre. Nomad takes the easy way out here and simplified the concepts for developers into just three things: job \_ group \_ task</code></pre> Job: Job is a collection of different groups. Job is where the constraints for type of scheduler, update strategies and ACL is placed.</li> Group: Group is a collection of different tasks. A group is always executed on the same Nomad client node. You’ll want to use Groups for use-cases like a logging sidecar, reverse proxies etc.</li> Task: Atomic unit of work. A task in Nomad can be running a container/binary/Java VM etc, defining the mount points, env variables, ports to be exposed etc.</li> </ul> If you’re coming from K8s you can think of Task as a Pod and Group as a Replicaset. There’s no equivalent to Job in K8s. BUT! The coolest part? You don’t have to familiarise yourself with all different types of Replicasets (Deployments, Daemonsets, Statefulsets) and different ways of configuring them. Want to make a normal job as a periodic job in Nomad? Simply add the following block to your existing Job: periodic { cron = "@daily" }</code></pre> You want to make a service run as a batch job (on all Nomad nodes – the equivalent of Daemonset in K8s)? Simply make the following change to your existing job: -type="service" +type="batch"</code></pre> You see this is what I mean by the focus on UX. There are many many such examples which will leave a nice smile on your face if you’re coming from K8s background. I’d recommend reading Internal Architecture</a> of Nomad if you want to understand this in-depth. Architecture#</a></h2> Tech stack for Hydra: Tailscale VPN: Serves as a mesh layer between my laptop/mobile and DO server. Useful for exposing internal services.</li> Caddy for reverse proxying and automatic SSL setup for all services. I run 2 instances of Caddy: Internal: Listens on Tailscale Network Interface. Reverse proxies all private services.</li> Public: Listens on DO’s Public IPv4 network interface. Reverse proxies all public-facing services.</li> </ul> </li> Terraform: Primary component to have IaC (Infra as Code). Modules to manage: Cloudflare DNS Zone and Records</li> DO Droplet, Firewall rules, SSH Keys, Floating IPs etc.</li> Nomad Jobs. Used for running workloads after templating env variables, config files in Nomad job files.</li> </ul> </li> </ul> Complexity of Nomad vs Kubernetes#</a></h2> </a> Nomad shines because it follows the UNIX philosophy of “Make each program do one thing well”. To put simply, Nomad is just a workload orchestrator. It only is concerned about things like Bin Packing, scheduling decisions. If you’re running heterogeneous workloads, running a server (or a set of servers) quickly becomes expensive. Hence orchestrators tend to make sense in this context. They tend to save costs by making it efficient to run a vast variety of workloads. This is all an orchestrator has to do really. Nomad doesn’t interfere in your DNS setup, Service Discovery, secrets management mechanisms and pretty much anything else. If you read some of the posts at Kubernetes Failure Stories</a>, the most common reason for outages is Networking (DNS, ndots etc). A lot of marketing around K8s never talks about these things. I always maintain “Day 0 is easy, Day N is the real test of your skills”. Anyone can deploy a workload to a K8s cluster, it’s always the Day N operations which involve debugging networking drops, mysterious container restarts, proper resource allocations and other such complex issues that require real skills and effort. It’s not as easy as kubectl apply -f</code> and my primary gripe is with people who miss out on this in their “marketing” pitches (obvious!). When to use Nomad#</a></h2> Nomad hits the sweet spot of being operationally easy and functional. Nomad is a great choice if you want to: Run not just containers but other forms of workloads.</li> Increase developer productivity by making it easier to deploy/onboard new services.</li> Consistent experience of deployment by testing the deployments locally.</li> (Not joking) You are tired of running Helm charts or writing large YAML manifests. The config syntax for Nomad jobs is human friendly and easy to grasp.</li> </ul> Nomad is available as a single binary. If you want to try it locally, all you need is sudo nomad agent -dev</code> and you’ll have a Nomad Server, Client running in dev mode along with a UI. This makes it easy for the developers to test out the deployments locally because there’s very little configuration difference between this and production deployment. Not to forget it’s super easy to self-host Nomad clusters. I’m yet to meet anyone who self hosts K8s clusters in production without a dedicated team babysitting it always. Once you eliminate the “blackbox” components from your stack, life becomes easier for everyone. When to not use Nomad#</a></h2> If you’re relying on custom controllers and operators. Operator Pattern</a> is a new way of managing large complex distributed systems (like databases, job queues etc). There are a lot of community built operators which help in reducing the effort to run these services. However, all of these are tied deeply into the “Kubernetes” ecosystem. If you find yourself running any of such operators, it’ll be tough (not impossible) to translate the same in Nomad ecosystem.</li> </ul> I genuinely cannot think of any other reason to not use Nomad! Practical Scenarios#</a></h2> Since I migrated a couple of workloads from my DO docker containers setup to Nomad, I’d demonstrate a few use cases which might be helpful if you want to start migrating your services to Nomad Accessing a Web service with Reverse Proxy#</a></h3> Context: I’m running Caddy as a reverse proxy for all the services. Since we discussed earlier, Nomad only is concerned about scheduling, so how exactly do you do Service Discovery? You need Consul (or something like Consul, Nomad has no hard restrictions) to register a service name with it’s IP Address. Here’s how you can do that: In the .task</code> section of your Nomad job spec, you need to register the service name with the port you’re registering and additional tags as metadata (optional): service { name = "gitea-web" tags = ["gitea", "web"] port = "http" }</code></pre> Nomad’s template</a> uses consul-template</code> behind the scenes. This is a small utility which continuously watches for Consul/Vault keys and provides the ability to reload/restart your workloads if any of those keys change. It can also be used to discover the address of the service registered in Consul. So here’s an example of Caddyfile</code> using Consul Template functions to pull the IP address of the upstream gitea-web</code> service: git.mrkaran.dev { {{ range service "gitea-web" }} reverse_proxy {{ .Address }}:{{ .Port }} {{ end }} }</code></pre> When a job is submitted to Nomad, a rendered template is mounted inside the container. You can define actions on what to do when the values change. For eg on a redeployment of Gitea container, the address will most likely change. We’d like Caddy to automatically restart with the new address configured in the Caddyfile in that case: template { data = <<EOF ${caddyfile_public} EOF destination = "configs/Caddyfile" # Rendered template. change_mode = "restart" }</code></pre> Using change_mode</code></a> we can either send a signal</code> or restart the task altogether. Binding to different network interfaces#</a></h3> I run a public instance of Gitea but I wanted to restrict the SSH access only to my Tailscale network. Nomad has an interesting feature host_network</code></a> which lets you bind different ports of a task on different network interfaces. network { port "http" { to = 3000 } port "ssh" { to = 22 # Need a static assignment for SSH ops. static = 4222 # SSH port on the host only exposed to Tailscale IP. host_network = "tailscale" } }</code></pre>Templating Env Variables#</a></h3> NOTE: This is not recommended for production. Nomad doesn’t have any templating functionalities, so all the config must be sourced from Consul and secrets should be sourced from Vault. However in the time constraint I had, I wanted to understand Nomad and Consul better and use Vault at a later stage</a>. I needed a way to interpolate the env variables. This is where Terraform comes into picture: resource "nomad_job" "app" { jobspec = templatefile("${path.module}/conf/shynet.nomad", { shynet_django_secret_key = var.shynet_django_secret_key, shynet_postgresql_password = var.shynet_postgresql_password }) hcl2 { enabled = true } }</code></pre> We can pass the variables from Terraform (which can be sourced by TF_VAR_</code> in your local env) to the Nomad job spec. Inside the job spec we can use env</code> to make it available to our task: env { DB_PASSWORD = "${shynet_postgresql_password}" DJANGO_SECRET_KEY = "${shynet_django_secret_key}" }</code></pre>Running a backup job on the host#</a></h3> I use restic</code> to take periodic backups of my server and upload to Backblaze B2. Since Nomad supports running tasks as a different isolated environment (chroot</code>) using exec</code> driver and even without isolation using raw_exec</code> driver, I wanted to give that a try. I’ve to resort using raw_exec</code> driver here because /data</code> file path on my host was not available to the chroot’ed environment. job "restic" { datacenters = ["hydra"] type = "batch" periodic { cron = "0 3 * * *" time_zone = "Asia/Kolkata" prohibit_overlap = true } ... task "backup" { driver = "raw_exec" config { # Since `/data` is owned by `root`, restic needs to be spawned as `root`. # `raw_exec` spawns the process with which `nomad` client is running (`root` i.e.). command = "$${NOMAD_TASK_DIR}/restic_backup.sh" } } ... }</code></pre> You can follow the rest of the config here</a>. Scope of Improvements#</a></h2> Nomad has been an absolute joy to work with. However, I’ve spotted a few rough edge cases which I believe one should be aware of: host_network</code> property sometimes gets ignored when doing a modification to service</code>. I’ve opened an issue</a> upstream but looks like other people are facing similar behaviours here</a> and here</a>.</li> host_network</code> as of present cannot</a> bind to a floating IP address (DigitalOcean/GCP etc). I’ve to resort to using my droplet’s public IPv4 address for now.</li> I tried using Consul Connect (service mesh with mTLS) but looks like again because of host_network</code>, I’m unable</a> to use it.</li> Nomad CLI can definitely be improved</a> for a much more consistent experience. I particularly missed using kubectl</code> when using nomad</code>.</li> </ul> That apart, I ended up sending a PR</a> to upstream addressing a CLI arg ordering issue. Gotchas:#</a></h3> On a Nomad server already bootstrapped, if you try changing server.bind_addr</code>, it won’t have any effect. I almost pulled my hair debugging this, ultimately deleting the data_dir</code> of the server resolved the issue for me.</li> I’m running DB and the App together as a single “group” in my setup configs. Don’t do this in production. Whenever you restart the job, the group will restart both the containers. The side effect of this is pretty interesting: Since we use Consul to fetch the DB Host, the app may start before the DB boots up and registers its new address with Consul. I will fix the dependency in a future version but since I’m running fewer workloads and there are automatic retries, it’s okay enough for me to keep it like this.</li> </ul> Community#</a></h2> Nomad’s community is pretty small compared to Kubernetes. However, the folks are super responsive on Gitter</a>, Discourse</a> and Github Issues. A few noteworthy mentions: @the-maldridge</a> helped me with my doubts in Gitter.</li> @tgross</a> who is super responsive on Github issues and does an excellent job at housekeeping the issues.</li> @shantanugadgil</a> who is also pretty active in the community.</li> </ul> Nomad’s ecosystem is still in its nascent stage and I believe there are a lot of contribution opportunities for folks interested in Golang, Ops, Distributed Systems to contribute to Nomad. The codebase of Nomad is approachable and there are quite a few key areas which can be contributed to: Docs: More examples, practical use cases.</li> Nomad Job files: There are many helm charts available to follow best practices. Something similar in Nomad will definitely be interesting.</li> Nomad Gotchas: Since K8s is widely used and has a much larger adoption, it’s only natural that the failure stories of K8s are highlighted a lot. Nomad being a pretty smaller community, we need more debugging and “things that went wrong” reference materials. You learn more from failures than 101 setup guides :)</li> </ul> Final Thoughts#</a></h2> I think I’m sold on Nomad. I’ve used Kubernetes in prod for 2 years but if you were to ask me to write a Deployment spec from scratch (without Googling/kubectl help) I won’t be able to. After writing Nomad configs, I just can’t think of the sheer amount of boilerplate that K8s requires to get an application running. Nomad is also a simpler piece to keep in your tech stack. Sometimes it’s best to keep things simple when you don’t really achieve any benefits from the complexity. Nomad offers less than Kubernetes and it’s a feature, not a bug. Fin! Discussions#</a></h4> HackerNews</a></li> Lobster</a></li> Twitter</a></li> </ul> Setup Gitlab Runner with AWS ECR 2021-01-29T08:10:55+05:30 There are some things you expect to just work. Sadly trying to make Gitlab Runner</a> with AWS ECR</a> turned out to be quite a daunting task and the little documentation in this area doesn’t help. There’s even a 4 years old issue</a> and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been. Anyway, since I spent a lot of time figuring out how to make a Private Registry work with a cross-account ECR, I’m documenting these steps hoping it’ll help someone someday :). The Problem#</a></h3> There are mainly 2 seemingly same but different problems when it comes to using ECR. Let’s discuss both of them separately: Pulling a private image from ECR using the Docker Executor. For eg, if your gitlab-ci.yml</code> looks like:</li> </ul> test-pull: image: $PRIVATE_ECR_IMAGE script: - echo "Hello World!"</code></pre> In this case, the Docker Executor needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE</code>. Pulling a private image inside the job. For eg, if you’re using Kaniko:</li> </ul> docker-build: image: gcr.io/kaniko-project/executor:debug script: - | /kaniko ... # Inside this step, we use a PRIVATE_ECR_IMAGE defined in our `Dockerfile`.</code></pre> In this case, Kaniko</a> needs to be “authenticated” to AWS ECR so that it can pull $PRIVATE_ECR_IMAGE</code>. NOTE I prefer Kaniko over DIND</a> as it is faster, doesn’t require running the privileged container, caching is simplified, and is in general a lot simpler to setup. The Solution#</a></h3> So, for the first case, where you want to authenticate the Docker Executor to AWS ECR, you’ll need 2 things: Setup DOCKER_AUTH_CONFIG</code> environment variable to { "credsStore": "ecr-login" }</code> in the config.toml</code> of the runner. For eg:</li> </ol> [[runners]] name = "Test" url = "https://gitlab.internal/" token = "REDACTED" executor = "docker" environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]</code></pre> Now, we’ve specified the Credential Store for Docker, but we don’t have this binary docker-credential-ecr-login</code> in our runner. AWS provides amazon-ecr-credential-helper</a> which is a neat way of automatically authenticating with AWS ECR based on your Access Keys/IAM role. What does automatic mean here? So, the normal docker login</code> is a basic auth command, where if you’ve to log in to ECR, you need to do something like:</li> </ol> aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com</code></pre> This is problematic because the authorization token is valid for 12 hours. Further, you’ve to log in to multiple registry IDs separately. Managing this is a nightmare, so Docker instead of just relying on Basic Auth, came up with a neat mechanism: docker-credential-helpers</a>. This allows you to keep your secret tokens in your Keystore. A new credential helper can be written in Go</code> which implements the credentials.Helper</code> interface. This is what amazon-ecr-credential-helper</a> does by offering various ways like AWS IAM Roles, Assumed Roles, Access Keys, etc to authenticate with ECR. This is where I stumbled the most. I downloaded the binary from the Github Releases</a> but this binary is statically compiled with muslc</code> libraries. However gitlab/gitlab-runner</code> is based on the ubuntu</code> docker image, so the above binary never worked. The strangest thing was the unhelpful error message that sh</code> returned as explained in this post</a>. To make things easier, I baked my own gitlab-runner</code> image with the above binary compiled inside the image using go get</code>: FROM ubuntu:20.04 AS build ENV DEBIAN_FRONTEND=noninteractive RUN : \ && apt-get update \ && apt-get install --no-install-recommends -y git golang-go ca-certificates \ && rm -rf /var/lib/apt/lists/* \ ; RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login WORKDIR /build RUN mv /root/go/bin/docker-credential-ecr-login . FROM gitlab/gitlab-runner:v13.8.0 AS deploy COPY --from=build /build/docker-credential-ecr-login /usr/local/bin/docker-credential-ecr-login</code></pre> The above image bakes in the docker-credential-ecr-login</code> binary and also puts it under /usr/local/bin</code> so it’ll be available under $PATH</code> to the Docker engine. With the above 2 things, if the runner’s server (EC2 instance/K8s pod) has access to the ECR image, it should be able to pull. Now coming to the 2nd problem, where we wanted kaniko</code> to authenticate to ECR, things are a bit simpler: Kaniko comes with docker-credential-ecr-login</code> baked in. All you need to do is add the following to ~/.docker/config.json</code> as explained here</a>.</li> </ul> { "credsStore": "ecr-login" }</code></pre> Next, we need to mount the AWS Credentials to Kaniko’s image so that it can use AWS SDK to perform a login to ECR. We do that by using volumes</code> of the runner</code>: [runners.docker] volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]</code></pre> This mounts the aws-credentials</code> file from the host inside the container which the runner spawned (kaniko</code> in this case). A sample aws-credentials</code> file if you’re using a cross-account access can look like: [default] role_arn=arn:aws:iam::ACCOUNT_ID:role/assume-role-{{ROLE_NAME}} credential_source=Ec2InstanceMetadata region=ap-south-1</code></pre> You can put in your normal AWS Keys or leave them blank if you want to use your IAM Role. With mounting the AWS Credentials inside Kaniko’s container, you can authenticate to cross-account ECRs as well (once you set up the whole assumed-role/trusted entities flow). Sample Runner#</a></h3> If you want to take a look at how the complete flow looks like: Start the Runner service using docker-compose</code>:</li> </ul> version: '3.7' services: test: # As explained above, this image has `docker-credentials-ecr-login` baked in. image: {{ACCOUNT_ID}.dkr.ecr.ap-south-1.amazonaws.com/custom/gitlab-runner:13.8.0 restart: always volumes: # Automatically created; config.toml. - './test/runner-config:/etc/gitlab-runner' # Mount AWS Credentials for Docker Executor to authenticate. - './test/aws-credentials:/root/.aws/credentials:ro' # Mount Docker Socket so that executor can communicate with it. - '/var/run/docker.sock:/var/run/docker.sock'</code></pre> Register a new runner:</li> </ul> docker-compose exec test register</code></pre> Fill in the basic info and edit ./test/runner-config.toml</code> with the following options: [[runners]] name = "Test" executor = "docker" environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"] [runners.docker] volumes = ["/home/ubuntu/runners/test/aws-credentials:/root/.aws/credentials:ro"]</code></pre>Conclusion#</a></h2> Honestly, this was a lot of trial and error to figure out how to use private images with Gitlab. Some important links and references that helped me figure this out: https://gitlab.com/bmares/gitlab-runner-ecr-auth-example/</a></li> https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_84649153</a></li> </ul> Fin! Designing a simple Job Queue in Golang 2020-12-01T08:10:55+05:30 In this post we’ll see how to create a simple job queue in Golang. There are tonnes of libraries and posts out there doing overly complicated stuff, however if your need is pretty minimal or you want to understand the concepts from ground up, this post aims to do just that and nothing more. We’ll be using concepts like WaitGroup</a>, Channels</a> and Contexts</a> to build our own Job Queuing mechanism. It primarily involves 2 components: Queue: A queue which has a list of items waiting to be processed. </li> Worker: A worker constantly listening to that queue and processing the events as desired. </li> </ul> With these 2 main ideas behind us, let us create our sample structure: package dispatch type Dispatcher interface { // Push takes an Event and pushes to a queue. Push(Event) error // Run spawns the workers and waits indefinitely for // the events to be processed. Run() } // EventDispatcher represents the datastructure for an // EventDispatcher instance. This struct satisfies the // Dispatcher interface. type EventDispatcher struct { Opts Options Queue chan models.Notification Finished bool } // Options represent options for EventDispatcher. type Options struct { MaxWorkers int // Number of workers to spawn. MaxQueueSize int // Maximum length for the queue to hold events. } // NewEventDispatcher initialises a new event dispatcher. func NewEventDispatcher(opts Options) (Dispatcher) { return EventDispatcher{ Opts: opts, Queue: make(chan Event, opts.MaxQueueSize), Finished: false, } }</code></pre>Pushing to Queue#</a></h3> Now that we have our basic structure ready, let’s write a function to push events to queue. A queue is simply a channel. We have created a new queue of size MaxQueueSize</code> while initialising the EventDispatcher</code>. Queue: make(chan Event, opts.MaxQueueSize)</code></pre> To push events into it, we’ll simply do: d.Queue <- event</code>. This adds a new item (event</code>) of type Event</code> to our queue. // Push adds a new event payload to the queue. func (d *EventDispatcher Push(event Event) error { if d.Finished { return errors.New(`queue is closed`) } d.Queue <- event return nil }</code></pre>Listening to Queue#</a></h3> So the client is calling Push()</code> on our EventDispatcher</code> and events are being pushed in the channel. But there’s no one reading from this channel so far. Let’s fix that by spawning workers, who will listen on the channel indefinitely and process the events: for { select { case event <- d.Queue: event.Process() } }</code></pre> In the above snippet, we are simply looping indefinitely to scan through all items in the queue. event <- d.Queue</code> is basically fetching the item from the channel and assigning a value to it. event.Process()</code> is a dummy function but it basically indicates that whatever processing that needs to be done should be handled here. Right now, you’ll be wondering two things: If this is an infinite loop, how do we guarantee it runs forever?</li> How do I spawn more workers if I need concurrency?</li> </ul> To address these problems, let’s add in WaitGroups and GoRoutines to our mix. WaitGroups will help us keep a count of workers which have been spawned and until each one of them finishes processing, wait groups will keep blocking indefinitely using wg.Wait()</code>. And to bring in more workers, we’ll simply spawn them with GoRoutines: go func() { for { select { case event <- d.Queue: event.Process() } } }()</code></pre> Now, spawning n</code> Goroutines is just a matter of a simple for loop over this: for i:=0; i<d.Opts.MaxWorkers; i++{ wg.Add(1) // Add a wait group for each worker go func() { for { select { case event <- d.Queue: event.Process() } } }() }</code></pre> Perfect! But hang on! We have missed a critical thing. How do we handle cancellations? For eg, when your program shuts down, we should clean up all the Goroutines spawned and process the remaining messages in queue. For that, we need to listen to a Cancellation</code> channel. The only purpose of this channel is to listen for SIGINT or SIGTERM signals and whenever either of them is received, we should flush our events. Here’s how the client would initialise a context: // Create a channel to relay `SIGINT` and `SIGTERM` signals. closeChan := make(chan os.Signal, 1) signal.Notify(closeChan, os.Interrupt, syscall.SIGTERM) ctx, cancel := context.WithCancel(context.Background())</code></pre> And in the main thread, the client would block on closeChan</code> channel like: // Listen on close channel indefinitely until a // `SIGINT` or `SIGTERM` is received. <-closeChan // Cancel the context to gracefully shutdown. cancel()</code></pre> When cancel()</code> is called, it does something special. It passes a value to ctx.Done()</code> channel. We can listen to this channel in the .Run()</code> function and flush pending events accordingly: case <- ctx.Done(): // Ensure no new messages are added. d.Finished = true // Flush all events. e.Flush() // This Goroutine has finished processing. wg.Done()</code></pre> Stitching all pieces together, we finally have: package dispatch type Dispatcher interface { // Push takes an Event and pushes to a queue. Push(Event) error // Run spawns the workers and waits indefinitely for // the events to be processed. Run() } // EventDispatcher represents the datastructure for an // EventDispatcher instance. This struct satisfies the // Dispatcher interface. type EventDispatcher struct { Opts Options Queue chan models.Notification Finished bool } // Options represent options for EventDispatcher. type Options struct { MaxWorkers int // Number of workers to spawn. MaxQueueSize int // Maximum length for the queue to hold events. } // NewEventDispatcher initialises a new event dispatcher. func NewEventDispatcher(opts Options) (Dispatcher) { return EventDispatcher{ Opts: opts, Queue: make(chan Event, opts.MaxQueueSize), Finished: false, } } // Push adds a new event payload to the queue. func (d *EventDispatcher Push(event Event) error { if d.Finished { return errors.New(`queue is closed`) } d.Queue <- event return nil } // Run spawns workers and listens to the queue // It's a blocking function and waits for a cancellation // invocation from the Client. func (d *EventDispatcher Run(ctx context.Context) { wg := sync.WaitGroup{} for i := 0; i < d.Opts.MaxWorkers; i++ { wg.Add(1) // Add a wait group for each worker // Spawn a worker go func() { for { select { case <-ctx.Done(): // Ensure no new messages are added. d.Finished = true // Flush all events e.Flush() wg.Done() return case e <- d.Queue: e.Process() } } }() } wg.Wait() } // Push adds a new event payload to the queue. func (d *EventDispatcher Push(event Event) error { if d.Finished { return errors.New(`queue is closed`) } d.Queue <- event return nil }</code></pre> This post doesn’t cover how to flush or process the events as these are implementation specific details. This is a pretty barebones structure and you can modify the code according to your usecase. Fin! Analysis Paralysis - CrashLoopBackoff 2020-11-27T08:10:55+05:30 Have you ever been stuck in a situation where you’ve spent countless hours digging through lot of options, overthinking their pros/cons and having a tough time to decide? If yes, I’ve been in the same boat myself and the more I think about this, I think it’s causing some serious issues in my productivity. Since I’ve started to self host a lot of services, I’m overthinking and overanalysing the software choices available to do a certain task. For eg, a personal wiki, sounds such a simple and common use case but I’ve personally tried countless choices. I’ve read and searched for these discussions on lobster/HN/Reddit discussions boards and that has been a real time sink for me, now that I look back. I’ll try an option for maybe a week or two at best and then jump on to something else in search of the perfect solution. It’s hard for me to stick to a solution. And it’s even more embarassing when you tell your friends about the “latest new distro” you’re using, announce it to the world on Twitter and when someone asks, “hey how’s that OS working for you” after a month, you tell them “Oh, I switched from it last week”. I’m not making any of this up, I’ve been in these shoes myself and honestly it’s not a good feeling. In the search of perfect I tend to lose sight of things that are important. Things which are slow, boring but they work are more reliable than a software which changes fast and breaks often. I want to fail cheap and quickly rather than spending a lot of time and still eventually failing. No, this post doesn’t have any answers on how to fix this problem. Well, if I myself knew then I’d not be writing this but giving a TEDx talk on it. These are just my observations and something that I intend to actively fix. Boring is better. I want to stick to stable choices when it comes to trivial things in life. I don’t wanna seek excitement or joy from switching to a cool new distro. Instead, by actually doing some important project at my work or building my side projects, something that gives me a genuine feeling of happiness. After all grass always seems to look greener on the other side. I don’t usually pen down my thoughts like these, but I want to refer to this public post as a reminder for myself the next time I’m about to make the same mistake. Fin. UPDATE (2021-11-02): There’s a really nice video</a> I came across today on the same subject. I’d highly recommend you to give this a listen if you found the above post relatable. Migrating my blog to Zola 2020-11-08T08:10:55+05:30 I’ve been writing on this blog for about 2 years now. This has been the longest I’ve stuck on to the same technology stack for my blog. I’ve previously jumped from a Jekyll based static site to a Medium blog before finally settling for Hugo</a>. I’ve been using Hugo since 2018 but I don’t recall as to why I went ahead with it. Maybe it was increasingly popular at that time and everyone touted Hugo as the solution to Static Site Generator (referred to as SSG from here on). There are 1000s of SSGs and at least a dozen of websites which lists all the SSGs out there. This is crazy by any standards. Hugo started as a generic blog generator but over the years it has become a website generator. It’s no longer aimed at people who just want to have a small little static website/blog but supports all the use cases for people building full-fledged static websites. IMHO these two goals are overarching however this has resulted in a simple project to become incredibly complex over time. Tipping Point#</a></h3> Anyway, so I wanted to change the look of the homepage on my website so I decided to look at Hugo’s documentation. Hugo’s documentation is great for someone who knows what exactly are they looking for. The documentation is so huge that you simply cannot grok it in one evening. I had zero ideas on how to customise the damn homepage of my blog and after spending hours buried in the documentation I was able to kind of figure the solution but it was unintuitive, to say the least. Apparently, to override any template from the theme, you have to mirror the directory structure of the theme in your root directory. Which meant, I needed to look at the source code of the theme, figure out the project structure, copy-paste all the folder names and put my override of index.html</code> there. Which, BTW magically overrides it. This whole magic thing is BS and I am being strongly opinionated here. There is more than one way to do something in Hugo. Different theme authors use different styles, which makes the whole thing even more complex. It also means for my customisations to work across themes, well you guessed it right: it’s impossible. Recently I discovered that I was unable to preview my Hugo website locally without internet because I had a Twitter shortcode</a> in one of my blog post (which makes an API call to Twitter to render a nice card preview). The site completely failed to render instead of just logging a warning. Bollocks. The tipping point for me, however, was when the theme I was using stopped working with the latest version of Hugo at that point. So, picture this – You make dozens of custom changes and then one update just breaks your website. Now not only you have to fix your shit but the theme you were using, you’ve to make upstream changes to the theme or maintain your own fork. And no, this is not a one-off experience. Hugo upgrades are a joke, they are known to break very very often. I was done at this point. I didn’t want to deal with this BS of continuously fighting the generator for my blog. A fresh change#</a></h3> Being a practitioner of Yak Shaving</a>, I discussed the idea of a “tinyhugo” with Kailash</a> and Sarat</a>. We’d arrived at a spec and I started writing some code to pander to my NIH syndrome. However, I was still not convinced that a simpler solution doesn’t exist. I spent countless hours exploring other alternatives. I’d used Lektor</a>, Pelican</a>, Eleventy</a> before finally stumbling upon Zola</a> from HN/Lobster discussions. I’ve got to say, the landing page gave a fresh feeling - one that I’ve not seen with any other alternatives. In fact quite opposite to the Eleventy landing page which looks like an over-engineered piece of software to generate websites (Not hating on it, there might be use cases for it, but the JS tooling and dependency system is something that I would not want to touch with a 10ft pole). Zola’s primary appeal to me was that like Hugo it’s extremely fast and comes as a single binary no dependency package. I looked at the docs the first impression was they are concise enough to get a basic idea. Zola is strongly opinionated, even to the extent of dictating a project structure and sometimes filenames too. I actually preferred this over the magic Hugo does. In less than 2 hours I was able to port the home page of my blog (and tweak it to my liking) in Zola. I decided to abandon my own tinyhugo</code> attempt because for the very fact Zola fits my needs very well. The thing that I really loved about Zola is how it enforces a separation between Section</a> and Pages</a>. The section represents a “collection” of posts. So a blog can be a section, and I can have another section called “Book Reviews”. I could easily tell Zola where to look for the templates by specifying the same in content/book_reviews/_index.md</code>. I don’t have to read Hugo docs or do Google-fu to figure this out, it’s right there in the docs and very apparent. For the record, I still don’t know how to customise different templates for different sections in Hugo, but I couldn’t care less. Migration#</a></h3> The migration was pretty straightforward – I had to copy the content folder</code>s of my blog (which are just a bunch of .md</code> files) and replace YAML</code> frontmatter to TOML</code>. There were a few variable changes that I needed to do manually but since they were a manageable 20-25 posts, I did it by hand. I could potentially automate but then rabbit deep in the rabbit hole of Yak Shaving. The good part was that I was able to retain the same URL structure for my new blog because the URL scheme was based on the file paths. I spent some time porting hugo-ink</a> to Zola and did minor CSS tweaks to it. Zola uses the Tera language for templating and it’s much more pleasing to eyes than the Go Template syntax. Zola comes with pretty neat features like Search, RSS/Atom Feeds, Syntax Highlighting and SASS->CSS Processors. What took me time however was to figure out how to get opengraph</code> tags in each page. Hugo provides nifty template</a> for this use case but Zola is pretty barebones like that. People who care a lot about SEO need to spend some extra efforts here. Future#</a></h3> Zola is still a pretty new kid on the block but the author shares the same frustration about Hugo: it personally drives me insane, to the point of writing my own template engine and static site generator. Yes, this is a bit biased. – Source</a> </blockquote> This also reflects in the issues/PRs I’ve seen for Zola and the author is opinionated about not adding features which would make Zola complicated. Overall I am very happy with the switch and it was long due. I feel more confident in tweaking certain sections of my website. I plan to open-source the current theme in the next few days. You can read the Source Code</a> of this website if you’d like to explore how this website is built. Fin! Import existing Route53 records in Terraform 2020-10-18T08:10:55+05:30 Terraform has a straightforward way of importing existing records (managed outside Terraform) via terraform import</code> command. The usage is documented here</a> and works well if you have a handful of records to import. However when you work with custom Terraform modules and have a whole bunch of records to be imported, you’d look out ways to script the entire workflow. I did this a few weeks back at work and thought to share a solution which works well for my usecase. How it works#</a></h3> The task consists of 3 parts: 1. Import all existing records in a hosted zone using AWS CLI</code>.#</a></h4> aws route53 list-resource-record-sets --hosted-zone-id XXX > data/company-tld.json</code></pre># Loads the zone records in a dict def load_records(zone_file=ZONE_FILE): with open(zone_file) as record_file: data = json.load(record_file) return data</code></pre>2. Import the record in Terraform state.#</a></h4> To do this, Terraform CLI comes with an import</code> command. However for import</code> to work, you need to have a resource declaration in your Terraform file already. From the official documentation</a>: Because of this, prior to running terraform import it is necessary to write manually a resource configuration block for the resource, to which the imported object will be mapped. </blockquote> To overcome this restriction, we will create a dummy.tf</code> and programatically write the configuration block for each record. # Writes the dummy Terraform template which is required # before `terraform import` runs. def template_dummy_file(resource_name): add_dummy_record = Template( """ resource "aws_route53_record" "$resource_name" { # (resource arguments) } """ ) dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf") with open(dummy_file_path, "a") as f: f.write(add_dummy_record.substitute(resource_name=resource_name))</code></pre> AWS Route53 module can import aws_route53_record</code> as decsribed here</a>. We will run this command as a subprocess. # Shells out `terraform import` command in the host OS. def terraform_import(resource_name, resource_type): import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}" run(import_command, shell=True, check=True)</code></pre>3. (Optional) Move Resources in a Module#</a></h4> In case you are using a Module to manage AWS Route53 resources, you’ll need to move the declaration from resource</code> to module</code> configuration block. This is described more in detail here</a>. The module declaration/naming would depend on how the module is configured. To demonstrate, the module I use internally requires the name to be of the format resource_name-resource_type</code>. To achieve this, you can call terraform state mv</code> as a subprocess: # Shells out `terraform state mv` command in the host OS. def terraform_move(resource_name, resource_type): mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'" run(mv_command, shell=True, check=True)</code></pre> That’s it! Running terraform plan</code> should now show you the changes and if you imported every record correctly you should not see any drift from the real world state. You can view the entire script here: import json from os import getenv, path from string import Template from subprocess import run from sys import exit ZONE_ID = getenv("ZONE_ID") MODULE_NAME = getenv("MODULE_NAME") ZONE_FILE = getenv("ZONE_FILE") TERRAFORM_DIR = getenv("TERRAFORM_DIR") # Returns the variable key if not present in ENV. def check_env_vars(): if not ZONE_FILE: return "$ZONE_FILE" if not ZONE_ID: return "$ZONE_ID" if not MODULE_NAME: return "$MODULE_NAME" if not TERRAFORM_DIR: return "$TERRAFORM_DIR" return "" # Loads the zone records in a dict def load_records(zone_file=ZONE_FILE): with open(zone_file) as record_file: data = json.load(record_file) return data # Writes the dummy Terraform template which is required # before `terraform import` runs. def template_dummy_file(resource_name): add_dummy_record = Template( """ resource "aws_route53_record" "$resource_name" { # (resource arguments) } """ ) dummy_file_path = path.join(TERRAFORM_DIR, "dummy.tf") with open(dummy_file_path, "a") as f: f.write(add_dummy_record.substitute(resource_name=resource_name)) # Shells out `terraform import` command in the host OS. def terraform_import(resource_name, resource_type): import_command = f"terraform import aws_route53_record.{resource_name} {ZONE_ID}_{resource_name}_{resource_type}" run(import_command, shell=True, check=True) # Shells out `terraform state mv` command in the host OS. def terraform_move(resource_name, resource_type): mv_command = f"terraform state mv aws_route53_record.{resource_name} 'module.{MODULE_NAME}.aws_route53_record.route53_record[\"{resource_name}-{resource_type}\"]'" run(mv_command, shell=True, check=True) if __name__ == "__main__": missing = check_env_vars() if missing: exit(f"Required env variable {missing} is missing.") records = load_records() for i in records.get("ResourceRecordSets"): resource_name = i.get("Name") resource_type = i.get("Type") template_dummy_file(resource_name) terraform_import(resource_name, resource_type) terraform_move(resource_name, resource_type) print(f"Imported {resource_name}")</code></pre> Hope this tiny Python script helps you transition your AWS Route53 records neatly and effortlessly! Fin! Setting up a RIPE Atlas Probe 2020-10-03T08:10:55+05:30 Twitter is an amazing thing! For all the shitposting and meme-ing that’s done there, there are some really cool people you get to interact with which wouldn’t have been possible IRL. I happened to stumble upon Swapneel</a> last year in Bangalore at a meetup. He occasionally posts about RIPE Atlas</a> Probes measurements on his Twitter feed and that made me curious and learn more about the Atlas Probe. Call it the Baader-Meinhof effect but just when I was reading his blog post</a> on RIPE Atlas, I found about a Hasgeek workshop</a> which is being conducted by Swapneel himself and well, that’s how I basically got interested in setting up a probe myself. What is RIPE Atlas Probe#</a></h2> A Probe</a> is a device used to measure various metrics like DNS, SSL/TLS, NTP, ping, traceroute etc for an upstream target. A network of such probes is useful not just to internet researchers, network engineers, ISP operators, public activists but also to common folks like me who are simply curious about the network/WWW in general. The idea behind this is simple – you get some virtual credits for hosting a probe. You can make use of these credits to run your own measurements to gain insights about the health of your network. The data from these measurements are made publicly available by RIPE NCC and anyone can see these results. There are 2 ways to host a Probe: Software and Hardware. Since the start of this year, RIPE Atlas is available as software packages for various platforms. Before this, the only way to set up a Probe was to host a Hardware probe by applying for one at RIPE NCC website. These hardware probes are not shipping to India since quite some time as Swapneel mentioned and if you’re in India, your only choice, for now, is to set up a software probe. A few key differences in the 2 kinds of probes: Initial Setup: Hardware probe is a plug-n-play device, no setup required. That makes it attractive for a lot of people, to just get a small device and host it.</li> Spare Compute: For hosting a probe it is recommended that the probe stays online for as long as possible. This means you need to have a spare compute lying around, like a RaspebrryPi, a VM, a server etc. It’s not really recommended to host a probe on your laptop/desktop - something which has ad-hoc usage pattern. For people who don’t have any spare compute, a hardware probe is more suited for them.</li> Tampering Result: With the software probe, there are easy ways of tampering the measurement result, which might give a bad data point to people running the measurements. For eg, a software probe could tamper with the DNS queries for a particular upstream and fool you to believe that there’s a problem with the uplink or the upstream target.</li> </ul> You can read more about Probes here</a> and this FAQ</a> section is highly recommended to clear all the basic doubts you might have about Probes. Setting up a Software Probe in a Container#</a></h2> Well, Containers are awesome! It’s easier to do a docker run</code> than figure out the installation instructions for your OS, ensure dependencies are installed and insert 10 other things you have to do. So, I decided to use jamesits/ripe-atlas</a> Docker image to host a probe on my RPi 4. However, in case you don’t want to use Containers for a reason best known to you, you can visit the official docs</a> and find out instructions for your platform. You can follow the below steps to set up a probe via Docker: 1. Create a RIPE NCC Account#</a></h4> You need to register for a RIPE NCC Access Account before proceeding further. Visit the registration</a> page and create an account. 2. Start the container#</a></h4> docker run --detach --restart=always --log-opt max-size=10m \ --cap-add=SYS_ADMIN --cap-add=NET_RAW --cap-add=CHOWN \ --mount type=tmpfs,destination=/var/atlasdata,tmpfs-size=64M \ -v /var/atlas-probe/etc:/var/atlas-probe/etc \ -v /var/atlas-probe/status:/var/atlas-probe/status \ -e RXTXRPT=yes \ --name ripe-atlas --hostname "$(hostname --fqdn)" \ jamesits/ripe-atlas:latest</code></pre> NOTE: In the above case, since I am running the Probe on my RPi which is why I am using latest-armv7l</code> tag for the jamesits/ripe-atlas</code> image. In case you are doing this on an amd64</code> machine, you should use the latest</code> tag only. You can find more options for the container here</a>. 3. Apply for a Software Probe#</a></h4> You need to register for a software probe here</a> and fill in the details as mentioned below. ASN: You can find the ASN of your public IP by visiting ip2asn.com</a>. You need to enter your public IP here which you can find it by visiting ifconfig.co</a>.</li> </ul> Public Key: You need to fill the public key generated by your Probe:</li> </ul> cat /var/atlas-probe/etc/probe_key.pub</code></pre> Note: We are mounting the /var/atlas-probe/etc</code> on the host path inside the container at /var/atlas-probe/etc</code>. This ensures that if you stop/remove the container, your public/private key pair isn’t lost and there’s no need to regenerate/apply for a Probe again. However in case you lose this, the Probe will attempt to generate a new pair and you might need to change the public key at that point of time. So be careful to take a backup of the generated key and store somewhere safe. After submitting the form, you will receive an email which will have information about your Probe ID. 4. Wait patiently#</a></h4> After filling the form you need to wait roughly for 15 minutes for your probe to be added to the global network of Atlas Probes. You will receive an email saying your Probe is live now. Creating Measurements#</a></h2> It’s pretty simple to create measurements, both via UI and a Python library</a> developed by the RIPE community. However, for the ease and convenience I will demonstrate how to create a traceroute measurement from the UI itself: Note: You get around 15 Credits/minute for hosting a probe. In case you just deployed a probe you might not have enough credits to perform this measurement. You can come back later to do this when you have sufficient credits. Visit the dashboard</a> and headover to Measurement</a> tab. On clicking the Create a Measurement</code> button you will be presented a form with various options. Let’s try a Traceroute to mrkaran.dev</code> for this measurement: To limit the number of probes, I am going to select all Probes in India, by clicking on +New Set -wizard</code> option in the Probe Selection tab. After submitting the form, you will be presented with a Measurement ID which you can use to download the data at a later point as well. Visit the Measurement ID page and wait for a couple of minutes for all probes to perform the measurements: That’s it! You can explore other types of Measurements and different options while performing them to customise your use case. Summary#</a></h2> Atlas Probes are pretty easy to host and they are crucial in giving valuable insights about how different clients connect to upstreams and spot issues in Networks. If you can host a probe, you should seriously consider doing that and help increase the number of probes in the network. As of today, the number of probes in India is just around 113 and that is seriously very less. If you need any help hosting one, don’t hesitate to contact me on my Twitter. Oh, and for the record, I am hosting 2 Probes currently. Probe 1000991</a> hosted on my local RPi connected to my home network (ASN24560) and Probe 1001117</a> hosted on a DigitalOcean droplet in Bengaluru (ASN14061). Fin! Monitoring my home network 2020-09-18T08:10:55+05:30 I like monitoring stuff. That’s what I do at work and when my home ISP started giving me random problems and I decided it would be nice to monitor my home network as well. There are a couple of ways to go around this, a very popular and OSS solution is SmokePing</a>. SmokePing is written in Perl and is used to visualise network latencies. It’s quite a great solution but for my current stack which involves Prometheus and Grafana, it meant I had to deploy a standalone tool separate from my monitoring stack - something which I wanted to avoid. So, I looked for other solutions and luckily happened to stumble upon oddtazz</a> in one of the common Telegram groups where he shared his solution for the above: Telegraf ICMP plugin</a> and Grafana. This is exactly what I’ve been looking for but for some reason, I had wrongly assumed Telegraf needs InfluxDB to store the data. Googling a bit more, I found Telegraf supports</a> Prometheus format (amongst a huge list of others!) but this wasn’t so clear in their docs. I decided to run a Telegraf agent in my RPi connected to my home router over LAN and scrape metrics using Prometheus and visualise graphs in Grafana! For the non-patient readers, here’s what my dashboard looks like!: Setup#</a></h2> To get started, we need to download Telegraf</a> and configure the Ping</a> plugin. Telegraf has the concept of Plugins for Input, Output, Aggregating and Processing. What this basically means is that you can configure multiple input plugins like DNS, ICMP, HTTP and export the data of these plugins in a format of your choice with Output plugins. This makes Telegraf extermely extensible, you could write a plugin (in Go) of your choice if you fancy that as well! Here’s what my telegraf.conf</code> looks like: # Input plugins # Ping plugin [[inputs.ping]] urls = ["mrkaran.dev", "tailscale.mrkaran.dev", "floyd.mrkaran.dev", "1.1.1.1", "kite.zerodha.com", "google.com", "reddit.com", "twitter.com", "amazon.in", "zerodha.com"] count = 4 ping_interval = 1.0 timeout = 2.0 # DNS plugin [[inputs.dns_query]] servers = ["100.101.134.59"] domains = ["mrkaran.dev", "tailscale.mrkaran.dev", "floyd.mrkaran.dev", "1.1.1.1", "kite.zerodha.com", "google.com", "reddit.com", "twitter.com", "amazon.in", "zerodha.com"] # Output format plugins [[outputs.prometheus_client]] listen = ":9283" metric_version = 2</code></pre> Firstly, so nice to see an Ops tool not using YAML</code>. Kudos to Telegraf for that. I’d love to see other tools follow suit. Getting back to the configuration part, input.plugin</code> is a list of plugins that can be configured and I have configured the Ping and DNS plugin in my config. The output</code> is in Prometheus format so it can be scraped and ingested by Prometheus’ time-series DB. Running Telegraf#</a></h3> With the above config in place, let’s try running the agent and see what metrics we get. I am using official</a> Docker image to run the agent with the following config: docker run --name telegraf-agent --restart always -d -p 9283:9283 -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro telegraf</code></pre> After running the above command, you should be able to see the metrics at localhost:9283/metrics</code> $ curl localhost:9283/metrics | head % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP dns_query_query_time_ms Telegraf collected metric # TYPE dns_query_query_time_ms untyped dns_query_query_time_ms{dc="floyd",domain="amazon.in",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 124.096472 dns_query_query_time_ms{dc="floyd",domain="google.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 136.793673 dns_query_query_time_ms{dc="floyd",domain="kite.zerodha.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 122.780946 dns_query_query_time_ms{dc="floyd",domain="mrkaran.dev",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 137.915851 dns_query_query_time_ms{dc="floyd",domain="twitter.com",host="work",rack="work",rcode="NOERROR",record_type="NS",result="success",server="100.101.134.59"} 111.097483</code></pre> Perfect! Now, we’re all set to configure Prometheus to scrape the metrics from this target. In order to do that you need to add a new Job</a>: - job_name: "ispmonitor" scrape_interval: 60s static_configs: - targets: ["100.94.241.54:9283"] # RPi telegraf Agent</code></pre> In the above config, I am plugging my Tailscale IP assigned to my RPi on the port where Telegraf agent is bound to. This is one of the many reasons why Tailscale is so bloody awesome! I can connect different components in my network to each other without setting up any particular firewall rules, exposing ports on a case by case basis. Sidenote: If you haven’t read Tailscale’s amazing NAT Traversal blog post</a>, do yourself a favour and check it out after you finish reading this one ofcourse! Anyway, coming back to our Prometheus setup, we can see the metrics being ingested: Show me the graphs#</a></h2> Now comes the exciting bit – making pretty graphs. First, let’s discuss what’s the most important data I can extract out of Ping</code> and DNS</code> plugins. These plugins export decent amount of data, but a good rule of thumb while making dashboards is to optimise signal v/s noise ratio. We’ll do that by filtering out only the metrics that we care for. Let’s checkout all the metrics exported by Ping</code> plugin: $ curl localhost:9283/metrics | grep ping | grep TYPE # TYPE ping_average_response_ms untyped # TYPE ping_maximum_response_ms untyped # TYPE ping_minimum_response_ms untyped # TYPE ping_packets_received untyped # TYPE ping_packets_transmitted untyped # TYPE ping_percent_packet_loss untyped # TYPE ping_result_code untyped # TYPE ping_standard_deviation_ms untyped # TYPE ping_ttl untyped</code></pre> Perfect! So, from the above list of metrics, the most important ones for us are: ping_average_response_ms</code>: Avg RTT for a packet</li> ping_maximum_response_ms</code>: Max RTT for a packet</li> ping_percent_packet_loss</code>: % of packets lost on the way</li> </ul> With just the above 3 metrics, we can answer questions like: Is my ISP suffering an outage?</li> </ul> If yes, ping_percent_packet_loss</code> should be unusually higher than normal. This usually happens when the ISP has routing is borked and that causes the packet to be routed in a less optimized way and as a side effect packet loss becomes one of the key metrics to measure here. Is the upstream down?</li> </ul> If yes, ping_average_response_ms</code> over a recent window should be higher than a window compared to a previous time range when things were fine and dandy. This can either mean 2 things: Either your ISP isn’t routing correctly to the said upstream or the CDN/Region where your upstream is faced an outage. This is quite a handy metric for me to monitor! How many times have your friends complained “xyz.com</code> isn’t working for me” and when you try to load, it’s fine from your end? There are a lot of actors at play but ping</code> is usually the most simple and quickest way to detect whether an issue persists or not. Of course, this doesn’t work for hosts which block ICMP packets altogether. They are not rare either, like netflix.com</code> and github.com</code> both block ICMP probes for example. For my use case, this wasn’t a major issue as I was able to still probe a decent amount of upstreams all over the world. With that out of the way, let’s break the dashboard into different components and see what goes behind them. Ping Response Panel#</a></h3> To plot this, simply choose a Stat</code> visualisation with the query ping_average_response_ms{url="$url"}</code>. Repeat this panel for the variable $url</code> and you should be able to generate a nice row view like this. Additonally you can choose Thresholds and the Unit to be displayed in the panel with these options. Ping Response Time Graph#</a></h3> The next graph is interesting, it lets me visualise the avg, min, max ping response time as well as the % packet loss plotted on the Y2 (right Y) axis. Availability Panel#</a></h3> An interesting query to calculate uptime (just in the context whether the upstream is reachable) is: 100 - avg_over_time(ping_percent_packet_loss[2m])</code></pre> Since I scrape metrics at an interval of 1m</code>(in order to not ping too frequently and disrupt my actual browsing experience), in this query I am averaging the data points for the metric ping_percent_packet_loss</code> in a [2m]</code> window. DNS Response Time Graph#</a></h3> We can similarly query the DNS response time by visualising the average response time for a DNS query. This might be useful only to people self-hosting their DNS servers. Conclusion#</a></h2> So with a pretty simple and minimal OSS solution, I was able to setup monitoring for my home network! Over the last few days whenever my ISP had slightest of trouble, I can correlate it with my metrics! I mean I still can’t do anything about it cause the other person on ISP’s customer support is “Did you try rebooting your router” – the quintessential solution to all tech problems. Wish we could reboot this entire damn 2020 as well, but one could hope! Shoot me for any questions on my Twitter @mrkaran_</a> :) Fin! Using Tailscale for home server 2020-04-23T08:10:55+05:30 For those of you who are new to my blog, I had written about my home server setup</a> (hydra</code>) a few months back. Since then I’ve tweaked the setup a bit and made some changes to how I organise/deploy applications inside my cluster. I’ll be talking about the updates made so far and the reason behind them. A brief overview of what has changed from hydra</code> v0.1 to hydra</code> v0.2: Replaced Wireguard with Tailscale </li> Added a new worker node (residing in DigitalOcean, Bengaluru) to the existing RPi k3s cluster. </li> Shifted from PiHole to Adguard DNS upstreaming to Unbound. </li> Containerised all workloads and deployed on K8s (using k3s</code> platform). </li> </ul> The setup looks something like this now: Let’s take a quick look at some of the above things in detail! Using Tailscale#</a></h2> So, before we jump to why I started using Tailscale, let’s address a few things about Wireguard which bothered me. Adding new devices is a real PITA and quite often laziness kicks in to generate a QR code or Private/Public key pair. So, for those of you unaware, Wireguard needs a client config to add the server endpoint and it’s public key for the client to communicate with the server. You need to add the client’s private key on the server-side as well for the exchange of encrypted packets to happen. Doing all of this manually has been one of the reasons I’ve not added/updated my devices as regularly as I’d like to do. For eg, on my recently bought iPad, I just haven’t bothered to do all of this, cause ugh I am lazy? </li> Having a central VPN server to talk to my RPi from my local network just doesn’t seem right you know? Especially when both devices are literally lying in the same room, sending the packets to a server somewhere in Electronic City (DO, blr region) from JP Nagar (where I live) feels like totally unnecessary. I really needed a mesh network to reduce wasted bandwidth and latencies. Having a central VPN server is also a SPOF, not that my home-server runs any mission-critical workloads, but still good to avoid where we can. Where else can I flex my ops-chops skills if not my home server, eh? </li> </ul> I started looking at different mesh VPN setups and Tailscale attracted me the most. I heard about Tailscale first time when I saw bradfitz’s</a> post about leaving Google and joining Tailscale on HN. Interestingly, Tailscale is built on top of wireguard-go</a> (the userspace implementation of Wireguard</a> kernel module). Since I was already familiar with Wireguard and had been using it for almost a year, I got curious on Tailscale. Tailscale basically sets up a point-to-point encrypted network across all your devices. What that means is there’s no “relay” (like Wireguard server) and clients can talk to each other directly. While it may seem that it is easy to set this up with a bunch of shell scripting-foo (just add all Wireguard peers in each peer config), Tailscale attempts to do a lot of the heavy lifting by making the network seamless and handling authentication. Coordination Server#</a></h3> So, when we add a new device to Wireguard server, we basically need to generate a private/public key pair as explained above. When you’re setting up Tailscale agent, it does all of this in the background and asks you to authorise to a central co-ordination server. Now, this server is only used to exchange information about each peer in your network. The tailscale agent uploads the private/public key information of the peer you are currently on and any time any new peer joins the network, all of the agents’ configurations are updated real-time. The coordination server periodically checks for any new changes and pushes the updates to the agent on each node. The real juice is that they authenticate via OAuth, OIDC or SAML. Which means that you can use your existing 2FA configurations to authenticate a new node in the network. However, this might be a point of concern for some users, but I chose convenience here. Also, since the traffic flows through the nodes directly, and is encrypted there’s not much to worry here. I’ve been following Tailscale closely and they do plan to opensource the co-ordination server in future, so maybe when that happens I’ll self-host it. NAT Traversal#</a></h3> Apart from handling the auth, Tailscale does a pretty good job at handling connectivity across different network topologies. So, basically, if you want to peer 2 devices over the public internet you’d need to perform NAT Traversal. There are several NAT hole punching techniques which allow traversing NAT but since they are not standardised and sometimes due to erratic NAT behaviour, it poses quite a challenge to do it seamlessly. And, I’m not even talking about roaming networks as of yet. Tailscale agents can perform NAT traversal using ICE</a> and STUN</a>. What all of this practically means is if you’re sitting in a cafe somewhere and you want to access any of your internal service it is possible without messing around any firewall ports :D TL;DR: I decided to give Tailscale a shot and was quite impressed by how easy it was to setup. You can refer to the official docs for more instructions but I was able to set it up across 2 RPis and have them talk to each other under 15 minutes. I think that’s quite an awesome feat in itself. The only sad bit is they don’t have an Android app yet, so I am eagerly waiting for it. Hybrid K3s Cluster#</a></h2> So, I am running a k3s</code> cluster on my RPi. At that time I was still using DO for running Pihole, Unbound, public DNSCrypt resolver etc. I decided to standardise the ad-hoc deployments to manage them efficiently. It also allowed me to play around with more on K8s which was my original goal behind buying these Pi was. Now, I’ve 2*RPi nodes, the k8s master node runs on the 4GB RPi while the 2GB variant serves as a worker node. I decided to get a bit fancy with my setup and hooked up the k3s</code> agent installation script on a DO node andta-da! I have a multi-arch(amd64</code> and arm64</code>), hybrid (bare metal and cloud) K8s cluster ready! I think if I sprinkle some ML/AI + Bitcoins in the above setup, I’m all set to raise VC funding for hydra</code>. I wanted to learn Terraform</a> as part of my work at my org as well, so I created + managed the entire droplet through terraform</code>. The script</a> has modules to provision a droplet, attach an elastic IP, configure firewall rules and add my SSH key to the server. Quite easy to manage and I am generally a hater of GUIs, so Terraform is indeed a blessing in disguise. I know some of my opinions are a bit strong but don’t worry I get meme’d/burnt for this almost every day by my work colleagues. </blockquote> Pihole to Adguard#</a></h2> While Pihole works really well for blocking Ads it had some features lacking, particularly DOT and DOH support out of the box. I decided I’ll shift to Adguard as the codebase is in Go - something which I am a bit familiar with and also the UI feels a bit sleek and refreshing too! Accessing internal services#</a></h3> A major challenge for me was however to configure access to internal services on the K8s cluster. Since I have bare metal worker nodes, it’s not possible to deploy a cloud load balancer. For now, I went with a really simple old school solution, to expose an Nginx proxy front-ending all my services through NodePort. I am planning to look at Traefik or Istio for this but I wanted to just shipit! at this point. Here’s a very basic example of an nginx</code> config for ip.mrkaran.dev</code> that I run on my cluster: server { server_name ip.mrkaran.dev; # reverse proxy location / { proxy_pass http://100.96.239.6:30506; # tailscale IP, connecting to NodePort service include fluff/proxy.conf; } }</code></pre> 30506</code> port is exposed by the Service</code> object backing up the pods for that application. Since a NodePort</code> service is available on any K8s node, you can give the Tailscale IP for any node and the routing will be handled by kube-proxy</code>. Challenges faced#</a></h2> Mesh or mess#</a></h3> Setting all of this up didn’t come without its own challenges or hurdles. Right off the bat, the first problem was that I was seeing really high latencies from my RPi node to DO node through Tailscale. Now, since both the nodes are physically in Bangalore and they are connecting to each other (or that’s what I had presumed), I didn’t expect latencies to be as high as 500-600ms. Bollocks! Eventually, I’d figured thanks to my super restrictive rules on the firewall of DO node, I had blocked all inbound UDP connection. That means NAT hole punching through STUN is simply not possible. In such cases, Tailscale forwards all packets to an encrypted TURN server(called DERP - Designated Encrypted Relay for Packets), which basically are TCP encrypted relays. Tailscale manages this network of DERPs and the one I got connected to was somewhere in the USA. Bottom line, I was all pumped up to talk to a DO node from my RPi node but as it turns out (no pun intended) my packets were flowing through the USA! Ah so bad. Anyway, opening up UDP from the Tailscale subnet, fixed the issue and latencies were back to being sub 10ms. Yay! Overlay over an overlay#</a></h3> Next up, were problems with K3s networking from the DO node to RPi node. The DO node was in a NotReady</code> state because the agent couldn’t reach the server: Apr 14 09:00:33 hydra-control k3s[19746]: I0414 09:00:33.306650 19746 log.go:172] http: TLS handshake error from 100.97.222.106:51516: read tcp 100.96.239.6:6443->100.97.222.106:51516: read: connection reset by peer</code></pre> Through some trial and error and reading the docs</a>, I figured that flannel</code> is running as a CNI in k3s</code>. Now the problem is flannel itself is an overlay network. But… Tailscale already is an overlay network (Wireguard) so the packets are not being routed correctly and being dropped halfway in the master node (I am guessing the DNAT/SNAT translation botched up here). The trick was to just change</a> flannel backend to run in the host namespace only. That solved the above issue for me. However, I still had one more issue. The DO node’s public IP was being advertised, while the agent was running on Tailscale network interface so the master was never able to reach the agent. Similarly, when the agent tried to communicate with the server, the private IP of the node was being advertised. Setting the --node-external-ip <tailscale-ip></code> in k3s</code> config seemed to have fixed the problem. Now all of the nodes in the cluster had proper Tailscale IPs advertised and the node went to Ready</code> state, at last! Who let the DNS out#</a></h3> So, I’ve a chicken and egg problem in my setup. Since my laptop</code> runs a Tailscale agent and whenever I boot up my systemm, Tailscale</code> attempts to posts logs to log.tailsclae.io</code> and fails to start if it cannot. The problem here is who resolves the DNS for me? I run a local DNS server with CoreDNS</a> forwarding my queries to Adguard. Now if I can’t reach Adguard (since Tailscale agent hasn’t initialised), how am I supposed to resolve log.tailscale.io</code>? I did what any sane guy would do, write a simple hacky bash script: #!/bin/bash sudo chattr -i /etc/resolv.conf sudo echo 'nameserver 1.1.1.1' > /etc/resolv.conf echo "changed dns server to 1.1.1.1" sudo tailscale up sudo echo 'nameserver 127.0.0.1' > /etc/resolv.conf echo "changed dns server back to 127.0.0.1" sudo chattr +i /etc/resolv.conf</code></pre> Yes, it’s quite insane. Also, I’ve not been able to figure out how to stop NetworkManager</code> from changing my /etc/resolv.conf</code> so I rely on a hack (documented in Arch official docs</a>), to lock</code> the file so any process cannot modify it. Quirky, but works! Storage#</a></h3> Unfortunately, I don’t have any external HDD/SSD so I am postponing running any stateful workloads till I get one soon (whenever lockdown gets over in my area). I plan to deploy an NFS server so I can run stateful workloads across any node and have redundancy in form of cloud backups. I’ve also heard cool things about Longhorn</a> but unfortunately, it doesn’t have ARM support</a>. Final Thoughts#</a></h2> Well, I am quite stoked by my current setup right now. Learnt a bunch of cool things around NAT traversal, zero-trust networking, realised the old days of LAN were so much better (not that I am that old to have experienced, my first internet connection was rather broadband at home, not even dial-up). Tailscale opens up a lot of new opportunities for corporate VPNs and it is something definitely to watch out for as they continue improving the product. Also, I was super elated when bradfitz</a> himself had commented on my tweet, not gonna lie! {{< tweet 1249552822674149376 >}} Credits#</a></h3> First, a major thanks to my friend sarat</a> with whom I pair debugged a lot of the above issues and since he also runs a home server, he was my go-to person to ask doubts! Here are some links that I collected while setting all of this up and might be useful as references if you’re planning a similar setup! How Tailscale Works</a></li> What are these 100.x.y.z addresses?</a></li> k3s config reference</a></li> P2P across NAT</a></li> Remembering the LAN</a></li> Cloudskew</a> (wonderful tool for creating architecture diagrams)</li> </ul> If you’ve any questions please find me on Twitter @mrkaran</a>. You can find all of the setup on GitHub</a>. Till next time, if we survive the pandemic! DNS Lookups in Kubernetes 2020-02-02T10:57:55+05:30 One of the primary advantages of deploying workloads in Kubernetes is seamless application discovery. Intra cluster communication becomes easy with the concept of Service</a> which represents a Virtual IP backing a set of Pod IPs. For example, if vanilla</code> service needs to talk to a chocolate</code> service, it can directly use the Virtual IP for chocolate</code>. Now the question is who resolves the DNS query for chocolate</code> and how? DNS resolution is configured in Kubernetes cluster through CoreDNS</a>. The kubelet configures each Pod’s /etc/resolv.conf</code> to use the coredns pod as the nameserver</code>. You can see the contents of /etc/resolv.conf</code> inside any pod, they’ll look something like: search hello.svc.cluster.local svc.cluster.local cluster.local nameserver 10.152.183.10 options ndots:5</code></pre> This config is used by DNS clients to forward the DNS queries to a DNS server. resolv.conf</code> is the resolver configuration file which has information about: nameserver : Where the DNS queries are forwarded to. In our case this is the address of CoreDNS</code> service. </li> search: Represents the search path for a particular domain. Interestingly google.com</code> or mrkaran.dev</code> is not FQDN (fully qualified domain name). A standard convention that most DNS resolvers follow is that if a domain ends with .</code> (representing the root zone), the domain is considered to be FQDN. Some resolvers try to act smart and append the .</code> themselves. So mrkaran.dev.</code> is an FQDN but mrkaran.dev</code> is not. </li> ndots: This is the most interesting value and is the highlight of this post. ndots</code> represents the threshold value of the number of dots in a query name to consider it as a “fully qualified” domain name. More on this later, as we discover the flow of DNS lookup. </li> </ul> Let’s check what happens when we query for mrkaran.dev</code> in a pod. $ nslookup mrkaran.dev Server: 10.152.183.10 Address: 10.152.183.10#53 Non-authoritative answer: Name: mrkaran.dev Address: 157.230.35.153 Name: mrkaran.dev Address: 2400:6180:0:d1::519:6001</code></pre> For this experiment, I’ve also turned on CoreDNS logging level to all</code> which makes it highly verbose. Let’s look at the logs of coredns</code> pod: [INFO] 10.1.28.1:35998 - 11131 "A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000263728s [INFO] 10.1.28.1:34040 - 36853 "A IN mrkaran.dev.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000214201s [INFO] 10.1.28.1:33468 - 29482 "A IN mrkaran.dev.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000156107s [INFO] 10.1.28.1:58471 - 45814 "A IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 56 0.110263459s [INFO] 10.1.28.1:54800 - 2463 "AAAA IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 68 0.145091744s</code></pre> Whew. So 2 things piqued my interest here: The query iterates through all search paths until the answer contains a NOERROR</code> code (which the DNS clients understand and store it as the result). NXDOMAIN</code> simply indicates no record found for that domain name. Since mrkaran.dev</code> isn’t an FQDN (according to ndots=5</code> setting), the resolver looks at search path and determines the order of query. </li> A</code> and AAAA</code> records are fired parallelly. This is because single-request</code> option in /etc/resolv.conf</code> has a default configuration to perform parallel IPv4 and IPv6 lookups. You can disable this using single-request</code> option. </li> </ul> Note: glibc</code> can be configured to send these requests sequentially but musl</code> cannot, so Alpine users must take note. Playing around with ndots#</a></h3> Let’s play around with ndots</code> a bit more and see how it behaves. The idea is simple, for the DNS client to know whether a domain is an absolute one or not is through this ndots</code> setting. For example, if you query for google</code> simply, how will the DNS client know if this is an absolute domain. If you set ndots</code> as 1</code>, the DNS client will say “oh, google</code> doesn’t have even one 1 dot, let me try going through search list. However, if you query for google.com</code>, the search list will be completely ignored since the query name satisfies the ndots</code> threshold (At least one dot). We can see this by actually doing it: $ cat /etc/resolv.conf options ndots:1</code></pre>$ nslookup mrkaran Server: 10.152.183.10 Address: 10.152.183.10#53 ** server can't find mrkaran: NXDOMAIN</code></pre> CoreDNS logs: [INFO] 10.1.28.1:52495 - 2606 "A IN mrkaran.hello.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000524939s [INFO] 10.1.28.1:59287 - 57522 "A IN mrkaran.svc.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000368277s [INFO] 10.1.28.1:53086 - 4863 "A IN mrkaran.cluster.local. udp 39 false 512" NXDOMAIN qr,aa,rd 132 0.000355344s [INFO] 10.1.28.1:56863 - 41678 "A IN mrkaran. udp 25 false 512" NXDOMAIN qr,rd,ra 100 0.034629206s</code></pre> Since mrkaran</code> didn’t specify any .</code> so the search list was used to find the answer. Note: ndots</code> value is silently capped to 15</code> and is 5</code> in Kubernetes as default. Handling this in Production#</a></h3> If your application is of the nature that makes a lot of external network calls, the DNS can become a bottleneck in case of heavy traffic since a lot of extra queries are made before the real DNS query is even fired. It’s quite uncommon to see applications appending the root zone in the domain names, but that can be considered as a hack. So instead of using api.twitter.com</code>, you can hardcode your application to include api.twitter.com.</code> which would force the DNS clients to do an authoritative lookup directly on the absolute domain. Alternatively, since K8s 1.14, the dnsConfig</code> and dnsPolicy</code> feature gates have become stable. So while deploying a pod you can specify ndots</code> setting to something lesser, say 3</code> or if you want to be really aggressive you can turn it down to 1</code>. The consequences of this will be that every intra-node communication now has to include the full domain. This is one of the classic tradeoffs where you have to choose between performance and portability. If the app doesn’t demand super low latencies, I guess you need not worry about this at all since DNS results are cached internally too. References#</a></h3> I got to know about this peculiarity first, in a K8s meetup</a> which I went to, last weekend where the folks mentioned about having to deal with this. Here are some additional links you can read on the web: Explainer on why ndots=5 in kubernetes</a></li> Great read on how ndots affects application performance</a></li> musl and glibc resolver inconsistencies</a></li> </ul> Note: I’m particularly not using dig</code> in this post. dig</code> apparently automatically adds a .</code> (root zone identifier) to make the domain an FQDN one without even first going through the search path. I’ve mentioned about this briefly in one of my older</a> posts. Nonetheless, it’s quite surprising to see that you need to give a flag to make it behave in what seems to be a standard way. It’s always DNS, isn’t it ;)#</a></h4> Fin! Update#</a></h3> 2020-12-24: In case you want to play around with ndots</code> and DNS more, I’ve worked on a DNS Client which lets you tweak these params on the fly. Feel free to checkout doggo</a> if interested! Resize PVC in Kubernetes 2020-01-14T08:10:55+05:30 Well, the title is self explanatory so let’s begin! First off, we need to ensure that the StorageClass</code> which was used to provision the PVC has the correct configuration. From the official docs</a>: You can only expand a PVC if its storage class’s allowVolumeExpansion field is set to true. </blockquote> So, let’s inspect our storage class: $ kubectl get sc # sc is short for storageclass NAME PROVISIONER AGE gp2 (default) kubernetes.io/aws-ebs 8d $ kubectl describe sc/gp2 # output redacted to focus only on the field we're concerned with Name: gp2 AllowVolumeExpansion: True</code></pre> If AllowVolumeExpansion</code> is set to True</code> you can skip the below step. If it’s not true, you need edit the field allowVolumeExpansion</code> as true</code>: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" name: gp2 parameters: fsType: ext4 type: gp2 provisioner: kubernetes.io/aws-ebs reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true</code></pre> Once the StorageClass</code> config is correct, all we need to do is update with the new size. So, for example if size of the PVC was 15GB</code> orginally: spec: resources: requests: storage: 15Gi</code></pre> To update it to 30GB</code>, you simply need to edit spec.resources.requests</code> field: spec: resources: requests: storage: 30Gi</code></pre> We now need to “apply” the updated PVC manifest. $ kubectl apply -f pvc.yml</code></pre> Let’s take a look at the PVC: $ kubectl describe pvc/myclaim # output redacted for brevity ... Conditions: Type Status LastProbeTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Tue, 14 Jan 2020 20:52:21 +0530 Waiting for user to (re-)start a pod to finish file system resize of volume on node. ...</code></pre> So, basically what FileSystemResizePending</code> means is that while PVC is in use, we have to either restart or delete the underlying Pod using the PVC. At the time of writing this, ExpandInUsePersistentVolumes</code> is still in beta</a> and has to be enabled as a feature gate. Sadly, EKS is still on 1.14</code> (while the world has moved to 1.17, such sloooow release cycles!), so I couldn’t enable this in my case. Once the pod is restarted, the expanded disk is automagically available! Let’s verify this: kc get pvc/myclaim -o=jsonpath="{.status.capacity.storage}" 30Gi</code></pre> Now, compare this with the standard way of resizing an EBS volume in EC2 instance. You need to first modify</a> the volume size using AWS EBS API and then in the EC2 instance, use a combination of growpart</code> and resize2fs</code> to extend</a> the resized volume. This sounds much more cumbersome than simply updating the storage field in PVC manifest! Fin! sshuttle - A better ssh tunnel 2020-01-12T09:57:55+05:30 The Motivation#</a></h2> Sometime back I had to access a Kubernetes API server which was firewalled to a private VPC network. I didn’t want to setup a separate bastion instance just to access this cluster, cause TBH bastions are kinda redundant in K8s as every task can be performed through the client-server APIs using kubectl</code>. So, all I needed was access to this API server from a trusted network in a secure way. Thanks to my friend @sarat</a>, I got to know about sshuttle</a>. sshuttle</code> is quite unique in the sense that it’s not really a VPN but acts like one (for most practical purposes). sshuttle</code> lets you access an internal network through a trusted node inside the VPC, without you having to deal with the mess of port forwarding or VPNs. The basic idea is pretty simple, sshuttle</code> starts a local python</code> server in your host machine and creates iptables</code> rules to route the destination packets of the specified CIDR blocks to this local server. At the server, the packets are multiplexed over an ssh</code> session and sent to the server. The server disassembles the multiplexed packet and the routes them to upstream. So, basically this is a clever hack to avoid TCP over TCP (which again is a mess on unreliable networks). Multiplexed streams on ssh</code> is just a single stateful TCP connection (as compared to VPN connections which are stateless). Now you must be wondering, how come the target server disassembles the packets. Yes, there needs to be some kind of sshuttle</code> daemon running which does that for you. This is where sshuttle</code> does some magic, it automagically deploys a python script on your target host to perform this task. So yes, for sshuttle</code> to work, both the client and target need to have python </code> and iptables</code> installed. Usage#</a></h3> sshuttle -r user@port x.x.x.x</code> All the packets routed to the CIDR block will now go through sshuttle</code> daemon, since it configured iptables</code> rules for them. Also, sshuttle</code> starts a local python server on your host machine. You can see it using netstat</code>: $ sudo netstat -tunapl | grep python Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:12300 0.0.0.0:* LISTEN 27425/python</code></pre> There’s a python</code> server listening on port 12300</code> in my host machine. To actually verify, this indeed is started by sshuttle</code>, you can use pstree -p | less</code> and search for sshuttle</code>. Here you can see sshuttle</code> did indeed start a python</code> server and the PID (27425</code>) matches with the one we saw in netstat</code> command. -zsh(13201)---sshuttle(27425)-+-ssh(27446)---ssh(27447) `-sudo(27427)---python(27445)</code></pre> You can even forward DNS queries with the --dns</code> flag. This is super helpful if you have something like Route53 to host your DNS records on a private zone (for eg tld like .internal</code>). Better than SSH tunnels?#</a></h3> Yes, you can also port forward with ssh using: ssh -nNT -L <local-port>:{upstream-host}:{upstream-port} user@remote</code> The problem with ssh</code> tunnels is that they experience frequent packet loss on a normal WiFi connection and it’s quite frustrating to deal with them. Moreover, sometimes you need access to multiple ports in your private network which requires you to explictly provide them with -L</code> flag which I find it as cumbersome. Also, you cannot forward DNS queries (over UDP) since ssh</code> can only do TCP. sshuttle</code> has made my life so simple! Fin! Use netcat for port scanning 2020-01-11T08:10:55+05:30 Quite often you’d need to check if a port on a target node is opened or blocked by firewall. I’ve always used telnet</code> to test that but it has a few drawbacks: Need to use dirty hacks</a> in shell scripts to auto close the connection. Also, telnet outputs some errors to /dev/stdout</code> instead of the standard /dev/stderr</code> which makes it harder to use in scripts. </li> Non standard implementation across different OSes. On Alpine Linux (mostly used in containers), if you install telnet using the /busybox-extras</code> package, the behaviour is different from what it is on standard Ubuntu/Arch environments. I’ve even faced weird issues on Alpine where telnet will simply wait endlessley for the connection to be established, while netcat would not indicate any issues. </li> Telnet is actually a protocol and the telnet-client initiates the negotiation</a> with the server before a connection is established. </li> </ul> So, after all these issues, I looked at other tools to eventually replace telnet</code> with something better. I tried nmap</code> which is also a port scanner, but is unreliable since a lot of hipster sysadmins drinking the security koolaid block port scanning tools like these. I wanted a dependable tooling and after a bit of Google-fu, I stumbled across netcat</code>. netcat</code> is basically a swiss army knife to perform all kind of ops with TCP/UDP. You can create a file server, chat client/server, TCP client etc. We are simply interested in the port scanning abilities of this for this blog post, so let’s actually see how to use it for the same. Note: Install netcat-openbsd</code> as it is a rewritten version of netcat-traditional</code> with some more bells and whistles. The basic syntax for port scanning looks like: nc -z host port</code> -z</code> tells nc to not send any data, just scan for any process listening on the target port. This is much better (and faster) than telnet</code> client initiating a connection with the upstream. To make it more usable however, let’s pepper our command with some helpful flags: nc -vz -w 3 host port</code> -v</code> turns on verbose mode which outputs diagnostic messages. -w</code> adds the timeout for the connection to be established. If you want to set a timeout in telnet</code> there’s a hack</a> for it. You can even supply a range of ports to netcat like: nc -vz -w 3 host 8000-9000</code> Quick Tip: You can also give an alias for port instead of the number. For example: $nc -vz -w 3 google.com https Connection to google.com 443 port [tcp/https] succeeded! $nc -vz -w 3 google.com ssh nc: connect to google.com port 22 (tcp) timed out: Operation now in progress nc: connect to google.com port 22 (tcp) failed: Network is unreachable</code></pre> Hope this post pretty much sums up the usage of netcat for port scanning! Read the man page</a> for more info. Fin! kubectl wait 2020-01-01T08:10:55+05:30 For the longest time I’ve had these commands in my .gitlab-ci.yml</code> file for a K8s CD pipeline: ... - kubectl apply -k overlays/prod - echo "Waiting for 15 seconds for pods to be restarted" && sleep 15 - kubectl get po ...</code></pre> So, basically I apply the changes to cluster using kubectl apply</code> and wait for arbitary decided time (15 seconds) to see the pod status</code>, hoping by that time the new deployments would have been active and old pods would be deleted. As the traditional SRE saying goes Hope is not a strategy this was clearly hacky and I knew it back then, just didn’t priortise enough to find a replacement. Recently got to know about kubectl wait</code> and woah, this is exactly what I needed. I can wait till either the condition is true or a timeout happens, whichever is earlier. This is so much better than the previous hack. kubectl wait --for=condition=available --timeout=60s --all deployments</code></pre> Here the condition</code> depends on the resource you are selecting. You can see the values for Conditions</code> using kubectl describe <resource></code>. For eg, for deployment and pods: $ kc describe deployments/{deployment_name} | grep Conditions -A 5 Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable</code></pre>$ kc describe pods/{pod_name} | grep Conditions -A 5 Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True</code></pre> So, now you’ll set the value for condition</code> according to your choice. This will be pretty useful in CI/CD pipelines. That’s pretty much it. Unrelated, but I thought about doing more such short posts and be consistent with more of writing. If you liked the short and precise format or have any feedback on it, do reach out to me on Twitter</a>. Happy New Year :) Fin! Introducing kubekutr 2019-12-30T08:10:55+05:30 kubekutr</a> was born out of my frustration of organising K8s resource manifests files/directories. For the uninitiated, K8s lets you hold the state of your cluster declaratively</a> as “manifest” files. K8s does so by a lot of asynchronous control loops to check whether the desired state matches the real world state and in case of drifts, it resets back to the user desired state (I oversimplified this, but hope you get the drift ;)). These files are predominantly YAML</code> but there’s support for JSON</code> as well. Anyway, to create these manifest files for a production level project is quite a bit of manual labour. The API spec of every resource in Kubernetes is quite daunting</a> and overwhelming. There are tools like Helm which abstract away the complexity of these YAML with it’s own templating system. There are quite a lot of these charts available for 3rd party apps here</a>. The idea is you populate the Chart with just your own config “values” and you’ve a deployment ready in no time. Admittedly this works quite well for something you want to take out for a quick spin but personally I am not quite a fan of hiding away the complexity with a magic layer. Also, the problem with Helm was the “Chart” (and templates) still had to be written by someone if you have a bespoke</a> application. Helm is more geared towards common off the shelf apps like DBs, Key Value stores, web proxies etc. I found out kustomize</a> few months back and quite happy with it’s approach towards managing manifests. The basic idea behind kustomize</code> is that you create a base and any kind of “customisations” must come as overlays. This is such a powerful technique over wrangling templates. kustomize</code> A common approach is to name these overlays based on the environment. For example, dev</code> deployment can have replicas: 1</code> for a pod, but prod</code> can apply a “patch” to update with repliacs: 3</code>. This way of separating two environments helps a lot when you follow GitOps</a> approach of deployment. All fine and good, until I realised I spent way too much time on copy-pasting the bases for different projects and manually editing these files for the new project config. Then I did what any other programmer would do, spend some more time to automate :P And that is how kubekutr</code> was born. (Quite an anticlimax I know!) kubekutr</code> is a really simple tool to bootstrap a Kustomize base</code>. kubekutr</code> reads a config file, templates out different resources and produces them as YAML files. Now, I know a lot of you reading this would be going Another damn templating solution in your mind and while that reaction is warranted, given that we have 200+ tools</a> in the community (everyone trying to solve similar problems in their own ways), I legit could not find a simple enough tool which would let the boring part of scaffolding a base out of my way and let me focus on what’s more important: the actual deployment. Hence I just decided to roll out my own solution which is the best one according to IKEA effect</a> (just kidding). Workflow#</a></h3> So, let’s say you need to create a Nginx deployment, the kubekutr</code> config.yml</code> would look something like this: deployments: - name: nginx replicas: 1 labels: - name: 'service: nginx' containers: - name: nginx image: 'nginx:latest' portInt: 80 portName: nginx-port services: - name: nginx type: ClusterIP port: 80 targetPort: 80 labels: - name: 'service: nginx' selectors: - name: 'service: nginx'</code></pre> To create the base</code>: kubekutr -c config.yml scaffold -o nginx-deployment</code> nginx-deployment</code> folder is initialised and you can view deployments/nginx.yml</code> and service/nginx.yml</code> which kubekutr</code> created. $ tree nginx-deployment |-- base | |-- deployments | | `-- nginx.yml | |-- ingresses | |-- services | | `-- nginx.yml | `-- statefulsets</code></pre>$ cat base/deployments/nginx.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: service: nginx spec: replicas: 1 selector: matchLabels: service: nginx template: metadata: labels: service: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: nginx-port</code></pre>$ cat base/services/nginx.yml apiVersion: v1 kind: Service metadata: name: nginx labels: service: nginx spec: ports: - port: 80 targetPort: 80 protocol: TCP type: ClusterIP selector: service: nginx</code></pre> You can now use the generated folder as a Kustomize base</code>. Non Goals#</a></h3> kubekutr</code> isn’t meant to replace the existing tools, it’s just a real simple cookie cutter approach to kustomize</code> bases and that’s pretty much it. kustomize</code> is native to Kubernetes and exposes the full API spec to end users. I feel that is much more better approach than templating solutions, the users must be exposed to the standard conventions rather than a random tool’s own config fields. The benefits are the same conventions can then be used across a wide variety of tools (like kubekutr</code>) and users are in better control of the underlying resources. Adding a layer of magic also makes it harder to debug when shit goes down. Hence kubekutr</code> chose kustomize</code> to do all the heavy lifting of managing manifests. There’s a lot of scope of improvements, but I wanted to just Ship It! and get some initial feedback. Let me know your thoughts on this :) Fin! A quick primer on dig 2019-11-11T10:57:55+05:30 Dig is a DNS lookup utility developed by BIND</a> which helps a lot while troubleshooting DNS issues (which are more common than you probably think #hugops). I use dig</code> fairly often and thought to write an introductory guide on how you can use dig</code> with some practical examples that’ll help you dig</code> through DNS issues faster (sorry for the lame pun, couldn’t resist.) Basics#</a></h2> The most basic and common usage for dig</code> is to query the authorative servers for a particular domain and retrieve the IP. If it’s an IPv4 then you should be looking at A</code> record, while if it’s IPv6 then AAAA</code> record is your friend. Let’s see the DNS records for the site you’re currently on: ➜ ~ dig mrkaran.dev ; <<>> DiG 9.10.6 <<>> mrkaran.dev ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23292 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1220 ;; QUESTION SECTION: ;mrkaran.dev. IN A ;; ANSWER SECTION: mrkaran.dev. 60 IN A 206.189.89.118 ;; Query time: 6 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Tue Oct 29 23:13:31 IST 2019 ;; MSG SIZE rcvd: 67 </code></pre> This is the most basic example for dig</code>. Let’s explore some of the additional options. Keep it short#</a></h3> dig +short</code> keeps the information to bare minimum and only displays the ANSWER</code>. dig +short mrkaran.dev 206.189.89.118</code></pre>Nameserver details#</a></h3> If you want to find the Nameserver</code> for your DNS records, you can use the query type ns</code>. $ dig mrkaran.dev ns +short alec.ns.cloudflare.com. cruz.ns.cloudflare.com.</code></pre> ns</code> is one of the many query types you can use to indicate which type of DNS record you want to fetch. Default is A</code> record which returns the IPv4 address of the domain (unless it’s a root domain, in which case the default query type is NS</code>). Some other examples of query types are mx</code>, AAAA</code>, TXT</code> etc. Fun Fact: ANY</code> query type has become obsolete</a> as per the new RFC8482</a> and DNS operators can choose to not respond to this query. The reason for this is that the payload response size for an ANY</code> query is quite huge (since it has to return all type of DNS records) and this could affect the performance of authoritative servers in case of a DNS amplification</a> attack. Using different DNS server#</a></h3> Let’s say you want to switch to a different resolver, you can use @</code> followed by the address of your DNS server. $ dig mrkaran.dev @9.9.9.9</code></pre>Reverse DNS Lookup#</a></h3> This one’s actually pretty cool. dig -x</code> lets you query the IP and retrieve the hostname details for that IP. dig -x 206.189.89.118</code></pre>Multiple queries#</a></h3> You can input a list of domain names and pass the file with the arg -f</code> to dig. $ cat digfile mrkaran.dev joinmastodon.org zoho.com</code></pre> To list down all MX records for the domains in a file, you can use something like: $ dig -f digfile +noall mx +answer mrkaran.dev. 242 IN MX 10 mx.zoho.in. mrkaran.dev. 242 IN MX 20 mx2.zoho.in. mrkaran.dev. 242 IN MX 50 mx3.zoho.in. joinmastodon.org. 21599 IN MX 10 in1-smtp.messagingengine.com. joinmastodon.org. 21599 IN MX 20 in2-smtp.messagingengine.com. zoho.com. 299 IN MX 10 smtpin.zoho.com. zoho.com. 299 IN MX 20 smtpin2.zoho.com. zoho.com. 299 IN MX 50 smtpin3.zoho.com.</code></pre>Search List#</a></h3> I learnt this recently while debugging a DNS issue in one of the Kubernetes pods. Dig doesn’t use search paths by default, so if you have a service say redis</code> inside a namespace dig won’t fetch any result: $ dig redis +short # empty output, indicates no record found</code></pre> This is because a service name in Kubernetes is of the form service.namespace.svc.cluster.local</code>. So, we should actually be querying for redis.myns.svc.cluster.local</code> and we’ll get our result. But isn’t that too long and painful (sorry for the pun) to type? So, there’s another option +search</code> which can be used to find all domains matching the search path defined in /etc/resolv.conf</code> namesever configurations. $ cat /etc/resolv.conf nameserver 10.100.0.10 search myns.svc.cluster.local svc.cluster.local cluster.local</code></pre> We can now query for redis</code> with this search list: dig redis +search +short 10.100.32.73</code></pre>DNSSec Validation#</a></h3> dig</code> even lets you validate the DNS records you received using DNSSEC</code> validation. $ dig mrkaran.dev +dnssec ; <<>> DiG 9.10.6 <<>> mrkaran.dev +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36275 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 1452 ;; QUESTION SECTION: ;mrkaran.dev. IN A ;; ANSWER SECTION: mrkaran.dev. 20 IN A 178.128.17.49 mrkaran.dev. 20 IN RRSIG A 13 2 20 20191112173050 20191110153050 34505 mrkaran.dev. Tl3zD6EqfVRvZi79ahePQcAXnbSUY9ZEYx/KwXnDUyonlrCKuBHzIYYC MJoVns410+sOwbIrcAdLgx+eiMYqRQ== ;; Query time: 65 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Mon Nov 11 22:01:01 IST 2019 ;; MSG SIZE rcvd: 163</code></pre> The important bit to note here is the ad</code> flag set which represents Authenticated Data. The records will only be returned if the validation succeeds (unless you also specify +cd</code> which indicates Checking Disabled flag.) On a server which doesn’t have DNSSEC enabled, you can see no records are returned with the +dnssec</code> flag. $ dig dnssec-failed.org +dnssec ; <<>> DiG 9.10.6 <<>> dnssec-failed.org +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19886 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;dnssec-failed.org. IN A ;; Query time: 335 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Mon Nov 11 22:03:50 IST 2019 ;; MSG SIZE rcvd: 35</code></pre> That pretty much broadly covers some practical examples with dig</code>. I will soon write a detailed post on how DNSSEC</code> validation works and why it needs to be mainstream. Fin! GitOps approach to Continuous Delivery for Kubernetes 2019-11-11T10:57:55+05:30 In this post, I’d like to share my experience and learnings about configuring a deployment pipeline for Kubernetes. I’ll be using Gitlab CI/CD and AWS EKS to demonstrate the concept, but the core idea remains the same: all changes must come declaratively from a single source of truth. GitOps</code> is a relatively newer term in the town but goes back to the fundamentals of Infra as Code. GitOps fundamentally is an operating model to perform tasks on Kubernetes related to deployments, configuration, secrets and monitoring workloads. All kind of changes must be performed via a single place, which happens to be a git</code> repo. Benefits of that are what basically benefits of version controlling the code is. So why treat infra as any different? git</code> happens to be the single source of truth for your infra, rollbacks are easy as reverting to last known good configuration and every change can be observed/verified. Goals#</a></h2> A lot of tutorials/blog posts hitherto cover a very basic scenario where they do kubectl apply</code> and voila the deployment’s live. However we all know things are very different (to say the least) in production, so this post will cover all aspects of deployment: Creating Manifests</li> Environment Promotion</li> Handling config and secrets</li> Authorization of CI/CD in the cluster</li> </ul> Basics#</a></h3> A GitOps workflow looks like: Push vs Pull#</a></h3> There are 2 approaches to how you can handle deployments to a cluster. In a Pull based approach, the cluster runs a synch controller program which continmioously syncs the state of cluster with a Git repo. Any changes you make to the Git repo will be synced automatically in cluster. The idea is that there should be no drift in the desired state via Git repo and the actual state of cluster. Flux, Argo are good tools if you want a Pull based pipeline. The merits of Pull based pipeline is it’s more secure, since the deployment is actually happening inside cluster and no external sytstem needs to communicate to your production infra. The demerits are sometimes you’ve to wait for the changes to be synced (every controller runs these sync process in a loop with a sleep which can be configured). Also using any kind of preprocessing tools like Kustomize becomes difficult, since Flux just syncs the state and applies those changes. Handling of secrets is yet another concern, you need to look . And finally GitOps is a relatively newer tech in market so GitOps tooling is still nascent and like with any other relatively (non battle tested) software you’re gonna find bugs. Push Approach however is a traditional CD approach, where the CD server talks to the cluster and applies changes through commands. In context of normal EC2 deployments, those commands could be SSH into server, running ansible playbook etc. In context of K8s however kubectl</code> does the magic for us. The CD server needs to talk to the K8s API server and run kubectl commands to change the cluster state. The merits of this approach are you can run all sorts of commands inside deployment pipeline and make it fully customisable. Handling of secrets also can be handled natively (like Gitlab env variables) or encrypted in git</code>. The demerits is that your production cluster is now exposed to your CD server. Overall, if you have an airgapped CD server with no inbound ports open, access controll the user auth to CD, I found the Push approach to be more preferable. YMMV. Writing the pipeline#</a></h2> I’ve created a docker image</a> eks-gitops</code> which I’ll be using throughout the pipeline. This container image contains popular tools like kustomize</code>, kubeval</code> etc and scripts to configure access to cluster using kubectl</code> using aws-iam-authenticator</code>. I’ve written more about how RBAC works inside EKS here</a>. Excerpt from .gitla-ci.yml</code>: # Use this as base image for all jobs unless overriden default: image: name: mrkaran/eks-gitops:latest entrypoint: ["/bin/sh", "-c"] ### Pipeline stages: - validate - deploy</code></pre>Prepare the manifests#</a></h3> I use Kustomize</code> to prepare the manifests. Advantage of Kustomize</code> is writing template free YAMLs but still be able to customise them heavily using overlays. For different environments, you can apply certain changes like increasing resource requests, adding more storage, while keeping the base same. Here’s a folder structure (from a real GitOps repo) I follow for manifests: . ├── base │ ├── deployments │ │ ├── app.yml │ │ ├── celery.yml │ │ └── nginx.yml │ ├── ingresses │ │ └── web.yml │ ├── kustomization.yaml │ ├── services │ │ ├── app.yml │ │ ├── nginx.yml │ │ └── redis-headless.yml │ ├── statefulsets │ │ └── redis.yml │ └── volumes │ └── redis.yml ├── kubekutter.yml ├── Makefile ├── overlays │ ├── dev │ └── prod │ ├── configs │ │ ├── app-config.env │ │ └── app-nginx.conf │ ├── kustomization.env.yml │ ├── namespace.yml │ ├── patches │ │ ├── configure-configmap-volume.yml │ │ ├── modify-alb.yml │ │ └── resource-limits.yml │ └── rbac.yml └── README.md</code></pre> Shameless Plug: I created kubekutr</code> which makes managing of these manifests using kustomize</code> a breeze. Some things to note here: Inside base/</code>, I keep all the base</code> resources required for the app to run. The resources can be Service</code>, Deployment</code>, Ingress</code> etc.</li> Inside overlays</code> there are multiple folders for different environment. This is very crucial as we want to separate the production config with a UAT config. Last-mile configuration to the base becomes very easy with this folder structure, since you only now need to build the manifests by targetting a specific folder in CI.</li> Inside overlays/{env}/patches</code> are all the “patches” you want to do to the base resource. Think like replica count, ALB subnets (since different env can be in different VPCs), increasing resource limits and stuff like that.</li> rbac.yml</code> abd namespace.yml</code> is the only missing piece because it’s like a chicken and egg problem. I cannot deploy directly (at first go) from a CI/CD if I don’t have a namespace created since the CD server is configured only has limited namespaced restricted access. So unless I create a namespace, add proper RBAC for the CD server I cannot do any deployments from CD. Note however this is only a first time step, which I guess is okay.</li> </ul> Lint yo manifest#</a></h3> I’m using kubeval</code> to lint the manifests. The manifests have to prepared by kustomize</code>. CI_ENVIRONMENT_NAME</code> is set by Gitlab</code> when you specify an environment for a job. Don’t sweat about this part, I’ll describe it more as we proceed. # Validate the yaml using kubeval .lint: extends: .prepare-manifest stage: validate script: - echo "Linting manifest for ${CI_ENVIRONMENT_NAME}" - kustomize build overlays/$CI_ENVIRONMENT_NAME --load_restrictor none | kubeval</code></pre>Setup environment#</a></h3> You can define environment name as: # Create an environment to record all jobs for this env .prod: &prod environment: name: prod url: https://prod.site .dev: &dev environment: name: dev url: https://dev.site</code></pre> Any job which is to be executed in a particular environment can include this variable and CI_ENVIRONMENT_NAME</code> will be automatically set. A cool feature of Gitlab is that you can restrict Variables scoped to the environment they are defined in. Configure Secrets#</a></h3> All secrets are defined as Environment Variables in Gitlab CD pipeline. While running the job, the runner has access to these variables and with the help of secretGenerator</code> in kustomize</code>, the Secret</code> is created. I use secretGenerator</code> because any time a K8s secret changes, kustomize appends with a new suffix, which makes the Replication Controller believe that they deployment has changed. So a new deployment is automatically triggered. Authenticate to cluster#</a></h3> EKS uses aws-iam-authenticator and uses IAM access roles to allow the cluster to perform actions. Since this is a push based pipeline, you need to allow the access from your CD server to port 443. Deploy changes#</a></h3> This is as simple as kubectl apply</code> which configures all the changes and diff between cluster and real world state. Here’s a full gitops repo if you’re interested in checking it out: Intro to RBAC in EKS 2019-11-01T18:10:55+05:30 EKS uses a custom authenticator tool called “aws-iam-authenticator”. The basic idea is to make the auth flow in EKS easier by using the tools you already use in AWS. To wrap your head around the flow, consider three separate entities: A kubernetes resource (entire namespace, specific pods/configs etc)</li> An action (get, watch, list, create, delete etc)</li> An IAM role/user created on AWS</li> </ul> Your usecase might be to give an IAM role access to a Kubernetes namespace for example with certain restricted actions. Since K8s is basically client-server communication with the API server, we must need to perform the following two things for every single API request which goes to the control plane: Authentication</li> </ul> kubectl</code> talks to the API server using the token generated by aws-iam-authenticator</code>. The API server passes on this info to the AWS servers which validated if the originating call is coming from a valid IAM user or not. If not, the access is denied from K8s. Authorization</li> </ul> Once the IAM user is validated, the IAM user needs to be mapped to a user to perform Authorization so K8s API server can know whether the action/resource requested for is to be allowed or not. This is where the aws-auth-cm.yml</code> comes into the picture. It is basically a map of all IAM users with internal K8s groups or users created. The roles are assosciated to these users/groups so once the mapping is done, K8s API server can know what to do with the API request. How does EKS know my IAM?#</a></h3> If you configured your KUBECONFIG</code> correctly using aws eks update-kubeconfig</code> then you’ll find the below lines in your config file. This basically runs a aws cli</code> command to find your IAM user/role configured on your system to produce a token. exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - token - -i - eks-zero-public command: aws-iam-authenticator</code></pre>Show me the YAML already#</a></h3> To create a Role and RoleBinding</li> </ul> apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: coolapp name: fullaccess rules: - apiGroups: [""] resources: ["*"] verbs: ["*"] - apiGroups: ["apps"] resources: ["*"] verbs: ["*"] - apiGroups: ["batch"] resources: ["*"] verbs: ["*"] - apiGroups: ["extensions"] resources: ["*"] verbs: ["*"] - apiGroups: ["autoscaling"] resources: ["*"] verbs: ["*"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: 10xdevs-fullaccess namespace: coolapp subjects: - kind: Group name: 10xdevs namespace: coolapp roleRef: kind: Role name: fullaccess apiGroup: rbac.authorization.k8s.io</code></pre> So, for all the normal folks who don’t grok YAML as fast as 10x devops engineers, I am basically creating a Role fullaccess</code> with all permissions for a namespace coolapp</code> (innovative, ikr). Then I am binding</code> this Role to a Group called 10xdevs</code> so that the group is allocated the role which has permissions. We will use this group 10xdevs</code> to map our AWS user now. To create the map of IAM Role/User ARN with the above Role</li> </ul> Edit your aws-auth-cm.yml</code> and add the below stuff: apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles ... <skipping> ... mapUsers: | - userarn: arn:aws:iam::<account-id>:user/<user-name> username: 10xdevs-fullaccess # RoleBinding name created in previous step groups: - 10xdevs # Group name created in previous step</code></pre>Verify it yourself#</a></h3> kubectl auth can-i create pods --all-namespaces</code>: should fail </li> kubectl auth can-i create pods -n coolapp</code>: should work </li> </ul> Fin! Kubernetes cluster on RPi 2019-09-22T08:10:55+05:30 So, I got hold of 2 Raspberry Pi4 (still limited stocks in India) recently and wanted to build a Kubernetes cluster. Don’t ask why cause that would be pointless. I’ve little experience with a managed Kubernetes workload (Amazon EKS, which btw deserves its post 😝) but never really played around any of the K8s internal stuff yet. In this post, I’ll show you how I got a lightweight Kubernetes distro: K3s up and running. k3s</a> is a pretty great Kubernetes distro which passes the K8s Conformance tests. On ARM architectures, you’re pretty much resource-bounded and you want the resource footprint of your infra to be as minimal as possible. In k3s there are a few changes such as the persistent layer of K8s is replaced with SQLite instead of etcd, unusable (legacy/alpha) features of K8s are removed and cloud-provider plugins are not bundled (but can be installed separately). All of this together means just 40MB of a binary to run the cluster and ~250MB of memory usage on an idle cluster. Awesome, team Rancher</a> :) Automation is the key and even though I have just 2 nodes on RPi, I Ansible-ized the setup which I am hoping would save time in future if I add more nodes to the setup. Hardware#</a></h2> 1x RPi4 4GB</a> and 1x RPi4 2GB</a> variant</li> 2x Samsung EVO micro SD Card</a></li> 2x USBC Cables</a></li> 1x Amker Power Port</a> (don’t compromise on the power supply, give enough juice so RPi doesn’t throttle)</li> 1x TP-Link Network Switch</a> (my router has only 1 usable LAN port)</li> 2x CAT5 LAN cables</a> (keeping it basic, you can get fancy flat LAN cables if you wish to)</li> </ul> This is how the final setup looks like: {{< tweet 1175659256764219392 >}} Setting up RPi#</a></h3> I downloaded Raspbian Buster Lite</a> because it’s the easiest to setup. Next step is to flash the SD card and for that, I used Etcher</a>. To enable SSH access, you need to create an empty file ssh</code> in the root volume. sudo touch /boot/ssh</code></pre> Once all sorted, we can use Ansible to set up the basic OS stuff, like changing the default password, enabling password-less SSH login, timezone & locale settings, changing hostname etc. I’ll be sharing relevant Ansible snippets, if interested you can check out the complete playbook at mr-karan/hydra repo</a>. We need to enable container features on the RPi so that containerd</code> can run. Containers like Docker make use of cgroups</code> (Linux kernel feature) which allows them to put resource limits on container processes like CPU and Memory. To enable cgroups</code>, you need to edit /boot/cmdline.txt</code>. - name: Add cgroup directives to boot command line config lineinfile: path: /boot/cmdline.txt regexp: '((.)+?)(\scgroup_\w+=\w+)*$' line: '\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory' backrefs: yes</code></pre> Quick Tip: If you’re going to use RPi as a headless server (not connecting with any monitor) you can reduce the GPU Memory to lowest possible (16M) - name: Set GPU memory split to 16 MB lineinfile: path: /boot/config.txt line: "gpu_mem=16" create: yes</code></pre>Deploy K3s cluster#</a></h3> Next, we’ll come to the actual stuff, where we’ll download K3s binary and run as a systemd service. There’s a handy shell script</a> to bootstrap the cluster provided by the Rancher team which setups the whole thing in one command. But if you wanna learn/play around I’d recommend you do things the hard way (it’s not all that hard tho. ((twss!)). - name: Download k3s binary armhf get_url: url: https://github.com/rancher/k3s/releases/download/{{ k3s_version }}/k3s-armhf dest: /usr/local/bin/k3s owner: root group: root mode: 755 when: ( ansible_facts.architecture is search("arm") ) and ( ansible_facts.userspace_bits == "32" )</code></pre> On the worker node, the process is similar, except you have to run the K3s with agent</code> compared to server</code> argument in the control plane. Things get a bit interesting here though. You need to give the cluster server URL along with it’s token in the command. A unique token is generated by the server at (/var/lib/rancher/k3s/server/node-token</code>) which is used to join the worker nodes. I did a bit of google-fu and got to know about this neat little Ansible module set_fact</code></a> which lets you “store” a variable from one host and use it in a second host. Every Ansible host maintains a Python dict of “host facts”. In the second node, I access the cluster’s host fact dict</code>, fetch the variable and use it in its systemd service template. Neat, ain’t it? Ansible has so many modules, it is mind-boggling. Reading and storing the variable as a “host” fact: # on the cluster - name: Read node-token from control node slurp: src: /var/lib/rancher/k3s/server/node-token register: node_token - name: Store control node-token set_fact: k3s_cluster_token: "{{ node_token.content | b64decode | regex_replace('\n', '') }}"</code></pre> Using the variable from the server host, in a template on the agent host: # on the agent (vars.yml) k3s_server_address: "{{ hostvars[groups['control'][0]].k3s_server_address }}" k3s_cluster_token: "{{ hostvars[groups['control'][0]].k3s_cluster_token }}" # use the value in a template ... [Service] ExecStart=/usr/local/bin/k3s agent --server {{ k3s_server_address }} --token {{ k3s_cluster_token }} ...</code></pre> P.S. Shoutout to Ansible tho. It is one of my fav infra tooling available out there. It has some gotchas that you need to be aware of but by and large, the experience has been quite pleasant. On the cluster, you should be able to see the nodes. Team #SelfHost#</a></h3> I am planning to host Bitwarden, Gitea and a Nextcloud instance on this cluster. Also will be using this as a testbed to play around with K8s internals. Stay tuned as I explore more of this! Cheers! :) My Personal Networking Setup 2019-06-29T18:10:55+05:30 If I have nothing to hide then I have nothing to show either </blockquote> When you visit a site from your browser, there are several entities which have access to this information in one request-response cycle. Starting from the router you connect to, to the DNS resolver you use to resolve the IP Address of the site, and finally, the ISP which has access to the raw network packets (unencrypted or encrypted). They are all insecure by default and most users on the internet do not care about their privacy. ISPs can very easily profile you by the queries you do and sell that data to 3rd party Ad companies. There are still a huge number of sites on the world wide web which don’t have a basic SSL encryption turned on. Even with the HTTPS turned on your ISPs can simply see the unencrypted DNS queries and figure out the site you’re visiting. Browser fingerprinting is another well-known technique where you can be easily profiled based on the different parameters exposed by your browser and generate patterns in your browsing habits. If you’re a human you already care about your privacy. You won’t give out your Whatsapp chats to a random stranger on the streets to read, you won’t leave the door open while bathing, you won’t leave your credit card details out in the open, you will have a closed private discussion with your second half. Everyone does care about privacy in real life and everyone should care about the same online too. “Some” social media companies exist solely on the data you provide. They are in the business of selling data and they don’t care a bit about your privacy. Every ad company out there wants an online persona about you, to feed you “personalized” recommendations. You may argue that they are “useful” but you cannot deny the fact that they are invading your privacy. You won’t like unwanted relationship advice from your neighbour when he hears you both fighting in the middle of the night. No one likes invaders, period. Correlation of data points is easy to do in this age of ML/AI (Malai). Re-iterating an example I read somewhere on the interwebz, say you search for a very generic term “cancer”, this alone doesn’t indicate that you have cancer but merely is suggestive of the fact that you are interested in knowing more about the disease. Now maybe a few days later you search for the phrase “buy a wig online”. These 2 search queries put together can suggest that you’ve had chemotherapy done recently. This is a piece of information you might not want to hide, but you do want to have an option right? That’s the whole fight against companies which invade your privacy is. You have absolutely no choice. You’re devoid of any choice. Google is your friend </blockquote> You may not even realise but we’ve all been guilty of Googling some random shit for fun, googling about anxiety or depression when feeling low, how to lose weight and all such seemingly embarrassing questions and we simply confront these things to Google thinking that no one is spying on us or being judgemental of us and we’re in a safe zone. What if this “friend” is the one who’s stabbing your back. No surprises there! You don’t even know but these search engines might have already shared all such queries with several 3rd party providers and business who are on the lookout for such data. I am scared </blockquote> I hope by now you’re convinced that you want some basic privacy online. If not, stop reading here because if this didn’t persuade you enough, nothing will and the rest of the blog is pretty useless for you anyway, so thanks for stopping by! For the rest of us folks, there are a few practical things we can immediately start which. These are the things which have very less involvement of effort but are a good kickstart. Use DNS over HTTPS (DoH) based resolver (personal choice: 1.1.1.1)</li> Use a browser which doesn’t ship with its own god damn resolver</a> (personal choice: Firefox)</li> Use an Ad-Blocker (personal choice: uBlock Origin)</li> </ul> Only with the above 3 steps you’ve achieved: ISPs cannot see your DNS queries in plain text</li> Websites cannot serve you random js code in the name of ads</li> A web browser which is more focussed towards privacy in general</li> </ul> If you’re someone who wants to spend a bit more time (trust me it won’t take even one hour) and a bit of money (one movie ticket a month) you should host your personal VPN. VPNs are awesome because you mitigate a lot of privacy invaders and you are on an anonymous browsing mode generally. Theoretically, we have shifted the trust from the ISP to the cloud provider where VPN is hosted, yes. But as long as you’re not doing anything illegal in the eyes of the state, you should be fine. Protip: Don’t use VPN to do something stupid. You are always trackable. The whole exercise is to not allow bad actors to access your personally identifiable information online and have a non-intrusive browsing experience. If you think by using VPNs you can get away with doing something which is illegal by your state then you’re wrong. Tools I use#</a></h2> VPN: Wireguard</a></li> Adblocking: Pi-hole</a></li> DNS Resolver: Unbound</a></li> </ul> The setup is pretty straightforward. I use Wireguard client on my MBP and an Android device to connect to the Wireguard VPN server hosted on a $5 DO</a> droplet in Bengaluru, India. For ad-blocking, I use Pi-hole, which does DNS based ad-blocking. And finally to resolve the DNS queries I use vanilla unbound without any forwarders. Read the following sections to know more on each of the above pieces. Setting up wireguard VPN#</a></h2> Wireguard gets all the love from me (thanks to @sarat</a> for telling me about this). It is probably the best option out there if you’re looking to self-host a VPN. Wireguard gets a lot of things right, first being the ease of setup. Literally takes a few commands and you’re all set. Wireguard is very lightweight and consumes minimal resources. Wireguard has better encryption and is a lot faster</a> than IPvsec or open VPN. Cloudflare’s new app Warp is also based on Wireguard. Oh, and did I mention that Wireguard seamlessly transitions when you switch networks (which happens all the time when you’re on shitty 4G network/public WiFis). Wireguard basically has 2 parts. One is the server and the other is the client. Wireguard sits in its own separate network namespace and uses this namespace as an init namespace from where the traffic is received or sent. This namespace is now responsible for flowing your traffic to the other actual interface your network card on the device (which is probably wlan or eth). Installing Wireguard#</a></h3> $ sudo add-apt-repository ppa:wireguard/wireguard $ sudo apt-get update $ sudo apt-get install wireguard-dkms wireguard-tools linux-headers-$(uname -r)</code></pre>Generating Public/Private key pair#</a></h3> $ umask 077 $ wg genkey | tee server_private_key | wg pubkey > server_public_key $ ls server_private_key server_public_key</code></pre>Configuring Wireguard (server)#</a></h3> $ touch /etc/wireguard/wg0.cong $ vim /etc/wireguard/wg0.conf # Add the following lines and modify the values [Interface] # Configuration settings for a separate network interface Address = 10.200.200.1/24 # You can choose any private subnet SaveConfig = false # Wireguard can configure additional peers automatically without reloading wireguard, for some reason this didn't work well for me PrivateKey = <redacted> # Output of the `server_private_key` generated in the above setup ListenPort = 51820 # Default port # Add the peers (clients which connect to the wireguard server) [Peer] # MBP PublicKey = <laptop_public_key> AllowedIPs = 10.200.200.2/32 [Peer] # Android PublicKey = <phone_public_key> AllowedIPs = 10.200.200.3/32</code></pre>Configuring Wireguard (client)#</a></h3> Repeat the step of generating a Public/Private key pair. You can also take a look at Subspace</a> which is a nice GUI tool which helps you create additional profiles for your devices where generating key/pair is not convenient like mobile phones. I didn’t get the time to set it up personally, so I generated public/private keys from my laptop itself and then configured it manually using an Android app Viscerion</a> which is a wireguard client app. </li> For the client, your config file should look like: </li> </ul> [Interface] PrivateKey = <client_private_key> Address = 10.200.200.2/32 [Peer] PublicKey = <server_public_key> AllowedIPs = 0.0.0.0/0 Endpoint = <public_ip_droplet>:51820 PersistentKeepalive = 25</code></pre>Recap#</a></h3> If you got overwhelmed at this point, let’s recap what just happened. We have 2 config files, one for the server and one for the client which happens to be my laptop. On each of the devices, generate a public/private key pair. On the server side, while configuring peer, give the public key of the client. On the client side, while configuring peer,give the public key of the server. This is similar to how ssh</code> works. Wireguard uses Curve25519</code> crypto technique to generate a public/private key pair, which honestly looks so better than lengthy ECDSA/RSA ones :P Now let us start the wireguard service. wireguard provides a nice wrapper wg-quick</code> which does the following things when you start: sudo wg-quick up wg0</code></pre>[#] ip link add wg0 type wireguard [#] wg setconf wg0 /dev/fd/63 [#] ip -4 address add 10.200.200.1/24 dev wg0 [#] ip link set mtu 1420 up dev wg0</code></pre> You can verify a new network interface now by: $ ip addr | grep wg0 5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 inet 10.200.200.1/24 scope global wg0</code></pre> You can also view the wireguard connection status by: $ sudo wg show interface: wg0 public key: <REDACTED> private key: (hidden) listening port: 51820 peer: <REDACTED> allowed ips: 10.200.200.2/32</code></pre>Almost There, But Not Quite#</a></h3> Turn on your wireguard client and you will notice a strange thing. At this point, you’re not able to browse the internet but you are able to connect to the wireguard server and even wireguard is acknowledging that (notice the last two lines in the following snippet): $ sudo wg show interface: wg0 public key: <REDACTED> private key: (hidden) listening port: 51820 peer: <client_public_key> endpoint: <client_public_ip>:64882 allowed ips: 10.200.200.2/32 *latest handshake: 8 seconds ago* *transfer: 754.60 KiB received, 5.59 MiB sent*</code></pre> Wireguard status shows that our client can reach the wireguard server. But still, we’re unable to browse the internet on our client. To debug this further, let’s use tcpdump</code> and monitor the packets coming in wg0</code> interface. Since tcpdump’s output can be overwhelming and contains a lot of noise, let’s filter to monitor only the ICMP packets. So we’ll be using ping</code> from a client which is the easiest way to send ICMP packets from point A to point B. # on wireguard client ➜ ~ ping $(curl icanhazip.com) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14 100 14 0 0 16 0 --:--:-- --:--:-- --:--:-- 16 PING <server_ip_redacted> (<server_ip_redacted>): 56 data bytes 64 bytes from <server_ip_redacted>: icmp_seq=0 ttl=64 time=112.126 ms 64 bytes from <server_ip_redacted>: icmp_seq=1 ttl=64 time=142.980 ms ^C --- <server_ip_redacted> ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss</code></pre># on wireguard server $ sudo tcpdump -nni wg0 -Q in icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes 14:26:31.991498 IP 10.200.200.2 > <server_ip_redacted>: ICMP echo request, id 52630, seq 0, length 64 14:26:33.027288 IP 10.200.200.2 > <server_ip_redacted>: ICMP echo request, id 52630, seq 1, length 64</code></pre> Okay, this is getting interesting. wg0 is definitely receiving packets from our client. Arghhh. Now does it strike to you? Our actual network routing is through eth0</code> (on DO usually) and wg0</code> is just some interface created by Wireguard. They have no “connection” (no pun intended) between them. Let us confirm if this actually is the problem by detecting ICMP traffic on this interface(eth0</code>) using the same command: # on wireguard server $ sudo tcpdump -nni eth0 -Q in icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type RAW (Raw IP), capture size 262144 bytes ... # nothing happens even if we are sending PING from client</code></pre> Wow, see? No ICMP packets received on eth0</code>. So that indeed is the problem and we have zero’ed it down using tcpdump</code>. (Note to self: Learn more Linux debugging utils, these things are a godsend!) In order to fix this, we need to do 2 things: IP Forwarding</li> </ul> $ vim /etc/sysctl.conf # check for this line and replace the value from 0 to 1 net.ipv4.ip_forward = 1 $ sudo sysctl -p</code></pre> IP Tables Rules</li> </ul> We need to set up NAT between eth0</code>(could be different for you) and wg0</code>. This can be done using iptables</code> and wireguard actually has a nice mechanism to run custom commands using PostUp</code>/PostDown</code> signals. # on wireguard server $ vim /etc/wireguard/wg0.conf # add these lines in [Interface] section PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Configure iptables to setup a NAT on eth0 and forward the packets (ipv4 and ipv6) on interface wg0 to eth0 PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE # Delete the rule since when wireguard is down, wg0 doesn't exist</code></pre> Now everything’s set up and we are browsing the internet privately using a VPN. Ensure</a> that your public IP is of the VPN server when you’re browsing and it’s not leaking. Setting up Pi-hole#</a></h3> Installing Pi-hole is as simple as curl -sSL https://install.pi-hole.net | bash</code></pre> You can read more about installation</a> in the official docs. A nice GUI is always a plus, so make sure you enable that option while installing Pi-hole. Now, we need to configure our Wireguard client to use Pi-hole as a nameserver for DNS resolution. # on the client add the following line sudo vim /etc/wireguard/wg0.conf [Interface] ... DNS = <vpn_server_public_ip> ...</code></pre> Pi-hole runs on port 53 and accepts DNS queries over UDP. Any query is first checked by Pi-hole in the blacklist. If it’s present in the blacklist, it’s immediately dropped. If not, Pi-hole will forward our DNS query to one of the forwarder configured (1.1.1.1 for eg or our custom server, explained in the next step). Setting up Unbound DNS#</a></h3> I don’t mind trusting Cloudflare. But I simply don’t have to. :) And once you make peace with the fact that you don’t need 3rd party companies controlling your networking stack, you’ll sleep better. I have setup Unbound without any forwarders. Unbound is a recursive resolver which supports DNSSec and caching mainly. Unbound first checks if the query exists in cache and if it does, it directly returns the “answer”. Otherwise it talks to the root nameserver and then the whole DNS dance happens. Since our DNS query is now split into multiple parts, where each nameserver is only being queried for a part of the FQDN (also known as QNAME minimialistion), it becomes a lot harder for anyone to intercept or reconstruct your DNS queries. You can install unbound using $ sudo apt-get install unbound</code></pre> To start using Unbound, we need a file root.hints</code> which contains information about root nameservers. You can cron this to fetch a new copy every once in 3-4 months, it hardly changes. $ wget -O root.hints https://www.internic.net/domain/named.root $ sudo mv root.hints /var/lib/unbound/ $ sudo service unbound restart</code></pre> You can verify if the DNS queries are being resolved by: # 6363 is where I have configured my Unbound server to listen dig mrkaran.dev @127.0.0.1 -p 6363</code></pre> Pi-hole official docs have a great</a> explainer on how to configure Unbound with Pi-hole so I won’t be repeating the steps here again. You can configure Pi-hole to forward accepted DNS queries from port 53</code> (standard) to 127.0.0.1#6363</code> (unbound). The Endgame#</a></h2> I plan to self-host a couple of more things. DNS is something I am really interested in and in future I plan to host my own DNScrypt server soon-ish. I believe if you own your data you’re in better control of your digital identity. Watching too much of Black Mirror added to the paranoia to an extent, I suppose! I grew up in the late nineties and I’ve seen the internet primarily as a set of the decentralized toolchain. There’s no reason we should let go any of that to the hands of a few corp giants and make it centralized. Ending this long-ish post by a beautiful quote: Study after study has shown that human behaviour changes when we know we’re being watched. Under observation, we act less free, which means we effectively are less free. ― Edward Snowden </blockquote> Cheers! :) Common Docker Mistakes - Episode 1 2019-04-15T10:57:55+05:30 So, off late, I’ve been dabbling a lot with Docker to explore the world of containerization (too late to the party, eh?). I plan to write about some common docker gotchas. The plan is to document these learnings so they might help someone getting started with docker and also serve as a reference for myself in the future. Let me be clear, if you read the docs you will find the exact same information and there’s nothing new that I have discovered. It took me some time to get around the following issues and I believe some of you might be struggling with the same. I just feel when you’re starting out with completely new technology, things quickly can become overwhelming and it’s A-OK to feel so. The important part is to not get intimidated by it and focus on learning the basics. Different pieces start coming together and there you have a solved puzzle :) The mysterious case of bind mounts and volumes#</a></h2> Ah! Storage. It’s always not a rosy scenario whenever someone mentions storage and containers in the same sentence. Anyway, so I had this requirement where I needed 2 containers to share data. Either of the containers could modify this data, so it made for a strong use case for volumes. But for some strange reason, I decided to use bind mounts. My thought process was that I’ll bind the mount path of the host to container and both of them could share the data. Now, I know, I know all the docker veterans already facepalming so hard, but in case anyone new to docker is reading it, it works the exact way I described. The host path will be mounted ON the container, so if your host path is empty, so will your container be. It took me quite some time to figure this out because of the side effect of it. I had this line in my docker-compose: volumes: - type: bind source: /etc/custom/data target: /</code></pre> As you can guess, I am mounting an empty folder etc/custom/data</code> on the root directory of the container /</code>. This was an nginx</code> container and I got the weird error that nginx</code> executable isn’t found. It became clear that I have obviously done something wrong. After reading the documentation, it became clear that I had to use something like Named Volumes</a> and use the same volume label for both the containers. Here’s the correct docker compose example (I have removed the unnecessary fluff and only included the volumes part): nginx: volumes: - type: volume source: assets-vol target: /usr/share/nginx/frontend frontend: volumes: - type: volume source: assets-vol target: /frontend/dist volumes: assets-vol:</code></pre> assets-vol</code> is a named volume and can be managed using docker API. CMD vs RUN#</a></h2> Now, this is particularly interesting. So I have a volume mount as shown in the previous example, and quite naively I am copying some files from container 1 to container 2 at build stage. I get the error that this path doesn’t exist. I am seriously reconsidering my life decisions right now. RUN cp /frontend/dist /usr/share/nginx/frontend</code> And then it became apparent that the volume is only mounted while the container is running, not when it is building. So I had to use CMD. CMD cp /frontend/dist /usr/share/nginx/frontend</code> Why RUN</code> doesn’t work and CMD</code> works, you might ask? This is because that’s how docker volumes are in nature. They bypass the unionFS (which is used to build docker images). UnionFS chain together a layer of images and build new images on top. Whenever you run a container from an image, a new layer is created for the container process. If you specify a path as a volume, this path doesn’t get committed</code> to the container data layer and is bypassed. So, TL;DR, volumes are really only accessible when the container is running and you can’t access them while building. Epilogue#</a></h3> I plan to share more such silly mistakes of mine while exploring more of Docker (and hopefully running production workloads on it soon!). It really is fun though, believe me :) Setting up Kong API Gateway - Part 2/2 2019-03-23T11:57:55+05:30 Managing Kong using UI#</a></h1> Konga</a> is an unofficial project which is basically an UI for Admin API of Kong. This post is about how to setup Konga and configuring an upstream service. We will also add an authentication layer to our upstream. Docker to the rescue#</a></h2> We will use Docker to quickly setup Konga as it a Node.js project and it is a lot of trouble building node projects on server, so let’s just use Docker. Our docker-compose.yml</code> for reference: version: "3.1" services: db: image: postgres restart: always ports: - 5432:5432 environment: POSTGRES_PASSWORD: "<redacted>" volumes: - /home/ubuntu/docker/volumes/postgresql/:/var/lib/postgresql/data networks: - dockergalaxy app: image: pantsel/konga:latest restart: always ports: - 1337:1337 environment: DB_URI: "postgresql://user_redacted:pass_redacted@db:5432/db_redacted" DB_ADAPTER: "postgres" networks: - dockergalaxy nginx: image: nginx:latest ports: - 80:80 - 443:443 volumes: - /home/ubuntu/docker/volumes/nginx/conf/:/etc/nginx/conf.d - /home/ubuntu/docker/volumes/nginx/ssl/:/etc/ssl/certs/konga/ networks: - dockergalaxy networks: dockergalaxy:</code></pre>docker-compose pull docker-compose up -d</code></pre> Verify, if all 3 containers are up. $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5f20abcd81fc nginx:latest "nginx -g 'daemon of…" 29 hours ago Up 28 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp deployment_nginx_1 ff20abcd24a66 pantsel/konga:latest "/app/start.sh" 30 hours ago Up 29 hours 0.0.0.0:1337->1337/tcp deployment_app_1 5848abcd27d2a postgres "docker-entrypoint.s…" 30 hours ago Up 29 hours 0.0.0.0:5432->5432/tcp deployment_db_1 </code></pre> Konga is running on port 1337</code> and you can verify the same by doing a curl</code>: curl http://localhost:1337 # should return HTTP 200</code></pre>Adding an Upstream to Kong#</a></h2> Visit Konga Admin Dashboard to login. If this is a first time login, you need to activate the connection to Kong’s Admin API by visiting Connections tab. This step needs to be done by each admin user, individually. Some common terminology before we begin setting up our APIs: (Source</a>) Term</th> Description</th></tr></thead> client</code></td> Refers to the downstream client making requests to Kong’s proxy port.</td></tr> upstream service</code></td> Refers to your own API/service sitting behind Kong, to which client requests are forwarded.</td></tr> Service</code></td> Service are abstraction of each of your own upstream services.</td></tr> Route</code></td> Routes are entrypoints into Kong, and defining rules for a request to be matched, and routed to a given Service.</td></tr> Plugin</code></td> This refers to Kong “plugins”, which are pieces of business logic that run in the proxying lifecycle. Plugins can be configured for an individual route, or a service or globally. An example of plugin which we use is Key Auth</code> plugin for authentication.</td></tr> </tbody></table> Adding New Service#</a></h2> Visit the Services section and click on Add new Service. Enter the following details for your upstream service here. You can refer to the below screenshot for reference: Field</th> Value</th></tr></thead> Name</code></td> Add a unique name for your upstream service.</td></tr> Description</code></td> Service description.</td></tr> Tags</code></td> List of tags to identify a group of services together. Press ENTER for any kind of array values in Konga UI.</td></tr> URL</code></td> Shorthand for setting Host</code>, Path</code>, Protocol</code> with just one value. Note that this is only a feature in Konga, Kong doesn’t have it, when using it with the Admin API calls directly.</td></tr> </tbody></table> Verify the details once and click on Save. Next, we’ll see how to add routes. Adding Routes to Services#</a></h2> Visit the Services section and click on the service entity you just created. Go to the routes section and add the a new route entry for the service. You can refer to the below screenshot for reference: Field</th> Value</th></tr></thead> Name</code></td> Add a unique name for your route.</td></tr> Hosts</code></td> Kong checks for the hostname present in the incoming request’s header. If you specify this value then the Hostname must be present for Kong to match the request to this route. This is suitable only if you want to block any request made outside this hostname. You can leave it null if not needed.</td></tr> Path</code></td> List of paths present in incoming request. This is required to namespace the upstream endpoints. The client must send this prefix in the request, Kong will try to match any request’s path in this list of paths and based on the settint of strip_path</code> the request will be proxied.</td></tr> Strip Path</code></td> Boolean value, which configures Kong to strip the matching path from the incoming request to the upstream URL.</td></tr> </tbody></table> How routing actually works#</a></h3> Kong has the ability to configure really complex routing endpoints based on your usecases. For the simplicity and keeping this guide as generic, the basic (but most common) use case is discussed below by taking an example. Let’s say your upstream URL(service</code>) is http://jsonplaceholder.typicode.com/</code>. To setup Kong for this service, we will simply add a route, with the path as /fake</code>. Here the path</code> acts as a namespace, to differentiate between different services. This can be helpful to avoid route collision if there are a lot of upstream services configured. Consider the upstream API endpoint is https://jsonplaceholder.typicode.com/todos/</code>. If we specify Kong to have the path</code> as /fake</code> and set the strip_path</code> as True</code> then our incoming request should look like https://kongapigateway.com/fake/todos</code>. Kong will try to match the path /fake</code> in this incoming request and look for the routes where the path is /fake</code>. Since it found the correct route, and we have set strip_path</code> to True</code>, Kong will just remove this particular path</code> prefix while reverse proxying to the upstream URL. In this way, our upstream doesn’t need to be concerned about the path</code> prefix as well. Adding Consumers#</a></h2> Step 1: Visit the Consumers section to add consumers for your API. Here the consumers doesn’t really mean 1:1 users, it could be a particular production service wanting to consume another service’s API. </li> Step 2: Click on Add new consumer and enter the following details: </li> </ul> Step 3: Add the remaining details in the groups section. Visit the Credentials sections, click on API Keys</code> (since we are using Key Auth plugin for authentication) and simply click on Submit button, since Kong will auto generate the API key for you (which most likely will be more secure than any random key you will enter).</li> </ul> Step 4: Leave the other details as it is. We will add this consumer to our Services page. Visit the services tab, click on the service entity you created and go to Plugins section. Click on Add New Plugin and the select Key Auth from the list. </li> </ul> In the modal that opens up, you need to specify the consumer UUID which was created (or leave it as blank for all consumers to access). More on controlling the access, is present later in the next section. Step 5: This feels a bit out of the flow, but after checking out Kong docs, there’s no alternate for this. In the previous step you would have been wondering, what if you need multiple consumers for the service, and want to identify them individually. Since Kong requires you to give either the Consumer UUID or leave it blank so all consumers are tied, the way around this is to use another plugin, called ACL Plugin. With this plugin, you can whitelist a group of consumers, so only the whitelisted consumers can access the API even though you specified All Consumers in the Key Auth plugin. ACL Plugin must be used in conjuction with another authentication plugin.</li> </ul> You can refer to the below screenshot for reference: NOTE: This is the same group, which you created at the time of Consumer creation. You can group different consumers based on the service they consume, hence the naming convention one can follow is -consumers. You can see all the eligible consumers for the service, in Eligible Consumers tab. Using Kong as an end user#</a></h2> You need to replace your upstream API endpoints and all other custom authentication with just the Kong’s URL, the namespace for the service, and add the authentication keys in header while sending the request. For example: If your original request is: curl -i -X GET --url http://jsonplaceholder.typicode.com/todos/</code></pre> The modified request becomes: curl -i -X GET --url https://kongapigateway.com/fake/todos/ --header "X-ACCESS: ENTER_KEY_HERE"</code></pre>Accessing user authentication details in your API#</a></h3> You can remove all custom authentication method in your API services as Kong forwards the user information in the headers. You can use the following headers while processing the incoming request to identify the users and implement custom business logic: Header Key</th> Value</th></tr></thead> X-Consumer-ID</code></td> ID of the Consumer on Kong</td></tr> X-Consumer-Custom-ID</code></td> custom_id of the Consumer (if set)</td></tr> X-Consumer-Username</code></td> username of the Consumer (if set)</td></tr> X-Anonymous-Consumer</code></td> will be set to true when authentication failed, and the ‘anonymous’ consumer was set instead.</td></tr> X-Credential-Username</code></td> the username of the Credential (only if the consumer is not the ‘anonymous’ consumer)</td></tr> </tbody></table> The End#</a></h2> Hope you liked the two part series of setting up and managing your Kong cluster. Do reach out to me @mrkaran_</a> in case of any feedback for the post. Thanks! Setting up Kong API Gateway - Part 1/2 2019-03-23T10:57:55+05:30 Kong#</a></h1> Kong</a> is an API Gateway, which basically reverse proxies every incoming request to the upstream URL. It is pretty useful if you have a lot of internal services which need to talk to each other (through HTTP) but you don’t want to keep managing the authentication layer, rate limiting, hosts whitelisting and other such things in every service. Kong acts as a central entrypoint to all other services’ API endpoints and all the common fluff is heavylifted by Kong’s API layer. Kong follows a plugin</a> approach, which makes it extensible and you can even make your own plugins</a>. Using plugins, it is possible to modify the request, add authentication layer at the Kong layer, forward user meta information headers to the upstream. Amongst other API Gateway solutions, Kong is pretty straightforward to get started with and has a nice community support as well. All of the actions to configure your API endpoints and manage them can be done through a RESTful Admin API. The following setup guide describes how the infra is setup for High Availability of a cluster of Kong nodes. Infra Setup#</a></h2> Each individual Kong node is stateless, since it is always connected to an external datastore. In this tutorial, we will provision 2 nodes for Kong. The instances are frontended by Amazon’s ELB which routes the traffic internally to either of a Kong node using internal DNS. Kong requires a datastore to fetch the information about upstream APIs, consumers, routing mechanisms, plugins, so each of the Kong node must be in sync with the other Kong node. We will achieve the same by using Cassandra as our database for Kong, which is being run in a clustering mode. Cassandra uses it’s gossip mechanism to ensure the other Cassandra node is upto-date with any new changes to the data. Scaling Kong#</a></h3> To scale Kong in future, we can keep on adding multiple Kong nodes horizontally and attaching them to one of the Cassandra node. This way we can have multiple Kong nodes in one cluster each pointing to one central Cassandra datastore. Setting up the cluster#</a></h2> We will use Ansible to automate the task of setting up Kong+Cassandra in each of 2 nodes. You can refer to the playbook</a> which will do the job. --- # https://github.com/mr-karan/kong-ansible # Playbook to install the Cassandra and Kong - hosts: "{{control_host}}" remote_user: "{{control_user}}" become: yes roles: - role: java - role: cassandra - role: kong</code></pre> After you run the playbook, there are a couple of important things which needs to be configured in order to have an HA setup. This setup guide assumes the playbook is run individually on 2 servers: srvr A</code> and srvr B</code>. Important Directory Paths#</a></h3> Local location</th> Description</th></tr></thead> /usr/local/bin/kong</code></td> Kong executable binary</td></tr> /usr/local/kong</code></td> All the settings and logs are available under a namespaced directory, which will be called as PREFIX</code> in further sections.</td></tr> /etc/systemd/service/kong.service</code></td> Managing Kong as systemd service</td></tr> /etc/systemd/service/cassandra.service</code></td> Managing Cassandra db as systemd service</td></tr> /etc/cassandra/cassandra.yaml</code></td> Config for Cassandra</td></tr> /etc/kong/kong.conf</code></td> Config for Kong</td></tr> </tbody></table> First Steps#</a></h3> Setting up Cassandra#</a></h4> Let’s setup Cassandra first and run in clustering mode. Do these steps in both of the servers. Stop any running cassandra node: sudo service cassandra stop</code></pre></li> Edit the cassandra config file and update the following values: cluster_name: 'KongAPICluster' seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "<private_ip_srvrA>,<private_ip_srvrB>" listen_address: <private_ip_srvr> start_rpc: true</code></pre></li> Start cassandra on both the servers and check the status: sudo serice cassandra start</code></pre></li> Verify Cassandra clustering: sudo nodetool status # Give it a couple of seconds (30-45) for both nodes to warm up and discover each other.</code></pre> The output of above command should look like: $ sudo nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN <REDACTED> 467.26 KiB 256 100.0% <REDACTED> rack1 UN <REDACTED> 496.36 KiB 256 100.0% <REDACTED> rack1</code></pre></li> Troubleshooting Cassandra: cqlsh unable to connect to cassandra</code> server: cqlsh</code> has a known bug in some versions with Python2.7 where it cannot connect to the cassandra server. Do the following steps to fix: sudo pip install cassandra-driver export CQLSH_NO_BUNDLED=TRUE</code></pre></li> Unable to discover the other cassandra cluster: This usally happens because of network connectivity issues. Verify both nodes are able to talk to each other by running Cassandra in single cluster mode and then issue the following commands: # in srvrA netstat -lntvp | grep cassandra # should be present (port 9042 usually) # in srvrB, check similarly... # in srvrA telnet private_ip_srvrB 9042 # should connect # in srvrB, check similarly...</code></pre></li> </ul> </li> </ol> Setting up Kong#</a></h4> Let’s setup each Kong node in the cluster as following. Stop any running kong instance: sudo service kong stop</code></pre></li> Edit the kong config file and update the following values: ... admin_listen = <private_ip_srvrA>:8001, <private_ip_srvrA>:8444 ssl database = cassandra db_update_propagation = 10 #seconds ...</code></pre></li> Start kong on both the servers and check the status: sudo serice kong start</code></pre></li> Verify if Kong is running: sudo service kong status</code></pre></li> Run Kong Migrations: Run the migrations on only one cluster. Since the datastores will be in sync(eventual consistency thanks to Cassandra), we don’t have to run migrations on the second cluster. kong migrations -c /etc/path/to/config</code></pre></li> Troubleshooting Kong: Check if Kong is actually running by sudo service kong status</code>. You can check for logs in $PREFIX/logs/error.log</code>.</li> If Kong is not running, you can try running kong check</code> for checking if the config file is correct. Kong additionally provides a health check command, which can be executed using kong health</code>.</li> </ul> </li> </ol> Managing Kong#</a></h2> Kong comes with Admin API</a> to manage all aspects of Kong. There is also an unofficial project which is aN UI on top of Kong’s API and it comes pretty handy to configure upstream endpoints, adding plugins etc. To read more about it, you can continue reading the second part of the series here</a> Adding Prometheus configuration to your CI/CD workflow 2019-03-17T10:57:55+05:30 Prometheus</a> configurations can turn into a mess in no time if you have a lot of different jobs scraping different targets. Certainly you can use tools like jsonnet</a> to keep your YAML</code> files DRY but this post is not about that. I initially started off by writing one job to scrape a set of EC2 instances in a particular AWS VPC. Over the time, I had a requirement to do it over 3-4 different targets, each of them had their own rules and different type of exporters as well. The whole practice of SSH</code>ing into the server, opening the config file in vim</code> and editing in the server didn’t just feel right. No surpises for guessing that I f*cked up a few times with config errors (YAML sigh) and finally decided that I need a better solution for this. CI/CD for config files#</a></h2> “Wait, so now you’re telling me that I need to setup a full fledged pipeline just for a few config files?”, must have been the thought that echo’ed in your head. Before you jump to the conclusion that it’s overdoing things, IMHO it’s not. And it’s not that difficult either to set it up, so why not? After integrating CI in your workflow, you can be confident that no bad syntax in your PromQL queries or in general the YAML</code> syntax would break your monitoring system. “That sounds amazing, show me teh code already”. Our Pipeline is fairly fairly simple: Lint the code using promtool</a>.</li> Push to S3</li> Write a shell script to pull from S3, put the config in right places and restarts Prometheus systemd service</li> </ul> You can automate the last step too, but I just wanted to keep atleast one manual check in this system so I decided against it. And since I don’t have a distributed Prometheus setup yet, it’s simple to keep things the old school way here . You can do pretty much all of the steps in the above pipeline of any CI of your choice, but I am using Gitlab</a>, so here’s a sample .gitlab-ci.yml</code> file: stages: - lint - deploy prometheus-lint: stage: lint # run this job on stage lint image: golang:1.11-alpine # pull go1.11 image from official docker repo before_script: - apk update && apk add git # install git script: - GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go get -u github.com/prometheus/prometheus/cmd/promtool # fetches promtool package - $GOPATH/bin/promtool check rules rules/alerts/* # run the `check rules` command for my prometheus rule files push-to-s3: stage: deploy # run this job on stage deploy environment: name: production # tag the job metadata in this environment. useful to quickly revert deploys when shit hits the fan image: python:3.7-alpine script: - pip install awscli # adds aws cli tools - aws s3 sync prometheus/rules s3://mybucket/rules/ # pushes our rule config files to s3 only: refs: # only allow deploy if branch is master. - master when: manual # trigger this job manually</code></pre> A brief explanation for the above file: stages: Used to define multiple stages in our pipeline and they are executed in order.</li> jobs: prometheus-lint</code> is one job which will be run in the stage lint</code>. I am using Docker executor with Gitlab runner, so the runner agent talks to the docker executor. Gitlab-CI file is basically an API abstracted to hide away these details from the end user so all the CI/CD files look pretty much the same but behind the scenes the way they are executed totally depends on the executor you choose. Since we are using Docker executor, the image</code> tag is picked up and the same docker image is pulled from a public docker repo for this job.</li> before_script</code> is basically an event which is called before we begin running our actual CI stuff. You can add your project dependencies here.</li> script</code> is a list of commands to be executed inside your environment (container for us).</li> </ul> Now that we have our basics about the CI/CD pipeline in place, let us see what each of the jobs is actually doing: prometheus-lint</code>: Installs promtool</code> binary and runs it against our rules/</code> folder. The default working directory is our repository itself, so we didn’t have to give the absolute path. </li> push-to-s3</code>: Installs aws</code> cli tools, so we can push the files to s3. In case you are wondering where are the access key and secret key present, I have added them as protected variables in my project settings, so only the protected branches (eg master</code>) can access them. </li> </ul> Automate everything? Nah mate#</a></h3> This is how the deploy.sh</code> script looks like: #!/bin/sh aws s3 sync s3://mybucket/rules/ /etc/prometheus/config/rules/ promtool check config /etc/prometheus/prometheus.yml sudo service prometheus restart</code></pre> As much as I’d like to be a cool hipster and run /deploy</code> from my slack bot (last I checked, there’s a legit term for this: ChatOps), I simply don’t prefer that. Having a human intervention before doing critical deployments like this is OK, IMHO. I don’t update the Prometheus config often, so I don’t mind actually SSH-ing into the single instance and triggering a shell script which does the job for me. I also don’t have a distributed Prometheus setup as of yet. Things will definitely change based on your requirements and there’s no one size that fits all. I’m much more confident now with my config changes and don’t have to pray to the server overlords everytime I restart Prometheus. It just works . Makefile for Golang projects 2018-10-11T18:10:55+05:30 Makefile is an awesome tool to group together a bunch of different rules and automate your build process. Makefile is used by make</code> which is essentially a file generator tool. It is generally used to compile and build programs from source by following rules</code> listed in the Makefile. People use Makefile for a lot of different purposes as well, for example converting md</code> to html</code> and publish these files to the web server. Every makefile you see, is composed of rules. A rule</code> is declaration of a target</code> and the commands to be executed to generate the target. A target can be a file or an action to be performed (more on that later). This is how a rule looks like in a Makefile target: dependencies recipe</code></pre> When you run make target</code>, make</code> searches for the rule which begins with this target</code> and executes the dependecies (if required). It then runs a bunch of commands which are listed in the recipe. An important thing to understand here is that make</code> tracks the dependencies by their last modified time. So if the dependencies haven’t changed, then make will complain with make: 'target' is up to date.</code> Enough of theory, let’s get our feet wet by writing our first Makefile. One important thing about Makefile</code> is that you need to use tabs</code> and not spaces</code>. It is one of the rare *nix</code> programs which is whitespace aware and this has been mentioned in The Unix-Haters Handbook</a> as well. To begin with, let’s write a simple rule which removes any tempory object files using go clean</code> and previous binary file using good ol rm</code>: clean: go clean rm -f sample.bin</code></pre> The target</code> here is clean</code>. There is something special going on here though. Imagine we have a file called clean</code> in our source directory? Let us try to run make clean</code> now Our directory structure:. ├── Makefile ├── clean └── sample.bin</code></pre> On running make clean</code>: make: `clean' is up to date.</code></pre> Every target in Makefile by default is a file target</code>. In our case clean</code> is a file target and make</code> tries to build this file clean</code> but since we already have a file with the same name clean</code> in our directory, make</code> is complaining there’s nothing to do. Moreover, in this case our rule is more of an action</code> rather than building files. So for all such scenarios, make</code> provides an easy way where we can instruct it to just run the rule and ignore any filename in our directory. This is called a PHONY</code> target which is a special kind of target. PHONY</code> is just a way in make</code> to forcefully run a target and not care about generating files. Our aim with this rule is to run 2 commands and that’s about it. So this is the perfect example for using PHONY</code>. We can add .PHONY</code> target to our Makefile simply by this line: .PHONY : clean clean: go clean rm -f sample.bin </code></pre> Now when we run make clean</code> we get our expected output go clean rm -f sample.bin</code></pre> Let us extend our Makefile to do some common tasks: .PHONY : build run fresh test clean test: go test build: go build run: ./sample.bin clean: go clean rm -f sample.bin</code></pre> If you have worked on any Golang project, these are very trivial actions on any Golang project. You will soon realise the power of Makefile when you have to do these steps repeatedly. Some people might argue then you can use aliases or simple shell scripts for the same. I vehemently disagree with that. Reason being, make</code> is a much more powerful tool than just running commands. make</code> has support for dependency tracking and it will only rebuild whatever is required. If you are working on a huge project where the build times are to the tune of hours, you will soon realise why shell scripts are inferior. Ofcourse, someone can point that they can write a shell script to do even that, by fetching the last modified time but why do the extra work when there’s an already existing tried and tested tool? make</code> also has support for parallel task execution, so you can just pass the flag -j {num}</code> to make</code> and it will run these {num} jobs parallely. All these benefits will be apparent for larger projects, but it is a good habit to write Makefile even for smaller projects. We will now make our Makefile a bit more sophisticated and introduce variables. If you want to custom name your binary, or inject variables at compile time</a>, you can declare these variables, for example: BIN := my-awesome-pro.bin HASH := $(shell git rev-parse --short HEAD) COMMIT_DATE := $(shell git show -s --format=%ci ${HASH}) BUILD_DATE := $(shell date '+%Y-%m-%d %H:%M:%S') VERSION := ${HASH} (${COMMIT_DATE})</code></pre> We can modify our Makefile to use these variables: build: go build -o ${BIN} -ldflags="-X 'main.buildVersion=${VERSION}' -X 'main.buildDate=${BUILD_DATE}'" run: ./${BIN} test: go test clean: go clean rm -f ${BIN}</code></pre> We can auto version our builds and pass variables during the build time with go linker tool, on passing the -X</code> flag. That’s really neat, now whenever we do a make build</code> we get new version of the build automagically. So now we have a working Makefile which helps us with trivial things, but everytime if we need to change something in our program and check, we still need to do these steps manually: make clean</code>, make build</code> and make run</code>. Won’t it be awesome if we could tell Makefile to do all this with just one command? Programmers are lazy creatures after all . In the beginning we saw a target is composed of recipe and dependencies. So we can just create a new PHONY</code> target with all these dependencies and any recipe if we want optionally. fresh: clean build run</code></pre> We created a new target</code> which depends on clean</code> to run first, then build</code> and finally run</code>. So everytime if we make some change in our Go program, all we need to run is make fresh</code>. Awesome, isn’t it? We will finally add our last target which is a highly opinionated way of generating binaries for different OS and architectures. prod: goreleaser --rm-dist --snapshot cp dist/linux_amd64/${BIN}-linux.bin . rm -rf dist</code></pre> This target runs goreleaser</code> which is a build automation tool. It then copies the required linux binary to the source directory and removes all the other junk. You can even extend your Makefile to commit files to a repo, and rsync these binaries to the production server or initiate your CI/CD build process. The reason I like Makefile is because it serves as a living documentation for your project on how to build/deploy the project making it easier for new contributors to get started. Some additional information#</a></h5> If you run make</code> without passing any target name, make</code> will run the first target present in the Makefile. To override this, you should set .DEFAULT_GOAL</code> setting and override the target</a> which you want to make as default. </li> .PHONY</code> is just one way to tell make</code> that it is a special kind of target, you can also do the same by creating a target without any recipe. Read this</a> to know more. </li> </ul> To know more about Makefile</code>, you can read the manual here</a> I hope you now appreciate Makefile</code> and try it out in your next project. I’d love feedback on this blog post, do reach me out at twitter</a> or email</a> Fin! Bi-Weekly Music Post Part 4 2018-07-28T21:10:55+05:30 I am writing this post exactly after 30 days of the last one, caught up with some work and I couldn’t write a new post, apologies! To make up for the delay, I have some absolutely gold reccos this time which I believe you will like it. Isaac Gracie - reverie#</a></h3> </a> Tash Sultana - Jungle#</a></h3> </a> 18th Dec - Tienas#</a></h3> </a> Pareek - Ariana and Amrina, Coke Studio Explorer 2018#</a></h3> </a> When The Curtain Falls - Greta Van Fleet#</a></h3> </a> Poppy Ackroyd - Paper#</a></h3> </a> Nothing Personal - Night Riots#</a></h3> </a> Do reach out to me @mrkaran_</a> with your dose of music reccos. Fin! Bi-Weekly Music Post Part 3 2018-06-28T21:30:55+05:30 Since the past one week, I’m having troubles in connecting to Spotify from India and my #1 source for discovering music is gone. I’ve started using Saavn</code> but it has crappy recommendations and too much of “Bollywoody”/crap shit thrown all over. Also, it requires location access to suggest new music. LOL is this a joke or what? Like seriously, can’t think of a single reason why a ML algo would require my location to suggest me music. Anyway, this week’s recommendation might be a bit more “Desi” as I’ve discovered some real good bands from our country and our beloved neighbours(ofcourse!) Mist of Capricorn - Agam#</a></h3> </a> Rangapura Vihaara - Agam#</a></h3> </a> Toh Phir Aao - Levi’s Live Session 7 by Mustafa Zahid & ROXEN#</a></h3> </a> Aankhon Kay Sagar - Shafqat Amanat Ali, Coke Studio Pakistan, Season 2#</a></h3> </a> Hit Me Up - The PropheC#</a></h3> </a> Tareefan Reprise - Lisa Mishra#</a></h3> </a> Message In A Bottle - The Police#</a></h3> </a> Agnes - Glass Animals#</a></h3> </a> Psycho Killer - Talking Heads#</a></h3> </a> Do reach out to me @mrkaran_</a> with your dose of music reccos. Fin! Bi-Weekly Music Post Part 2 2018-06-14T12:30:55+05:30 Ive looped through Nescafe Basement like 100 times the past two weeks and still cant get enough of it. This is the kind of music I like, the kind that just grows on you until you loop endlessly and then hate it Anyway, the following is a list of songs that I have discovered and liked the most in the past two weeks. Tattva - aswekeepsearching#</a></h3> </a> Koothu Over Coffee - Agam#</a></h3> </a> Tere Ishq Mein Jo Bhi - Nescafe Basement#</a></h3> </a> Houndmouth - Sedona#</a></h3> </a> Heavy - Collective Soul#</a></h3> </a> The Youth - George Taylor#</a></h3> </a> Porcupine - Pigeons Playing Ping Pong#</a></h3> </a> Misirlou - Ashar Kazi & Shruti Naik (cover)#</a></h3> </a> Redbone - Childish Gambino (ukulele cover)#</a></h3> </a> Do reach out to me @mrkaran_</a> with your dose of music reccos. Fin! Receiving notifications from Supervisor 2018-06-03T10:57:55+05:30 Supervisor Events#</a></h3> I had a seemingly simple task which was to receive notifications any time a process managed by Supervisor restarts. I wanted a generic solution where I could get notifications for any change in the process state. Supervisor Events</code> saved my day, although I would admit it wasn’t straightforward to set up. Supervisor uses STDIN/STDOUT</code> mechanism to communicate with the event listener. You need to configure your event listener in such a way that it can understand the STDIN</code> sent by Supervisor and also communicate back using STDOUT</code>. You can write this event handler in any language you like as long as you conform to the specially formatted messages that Supervisor sends and expects. I had struggled the most at this step and my google-fu</code> didn’t help much in this case. Supervisor by default will send these events even if no listener is configured. Once you have your own listener setup, you can execute any task you want, eg: send email/telegram/slack messages etc. In order to configure your event listener, you need to add it to your supervisor.conf</code>. Here’s an example configuration: [eventlistener:wowevent] command=/home/work/testevent/test.py events=PROCESS_STATE_STARTING process_name=%(program_name)s_%(process_num)s numprocs=1 autorestart=true stderr_logfile=/home/work/testevent/logs/event_err.log stdout_logfile=/home/work/testevent/logs/event.log</code></pre> For the program to know that it has to send a notification at wowevent</code> pool, you need to add the events</code> key to the program</code> section of supervisor.conf</code>. [program:myprog] ... events=PROCESS_STATE_STARTING ...</code></pre> Now everytime myprog</code> is about to start, it will send an event to wowevent</code> event pool. Your event listener which is configured at /home/work/testevent/test.py</code> will handle the notification and execute tasks which you want to perform. There are a bunch of event states that Supervisor captures, I was interested in knowing when my process has started, so I used PROCESS_STATE_RUNNING</code>. You can take a look at all different event types here</a>. During all this experimentation I came across a bug (which I later found it is a known issue</a>, and an open bug since 4 years now). If you’ve been using Supervisor I am sure at least once you’ve been bitten by not rereading</code> the config file and wondering why Supervisor isn’t picking up changes in config file when you restart the Supervisor. So reread</code> and update</code> becomes a muscle memory after this event . The bug with events is that if you make any changes to the event group, reread</code> doesn’t pick up this change. I initially thought this must be the standard way Supervisor is behaving because I didn’t change any program</code> group. I randomly decided to change the event listener name and BAM! Supervisor read the new configuration and everything suddenly works! ARGH. Why do complex problems have such simple solutions (not a solution, rather a workaround, but you get the drift right?) Supervisor Event Listener Protocol#</a></h3> Supervisor sends a header which is a key value pair of meta-attributes about the process and event. This header looks something like ver:3.0 server:supervisor serial:208 pool:mylistener poolserial:0 eventname:PROCESS_STATE_RUNNING len:69</code></pre> The event listener mylistener</code> will be in ACKNOWLEDGED</code> state when Supervisor sends the event PROCESS_STATE_RUNNING</code>. Now that the myslistener</code> state has received this ACKNOWLEDGED</code> state, the event listener will send READY</code> back to Supervisor. This is to let Supervisor know that the listener has received the notification. Supervisor puts the listener to BUSY</code> state now and here you can do write your custom task. Supervisor waits for the task to get executed and when it does, the listener needs to communicate back with the result. This process is one full request-response cycle. Let us write a simple Python script which will listen to Supervisor event notification and communicate back, all in a protocol Supervisor understands. import sys import requests import logging logger = logging.getLogger('event_listener') handler = logging.FileHandler('/home/user/prod/events/logs/response.log') formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) logger.setLevel(logging.DEBUG)</code></pre> Here I am just setting up basic logging structure since I want to keep the stdout</code> messages separate from what Supervisor uses. def write_stdout(s): # only eventlistener protocol messages may be sent to stdout sys.stdout.write(s) sys.stdout.flush() def write_stderr(s): sys.stderr.write(s) sys.stderr.flush()</code></pre> We have written helper functions which will be used to communicate with Supervisor. Now comes the main</code> part (Sorry for bad pun! ) def main(): while 1: # Hey Supervisor, I'm ready for some action write_stdout('READY\n') # Reading the header from STDIN line = sys.stdin.readline() write_stderr(line) # read event payload and print it to stderr headers = dict([ x.split(':') for x in line.split() ]) data = sys.stdin.read(int(headers['len'])) write_stderr(data) # add your events here notify_user() # transition from READY to ACKNOWLEDGED write_stdout('RESULT 2\nOK') logger.debug("It's all fine and dandy") if __name__ == '__main__': main()</code></pre> Let us break this into pieces. write_stdout('READY\n')</code></pre> We flush READY</code> with a linefeed character (\n</code>) to STDOUT</code>. Supervisor has put mylistener to BUSY</code> state now. line = sys.stdin.readline() write_stderr(line)</code></pre> line</code> would be the header which we discussed previously. data = sys.stdin.read(int(headers['len']))</code></pre> This part is interesting. Here, we capture the len</code> key from the header and read the next STDIN</code> line up to this many chars. The data</code> would consist of our event payload. Event payload looks something like: processname:prog-restartv3_0 groupname:prog-restartv3 from_state:STOPPED tries:0ver:3.0 server:supervisor serial:25 pool:prog-restartv3 poolserial:3 eventname:PROCESS_STATE_STARTING len:76`</code></pre>notify_user()</code></pre> This is the handler where you can send an email, send a request to API, log it to file etc. write_stdout('RESULT 2\nOK')</code></pre> Finally, we tell Supervisor to put the listener from BUSY</code> to ACKNOWLEDGED</code> state, by sending a result structure. The result could be FAIL</code> or OK</code>, so you need to send RESULT</code> followed by the length of the state variable. For example for OK</code> you will send RESULT 2\nOK</code> but for FAIL</code> you have to send RESULT 4\nFAIL</code>. That’s pretty much all you need to start receiving notifications from Supervisor every time your program changes its state. If you found this article useful, I’d love if you share this on Twitter</a> or Facebook</a> and let your friends know about it too.

Karan Sharma - Posts

Building an LLM-Assisted Incident Investigation Tool

A Web Terminal for My Homelab with ttyd + tmux

Architecture#</a></h2> Browser -> Caddy -> ttyd -> nsenter -> su - karan -> tmux(main)</span></span></code></pre> Two decisions matter most:</p> ttyd</code> handles terminal-over-websocket behavior well.</li>

tmux profile for agentic workflows#</a></h2> I tuned tmux for long-running agent sessions, not just manual shell use.</p> Long-run defaults#</a></h3> history-limit 200000</code></li> remain-on-exit on</code></li> window-size latest</code></li>

Long-run defaults#</a></h3> history-limit 200000</code></li> remain-on-exit on</code></li> window-size latest</code></li>

Better operational visibility#</a></h3> status line shows host + session + path + time</li> pane border shows pane number + current command</li>

Copy/paste that is not annoying#</a></h3> This was a big pain point, so I added both workflows:</p> Browser-native copy</strong></p>

Setting Up LiteLLM with AWS Bedrock

Authentication#</a></h2> This is where the first gotcha lives. Bedrock doesn’t use API keys. It authenticates via standard AWS credentials:</p> Environment variables: AWS_ACCESS_KEY_ID</code> + AWS_SECRET_ACCESS_KEY</code></li> AWS profile: AWS_PROFILE</code></li>

Why Plain-Text Ledger is Powerful for Gullak

CLIs are the New AI Interfaces

Logchef v1.0: The Journey to a Real Log Viewer

What’s Not in 1.0#</a></h2> Some things didn’t make the cut:</p> Live tail</strong>: Streaming logs in real-time. Still on the roadmap.</li>

Fixing a CIBIL Score Disaster with AI

AI and Home-Cooked Software

State of My Homelab 2025

TIL: WireGuard's Misleading "No Route to Host" Error

Announcing Logchef

Stack#</a></h2> Backend: Written in Go for performance and concurrency.</li> Metadata Storage: Uses SQLite for lightweight management of users, teams, Clickhouse source connections, and query collections. It’s simple and perfectly suited for this kind of a metadata store.</li>

Trying out NixOS

Automating Badminton Game Alerts

Cleaning up Notes with LLM

2024: A Year In Review

Life#</a></h2> Got married to the prettiest and dearest Saumya 💗</li> Did my first international trip, exploring Europe</li> Bought a fun toy - Maruti Jimny 4x4</li> Relocated to Bangalore after working from home for 4+ years since Covid</li>

Travel#</a></h2> Switzerland</li> Italy</li> Austria</li> Binsar</li> Cochin</li> Ajmer</li> Ranthambore</li> Pondicherry</li>

How I use LLMs

Self Hosting Outline Wiki

Building an expense tracker app