<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://davidlow.ai/feed.xml" rel="self" type="application/atom+xml" /><link href="https://davidlow.ai/" rel="alternate" type="text/html" /><updated>2026-11-16T14:53:55+00:00</updated><id>https://davidlow.ai/feed.xml</id><title type="html">David Low</title><subtitle>David Low, Head of Technology at Vertex Holdings and fractional Chief AI Officer at HeyHi. Notes on building AI that gets used.</subtitle><entry><title type="html">From Tab Completion to 100% Code Written by Agents</title><link href="https://davidlow.ai/from-tab-completion-to-agent-default/" rel="alternate" type="text/html" title="From Tab Completion to 100% Code Written by Agents" /><published>2026-03-02T13:02:00+00:00</published><updated>2026-03-02T13:02:00+00:00</updated><id>https://davidlow.ai/from-tab-completion-to-agent-default</id><content type="html" xml:base="https://davidlow.ai/from-tab-completion-to-agent-default/"><![CDATA[<p>From tab completion to 100% code written by agents.</p>

<p>That escalated quickly.</p>

<p>I still remember the first time I used GitHub Copilot in 2021.
You typed a comment, hit tab, and half the function appeared. It felt futuristic.</p>

<p>Fast forward a few years.</p>

<p>Now we are talking about workflows where agents plan, write, test, refactor, and iterate on code with minimal supervision. Not suggestions. Not snippets. Entire task ownership.</p>

<p>Recently, Cursor cofounder Michael Truell shared data showing how agent usage has overtaken simple tab completion in product usage metrics.</p>

<p>That chart is telling.</p>

<p><img src="/assets/images/posts/agents-default-chart.jpg" alt="Agents requests per tab accept trend, Jan 2025 to Jan 2026" /></p>

<p><em>Agent requests per tab accept rising through 2025 and crossing 1x by Jan 2026 — the point where agent workflows overtake tab completion.</em></p>

<p>It shows agent requests per tab accept climbing steadily through 2025, eventually crossing 1x. In other words, developers are triggering full agent workflows more often than accepting inline completions.</p>

<p>That is a profound behavioural shift.</p>

<p>Tab completion augments the developer.
Agents assume responsibility for chunks of the workflow.</p>

<p>This is not just a UI evolution. It is a transfer of execution.</p>

<p>If agents become the default way code is produced, several things follow:</p>

<ol>
  <li>
    <p><strong>Architecture matters more than syntax.</strong>
You are no longer optimising keystrokes. You are designing systems agents can operate safely within.</p>
  </li>
  <li>
    <p><strong>Control becomes strategic.</strong>
Who reviews? Who authorises? What guardrails exist? When an agent can touch multiple files and refactor broadly, governance is not optional.</p>
  </li>
  <li>
    <p><strong>Headcount is no longer a clean proxy for output.</strong>
If agents handle more routine engineering surface area, the leverage per developer increases significantly.</p>
  </li>
</ol>

<p>We are moving from human-written code with AI assistance to human-supervised systems with AI execution.</p>

<p>That is a different era.</p>

<p>Copilot showed us that AI could complete our thoughts.
Agent workflows are starting to complete our tasks.</p>

<p>The real leadership question is not whether this works.
The data suggests it already does.</p>

<p>The question is how much execution authority we are prepared to delegate, and how we intend to control it.</p>

<p>The shift from tab completion to agent default may look incremental on a chart.</p>

<p>It is not incremental inside the organisation.</p>

<hr />

<p>Originally posted on LinkedIn:
<a href="https://www.linkedin.com/posts/davidlowjw_from-tab-completion-to-100-code-written-share-7434219643118034944-Lo-9">https://www.linkedin.com/posts/davidlowjw_from-tab-completion-to-100-code-written-share-7434219643118034944-Lo-9</a></p>]]></content><author><name></name></author><category term="blog" /><category term="ai" /><category term="software-engineering" /><category term="agents" /><category term="developer-tools" /><summary type="html"><![CDATA[A shift from inline AI assistance to full agent-driven execution in software engineering.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://davidlow.ai/assets/img/v2/og.jpg" /><media:content medium="image" url="https://davidlow.ai/assets/img/v2/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">My AI Server Froze and I Couldn’t Tell You Why – Here’s What I Fixed</title><link href="https://davidlow.ai/my-ai-server-froze-and-i-couldnt-tell-you-why/" rel="alternate" type="text/html" title="My AI Server Froze and I Couldn’t Tell You Why – Here’s What I Fixed" /><published>2026-03-01T12:00:00+00:00</published><updated>2026-03-01T12:00:00+00:00</updated><id>https://davidlow.ai/my-ai-server-froze-and-i-couldnt-tell-you-why</id><content type="html" xml:base="https://davidlow.ai/my-ai-server-froze-and-i-couldnt-tell-you-why/"><![CDATA[<p>Recently, my home AI inference server – a GMKTEC EVO X2 AI Mini PC (AMD Ryzen AI MAX+ 395 with 128 GB unified memory) – silently froze after 49 hours of uptime. No warning. No crash dump. No kernel panic. The system journal, which faithfully records every event on a Linux machine, just… stopped. One moment it was logging a routine data collection. Twenty minutes later, I was standing at the machine holding the power button.</p>

<p>SSH was dead. The network was unreachable. My first instinct was to blame a network outage or maybe overheating. But when I sat down to investigate after the hard reboot, neither turned out to be true. Temperature had been a cool 37°C. Memory was only 15% utilized. There was no out-of-memory event, no thermal throttle, nothing obvious.</p>

<p>What ultimately cracked the case was <code class="language-plaintext highlighter-rouge">sysstat</code> – a system monitoring tool that had been quietly running in the background, recording CPU and memory snapshots every 10 minutes. Those snapshots were the <strong>only</strong> forensic data that survived the freeze. They let me rule out overheating and memory exhaustion, and narrow the root cause down to a kernel workqueue deadlock that had been progressively worsening over two days.</p>

<p>The diagnosis was a success. But it also exposed a gap: 10-minute intervals are coarse. The memory readings jumped straight from “everything’s fine” to “LINUX RESTART.” If I’d had 1-minute resolution, I might have caught the degradation pattern in real time – or at least had a much sharper picture of the minutes leading up to the hang.</p>

<p>So I fixed that. Here’s how, and why you should too.</p>

<hr />

<h2 id="why-this-matters-if-you-run-local-ai">Why This Matters If You Run Local AI</h2>

<p>More of us are running large language models on our own hardware. Whether you’re serving a model through Ollama, running inference with llama.cpp, or hosting a vLLM endpoint for your team, the appeal is clear: no API costs, full control, your data stays local.</p>

<p>But unlike a managed cloud GPU instance, nobody is monitoring your local server for you. There’s no CloudWatch dashboard, no PagerDuty alert, no auto-scaling safety net. When something goes wrong at 11 PM on a Tuesday, the first sign is usually “I can’t reach the machine anymore.”</p>

<p>AI inference workloads are also <strong>bursty</strong> in a way that traditional server loads aren’t. A large model loading into memory can consume 60 GB of RAM in seconds. A batch of concurrent requests can pin every CPU core while the model generates tokens. These spikes happen fast, and they resolve fast. A monitoring system that samples every 10 minutes will miss most of them entirely.</p>

<p>The GMKTEC EVO X2 runs an AMD Ryzen AI MAX+ 395 with 128 GB of unified LPDDR5X memory and an integrated Radeon 8060S GPU with 40 compute units. Thanks to AMD’s Variable Graphics Memory, the GPU can dynamically claim up to 96 GB of that shared memory pool as VRAM – enough to load a 70B-parameter model entirely on-device. It’s not a datacenter machine – it sits on my desk. But it runs real workloads, and when it goes down, I need to know why. That’s the whole point of monitoring: not pretty graphs, but <strong>answers after something goes wrong</strong>.</p>

<hr />

<h2 id="what-is-sysstat">What is sysstat?</h2>

<p>Think of <code class="language-plaintext highlighter-rouge">sysstat</code> as your Linux machine’s flight recorder. Just like the black box on an aircraft continuously records flight data so investigators can reconstruct what happened after an incident, <code class="language-plaintext highlighter-rouge">sysstat</code> continuously records system performance data – CPU usage, memory utilization, disk I/O, network throughput, and more – so you can look back and see exactly what was happening at any point in time.</p>

<p>The good news: if you’re running Ubuntu (or most Debian-based distributions), <code class="language-plaintext highlighter-rouge">sysstat</code> is likely already installed. You can check with:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">dpkg <span class="nt">-l</span> | <span class="nb">grep </span>sysstat</code></pre></figure>

<p>Under the hood, <code class="language-plaintext highlighter-rouge">sysstat</code> is a collection of tools that work together:</p>

<ul>
  <li><strong><code class="language-plaintext highlighter-rouge">sadc</code></strong> (System Activity Data Collector) – the engine. It reads performance counters from the kernel and writes them to a binary data file. You never run this directly.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">sa1</code></strong> – a wrapper that tells <code class="language-plaintext highlighter-rouge">sadc</code> to take a snapshot. This is what runs on a schedule.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">sa2</code></strong> – generates daily summary reports from the collected data.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">sar</code></strong> (System Activity Reporter) – your main interface. This is the command you use to query historical data: “show me CPU usage from last Thursday between 7 PM and midnight.”</li>
</ul>

<p>The data lives in <code class="language-plaintext highlighter-rouge">/var/log/sysstat/</code>, with one binary file per day named <code class="language-plaintext highlighter-rouge">saDD</code> (where DD is the day of the month – <code class="language-plaintext highlighter-rouge">sa25</code> for the 25th, for example). By default, files are retained for 7 days and then cleaned up automatically.</p>

<p>Out of the box on Ubuntu, <code class="language-plaintext highlighter-rouge">sysstat</code> collects a snapshot every <strong>10 minutes</strong> and records a <strong>basic set of metrics</strong> – CPU, memory, swap, and disk I/O. That’s a reasonable default for a general-purpose server, but it’s not enough for our use case. We want:</p>

<ol>
  <li><strong>1-minute intervals</strong> – fine enough to catch the bursty spikes that AI workloads produce.</li>
  <li><strong>All available metrics</strong> – adding network interfaces, interrupts, power management, hugepages, and more on top of the basics.</li>
</ol>

<p>Both of these are simple configuration changes. No new software to install.</p>

<hr />

<h2 id="the-setup-three-changes-five-minutes">The Setup: Three Changes, Five Minutes</h2>

<p>There are three things to change, and I’ll explain why each one matters.</p>

<h3 id="step-1-override-the-collection-timer-to-1-minute">Step 1: Override the collection timer to 1 minute</h3>

<p>On modern Ubuntu, <code class="language-plaintext highlighter-rouge">sysstat</code> is triggered by a systemd timer called <code class="language-plaintext highlighter-rouge">sysstat-collect.timer</code>. By default, it fires every 10 minutes. We want every 1 minute.</p>

<p>The right way to change a systemd timer’s schedule is with a <strong>drop-in override</strong> – a small config file that supplements the original without modifying it. This is important because the original file lives in <code class="language-plaintext highlighter-rouge">/usr/lib/systemd/system/</code> and belongs to the <code class="language-plaintext highlighter-rouge">sysstat</code> package. If you edit it directly, a future package update will overwrite your changes. Drop-in overrides live in <code class="language-plaintext highlighter-rouge">/etc/systemd/system/</code> and are always preserved.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/systemd/system/sysstat-collect.timer.d

<span class="nb">cat</span> <span class="o">&lt;&lt;</span><span class="sh">'</span><span class="no">EOF</span><span class="sh">' | sudo tee /etc/systemd/system/sysstat-collect.timer.d/override.conf
[Timer]
OnCalendar=
OnCalendar=*:*:00
EOF</span></code></pre></figure>

<p>The first <code class="language-plaintext highlighter-rouge">OnCalendar=</code> (with no value) clears the default 10-minute schedule. The second line, <code class="language-plaintext highlighter-rouge">OnCalendar=*:*:00</code>, means “every hour, every minute, at second :00” – i.e., once per minute.</p>

<h3 id="step-2-switch-from-disk-only-to-full-statistics">Step 2: Switch from disk-only to full statistics</h3>

<p>The default configuration in <code class="language-plaintext highlighter-rouge">/etc/sysstat/sysstat</code> sets <code class="language-plaintext highlighter-rouge">SADC_OPTIONS="-S DISK"</code>, which adds block device statistics to the basic collection but still misses network interfaces, interrupts, power management, and other extended metrics. We want everything.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s/^SADC_OPTIONS="-S DISK"/SADC_OPTIONS="-S XALL"/'</span> /etc/sysstat/sysstat</code></pre></figure>

<p>The <code class="language-plaintext highlighter-rouge">-S XALL</code> flag enables collection of all available statistics: CPU utilization, memory usage, swap, network interfaces, NFS, sockets, hugepages, power management, and more. The storage overhead is minimal – roughly 5-6 MB per day, or about 40 MB for a full week of 1-minute data.</p>

<h3 id="step-3-disable-the-legacy-cron-job">Step 3: Disable the legacy cron job</h3>

<p>Here’s a subtlety that’s easy to miss. On Ubuntu 24.04, <code class="language-plaintext highlighter-rouge">sysstat</code> ships with <strong>both</strong> a systemd timer and a cron job, and both are active by default. The cron job runs at 5-minute offsets (<code class="language-plaintext highlighter-rouge">:05</code>, <code class="language-plaintext highlighter-rouge">:15</code>, <code class="language-plaintext highlighter-rouge">:25</code>, …) while the timer runs on the tens (<code class="language-plaintext highlighter-rouge">:00</code>, <code class="language-plaintext highlighter-rouge">:10</code>, <code class="language-plaintext highlighter-rouge">:20</code>, …). Together they produce a sample every 5 minutes – but they’re not aware of each other, and with our 1-minute timer, the cron becomes redundant noise.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s|^5-55/10|#5-55/10|'</span> /etc/cron.d/sysstat</code></pre></figure>

<p>This comments out the cron schedule line, leaving the systemd timer as the sole driver.</p>

<h3 id="apply-the-changes">Apply the changes</h3>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>systemctl daemon-reload
<span class="nb">sudo </span>systemctl restart sysstat-collect.timer</code></pre></figure>

<h3 id="verify">Verify</h3>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">systemctl status sysstat-collect.timer</code></pre></figure>

<p>You should see the drop-in override listed and a <code class="language-plaintext highlighter-rouge">Trigger</code> timestamp roughly 1 minute in the future. After a couple of minutes, confirm data is flowing:</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sar <span class="nt">-u</span> | <span class="nb">tail</span> <span class="nt">-5</span></code></pre></figure>

<p>The timestamps should be 1 minute apart:</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">08:55:02 PM     all      0.20      0.00      0.19      0.19      0.00     99.43
08:56:11 PM     all      0.21      0.00      0.19      0.19      0.00     99.42
08:57:11 PM     all      0.20      0.00      0.18      0.20      0.00     99.42
08:58:02 PM     all      0.19      0.00      0.19      0.19      0.00     99.42
08:59:01 PM     all      0.19      0.00      0.19      0.19      0.00     99.43</code></pre></figure>

<p>That’s it. Five minutes of work, and your machine now records a comprehensive performance snapshot every minute, automatically, surviving reboots, with 7-day retention and zero ongoing maintenance.</p>

<hr />

<h2 id="reading-the-data-sar-in-practice">Reading the Data: sar in Practice</h2>

<p>Having the data is only useful if you can query it. Here’s a quick tour of the commands you’ll actually use.</p>

<h3 id="cpu-utilization">CPU Utilization</h3>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sar <span class="nt">-u</span></code></pre></figure>

<p>This shows today’s CPU data. The columns that matter most:</p>

<table>
  <thead>
    <tr>
      <th>Column</th>
      <th>What it means</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">%user</code></td>
      <td>Time spent on user-space processes (your applications, your model inference)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">%system</code></td>
      <td>Time spent on kernel operations (memory management, I/O)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">%iowait</code></td>
      <td>Time the CPU was idle while the system had an outstanding disk I/O request</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">%idle</code></td>
      <td>Time the CPU had nothing to do</td>
    </tr>
  </tbody>
</table>

<p>For AI inference, watch <code class="language-plaintext highlighter-rouge">%user</code> (this is where your model runs) and <code class="language-plaintext highlighter-rouge">%iowait</code> (high values here mean your storage is a bottleneck, possibly during model loading).</p>

<h3 id="memory-utilization">Memory Utilization</h3>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sar <span class="nt">-r</span></code></pre></figure>

<p>Key columns:</p>

<table>
  <thead>
    <tr>
      <th>Column</th>
      <th>What it means</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">kbmemfree</code></td>
      <td>Truly free memory (not used for anything)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">kbmemused</code></td>
      <td>Memory in active use</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">%memused</code></td>
      <td>Percentage of total memory in use</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">kbbuffers</code> / <code class="language-plaintext highlighter-rouge">kbcached</code></td>
      <td>Memory used by the kernel for disk caching (reclaimable under pressure)</td>
    </tr>
  </tbody>
</table>

<p>When a large model loads, you’ll see <code class="language-plaintext highlighter-rouge">%memused</code> jump sharply. If it climbs toward 90%+ and stays there, you’re flirting with swap usage or OOM conditions.</p>

<h3 id="network">Network</h3>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sar <span class="nt">-n</span> DEV</code></pre></figure>

<p>Useful if you’re serving inference over the network and want to see traffic patterns – or if you need to confirm the network was actually active before an incident, as I did during my freeze investigation.</p>

<h3 id="querying-historical-data">Querying Historical Data</h3>

<p>This is where <code class="language-plaintext highlighter-rouge">sar</code> really shines for post-incident analysis. You’re not limited to today’s data.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Show CPU usage from a specific date (the 25th of the current month)</span>
sar <span class="nt">-u</span> <span class="nt">-f</span> /var/log/sysstat/sa25

<span class="c"># Show memory usage in a specific time window</span>
sar <span class="nt">-r</span> <span class="nt">-s</span> 19:00:00 <span class="nt">-e</span> 23:30:00

<span class="c"># Combine both: memory on Feb 25, 7 PM to 11:30 PM</span>
sar <span class="nt">-r</span> <span class="nt">-f</span> /var/log/sysstat/sa25 <span class="nt">-s</span> 19:00:00 <span class="nt">-e</span> 23:30:00</code></pre></figure>

<p>During my freeze investigation, this exact capability is what ruled out an out-of-memory condition. The <code class="language-plaintext highlighter-rouge">sar -r</code> output from that evening showed memory usage steady at 14-15%, right up until the journal stopped. No spike, no leak, no OOM. That single query eliminated an entire class of hypotheses and redirected the investigation toward the kernel.</p>

<p>With 1-minute resolution, these queries become even more valuable. Instead of seeing six data points in the final hour before an incident, you get sixty.</p>

<hr />

<h2 id="whats-next">What’s Next</h2>

<p>There’s one blind spot this setup doesn’t cover: <strong>GPU and VRAM</strong>.</p>

<p>If you’re running model inference on a GPU – whether discrete or an integrated one like the Radeon 8060S on my GMKTEC EVO X2 – you’ll want to track GPU utilization and VRAM pressure. The Ryzen AI MAX+ 395 can allocate up to 96 GB of unified memory as VRAM, so knowing how much is actually in use matters. sysstat doesn’t collect GPU metrics, but on AMD systems, the kernel exposes them through sysfs files that are trivial to read and log.</p>

<p>A natural next step would be a lightweight script on a systemd timer that reads these sysfs values every minute and appends them to a CSV – the same philosophy as the sysstat setup: no new dependencies, minimal overhead, data that survives reboots. If your needs eventually grow beyond historical forensics into real-time dashboards and alerting, Prometheus + Grafana is the well-trodden path. But for a single local inference server, the approach in this article is hard to beat for simplicity-to-value ratio.</p>

<hr />

<h2 id="the-bottom-line">The Bottom Line</h2>

<p>Running a local AI inference server is one of the most rewarding things you can do as a practitioner. But “local” means “your responsibility.” When something goes wrong – and it will – the quality of your monitoring determines whether you spend twenty minutes diagnosing the problem or twenty hours guessing.</p>

<p><code class="language-plaintext highlighter-rouge">sysstat</code> is already on your machine. The setup takes five minutes. The storage cost is negligible. And the next time your server hangs at 11 PM, those 1-minute snapshots might be the only evidence that survives.</p>

<p>The best time to set up monitoring is before you need it. The second best time is now.</p>]]></content><author><name>davidlow</name></author><category term="blog" /><category term="linux" /><category term="monitoring" /><category term="sysstat" /><category term="ai-inference" /><category term="ubuntu" /><summary type="html"><![CDATA[How a real system freeze on my local AI inference server led me to set up 1-minute sysstat monitoring -- and why every local LLM setup needs it.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://davidlow.ai/assets/img/v2/og.jpg" /><media:content medium="image" url="https://davidlow.ai/assets/img/v2/og.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>