AI Breaks Developer Boundaries 3 - OS and Devices

Series · AI Breaks Developer Boundaries

Article series · Ongoing

Episode 3 · AI Breaks Developer Boundaries 3 - OS and Devices

This final article in the three-part series looks at how to implement across the boundary, with code and commands rather than abstractions. AI lowers this particular barrier well: it can connect the starting OS API, the callback that returns data, the log to open, and the sequence that joins a board to a server.

The examples form one implementation map: Windows UI input, Android BLE, and a Raspberry Pi toy car. The game-automation example discusses only the UI-input layer. Process memory, hooks, kernel drivers, and anti-cheat evasion are outside this article.

1. The UI layer of an auto mouse: turn coordinates into SendInput events

At the outermost Windows layer, an auto mouse is straightforward. Pick the screen pixel to click, normalize it to the 0–65535 absolute-coordinate range, then put move, left-button-down, and left-button-up records into an INPUT array for SendInput. Microsoft's SendInput reference describes this as serial insertion of keyboard and mouse events into the input stream.

// Win32 / User32.lib. Click one virtual-desktop pixel coordinate.
#include <windows.h>
#include <array>

bool clickAt(long pixelX, long pixelY) {
    const long left = GetSystemMetrics(SM_XVIRTUALSCREEN);
    const long top = GetSystemMetrics(SM_YVIRTUALSCREEN);
    const long width = GetSystemMetrics(SM_CXVIRTUALSCREEN);
    const long height = GetSystemMetrics(SM_CYVIRTUALSCREEN);
    const long normalizedX = (pixelX - left) * 65535L / (width - 1);
    const long normalizedY = (pixelY - top) * 65535L / (height - 1);

    std::array<INPUT, 3> input{};
    input[0].type = INPUT_MOUSE;
    input[0].mi.dx = normalizedX;
    input[0].mi.dy = normalizedY;
    input[0].mi.dwFlags = MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE |
                          MOUSEEVENTF_VIRTUALDESK;
    input[1].type = INPUT_MOUSE;
    input[1].mi.dwFlags = MOUSEEVENTF_LEFTDOWN;
    input[2].type = INPUT_MOUSE;
    input[2].mi.dwFlags = MOUSEEVENTF_LEFTUP;
    return SendInput(static_cast<UINT>(input.size()), input.data(), sizeof(INPUT))
           == input.size();
}

MOUSEEVENTF_ABSOLUTE accepts normalized rather than literal pixel coordinates. On a multi-monitor desktop, pair it with MOUSEEVENTF_VIRTUALDESK and the virtual desktop's left, top, width, and height. For a non-fixed button, the next pipeline is screen capture, template matching or OCR, button-center calculation, then clickAt(). That is the technical backbone of UI automation.

cl /std:c++20 /EHsc auto_click.cpp user32.lib
.\auto_click.exe

SendInput is a User32 input-layer API. This article stays at OS-provided UI-event generation and screen-coordinate handling. UIPI and application-specific input handling remain runtime conditions to check.

<Implementation flow from Windows UI input to an Android BLE connection 3.1>

2. Android Bluetooth LE: from scan callback to GATT notification

For headphones or a sensor, three Android classes carry the BLE implementation. BluetoothLeScanner discovers peripherals and returns results through ScanCallback. A discovered BluetoothDevice connects through connectGatt(). Connection, service discovery, reads, and notifications return through BluetoothGattCallback.

On Android 12 and newer, declare and request the following permissions at runtime. Android's Bluetooth permissions guide explains BLUETOOTH_SCAN and BLUETOOTH_CONNECT.

<uses-permission android:name="android.permission.BLUETOOTH_SCAN" />
<uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />

This Kotlin example finds an owned BLE test device by advertised name, connects, then locates a documented service and characteristic UUID and enables notifications. Replace the sample UUIDs with the device's published GATT contract or values first observed in nRF Connect; do not invent them for a product.

private val serviceUuid = UUID.fromString("12345678-1234-1234-1234-1234567890ab")
private val notifyUuid = UUID.fromString("12345678-1234-1234-1234-1234567890ac")
private val cccdUuid = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb")

private val scanCallback = object : ScanCallback() {
    override fun onScanResult(type: Int, result: ScanResult) {
        if (result.device.name == "LAB_SENSOR") {
            bluetoothAdapter.bluetoothLeScanner.stopScan(this)
            result.device.connectGatt(this@MainActivity, false, gattCallback)
        }
    }
}

private val gattCallback = object : BluetoothGattCallback() {
    override fun onConnectionStateChange(gatt: BluetoothGatt, status: Int, state: Int) {
        if (status == BluetoothGatt.GATT_SUCCESS && state == BluetoothProfile.STATE_CONNECTED) gatt.discoverServices()
        else if (state == BluetoothProfile.STATE_DISCONNECTED) gatt.close()
    }
    override fun onServicesDiscovered(gatt: BluetoothGatt, status: Int) {
        val characteristic = gatt.getService(serviceUuid)?.getCharacteristic(notifyUuid) ?: return
        gatt.setCharacteristicNotification(characteristic, true)
        val cccd = characteristic.getDescriptor(cccdUuid) ?: return
        cccd.value = BluetoothGattDescriptor.ENABLE_NOTIFICATION_VALUE
        gatt.writeDescriptor(cccd)
    }
    override fun onCharacteristicChanged(gatt: BluetoothGatt, characteristic: BluetoothGattCharacteristic, value: ByteArray) {
        Log.d("BLE", "${characteristic.uuid}: ${value.joinToString { "%02x".format(it) }}")
    }
}

onConnectionStateChange() receives link state, onServicesDiscovered() gets the GATT table, and onCharacteristicChanged() receives data pushed by the device. API 33 added the ByteArray value callback signature; Android's BluetoothGattCallback reference documents why it is preferable.

3. When BLE will not connect: where nRF Connect, HCI, ATT, and Wireshark fit

Separate the layers instead of reciting tool names. nRF Connect is a mobile test app that scans peripherals and explores, reads, and subscribes to GATT services and characteristics. Use it to verify the UUID contract before debugging application code. ATT, the Attribute Protocol, is the data protocol through which GATT reads and writes characteristic values. HCI, the Host Controller Interface, is the packet layer between Android's Bluetooth host and the physical controller. Wireshark opens an HCI snoop capture to show whether advertising, connection, ATT reads/writes, and notifications actually occurred.

Start with framework and service state:

adb shell dumpsys bluetooth_manager
adb logcat -v time | grep -E "BluetoothGatt|BtGatt|Bluetooth|BLE"

Then enable Enable Bluetooth HCI snoop log in Developer options and restart Bluetooth. AOSP's Bluetooth debugging guide documents dumpsys bluetooth_manager, btsnooz.py BUG_REPORT.txt > BTSNOOP.log, and HCI-snoop collection.

adb bugreport ble-report.zip
unzip ble-report.zip -d ble-report
btsnooz.py ble-report/bugreport-*.txt > BTSNOOP.log
# Open BTSNOOP.log in Wireshark and inspect HCI and ATT packets by time.

The diagnostic sequence becomes concrete. If nRF Connect cannot see the service, start with advertising, connection, or the peripheral GATT itself. If nRF works but the app does not, inspect permission, UUID, callback order, and descriptor write. If HCI contains a write but no notification, inspect peripheral firmware and characteristic subscription. Linking code, OS, and controller logs on one timeline is exactly where AI compresses discovery time.

4. Raspberry Pi car: join motor, sensor, and UART in one process

The minimum toy-car stack is a Raspberry Pi, an H-bridge motor driver, a distance sensor, and a helper MCU or sensor over UART. Do not attach a motor directly to GPIO; use an H-bridge such as a TB6612FNG or L298N. Raspberry Pi's GPIO hardware guide provides the same guidance and exposes pinout for local pin inspection.

pinout
ls -l /dev/serial0 /dev/ttyACM* /dev/ttyUSB* 2>/dev/null
sudo usermod -a -G gpio "$USER"
python3 -m pip show gpiozero pyserial requests

This example uses gpiozero.Motor for the H-bridge IN1/IN2 pins and DistanceSensor for an HC-SR04-style trigger/echo sensor. Pin numbers are BCM numbers; adapt them to real wiring, driver-board enable, and standby pins.

# car_control.py
from gpiozero import Motor, DistanceSensor
from time import sleep, monotonic

motor = Motor(forward=17, backward=27, enable=18)
range_sensor = DistanceSensor(echo=24, trigger=23, max_distance=2.0)

while True:
    distance_m = range_sensor.distance * 2.0
    command = "forward" if distance_m > 0.35 else "stop"
    if command == "forward": motor.forward(0.45)
    else: motor.stop()
    print(f"ts={monotonic():.3f} distance_m={distance_m:.3f} command={command}", flush=True)
    sleep(0.05)

python3 -u car_control.py | tee car-control.log
tail -f car-control.log

For an Arduino or sensor board on UART, send a line of JSON from the MCU and read it on the Pi. This is the simplest telemetry path between a board and a Linux application.

void setup() { Serial.begin(115200); }
void loop() {
  int raw = analogRead(A0);
  Serial.print("{\"raw\":"); Serial.print(raw);
  Serial.println(",\"source\":\"mcu\"}");
  delay(100);
}

import json, serial
with serial.Serial("/dev/ttyACM0", 115200, timeout=1) as uart:
    while True:
        line = uart.readline().decode(errors="replace").strip()
        if line: print(json.loads(line))

<Data flow from Raspberry Pi sensors and UART through motor, camera, and server paths 3.2>

5. Send camera frames to a server with Picamera2 and HTTP

Adding a camera lets the car send frames as well as scalar sensor data. On current Raspberry Pi OS, Picamera2 runs on the libcamera/rpicam stack. Raspberry Pi's camera documentation and the Picamera2 manual document the capture_array() path to a NumPy frame.

rpicam-hello -t 3000
sudo apt install -y python3-picamera2 python3-opencv python3-requests

# camera_uploader.py
from picamera2 import Picamera2
import cv2, requests, time

SERVER = "http://192.168.0.10:8080/telemetry/frame"
camera = Picamera2()
camera.configure(camera.create_preview_configuration(main={"size": (640, 480), "format": "RGB888"}))
camera.start()

while True:
    frame = camera.capture_array("main")
    ok, jpeg = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, 75])
    if ok:
        response = requests.post(SERVER,
            files={"frame": ("frame.jpg", jpeg.tobytes(), "image/jpeg")},
            data={"source": "toy-car", "captured_at": str(time.time())}, timeout=2)
        print("upload", response.status_code, len(jpeg))
    time.sleep(0.2)

At first the server only needs to receive the frame and return a status code. Later it can separate motor commands through MQTT or return object-detection results for the Pi to consume. AI is particularly useful here because it joins rules scattered across GPIO pins, UART baud rates, camera formats, HTTP payloads, and system-service logs into one executable first draft.

The point of Part 3 is not that AI writes safety language. It is that it connects the first implementation path across UI events, GATT callbacks, HCI/ATT packets, UART telemetry, GPIO motor control, and camera HTTP uploads. That connection lowers the embedded entry barrier for web developers and the OS/network barrier for embedded developers.

AI is not magic that returns a finished product from every domain. Through a developer CLI workflow and AI coding tools, it is an explorer that makes a developer move in the order which class → which callback → which log → which port → which test. Once that flow is run and corrected, the walls built by languages and platforms are substantially lower.

Previous episodes

1. Episode 2 AI Breaks Developer Boundaries 2 - Open Code and Interfaces
2. Episode 1 AI Breaks Developer Boundaries 1 - Why Walls Form

View the first episode