Spectrum C API Reference

The Spectrum C API is the low-level foundation of the SDK. It is designed for ABI stability, high performance, and extensibility. All higher-level language bindings (C++, Python, Kotlin, Swift) are built on top of this C interface.

Design Philosophy

1. ABI Stability

The API uses a C-first design to ensure binary compatibility across compiler versions and languages.

No C++ classes at the boundary: All interfaces are C structs with function pointers.
Extension Chains: Structures use a pNext pattern (similar to Vulkan) to allow adding fields without breaking existing layouts.
Reserved Padding: Enums and structs have reserved ranges/padding for future use.

2. Zero-Copy & Memory Ownership

Packets own the data. Passing a packet to a plugin transfers access, but ownership rules are explicit (reference counting or strict lifecycle).
Zero-Copy: Data is passed by reference. GPU buffers are shared via handles (e.g., AHardwareBuffer, cl_mem, VkDeviceMemory) without CPU round-trips when possible.

3. Explicit Context

Contexts abstract the execution backend (OpenCL, Vulkan, Metal).
Plugins receive a context at initialization and must use it for resource allocation to ensure sharing.

Architecture Overview

The Spectrum runtime is composed of the following core entities:

graph TD
    App[Application] -->|Configures| Graph[Spectrum Graph]
    Graph -->|Manages| Node[Nodes]
    Node -->|Wraps| Plugin[Plugin Instance]
    Plugin -->|Uses| Context[Spectrum Context]
    Plugin -->|Processes| Packet[Spectrum Packet]
    Packet -->|Carries| Data[Image/Tensor Data]
    Packet -->|Has| Metadata[Metadata]

Graph: The orchestrator. Parses JSON, loads plugins, manages data flow.
Plugin: The worker. Implements image processing logic (e.g., "Face Detect", "Grayscale").
Context: The hardware interface. Manages GPU devices and command queues.
Packet: The data carrier. Holds image frames or tensor data.
Metadata: Side-channel data. Configuration parameters or analysis results.

Core API Modules

1. Context (`context.h`)

The Context abstracts the compute backend. It is essential for sharing GPU resources between plugins.

Key Structures:

spectrum_context_t: Opaque handle to the context.
spectrum_context_create_info_t: Configuration for creation.

Example: Creating a Vulkan Context

#include <spectrum/c/context.h>
#include <spectrum/c/context_vulkan_ext.h>

// 1. Define Vulkan extension
spectrum_context_vulkan_ext_t vk_ext = {
    .structure_type = SPECTRUM_STRUCTURE_TYPE_CONTEXT_VULKAN_EXT,
    .device_index = 0, // Use first device
    // .instance, .physical_device, .device can be passed here to share existing VK state
};

// 2. Create Info
spectrum_context_create_info_t create_info = {
    .structure_type = SPECTRUM_STRUCTURE_TYPE_CONTEXT_CREATE_INFO,
    .backend = SPECTRUM_BACKEND_VULKAN,
    .next = &vk_ext // Chain the extension
};

// 3. Create
spectrum_context_t* ctx = NULL;
spectrum_status_e status = spectrum_context_create(&create_info, &ctx);

if (status == SPECTRUM_STATUS_SUCCESS) {
    // Use context...
    spectrum_context_destroy(ctx);
}

2. Packet (`packet.h`)

Packets are the currency of the graph. They hold data descriptions (format, dimensions) and the actual memory.

Key Structures:

spectrum_packet_t: Opaque handle.
spectrum_packet_description_t: Metadata about the image (width, height, format).

Example: Importing Host Memory

#include <spectrum/c/packet.h>

// 1. Describe the data
spectrum_packet_description_t desc = {
    .structure_type = SPECTRUM_STRUCTURE_TYPE_PACKET_DESCRIPTION,
    .format = SPECTRUM_FORMAT_RGB_INTERLEAVED,
    .width = 1920,
    .height = 1080,
    .channels = 3,
    .data_type = SPECTRUM_DATA_TYPE_UINT8
};

// 2. Wrap host pointer
uint8_t* my_pixels = malloc(1920 * 1080 * 3);
spectrum_packet_handle_host_t host_handle = {
    .host_buffer = my_pixels
};

// 3. Create Import Handle
const void* import_handle = spectrum_packet_import_from_host(&host_handle);

// 4. Create Packet
spectrum_packet_t* packet = spectrum_packet_create_from_handle(&desc, import_handle, NULL);

// ... use packet ...

spectrum_packet_destroy(packet); // Does not free my_pixels, only the wrapper
free(my_pixels);

3. Metadata (`metadata.h`)

Metadata is a strongly-typed key-value store for configuration and results. It is distinct from Packets to allow lightweight parameter updates.

Key Functions:

spectrum_metadata_create
spectrum_metadata_set
spectrum_metadata_get

Example: Setting Configuration

#include <spectrum/c/metadata.h>

spectrum_metadata_t* meta = spectrum_metadata_create(4096);

// Set an integer value
int32_t exposure = 100;
spectrum_metadata_entry_t entry = {};
spectrum_metadata_value_ro_t val = { .i32 = &exposure };

spectrum_create_metadata_entry_with_value(
    &entry, "exposure_time", SPECTRUM_DATA_TYPE_INT32, val);

spectrum_metadata_set(meta, &entry);

Plugin API (`plugin_api.h`)

This is the interface you implement to create custom nodes. A plugin is a shared library exporting spectrum_plugin_get_callbacks.

Lifecycle

Load: Host calls spectrum_plugin_get_callbacks.
GetCapabilities: Host asks "What can you do?" (Ports, Backends).
Create: Host creates an instance.
Negotiate: Host and Plugin agree on formats.
Initialize: Host provides final config and Context.
Process: Repeatedly called with input/output packets.
Destroy: Cleanup.

Negotiation Flow

Negotiation allows the graph to adapt. For example, a "Grayscale" plugin might accept any resolution but force the output format to GRAY8.

Forward (negotiate_output_description): Input is known. Plugin says what outputs it can produce. Example: Input is 1920x1080 RGB. Plugin says "I can output 1920x1080 GRAY".
Backward (negotiate_input_description): Output is constrained (e.g., by the screen). Plugin says what inputs it needs. Example: Output must be 1080p. Plugin says "Then I need 1080p input".

Example: Implementing Process

High-Performance Processing with Halide

For image processing kernels, we strongly recommend using Halide instead of writing raw loops. The SDK provides built-in support for obtaining Halide buffers from packets, handling all necessary synchronization and memory mapping.

Recommended Pattern:

Sync the packet to ensure the buffer is ready (spectrum_packet_sync_to_buffer).
Get the underlying halide_buffer_t directly (spectrum_packet_get_halide_buffers).
Invoke the generated Halide kernel.

#include <HalideBuffer.h>
#include "generated_halide_kernel.h" // e.g., halide_rgb_to_gray.h

spectrum_status_e process_with_halide(
    spectrum_packet_t* in_pkt,
    spectrum_packet_t* out_pkt)
{
    // 1. Sync packets to ensure CPU-accessible buffers are ready
    SPECTRUM_RETURN_IF_ERROR(spectrum_packet_sync_to_buffer(in_pkt));
    SPECTRUM_RETURN_IF_ERROR(spectrum_packet_sync_to_buffer(out_pkt));

    // 2. Get Halide Buffers directly from the SDK
    int num_in = 0, num_out = 0;
    struct halide_buffer_t** in_bufs = spectrum_packet_get_halide_buffers(
        in_pkt, &num_in, SPECTRUM_HALIDE_BUFFER_VIEW_AUTO);
    struct halide_buffer_t** out_bufs = spectrum_packet_get_halide_buffers(
        out_pkt, &num_out, SPECTRUM_HALIDE_BUFFER_VIEW_AUTO);

    if (!in_bufs || num_in == 0 || !out_bufs || num_out == 0) {
        return SPECTRUM_STATUS_INVALID_ARGUMENT;
    }

    // 3. Invoke Kernel (passing the raw halide_buffer_t*)
    // Note: Generated kernels typically accept halide_buffer_t* directly.
    int result = halide_rgb_to_gray(in_bufs[0], out_bufs[0]);

    return (result == 0) ? SPECTRUM_STATUS_SUCCESS : SPECTRUM_STATUS_INTERNAL_ERROR;
}

Graph API (`graph.h`)

The Graph API allows applications to compose plugins into pipelines.

1. Defining the Graph

Graphs are defined using JSON.

{
  "nodes": [
    { "name": "cam", "plugin": "camera_source" },
    { "name": "gray", "plugin": "grayscale_filter" },
    { "name": "screen", "plugin": "screen_sink" }
  ],
  "connections": [
    { "source": "cam.out", "target": "gray.in" },
    { "source": "gray.out", "target": "screen.in" }
  ]
}

2. Creating and Binding

#include <spectrum/c/graph.h>

// 1. Create Graph
spectrum_graph_create_info_t info = {
    .structure_type = SPECTRUM_STRUCTURE_TYPE_GRAPH_CREATE_INFO,
    .graph_json = json_string
};
spectrum_graph_t* graph;
spectrum_graph_create(&info, &graph);

// 2. Bind External Outputs (e.g., to a Surface)
spectrum_graph_output_binding_t binding = {
    .port = "screen.in",
    .packet = my_surface_packet
};
spectrum_graph_output_bindings_t bindings = { .bindings = &binding, .binding_count = 1 };
spectrum_graph_bind_outputs(graph, &bindings);

// 3. Start
spectrum_graph_start(graph);

Utilities

Logging (`log.h`)

Use these macros to log to the platform's standard output (Logcat on Android, stdout on Linux/macOS).

#include <spectrum/c/log.h>

SPECTRUM_LOGI("MyPlugin", "Initialized with width=%d", width);
SPECTRUM_LOGE("MyPlugin", "Fatal error: %s", error_msg);

Types (`types.h`)

Common types and helpers.

spectrum_status_e: Return code for all functions.
spectrum_status_to_string(status): Helper for debugging.

Design Philosophy​

1. ABI Stability​

2. Zero-Copy & Memory Ownership​

3. Explicit Context​

Architecture Overview​

Core API Modules​

1. Context (context.h)​

2. Packet (packet.h)​

3. Metadata (metadata.h)​

Plugin API (plugin_api.h)​

Lifecycle​

Negotiation Flow​

High-Performance Processing with Halide​

Graph API (graph.h)​

1. Defining the Graph​

2. Creating and Binding​

Utilities​

Logging (log.h)​

Types (types.h)​