Compare commits

...

5 Commits

Author SHA1 Message Date
66feb91821 ci: pass docker push input as a string, not a boolean
All checks were successful
🚦 CI / changes (push) Successful in 6s
🚦 CI / docker (push) Successful in 2m57s
🚦 CI / test (push) Successful in 8m7s
🚦 CI / docs (push) Successful in 6s
Gitea's act_runner mangles boolean workflow_call/dispatch inputs passed
from an expression -- they arrive as false regardless of value. Declare
`push` as a string ("true"/"false") and compare with == 'true' so the
builder image is pushed on non-PR events again.
2026-07-04 15:07:27 +02:00
903dd4eea5 docs: update README examples table and add documentation link
07_python_network and 08_python_subport now work and run as CI smoke
tests, so drop their "(pending)" markers and describe what they actually
demonstrate. Also surface the hosted documentation link at the top and
re-render README.md from README.md.in.
2026-07-04 15:07:12 +02:00
298c9e770b examples: run Python examples 07 and 08 as CTest smoke tests
Neither Python example was registered as a test, so `ctest -L examples`
in CI skipped them entirely -- which is how 08's missing Python node
went unnoticed.

Add a kpn_python_example() helper (gated on KPN_BUILD_PYTHON) that runs
each script with PYTHONPATH pointed at the freshly-built module, so it
does not depend on cwd or a hard-coded build/python path, and register
07 and 08. Also del the network in 07 for deterministic teardown.
2026-07-04 14:44:30 +02:00
2b0873b61b examples: make 08_python_subport run a real Python node
The example was named "python subport" but its graph was entirely C++
(ProduceNode -> DoubleItNode); Python only tapped the output, leaving a
dangling "#todo: return value to network".

Rewrite so the only node in the graph is a pure-Python py_triple, driven
from both ends via the subport taps: net.write() injects inputs and
net.read() pulls results back, closing the round trip. Also del the
network at the end so its callable cycle is reclaimed deterministically.
2026-07-04 14:44:24 +02:00
c4538f03ca python: fix nanobind Network reference leak via GC type slots
The Python Network holds each PyNode's callable, forming an
uncollectable instance -> callable -> globals() -> instance cycle that
tripped nanobind's leak check at interpreter shutdown.

Implement tp_traverse/tp_clear type slots on the Network binding so
Python's cyclic collector can see through the C++-held callables and
break the cycle. PyNode exposes its callable; PyNetwork visits and
clears them. Wired into both binding sites (auto_bind and the legacy
register_py_network).
2026-07-04 14:43:47 +02:00
10 changed files with 157 additions and 34 deletions

View File

@ -63,7 +63,8 @@ jobs:
if: ${{ needs.changes.outputs.dockerfile == 'true' }} if: ${{ needs.changes.outputs.dockerfile == 'true' }}
uses: ./.gitea/workflows/docker.yaml uses: ./.gitea/workflows/docker.yaml
with: with:
push: ${{ github.event_name != 'pull_request' }} # Explicit string, not a boolean expression (act_runner mangles bools).
push: ${{ github.event_name == 'pull_request' && 'false' || 'true' }}
# Runs after docker (if docker ran). A skipped docker job is fine; a failed # Runs after docker (if docker ran). A skipped docker job is fine; a failed
# one blocks this via !failure(). Re-run tests when code OR the image changed. # one blocks this via !failure(). Re-run tests when code OR the image changed.

View File

@ -4,19 +4,22 @@ name: '🐳 Builder Image'
# It is called by ci.yaml only when Dockerfile.builder or docs/requirements.txt # It is called by ci.yaml only when Dockerfile.builder or docs/requirements.txt
# change. It runs on the host runner (NOT inside the builder container) because # change. It runs on the host runner (NOT inside the builder container) because
# it needs the Docker CLI/daemon. # it needs the Docker CLI/daemon.
# Note: `push` is a STRING ("true"/"false"), not a boolean. Gitea's act_runner
# mangles boolean inputs passed from an expression (they arrive as false), so we
# pass an explicit string and compare with == 'true' below.
on: on:
workflow_call: workflow_call:
inputs: inputs:
push: push:
description: 'Push the built image to the registry' description: 'Push the built image to the registry ("true"/"false")'
type: boolean type: string
default: true default: 'true'
workflow_dispatch: workflow_dispatch:
inputs: inputs:
push: push:
description: 'Push the built image to the registry' description: 'Push the built image to the registry ("true"/"false")'
type: boolean type: string
default: true default: 'true'
jobs: jobs:
build: build:
@ -48,7 +51,7 @@ jobs:
. .
- name: Push builder image - name: Push builder image
if: ${{ inputs.push }} if: ${{ inputs.push == 'true' }}
run: | run: |
docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:${{ github.sha }} docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:${{ github.sha }}

View File

@ -1,4 +1,4 @@
# KPN++ Builder Image (CI: pipeline trigger) # KPN++ Builder Image (CI: pipeline trigger v2)
# Pre-built image with GCC, CMake, Ninja, and Python dev headers for building and testing KPN++ # Pre-built image with GCC, CMake, Ninja, and Python dev headers for building and testing KPN++
# Build: docker build -f Dockerfile.builder -t gitea.tourolle.paris/dtourolle/kpnpp-builder:latest . # Build: docker build -f Dockerfile.builder -t gitea.tourolle.paris/dtourolle/kpnpp-builder:latest .
# Push: docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest # Push: docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest

View File

@ -2,6 +2,8 @@
A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind. A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind.
📖 **[Documentation](https://pages.tourolle.paris/dtourolle/kpn/)**
--- ---
## Requirements ## Requirements
@ -398,8 +400,8 @@ Violating the second rule deadlocks.
| `04_storage_policy` | `channel_storage_policy` default and specialisation | | `04_storage_policy` | `channel_storage_policy` default and specialisation |
| `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` | | `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` |
| `06_watchdog` | Watchdog interval, stall detection | | `06_watchdog` | Watchdog interval, stall detection |
| `07_python_network` | PyNetwork, pure Python node *(pending)* | | `07_python_network` | PyNetwork with a pure-Python node between a C++ source and sink |
| `08_python_subport` | `net.read`, `net.write`, sub-port tap *(pending)* | | `08_python_subport` | Drive a Python node from Python via `net.write`/`net.read` sub-port taps |
| `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 | | `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 |
Run the cell-shading example: Run the cell-shading example:

View File

@ -2,6 +2,8 @@
A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind. A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind.
📖 **[Documentation](https://pages.tourolle.paris/dtourolle/kpn/)**
--- ---
## Requirements ## Requirements
@ -220,8 +222,8 @@ Violating the second rule deadlocks.
| `04_storage_policy` | `channel_storage_policy` default and specialisation | | `04_storage_policy` | `channel_storage_policy` default and specialisation |
| `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` | | `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` |
| `06_watchdog` | Watchdog interval, stall detection | | `06_watchdog` | Watchdog interval, stall detection |
| `07_python_network` | PyNetwork, pure Python node *(pending)* | | `07_python_network` | PyNetwork with a pure-Python node between a C++ source and sink |
| `08_python_subport` | `net.read`, `net.write`, sub-port tap *(pending)* | | `08_python_subport` | Drive a Python node from Python via `net.write`/`net.read` sub-port taps |
| `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 | | `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 |
Run the cell-shading example: Run the cell-shading example:

View File

@ -30,3 +30,8 @@ net.build()
net.start() net.start()
time.sleep(0.1) time.sleep(0.1)
net.stop() net.stop()
# Drop the network deterministically: it holds the Python callable, which forms
# a reference cycle via globals(). Deleting the global breaks it so the network
# is reclaimed now rather than lingering to interpreter shutdown.
del net

View File

@ -1,38 +1,55 @@
""" """
08_python_subport tap a C++ node's output from Python using net.read(). 08_python_subport drive a *Python* node from Python via write()/read() taps.
Graph: Graph:
[ProduceNode] --int--> [DoubleItNode] --int--> (tapped by net.read()) (fed by net.write()) --int--> [py_triple] --int--> (tapped by net.read())
The sink is Python: instead of connecting a PrintItNode, we call net.read() Unlike 07, there is no C++ source or sink here: the only node in the network is
to pull values out of DoubleItNode's output directly into Python. a pure-Python function, py_triple. Python plays *both* the producer and the
We also demonstrate net.write() by injecting a value into DoubleItNode's input. consumer by using the subport taps:
* net.write("py", 0, v) injects v into py_triple's input (Python -> network)
* net.read("py", 0) pulls py_triple's output back out (network -> Python)
This closes the loop the old version left as a "#todo": a value flows from
Python, through a Python node running inside the network, and back to Python.
""" """
import sys import sys
import time sys.path.insert(0, "build/python") # for `python examples/.../example.py` from repo root
import threading
sys.path.insert(0, "build/python")
import kpn_python as kpn import kpn_python as kpn
def py_triple(x: int) -> int:
return x * 3
net = kpn.Network() net = kpn.Network()
net.add("src", kpn.make_produce()) # The whole network is a single Python node with a tapped input and output.
net.add("dbl", kpn.make_double_it()) net.add_node("py", py_triple, inputs=["int"], outputs=["int"])
net.connect("src", 0, "dbl", 0)
net.build() net.build()
net.start() net.start()
# Collect a few values from DoubleItNode's output via Python tap # Push values in from Python and read the Python node's results back out.
inputs = [1, 2, 7, 10, 100]
results = [] results = []
for _ in range(5): for v in inputs:
val = net.read("dbl", 0) net.write("py", 0, v) # Python -> py_triple input
results.append(val) results.append(net.read("py", 0)) # py_triple output -> Python
net.stop() net.stop()
print("values read from C++ DoubleItNode output:", results) print("inputs written from Python: ", inputs)
assert all(v == 84 for v in results), f"expected all 84, got {results}" print("outputs read from py_triple:", results)
print("all correct (42 * 2 = 84)")
expected = [v * 3 for v in inputs]
assert results == expected, f"expected {expected}, got {results}"
print("all correct (x * 3 computed by a Python node inside the network)")
# Drop the network deterministically. The network holds the Python callable,
# which (via globals) forms a reference cycle; deleting the global breaks it so
# the network is reclaimed promptly instead of lingering to interpreter exit.
del net

View File

@ -13,6 +13,25 @@ function(kpn_example name)
) )
endfunction() endfunction()
# Register a Python example script as a CTest smoke test. Runs the script with
# PYTHONPATH pointing at the freshly-built kpn_python module, so it does not
# depend on the caller's working directory or a hard-coded "build/python" path.
function(kpn_python_example name)
if(NOT KPN_BUILD_PYTHON)
return()
endif()
add_test(
NAME example_${name}
COMMAND ${CMAKE_COMMAND} -E env
"PYTHONPATH=$<TARGET_FILE_DIR:kpn_python>"
${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/${name}/example.py
)
set_tests_properties(example_${name} PROPERTIES
TIMEOUT 15
LABELS examples
)
endfunction()
kpn_example(01_hello_pipeline) kpn_example(01_hello_pipeline)
kpn_example(02_named_ports) kpn_example(02_named_ports)
kpn_example(03_multi_output) kpn_example(03_multi_output)
@ -31,7 +50,9 @@ if(KPN_WEB_DEBUG)
target_link_libraries(14_debug_hub PRIVATE kpn) target_link_libraries(14_debug_hub PRIVATE kpn)
kpn_target_enable_web_debug(14_debug_hub) kpn_target_enable_web_debug(14_debug_hub)
endif() endif()
# 07 and 08 are Python scripts no compiled target needed. # 07 and 08 are Python scripts no compiled target, but run as smoke tests.
kpn_python_example(07_python_network)
kpn_python_example(08_python_subport)
# 09 requires OpenCV only build if found # 09 requires OpenCV only build if found
find_package(OpenCV QUIET COMPONENTS core imgproc highgui videoio) find_package(OpenCV QUIET COMPONENTS core imgproc highgui videoio)

View File

@ -247,7 +247,7 @@ void bind_network(nb::module_& m) {
nb::class_<IVariantNode<Variant>>(m, "INode"); nb::class_<IVariantNode<Variant>>(m, "INode");
nb::class_<Net>(m, "Network") nb::class_<Net>(m, "Network", nb::type_slots(network_type_slots<Variant>()))
.def("__init__", [](Net* self) { .def("__init__", [](Net* self) {
new (self) Net(); new (self) Net();
register_all_converters<Registry>(*self); register_all_converters<Registry>(*self);

View File

@ -35,6 +35,16 @@ public:
using VNode = IVariantNode<Variant>; using VNode = IVariantNode<Variant>;
using VChannel = IVariantChannel<Variant>; using VChannel = IVariantChannel<Variant>;
// ── GC support ────────────────────────────────────────────────────────────
// Visit every Python object this network transitively holds (currently the
// callable of each PyNode). Used by the Network type's tp_traverse slot so
// Python's cyclic GC can discover instance → callable → globals() cycles.
// Defined out-of-line below, once PyNode is a complete type.
template<typename Fn>
void visit_python_objects(Fn&& visit) const;
// Drop all Python references held by nodes, breaking any cycle (tp_clear).
void clear_python_objects();
// ── Builder API ─────────────────────────────────────────────────────────── // ── Builder API ───────────────────────────────────────────────────────────
void add(std::string name, std::shared_ptr<VNode> node) { void add(std::string name, std::shared_ptr<VNode> node) {
@ -367,6 +377,14 @@ public:
out_channels_[i] = std::move(ch); out_channels_[i] = std::move(ch);
} }
// ── GC support (tp_traverse / tp_clear on the owning Network) ──────────────
// The node holds a Python callable, which typically forms an
// instance → callable → globals() → instance cycle. Expose the callable so
// the Network's GC slots can traverse and clear it. See bindings.hpp's
// network_tp_traverse/network_tp_clear.
const nb::object& python_callable() const { return callable_; }
void clear_python_callable() { callable_ = nb::object(); }
private: private:
void run_loop() { void run_loop() {
while (!stop_flag_.load(std::memory_order_relaxed)) { while (!stop_flag_.load(std::memory_order_relaxed)) {
@ -439,6 +457,60 @@ private:
NodeStats stats_; NodeStats stats_;
}; };
// ── PyNetwork GC helpers (defined here: PyNode is now complete) ────────────────
template<typename Variant>
template<typename Fn>
void PyNetwork<Variant>::visit_python_objects(Fn&& visit) const {
for (const auto& [name, node] : nodes_)
if (auto* py = dynamic_cast<const PyNode<Variant>*>(node.get()))
visit(py->python_callable());
}
template<typename Variant>
void PyNetwork<Variant>::clear_python_objects() {
for (auto& [name, node] : nodes_)
if (auto* py = dynamic_cast<PyNode<Variant>*>(node.get()))
py->clear_python_callable();
}
// ── GC type slots for the Network binding ─────────────────────────────────────
// The Network holds Python callables (via PyNode), forming uncollectable
// instance → callable → globals() → instance cycles at interpreter shutdown.
// These slots let Python's cyclic collector traverse and break them, silencing
// nanobind's leak warnings. See the nanobind "Reference leaks" documentation.
template<typename Variant>
int network_tp_traverse(PyObject* self, visitproc visit, void* arg) {
Py_VISIT(Py_TYPE(self));
if (!nb::inst_ready(self))
return 0;
auto* net = nb::inst_ptr<PyNetwork<Variant>>(self);
int rv = 0;
net->visit_python_objects([&](const nb::object& obj) {
if (rv == 0 && obj.is_valid())
rv = visit(obj.ptr(), arg);
});
return rv;
}
template<typename Variant>
int network_tp_clear(PyObject* self) {
auto* net = nb::inst_ptr<PyNetwork<Variant>>(self);
net->clear_python_objects();
return 0;
}
template<typename Variant>
PyType_Slot* network_type_slots() {
static PyType_Slot slots[] = {
{ Py_tp_traverse, reinterpret_cast<void*>(&network_tp_traverse<Variant>) },
{ Py_tp_clear, reinterpret_cast<void*>(&network_tp_clear<Variant>) },
{ 0, nullptr }
};
return slots;
}
// ── register_py_network (legacy helper) ─────────────────────────────────────── // ── register_py_network (legacy helper) ───────────────────────────────────────
// Registers PyNetwork<Variant> with the given nanobind module. // Registers PyNetwork<Variant> with the given nanobind module.
// Prefer bind_network<Registry> from auto_bind.hpp for new code. // Prefer bind_network<Registry> from auto_bind.hpp for new code.
@ -447,7 +519,7 @@ template<typename Variant>
void register_py_network(nb::module_& m, const char* class_name = "Network") { void register_py_network(nb::module_& m, const char* class_name = "Network") {
using Net = PyNetwork<Variant>; using Net = PyNetwork<Variant>;
nb::class_<Net>(m, class_name) nb::class_<Net>(m, class_name, nb::type_slots(network_type_slots<Variant>()))
.def(nb::init<>()) .def(nb::init<>())
.def("connect", &Net::connect, .def("connect", &Net::connect,
nb::arg("src"), nb::arg("out_idx"), nb::arg("src"), nb::arg("out_idx"),