Function Isolation

Function isolation is a crucial feature of PyneCore that enables the precise replication of Pine Script’s unique execution model. This document explains how function isolation works, why it’s necessary, and how it’s implemented in PyneCore.

What is Function Isolation?

In Pine Script, every function call creates an isolated environment with its own copies of variables. This is different from traditional programming languages where functions share access to the same variables. Function isolation means:

  1. Every function call gets its own copies of Series and Persistent variables
  2. State changes made within a function remain isolated to that specific function call
  3. Multiple calls to the same function maintain separate states

This behavior is essential for correctly implementing indicators and strategies in a bar-by-bar execution model.

Why Function Isolation is Necessary

Function isolation solves several critical requirements for Pine Script compatibility:

1. Per-Call Variable State

Consider the following Pine Script example:

//@version=6
indicator("Function Call State Example")

myFunc() =>
    var count = 0
    count := count + 1
    count

plot(myFunc())
plot(myFunc())

In Pine Script, this would plot two different lines because each call to myFunc() has its own isolated state for the count variable. Without function isolation, both calls would share the same variable, resulting in identical values.

2. Correct Historical Behavior

For indicators like moving averages, each function call needs to maintain its own buffer of historical values. Multiple moving averages with different periods need to maintain separate data buffers even though they use the same underlying function.

Implementation in PyneCore

PyneCore implements function isolation with per-instance state vectors: every function instance’s state (persistent variables, series buffers, the state of its own callees) lives in a plain Python list whose slots are assigned at transform time. Two components cooperate:

1. State Vectors and Layouts

The state-related AST transformers (see AST Transformation) give every state-carrying function:

  • a hidden first parameter (__state__) — its state vector, and
  • a __pyne_layout__ attribute that describes how to build one.

A layout is a plain dict with these keys:

  • init — template value for every slot; instantiation is essentially list(init)
  • series — (slot, max_bars_back, elem) triples; instantiation puts a fresh SeriesImpl circular buffer into these slots. elem is the statically known element type name ('float' selects the native nan as the series’ out-of-range na value) or None
  • varip — slot indexes excluded from the var rollback on intra-bar re-execution
  • children — (slot, call_id, in_loop) triples describing the function’s isolated call sites
  • names — per-slot debug names (used only by debug tooling)
  • pin — present only for a scope whose body resolves something per instance: the slot that holds the instance vector (see core/instance_state.py)
  • compacted — present and true only for @pyne lib modules, whose series are na-compacted rolling windows

The key idea is the parent-slot scheme: the state of a callee instance occupies a dedicated child slot of the caller’s state vector, assigned at transform time. All live state therefore forms a tree hanging off a few root vectors. There is no global instance cache and no key lookups — reaching an instance’s state is a list index away, and dropping a parent releases its entire subtree through normal garbage collection.

2. Root Vectors (core/instance_state.py)

The runner creates root state vectors for the entry points it drives directly — the script’s main(), imported library mains and request.security() processes — via create_root(key, layout). Everything else lives in the tree below them. The same module also provides:

  • reset() — clears the child slots of all roots (drops every function instance) between runs
  • RootVarSnapshot — snapshot/restore of the roots’ var slots for a discarded re-execution (a calc_on_order_fills fill, a live intra-bar tick, a request.security developing re-tick); varip slots and child instances are excluded
  • RootChildSnapshot — the same for the function-instance subtrees below the roots, series slots included

Both restore in place. Arrays, maps, matrices and UDT instances are references in Pine, so the baseline keeps each such object next to a copy of its content and the restore writes the content back into that same object, recursively. Two variables naming one array still name one array after a rollback — in the same scope, in different function instances, or through a container element or a UDT field — and a mutation that reached a nested object is undone with it.

  • explain_state(func, state) — renders a state vector as a readable name -> value dict for debugging

Call Routing

At transform time every call site is classified, because the best possible emission depends on what the callee is:

  • fast — the callee is provably state-carrying (a same-module def, or an imported @pyne function with a layout). Emission: child slot with the __resolve_slot__ cold path.
  • direct — the callee is provably stateless (e.g. a transformed function with no state slots). Emission: plain call, untouched.
  • uniform — the callee cannot be resolved at transform time (function-valued variables, parameters, conditionally rebound names, overload dispatchers). Emission: anchor slot with __bind_any__ and an identity check.
  • skip — builtins, stdlib, types/constructors, module properties, plot/log-style display functions, and the synthetic series-slot method calls emitted by the Series transformer. Untouched.

Where the classification comes from

The transformer never guesses: a site whose callee it cannot prove anything about routes uniform, which is correct for any callee and only slower. What it can prove comes from four places:

  • Same-module definitions are classified from the module’s own slot layout plus a fixpoint over its call graph: a function carries state if it has slots of its own or any call site that is not direct.
  • NON_TRANSFORMABLE_FUNCTIONS in transformers/function_isolation.py lists the callees that skip by name, before anything is resolved: the stateless functions of plain (untransformed) modules, which nothing else could prove stateless — the lib display, drawing, math and string functions, the cast_* helpers, method_call, and the like. An entry must never name a function that keeps per-call-site state.
  • pynecore.lib callees are looked up in the routes section of the generated transformers/lib_types.json, keyed the way a script spells them (lib.ta.sma). A path not listed there routes uniform. The lib is not imported to decide the route.
  • Every other imported module (a user Pyne library, a plain module, the rest of pynecore) is imported at transform time and the callee object is inspected.

One function, classify_callee, turns a callee object into a route: an Exported proxy or an overload dispatcher (__pyne_bind__) is uniform; a class, a classmethod, a builtin or a module property is skipped; a __pyne_layout__ attribute proves it state-carrying (the fast route, or its fast-shared variant for a machine marked __pyne_shared_call_site__); a function of a transformed module without a layout is direct. The transform runs it on the live object of a non-lib callee, and scripts/lib_type_collector.py runs it on every lib callee path to generate the routes section.

The reason for the generated table is the bytecode cache. A cached script is valid as long as the pipeline hash matches, and the hash covers transformers/ (with lib_types.json) but not the lib sources. A route read off the live lib would let an edit of lib/ta.py change a builtin from stateless to state-carrying while every cached script that calls it keeps the old emission, and crash on the missing __pyne_layout__. Read from the table, the same edit leaves the cache valid only until the table is regenerated, and the regeneration changes the hash.

When you change a lib function in a way that moves its route — give a builtin state or take its state away, turn it into an @overload group, make it a module property, mark it __pyne_shared_call_site__, or add a name to a module’s __all__ — regenerate the registry and commit it with the change:

python scripts/lib_type_collector.py

tests/t00_pynecore/ast/test_104_lib_types_registry.py regenerates the registry in memory and fails, naming every changed route, until the committed file matches. When you add a new stateless function that compiled scripts call, decide its route in the same change as well: list it in NON_TRANSFORMABLE_FUNCTIONS if it keeps no per-call-site state, or leave it out (it then routes uniform, or fast/direct when it lives in a transformed lib module) and say why in a comment next to the function.

Fast path

The child slot is filled on first call and reused afterwards — the hot path is a single list read:

lib.ta.sma(__st·__ if (__st·__ := __state__[1]) is not None
           else __resolve_slot·__(__state__, 1, lib.ta.sma), lib.close, 14)

Fast path in a loop

A call site inside a loop keeps one instance, shared by every iteration. TradingView keeps one state per call site however often a loop body executes it on a bar (a var counter in a called function counts 1, 2, 3 on one bar and 4, 5, 6 on the next). The whole guard folds into a helper, __loop_state__, which also restores the callee’s builtin machines to their bar-start state before every same-bar re-execution; user state (Persistent variables) accumulates across the iterations:

def main(__state__):
    total = 0
    for length in (5, 10, 20):
        total += counter(__loop_state·__(__state__, 0, counter))

The slot holds a [state, last_bar, snapshot] cell that is private to the helper: the first call of a bar refreshes the snapshot, every further call on that bar restores it.

Uniform path

When the callee is only known at runtime, the call site gets an anchor slot holding a (callee, bound) pair. The emission checks the callee’s identity: on a hit it calls the cached binding, on a miss (first call, or the callee changed) __bind_any__ rebinds with fresh state:

(__b·__[1] if (__b·__ := __state__[7]) is not None and __b·__[0] is f
 else __bind_any·__(__state__, 7, f))(x)

__bind_any__ handles what the callee turns out to be at runtime: state-carrying functions get a fresh state vector baked into a partial, exported library functions are unwrapped, overload dispatchers are bound through their __pyne_bind__ factory, and plain callables pass through as-is. Loop-shaped uniform sites fold the whole guard into __bind_loop__(__state__, 7, f)(x): one shared instance with the same same-bar rollback as the fast loop form.

Python forbids assignment expressions anywhere inside a comprehension’s iterable, so a straight-line site under an iterable folds its guard into one helper call instead (__slot_state·__ for the fast path, __bind_slot·__ for the uniform path).

Note: when the callee at a uniform site changes (g = a if cond else b; g(x)), the site is rebound with fresh state — state does not survive an a -> b -> a swap.

Call Site Identifiers

Each isolated call site gets an identifier built from the scope path, the callee path and an ordinal — for example main·lib.ta.sma·0. These appear in the layout’s children tuple and in the debug names, so a state tree can be navigated by eye. The Unicode middle dot (·) separator prevents collisions with underscores in function names.

The same separator guards every name the transformer injects into script scope: the generated temporaries (__st·__, __b·__, __i·__, __cnt·0__) and the runtime helper imports, which are aliased on the way in (from pynecore.core.instance_state import __resolve_slot__ as __resolve_slot·__). A script variable spelled like a helper (__slot_state__) therefore neither shadows the injected name nor overwrites it.

The middle dot is a legal identifier character in Python, so the namespace is closed from the other side as well: the import hook rejects Pyne code that spells · in any identifier, with a SyntaxError pointing at the name. Strings and comments are unaffected — only identifiers are reserved.

Example: Transformed Code

Original code:

"""
@pyne
"""
from pynecore import Series

def main():
    def t():
        a: Series[float] = 1
        a += 1
        return a[1]

    a = t()
    print(a)
    b = t()
    print(b)

Transformed code:

"""
@pyne
"""
from pynecore.core.instance_state import __resolve_slot__ as __resolve_slot·__
__pyne_slot_layout__ = {'main': {'init': (None, None), 'series': (), 'varip': (), 'children': ((0, 'main·t·0', False), (1, 'main·t·1', False)), 'names': ('main·t·0', 'main·t·1')}, 'main·t': {'init': (None,), 'series': ((0, None, 'float'),), 'varip': (), 'children': (), 'names': ('a',)}}

def main(__state·main__):

    def t(__state__):
        a = __state__[0].add(1)
        a = __state__[0].set(a + 1)
        return __state__[0][1]
    t.__pyne_layout__ = __pyne_slot_layout__['main·t']
    a = t(__st·__ if (__st·__ := __state·main__[0]) is not None else __resolve_slot·__(__state·main__, 0, t))
    print(a)
    b = t(__st·__ if (__st·__ := __state·main__[1]) is not None else __resolve_slot·__(__state·main__, 1, t))
    print(b)
main.__pyne_layout__ = __pyne_slot_layout__['main']

The two call sites of t got child slots 0 and 1 of main’s state vector, so the two calls keep fully independent series buffers — exactly the Pine Script behavior. Note that main itself takes a state vector too (__state·main__, qualified because it contains a nested definition); the runner provides it as a root vector.

For decorated definitions the layout attach uses a decorator form (@__attach_layout__(...) in the innermost position) so the layout lands on the raw function before any other decorator wraps it.

Performance Considerations

The slot scheme was designed for the bar-by-bar hot loop:

  1. After the first call, reaching a callee’s state is a single list index read — no dictionaries, no key construction, no cache lookups
  2. Function objects are never recreated or wrapped at runtime; the emitted code calls them directly with the state vector as hidden argument
  3. Only provably stateful call sites pay the slot check; stateless calls run untouched
  4. State slots are read and written with literal int indexes — the fastest state access Python offers
  5. Dropping an instance (e.g. on reset()) releases its whole subtree through normal garbage collection — no global registry to invalidate

Summary

Function isolation is a fundamental PyneCore feature that enables accurate Pine Script compatibility. By giving every function instance its own state vector, parked in a slot of its caller’s state vector, PyneCore ensures that indicators, strategies, and other scripts behave correctly in the bar-by-bar execution model — with hot-path costs reduced to plain list indexing.