The robustness command runs a strategy’s backtest once and tests how much of its result could be luck. What the tests do and how to read them is in Robustness Tests; this page is the command reference.

Basic Usage

pyne robustness SCRIPT DATA [OPTIONS]
pyne robustness SCRIPT --from-trades TRADE_CSV [OPTIONS]

Where:

  • SCRIPT: The strategy, as Pyne code (.py) or as a Pine Script (.pine) file that is converted to Pyne code first (a PyneSys API key is needed, see Running Scripts)
  • DATA: The data file (.ohlcv, .csv, .json, .txt) or a provider string, as for pyne run
  • TRADE_CSV: A trade CSV written by pyne run, analysed without running the backtest
# Test a strategy with the default settings
pyne robustness ema_cross binance_BTCUSDT_15

# Both coin-flip tests, a 2x target and no leverage in the account Monte Carlo
pyne robustness ema_cross binance_BTCUSDT_15 --coinflip both --target 2 --max-leverage 1

# The exact direction test on 4 processes, every one of its 200 backtests
pyne robustness ema_cross binance_BTCUSDT_15 --exact --early-stop off --jobs 4

The command prints a summary table and writes the results as CSV files.

Command Options

Data Options

The same as for pyne run:

  • --from, -f: Start date (UTC), days back from now, or -N bars back
  • --to, -t: End date (UTC) or days back from now
  • --timeframe, -tf: Chart timeframe in TradingView format (e.g. 60, 1D)
  • --security, -sec: Security data, "TIMEFRAME=data_name" or "SYMBOL:TIMEFRAME=data_name"
  • --api-key, -a: PyneSys API key for Pine Script compilation

Input Options

  • --from-trades: Analyse a trade CSV written by pyne run instead of running the backtest. The file has no stops and no point value, so the results are in account currency, the cost test is skipped, and the coin-flip tests, which run the strategy, do not run.

Test Options

  • --unit: The unit trades are measured in: auto (default; R when every trade has a stop, account currency otherwise), r, currency or percent
  • --draws: Resamples of every random test (default: 10000)
  • --seed: Seed of every random test (default: 0). The same trades, settings and seed give the same numbers.
  • --mc-method: Monte Carlo method: bootstrap (default; draws trades with replacement) or shuffle (reorders the trades)
  • --block: Bootstrap block: week (default), month or trade
  • --confidence: Coverage of the intervals, percent (default: 95)
  • --cost-bps: Extra cost scenarios per side, basis points of the traded value (default: 0,2,5,10)
  • --periods: Calendar breakdowns, any of year (default), month, weekday, comma separated

Account Options

  • --capital: Starting account of the account Monte Carlo (default: the equity the first trade opened with)
  • --risk: Account risked per trade when results in R are compounded, percent (default: 1)
  • --max-leverage: Largest position of the account Monte Carlo of results in R, as a multiple of the account (1 = no leverage). A trade whose stop is too close to risk --risk within the limit risks less. Without the option there is no limit.
  • --ruin: Loss of the starting account that counts as ruin, percent (default: 50)
  • --target: Multiple of the starting account counted as reaching the target (e.g. 2)

Coin Flip Options

  • --coinflip: direction (default; a coin decides long or short on every real entry), timing (the real trades enter at random moments), both or none
  • --coinflip-runs: Backtests of the tests that run one per draw: timing, and direction with --exact (default: 200)
  • --exact: Direction test: a full backtest per draw instead of pairing the trades of one flipped backtest. Needed when the strategy’s later entries depend on its earlier trades: without it, fewer than 90% paired trades give no verdict.
  • --early-stop: When the tests that run one backtest per draw may stop before --coinflip-runs: edge (default; once it is clear whether the real rules beat 95% of the random draws), better (once it is clear whether they beat half of them) or off (every backtest runs)
  • --jobs, -j: Backtests those tests run at the same time, each in its own process (default: the CPU count minus one). The results do not depend on it.

Output Path Options

  • --output, -o: Directory of the result files (default: workdir/output/)
  • --json: Also write the whole result as <script>_robustness.json

Output Files

FileContents
<script>_robustness.csvOne row per measured figure (columns below)
<script>_montecarlo.csvP5–P95 path bands by trade number: running total and account
<script>_costs.csvThe statistics under each extra cost
<script>_periods.csvThe statistics by year, month or weekday
<script>_coinflip_<mode>.csvEvery random total of a coin-flip test, with the real total
<script>_robustness.jsonWith --json: the whole result

The columns of <script>_robustness.csv:

ColumnContents
TestBacktest, Hit rate, Bootstrap, Monte Carlo, Coin flip, …
MetricThe measured figure
UnitR, currency, %, bps or trades
ValueThe figure
Low / P5Lower end of the interval, or the 5th percentile
Median / P50Median of a random distribution
High / P95Upper end of the interval, or the 95th percentile
DrawsResamples or backtests behind the figure
SeedSeed of the random test
MethodHow the figure was made
VerdictThe figure in plain words, where there is one