Simulations#
This approach is tailored to situations where the systems under test are virtual and testing occurs through modeling and simulation. This approach handles situations such as those where simulation time is preferred to wall clock time, there are a large number of varied test cases with different parameters, and most analysis is statistical rather than unit level.
While very similar to the tagged approach, the biggest difference and advantage of this approach for the simulation use case is that every run of a model lines up on the same timeline, so comparing runs takes no time alignment. The drawback is that, as with the tagged approach, every point must be correctly tagged when loaded into Nominal.
Implementing this approach#
In this approach, each simulation model gets one dataset. Every build and every execution of that model writes to that dataset, and each point is tagged with data that allows downstream filtering.
For example, tag each point with model, execution-id, and parameter-set values. These tags pick out the data from one execution, the job that assets do in the tagged approach and runs do in the default approach.
Comparing executions doesn’t depend on when each one ran, so write every execution’s timestamps relative to a common start time, such as 0, and tell executions apart by their tags instead of by time. Every run of a model then overlays every other run with no alignment.
Conceptually, that looks like:
---
config:
layout: elk
---
flowchart BT
classDef green fill:#dcfce7,stroke:#15803d,color:#1f2937
classDef red fill:#fee2e2,stroke:#b91c1c,color:#1f2937
classDef purple fill:#e0e7ff,stroke:#4338ca,color:#1f2937
subgraph dataset1["Dataset: Model A"]
row1["model=A<br>execution-id=1<br>parameter-set=34<br><br>0.0 s · 100.0<br>0.1 s · 101.3<br>..."]:::purple
row2["model=A<br>execution-id=2<br>parameter-set=77<br><br>0.0 s · 100.0<br>0.1 s · 98.7<br>..."]:::purple
row3["model=A<br>execution-id=3<br>parameter-set=34<br><br>0.0 s · 100.0<br>0.1 s · 102.9<br>..."]:::purple
end
style dataset1 fill:#f5f7ff,stroke:#4338ca,color:#1f2937
run1["Run: Model A, execution 1"]:::red
row1 --> run1
run2["Run: Model A, execution 2"]:::red
row2 --> run2
run3["Run: Model A, execution 3"]:::red
row3 --> run3
click row1,row2,row3 href "../../#time-series-data" "Time-series data"
click run1,run2,run3 href "../../#runs" "Runs"
Unlike the other approaches, you don’t create any assets. Each run is created directly from the dataset, filtering the dataset to that run’s tags. Under the hood, Nominal automatically creates an asset to implement the linkage. You can ignore those assets.
---
config:
layout: elk
---
flowchart BT
classDef green fill:#dcfce7,stroke:#15803d,color:#1f2937
classDef red fill:#fee2e2,stroke:#b91c1c,color:#1f2937
classDef purple fill:#e0e7ff,stroke:#4338ca,color:#1f2937
classDef auto fill:#f3f4f6,stroke:#6b7280,stroke-dasharray: 2 2,color:#1f2937
subgraph dataset1["Dataset: Model A"]
raw["0.0 s · 100.0 · execution-id=1<br>0.0 s · 100.0 · execution-id=2<br>0.0 s · 100.0 · execution-id=3<br>0.1 s · 101.3 · execution-id=1<br>..."]:::purple
end
style dataset1 fill:#f5f7ff,stroke:#4338ca,color:#1f2937
run1["Run: Model A, execution 1"]:::red
asset1["Automatic Asset<br>(Ignore)"]:::auto
asset1 --> run1
raw -->|"ref name=default<br>filter model=A<br>and execution-id=1"| asset1
run2["Run: Model A, execution 2"]:::red
asset2["Automatic Asset<br>(Ignore)"]:::auto
asset2 --> run2
raw -->|"ref name=default<br>filter model=A<br>and execution-id=2"| asset2
click run1,run2 href "../../#runs" "Runs"
click asset1,asset2 href "../../#asset" "Asset"
click raw href "../../#time-series-data" "Time-series data"
Because every run shares the same ref name and the same timeline, workbooks can overlay runs of a model without aligning them, and you can execute one checklist on every run to see each check’s result on each run.
---
config:
layout: elk
---
flowchart BT
classDef green fill:#dcfce7,stroke:#15803d,color:#1f2937
classDef red fill:#fee2e2,stroke:#b91c1c,color:#1f2937
classDef purple fill:#e0e7ff,stroke:#4338ca,color:#1f2937
classDef blue fill:#e0f2fe,stroke:#0369a1,color:#1f2937
classDef hidden fill:none,stroke:none
spacer[" "]:::hidden
dataset1[("Dataset<br>Model A")]:::purple
run1["Run: Model A, execution 1"]:::red
dataset1 --> run1
run2["Run: Model A, execution 2"]:::red
dataset1 --> run2
run3["Run: Model A, execution 3"]:::red
dataset1 --> run3
workbook1["Workbook for<br>Execution review<br><br>Points at one run at a time"]:::blue
run1 -.->|"pointing at execution 1"| workbook1
workbook2["Workbook for<br>Execution comparison<br><br>Points at many runs"]:::blue
run1 --> workbook2
run2 --> workbook2
run3 --> workbook2
workbook1 ~~~ spacer
workbook2 ~~~ spacer
click dataset1 href "../../#time-series-data" "Time-series data"
click run1,run2,run3 href "../../#runs" "Runs"
click workbook1,workbook2 href "../../../workbooks/" "Workbooks"
The biggest downside to one run per execution is that with a very large number of executions per model it can be challenging to manage the runs. For larger campaigns, see Large campaigns.
Extensions#
Large campaigns#
Use this extension if a model has hundreds or thousands of executions, as in a Monte Carlo analysis.
To implement this extension, create one run per model instead of one per execution. The run’s data is filtered on model only, so every execution of that model lands in that one Nominal run. In a workbook, use Split by tags in a variable’s Series settings to split the run’s data into one series per execution-id.
---
config:
layout: elk
---
flowchart BT
classDef green fill:#dcfce7,stroke:#15803d,color:#1f2937
classDef red fill:#fee2e2,stroke:#b91c1c,color:#1f2937
classDef purple fill:#e0e7ff,stroke:#4338ca,color:#1f2937
classDef blue fill:#e0f2fe,stroke:#0369a1,color:#1f2937
classDef auto fill:#f3f4f6,stroke:#6b7280,stroke-dasharray: 2 2,color:#1f2937
subgraph dataset1["Dataset: Model A"]
raw["0.0 s · 100.0 · execution-id=1<br>0.0 s · 100.0 · execution-id=2<br>...<br>0.0 s · 100.0 · execution-id=1000<br>..."]:::purple
end
style dataset1 fill:#f5f7ff,stroke:#4338ca,color:#1f2937
run1["Run: Model A<br>(all executions)"]:::red
asset1["Automatic Asset<br>(Ignore)"]:::auto
asset1 --> run1
raw -->|"ref name=default<br>filter model=A"| asset1
workbook1["Workbook for<br>Model analysis<br><br>Splits by execution-id"]:::blue
run1 --> workbook1
click run1 href "../../#runs" "Runs"
click asset1 href "../../#asset" "Asset"
click raw href "../../#time-series-data" "Time-series data"
click workbook1 href "../../../workbooks/" "Workbooks"
The biggest downside to this extension is that individual executions are no longer runs in Nominal, so they can’t have their own events, properties, or checklist results.
Comparing simulations with physical tests#
Use this extension if you want to validate a model against physical test data.
To implement this extension, give the simulation the same channel names as the physical unit’s data and use the same ref name on both sides. If the physical units follow the default approach, that means default. A workbook can then overlay a simulation run and a physical test run channel for channel, with a time offset to line up their start times.
---
config:
layout: elk
---
flowchart BT
classDef green fill:#dcfce7,stroke:#15803d,color:#1f2937
classDef red fill:#fee2e2,stroke:#b91c1c,color:#1f2937
classDef purple fill:#e0e7ff,stroke:#4338ca,color:#1f2937
classDef blue fill:#e0f2fe,stroke:#0369a1,color:#1f2937
asset1["Asset: UUT1"]:::green
dataset1[("Dataset<br>UUT1")]:::purple
dataset1 -->|"ref name=default"| asset1
run1["Run: Test A<br>on UUT1"]:::red
asset1 --> run1
dataset2[("Dataset<br>Model A")]:::purple
run2["Run: Model A, execution 7"]:::red
dataset2 -->|"ref name=default"| run2
workbook1["Workbook for<br>Model validation<br><br>Looks for `default`.(channel)<br>in each run"]:::blue
run1 --> workbook1
run2 --> workbook1
click asset1 href "../../#asset" "Asset"
click dataset1,dataset2 href "../../#time-series-data" "Time-series data"
click run1,run2 href "../../#runs" "Runs"
click workbook1 href "../../../workbooks/" "Workbooks"
If the physical units also share a source that tags its data with the unit under test, such as a ground station, see Combining the default and tagged approaches.