The shortest distinction
Update is a change that inherits the same history, Branch is when multiple histories are run from the same starting point, Stop rule is a pre-rule for what results to cancel or suspend, and Kill switch is an emergency stop method in case of danger. They are similar, but their roles are different.
Why is this distinction necessary?
When it comes to WBE and identity, the system is not fixed. If you learn, it will change, if you copy it, it will separate, and if you use it, you will have to decide whether to stop it. If this is left vague, it is easy to read things too strongly, such as, ``It's better than the previous version, so it's a healthy progression of the same system,'' or ``Even after the divergence, it's all the same person.''
First, separate terms
| Term | Rough meaning | Important points here |
|---|---|---|
| update | Introducing new learning or setting changes to the same system. | If you do not record what has changed in the differences, the meaning of the evaluation will be ambiguous. |
| version | The ID assigned to the fixed state at that point. | It is important to be able to go back and compare without overwriting previous versions. |
| branch | Separate update histories start running from the same starting point. | Since the history after branching is different, it is safer not to mix them in the same evaluation unit. |
| stop rule | These are advance rules that determine which results should be used to cancel, suspend, or withdraw. | It will be stronger as a verification if you fix it first rather than deciding after seeing the results. |
| Kill Switch / Isolation | This is a mechanism that stops or disconnects operations when a safety problem occurs. | Unlike stopping rules, the main purpose is to ensure safety rather than statistical judgment. |
Minimum check to see if it is OK to update
| Check items | The minimum thing I want to do |
|---|---|
| Clarification of changes | Write down the changes made to the data, preprocessing, model, and evaluation metrics. |
| Save previous version | Does not overwrite the previous version, leaving a state where it can be rerun with the same input. |
| Fixing comparison conditions | Compare with the previous version using the same test, the same evaluation metrics, and the same baseline. |
| Recheck for leaks | Recheck that the train/test boundaries and normalization procedure are not corrupted by updating. |
| Handling in case of failure | Decide first where to suspend or cancel updates in the event of performance deterioration or instability. |
Just because your score has increased doesn't mean it's a safe update. Since the numbers can increase due to leaks, evaluation changes, or convenient subset selection, it is necessary to keep a set of differences and comparison conditions.
What to fix when a branch occurs
| What you want to fix | Reason |
|---|---|
| Branch point | This is to clarify which version and data point the branch was made from. |
| Parent-child relationship | Enables auditing of which branch inherited which parent version. |
| Individual ID / Instance ID | This is to avoid confusing multiple histories as the same individual. |
| Evaluation period | This is to distinguish between whether they are similar just after the branch or whether they are similar for a long time. |
| Attribution rules | This is to fix which results are recorded as achievements/failures of which branch. |
Particularly in discussions of identity, it is dangerous to collectively refer to all results after branching as the "principal." At a minimum, you should recordup to what point you treat them as the same unit ofevaluation.
Stopping rules and kill switches are different
| How it works | When to decide | Main purpose |
|---|---|---|
| Stopping rules | Before experiments and evaluation. | This is to avoid leaving the failure judgment and withdrawal line behind. |
| Hold / freeze | This is when the problem was discovered. | This is to temporarily suspend updates and publishing until the cause is determined. |
| Kill Switch / Isolation | Implement it in advance in case of safety issues. | This is to quickly stop any dangerous behavior or deviation. |
``I stopped because the results were bad'' is about stopping rules, and ``I stopped immediately because it was dangerous'' is about safety. The same word ``stop'' has different meanings when it comes to handling evidence and safety measures.
Particularly in closed-loop implementations, safe stopping for delays and abnormal behavior becomes additionally important. If you want to organize only real-time system entrances first, please see Wiki: Closed loop, delay, jitter, and safety stops.
The minimum log you want to keep
Checklist
- version ID:Which version have you just evaluated?
- parent / branch ID:Where you branched from.
- Change Diff:What changed in the data, preprocessing, model, and evaluation?
- Comparison results: Differences, improvements, deteriorations, and uncertainties from the previous version. </li>
- Stopping decision:What rules were touched or not touched?
- Failure example:Conditions that did not work or collapsed.
</ul> </div> </section>3 questions when reading strong arguments
- Are there any differences before and after the update?If it's unclear what has changed, it's safe not to read too much into the meaning of the improvement.
- Do you have a branch ID and observation period? Check if you have combined multiple histories into one.
- Have the stopping rules been determined in advance?Check whether the handling when a bad result is obtained has not been rewritten later.
Where to go back next
If you want to go back to I5 / I8 / V7 of Roadmap, please use Technology Roadmap, if you want to go back to Registry / Audit of verification platform, please use Verification Platform, and if you want to go back to L0 practice, please use Hands-on.