Skip to content

The Insights Piece — anatomy, data-load gate, and rollout

The Insights piece (built by Mark Wright) is a reusable form section that shows a coloured grid of property insights (icon, kind, type, text, clients-affected) loaded from the property in question. It is intended to replace the old ad-hoc property alerts / warnings and be added to all regularly-used forms.

A standalone Insights form also exists, so insights for a property can always be viewed even on forms that don't yet embed the piece.

Anatomy — three XML parts (canonical: Insights_59.form, also Routine Tenancy Visit_13)

  1. GenBPInsight group — two-part:
  2. a top-level <FormItem type="Group" name="GenBPInsight"> (the binding handle), and
  3. a <Group name="GenBPInsight"> block inside <Groups> carrying the member items z_Insight_Emojicon(+ColouredInternal), z_Insight_Kind(+Coloured), z_Insight_Type(+Colour), z_Insight_Text(+Coloured), z_Insight_ClientsAffected(+Colour), z_Insight_Colour, z_Insight_Detail, z_Insight_Type_Negation. The *Coloured/*Colour items are base.fontcolor(z_Insight_Colour) calcs.
  4. Toggle logic (top-level items):
  5. z_Insights — reads GetConfigurationValue("Insights") (the global on switch)
  6. z_Insights_Switch — option list Show Insights;Hide Insights; (user toggle)
  7. z_Insights_Visible — computed: Show if switch=Show, or if config=True and switch is null; else Hide.
  8. Display run in the Main page FormElements (placed after the form header):
  9. the Show/Hide switch control — an OptionsListRadioButtonElement bound to z_Insights_Switch (push-buttons "Show Insights" / "Hide Insights"). This is a separate display element from the z_Insights_Switch data item — the data item alone gives you no on-screen toggle (see the "missing switch" bug below).
  10. a gate SectionBreakElement whose Visible = z_Insights_Visible == "Show Insights",
  11. the GroupDataGridElement bound to GenBPInsight (caption "Insights", 5 columns). Set showAdd=False (no Add button — users must not add insight rows) and keep showEdit=True (users tap a row to view its detail). Per Mark in the catch-up: "we don't need the show ad… not going to be adding anything" / "they want to edit because it allows them to go into it and have a look inside."
  12. a trailing spacer SectionBreakElement.

"Missing switch" bug (fixed 2026-06-05): the original injector added the z_Insights_Switch data item but not the switch display control, so injected forms rendered the panel with no toggle and (config defaulting on) showed it unconditionally. inject.py now always adds switch_control.frag.

Legacy property-alerts inverse-hide: the old "Property Alerts" section is gated to hide when insights are shown — its Visible becomes z_Insights_Visible != "Show Insights" (kept in place, not deleted, so toggling insights off restores it). Many forms also carry other warning sections (e.g. "Property Warnings", "Client Warning", "Alerts and Warnings") that still need manual tidy-up per Peter.

Data-load gate (critical gotcha)

The grid only shows rows if the task the form runs in has its GenBPInsight group filled at runtime ([Versaa].[usp_all_insights], keyed on GenBP_Key_PropertyId). There are three ways that happens (see the verified route→loader map):

  1. Loader-route — the launch/load workflow runs the insight query (Property Search, FRA Search, RWO QL Data, Client Visit Load, Estate Search, Order Appointment).
  2. Inherited — the form runs inside a parent task that was already loaded (e.g. RWO sub-tasks UvHWS / NIBE / Damp & Mould inherit the RWO's insights). Display-only.
  3. Self-trigger — the form fires *.AppendPropertyInsights itself: a RequestPropertyInsights request item (messageName=AppendPropertyInsights, items=GenBP_Key_PropertyId;, copied from Family Survey 2 MW) + an autoSend RequestButtonElement (the fire-once, self-hiding pattern from Asbestos Data). Use this only for forms whose task is not already loaded (e.g. Void Survey via Create Jobs).

Consequences:

  • Adding the display grid to a form does not guarantee data appears — confirm in the app, per form.
  • ⚠️ Duplication rule: the Append Property Insights handler appends, it does not replace. If a task gets insights from two sources you get duplicate rows. Putting a self-trigger on an inherited sub-task (e.g. UvHWS inside RWO) duplicates — autoSendOnceOnly and hiding the button do not fix it because the copies come from different sources. So: inherited sub-task → display-only; self-trigger → only where nothing else loads.
  • Forms with no property context (e.g. Van Check) show an empty grid — skip.

Verified 2026-06-05: Unvented Hot Water Storage v9 is Active in Dev with display-only (inherits from the Repair Works Order) — panel populates, single copy, Show/Hide works, Add disabled, click-in to view works, tasks still sync. Note: Management Studio assigns the next sequential version on import — the manifest version in the .ftpackage is cosmetic.

The workflow-side load is gated by the Insights_OnSwitch workflow variable (workflow_variables/…/Insights_OnSwitch.variable, "1 = On, 0 = Off", currently true) — a single global variable, not a per-form or Management-Studio toggle. Don't confuse it with the form items z_Insights_Switch (the user's Show/Hide toggle) or z_Insights (GetConfigurationValue("Insights")). See insights-rollout-plan.md "the three switches".

Audit (2026-06-04, against a freshly re-pulled snapshot) — which forms have insights

Of 206 forms (latest version each), 27 have the insights grid (count rose from 25 once the stale snapshot was refreshed — e.g. Stock Condition Survey appeared at v48 with the full piece). Re-run this audit only against a fresh Dev pull (see the snapshot-lag gotcha). The authoritative test is a GroupDataGridElement bound to GenBPInsight with Column formItemName="z_Insight_..." (column spelling varies by form age — z_Insight_EmojiconColoured vs ...ColouredInternal, ClientsAffectedColour vs ...Coloured — so match the binding, not one column name).

Already done include: Repair Works Order, Client Visit, Building Safety Check, Awaabs Triage, all Asbestos forms, Pre-Leaving Visit, Routine Tenancy Visit, Fire Risk Assessment, Red Flag Reporting, Balcony Inspection, and others. FRA Review was completed by Mark as v68 (all_versaa_forms/FRA_Review_04062026.ftpackage).

Rollout — who does it, and how (corrected 2026-06-04)

The rollout is owned by Mark and done MANUALLY in Management Studio Dev, form-by-form, prioritised by usage. This is what the catch-up actually agreed (Mark copies the group between forms in the MS UI; "copy that entire section and paste it across"). There is no programmatic-package step in the agreed process.

Peter's "list" is a May usage/completion report — NOT a "needs insights" list. It ranks how many of each task type were completed in May, so Mark can do the most-used forms first. Peter: "it's not 100% accurate… but it gives you a good idea of what is used the most." Treating it as a to-do list of forms-missing-insights was a misread.

Retired experiment: the programmatic injector

An insights_injector was built (2026-06-04) to inject the piece and emit one .ftpackage per "gap" form from the usage list. It has been stood down — do not import its packages. Three reasons it was wrong: 1. Built on a stale snapshot (see gotcha below) → its packages would revert Mark's newer Dev versions on import (it built SCS v44 while Dev was already on v48; Void Survey v28 vs Dev v30; Client v69 vs Dev v71). 2. Sub-form breakage — Client v69 (a sub-form) broke task sync (see next section). 3. The grid is inert without workflow work (data-load gate) — injecting form XML only does half the job; the launch-workflow data-push is a manual per-form task.

The packages now live in versaa-rag/insights_injector/out/packages/_retired/. rollout.py is now audit-only by default (build is opt-in behind --build and still must not be imported without reconciling versions with Mark). The useful part that remains is the audit (classify_forms.py → insights_audit.csv) for tracking coverage: - versaa-rag/insights_injector/insights_rollout_status.md / .csv — current gap list - versaa-rag/insights_injector/insights_audit.csv — raw tier per form

Snapshot-lags-Dev gotcha (critical for any audit)

Exported .ftpackages — and therefore everything in extracted/ — trail the live Dev state. The MS export captures the latest exported version of each item, but Mark's in-progress edits live in Management Studio and aren't in a package until re-exported. Concretely: today's first export captured Stock Condition Survey v43 (no insights) while Dev was actually on v48 (full insights); the CLAUDE.md §10 log records the same for Void Survey ("package captures v27; Dev is on v30 — MS-only v28–v30").

Always re-export from Dev and re-extract (run_dev_pull.py, or a fresh export + extract_all_categories.py) immediately before auditing or doing anything version- sensitive. Auditing a stale snapshot is exactly what produced the false "these forms lack insights" list.

Held / excluded (still valid): Van Check (confirm property context), Gas Servicing (skip — offshoot of Works Order), FRA Review (done by Mark). Sub-form sections (no standalone launchable form, inherit from parent): Safeguarding Report, Estate Inspection, Pets Questionnaire, Client (sub-form of Client Visit).

CRITICAL gotcha — never inject the piece into a sub-form (2026-06-04)

Client_69_insights broke task sync in Dev. Root cause: Client is a sub-form, not a standalone form, and its parent already has Insights.

The Client form (category Tenancy) is loaded as a sub-form by Client Visit (via the Client Subform Load item) and is reached through the CRM / EDI flows. Client Visit v100 already carries the full Insights piece (GenBPInsight group + 25 z_Insight_* items), so Insights already renders in the Client area at runtime — inherited from the parent.

Injecting the piece into the Client sub-form (v69) added a second GenBPInsight group + duplicate z_Insight_* items. When the device merges the sub-form into the parent it sees the same group/items declared twice → the form fails to provision on the device (Management Studio import still succeeds — the file is well-formed, GUID preserved, all formula ids unique/valid, no parse error). Symptom: tasks created via Client Search did not sync to Aareon Mobile while v69 was active; reverting to v68 released them immediately.

Rules: - A form's eligibility for injection is not just "lacks the grid" — it must be a standalone launchable form. Sub-forms must be excluded, because they inherit the piece from their parent and a duplicate group breaks the runtime merge. - Detect sub-forms by searching all forms for a name="<FormName> Subform Load" item (the load-control convention) — if any parent loads the target, do not inject. - A clean MS import does not mean the form works: device provisioning is a separate compile step. Confirm new tasks still sync after activating a new version. - Client was the only sub-form among the injector targets. All injector packages are now retired (out/packages/_retired/); none should be imported. In current Dev, Client is on v71 and inherits Insights from its parent Client Visit.

classify_forms.py now carries the sub-form exclusion (it scans for <X> Subform Load items and flags any target loaded by a parent as is_subform), and rollout.py excludes those automatically.

The injection toolkit

versaa-rag/insights_injector/ (run with versaa-rag\.venv\Scripts\python.exe):

Script Purpose
classify_forms.py Tier every latest-version form (DONE / NONE) → insights_audit.csv
merge_usage.py Join Peter's May counts to the audit → usage-prioritised gap list
extract_reference.py String-slice the canonical blocks from Routine Tenancy Visit_13 → templates/
inject.py Idempotent string-splice of the missing components into a form, regenerate all formula/element ids, place the grid after the header, gate Property Alerts, validate
build_package.py Wrap a modified .form into a single-form .ftpackage (CLAUDE.md §8, UTF-8 no BOM) — used only behind rollout.py --build
rollout.py Audit-only by default — writes the gap-list tracking tables. The inject+package path is SUPERSEDED and gated behind --build; do not import its output.
components.py / diff_conversion.py Inspection helpers

Supported use today = the audit. classify_forms.py (coverage tiers + sub-form detection) and the rollout.py tables are the useful, maintained outputs. The inject/build path is kept for reference only — the rollout itself is Mark's manual job.

Why string-splice, not lxml: an lxml round-trip rewrites the whole file's formatting (<Value></Value> → <Value/> etc., ~20k diff lines). The toolkit edits the raw text so each form stays byte-identical except at the insertion points (verified: purely additive, zero content deletions). Every injected formula id is regenerated to a fresh 32-lowercase-hex value and every element GUID re-minted, so there are no id collisions (CLAUDE.md §8). Validation tolerates pre-existing duplicate ids (Client and Complaint ship with them and import fine) — it only fails on duplicates the injection itself introduces.

To refresh the audit: first re-pull Dev (run_dev_pull.py, or fresh export + extract_all_categories.py), then python versaa-rag\insights_injector\classify_forms.py and python versaa-rag\insights_injector\rollout.py (audit-only). Never audit a stale snapshot — see the snapshot-lag gotcha.

Deployment & control docs

The injector packages are retired and must not be deployed. Mark adds the piece directly in Dev and migrates it Dev → Test → Live per CLAUDE.md §9 (confirm grid populates + Show/Hide toggle + legacy alerts hide at each stage; the grid needs the launch-workflow data-push to show data). The rollout.py audit tables are the coverage tracker, not a deploy queue. Control documents (CLAUDE.md §12) are updated per form as Mark confirms each in Dev.