Phase 3: Import and migration tooling

Status: approved by the owner on 2026-09-29. Rulings I-1..I-8 are the owner’s; I-9 and I-10 were taken as recommended with no objection and may still be changed. Every ruling is binding. Branch phase-3-import, based on main 8db4dd50. Roadmap: Phase 3, “Import / migration tooling”. Architecture spec §19 (Phase 3), §19.1, §11A.

Why this lane is first

§19 calls import “a go-to-market requirement, not a convenience: nobody leaves AutoCount if their history cannot come with them”, and §19.1 puts import and layout design ahead of everything else in Phase 3 because they are what let a reseller deploy without engineering.

What exists today (surveyed 2026-09-29)

• Two importers, and neither is general. Bank statements take a mapped CSV (platform/banking: CSVMapping, ParseCSV, 5 MB, file SHA-256 dedupe). Budgets take a fixed-header CSV and report every bad line at once (accounting/budget/csv.go: CSVError, 1 MB, 5000 rows, 50 errors), then save through the same function the screen uses. That second pattern is the one to generalize.
• No XLSX reader. excelize/v2 is in go.mod but only tests import it; the XLSX writer is hand-rolled (reporting/render/xlsx.go).
• Chart of accounts: POST /v1/accounting/accounts (accounting.account.manage). Codes are unique per org; control kinds, system_role (retained_earnings only) and owner modules are fixed once set. The MY country pack seeds the chart at provisioning step 70 and never seeds over an existing one.
• Parties: customers and suppliers are created one at a time (CreateCustomer, CreateSupplier); dedupe is on code only. legal_name must be English or Malay (00024). Supplier bank details are maker-checker (bank_verifier_differs).
• Opening balances: only a manual journal with opening: true, dated on the first day of the earliest fiscal year, and on subledger accounts only for sst_payable / wht_payable (ledger/post.go). Manual lines on ar, ap, ap_trade, employee_advances, petty_cash_ap and owner-module accounts are always refused. There is no web form (DEPLOY_NOTES).
• The subledger is derived from documents. Aging, statements and allocation read invoices, credit notes, receipts and payments, and a drift test requires Σ open items per party to equal the control account per party (receivables/subledger_test.go; AP has the same rule).
• Fiscal years are contiguous and none can be created before the earliest one. Provisioning creates the year containing today.
• Documents: references are allocated gap-free (INV-2026-0001), UNIQUE (org_id, reference); an old AutoCount number cannot be kept as the reference. Issuing a sales document in an in_scope organization queues its MyInvois submission in the same transaction (einvoice/guard.go).

Owner rulings (2026-09-29)

• I-1. Import tooling is the first Phase 3 lane.
• I-2. Unpaid customer and supplier balances come in as one opening document per unpaid invoice (and per unapplied credit note or unallocated payment), not one total per party, so aging, statements and allocation work from the first day.
• I-3. Fully settled documents from before go-live come in as a read-only archive: searchable and viewable under their original numbers, never posted to the ledger.
• I-4. Files are CSV or XLSX, mapped column by column, with saved mappings and downloadable templates. AutoCount’s own export formats wait for real sample files.

The go-live model

Three dates:

• S, the start of the organization’s earliest fiscal year (the go-live anchor ledger.Post already uses).
• C, the cut-over date: the last day the old system is the book of record.
• G = C + 1, the first day this system is.

S ≤ G, and G falls inside the earliest fiscal year. A go-live on the first day of a fiscal year is S = G.

What is imported, and how it lands in the ledger:

#	Import	Ledger entry	Dated
A	Trial balance at S − 1 (prior year end), balance-sheet accounts	One opening entry, source_kind = 'opening_balance'	S
B	Net movement per account for each fiscal period from S to C (only when G > S)	One entry per period, source_kind = 'migration_period'	that period’s last day
C	Open items at C: unpaid invoices, unapplied credit notes, unallocated payments	One posting per opening document, through AR/AP posting rules	C

The migration clearing account. A and B never touch a subledger control account. Every amount the old trial balance shows on ar, ap or ap_trade goes to a new equity account with system role migration_clearing instead. Each opening document C posts its own control line against the same clearing account (Dr ar / Cr clearing for a sales invoice). If the old books are consistent, clearing nets to exactly zero once A, B and C are in. That is the migration’s proof: the old trial balance’s receivables and payables agree, to the cent, with the open items that replaced them.

Why this shape:

• The subledger rule holds without exception. The only lines ever posted to ar or ap come from documents, as today. No manual control-account line is ever allowed.
• This year’s profit and loss is complete. B carries the months before go-live period by period, so the year’s P&L, SST and budget-vs-actual are right, not missing January to September.
• Nothing before S is needed. No fiscal year is created before the earliest one.
• One known distortion. For periods before G, the balance sheet shows receivables and payables inside Migration clearing, not on their own lines (I-7 asks the owner to accept this).

Finishing the migration. “Finish migration” refuses unless clearing is zero, and records who finished it and when (audit row). After that, no A, B or C import is accepted. The first period close also refuses while an unfinished migration has a non-zero clearing balance. Reopening is I-10.

Lanes and migration ranges

Phase 3 takes 00600–00699. This lane:

Lane	What	Range
M1	Import framework: jobs, parsing, mapping, validate-then-commit	00600–00609
M2	Chart of accounts	00610–00619
M3	Customers and suppliers	00620–00629
M4	Opening trial balance and period movements (A, B), clearing account, finish	00630–00639
M5	Opening documents (C)	00640–00649
M6	Historical archive	00650–00659
M7	Web: import screens and the migration checklist, saved mappings	00660–00669

M2 and M3 can be built in parallel after M1; M4 needs M2; M5 needs M3 and M4; M6 needs M3.

M1. Import framework

• import_job (tenant table, append-only): one row per committed import: id, org_id, kind, file_name, file_format, file_sha256, file_bytes, mapping jsonb, row_count, committed_by, committed_at. It is the evidence of what was imported, kept like any record. kind is a vocabulary string (chart_of_accounts, customers, suppliers, opening_trial_balance, period_movements, opening_sales_documents, opening_purchase_documents, archive_documents); each lane adds its own.
• Saved mappings (import_mapping, per (org_id, kind, name), so a reseller maps an AutoCount export once) arrive with the web screens in M7, their only user. Until then a request carries its mapping, and a file whose headers are the template’s needs none.
• Upload is JSON with content_base64, like bank statements, up to 5 MB. Import files are not receipts: they do not go through documents, whose allowed types are images and PDF.
• Parsing: CSV through encoding/csv with the banking package’s encoding rule (UTF-8, BOM stripped, ISO-8859-1 fallback); XLSX through excelize (first sheet, or a named one), which moves from test-only to runtime. Both produce the same [][]string; nothing below the parser knows which it was.
• Two calls, one code path, like the budget import:
  1. Validate is a dry run: it parses, maps and saves every row through the same functions the screens and API use, inside a transaction that is then rolled back. It returns every line error with its file line number, capped at 200, and stores nothing.
  2. Commit sends the same file and mapping again and runs the identical code in a transaction that commits: all rows or none. It writes the import_job row and one audit row naming it. There is no second set of business rules, so a file that validates clean commits the same way (the only difference is data that changed in between).
• Idempotency: the same (kind, file_sha256) cannot be committed twice (409 import_already_committed, naming the earlier job).
• Limits: 5 MB, 20 000 rows. Larger migrations split the file.
• Permission: imports.run, granted to the Owner role only (I-9).
• Refusals are coded, with en/ms/zh sentences, like every Phase 2 refusal.

M2. Chart of accounts

• Columns: code, name, parent_code, type, header (yes/no), currency (optional), maps_to (optional).
• The seeded MY chart stays. A row whose code already exists is refused unless it maps to the same type, in which case it is a no-op; the file is not allowed to rename the seeded chart.
• Mapping instead of creating. maps_to names an existing account by its code: a seeded control account (Trade receivables 1540, Trade payables 2105, a bank account, SST payable 2140, …), retained earnings, or any other account the customer wants to keep. No account is created; the file’s code is stored as an alias (account_alias, unique per org and never equal to a real account code), so later imports (A, B, C) can name accounts by the old system’s codes. The row’s type must match the target’s.
• Imported accounts are always ordinary. The file has no control-kind or system-role column, so control and system accounts can only be mapped to. If the old system’s debtors control account is created instead of mapped, its trial-balance amount lands on an ordinary account, and Migration clearing does not reach zero: “Finish migration” refuses, which is the intended catch.
• Parents may appear after children in the file; the importer orders by depth.
• An account name follows chart.CreateAccount’s rules, as from the screen. Account names are not record text today, so I-8 does not refuse a Chinese account name.

M3. Customers and suppliers

• One kind per file. Columns follow CreateCustomer / CreateSupplier: code, legal_name, registration_type, registration_no, tin, sst_registration_no, address…, email, phone, credit_terms, and for suppliers the bank fields.
• Create only. An existing code is a line error. Updating records by import is out of v1.
• Supplier bank details arrive unverified. The existing maker-checker applies: a second member verifies them before any payment. Import does not bypass it.
• Possible duplicates are warnings, not errors: a row whose TIN or registration number already belongs to another party is reported and still created. Merging parties (the AR spec defers it here) is out of v1.
• Record language: legal_name must be English or Malay (§11A.3). A Chinese-only name is a line error. See I-8.
• As built: the fields are code, legal_name, party_kind, tin, registration_type, registration_no, country_code, address_line1, address_line2, city, postcode, state, email, phone, payment_terms_days, currency, tax_code, active, plus sst_registration_no, revenue_account for customers and expense_account, wht_code, bank, bank_account_no, bank_account_name for suppliers. Only code and legal_name are required; the rest default as the screen does (organization, MY, 30 days, the base currency, active). Accounts resolve by code or by an M2 alias; a bank by its list key or name. Each row runs in its own savepoint, so a database refusal (a duplicate code) is reported on its line and the rows after it are still checked. Warnings come back beside the counts and never stop a commit.

M4. Opening trial balance and period movements

• A: columns account (code or alias), debit, credit as at S − 1. Balance-sheet accounts only; a P&L account is a line error (its prior-year balance is already in retained earnings). Debits must equal credits.
• B: columns period (YYYY-MM or the period's name), account, debit, credit, one row per account per period from S to C. Each period must balance. Periods from G onward are refused.
• Control accounts: amounts on ar, ap and ap_trade are posted to Migration clearing instead, and the job records the redirected totals. Amounts on the other refused kinds (employee_advances, petty_cash_ap, owner-module accounts) are line errors in v1: those balances must be settled in the old system before cut-over, or entered through their own flows.
• Ledger change: ledger.Post gains source_kind = 'migration_period'. It is allowed only dated from S to C, only for an organization whose migration is not finished, only with imports.run, and never on a control or owner-module account. The opening rule is unchanged.
• Provisioning: a step migration_clearing (order 79) creates the equity account with the new system role; account_system_role_known gains it.
• Periods before G must be open to post A and B. They may be closed once the migration is finished.
• As built: the migration’s state is org_migration (one row per organization, amendable, every change audited): C is set by PUT /v1/imports/migration and must be S − 1 or the last day of one of the earliest year’s periods but the last, so every period before G is whole and G stays inside that year. It cannot move before a period movement already posted. A is columns account, debit, credit; B is period (YYYY-MM), account, debit, credit (fiscal periods have no names). Blank or zero rows are skipped, amounts are never negative or finer than a cent, and v1 takes base-currency accounts only. A second A, or a period already in B, is refused until the earlier entry is reversed. The redirected totals come back as redirected and are in the commit’s audit row. SST and WHT payable take their balance through A only; a movement on them in B is a line error (the manual-line rule). The close refusal is a reconciliation check, migration_clearing, so it shows beside the others. The ledger also refuses any other manual line on Migration clearing (migration_clearing_manual), so the migration’s proof cannot be fixed by hand. Finish and reopen (I-10) are POST /v1/imports/migration/finish and /reopen.

M5. Opening documents

• Kinds:
  • customers: unpaid sales invoice, unapplied credit note, unallocated receipt
  • suppliers: unpaid purchase invoice, unapplied credit note, unallocated payment
• They reuse the existing tables, not a new one, so aging, statements, allocation and receipts work unchanged. New columns on sales_invoice, purchase_invoice, customer_receipt and supplier_payment:
  • is_opening boolean NOT NULL DEFAULT false
  • legacy_reference text (the old system’s number, unique per org and kind, searchable, printed)
  • legacy_einvoice_uuid text (optional)
• One amount, not lines: an opening document carries its outstanding amount at C, in its currency and base, plus its original document date, due date and original total for reference. Its tax is already in the trial balance, so it posts no tax.
• Posting: AR and AP each register a posting rule for opening documents: Dr ar / Cr Migration clearing, and the mirror for payables, dated C, with party_id set so the subledger rule holds. Modules still never write journal lines themselves.
• They enter in the issued / approved state directly, with no workflow run: they were approved in the old system. The audit row names the import job.
• Numbering: each gets a reference from its own gap-free sequence (OB-INV-0001, OB-PI-0001, …), so opening documents never consume the organization’s real INV sequence. The legacy number is what users see first.
• MyInvois: an opening document is never submitted. Because it enters without the issuing transition, einvoice/guard.go never queues it. A drift test asserts that no einvoice_submission row ever references an opening document, with the presence check that a normally issued invoice in the same org does get one.
• Aging uses the original document and due dates, so a two-year-old unpaid invoice ages as two years old.
• As built: two kinds, opening_sales_documents (type invoice, credit_note or receipt; customer, number, date, due_date, currency, fx_rate, outstanding, original_total, einvoice_uuid) and opening_purchase_documents (supplier and supplier_invoice_no in their place). Saved by receivables.CreateOpeningDocument / CreateOpeningReceipt and payables.CreateOpeningInvoice. The workflow instance is created directly in the posted state by workflow.StartPosted (no decision, no guard, nothing published), then the document is posted by posting.Dispatch through the ordinary rules. The accrual rules have an opening branch (control against clearing, one amount, on the event’s date), and an opening receipt deposits to Migration clearing, so the receipt rule already posts it right. Each takes its reference from its own sequence (opening:<kind>: OB-INV-2026-0001, OB-CRN-, OB-RCT-, OB-PI-, OB-CN-). supplier_payment gets no opening columns: AP has no payment on account (a payment must be fully allocated), so a supplier prepayment comes in as an opening supplier credit note, whose effect on what is owed is the same. A receipt is base-currency only; a foreign-currency document needs fx_rate. An opening customer credit note names no original invoice, the one relaxation of sales_invoice_credit_note_shape. The legacy columns are frozen by trigger. When go-live is the first day of the year (C = S − 1, which no period covers), the documents post on S with the opening balance.

M6. Historical archive

• archived_document (tenant table, append-only, seven-year retention list): kind, legacy_reference, document_date, party_id (when matched) or party_name, currency, total, lines jsonb, legacy_einvoice_uuid, import_job_id, plus optional attachments through the existing documents store.
• Never posted, never in any report or balance. It is evidence, not accounting. Search finds it by legacy number and party; the party page lists it.
• Record text must be English or Malay, as for every record (I-8).
• As built: two kinds, archive_sales_documents and archive_purchase_documents, with type (invoice, credit_note, and receipt or payment), number, date, customer or supplier (a code) or party_name (a party not carried over), currency (the base one when blank), total, einvoice_uuid, and line_description, line_quantity, line_unit_price, line_amount. One row per line; rows with the same type and number are one document and must agree on everything but the line. kind is the live document’s (sales_invoice, customer_receipt, supplier_payment, …). A document is dated on or before C once C is set (else today), is refused when its number is already archived or was imported as an open item of the same kind, and is written by the import package itself (its own table, like import_job). Read at GET /v1/imports/archive/sales and /purchases (q matches the old number, the old party name or this system’s, plus kind, party_id and a cursor), and /{id}, behind receivables.invoice.read and payables.invoice.read, not imports.run. Scans attach at POST /v1/imports/archive/{sales,purchases}/{id}/attachments (imports.run): a document the caller uploaded, recorded in archived_document_attachment (00651, append-only) and answered for by the archived_document binder, so one scan backs one record in all consumers and is read by whoever reads that side’s invoices.

M7. Web

• Settings → Import: pick a kind, download its template, upload, map columns (or pick a saved mapping), see the validation errors as a table of line, column and message, then commit.
• Migration checklist: the steps in order (chart → customers and suppliers → opening balance → period movements → opening documents → archive → finish). It shows the live Migration clearing balance and, once finished, who finished it.
• Strings in en, ms and zh (i18nparity).
• As built: /imports (Manage → Import), /imports/migration (Manage → Migration) and /archive (Accounting → Archive), plus an archive section on the customer and supplier drawers. The screen reads a file’s header row through POST /v1/imports/headers (nothing stored) and matches each field to the header spelled like it; the template is the kind’s field names from GET /v1/imports/kinds, built in the browser. Import is enabled only after a clean check of the same kind, file, sheet and mapping. Saved mappings are import_mapping (00660, one per kind and name, operational: freely replaced and deleted, since each import keeps the mapping it used on its import_job row), at GET/PUT /v1/imports/mappings and DELETE /v1/imports/mappings/{id}. Line problems are the server’s English sentences; every coded refusal has an en/ms/zh sentence. The archived document’s dialog lists its scans and, for the Owner, uploads and attaches more.

Drift tests this lane adds

Each is written presence-first and seen failing once (AGENTS.md):

• Every import commit uses the domain save functions: a guard that the import package issues no INSERT into business tables of its own.
• Opening documents are never submitted to MyInvois (M5).
• Subledger equals control per party after a migration (reusing the AR/AP drift tests, with opening documents in the fixture).
• Migration clearing is zero at finish, and finish refuses otherwise.
• migration_period entries are refused on control accounts, after G, and after finish.
• Validation and commit agree: a file that validates clean commits, and one line changed to break a rule fails both with the same line number.

Out of v1

AutoCount’s native export formats (need samples); updating existing records by import; merging duplicate parties; employee advances, petty cash floats and payout profiles; bank statement history (banking has its own importer); inventory, fixed assets and payroll (Phases 4 and 5); import through the operator console.

Further rulings (2026-09-29)

• I-5. Go-live may fall in any month. An opening balance at S plus period movements up to C, as described in the go-live model. Go-live is not restricted to the first day of a fiscal year.
• I-6. An opening document carries its outstanding amount at C. The original total is kept for reference only; payments already received in the old system are not re-entered.
• I-7. Accepted: balance sheets for periods before G show receivables and payables inside Migration clearing. It is the price of never allowing a manual line on a control account.
• I-8. Chinese-only names and descriptions are refused on import (§11A.3: record text is English or Malay). The error names the line and column, so the customer adds a romanized or Malay name. A Chinese display name can still be added as a zh translation after import where the entity has translation rows.
• I-9. Only the Owner may import (imports.run). It can be granted to another role like any permission. (Taken as recommended.)
• I-10. A finished migration may be reopened by the Owner, with a mandatory reason, while no period from G onward is closed; audited. (Taken as recommended.)