Skip to Content
Onshore is in early access. Don’t load real customer data until the items in Known limits are closed.
DevelopersTask guidesStore values at the source

Store values at the source

The goal: a sensitive value (an ID number, a bank account, a date of birth) goes into the vault the moment it enters your system, and everything downstream (your database, logs, analytics, support tools, backups) only ever sees a token.

Where to tokenize

Tokenize at the first place that receives the value, and nowhere else:

The value arrives…Tokenize inUse
Typed by a customer in your web or mobile appThe browser or app, so it never reaches youSecure components
In an API request from a partnerYour API handler, before logging or savingtokenize
In a file upload or importThe import job, row by rowtokenize with continueOnError
In free text (a support message, a note)Where the text is receiveddetect with tokenize
In traffic you can’t change (a vendor’s API)The vault’s HTTP proxyHTTP proxy

Make sure the raw value isn’t logged on its way in. Request-logging middleware, error trackers (Sentry, Bugsnag) and APM tools often capture request bodies. Exclude these fields, or tokenize before the logger sees the request.

Choose a token kind for each field

Each field in the vault has a token kind, set in the console when you add the field. It can’t be changed later, because existing tokens depend on it.

Token kindSame value twiceSearchable by valueValue can be updatedChoose it for
Same value, same token (deterministic)Same token, existing: trueAlwaysNoAnything you join, deduplicate or look up: national IDs, emails, phone numbers, account numbers
New token each time (cell)Different tokensIf marked searchableYes, with updateValueValues that change, or shouldn’t be linkable across records: addresses, notes, names
Expiring (transient)New token; stops working after the field’s time limitNoNoValues needed briefly: an ID number checked at onboarding and then not kept

Two more settings matter here:

  • Unique: on a cell field, storing a value already present fails with 409 duplicate_value. Use it to enforce “one account per national ID” in the vault itself.
  • Format-preserving tokens: the token has the same shape as the value (a 16-digit token for a 16-digit number) for systems that validate formats. Opaque tokens (tok_uk_nino_…) are the default and the safer choice, because nobody can mistake them for a real value.

Group values by person with subject_ref

Pass your own customer ID as subject_ref with every value that belongs to a person. It costs nothing and it’s what makes these possible later:

  • finding everything held on a person (searchSubjects),
  • exporting it for an access request, and erasing it on request (Data subject requests),
  • restricting a person so nothing about them reveals.

Without subject_ref, a value can’t be tied back to anyone, so it can’t be found for a request.

A sign-up handler

app.post("/signup", async (req, res) => { const { name, email, nationalId, dateOfBirth } = req.body; const customerId = newCustomerId(); const { id: recordId, tokens } = await vault.insertRecord(VAULT, { full_name: name, email, national_id: nationalId, date_of_birth: dateOfBirth, }, customerId); await db.customers.insert({ id: customerId, onshore_record: recordId, email_token: tokens.email, national_id_token: tokens.national_id, dob_token: tokens.date_of_birth, name_token: tokens.full_name, }); res.status(201).json({ customerId }); });

What to store in your database: the tokens, and the record ID if you use getRecord. Your columns become national_id_token instead of national_id. Tokens are safe to log, index, copy to a warehouse and send to vendors.

Handling a bad value

If a value fails its field’s format check, the whole record is refused with 422 validation_failed, and nothing is stored. The error names the problem but never echoes the value. Show the person a form error. The check runs in the vault, so you don’t have to keep validation rules for 18 countries’ ID formats in your own code, although checking basic shape in the form first gives a quicker response.

Migrating an existing database

You have a customers table with a plain national_id column. Moving it into the vault without downtime:

Add a token column

ALTER TABLE customers ADD COLUMN national_id_token text;

Write both, for new rows

Change your code so new and updated rows are tokenized as well as written to the old column. Run this for a while, so no new row is missed by the backfill.

Check the old data first

A dry run checks every value against the field’s format and stores nothing:

const rows = await db.query("SELECT id, national_id FROM customers WHERE national_id_token IS NULL"); const check = await vault.tokenize(VAULT, rows.map((r) => ({ field: "national_id", value: r.national_id, subject_ref: r.id })), { dryRun: true, continueOnError: true }); const bad = check.filter((r) => r.error); console.log(`${bad.length} of ${rows.length} would be refused`); for (const b of bad) console.log(rows[b.index].id, b.error!.code); // ids and codes, never values

Fix or decide on the bad rows before going on: old data often holds placeholders like N/A or 000000000.

Backfill in batches

const BATCH = 2000; for (;;) { const rows = await db.query( "SELECT id, national_id FROM customers WHERE national_id_token IS NULL AND national_id IS NOT NULL ORDER BY id LIMIT $1", [BATCH]); if (rows.length === 0) break; const results = await vault.tokenize(VAULT, rows.map((r) => ({ field: "national_id", value: r.national_id, subject_ref: r.id })), { continueOnError: true }); await db.tx(async (t) => { for (const r of results) { if (r.token) await t.query("UPDATE customers SET national_id_token=$1 WHERE id=$2", [r.token, rows[r.index].id]); else await t.query("UPDATE customers SET national_id_migration_error=$1 WHERE id=$2", [r.error!.code, rows[r.index].id]); } }); }

The field is deterministic, so re-running the backfill after a crash returns the same tokens (existing: true) rather than storing duplicates.

Switch reads to tokens

Change code that used national_id to use the token, and reveal only where a person really needs to see the value.

Drop the old column

Once nothing reads it, remove the column, and purge it from backups on your normal backup schedule.

ALTER TABLE customers DROP COLUMN national_id;

For bulk loads of millions of rows, the console’s Import data page and the rv import command run the same checks with progress and a report: see Import data.

Next

Last updated on