what does "rebuild data file" actually do?

m w

Hi - I was playing around with different keychains and at one point I deleted all records from the active keychain which had 776 records, deleted the trash and created just a couple of test secure notes. I saw the keychain bundle shrink from around 32 MB to below 2 MB. Then I performed a "clear cache" and "rebuild data file" operation and was surprised that during the rebuilding it was processing 914 records. I repeated this process a few times but it always kept processing 914 records. So I had a look in the ~/Dropbox/1Password/1Password.agilekeychain/data/default folder and found 917 items in there. And yet the keychain only contained 2 secure notes. Later I restore all records using the "File > Restore" command and the keychain was back at 32 MB. Rebuilding the data file again processed 917 files and in the default folder I still had 917 records.

Therefore my question: what does "rebuild data file" actually do?
And in addition: is it right to have 917 items in the keychain default folder if the keychain contains only 2 items?

Cheers
Michael

MikeT

Hi Michael,

That is normal. When you empty your trash, the items are not gone but simply tombstoned. In other words, the data values are deleted for good but the metadata is left behind. The reason is because it's used to sync permanent deletions to other endpoints such as your iOS app, browser extensions and so on. If we deleted the file itself, the endpoints would think it's missing and syncs the data back to the data file.

Rebuilding basically reads the data, copy it and recreate it into a new item that's exactly like the first one, it's a way to refresh the items to get rid of any stale stuff. This also does the same for the tombstoned items.

Here's an example of a tombstoned item if you look at it:

{
    "uuid": "0B3DD335F48E432BB8A0D47307A4D7C3",
    "updatedAt": 1360806914,
    "locationKey": "",
    "openContents": {
        "contentsHash": "fe62a730"
    },
    "keyID": "E4AFA4BA5A814571B85938DADE701C1C",
    "trashed": true,
    "title": "",
    "location": "",
    "encrypted": "U2FsdGVkX18OUZakhn9MBGtJc48Pj/HKTqX7uonStyE=\u0000",
    "createdAt": 1278191192,
    "typeName": "system.Tombstone"
}

m w

Thanks for your explanation Mike - that makes sense. However, does this mean that the data folder will grow indefinitely? Or will it start pruning those tombs once they synced across all devices (how would it know that all devices have been synced though) or after a certain time?

Cheers
Michael

MikeT

Hi Michael,

Yes, it can grow over time despite the tombs being very small. We're trying to find a better way to handle this. This is one of the reasons why the sync process can be the most difficult part of many apps, it's in their nature to make the entire app 10x more complex.

Right now, the cleanest way to get rid of those tombs are to export the data and import it in a new data file, which is probably something we can do in the future with a hidden option. The thing about doing this, is that you'll have a separate data file ID, which kicks in a security alarm in the 1Password Helper to block the sync between the browser extensions and the main 1Password app. You'd have to reinstall the extensions to allow the new IDs.

The security policy is in place to prevent 1Password from accidentally syncing two different data files into the same file.

m w

Understood - thanks Mike.

MikeT

You're welcome!