Syncing data from instruments¶

Copying files off instrument computers by hand is slow. Files get lost, renamed or attached to the wrong sample. datalab has two ways to avoid it:

• Beholder runs on the instrument computer and sends files to datalab by itself. This is the recommended approach.
• Server shares let you browse a connected network folder from inside datalab and pick files by hand.

The quick version of both is in Attach files. This page has more detail.

Beholder ¶

Beholder is a small program that watches the folders where an instrument saves its data. When it finds a file whose path contains an item ID, it uploads the file and attaches it to that item.

Why use it¶

• No manual steps. Measure, save with the right name, and the file arrives.
• Files stay up to date. If a file grows during a long measurement, Beholder uploads the new version and replaces the old copy.
• Safe for instrument computers. Beholder only makes outgoing connections. The instrument computer does not need to be reachable from the network.
• Nothing is deleted. Beholder never removes anything from datalab.

How it decides where a file goes¶

Each watched folder has one or more patterns. A pattern picks the item ID out of the file's path. If no pattern matches, the file is skipped.

A pattern can also pick out:

• a group, so that new items are shared with that group,
• a collection, so that new items are added to it.

If the item does not exist yet, Beholder can create it. This only happens if it was set up with an item type to use. Otherwise it waits until someone creates the item.

For example, a battery-testing group might save files as P042/cell-7/run1.mpr. A pattern could read P042 as the project and 7 as the cell number, and attach the file to item P042-7.

What this means for you¶

• Name files and folders the way your group agreed. This is the only thing Beholder needs from you.
• Create the item first if your Beholder is not set up to create items.
• Be patient, or upload by hand. With default settings, a new file usually appears within an hour or two. Changes to known files are picked up faster.

Files that Beholder uploads are recorded as uploaded by the account that owns its API key. This is often a shared or administrator account rather than you.

Setting it up¶

Setting up Beholder is usually a job for an instrument owner or administrator.

1. Install Beholder on a computer that can read the instrument's data folder. There is a ready-made Windows program, so Python is not needed.
2. Create an API key for the account Beholder will use.
3. Write a configuration file listing the datalab instance, the folders to watch, and the patterns to use.
4. Use the dry run command to check which files would go where, before anything is uploaded.
5. Start Beholder. It has a small window showing its status.

The Beholder README has full instructions and every setting.

Server shares¶

Your administrator can connect folders on other computers to datalab. These can be local folders on the server, network drives, or computers reachable over SSH. They then appear in datalab as folder trees you can browse.

Adding a file from a share¶

1. Open the item.
2. In the Files section, click Add files from server.
3. Browse to the file, or search for it. The search starts with the item's ID.
4. Tick the files you want and click Load files.

datalab copies the file and attaches it to the item.

Keeping listings fresh¶

datalab stores a copy of each folder listing, so the tree opens quickly. The copy is refreshed about once an hour. If a new file is missing, click Update tree.

A file added from a share shows the server it came from. It also shows when it last changed on the server and when datalab last copied it.

Which should I use?¶

Many groups use both. Beholder handles the everyday measurements. Server shares cover older data and one-off files.

Bringing the two approaches together is on the roadmap.