Edit an NXtomo

The general workflow to edit an NXtomo is:

load it from disk -> modify it in memory -> save it to disk

In this example we edit the dummy_nxtomo.nx file, produced by the “Create an NXtomo (from scratch)” tutorial.

import os
from nxtomo import NXtomo

nx_tomo_file_path = os.path.join("resources", "dummy_nxtomo.nx")
nx_tomo = NXtomo().load(nx_tomo_file_path, "entry", detector_data_as="as_numpy_array")
print("nx_tomo type is", type(nx_tomo))
print("nx_tomo energy is", nx_tomo.energy)

nx_tomo type is <class 'nxtomo.application.nxtomo.NXtomo'>
nx_tomo energy is 13.6 kiloelectron_volt

You can then modify the values as shown previously and overwrite the file.

import pint

ureg = pint.UnitRegistry()

nx_tomo.energy = 13.6 * ureg.keV
nx_tomo.save(
    file_path=nx_tomo_file_path,
    data_path="entry",
    overwrite=True,
)
print("new energy is", NXtomo().load(nx_tomo_file_path, "entry").energy)

new energy is 13.6 kiloelectron_volt

Detector data is usually stored as an h5py virtual dataset. Large acquisitions can consume significant memory, so detector data can be loaded with several strategies:

• “as_data_url” (default): each VirtualSource is saved as a DataUrl to keep handling lightweight (see later in the tutorial).

• “as_virtual_source”: retrieves the original VirtualSource objects so you can edit them.

• “as_numpy_array”: loads all frames in memory for editing and writes everything back. Avoid this for real datasets as it triggers heavy I/O.

Clean

if os.path.exists(nx_tomo_file_path):
    os.remove(nx_tomo_file_path)
if os.path.exists("nxtomo_reconstruction.hdf5"):
    os.remove("nxtomo_reconstruction.hdf5")

Advanced usage: provide DataUrl objects to instrument.detector.data

The NXtomo described above can consume lots of memory when the detector is large or the number of projections is high.

A practical workaround is to supply DataUrl objects that may point to external files and let the FrameAppender manage them.

In this example we first store the metadata in the HDF5 file (and optionally a few frames). You can then append frame series sequentially with their rotation angles and image keys.

Create datasets in external files

Here we create datasets in external files and record the corresponding DataUrls.

These datasets must be 3D; otherwise the virtual dataset creation will fail.

import h5py
import numpy
from silx.io.url import DataUrl

detector_data_urls = []
for i_file in range(5):
    os.makedirs("output/external_files", exist_ok=True)
    external_file = os.path.join(f"output/external_files/file_{i_file}.nx")
    with h5py.File(external_file, mode="w") as h5f:
        h5f["data"] = numpy.arange(
            start=(5 * 100 * 100 * i_file), stop=(5 * 100 * 100 * (i_file + 1))
        ).reshape(
            [5, 100, 100]
        )  # of course here this is most likely that you will load data from another file

    detector_data_urls.append(
        DataUrl(
            file_path=external_file,
            data_path="data",
            scheme="silx",
        )
    )

Create a simple NXtomo and provide DataUrl entries for instrument.detector.data

my_large_nxtomo = NXtomo()

Provide all metadata except the frames. In this example we create a dataset with 180 projections.

my_large_nxtomo.instrument.detector.distance = 0.2 * ureg.meter
my_large_nxtomo.instrument.detector.x_pixel_size = (
    my_large_nxtomo.instrument.detector.y_pixel_size
) = (12 * ureg.micrometer)
my_large_nxtomo.energy = 12.3 * ureg.keV
# ...
my_large_nxtomo.sample.rotation_angle = (
    numpy.linspace(0, 180, 180, endpoint=False) * ureg.degree
)
my_large_nxtomo.instrument.detector.image_key_control = [0] * 180  # 0 == Projection

Provide the list of DataUrl objects to instrument.detector.data.

my_large_nxtomo.instrument.detector.data = detector_data_urls

os.makedirs("output", exist_ok=True)
my_large_nxtomo.save("output/my_large_nxtomo.nx", data_path="entry0000", overwrite=True)

This creates a virtual dataset under ‘instrument/detector/data’ that stores relative links from “my_large_nxtomo.nx” to the external files.

The data dataset now contains 180 frames (running the previous cell again keeps appending data).

If a DataUrl is provided instead of a NumPy array, NXtomo builds a virtual dataset and avoids duplicating data. Keep the files in the same relative locations.

Appended frames must have the same dimensions; otherwise the operation fails.

from h5glance import H5Glance

H5Glance("output/my_large_nxtomo.nx")

• output/my_large_nxtomo.nx
  • entry0000
    • beam
      • incident_energy → /entry0000/instrument/beam/incident_energy
    • data
      • data → /entry0000/instrument/detector/data
      • image_key → /entry0000/instrument/detector/image_key
      • image_key_control → /entry0000/instrument/detector/image_key_control
      • rotation_angle → /entry0000/sample/rotation_angle
    • instrument
      • beam
        • incident_energy [📋]: scalar entries, dtype: float64
      • detector
        • data [📋]: 25 × 100 × 100 entries, dtype: int64
        • distance [📋]: scalar entries, dtype: float64
        • field_of_view [📋]: scalar entries, dtype: UTF-8 string
        • image_key [📋]: 180 entries, dtype: int64
        • image_key_control [📋]: 180 entries, dtype: int64
        • x_pixel_size [📋]: scalar entries, dtype: int64
        • y_pixel_size [📋]: scalar entries, dtype: int64
      • source
        • name [📋]: scalar entries, dtype: UTF-8 string
        • probe [📋]: scalar entries, dtype: UTF-8 string
        • type [📋]: scalar entries, dtype: UTF-8 string
    • sample
      • rotation_angle [📋]: 180 entries, dtype: float64
    • definition [📋]: scalar entries, dtype: UTF-8 string

Check that the paths of the VirtualSource objects are relative (they must start with ‘./’).

with h5py.File("output/my_large_nxtomo.nx", mode="r") as h5f:
    dataset = h5f["entry0000/instrument/detector/data"]
    print("dataset is virtual:", dataset.is_virtual)
    for vs_info in dataset.virtual_sources():
        print("file name is", vs_info.file_name)
        assert vs_info.file_name.startswith("./")

dataset is virtual: True
file name is ./external_files/file_0.nx
file name is ./external_files/file_1.nx
file name is ./external_files/file_2.nx
file name is ./external_files/file_3.nx
file name is ./external_files/file_4.nx

tip

• You can also provide a list of h5py.VirtualSource objects to the detector.data attribute.

• To append frames to an existing dataset you can use the FrameAppender class from nxtomo.

Advanced use cases

Provide NXtransformations to NXdetector (detector flip, rotation, translation…)

Detectors may include image flips (up-down or left-right) introduced by the acquisition (BLISS-Tango) or manual adjustments such as rotations around an axis.

To describe them, NXdetector exposes a TRANSFORMATIONS group that defines the transformation chain. As of today (2023) only image flips are taken into account by nabu for stitching.

To provide such transformations you can supply a set of transformation entries like the following:

from nxtomo import NXtomo

my_nxtomo = NXtomo()

import pint
from nxtomo.utils.transformation import Transformation

_ureg = pint.UnitRegistry()

my_nxtomo.instrument.detector.transformations.add_transformation(
    Transformation(
        axis_name="rx",  # axis name must be unique
        transformation_type="rotation",
        value=180 * ureg.degree,
        vector=(
            1,
            0,
            0,
        ),  # warning: transformation are provided as (x, y, z) which is different of the usual numpy ref used (z, y, x)
    )
)

There are several utility classes that provide detector flips and basic transformation axes for you. Please consider using them.

from nxtomo.utils.transformation import (
    Transformation,
    DetYFlipTransformation,
    DetZFlipTransformation,
    TransformationAxis,
    TransformationType,
)
from nxtomo.nxobject.nxtransformations import NXtransformations

nx_transformations = NXtransformations()
nx_transformations.transformations = (
    DetYFlipTransformation(flip=True),  # vertical flip of the detector
    DetZFlipTransformation(
        flip=True, depends_on="ry"
    ),  # horizontal flip of the detector. Applied after the vertical flip
    Transformation(  # some translation over x axis. Applied after the horizontal flip
        axis_name="tx",
        value=0.02
        * ureg.millimeter,  # value can be a scalar - static value - of an array of value (one per frame expected)
        transformation_type=TransformationType.TRANSLATION,  # default unit for translation is SI 'meter'
        depends_on="rz",  #  Applied after the horizontal flip in the transformation chain
        vector=TransformationAxis.AXIS_X,
    ),
)
my_nxtomo.instrument.detector.transformations = nx_transformations

NXtransformations and NXsample

As of today the NXtomo application uses individual datasets (x_translation, y_translation, z_translation, rotation_angle) to describe sample transformations. Future versions may adopt an NXtransformations group, but this is not yet implemented, so keep using the original datasets for transformations.