DARLING#

the python [D]ark field x-ray microscopy [A]nalysis & [R]econstruction [L]ibrary for rapid data [IN]spection & [G]raphing

Authors#

darling is written and maintained by:

Axel Henningsson, Felix Tristan Frankus and Adam André William Cretton

affiliated with DTU. The core ideas of this library was originally written during a beamtime at ESRF id03D.

Until an associated journal publication is available, if you use this code in your research, we ask that you cite this repository.

If you are interested in collaborating with us on DFXM data analysis, please reach out to us at: naxhe@dtu.dk and we can discuss the possibilities.

Usecase#

Loading & plotting Data#

Darling collects many useful tools in the properties module.

For example, it is possible to model the angular intensity distribution of a scan using a Gaussian Mixture Model (GMM) and then color code the scan based on the first maxima of the GMM using a RGB color map known as a mosaicity map.

import numpy as np

# read some toy data in
path_to_data, _, _ = darling.assets.domains()  # replace with your own data
scan_id = "1.1"  # tope level key in the hdf5 file
dset = darling.DataSet(path_to_data, scan_id)

# model the intensity distribution using a GMM
features = darling.properties.gaussian_mixture(dset.data, k=4, coordinates=dset.motors)

# color code the scan based on the first maxima of the GMM
first_maxima_mean = np.concatenate(
   (features["mean_motor1"][..., 0, None], features["mean_motor2"][..., 0, None]),
   axis=-1,
)
rgbmap, colorkey, colorgrid = darling.properties.rgb(
   first_maxima_mean, norm="dynamic", coordinates=dset.motors
)

To visualize the resulting mosaicity map, you can use the following code:

import matplotlib.pyplot as plt

# plot the resulting mosaicity map
plt.style.use('dark_background')
fig, ax = plt.subplots(1, 1, figsize=(7,7))
im = ax.imshow(rgbmap)
plt.tight_layout()
plt.show()

https://github.com/AxelHenningsson/darling/blob/dev/docs/source/images/domains_mosa.png?raw=true

for more examples see the externally hosted documentation at https://axelhenningsson.github.io/darling/

Installation#

From source the key is simply to clone and pip install

git clone https://github.com/AxelHenningsson/darling.git
cd darling
pip install -e .

In general, you probably want to install in a fresh virtual environment as

python3 -m venv .venv_darling
source .venv_darling/bin/activate
git clone https://github.com/AxelHenningsson/darling.git
cd darling
pip install -e .

use

source .venv_darling/bin/activate

whenever you want to activate the environment. To add your env into a jupyter kernel such that you can use it in an interactive notebook you may add the following two commands:

pip install ipykernel
python -m ipykernel install --user --name=darling

Note on jupyter & the ESRF slurm cluster#

In the main ESRF slurm Python jupyter kernel it is possible to do the following hack to get the latest darling running.

git clone https://github.com/AxelHenningsson/darling.git
sys.path.insert(0, os.path.abspath('./darling'))
import darling

This trick is possible since that all dependencies of darling are already installed in the big Python jupyter kernel at ESRF.

The following snippet has also been verified to work on the ESRF slurm cluster 19 Dec 2024 in a browser terminal:

python3 -m venv .venv_darling
source .venv_darling/bin/activate
git clone https://github.com/AxelHenningsson/darling.git
cd darling
pip install -e .
pip install ipykernel
python -m ipykernel install --user --name=darling

This appraoch should work on other clusters as well, as long as some user permission to install exists.

Documentation#

Darling hosts documentation at https://axelhenningsson.github.io/darling/

properties#

Functions module for computation of data features over 4D or 5D fields. I.e computation of moments of mosa-scans strain-mosa-scans and the like.

As an example, in a DFXM strain-mosaicity-scan setting, using random arrays, the 3D moments in theta, phi and chi can be retrieved as:

import numpy as np
import darling

# create coordinate arrays
theta = np.linspace(-1, 1, 9) # crl scan grid
phi = np.linspace(-1, 1, 8) # motor rocking scan grid
chi = np.linspace(-1, 1, 16) # motor rolling scan grid
coordinates = np.meshgrid(phi, chi, theta, indexing='ij')

# create a random data array
detector_dim = (128, 128) # the number of rows and columns of the detector
data = 64000 * np.random.rand(*detector_dim, len(phi), len(chi), len(theta))

data = data.astype(np.uint16) # the collected intensity data for the entire scan

# compute the first and second moments such that
# mean[i,j] is the shape=(3,) array of mean coorindates for pixel i,j.
# covariance[i,j] is the shape=(3,3) covariance matrix of pixel i,j.
mean, covariance = darling.properties.moments(data, coordinates)

assert mean.shape==(128, 128, 3)
assert covariance.shape==(128, 128, 3, 3)

darling.properties.rgb(property_2d, norm='dynamic', coordinates=None)[source]#

Compute a m, n, 3 rgb array from a 2d property map, e.g from a first moment map.

NOTE: Only normalization ranges that covers the full range of the property_2d are accepted here. Consider marking values outside range by np.nan before calling in case such normalization is needed.

import matplotlib.pyplot as plt
import numpy as np
import darling

# create some phantom data
phi = np.linspace(-1, 1, 64)
chi = np.linspace(-1, 1, 128)
coord = np.meshgrid(phi, chi, indexing="ij")
property_2d = np.zeros((len(phi), len(chi), 2))
property_2d[..., 0] = np.cos(np.outer(phi, chi))
property_2d[..., 1] = np.sin(np.outer(phi, chi))

# compute the rgb map normalising to the coordinates array
rgb_map, colorkey, colorgrid = darling.properties.rgb(property_2d, norm="full", coordinates=coord)

plt.style.use("dark_background")
fig, ax = plt.subplots(1, 1, figsize=(7, 7))
im = ax.imshow(rgb_map)
plt.tight_layout()
plt.show()

alternatively; normalize to the dynamic range of the property_2d array

rgb_map, colorkey, colorgrid = darling.properties.rgb(property_2d, norm="dynamic")

plt.style.use("dark_background")
fig, ax = plt.subplots(1, 1, figsize=(7, 7))
im = ax.imshow(rgb_map)
plt.tight_layout()
plt.show()

Parameters:

property_2d (numpy array) – The property map to colorize, shape=(a, b, 2), the last two dimensions will be mapped to rgb colors.
coordinates (numpy array) – Coordinate grid assocated to the property map, shape=(m, n), optional for norm=”full”. Defaults to None.
norm (numpy array or str) – array of shape=(2, 2) of the normalization range of the colormapping. Defaults to ‘dynamic’, in which case the range is computed from the property_2d array max and min. (norm[i,0] is min value for property_2d[:,:,i] and norm[i,1] is max value for property_2d[:,:,i].). If the string ‘full’ is passed, the range is computed from the coordinates as the max and min of the coordinates. This requires the coordinates to be passed as well.

Returns:

RGB map of shape=(a, b, 3) and: the colorkey of shape (m, n, 3) and the grid of the colorkey of shape=(m, n).

Return type:

tuple of numpy array

darling.properties.kam(property_2d, size=(3, 3))[source]#

Compute the KAM (Kernel Average Misorientation) map of a 2D property map.

KAM is computed by sliding a kernel across the image and for each voxel computing the average misorientation between the central voxel and the surrounding voxels. Here the misorientation is defined as the L2 euclidean distance between the (potentially vectorial) property map and the central voxel such that scalars formed as for instance np.linalg.norm( property_2d[i + 1, j] - property_2d[i, j] ) are computed and averaged over the kernel.

NOTE: This is a projected KAM in the sense that the rotation the full rotation matrix of the voxels are unknown. I.e this is a computation of the misorientation between diffraction vectors Q and not orientation elements of SO(3). For 1D rocking scans this is further reduced due to the fact that the roling angle is unknown.

import matplotlib.pyplot as plt
import numpy as np
from scipy.ndimage import gaussian_filter

import darling

# create some phantom data
phi = np.linspace(-1, 1, 64)
chi = np.linspace(-1, 1, 128)
coord = np.meshgrid(phi, chi, indexing="ij")
property_2d = np.random.rand(len(phi), len(chi), 2)
property_2d[property_2d > 0.9] = 1
property_2d -= 0.5
property_2d = gaussian_filter(property_2d, sigma=2)

# compute the KAM map
kam = darling.properties.kam(property_2d, size=(3, 3))

plt.style.use("dark_background")
fig, ax = plt.subplots(1, 1, figsize=(7, 7))
im = ax.imshow(kam, cmap="plasma")
plt.tight_layout()
plt.show()

Parameters:

property_2d (numpy array) – The property map to compute the KAM from, shape=(a, b, m) or (a, b). This is assumed to be the angular coordinates of diffraction such that np.linalg.norm( property_2d[i,j]) gives the mismatch in degrees between the reference diffraction vector and the local mean diffraction vector.
size (tuple) – The size of the kernel to use for the KAM computation. Defaults to (3, 3).

Returns:

The KAM map of shape=(a, b). (same units as input.)

Return type:

numpy array

darling.properties.moments(data, coordinates)[source]#

Compute the sample mean amd covariance of a 3D, 4D or 5D DFXM data-set.

The data-set represents a DFXM scan with 1, 2 or 3 degrees of freedom. These could be mu, phi and chi, or phi and energy, etc. The total data array is therefore either 3d, 4d or 5d.

NOTE: Computation is done in parallel using shared memory with numba just: in time compiling. For this reason the data array must be of type numpy uint16.

Example in a DFXM mosaicity-scan setting using random arrays:

import numpy as np
import darling

# create coordinate arrays
phi = np.linspace(-1, 1, 8)
chi = np.linspace(-1, 1, 16)
coordinates = np.meshgrid(phi, chi, indexing='ij')

# create a random data array
detector_dim = (128, 128)
data = 64000 * np.random.rand(*detector_dim, len(phi), len(chi))

data = data.astype(np.uint16)

# compute the first and second moments
mean, covariance = darling.properties.moments(data, coordinates)

Parameters:

data (numpy array) – Array of shape=(a, b, m) or shape=(a, b, m, n) or shape=(a, b, m, n, o) where the maps over which the mean will be calculated are of shape=(m) or shape=(m, n) or shape=(m, n, o) respectively and the detector field dimensions are of shape=(a, b). Must be numpy uint16. I.e data[i, j, …] is a distribution for pixel i, j.
coordinates (tuple of numpy array) – Tuple of len=1, len=2 or len=3 containing numpy nd arrays specifying the coordinates in each dimension respectively. I.e, as an example, these could be the phi and chi angular cooridnates as a meshgrid.

Returns:

The mean map of shape=(a,b,…) and the: covariance map of shape=(a,b,…).

Return type:

tuple of numpy array

darling.properties.mean(data, coordinates)[source]#

Compute the sample mean of a 3D, 4D or 5D DFXM data-set.

The data-set represents a DFXM scan with 1, 2 or 3 degrees of freedom. These could be mu, phi and chi, or phi and energy, etc. The total data array is therefore either 3d, 4d or 5d.

NOTE: Computation is done in parallel using shared memory with numba just: in time compiling. For this reason the data array must be of type numpy uint16.

Example in a DFXM energy-mosaicity-scan setting using random arrays:

import numpy as np
import darling

# create coordinate arrays
theta = np.linspace(-1, 1, 7)
phi = np.linspace(-1, 1, 8)
chi = np.linspace(-1, 1, 16)
coordinates = np.meshgrid(phi, chi, theta, indexing='ij')

# create a random data array
detector_dim = (128, 128)
data = 64000 * np.random.rand(*detector_dim, len(phi), len(chi), len(theta))

data = data.astype(np.uint16)

# compute the first moments
first_moment = darling.properties.mean(data, coordinates)

Parameters:

data (numpy array) – Array of shape=(a, b, m) or shape=(a, b, m, n) or shape=(a, b, m, n, o) where the maps over which the mean will be calculated are of shape=(m) or shape=(m, n) or shape=(m, n, o) respectively and the detector field dimensions are of shape=(a, b). Must be numpy uint16. I.e data[i, j, …] is a distribution for pixel i, j.
coordinates (tuple of numpy array) – Tuple of len=1, len=2 or len=3 containing numpy nd arrays specifying the coordinates in each dimension respectively. I.e, as an example, these could be the phi and chi angular cooridnates as a meshgrid.

Returns:

The mean map of shape=(a,b,k) where k=data.ndim - 2.

Return type:

numpy array

darling.properties.covariance(data, coordinates, first_moments=None)[source]#

Compute the sample mean of a 3D, 4D or 5D DFXM data-set.

The data-set represents a DFXM scan with 1, 2 or 3 degrees of freedom. These could be mu, phi and chi, or phi and energy, etc. The total data array is therefore either 3d, 4d or 5d.

NOTE: Computation is done in parallel using shared memory with numba just: in time compiling. For this reason the data array must be of type numpy uint16.

Example in a DFXM energy-mosaicity-scan setting using random arrays:

import numpy as np
import darling

# create coordinate arrays
phi = np.linspace(-1, 1, 8)
chi = np.linspace(-1, 1, 16)
coordinates = np.meshgrid(phi, chi, indexing='ij')

# create a random data array
detector_dim = (128, 128)
data = 64000 * np.random.rand(*detector_dim, len(phi), len(chi))

data = data.astype(np.uint16)

# compute the first moments
first_moment = darling.properties.mean(data, coordinates)

# compute the second moments
covariance = darling.properties.covariance(data, coordinates, first_moments=first_moment)

Parameters:

data (numpy array) – Array of shape=(a, b, m) or shape=(a, b, m, n) or shape=(a, b, m, n, o) where the maps over which the mean will be calculated are of shape=(m) or shape=(m, n) or shape=(m, n, o) respectively and the detector field dimensions are of shape=(a, b). Must be numpy uint16. I.e data[i, j, …] is a distribution for pixel i, j.
coordinates (tuple of numpy array) – Tuple of len=1, len=2 or len=3 containing numpy nd arrays specifying the coordinates in each dimension respectively. I.e, as an example, these could be the phi and chi angular cooridnates as a meshgrid.
first_moments (numpy array) – Array of shape=(a, b, …) of the first moments as described in darling.properties.mean(). Defaults to None, in which case the first moments are recomputed on the fly.

Returns:

The covariance map of shape=(a,b,…).

Return type:

numpy array

darling.properties.gaussian_mixture(data, k=8, coordinates=None)[source]#

Model a 2D grid of 2D images with a 2D grid of gaussian mixtures.

For a data array of shape (m, n, a, b), each primary pixel (i, j) contains a (a, b) sub-array that is analyzed as a 2D image. Local maxima are identified within this sub-array, and segmentation is performed to assign a label to each secondary pixel.

Each segmented region is treated as a Gaussian, with mean and covariance extracted for each label. Additionally, a set of features is computed for each label.

Specifically, for each located peak, the following features are extracted:

sum_intensity: Sum of the intensity values in the segmented domain.
number_of_pixels: Number of pixels in the segmented domain.
mean_row: Mean row position in the segmented domain.
mean_col: Mean column position in the segmented domain.
var_row: Variance of the row positions in the segmented domain.
var_col: Variance of the column positions in the segmented domain.
var_row_col: Covariance of the row and column positions in the segmented domain.
max_pix_row: Row position of the pixel with the highest intensity.
max_pix_col: Column position of the pixel with the highest intensity.
max_pix_intensity: Intensity of the pixel with the highest intensity.

Additionally, when motor coordinate arrays are provided, the following features are included:

mean_motor1: Mean motor position for the first motor.
mean_motor2: Mean motor position for the second motor.
var_motor1: Variance of the motor positions for the first motor.
var_motor2: Variance of the motor positions for the second motor.
var_motor1_motor2: Covariance of the motor positions for the first and second motor.
max_pix_motor1: Motor position for the first motor of the pixel with the highest intensity.
max_pix_motor2: Motor position for the second motor of the pixel with the highest intensity.

Example:

import darling
import matplotlib.pyplot as plt

# import a small data set from assets known to
# comprise crystalline domains
_, data, coordinates = darling.assets.domains()

# compute all the gaussian mixture model features
features = darling.properties.gaussian_mixture(data, k=3, coordinates=coordinates)

# this is a dict like structure that can be accessed like this:
sum_intensity_second_strongest_peak = features["sum_intensity"][..., 1]

# plot the mean in the first motor direction for the strongest peak
plt.style.use("dark_background")
fig, ax = plt.subplots(1, 1, figsize=(7, 7))
im = ax.imshow(features["mean_motor1"][..., 0], cmap="plasma")
plt.tight_layout()
plt.show()

Parameters:

data (numpy array) – Array of shape=(a, b, m, n) or shape=(a, b, m) where the maps over which the mean will be calculated are of shape=(m, n) or shape=(m,) and the field is of shape=(a, b) such that data[i, j, :, :] or data[i, j, :] is a 2D or 1D intensity distribution for pixel i,j.
k (int) – The number of gaussians to fit to the data. Defaults to 8. this means that the k strongest peaks, with respect to intensity, will be fitted with gaussians. The remaining peaks will be ignored.
coordinates (numpy array) – array specifying the coordinates of shape=(2, m, n) or shape=(1, m). I.e, as an example, these could be the phi and chi angular coordinates or the mu rocking angle.

Returns:

features as a dictionary containing the extracted features for: each peak with keys as specified above. i.e features[“sum_intensity”][…, i] is a 2D image where each pixel holds the summed intensity of the i-th strongest peak. Likewise features[“mean_row”][…, i] is the mean row position of the i-th strongest peak etc.

Return type:

dict

peaksearcher#

This module defines a discrete peak searcher algorithm for 2D data.

This module enables segmentation of 2D data into domains of interest and allow for extraction of features from these domains. The algorithm is based on a gradient ascent in the data map moving in a manhattan-like fashion. The algorithm is implemented in numba and can be run in parallel.

The final result is a gaussian mixture model of a grid of images. The features extracted from the segmented domains are stored in a feature table. The feature table is a dictionary with keys corresponding to the features extracted from the segmented domains. The feature table can be converted to motor positions if motor positions are provided.

Specifically, for each located peak, the following features are extracted:

sum_intensity: Sum of the intensity values in the segmented domain.

number_of_pixels: Number of pixels in the segmented domain.

mean_row: Mean row position in the segmented domain.

mean_col: Mean column position in the segmented domain.

var_row: Variance of the row positions in the segmented domain.

var_col: Variance of the column positions in the segmented domain.

var_row_col: Covariance of the row and column positions in the segmented domain.

max_pix_row: Row position of the pixel with the highest intensity.

max_pix_col: Column position of the pixel with the highest intensity.

max_pix_intensity: Intensity of the pixel with the highest intensity.

Additionally, when motor coordinate arrays are provided, the following features are included:

mean_motor1: Mean motor position for the first motor.

mean_motor2: Mean motor position for the second motor.

var_motor1: Variance of the motor positions for the first motor.

var_motor2: Variance of the motor positions for the second motor.

var_motor1_motor2: Covariance of the motor positions for the first and second motor.

max_pix_motor1: Motor position for the first motor of the pixel with the highest intensity.

max_pix_motor2: Motor position for the second motor of the pixel with the highest intensity.

darling.peaksearcher.label_sparse(data)[source]#

Assigns pixels in a 2D image to the closest local maxima.

The algorithm proceeds as follows:

For a given pixel, find the highest-valued neighbor.
Move the pixel to this neighbor:
1. If the neighbor is already labeled, propagate the label back to the pixel.
2. If the pixel is a local maximum, assign it a new label.
3. Otherwise, repeat step 1 until a label is assigned.

This process ensures that each pixel is assigned to the nearest local maximum through a gradient ascent type climb.

To illustrate how the local maxclimber algorithm can separate overlapping gaussians we can consider the following example:

import matplotlib.pyplot as plt
import numpy as np
import darling

# Create a synthetic image with 9 gaussians with a lot of overlap
rng = np.random.default_rng(42)
x, y = np.meshgrid(np.arange(512), np.arange(512), indexing='ij')
img = np.zeros((512, 512))
for i in range(127, 385, 127):
    for j in range(127, 385, 127):
        img += np.exp(-((x - i) ** 2 + (y - j) ** 2) / (2 * rng.uniform(31, 61) ** 2))

# Label the image following the local max climber algorithm
labeled_array, nfeatures = darling.peaksearcher.label_sparse(img)

# The segmented image shows how the local maxclimber algorithm has segmented the image
# into 9 regions splitting the overlapping gaussians.
fig, ax = plt.subplots(1, 2, figsize=(14,7))
im = ax[0].imshow(img)
fig.colorbar(im, ax=ax[0], fraction=0.046, pad=0.04)
im = ax[1].imshow(labeled_array, cmap='tab20')
fig.colorbar(im, ax=ax[1], fraction=0.046, pad=0.04)
ax[0].set_title("Original image")
ax[1].set_title("Labeled image")
plt.tight_layout()
plt.show()

Parameters:: data (numpy.ndarray) – a 2D data map to process. shape=(m,n)
Returns:: a 2D array with the same shape as the input data number_of_labels (int): the number of labels assigned to the data map
Return type:: labeled_array (numpy.ndarray)

darling.peaksearcher.extract_features(labeled_array, data, k)[source]#

Extract features from a labeled array.

Parameters:

labeled_array (numpy.ndarray) – Label array with shape (m,n)
data (numpy.ndarray) – The underlying intensity data array with shape (m,n)
k (int) – number of segmented domains to keep features for The domain with the highest sum_intensity will be kept.

Returns:

a 2D array with the extracted features with indices following: a static _FEATURE_MAPPING dict.

Return type:

‘numpy array’

DataSet#

class darling._dataset.DataSet(data_source, scan_id=None)[source]#

Bases: object

A DFXM data-set.

This is the master data class of darling. Given a data source the DataSet class will read data from arbitrary layers, process, threshold, compute moments, visualize results, and compile 3D feature maps.

Parameters:: ( (data_source) – obj: string or darling.reader.Reader): A string to the absolute h5 file path location of the data, or a reader object implementing the darling.reader.Reader() interface.

reader (: obj: darling.reader.Reader): A file reader implementing, at least, the functionallity specified in darling.reader.Reader().

data (: obj: numpy.ndarray): The data array of shape (a,b,m,n,(o)) where a,b are the detector dimensions and m,n,(o) are the motor dimensions.

motors (: obj: numpy.ndarray): The motor grids of shape (k, m,n,(o)) where k is the number of motors and m,n,(o) are the motor dimensions.

h5file (: obj: string): The absolute path to the h5 file in which all data resides.

info()[source]#

load_scan(scan_id, scan_motor=None, roi=None)[source]#

Load a scan into RAM.

Parameters:

scan_id (str or list or str) – scan id or scan ids to load.
scan_motor (str) – path in h5file to the motor that is changing with the scan_id. Defaults to None. Must be set when scan_id is not a single string.
roi (tuple of int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded. The roi refers to the detector dimensions.

subtract(value)[source]#

Subtract a fixed integer value form the data. Protects against uint16 sign flips.

Parameters:: value (int) – value to subtract.

estimate_background()[source]#

Automatic background correction based on image statistics.

a set of sample data is extracted from the data block. The median and standard deviations are iteratively fitted, rejecting outliers (which here is diffraction signal). Once the noise distirbution has been established the value corresponding to the 99.99% percentile is returned. I.e the far tail of the noise is returned.

moments()[source]#

Compute first and second moments.

The internal attributes self.mean and self.covariance are set when this function is run.

Returns:

mean and covariance maps of shapes (a,b,2) and (a,b,2,2): respectively with a=self.data.shape[0] and b=self.data.shape[1].

Return type:

(tupe of numpy array)

kernel_average_misorientation(size=(5, 5))[source]#

Compute the KAM (Kernel Average Misorientation) map.

KAM is compute by sliding a kernel across the image and for each voxel computing the average misorientation between the central voxel and the surrounding voxels.

Parameters:: size (tuple) – The size of the kernel to use for the KAM computation. Defaults to (3, 3).
Returns:: The KAM map of shape=(a, b). (same units as input.)
Return type:: numpy array

integrate(axis=None, dtype=<class 'numpy.float32'>)[source]#

Return the summed data stack along the specified axes, avoiding data stack copying.

If no axis is specified, the integration is performed over all dimensions except the first two, which are assumed to be the detector dimensions.

Parameters:

axis (int or tuple, optional) – The axis or axes along which to integrate. If None, integrates over all axes except the first two.
dtype (numpy.dtype, optional) – The data type of the output array. Defaults to np.float32.

Returns:

Integrated frames, a 2D numpy array of reduced: shape and dtype dtype.

Return type:

numpy.ndarray

estimate_mask(threshold=200, erosion_iterations=3, dilation_iterations=25, fill_holes=True)[source]#

Segment the sample diffracting region based on summed intensity along motor dimensions.

Parameters:

threshold (int) – a summed count value above which the sample is defined.
erosion_iterations (int) – Number of times to erode the mask using a 2,2 structure.
dilation_iterations (int) – Number of times to dilate the mask using a 2,2 structure.
fill_holes (bool) – Fill enclosed holes in the final mask.

Returns:

Returns: a binary 2D mask of the sample.

Return type:

(numpy array)

compile_layers(scan_ids, threshold=None, roi=None, verbose=False)[source]#

Sequentially load a series of scans and assemble the 3D moment maps.

this loads the mosa data array with shape a,b,m,n,(o) where a,b are the detector dimension and m,n,(o) are the motor dimensions as ordered in the self.motor_names.

NOTE: This function will load data sequentially and compute moments on the fly. While all moment maps are stored and concatenated, only one scan (the raw 4d or 5d data) is kept in memory at a time to enhance RAM performance.

Parameters:

scan_ids (str) – scan ids to load, e.g 1.1, 2.1 etc…
threshold (int or str) – background subtraction value or string ‘auto’ in which case a default background estimation is performed and subtracted. Defaults to None, in which case no background is subtracted.
roi (tuple or int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded
verbose (bool) – Print loading progress or not.

to_paraview(file)[source]#

Write moment maps to paraview readable format for 3D visualisation.

The written data array will have attributes as:

cov_11, cov_12, (cov_13), cov_22, (cov_23, cov_33) : Elements of covariance matrix. mean_1, mean_2, (mean_3) : The first moments in each dimension.

Here 1 signifies the self.motors[0] dimension while 2 is in self.motors[2], (and 3 in self.motors[3], when the scan is 3D)

NOTE: Requires that 3D moment maps have been compiled via compile_layers().

Parameters:: file (string) – Absolute path ending with desired filename.

reader#

Collection of pre-implemneted h5 readers developed for id03 format.

NOTE: In general the file reader is strongly dependent on data collection scheme and it is therefore the purpose of darling to allow the user to subclass Reader() and implement their own specific data structure.

Once the reader is implemented in darling format it is possible to interface the DataSet class and use all features of darling.

class darling.reader.Reader(abs_path_to_h5_file)[source]#

Bases: object

Parent class for readers.

Parameters:: ( (abs_path_to_h5_file) – obj: str): Absolute file path to data.

abs_path_to_h5_file (: obj: str): Absolute file path to data.

__call__(scan_id, roi=None)[source]#

Method to read a single 2D scan

NOTE: This method is meant to be purpose implemented to fit the specific data aqusition: scheme used.

Parameters:

scan_id (str) – scan id to load from, these are internal kayes to diffirentiate layers.
roi (tuple of int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded. The roi refers to the detector dimensions.

Returns:

data (numpy array) of shape=(a,b,m,n) and type np.uint16 and motors (tuple of numpy array) of shape=(m,) and shape=(n,) and type np.float32. a,b are detector dimensions while m,n are scan dimensions over which teh motor settings vary.

class darling.reader.MosaScan(abs_path_to_h5_file)[source]#

Bases: Reader

Load a 2D mosa scan. This is a id03 specific implementation matphing a specific beamline mosa scan macro.

NOTE: This reader was specifically written for data collection at id03. For general purpose reading of data you must implement your own reader class. The exact reding of data is strongly dependent on data aqusition scheme and data structure implementation.

Parameters:: str (abs_path_to_h5_file) – absolute path to the h5 file with the diffraction images.

__call__(scan_id, roi=None)[source]#

Load a scan

this loads the mosa data array with shape N,N,m,n where N is the detector dimension and m,n are the motor dimensions as ordered in the self.motor_names. You may view the implemented darling readers as example templates for implementing your own reader.

Parameters:

scan_id (str) – scan id to load from, e.g 1.1, 2.1 etc…
roi (tuple of int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded

Returns:

data of shape=(a,b,m,n) and the: motor arrays as 2d meshgrids, shape=(k,m,n) where k is the number of motors used in the scan (typically 1,2 or 3).

Return type:

data, motors

class darling.reader.Darks(abs_path_to_h5_file)[source]#

Bases: MosaScan

Load a series of motorless images. This is a id03 specific implementation matphing aspecific beamline mosa scan macro.

Typically used to red dark images collected with a loopscan.

Parameters:: str (abs_path_to_h5_file) – absolute path to the h5 file with the diffraction images.

__call__(scan_id, roi=None)[source]#

Load a scan

this loads the static scan data array with shape a,b,m where a,b are the detector dimensions and m is the number of (motorless) images. You may view the implemented darling readers as example templates for implementing your own reader.

Parameters:

scan_id (str) – scan id to load from, e.g 1.1, 2.1 etc…
roi (tuple of int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded

Returns:

data of shape=(a,b,m) and an empty motor array.

Return type:

data, motors

class darling.reader.RockingScan(abs_path_to_h5_file)[source]#

Bases: MosaScan

Load a 1D rocking scan. This is a id03 specific implementation matphing aspecific beamline mosa scan macro.

A rocking scan is simply a set of 2D detector images collected at different rocking angles of the goniometer.

Parameters:: str (abs_path_to_h5_file) – absolute path to the h5 file with the diffraction images.

__call__(scan_id, roi=None)[source]#

Load a scan

this loads the rocking scan data array with shape a,b,m where a,b are the detector dimensions and m is the motor dimensions. You may view the implemented darling readers as example templates for implementing your own reader.

Parameters:

scan_id (str) – scan id to load from, e.g 1.1, 2.1 etc…
roi (tuple of int) – row_min row_max and column_min and column_max, defaults to None, in which case all data is loaded

Returns:

data of shape=(a,b,m) and the: motor arrays as an array of shape=(k, n) where k is the number of motors used in the scan (i.e k=1 for a rocking scan).

Return type:

data, motors

assets#

Module to load example data and phantoms.

darling.assets.path()[source]#

darling.assets.domains(scan_id='1.1')[source]#

darling.assets.rocking_scan()[source]#

darling.assets.motor_drift(scan_id='1.1')[source]#

darling.assets.mosaicity_scan(scan_id='1.1')[source]#

Load a (tiny) part of a 2D mosaicity scan collected at the ESRF ID03.

This is a central detector ROI for a 111 reflection in a 5% deformed Aluminium. Two layers are available with scan_id 1.1 and 2.1.

Parameters:: scan_id (str) – One of 1.1 or 2.1, specifying first or second layer scanned in the sample.
Returns:: Absolute path to h5 file. data (numpy array): Array of shape (a, b, m, n) with intensity data. data[:,:,i,j] is a noisy detector image (uint16) for phi and chi at index i, j. coordinates (numpy array): Array of shape (2, m, n) containing angle coordinates.
Return type:: data_path (str)

darling.assets.gaussian_blobs(N=32, m=9)[source]#

Phantom 2d scan of gaussian blobs with shifting means and covariance.

Parameters:

N (int) – Desired data array size which is of shape=(N,N,m,m).
m (int) – Desired data array size which is of shape=(N,N,m,m).

Returns:

Array of shape=(N, N, m, m) with intensity data, data[:,:,i,j] is a noisy detector image in type uint16 for motor x and y at index i and j respectively. coordinates (numpy array): array of shape=(2,m,m) continaning x and y coordinates.

Return type:

data (numpy array)

metadata#

The metadata module is intended to contain easy configuration for the different beamlines/ changes in beamline data storage formatting.

The ID03 object is used internally by darling to interface the ESRF ID03 beamline data storage. It contains specific information on how to fetch motor names, scan shapes and other meta-data from a h5 file.

class darling.metadata.ID03(abs_path_to_h5_file)[source]#

Bases: object

The configuration object parses & fetches meta-data on the used motors and scan configuration from the h5 file. This one is specific to the ID03 beamline at the ESRF following the bliss configuration.

Parameters:: abs_path_to_h5_file (str) – The absolute path to the h5 file.

__call__(scan_id)[source]#

Return a dictionary of scan parameters, including scan shape, motor names etc.

The possible scan_command options for ID03 are

ascan motor start stop intervals ...
fscan motor start stop steps ...
a2scan motor1 start1 stop1 motor2 start2 stop2 intervals ...
d2scan motor1 start1 stop1 motor2 start2 stop2 intervals ...
fscan2d motor1 start1 stop1 steps1 motor2 start2 stop2 steps2 ...

The used command should be located under title in the h5 file. This command is parsed to extract the scan shape, motor names and other meta-data parameters.

Parameters:

scan_id (str) – scan id to load from, e.g 1.1, 2.1 etc…

Returns:

scan_params, dictionary of scan parameters;: scan_params[“scan_command”], scan_params[“scan_shape”], scan_params[“motor_names”], scan_params[“integrated_motors”], scan_params[“data_name”]

Return type:

dict