gmane.science.simulation.h5md.user

[Provenance] Attaching metadata to PDF files

Peter Colberg — 2013-05-23T21:19:40

Hi all,

Sorry for the following off-topic post, but I have to share this tidbit.

Recently, I was looking for a way to attach metadata to a plot file in
PDF format, such as a list of files used to generate the plot, a set
of plot parameters, or even the plot script itself.

PDF has two different ways of storing metadata, in form of an info
dictionary with a predfined set of attributes, or additional streams
in XMP format. The former can be modified using the pdf backend of
matplotlib [1], but does not allow arbitrary fields. The latter,
free-form metadata is inaccessible from matplotlib.

But it turns out that PDF supports file attachments [2].

[1] http://matplotlib.org/api/backend_pdf_api.html#matplotlib.backends.backend_pdf.PdfPages.infodict
[2] http://blogs.adobe.com/insidepdf/2010/11/pdf-file-attachments.html

So I appended this small snippet to my plot script:

  path = tempfile.mkdtemp()
  fn = os.path.join(path, os.path.basename(args.output))
  fig.savefig(fn + ".pdf")
  f = h5.File(fn + ".h5", "w")

Re: Pre-averaged observables

Felix Höfling — 2013-05-16T08:43:08

Am 15.05.2013, 18:07 Uhr, schrieb Peter Colberg  
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


Fine :-)

May we add these optional fields for observable groups to the draft for  
0.1? I think of "error" (standard error of mean), "count" (number of  
samples), and "variance". Since the standard deviation is just  
sqrt(variance) it may be omitted.


Actually, "particle_count" or "particle_number" would be least ambiguous  
although it is a bit long. We may switch to "particles" simply.

Pierre, what do you think?

Regards,

Felix

Re: Pre-averaged observables

Peter Colberg — 2013-05-15T16:07:51

Hi Felix,

On Wed, May 15, 2013 at 12:10:13PM +0200, Felix Höfling wrote:

I use the above scheme, with the additional "error" and "count" datasets.



The attribute with the particle number is news to me…

Yes, the choice of "count" is indeed confusing.

We could introduce "particles" or similar instead?

By the way, when committing to the git repository, it would be great
if each commit reflects only one atomic idea. This makes it possible
to review and/or revert a change later on. (I mention this after
looking up the commit containing the particle number attribute.)

Regards,
Peter

Pre-averaged observables

Felix Höfling — 2013-05-15T10:10:13

Hi all,

sometimes, one wants to store pre-averaged observables, i.e. accumulated  
over a certain time span. For example, compute the pressure every 1000  
steps and compute the mean from 10 values, i.e. writing the data only  
every 10000 steps. Such a functionality is provided by LAMMPS and recently  
also by HALMD.

http://lammps.sandia.gov/doc/fix_ave_time.html
http://halmd.org/modules/observables/utility/accumulator.html

Now my question: how shall such data be stored in the H5MD observables  
group? Along with the mean value, one would like to store also the  
standard error (or the variance) and the number of accumulated values. One  
scheme would be to distribute this information over several groups under  
the roof of the observable's name:

obs1
   \-- mean
   |    +-- count
   |    \-- value
   |    \-- step
   |    \-- time
   |
   \-- error_of_mean
   |    +-- count
   |    \-- value
   |    \-- step
   |    \-- time
   |
   \-- count
        +-- count
        \-- value
        \-- step

Re: H5MD

Peter Colberg — 2013-05-07T21:05:36

Dear Valérie,

Welcome to the community of H5MD users!

On Mon, May 06, 2013 at 02:17:01PM -0500, Valérie Vallet wrote:

H5MD itself is a specification for storing molecular data using the
HDF5 [1] file format, so any HDF5 software can be used to read and
write H5MD files.

[1] http://www.hdfgroup.org/HDF5/

The HDF5 library provides interfaces for C and Fortran, which one
would typically use for the heavy-duty output in simulation codes.
The main “low-level” HDF5 API [2] seems vast at first, but you will
only need a small subset for reading and writing H5MD files.

[2] http://www.hdfgroup.org/HDF5/doc/RM/RM_H5Front.html#LowLevelAPIs

A good starting point for learning the C/Fortan interfaces are the
HDF5 tutorial [3] and the examples shipped with the HDF5 source [4].

[3] http://www.hdfgroup.org/HDF5/Tutor/introductory.html
[4] http://www.hdfgroup.org/ftp/HDF5/current/src/

If you are thinking of more high-level use, e.g., using Python
scripting, there is the excellent h5py [5] and documentation [6].

Re: Minor revisions before H5MD v1.0

Peter Colberg — 2013-05-06T15:36:56

Yes, I would prefer attaching "dimension" to the "box" group.

Peter

Re: Minor revisions before H5MD v1.0

Peter Colberg — 2013-05-06T15:29:46

Let's make it mandatory. If a "trajectory" subgroup is present, it has
to contain "box". If an "observables" group is present, it has to
contain "box".


Could we leave it up to the user at which times to sample box or the
observables? Before reading any "value" dataset, I seek [1] in its
related "step" dataset to find the corresponding index. Regardless of
whether a group of datasets shares the same "step" dataset or not.

[1] http://en.wikipedia.org/wiki/Binary_search_algorithm

This is my seeking routine (LuaJIT):

  local function find_step(group, step)
    local dset = group:open_dataset("step")
    local filespace = dset:get_space()
    local dims = filespace:get_simple_extent_dims()
    local data = ffi.new("uint32_t[1]")
    local memspace = hdf5.create_simple_space({1})
    filespace:select_hyperslab("set", {0}, nil, {1})
    local min, max = 0, dims[1] - 1
    while min <= max do
      local mid = min + bit.rshift(max - min, 1)
      filespace:offset_simple({mid})
      dset:read("native_uint32",

Re: unit attributes

Felix Höfling — 2013-05-06T15:23:07

Am 06.05.2013, 17:00 Uhr, schrieb Peter Colberg  
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


Actually not, I do not make use of the unit attribute. I don't know  
whether experimentalists have some standards to specify the unit (like  
length is always in "metres" times a power of 10). But some people may  
want to say that the unit of length is "\sigma", others prefer Ångstrøms  
or cm. I think a free string would be sufficient for the time being.

Felix

Re: Periodic and non-periodic boundary conditions

Peter Colberg — 2013-05-06T15:18:15

Let's start with an initial choice of "nonperiodic" and "periodic".

(See variable-length string array attribute [1] for implementation hint.)

LAMMPS has a non-periodic “shrink-wrapped” boundary condition [2],
where the walls follow the extent of all particles. These boundaries
are not "fixed", but still "nonperiodic".

[1] http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/attrvstr.c
[2] http://lammps.sandia.gov/doc/boundary.html

Peter

Re: Minor revisions before H5MD v1.0

Felix Höfling — 2013-05-06T15:15:45

Am 06.05.2013, 16:52 Uhr, schrieb Peter Colberg  
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


Yes, this is an example. The 3/2 reads actually d/2, this term is the  
ideal gas contribution and arises from kinetic energy fluctuations. I  
realised that one may compute the latter fluctuations as well, hoping that  
the value d/2 is reproduced.

Another example is eq. (2.84) in this book, where several observables  
enter and the system volume is needed. Hence, my suggestion is that such  
kind of analysis is only possible if /observables/box is present.

Cheers,

Felix

Re: Box image vectors

Peter Colberg — 2013-05-06T15:07:47

Good idea, since there are cases where the image vectors can be
discarded while positions are still stored in reduced form, e.g.,
after the equilibration phase of a simulation as input to the
measurement phase.


Ok, the data type of “image” is left unspecified.

Peter

Re: unit attributes

Peter Colberg — 2013-05-06T15:00:47

Do you have suggestions for the format of the unit attribute?

It would be great if the unit was not some free-form text, but
something adhering to a standard representation of SI units in
ASCII or UTF-8. Maybe not mandatory, more as a recommendation.

Peter

Re: Minor revisions before H5MD v1.0

Peter Colberg — 2013-05-06T14:52:14

Hi Felix,

On Mon, May 06, 2013 at 04:02:52PM +0200, Felix Höfling wrote:

Thank you for the clarification, I was ignorant of the physics.

You refer to this formula, right?

\langle\delta\mathscr{V}^2\rangle_{NVT} = k_B T^2 \left(C_V - \frac{3}{2} N k_B\right)

Peter

Re: Box image vectors

Felix Höfling — 2013-05-06T14:24:37

Am 05.05.2013, 22:19 Uhr, schrieb Peter Colberg
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


It is a good idea to support both choices. All of the subgroups are
optional, so the example in the Draft does not need to be extended, simply
the list of standardised group names.

Instead of making the interpretation of "position" dependent on the
existence of "image", I would prefer to attach an attribute to "position"
stating whether they are absolute or periodically reduced. In the latter
case, the image data group shall exist and is referred to. This helps to
ensure file consistency, e.g., if a user copied the position data but for
some reason not the image vectors.

Note that the image vector is not necessarily an integer, e.g., for
Lees-Edwards boundaries. For such boundaries, also the velocities need to
be corrected for the boundary conditions.

Felix

unit attributes

Felix Höfling — 2013-05-06T14:17:57

Hi,

I realised that the optional attribute "unit" is only mentioned for the  
"value" datasets in the trajectory group. Actually, the "time" dataset may  
also carry a unit. The same applies for the groups in /observables.

I would like to generalise and to modify the value/step/time structure of  
data groups as

data_group
  \-- value
      +-- unit
  \-- step
  \-- time
      +-- unit

Felix

Re: Minor revisions before H5MD v1.0

Felix Höfling — 2013-05-06T14:12:57

Am 05.05.2013, 21:12 Uhr, schrieb Pierre de Buyl  
<pdebuyl-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


Hi Pierre,

Looking up an attribute in different possible places might be a bit  
impractical for the H5MD reader. I have a different suggestion just posted  
to the original thread. Let me repeat it:

Interpretation of the data in /observables may depend on information about  
the spatial dimension or the system geometry. Such interpretation is only  
possible, if the optional group /observables/box is present and if its  
step/time datasets coincide with the those of the observable under  
consideration. Otherwise, an H5MD reader considers the file incomplete for  
the desired purpose and fails.

Apart from that we might add the attribute "dimension" to the box group  
itself. It costs basically no storage and is less cumbersome than checking  
whether the box has an attribute "offset" or a dataset "offset", opening  
it and retrieving its shape.

What do you think?

Regards,

Felix

Re: Minor revisions before H5MD v1.0

Felix Höfling — 2013-05-06T14:02:52

Am 03.05.2013, 18:37 Uhr, schrieb Peter Colberg
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


The box group has always been optional, Pierre agreed with me that it
should be mandatory along with each trajectory group. This did not solve
the problem that it is still optional in the /observables group. Maybe it
should become mandatory there as well?

It is not true that the interpretation of, e.g., the observable
"potential_energy" does not require the space dimension. It is for example
needed when analysing the fluctuations to compute the specific heat. (See
Allan & Tildesley, Chapter 2.5) But other quantitites may need the volume
as well, and so it would be consequent to

1) either make /observables/box mandatory if /observables is present,

2) or permit the interpretation of flucutations only if the optional group
/observables/box is present (otherwise the reader considers the H5MD file
as incomplete for the desired purpose and fails)

In both cases, there appears the same issue with a fluctuating b

Re: Periodic and non-periodic boundary conditions

Felix Höfling — 2013-05-06T13:40:41

Am 06.05.2013, 14:11 Uhr, schrieb Peter Colberg  
<pcolberg-66mfFHjVbkueFQavDyXPBQ< at >public.gmane.org>:


Hi Peter,

We should also have sheared systems in mind, namely boxes with  
Lees-Edwards boundary conditions or the SLLOD algorithm, which are very  
common methods in non-equilibrium simulations of fluids. If I understand  
the methods right, the image vector changes with time and is not anymore  
an integer multiple of the edge length. Nevertheless,  I think this would  
fit into our present scheme for the box specification, but would require  
an additional d-dimensional attribute "shear_rate".

As a modification to your suggestion, I would call the attribute  
"boundary" and give it string values for each direction, e.g. "fixed",  
"periodic", ...

You may find a short summary on the above methods here:

Chapter 6.3 in the Book by Evans and Morriss:
http://rsc.anu.edu.au/~evans/evansmorrissbook.php

Alternatively, I found a brief summary in Chapter 4.7 of this work:
http://www.swinburne.edu.au/ict/rese

Periodic and non-periodic boundary conditions

Peter Colberg — 2013-05-06T12:11:29

Hi all,

How should we store the boundary conditions of a cuboid/triclinic box?

  trajectory
   \-- group1
        \-- box
             +-- type
             +-- periodic

The periodic boundary conditions could be attached to the “box” group
as a D-dimensional attribute “periodic” of integer type, where each
element indicates whether the dimension is periodic (non-zero value)
or non-periodic (zero). Note that HDF5 does not support booleans [1].

[1] http://www.hdfgroup.org/hdf5-quest.html#bool

Does this scheme cover all possible boundary conditions in your simulations?

Peter

Box image vectors

Peter Colberg — 2013-05-05T20:19:28

Hi,

While implementing the trajectory I/O for my new simulation codes,
I noticed that H5MD does not yet specify box image vectors in the
trajectory subgroup(s).

I suggest to specify a trajectory subgroup as follows:

    trajectory
     \-- group1
          \-- position
          |    \-- value
          |    \-- step
          |    \-- time
          \-- velocity
          |    \-- value
          |    \-- step
          |    \-- time
          \-- force
          |    \-- value
          |    \-- step
          |    \-- time
          \-- image
          |    \-- value
          |    \-- step
          |    \-- time
          \-- species
          |    \-- value
          |    \-- step
          |    \-- time

The "image" group contains the signed integer box image vectors, which
represent the periodic image of a particle given periodic boundary
conditions. The group is optional. If present, "position" is to be
interpreted as periodically reduced coordinates. If absent, "position"
is to be interpreted as

Re: Minor revisions before H5MD v1.0

Peter Colberg — 2013-05-05T19:59:49

Thanks for the proposal Pierre, it is fine with me.

Peter