[SQM] (no subject)

Christopher Kyba christopher.kyba at wew.fu-berlin.de
Tue May 15 13:47:16 UTC 2012


Dear colleagues,

[Executive summary: A standard format for recording light-at-night data has
been developed, and you are being asked to endorse the format and make
comments if you notice something that needs improvement.]

The introduction of the SQM and the IYA Lightmeter have led to a large
number of permanent skyglow monitoring stations coming online. At the same
time, a number of individuals and groups have developed their own
non-commercial devices (e.g. NSBM from the IDA, and DigiLum from Henk
Spoelstra).  While these developments are excellent news for those
interested in monitoring light pollution, there exists no common standard
for recording measurements from the devices.  This has greatly hampered
efforts to compare measurements from different locations, and to develop
databases containing long term measurements from around the world.

At the recent Cabauw Lightmeter InterComparison (CLIC) workshop, a group of
light pollution researchers defined a proposal for a new, standard format
to be used for recording skyglow measurements (attached).  The goal of the
standardization is to make comparisons of future measurements easier,
regardless of whether the measurements were taken by currently available
detectors, or by new, future detectors.  For this reason, we have attempted
to define a format that anticipates the needs of potential future devices
(e.g. multiple channels with different filters or opening angles), but
without making the format so open as to not actually be helpful.

In addition to the attached definition of the format, two example data
files written in the new format (for the SQM and Lightmeter) are included.
The rational for various decisions is included at the bottom of this email.

The participants at the CLIC workshop reached a consensus on this data
format after an extremely long discussion.  It does not match the format
currently used by any of the workshop participants, and is not the way any
of us would choose to do things if left to our own devices.  Regardless of
this, we were all in agreement that the benefits of an agreed upon common
light pollution data format vastly outweigh the one time inconvenience of
developing a converter to transfer such data into our favorite format for
data analysis.  (The one place we could not reach consensus was on the
nature of the data delimiter.  If you care about this, please indicate your
preference when you endorse the format.  If one delimiter is chosen by over
50% of endorsers it will be used, otherwise an online poll will be held to
make the final decision.  The most popular delimiters at the CLIC workshop
were space and semicolon.)

We now invite you, as a member of the light pollution research community,
to comment on the proposed format, and hopefully to endorse it.  We ask
that you respond with your comments by May 29 (if you need more time than
that please let us know as soon as possible).  After that date, the format
will be finalized, and we will issue a final call for endorsements.  We
will then officially announce the new format in a letter to an astronomy
journal, including in the appendix a list of those who endorsed it.  We
will also provide Unihedron with a new perl script that writes out data in
this format.  The makers of the popular SQM Reader programs have also
indicated that they will provide an updated version at some point after the
format is finalized.

To endorse the proposed standard light at night data format, please send an
email to Dorien Lolkema <dorien.lolkema at rivm.nl>, including your name and
institution (if applicable).

If you have comments or suggestions regarding the format, you can either
let us know by email privately, or post a public comment on one of the
sqm at unihedron.com or LPResearch at yahoogroups.com lists.

Best regards,

Christopher Kyba and Dorien Lolkema




General principles guiding the design of the format
====================================

Anyone is free to write out two data files (e.g. in old and new format) if
they so choose.  It should therefore be a simple matter to adopt this
format without disrupting any current analysis program.

This is is a "level0" data format.  This means reporting of the raw values
that came from the device, and not calibrated values.  The level0 data file
should never be changed after it is written.  (This means that you should
not subtract off the 0.11 mag/arcsec^2 from an SQM when writing the file,
you should write out what the SQM actually reported.)

New files should be opened once per 24 hour period.  The decision of what
local or UTC time to open a new file is left up to the users, because some
devices also record data in the daytime.

The header should only appear once in each data file.  If the acquisition
program is restarted for some reason a new file should be opened.

The data must include both a universal and a local time.  Local time is
useful because there are occasionally events that occur repeatedly at a
particular local time (e.g. stadium lights going off), and because it makes
it easier for the user to examine the raw data.

UTC time was chosen over other options (e.g. JD, unix time) due to its
improved human readability and ease in handling leap seconds.

The data format is for stationary monitors only.  An almost equivalent
format will be announced for GLOBE at Night data (or for future devices in
which the output of the device contains a location).  If you take moving
data with an SQM, your acquisition program should still write out the SQM
data in this format, since it's the raw SQM data.

There should never be "blank" entries.  If a value is not applicable for
your device, then -9999 should be written.  If the device does not report a
value, or if the value is corrupted, then -9998 should be written.

Users are free to decide whether or not to release their data under the
ODbL (or other license).  We encourage everyone to use ODbL, but understand
that some researchers may have a need to embargo their initial data before
joining a present or future global network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://unihedron.com/pipermail/sqm/attachments/20120515/8ab870e9/attachment.html 
-------------- next part --------------
Definition of the proposed "light at night" standard data format
Version 0.1
May 15, 2012


Header information
==================
(examples in [], only one header allowed per filem seperate multiple entries in a single line with ", ")

# Data format version
# Number of header lines
# Data license information
# Type of device [SQM, IYAL, NSBM, DigiLum]
# Instrument ID [SQM-RIVM1]
# Data supplier [Constance Walker]
# Location name [Berlin Dahlem]
# Location coordinates (decimal degrees, elevation in m) [lat/lon/elev]
# Timezone [America/Mexico_City]
# Moving or Stationary observatory [STATIONARY / MOVING]
# Number of device channels
# Filter used for each channel [Johnson V, HOYA CM-500]
# Measurement direction (in degrees) for each channel (azimuthal angles increasing clockwise from 0 for directly North) [z1, az1, z2, az2]
# Field of view for each channel (in degrees) [FOV1, FOV2]
# Number of entries in each data line
# Device specific characteristics [see example files]
# User comments line 1
# User comments line 2
# User comments line 3
# Extra blank lines to allow for future changes in the header
# data line entries in human format [see example files]
# End of header statement


Data lines:
===========

UTC Date;UTC Time;Local Date; Local Time;Data

Dates and times are recorded as:

yyyy mm dd hh mm ss.ddd  (Leading and trailing zeros are mandatory!)

The data delimiter is not yet fixed.  The most popular candidates are
space and semicolon (i.e. " " and ";").


Unavailable data rules
======================
-9999 if not applicable
-9998 if not entered for some reason




File naming, and frequency of starting new files
================================================

Filename: YYYYMMDD_HHmmss_[Instrument ID].dat

example:  20120427_120000_SQM-RIVM1.dat


A new file should be started each day, at a time chosen by the user.

Files can only contain one header.  If for some reason the data taking
is interrupted and must be restarted, then a new file should be
opened.  Data should be written in this new file until the standard
"new file start time" of the user.
-------------- next part --------------
# Light Pollution Monitoring Data Format 0.1
# Number of header lines: 30
# This data is released under the following license: ODbL 1.0 http://opendatacommons.org/licenses/odbl/summary/
# Device type: SQM-LE
# Instrument ID: Dahlem_tower_le
# Data supplier: Christopher Kyba / Freie Universitaet Berlin
# Location name: Berlin Dahlem
# Position: 52.4577, 13.3107, 91
# Local timezone: Europe/Berlin
# Moving / Stationary: STATIONARY
# Number of channels: 1
# Filters per channel: HOYA CM-500
# Measurement direction per channel: 0., 0.
# Field of view: 20
# Number of entries per line: 8
# SQM serial number: 1687
# SQM firmware version: 4-3-21
# SQM cover offset value: -0.11
# SQM test ix: i,00000004,00000003,00000021,00001687
# SQM test rx: r, 18.73m,0000000004Hz,0000130978c,0000000.284s, 031.2C
# Comment: SQM installed on a tower.
# Comment: last cleaned on 20120511
# Comment: 
# blank line 23
# blank line 24
# blank line 25
# blank line 26
# UTC Date / UTC Time / Local Date / Local Time / Temperature / Counts / Frequency / MSAS
# YYYY;MM;DD;HH;mm;ss.ddd;YYYY;MM;DD;HH;mm;ss.dddd;Celcius;number;Hz;mag/arcsec^2
# END OF HEADER
2012;05;11;00;00;04.000;2012;05;11;00;00;04.000;31.2;130978;4;18.73
2012;05;11;00;00;09.000;2012;05;11;00;00;09.000;31.2;130978;4;18.73
2012;05;11;00;00;14.000;2012;05;11;00;00;14.000;31.2;130978;4;18.73
2012;05;11;00;00;19.000;2012;05;11;00;00;19.000;31.2;130978;4;18.73
2012;05;11;00;00;24.000;2012;05;11;00;00;24.000;31.2;130978;4;18.73
2012;05;11;00;00;29.000;2012;05;11;00;00;29.000;31.2;130978;4;18.73
-------------- next part --------------
# Light Pollution Monitoring Data Format 0.1
# Number of header lines: 30
# This data is released under the following license: ODbL 1.0 http://opendatacommons.org/licenses/odbl/summary/
# Device type: IYA Lightmeter (Mark 2.3)
# Instrument ID: DE_BERLIN_4
# Data supplier: Christopher Kyba / Freie Universitaet Berlin
# Location name: Berlin Dahlem
# Position: 52.4577, 13.3107, 91
# Local timezone: Europe/Berlin
# Moving / Stationary: STATIONARY
# Number of channels: 1
# Filter per channel: None
# Measurement direction per channel: 0., 0.
# Field of view: 180
# Number of entries per line: 6
# IYAL serial number: -9998
# IYAL firmware version: -9998
# IYAL test line: 23.3.2012;15:24:58;27,6;°C;2984760;0;0;ok;
# Comment: The shadow of our building falls on the lightmeter around noon
# Comment: 
# Comment: 
# blank line 21
# blank line 22
# blank line 23
# blank line 24
# blank line 25
# blank line 26
# UTC Date / UTC Time / Local Date / Local Time / Temperature / Counts
# YYYY MM DD HH mm ss.ddd YYYY MM DD HH mm ss.ddd Celcius number
# END OF HEADER
2012 03 23 15 24 59.000 2012 03 23 16 24 59.000 27.5 2984520
2012 03 23 15 25 01.000 2012 03 23 16 25 01.000 27.5 2984640
2012 03 23 15 25 02.000 2012 03 23 16 25 02.000 27.5 2984280
2012 03 23 15 25 03.000 2012 03 23 16 25 03.000 27.6 2984520
2012 03 23 15 25 04.000 2012 03 23 16 25 04.000 27.5 2984760
2012 03 23 15 25 05.000 2012 03 23 16 25 05.000 27.6 2984640
2012 03 23 15 25 06.000 2012 03 23 16 25 06.000 27.6 2984760
2012 03 23 15 25 07.000 2012 03 23 16 25 07.000 27.5 2984760
2012 03 23 15 25 08.000 2012 03 23 16 25 08.000 27.5 2985000
2012 03 23 15 25 09.000 2012 03 23 16 25 09.000 27.5 2984880
2012 03 23 15 25 10.000 2012 03 23 16 25 10.000 27.5 2984880
2012 03 23 15 25 11.000 2012 03 23 16 25 11.000 27.6 2984760


More information about the sqm mailing list