I’m starting a small project on how sampling, repeated timestamps, and gaps affect what we can conclude from RadLab data. I’d like to coordinate the work and find a few collaborators.
The first release and report are here. It checks 18 DosTel1/DosTel2 instrument-day files covering 21,500 readings. There are 58 timestamps with differing rate values; the code preserves them and leaves affected series out of integration until their meaning is clear. These may have a valid instrument explanation.
The package also includes a reusable SAA annotation-interval checker, synthetic examples, exact source queries, and tests. Its masking results compare numerical integrals of reported rates, not validated physical dose.
I’d welcome help with:
instrument and timestamp interpretation;
checking the data and reporting useful edge cases;
independently reproducing the results.
@james.casaletto@martin, would this be a useful scope to develop within the AWG? I’ve also offered validation help in the LEO Dosimetry thread and want to coordinate with existing work. I’m proposing a project here; formal subgroup recognition is still pending.
If you’re interested, reply with which part you’d like to help with. The short project brief has the starting scope.
This is a great scope, happy to help. My background is AI/ML and bioinformatics with a focus on dataset engineering and validation, catching things like leakage, split issues, and metadata inconsistencies on other space biology projects I’m contributing to.
I’d like to help with independently reproducing the results and checking the data for edge cases, specifically the 58 timestamps with differing rate values you flagged. Happy to dig into whether there’s a pattern (specific instrument, specific time window) before assuming it needs a physical explanation, that kind of check is usually where I can add the most value.
Thanks, Hitaeshi! I’ve emailed you the release and a suggested starting task: reproducing the results and checking the timestamp patterns. I’ll handle the code and instrument questions. We can keep findings in the GitHub issue, or email if that’s easier.
If you plan to work with the SPACEDOS data, it is important to know that it is a battery-powered standalone device, meaning its internal clock has a certain time drift. The data imported into RadLab has gone through post-processing that fitted the data to the geomagnetic field with an accuracy of approximately 1 minute.
Therefore, you cannot expect higher precision from the current RadLab data. To achieve that, an additional, more precise fit would need to be performed.
There is one more thing I’d be interested in. Different instruments have different integration times. Some even have a variable integration time depending on the flux. If we were to normalize the gain (different instruments have different sensitivities to various particle energies, but let’s neglect that for now and just normalize the gain), what is the difference in the cumulative dose due to this variable integration time? Here is an example comparing measurements from DosTel2 and SPACEDOS02.
Thanks, Martin. I added a small synthetic check. It confirms that duration-weighted interval averages preserve cumulative dose over the same complete window; simple averaging and gaps can change the result.
Could you share the exports behind the December 4, 2019 DosTel2/SPACEDOS02A-F plot? It would help to know each interval’s duration, whether timestamps mark its start/middle/end, and any rebinning or live-time normalization. I’ll keep the roughly one-minute alignment limit separate from timestamp precision, and treat the real comparison as conditional on the gain/response assumptions.
You can download the data directly from RadLab as CSV. I’d recommend starting with SPACEDOS02A-S01 rather than F, the F build had a firmware bug causing larger jumps in real time. I’ll share the raw data separately. Timestamps mark the end of the integration interval.
Thanks @martin , this is very useful, especially that the timestamps mark the end of the integration interval and that the RadLab series is accurate to about 1 minute after the geomagnetic fit.
From Abhi’s public DosTel snapshots, I found two things that may connect to your integration time question:
There are 58 timestamps with two different readings, and all 58 fall exactly on 5 minute boundaries.
In 44 of those 58, the first listed rate has more than 6 decimal places (for example 283.136608814696), while the second listed rate never does, and neither do the 116 neighboring rows. Many of the long values are consistent with being means of n six decimal values (n = 313 in 18 cases).
Could you tell us whether the DosTel export mixes native readings with an aggregated or processed series at the 5 minute marks, and whether DosTel timestamps also mark the end of the interval? If so, I can rerun my neighboring sample check using interval midpoints. I’m also happy to run the duration weighted cumulative dose comparison of SPACEDOS02A-S01 against DosTel2 once the exports are available.
I am Lissandra Kruse Fuganti, a master’s student in Applied Computing at UEPG, Brazil. Following your invitation for independent reproduction, I have prepared and executed a public Kaggle notebook using your pinned v0.1.1 release (commit 1a0b211ba57676a53d7165f92a93caaa06076c44) and its committed public RadLab snapshots.
Reproduced 18 instrument-day cases and 21,500 readings, with 58 conflicting timestamp groups and 16 series excluded from numerical integration by the release diagnostics.
Your verification script passed: structure, integers, strings and hashes matched exactly; finite floating-point results were within the specified 1e-12 relative/absolute tolerance.
A separate CSV grouping check, without importing the upstream diagnostic functions, confirmed 34 groups for DosTel1 and 24 for DosTel2. Each contains two source rows with distinct rates, and all 58 fall on five-minute boundaries in the recorded clock. Source hashes were also checked; neighboring readings and source row numbers are exported in independent_timestamp_neighbors.json.
The fictional annotation examples produced 3 parsed intervals in 2 files, no structural findings, and 1 midnight review candidate.
Kaggle environment: Python 3.13.15, Linux x86_64; successful full run in approximately 28 seconds. The notebook includes the pinned public code and snapshots and runs without Internet or GPU. Original code and analysis are credited to Abhi Singh, with the MIT license retained; preparation was AI-assisted.
These are reproduced observations already reported in your work, not a new physical finding or instrument validation. Both values are preserved. Timezone, detector direction/channel and interval semantics remain unresolved. I have not fully reproduced the later timestamp-review report, and exporting neighboring readings does not establish their instrument-level interpretation.
Would this be useful as an initial contribution, and is there a specific next check you would recommend? I prefer written, asynchronous instructions and feedback because I have difficulty speaking English and following live English conversations.
Best regards,
Lissandra Kruse Fuganti
UEPG, Brazil
Thanks, Lissandra, this is helpful. Could you next reproduce the timestamp-review report and review packet in PR #3? Please include your Python version and let me know if theres any differences.
I have completed the PR #3 reproduction using commit 4d41d6a5393b38c96f227907618e39201ee364a6 and published Version 2 of my Kaggle notebook:
The full Kaggle run completed successfully in 38.8 seconds with Python 3.13.15 and Node.js v20.19.0 on Linux x86_64. All 67 tests passed, with no skips, failures or errors.
The timestamp-review CSV and JSON summary, neighboring-sample outputs, and all three review-packet files (HTML, JSON and Markdown) reproduce byte for byte. The regenerated timestamp report confirms 58 differing-rate groups containing 116 source rows, all on five-minute boundaries.
Across the 15 files with committed reference counterparts, 14 are byte-identical. The only difference is in the older baseline diagnostics.json: 27 floating-point values differ, with a maximum absolute difference of 2.4513724383723456e-13. All pass the repository’s 1e-12 relative/absolute tolerance; structure, counts, strings and hashes pass the semantic checks. No discrepancy was found in the requested timestamp report or review packet.
I included a differences report in Markdown and HTML, a CSV listing all 27 differing values, full file hashes, test logs and environment details. They are available under outputs/differences-report and in RadLab_PR3_generated_review_packet.zip on the Output tab:
These checks establish computational reproducibility. I have not independently tested interactive browser save/reload behavior or validated the physical interpretation of the readings. Both repeated readings remain preserved.
Best regards,
Lissandra Kruse Fuganti
UEPG, Brazil
Thanks, Lissandra. I checked your outputs. The timestamp report and review packet match exactly, and the small baseline differences are within tolerance. Appreciate you running this.
I’d like to join in this project and help with the instrument/timestamp interpretation.
I did an independent reproduction: v0.1.1, Python 3.13.17, Linux, all 22 tests pass, verify_reproduction.py passes, and review_timestamps.py output matches the committed table exactly.
I also think the repeated timestamps have a processing explanation. In all 18 files the rows split into two series by number format: a raw series (~7 sig. digits, ~100 s spacing, 17,480 rows) and a 5-minute series (~15 sig. digits, 300 s spacing, 4,020 rows, every one on a 5-minute boundary). The values in the 5-minute series look like means of 3 readings. On the 5-minute grid the two series are nearly mutually exclusive (3,976 slots have only the 5-min value, 1,156 only a raw reading, 44 both), and those 44 are exactly the 44 “long+short” pairs among the 58 differing-rate groups. That pattern looks like a coalesce step between a raw product and a gridded product rather than an instrument fault. The remaining 14 groups are raw+raw and need a separate look.
Happy to create a classify_series.py + test that tags rows and emits the grid-occupancy table, if that fits the scope.
I’m an undergraduate mechanical engineering researcher at Chung-Ang University, currently conducting research at Tennessee Tech, and a member of the OSDR AI/ML AWG.
I’m interested in contributing to this project, particularly through sampling sensitivity analysis, numerical integration uncertainty, and reproducibility validation using MATLAB/Python.
I was wondering if the project is still accepting collaborators and whether there are plans to develop the work into a peer-reviewed journal paper.
I’d be happy to help with a specific analysis task if there’s an opportunity. Looking forward to hearing from you!
Thanks George, happy to have you! Can you share your script, rule and commit? Lets keep both readings and use descriptive tags while Kirill checks their origin. The latest version is PR #3 (67 tests).
Hi Yeongho, happy to have you join! Could you try different bin widths and pulse positions against known totals? MATLAB or Python is fine. A paper is possible, but theres no submission plan yet.
Thank you for the opportunity! I’d be happy to work on this using MATLAB.
Could you share the relevant dataset or code, along with the reference totals and any specific bin widths or pulse configurations you’d like me to test?
I’ll start by comparing the results across different settings and evaluating the numerical errors.
