LEO Dosimetry – SAA/GCR Separation Using ML - New subgroup

AWG Projects AIML AWG Open Projects
,
123

Hi @bose, welcome to the project, and thanks for offering to help!

We’re currently in the data-labeling stage. You can get started by following the Volunteer Guide and using the Labeling Tool linked above. Choose any day, complete the labeling, export the CSV, and upload it to the shared folder. There’s no need to reserve a day.

We’ll share more about the data analysis, ML, and software work as we move to the next stage. Please send me your email address in a private message so I can keep you updated.

Best,
Odysseas

3 Likes
124

Hi Abhi,

Thank you for the offer. We’d be happy to have you help with the validation side.

Any statistical evaluation of the labeled intervals would be valuable. There are still a few days left in the labeling phase, but feel free to work with the data collected so far.

For evaluation we currently have this script:

Feel free to build on it or improve it as you see fit.!

Best,
Martin

2 Likes
125

Final call for SAA manual labeling

Hi everyone, the labeling phase is approaching its deadline. If you would like to contribute, please complete any remaining SAA labeling you can and upload your CSV files to the shared folder before the phase closes. Every contribution helps us improve and validate the dataset.

You can find the Volunteer Guide and the SAA Manual Labeling Tool in the resources shared above. Please name each uploaded file YYYY-MM-DD_YourName.csv.

New team members: please also add yourselves to the SAA ML phonebook team sheet and fill in your skills. This will help us understand how you would like to contribute to the next stages of the project.

Thank you to everyone who has contributed so far!

3 Likes
126

Thanks, Martin! I can open the notebook. I’ll start with interval checks and duration summaries, keeping each volunteer’s labels separate, then compare agreement where the same periods were labeled. I’ll share the checker and a short report here for review.

1 Like
127

Hi,
My name is Hamza, and I’m from Morocco. I’m in my second year of college, and I’m interested in joining your project to practice what I’ve learned and improve my skills. I’m a beginner in data science, and I’m still learning.

1 Like
128

Hello @odylec and @martin!

My name is Sahithi. I recently joined the AI/ML AWG and came across the SAA/GCR separation project. I’m a CS student with experience in deep learning, ML systems, and research, and I’d love to get involved.

I saw that the manual labeling phase is wrapping up, so I was wondering if there are opportunities to contribute to the next stages, particularly model development, experiments, or evaluation. Happy to help wherever would be useful!

Thanks!

2 Likes
129

Hello colleagues,
we have completed another round of annotation for the SAA passes. I have evaluated the current data, and unfortunately, only half of the annotators produced usable data. Therefore, I suggest continuing with manual annotation for now.

image
image814×730 88.1 KB

However, if anyone would like to start training their ML model for the labeling, you can go ahead. I think about one week of manually annotated data is usable.

image
image3247×1140 302 KB

As for continuing with the manual annotation, @odylec will put together summary information for current annotators and newcomers to the group.

3 Likes
130

SAA/GCR Annotation – Quick Guide

Before starting, please review the instructions available at the top of the labeling tool and watch this short demonstration:

:movie_camera: SAA/GCR Annotation Video
“https://www.youtube.com/watch?v=0jNqs0UmlqE”

Important lessons from the latest annotation review

1. Check the full day and don’t miss passes

image
image771×302 68.3 KB

Please review the complete day carefully. The highlighted examples show peaks that were missed during annotation.

2. Don’t overlook faint spikes

image
image110×368 10.8 KB

Not every SAA passage appears as a large, obvious spike. Faint spikes like the example above should also be identified and annotated.

3. Use all three panels
Do not make the decision from the flux panel alone. Check all three panels and zoom in when needed, following the instructions provided in the labeling tool.

If you’re unsure about an annotation, please ask before submitting.

5 Likes
131

Newcomers, please add your details to the project’s “phonebook” here: SAA ML phonebook - Google Docs

Please include your skills next to your name so we know what we can reach out to you for.

You will also find links to other project resources there.

image|157px;x159px;
image|157px;x159px;157×159 6.1 KB

3 Likes
132

Thanks @martin and @odylec for the evaluation and the guide. I’ve added my details to the phonebook. Looking at the plot, my marks (n=87) are among the low overlap ones, so I’ll redo them following the three lessons (check the full day, catch faint spikes, use all three panels) before the next round.]

Two questions:

  1. Is ‘usable’ defined as the share of each marked interval that falls inside the NASA window, as on the plot’s x axis? If each annotator could get their own overlap numbers by day, we could see whether the problem is missed passes or shifted boundaries. [Skip this if the Volunteer CSV folder already has them.]
  2. For the ML option, which week is the usable one, and are the NASA windows available as a file? I’d like to try a baseline that uses that week as labels, with train and test split by day so neighboring days don’t leak into each other, and report boundary agreement with a tolerance of about 1 minute. Martin noted that the RadLab SPACEDOS timestamps were fitted to the geomagnetic field to about a minute. Since the NASA windows depend on the same field, should SPACEDOS data be evaluated separately to avoid circularity?
3 Likes
133

Hello everyone, @AIMLawg

I have prepared a sample annotation of 24 days to illustrate how I would like the data to be annotated.

Here are the key resources:

Feel free to annotate the same days multiple times, each manual annotation can differ, and that’s perfectly fine. Both new and returning volunteers are welcome to annotate days that have already been labelled.

You are also invited to participate in the statistical evaluation of the annotations, which you can find here: Volunteer CSV - Google Drive
This includes comparing how annotations differ across volunteers and benchmarking them against the SAA overflights identified using the NASA SRAG methodology (available in the original data: https://drive.google.com/file/d/1gpqXHXpY7BEdvzvAqR4PKHRmxHYzph4K/view?usp=sharing

Finally, you are welcome to start building your own ML models for automated SAA overflight detection. For model training, please use the following datasets: https://drive.google.com/file/d/1xyehqkV9UAqQ1JXwDnrz24X5IU_FhSJB/view?usp=sharing

For now, please split the sample data into training and test sets yourselves. As more volunteer-annotated data becomes available, we will expand these intervals. Finally, the models will be tested on a separate dataset that has not yet been annotated.

Happy AI training!

3 Likes
135

Hi @martin, thanks for putting together the 24-day sample annotation. It makes the target much clearer. Your talk on Wednesday also helped me see why careful human labels matter so much, since NASA’s method overestimates SAA passes.

One suggestion, building on what I raised at the meeting: the background changes over the day, so faint peaks are where annotators disagree most. It might help to let annotators mark each pass as “confident” or “uncertain,” so we can measure how much of the disagreement comes from faint peaks and how much from shifted start and end times.

I’d like to help in all three areas:

  1. Annotation: I’ll re-label the 24 reference days following your sample and the new guide (full day, faint spikes, all three panels), so there’s another independent set to compare against.
  2. Evaluation: I can build on the Colab notebook to make per-annotator, per-day comparisons against your reference and the NASA SRAG windows, separating missed passes from boundary shifts. Happy to coordinate with Abhi so we don’t duplicate his interval checks.
  3. ML: I’ll start on the physics-informed XGBoost baseline I signed up for, trained on the sample data with a train/test split by day, and share early results here.

Two questions:

  • For evaluation, should your 24-day sample be the main reference, given that the NASA windows overestimate passes?
  • Later on, could ML results help with the direction asymmetry you found between NW-to-SE and SW-to-NE passes? I’d be very interested in contributing there.

Thanks for leading this!
İdil

3 Likes
136

Hello @odylec and @martin,

Thank you for the volunteer invitation. I would like to contribute to all three areas, and have run an initial private, AI-assisted Python pilot locally using the supplied files. This has not yet been run or published on Kaggle.

1. Annotation support: prepared blinded review plots for three dates, an empty human-annotation template, and separately identified model-predicted intervals. Human review is pending. These are model predictions, not independent human labels.

2. Initial agreement audit: the supplied detector table contains 631,828 observations (15 October–31 December 2020). Martin’s interval file covers 24 dates; 194,827 observed samples fall on those dates. Using start-inclusive/end-exclusive intervals and provisionally treating unmarked observations on covered days as negatives, Martin versus the supplied SAA column gives sample-level F1 = 0.638 and recall = 0.471. A one-day comparison (15 October) between Frisko’s and Martin’s marks gives F1 = 0.897; this is only a schema-compatible initial example, not an assessment of all volunteers. Please confirm whether those dates are completely annotated and whether the Martin/Martin Kakona name variants should be combined.

3. ML baseline: chronological split of the 24 covered dates into 16 training, 4 validation and 4 local test dates. Inputs are only detector channels 9–27 (log1p); UTC, date, SAA, ddsi and flux are excluded. Logistic regression and Random Forest were compared with a majority-negative baseline; model and threshold selection used validation only. The selected Random Forest gives test F1 = 0.913 against Martin’s marks on 4–7 November, but only F1 = 0.575 and recall = 0.405 against the supplied SAA column.

The difference is important: reproducing human marks does not establish agreement with the SRAG criterion or operational performance. These are sample-level exploratory results, not event-level validation, a GCR/dose interpretation, or the project’s future independent final test. I would appreciate guidance on interval definitions and this disagreement before extending the analysis.

I have the executed notebook and result tables ready for review. Could you confirm the assisted-annotation protocol and whether the shared source data may be republished on Kaggle? I have not attached or publicly redistributed those source files. Written, asynchronous feedback works best for me.

Best regards,
Lissandra Kruse Fuganti
MSc student in Applied Computing — UEPG, Brazil

2 Likes
137

Happy to coordinate, İdil. I checked the updated reference and listed 67 SAA-flagged runs without overlapping human marks. I’m covering structure and timing. Could we coordinate on the passage and boundary comparisons?

138

Thanks @abhis01, that’s really helpful! Since you’re covering structure and timing, I’m happy to take the passage and boundary comparisons. I’ll compare entry/exit boundaries from the human labels against NASA’s flags (and later the ML outputs), tracking where they disagree and by how much. Could you share your list of the 67 runs? I’ll start with those so we don’t double up, and I can set up a shared sheet so we log results in the same format.

İdil

1 Like
139

@abhis01 I put it in the shared folder here: SAA Passage & Boundary Comparison - Google Sheets

1 Like
140

Thanks İdil, I’ve attached the 67 cases with source rows and UTC times. These are stretches flagged as SAA in the supplied data but not marked in Martin’s annotation file. Could you review them using all three panels and note any uncertain cases? We can compare boundaries separately for passages marked in both.

SAA_review_cases.md (13.1 KB)

1 Like
141

Thanks Abhi, got the file! I’ll go through all 67 cases across the three panels and mark each one as a clear SAA pass, not SAA, or uncertain, with a short note on why for the uncertain ones. I’ll aim to post the annotated list back here by Friday. After that I’m happy to do the boundary comparison on the passes we both marked. Would it help if I added start/end offsets in minutes for those, so we can see how much our boundaries drift from Martin’s?

İdil

1 Like
142

Thanks İdil, Friday sounds good! Yes, start/end offsets in minutes would help for the matched passes. Please use positive for later than Martin and negative for earlier.

2 Likes
143

Thank you all for your questions, suggestions, and analysis!

One general note relevant to all of you: SPACEDOS is a battery-powered standalone instrument, so its clock is not synchronised with any time reference. The timestamps were aligned to the L-shell parameter with an accuracy of approximately one minute. Please keep this in mind when interpreting boundary precision and comparing with NASA SRAG windows.

For human annotations, the data was split into days, but this division has no physical meaning. All statistics should always be computed on continuous data segments, regardless of calendar days.

I hope these answers help clarify things:

@lilmisscupcake
The NASA SRAG SAA windows are already part of the input data. You can find them here: https://drive.google.com/file/d/1gpqXHXpY7BEdvzvAqR4PKHRmxHYzph4K/view?usp=sharing

@idildoga.turkmen

  1. I don’t think a “confident/uncertain” flag is necessary. The assumption is that an annotator either sees a peak and marks it, or doesn’t see it and leaves it unmarked. If annotators disagree, that can be addressed statistically through decimation and we don’t need to aim to be better than any single annotator.

  2. For now, my 24-day annotation can serve as the training baseline for ML models. I hope these annotations will be refined and extended with the help of more annotators over time. The NASA SRAG method cannot be used as ground truth, but comparisons with it are valuable. For example, it would be interesting if someone finds a peak outside the SAA passes. I noticed a few such cases during annotation but skipped them, as they represent a different physical problem requiring a different explanation.

  3. ML models could eventually help explain SAA anisotropy, but I wouldn’t mix the two topics just yet. The group’s current goal is to produce ML models that perform comparably to a human annotator. We will then evaluate them against the NASA SRAG methodology and against human annotators across as large a dataset as possible. That should be the outcome of the group’s first joint paper.

@fuganti

  1. Yes, Martin and Martin Kakona are the same person, just one annotator.

  2. You are right that human annotations and the NASA SRAG windows do not match. However, it appears that all human-annotated intervals should fall within the NASA SRAG windows, so the human marks are a subset, not a contradiction.

  3. Publishing the data on Kaggle sounds like a great idea. As a first step, could you prepare a draft of the challenge description in a Google Drive document, including links to the relevant sources? I’ll review it and add anything that’s missing.

Thank you all for your contributions.

4 Likes