Difference ISN vs 10.7 Solar Flux vs High Resolution SN

[3gMike]

1 hour ago, Philalethes said:

but if that's the case it beats me what exactly they're using for the data.

It seems we may struggle to get an answer to that ! But, in any case, we can be fairly sure that Patrick's claim of a new Minimum will prove to be erroneous. That said, I still think it is possible that we will see an early maximum for this cycle, but that is a discussion to be had elsewhere.

[Philalethes]

On 5/7/2023 at 5:53 PM, 3gMike said:

It seems we may struggle to get an answer to that ! But, in any case, we can be fairly sure that Patrick's claim of a new Minimum will prove to be erroneous. That said, I still think it is possible that we will see an early maximum for this cycle, but that is a discussion to be had elsewhere.

I decided to send Jan an email about it, and there are two findings so far that can account for the differences:

1) The data provided by CLS apparently does not match the data directly from NRCan (the closest source to the Penticton data if I'm not mistaken), and most of the values after 2018 are apparently provisional or interpolated (there's no mention of what exactly that means that I've found). Many of those recent fluxes are actually a bit higher during those years than the data directly from NRCan, which is what he is using, accounting for part of the discrepancy.

2) He did mention that he's doing flare correction himself, which should presumably account for the rest of the difference; I've inquired about how exactly he's doing this, so I'll provide an update once I know more. I mentioned to him that this is likely the most contentious part of flux correction, and that there should hopefully be some way of doing it uniformly based on flare data. If not I imagine it could lead to resorting to ad-hoc adjustments as one sees fit, and thus biased data.

As per 1) I've downloaded the data directly from NRCan here and have redone some of the calculations with that in mind. Long-term the differences are minimal due to how this discrepancy primarily arose from somewhere in 2018, so I don't see any point in reposting the historical plots, the biggest differences are primarily that the differences in predicted flux and actual flux are actually somewhat smaller for SC25, so it actually fits more closely in that sense. The only plot I'll repost is the last one, since the 81-day average changes more significantly due to this difference, and the SC25 is a little smaller than what I showed; the new one looks like this:

This is still not below 170, so the remainder of the difference is due to the way he's correcting for flares. I'll be interested in what he says about that, and hopefully it's a reasonable method, but to my knowledge there's not as good data on flares going back as there is on the fluxes, so it probably does limit the dataset greatly. I'm also not completely sold on whether or not it's actually something you really want to do if you're trying to compare it to other proxies of Solar activity, but that's a different issue I suppose.

Edited May 8, 2023 by Philalethes
removed a few duplicates, data now in full agreement with that of Solen before flare-correction

[3gMike]

2 hours ago, Philalethes said:

I decided to send Jan an email about it, and there are two findings so far that can account for the differences:

1) The data provided by CLS apparently does not match the data directly from NRCan (the closest source to the Penticton data if I'm not mistaken), and most of the values after 2018 are apparently provisional or interpolated (there's no mention of what exactly that means that I've found). Many of those recent fluxes are actually a bit higher during those years than the data directly from NRCan, which is what he is using, accounting for part of the discrepancy.

2) He did mention that he's doing flare correction himself, which should presumably account for the rest of the difference; I've inquired about how exactly he's doing this, so I'll provide an update once I know more. I mentioned to him that this is likely the most contentious part of flux correction, and that there should hopefully be some way of doing it uniformly based on flare data. If not I imagine it could lead to resorting to ad-hoc adjustments as one sees fit, and thus biased data.

As per 1) I've downloaded the data directly from NRCan here and have redone some of the calculations with that in mind. Long-term the differences are minimal due to how this discrepancy primarily arose from somewhere in 2018, so I don't see any point in reposting the historical plots, the biggest differences are primarily that the differences in predicted flux and actual flux are actually somewhat smaller for SC25, so it actually fits more closely in that sense. The only plot I'll repost is the last one, since the 81-day average changes more significantly due to this difference, and the SC25 is a little smaller than what I showed; the new one looks like this:

This is still not below 170, so the remainder of the difference is due to the way he's correcting for flares. I'll be interested in what he says about that, and hopefully it's a reasonable method, but to my knowledge there's not as good data on flares going back as there is on the fluxes, so it probably does limit the dataset greatly. I'm also not completely sold on whether or not it's actually something you really want to do if you're trying to compare it to other proxies of Solar activity, but that's a different issue I suppose.

Good thinking. Let's see what Jan has to say.

However, the CLS file claims to source 10.7cm flux from Penticton / Ottawa (for early data).

I just did a quick search on flare corrections. For the 10.7cm record I only located 93 corrections in a total of 26120 readings. There have been no flare corrections in that record since  September 2017.

[Philalethes]

1 hour ago, 3gMike said:

Good thinking. Let's see what Jan has to say.

However, the CLS file claims to source 10.7cm flux from Penticton / Ottawa (for early data).

I just did a quick search on flare corrections. For the 10.7cm record I only located 93 corrections in a total of 26120 readings. There have been no flare corrections in that record since  September 2017.

What I meant to say was that I had already gotten a reply from him, and thus cleared up the data discrepancy. I agree that it's strange that the CLS file has all those provisional/interpolated values from 2018 and to today, especially since it appears to be updated continuously, but I did check some values here and there and found them to be different from the ones directly from NRCan. The ones I checked before that (where the flag is 0 in the CLS files) seemed to match the NRCan data. I switched to using that data instead, and then we agreed that the average for that would be 171.5 without correction for flares. It's ultimately the closest to the source, since NRCan are the ones collecting the data as far as I'm aware.

So now the discrepancy is primarily the flare correction. It seems like there's no such correction applied by NRCan themselves, so I believe the corrections in the CLS data is done by CLS themselves. I indeed also found only those same 93 corrected days there, which is what I mentioned earlier.

What Jan said was that he found the corrections of official datasets "lacking", hence why opting to do them himself instead. So that's essentially where the only remaining difference is, and since this correction is presumably done ad hoc for now it's not possible to reproduce for us until an exact method is devised. Personally I think more frequent measurements from more observatories would be better, and to then include increased flux from flares due to less chance of sampling errors. As I mentioned to him you could also have the reverse problem, e.g. a sudden drop in flux at the time of measurement, but I suppose that's not as common. Including all the variation seems the better solution if you want flux to be a measure of Solar activity in general to me.

As a last note, since the data from 2018 is what's most affected by changing to the data from NRCan, it should be mentioned that the 365-day average peak of SC25 now no longer is above that of SC24:

So I suppose Patrick gets to keep his claim in terms of flux for now. The rise in flux lately certainly indicates to me that it will be surpassed; checking quickly, simply keeping the daily flux at 140.0 for the next six months would be enough to surpass the SC24 peak due to the recent high values. Personally I expect even more, but we'll see.

[Patrick P.A. Geryl]

I improved my theory and submitted to Solar Physics.

High: 142.5 Sfu plus minus 6 percent

smoothed max: July 2023

smoothed max SSN: 104 in May 2023

 

 

[Philalethes]

55 minutes ago, Patrick P.A. Geryl said:

I improved my theory and submitted to Solar Physics.

High: 142.5 Sfu plus minus 6 percent

smoothed max: July 2023

smoothed max SSN: 104 in May 2023

The fluxes in terms of 365-day smoothing, I take it? Also, I assume this is based on Jan's own flare-corrected fluxes? I still think that's problematic, but at least these are some general predictions to go by; the SSN at least is still a relatively objective measure (I'm assuming that's the 13-month smoothed one by SILSO, the "official" one).

A max SSN of 104 sounds highly unlikely to me and many others, so let's see what time will bring, at the end of the year we'll have some indication.

[Patrick P.A. Geryl]

17 minuten geleden, Philalethes zei:

 

A max SSN of 104 sounds highly unlikely to me and many others, so let's see what time will bring, at the end of the year we'll have some indication.

Also plus minus 6 percent. Anyway May is the high.

[Patrick P.A. Geryl]

If you look at Solen Info you will see that the 2K SSN is close to the cycle 24 high while the 1K SSN lags a few percent more and the ISN IS OFF by 15 percent.

This shows there is something going on…

[3gMike]

4 hours ago, Patrick P.A. Geryl said:

If you look at Solen Info you will see that the 2K SSN is close to the cycle 24 high while the 1K SSN lags a few percent more and the ISN IS OFF by 15 percent.

This shows there is something going on…

Please be more specific. Exactly which Solen data are you referring to?

[Patrick P.A. Geryl]

54 minuten geleden, 3gMike zei:

Please be more specific. Exactly which Solen data are you referring to?

Look at the 365 day graphs under

[3gMike]

4 hours ago, Patrick P.A. Geryl said:

Look at the 365 day graphs under

I had already looked at them and that is why I asked you to be more specific. I do not think that they support what you are claiming.

For the sake of precision we need to note that the latest displayed data refers to the mid-point of the 365 day average, i.e. November 2022. Using those numbers we can compare with the maximum values in the previous cycle

For 2K data  242/250 gives current value as 97% of previous max.

For 1K data 156/170 gives current value as 92% of previous max.

For ISN data 104/117 gives current value as 89% of previous max.

I see nothing there to support your claim that ISN is off by 15%

[Patrick P.A. Geryl]

The flux is at 98%

2K at 97%. This is comparable.

so 1K and the ISN are too low in comparison.

Above that the ISN high from cycle 24 in comparison with the flux was lower then the other cycles…

[3gMike]

3 hours ago, Patrick P.A. Geryl said:

The flux is at 98%

2K at 97%. This is comparable.

so 1K and the ISN are too low in comparison.

Above that the ISN high from cycle 24 in comparison with the flux was lower then the other cycles…

You still have not explained why you claimed ISN is "off by 15%"

In any case I'm not sure that you have enough data to draw any meaningful conclusion.

[Patrick P.A. Geryl]

Op 7/6/2023 om 22:05, 3gMike zei:

You still have not explained why you claimed ISN is "off by 15%"

In any case I'm not sure that you have enough data to draw any meaningful conclusion.

At the end of May the 13 month smoothed flux was around 142 for November2022. According to my peer reviewed formula the ISN should be around 120. It was 101…

Edited June 10, 2023 by Patrick P.A. Geryl

[3gMike]

12 hours ago, Patrick P.A. Geryl said:

At the end of May the 13 month smoothed flux was around 142 for November2022. According to my peer reviewed formula the ISN should be around 120. It was 101…

That explains how you arrived at the figure of 15% low for the ISN, but that means nothing without also looking at the figures for 2K and 1K. Further, you cannot reasonably expect that your formula will be absolutely accurate for every point on the plot.

Since your formula is meant to predict 10.7cm flux it makes more sense to compare the predicted value of flux with the measured value. Doing this gives a different perception.

ISN = 101, 10.7 cm = 142, Calculated flux = (101 x 0.635) + 64 = 128

Calculated flux / Measured flux = 128/142 = 0.90, i.e. Prediction is 10% low

If we do the same calculation for the 1K and 2K values they happen to produce an absolutely accurate prediction (in this instance).

I then decided to examine the variation in predictions for different levels of activity. For each level I used three points in time - first for rising edge of Cycle 24, then falling edge of Cycle 24 and finally for rising edge of Cycle 25. Since you suggest that the 2K figures give the most accurate prediction of 10.7cm flux I used the 2K values as a reference point. I think that you may find the results informative.

As you can see, none of the series (1K,2K,ISN) give consistent results. What is particularly interesting is the range of variation in predicted results for each series....

2K predicted flux varies -6% to +5% compared to measured values

1K predicted flux varies -8% to +5% compared to measured values

ISN predicted flux varies -12% to +1% compared to measured values

So we can say that, for 10.7cm flux in the range 114 to 145 your formula tends to produce low values for the ISN prediction.

It is also worth noting that Ref 1 produced almost identical predictions for all three series, but they were all 6% to 8% low compared to measured value, then Ref 12 showed perfection for 1K and 2K series, with -9% for ISN. Other lines give a spread for each series, with ISN predictions generally being the lowest, but sometimes still being the most accurate (i.e. other series predicted too high).

I do not believe that you can make any reliable predictions based on this limited set of data.

[Patrick P.A. Geryl]

Major flaws in the ISN

compare February with June, The ISN should be about the same. We find a difference of 50! The high resolution 1K and 2K are about the same!

February

10.7 flux 163.2

ISN.  110.9

June

10.7 flux 166.8

ISN 163.4

[Jesterface23]

I wonder if there could be something in the 10.7 flux too. I'm not sure how much correlation there is, but the x-ray flare count this cycle seems fairly high.