Periodic Reporting for period 4 - Bits2Cosmology (Time-domain Gibbs sampling: From bits to inflationary gravitational waves)
Okres sprawozdawczy: 2022-10-01 do 2023-09-30
The detection of primordial gravity waves created during the Big Bang ranks among the greatest potential intellectual achievements in modern science. During the last few decades, the instrumental progress necessary to achieve this has been breathtaking. However, from the latest ultra-sensitive experiments such as BICEP2 and Planck, it is clear that instrumental sensitivity alone will not be sufficient to make a robust detection of gravitational waves. Contamination in the form of astrophysical radiation from the Milky Way obscures the cosmological signal by orders of magnitude. Even more critically are second-order interactions between this radiation and the instrument itself that lead to a highly non-linear and complicated problem. The defining goal of bits2cosmology is to integrate all steps from raw time-ordered data processing to high-level cosmological parameter estimation. Traditionally, this process has been carried out in a series of weakly connected steps, pipelining independent executables with or without human intervention. Some steps have mostly relied on frequentist statistics, while other steps have adopted a Bayesian approach; bits2cosmology will implement the first real-world CMB pipeline based on an end-to-end Bayesian approach called Commander3. Today, this defines the state-of-the-art in the community with respect to end-to-end Bayesian CMB analysis. This code has already been adopted as the main computational engine to re-analyze Planck LFI in collaboration with the BeyondPlanck project, and it has been adopted by the Cosmoglobe project to build a global model of the microwave and sub-mm sky. Commander3 will play a dominating role in the CMB community in the foreseeable future.
The bits2cosmology project started in April 2018, directly building on the recent Planck 2018 analysis and experience. According to the original plan, the first step would be to add support for time-ordered WMAP data in the then existing Commander2 code, which at the time only supported map-based component separation analysis. However, around the same time a project called BeyondPlanck was initiated that included many leading members of the original Planck LFI team. The primary goal of BeyondPlanck was to perform end-to-end analysis of the Planck LFI data, and deliver these products to the community. As such, there were uniquely valuable synergies that could be exploited between the bits2cosmology and BeyondPlanck projects: The BeyondPlanck collaboration included world-leading expertise on the Planck LFI data set, while the bits2cosmology project had world-leading expertise in CMB Gibbs sampling. For this reason, Planck LFI became the first priority for the bits2cosmology project as well.
This collaboration has been a resounding success, and the first major end-to-end Commander-based data release was presented to the public at the BeyondPlanck release conference on November 18-20, 2020. That release was based on a suite of 17 scientific papers describing both algorithmic breakthroughs and scientific applications. Prof. Hans Kristian Eriksen (PI of both bits2cosmology and BeyondPlanck) gave the opening talk at the conference, which was attended by more than 170 scientists from 27 institutions in 6 continents. Overall, the release conference was a resounding success for both BeyondPlanck, which now has delivered new state-of-the-art Planck LFI products to the community, and for the bits2cosmology project, which has managed to establish Commander3 as a new industry standard for end-to-end CMB analysis, and demonstrated this on a leading real-world data set.
In parallel, another ERC-funded project called Cosmoglobe (PI: Prof. Ingunn Kathrine Wehus at the University of Oslo) aims to combine all available state-of-the-art experiments into one coherent global model of the radio, microwave and sub-mm sky. As such, the algorithmic foundation developed in bits2cosmology represents an ideal framework for performing such work, and the work performed in bits2cosmology is already guaranteed to have a long-lasting impact because of the much larger Open Source Cosmoglobe project.
This collaboration has been a resounding success, and the first major end-to-end Commander-based data release was presented to the public at the BeyondPlanck release conference on November 18-20, 2020. That release was based on a suite of 17 scientific papers describing both algorithmic breakthroughs and scientific applications. Prof. Hans Kristian Eriksen (PI of both bits2cosmology and BeyondPlanck) gave the opening talk at the conference, which was attended by more than 170 scientists from 27 institutions in 6 continents. Overall, the release conference was a resounding success for both BeyondPlanck, which now has delivered new state-of-the-art Planck LFI products to the community, and for the bits2cosmology project, which has managed to establish Commander3 as a new industry standard for end-to-end CMB analysis, and demonstrated this on a leading real-world data set.
In parallel, another ERC-funded project called Cosmoglobe (PI: Prof. Ingunn Kathrine Wehus at the University of Oslo) aims to combine all available state-of-the-art experiments into one coherent global model of the radio, microwave and sub-mm sky. As such, the algorithmic foundation developed in bits2cosmology represents an ideal framework for performing such work, and the work performed in bits2cosmology is already guaranteed to have a long-lasting impact because of the much larger Open Source Cosmoglobe project.
Some of the main scientific breakthroughs made by bits2cosmology include:
1) Development of a new and efficient Gibbs-based CMB mapmaking algorithm that for the first time supports end-to-end error propagation of correlated noise at full angular resolution. This is an extremely important example of the synergies arising from the BeyondPlanck/bits2cosmology collaboration: The original idea was proposed by Dr. Elina Keihänen (BeyondPlanck), and then implemented and optimized in Commander by the bits2cosmology team. The result was a speed-up of almost one order of magnitude compared to the idea that was originally outlined in the bits2cosmology proposal, which was relied on traditional Conjugate Gradient-based maximum-likelihood mapmaking.
2) In-memory data compression. For the first time in CMB history, we now store time-ordered data compressed in memory, using so-called Huffman compression, which saves about a factor of five in the total amount of RAM needed for the analysis. This translates directly into a lower computational costs, and we are now in fact able to analyze the entire Planck LFI data set on one single compute node!
3) A detailed time-dependent model of the Planck LFI instrument, in which the instrument noise properties are allowed to vary from hour to hour. This new instrument model has in turn resulted in a fundamentally new understanding of both systematic effects in the LFI data and better sky maps overall.
4) A new Planck LFI gain model that uses external data from NASA's WMAP satellite mission to break internal degeneracies in the Planck data set. Likewise, using the new Planck data, we have isolated poorly measured modes in WMAP that later can be solved by also analyzing WMAP in terms of time-ordered data. As a result of this joint analysis, both experiments improve.
5) A new sky model, including novel constraints on polarized synchrotron emission. Importantly, we limit any spatial variations in the spectral index of polarized synchrotron emission, and show that this quantity is similar in two of the brightest regions in the sky. This is an important conclusion for future B-mode experiments.
6) The first ever cosmological constraints with true end-to-end Bayesian error propagation.
Overall, this development and demonstration was precisely the main goal for the entire bits2cosmology.
1) Development of a new and efficient Gibbs-based CMB mapmaking algorithm that for the first time supports end-to-end error propagation of correlated noise at full angular resolution. This is an extremely important example of the synergies arising from the BeyondPlanck/bits2cosmology collaboration: The original idea was proposed by Dr. Elina Keihänen (BeyondPlanck), and then implemented and optimized in Commander by the bits2cosmology team. The result was a speed-up of almost one order of magnitude compared to the idea that was originally outlined in the bits2cosmology proposal, which was relied on traditional Conjugate Gradient-based maximum-likelihood mapmaking.
2) In-memory data compression. For the first time in CMB history, we now store time-ordered data compressed in memory, using so-called Huffman compression, which saves about a factor of five in the total amount of RAM needed for the analysis. This translates directly into a lower computational costs, and we are now in fact able to analyze the entire Planck LFI data set on one single compute node!
3) A detailed time-dependent model of the Planck LFI instrument, in which the instrument noise properties are allowed to vary from hour to hour. This new instrument model has in turn resulted in a fundamentally new understanding of both systematic effects in the LFI data and better sky maps overall.
4) A new Planck LFI gain model that uses external data from NASA's WMAP satellite mission to break internal degeneracies in the Planck data set. Likewise, using the new Planck data, we have isolated poorly measured modes in WMAP that later can be solved by also analyzing WMAP in terms of time-ordered data. As a result of this joint analysis, both experiments improve.
5) A new sky model, including novel constraints on polarized synchrotron emission. Importantly, we limit any spatial variations in the spectral index of polarized synchrotron emission, and show that this quantity is similar in two of the brightest regions in the sky. This is an important conclusion for future B-mode experiments.
6) The first ever cosmological constraints with true end-to-end Bayesian error propagation.
Overall, this development and demonstration was precisely the main goal for the entire bits2cosmology.