The Standard Model (SM) of particle physics has been a tremendous success, particularly with the discovery of the Higgs boson in 2012 by the ATLAS and CMS collaborations. Since then, research has focused on measurement of the properties of the newly discovered particle. A deviation of a measurement from the SM prediction would be a sign of new physics “beyond” the SM (BSM). The BSM physics is required to answer many remaining open questions in the SM. Our project “hh” aims to observe new physics in the Higgs boson sector through the search for a Higgs boson pair, considering both resonant and non-resonant production. Many BSM theories predict the existence of heavy particles that can decay to a pair of Higgs bosons, such as minimal supersymmetric extension of the SM, twin Higgs and composite Higgs models. These heavy particles are identified as a resonance in the Higgs boson pair invariant mass spectrum. The enhancement of the non-resonant Higgs boson pair cross-section might occur through loop corrections involving new particles or through non-SM couplings. This project is based on the data collected in 2015 and 2016 by the ATLAS experiment in the Large Hadron Collider at CERN. The results of this project indicate non-significant deviations from the SM predictions. Tight upper limits are set in both the non-resonant and resonant modes in the mass range 260 GeV to 1 TeV, improving the sensitivity by a factor up to 10 compared to the previous ATLAS results. The Higgs boson self coupling, a fundamental parameter, cannot be directly probed with the currently available data but was significantly constrained in this project. The prospects of measuring the Higgs boson self-coupling in the high luminosity LHC (HL-LHC) were also studied within this project.