ESS will be home to the brightest neutron source in the world, along with a suite of instruments that will allow great leaps forward in neutron scattering research. All of these elements will be producing huge amounts of data, so it’s essential that there’s an infrastructure there to be able to handle and interpret this information. During 2019 the ESS Data Management and Software Centre, or DMSC, reached more than one milestone on the way towards successful First Science and user operations.
The ESS facility, and the researchers that visit, will be relying on the data gathered directly from the instruments in use, and the DMSC made great strides of progress during 2019. The team are responsible for delivering the scientific computing applications and infrastructure required, including developing software from the ground up that’s specifically suited to ESS.
“The neutrons from the beamline have computing requirements, whether that’s for controlling the experiments themselves or acquiring, processing and analysing data,” says Jonathan Taylor, Head of DMSC Division. “Basically, the raw data counts the neutrons which get scattered from a sample in a unit called time of flight. This arrives as a time stamp, but we need to convert this data into useful science units, and it’s a considerable amount of effort to get this to work. Thanks to our fantastic team, and collaborations with other experts around the world, we now have technology that’s more than capable of the task at hand.”
This software has been designed and modified throughout the construction phase, which reached completion in 2019. This period also saw the development of the core technologies for the various beamlines as well as software for a number of different types of neutron research. In addition to these developments, ESS also opened a data centre in Copenhagen which is equipped to store all data, and make it accessible to those who need it.
“In the construction phase we developed software for all these different areas and instruments,” Jonathan continues, “and we also developed our own skill set and skill base. When we started, there were five of us. Now there are 28, and we always try to make sure we have the right people in the right place, working on the right things. Then we also opened the data centre, which was another big step towards First Science from the data management side.”
Cooperation is key
The DMSC and the new data centre are located in Copenhagen, Denmark, which requires a high-performance network connection between the facility in Lund and the data management and software centre side. Once operations begin, there will be vast amounts of information travelling between these two cities, sparking ideas and inspiration on both sides of the border, in both host countries. This collaborative spirit has run through every part of the DMSC project, and of the ESS project as a whole.
“DMSC was reliant on in-kind contributions, and key parts were developed by our partners,” explains Jonathan. “One of the benefits of the software development community is that you can naturally work remotely, so collaboration is already seamless. When you develop alongside other neutron facilities, like we did, everyone benefits. We don’t have to rebuild things that already exist, and other developers benefit from their software being built upon.”
In-keeping with this approach, DMSC, just like the European Union itself, is a major proponent of FAIR (Findable, Accessible, Interoperable and Reusable) and open data. This means that, after a specified embargo has passed, anyone can access the research data and use it as a foundation for further research a deeper understanding. In addition to open data, the new data centre will be able to offer users a cloud resource for neutron science. This means rather than having to buy and build huge computers, scientists can remotely access the compute they need in the ESS data centre.
“This open data will contribute to the European Open Science Cloud,” says Jonathan. “This is a big group of projects making open data viable for the European research community. To make data open, it also needs to be findable, accessible, interoperable and reusable. This requires a lot of effort and thought, but once it’s done it’s a huge benefit to everyone.”
Prepared for a bright future
The DMSC decided that as a significant milestone, it would show that the software worked on an actual beamline. The tests were carried out in Berlin at the BER II reactor, and the test conditions replicated what will be seen when ESS is open for science.
“We wanted to show that we could do neutron scattering as we needed to,” explains Jonathan. “Our job is to tie all the technology and instrumentation together with a usable system, and a lot of effort went into this integration. The test highlighted things to be fixed and improved, but it also shows that the core software was more than suitable for ESS.”
All the hard work during this construction phase has paid off, as DMSC now has a fully operational data centre alongside a state-of-the-art, bespoke software stack built to make the most of the world-leading technology in place at ESS. This is all thanks to collaboration between experts in the DMSC working alongside many other teams from the science and technical directorate at ESS.
“Building scientific software is like science in general,” Jonathan says. “There’s more than one way to do things, and occasionally things will go wrong, so collaboration is essential. You need the right people in the right places who are able to offer the guidance you need. Science is only right because it’s collaborative.”