Getting Started

Motivations

A major advantage of our era is the easiness of gathering and recording data. Data can be found literary everywhere, from quantum scale to stars and galaxies. Great challenges are arising from this fact and the biggest of them all is how you can efficient manage all this information.

In particular for space data, technology has advanced so rapidly that every day, petabytes of useful information is recorded through satellites, other sensors around the globe, and simulations. However, a big part of this information is remain unused (or partly used) because the proper tools have not yet been created. New software packages can lead the way to win that challenge and fully (or partly) automate various processes that are necessary in order to transform them in a format that the scientists can understand and analyze.

This is the reason why we developed aidapy which is a high level Python package for the analysis of spacecraft data from heliospheric missions using modern techniques including data assimilation, machine learning (ML), artificial intelligence (AI), and advanced statistical models. The transition from proprietary software based on commercial languages to the new open-source community-developed packages in Python represent one of the main objective of aidapy. Our vision is to combine software tools that are already existing out there for different reasons, in a nice and efficient way. In our minds, this is a crucial and necessary task for the reducing of the knowledge “gap” that exist in several occasions between different scientific fields. aidapy has been created in order to fulfil the needs of a user with high programming skills as well as, for a user with basic programming knowledge.

Finally, we aim to build a new specific database for helping the ongoing trend in database standardization in space. Key heliophysics problems are selected to produce a database (AIDAdb) of new high-level data products that include catalogs of features and events detected by ML and AI algorithms from existing numerical simulations and observations. New simulations are also performed to enrich this database. Moreover, many of the AI methods developed in AIDA represent themselves higher-level data products, for instance in the form of trained models.

The transition to a big-data phase and the use of the big-data language of choice Python prepares the space community to the use of ML and AI. The latest AI developments and especially the methodology of deep neural networks are particularly suited for the identification of physical processes in space images and data. However, there is a barrier to overcome between the two very distinct communities of space scientists and experts in AI. Aidapy aims at bridging this gap linking the software developed in the AI community with heliospheric data.

We are developing aidapy to make it easier for a researcher in space science who does not have a background in computer science or in artificial intelligence to use some of the most advanced tools in this field and provide some concrete examples on how AI can help analyse data from simulations and observations, how AI can help discover physical processes hidden in the data and how it can make space weather forecasts.

Installation

The package aidapy has been tested only for Linux.

Using PyPi

aidapy is available for pip.

pip install aidapy

From sources

The sources are located on GitLab:

  1. Clone the GitLab repo:

Open a terminal and write the below command to clone in your PC the AIDApy repo:

git clone https://gitlab.com/aidaspace/aidapy.git
cd aidapy
  1. Create a virtual env

AIDApy needs a significant number of dependencies. The easiest way to get everything installed is to use a virtual environment.

  • Anaconda

You can create a virtual environment and install all the dependencies using conda with the following commands:

pip install conda
conda create -n aidapy
source activate aidapy
  • Virtual Env

Virtualenv can also be used:

pip install virtualenv
virtualenv AIDApy
source AIDApy/bin/activate
  1. Install the version you want via the commands:

For the “basic” version:

python setup.py install

For the version with the ML use cases:

pip install aidapy[ml]
  1. Test the installation in your PC by running. (Install both versions before running the tests)

python setup.py test

5) (Optional) Generate the docs: install the extra dependencies of doc and run the setup.py file:

pip install aidapy[doc]
python setup.py build_sphinx

Once installed, the doc can be generated with:

cd doc
make html

Dependencies

The required dependencies are:

Optional dependencies are:

Testing dependencies are:

Extra testing dependencies:

~