Getting Started¶
Motivations¶
A major advantage of our era is the easiness of gathering and recording data. Data can be found literary everywhere, from quantum scale to stars and galaxies. Great challenges are arising from this fact and the biggest of them all is how you can efficient manage all this information.
In particular for space data, technology has advanced so rapidly that every day, petabytes of useful information is recorded through satellites, other sensors around the globe, and simulations. However, a big part of this information is remain unused (or partly used) because the proper tools have not yet been created. New software packages can lead the way to win that challenge and fully (or partly) automate various processes that are necessary in order to transform them in a format that the scientists can understand and analyze.
This is the reason why we developed aidapy
which is a high level Python package for the analysis of spacecraft
data from heliospheric missions using modern techniques including data assimilation, machine learning (ML), artificial intelligence (AI), and advanced statistical models.
The transition from proprietary software based on commercial languages to the
new open-source community-developed packages in Python represent one of the main objective of aidapy
.
Our vision is to combine software tools that are already existing out there for different reasons, in a nice and efficient way.
In our minds, this is a crucial and necessary task for the reducing of the knowledge “gap” that exist in
several occasions between different scientific fields.
aidapy
has been created in order to fulfil the needs of a user with high programming skills as well as, for a user with
basic programming knowledge.
Finally, we aim to build a new specific database for helping the ongoing trend in database standardization in space. Key heliophysics problems are selected to produce a database (AIDAdb) of new high-level data products that include catalogs of features and events detected by ML and AI algorithms from existing numerical simulations and observations. New simulations are also performed to enrich this database. Moreover, many of the AI methods developed in AIDA represent themselves higher-level data products, for instance in the form of trained models.
The transition to a big-data phase and the use of the big-data language of choice Python prepares the space community to the use of ML and AI. The latest AI developments and especially the methodology of deep neural networks are particularly suited for the identification of physical processes in space images and data. However, there is a barrier to overcome between the two very distinct communities of space scientists and experts in AI. Aidapy aims at bridging this gap linking the software developed in the AI community with heliospheric data.
We are developing aidapy
to make it easier for a researcher
in space science who does not have a background in computer science or in artificial
intelligence to use some of the most advanced tools in this field and provide some concrete
examples on how AI can help analyse data from simulations and observations, how AI can help
discover physical processes hidden in the data and how it can make space weather forecasts.
Installation¶
The package aidapy has been tested only for Linux.
Using PyPi¶
aidapy
is available for pip.
pip install aidapy
From sources¶
The sources are located on GitLab:
Clone the GitLab repo:
Open a terminal and write the below command to clone in your PC the AIDApy repo:
git clone https://gitlab.com/aidaspace/aidapy.git
cd aidapy
Create a virtual env
AIDApy needs a significant number of dependencies. The easiest way to get everything installed is to use a virtual environment.
Anaconda
You can create a virtual environment and install all the dependencies using conda with the following commands:
pip install conda
conda create -n aidapy
source activate aidapy
Virtual Env
Virtualenv can also be used:
pip install virtualenv
virtualenv AIDApy
source AIDApy/bin/activate
Install the version you want via the commands:
For the “basic” version:
python setup.py install
For the version with the ML use cases:
pip install aidapy[ml]
Test the installation in your PC by running. (Install both versions before running the tests)
python setup.py test
5) (Optional) Generate the docs: install the extra dependencies of doc and run the setup.py file:
pip install aidapy[doc]
python setup.py build_sphinx
Once installed, the doc can be generated with:
cd doc
make html
Dependencies¶
The required dependencies are:
Python >= 3.6
scikit-learn >= 0.21
numpy >= 1.18
scipy >= 1.4.1
matplotlib >= 3.2.1
pandas >= 1.0.3
heliopy >= 0.12
sunpy >= 1.1.2
astropy >=4.0.1
xarray >=0.15
bottleneck >= 1.3.2
heliopy-multid >= 0.0.2
Optional dependencies are:
Testing dependencies are:
pytest >= 2.8
Extra testing dependencies:
~