Script your Data Analysis

For conducting an investigation of a large dataset, you will probably want to script your analysis after you have developed intuition for it through interactive exploration. The Data Lab will support scripted analysis through:

  • A Storage Manager to handle user Virtual Storage space
  • A Query Manager for synchronous and asynchronous queries to the Data Lab database, personal user database storage, and queries to external data services
  • A Job Manager to handle running of jobs on Data Lab servers
  • The ability to trigger computation automatically by assigning Capabilities to a user’s Virtual Storage space
  • Examples of running workflows inside Containers for easy transport of computing jobs
  • Custom scripted tools for specific science cases, include a quick-look photometric pipeline

Identification of the Hydra II dwarf galaxy through automated catalog filtering and differential convolution of the point source distribution. The Hydra II dwarf appears as the dark spot at lower left in the plot on the right, whereas the its color-magnitude diagram appears as the red points in the plot on the left. Retrieval of the catalog data and computation of the convolution took ~30 seconds from inside the NOAO network.

demonstration of the Kernel Density Estimation technique applied to Hydra II catalog data from within an iPython notebook

Re-discovery of the Eridanus II dwarf galaxy using the Data Lab quick-look photometry pipeline NIKE, compared to the original discovery by Koposov et al. (2015). NIKE processes a typical set of full DECam images in just a few minutes, including transfer time from the NOAO Science Archive.