0 votes
110 views
I am having some difficulty parsing the datalab website in terms of determining the best way to generate, for example, a csv with the DES Y1 photometry from des_dr1.galaxies that I can interface with on my local machine. I assume this is a trivial task, but I seem to be missing some piece of assumed background knowledge for actually accomplishing it!
asked Feb 14 by anonymous | 110 views

1 Answer

0 votes

    The answer is that there are many ways to do this (hence the potential for confusion).  In general, for questions like this, one place to start is the Data Lab User Manual, http://datalab.noao.edu/docs/manual/index.html  .  In the Introduction (http://datalab.noao.edu/docs/manual/UsingTheNOAODataLab/Introduction/index.html), we provide an overview of services that lay out many of the ways you can do a query and return the result to local storage.  The rest of the documentation provides more specifics on the services, although it, again, not entirely complete.  

Here is more detail on a few ways that you could proceed:

Through virtual storage.  Virtual storage is aimed at working with large files.

====

0. Log in.

1.  Write a query either through the Web Query Interface (http://datalab.noao.edu/query.php) or through the queryClient in a Jupyter notebook (datalab.noao.edu/devbooks).  In the query interface, you could e.g. write:

SELECT TOP 10 * FROM des_dr1.galaxies

and output to Virtual storage.  Note that this is in ADQL format.  Or, in a notebook on the notebook server:

from getpass import getpass

from dl import authClient, queryClient, storeClient

token = authClient.login(your_username,getpass())

query = 'select top 10 * from des_dr1.galaxies'

response = queryClient.query(token, adql = query, fmt = 'csv', out="vos://test.csv")

storeClient.ls (token, name='vos://test.csv')

shows that the file is there.

2.  Transfer the file from virtual storage to your machine.  To do this, you need install the datalab client software on your machine: https://github.com/noao-datalab/datalab-client .  It’s documented at http://datalab.noao.edu/docs/manual/UsingTheNOAODataLab/CommandLineTools/TheDatalabCommand/TheDatalabCommand.html#data-lab-command-line-client

Then:

datalab login (only needs to be done once)

datalab get —fr=vos://test.csv —to=test.csv

Through notebook interface

=====

1.  As above, but:

from dl.helpers import convert

query = 'select top 10 * from des_dr1.galaxies'

response = queryClient.query(token, adql = query, fmt = 'csv', out="vos://test.csv")

df = helpers.convert(response,'pandas')

df.to_csv(‘test.csv’)

2.  Use notebook file interface to download

Through TOPCAT

====

1.  Connect to Data Lab TAP service through TOPCAT by entering url http://datalab.noao.edu/tap in TOPCAT’s “Query remote databases…” window

2.  Issue query as above, e.g.

SELECT TOP 10 * FROM des_dr1.galaxies

3. Use TOPCAT File->Save table

We are working to make the Virtual storage space more user friendly, so that it works more like Dropbox.  Stay tuned!

answered Feb 14 by kolsen (1,180 points)

92 questions

79 answers

25 comments

439 users

Welcome to Data Lab Help Desk, where you can ask questions and receive answers from other members of the community.