dlinterface module

class dlinterface.DLInteract[source]

Main class for Data Lab interactions

get(section, param)[source]

Get a value from the configuration file.

save(section, param, value)[source]

Save the configuration file.

class dlinterface.Dlinterface(verbose=True)[source]

Data Lab python interface super-class with methods for each command.

broadcast(type=None, pars=None)[source]

Broadcast a SAMP message

cp(source=None, destination=None, verbose=True)[source]

Copy a file in Data Lab VOSpace.

Parameters:
  • source (str) – The name of the file in VOSpace to copy, e.g. file1.txt.
  • destination (str) – The new name of the file in VOSpace, e.g. newfile1.txt.

Example

Copy the file file.txt to newfile.txt.

dl.ls()
file1.txt

dl.cp('file1.txt','newfile.txt')

dl.ls()
file1.txt, newfile.txt
dropdb(table=None)[source]

Drop a user’s MyDB table.

Parameters:table (str) – The name of a specific table in mydb to drop.
Returns:list – The list of properties of table or all tables in mydb.
Return type:str

Example

Drop the MyDB table called table.

print dl.listmydb()
table
table2

dl.dropmydb('table')
table
table2

print dl.listmydb()
table2
get(source=None, destination=None, verbose=True)[source]

Get one or more files from Data Lab.

Parameters:
  • source (str) – The name of the source file on VOSpace, e.g. file2.txt.
  • destination (str) – The name of the local destination file, e.g. file1.txt.

Example

Get a query output table called table1_output.txt from VOSpace.

dl.get('table1_output.txt','table1_output.txt')
(1/1) [====================] [   9.1K] table1_output.txt
help(command=None)[source]

Print out useful help information on the Data Lab python interface and it’s commands.

launch(dir=None)[source]

Launch a plugin in Data Lab

listdb(table='')[source]

List the user’s MyDB tables.

Parameters:table (str) – The name of a specific table in mydb. If this is blank then all tables will be listed.
Returns:list – The list of properties of table or all tables in mydb.
Return type:str

Example

List the MyDB tables.

print dl.listmydb()
table
table2
ln(target=None, link=None)[source]

Link a file in Data Lab VOSpace.

Parameters:
  • target (str) – The name of the file in VOSpace to link to, e.g. file1.txt.
  • link (str) – The name of the link, e.g. file1link.

Example

Create a link called iamlink to the file file1.txt.

dl.ls()
file1.txt

dl.ln('file1.txt','iamlink')

dl.ls()
file1.txt, iamlink
login(user=None)[source]

Login to Data Lab using username.

Parameters:user (str) – The Data lab username. If this is not given, then the user will be prompted for the information.

Example


Login and give the username,

dl.login(‘myusername’) Enter password: *** Welcome to the Data Lab, myusername

or,

dl.login() Enter user: myusername Enter password: ** Welcome to the Data Lab, myusername
logout(unmount=None, verbose=True)[source]

Logout out of the Data Lab.

Example

Logout of Data Lab.

dl.logout()
'myusername' is now logged out of the Data Lab
ls(name='vos://', format='csv', verbose=False)[source]

List files in VOSpace.

Parameters:
  • name (str) – The name of a specific file to list. If name is blank then all files will be listed.
  • format (str) – The format to use.
  • verbose (bool) – Give more verbose output, or just a list of files. The default is verbose=True.
Returns:

results – The list of files in VOSpace.

Return type:

str

Example

List the files.

dl.ls()
test2  test1

Verbose listing of the files in the public/ directory.

dl.ls('public',verbose=True)
lrw-rw----  demo15      0B  17 May 2017 14:04:25  thisisalsoalink -> /public/smash2
lrw-rw----  demo15      0B  17 May 2017 13:58:04  thisisalink -> /smash1
-rw-rw-r--  demo15    3.4K  17 May 2017 09:40:13  smash2
-rw-rw-r--  demo15    3.4K  17 May 2017 07:34:54  smash1
drw-rw----  demo15      0B  17 May 2017 14:05:02  data/  tableingester,downloader,runner
mkdir(name=None)[source]

Create a directory in Data Lab VOSpace.

Parameters:name (str) – The name of the directory in VOSpace to create, e.g. results.

Example

Create the directory data1/.

dl.mkdir('data1')
mv(source=None, destination=None, verbose=True)[source]

Move a file in Data Lab VOSpace.

Parameters:
  • source (str) – The name the file in VOSpace to move/rename, e.g. file1.txt.
  • destination (str) – The new name of the file in VOSpace (e.g. newfile1.txt) or the directory to move it to.

Example

Rename the file file.txt to newfile.txt.

dl.ls()
file.txt

dl.mv('file.txt','newfile.txt')

dl.ls()
newfile.txt

Move the file output.fits to the results/ directory.

dl.ls()
output.txt, results

dl.mv('output.fits','results/output.fits')

dl.ls()
results/output.txt
put(source=None, destination=None, verbose=True)[source]

Put files into Data Lab VOSpace.

Parameters:
  • source (str) – The name of a local file to upload to VOSpace, e.g. file1.txt.
  • destination (str) – The name of the destination file with, e.g. file2.txt. The destination file can have the vos:// prefix but it is not required.

Example

Put a catalog called cat.fits into VOSpace.

dl.put('cat.fits','cat.fits')
(1 / 1) cat.fits -> vos://cat.fits
query(query=None, qtype='sql', fmt='csv', out=None, async=False, profile='default', verbose=True)[source]

Send a query to a remote query service.

Parameters:
  • query (str) –

    The query string that will be passed to the queryClient and then to the DB query manager. This can either be in the SQL or ADQL format (specified by the “type” parameter). For example,

    'select ra,dec from gaia_dr1.gaia_source limit 3'
    
  • qtype (str) – The query format, SQL or ADQL. SQL is used by default.
  • fmt (str) –
    Format of the result to be returned by the query. Permitted values are:
    • ‘csv’ the returned result is a comma-separated string that looks like a csv file (newlines at the end of every row)
    • ‘string’ same as csv
    • ‘array’ Numpy array
    • ‘structarray’ Numpy structured / record array
    • ‘pandas’ a Pandas data frame
    • ‘table’ in Astropy Table format
    • ‘votable’ result is a string XML-formatted as a VO table
  • out (str or None) – The output name if the results are to be saved to mydb (mydb://tablename), to VOSpace (vos://filename), or the local file system (file:// and other names with no prefix). The files are in csv format.
  • async (bool) – If True, the query is asynchronous, i.e. a job is submitted to the DB, and a jobID is returned. The jobID must be then used to check the query’s status and to retrieve the result (when status is COMPLETE). Default is False, i.e. synchroneous query.
Returns:

result – If async=False and out is not used, then the return value is the result of the query in the requested format (see fmt). If out is given then the query result is saved to a file or mydb. If async=True the jobID is returned with which later the asynchronous query’s status can be checked (dl.querystatus()), and the result retrieved (see dl.queryresults().

Return type:

str

Example

A simple query returned as a pandas data frame.

data = dl.query('SELECT * from smash_dr1.source LIMIT 100',fmt='pandas')
Returning Pandas dataframe

type(data)
pandas.core.frame.DataFrame

print data['ra'][0:3]
0    103.068355
1    103.071774
2    103.071598

Perform a query and save the results to a table called “table1.txt” in mydb.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='mydb://table1.txt')

 dl.listmydb()

Perform the same query and save it to a local file.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='table1.txt')

ls
table1.txt
queryhistory(async=None)[source]

Report the history of queries made so far.

Parameters:
  • async (bool) – A boolean (True/False) of whether to only show the ASYNC queries. By default all quries are shown.
  • Results
  • -------
  • information on part queries is output to the screen with the following (The) –
  • columns (query ID, submission time, query type (sql/adql), sync or async query, jobid (for async queries),) – output format, status of query (or number of returned rows if sync query), query string

Examples

Perform some queries and then list the history.

data1 = dl.query('select ra,dec from smash_dr1.source limit 100',fmt='csv')
Returning CSV formatted table as a string

data2 = dl.query('select ra,dec from smash_dr1.source limit 500',fmt='pandas')
Returning Pandas dataframe

data3 = dl.query('select ra,dec from smash_dr1.source limit 1000',fmt='structarray')
Returning Numpy structured / record array

dl.queryhistory()
1  2017-05-16 13:27:34  sql  SYNC  pandas  100  --  'select ra,dec,gmag from smash_dr1.object limit 100'
2  2017-05-16 13:27:40  sql  SYNC  csv  500  --  'select ra,dec,gmag from smash_dr1.object limit 500'
3  2017-05-16 13:27:46  sql  SYNC  structarray  1000  --  'select ra,dec,gmag from smash_dr1.object limit 1000'
queryprofiles(profile=None)[source]

List the available Query Manager profiles to use with a dl.query().

Parameters:profile (str) – The name of a specific Query Manager profile to check. If this is blank then all of the available profile names will be listed.
Returns:results – The list of properties of profile profile, or a list of all available profiles.
Return type:str

Example

List of available profiles.

dl.queryprofiles()
default,IRSA,HEASARC,Vizier,GAVO,SIMBAD,zeus1,SDSS-DR9,STScI-RegTAP,GALEX-DR6,dldb1

Get profile information on profile dldb1.

dl.queryprofiles(‘dldb1’) {u’accessURL’: u’http://dldb1.sdm.noao.edu:8080/ivoa-dal/tap‘, u’dbport’: 5432, u’password’: u’datalab’, u’description’: u’Development NOAO Data Lab TAP Service / Database on dldb1’, u’database’: u’tapdb’, u’host’: u’dldb1.sdm.noao.edu’, u’vosRoot’: u’vos://datalab.noao!vospace’, u’vosEndpoint’: u’http://dldb1.sdm.noao.edu:8080/vospace-2.0/vospace’, u’user’: u’dlquery’, u’vosRootDir’: u’/data/vospace/users’, u’type’: u’datalab’}
queryresults(jobid=None)[source]

Get the async query results.

Parameters:jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.
Returns:result – The result of the query in the requested format (see fmt in dl.query().
Return type:str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 3',async=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED

results = dl.queryresults(jobid)
print results
ra,dec
103.068354922718,-37.973538878907299
103.071774116284,-37.973599429479599
103.071597827998,-37.972329108796401
querystatus(jobid=None)[source]

Get the async query job status.

Parameters:jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.
Returns:status – The status of the query, which can be one of the following: QUEUED the query job is in queue and waiting to be executed. EXECUTING the query job is currently running. COMPLETED the query is done and the results are ready to be retrieved with dl.queryresults(). ERROR there was a problem with the query.
Return type:str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 100',async=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED
rm(name=None, verbose=True)[source]

Delete files in Data Lab VOSpace.

Parameters:name (str) – The name of the file in VOSpace to delete, e.g. file1.txt.

Example

Delete the file file1.txt.

dl.ls()
file1.txt, file2.txt

dl.rm('file1.txt')

dl.ls()
file2.txt
rmdir(name=None)[source]

Delete a directory in Data Lab VOSpace.

Parameters:name (str) – The name of the directory in VOSpace to delete, e.g. results.

Example

Delete the directory data1/.

dl.rmdir('data1')
schema(val='', fmt='text', profile='default')[source]

Print information about data service schema.

Parameters:
  • val (str) – Value to list ([[<schema>][.<table>][.<col>]]).
  • fmt (str) – Output format (csv|text|json).
  • profile (str) – Service profile.
Returns:

results – The schema information is printed to the screen.

Return type:

str

Example

Print out all the DL tables.

 datalab schema

 Schema Name   Description
-----------   -----------
   gaia_dr1   GAIA Data Release 1
       ivoa   IVOA ObsCore tables
   des_sva1   DES SVA1 Data Products
 tap_schema   TAP Schema Tables
       usno   USNO Astrometry Catalogs
  sdss_dr13
    neo_dr1   NEO Survey Data Release 1
     ls_dr3   The DECam Legacy Survey Data Release 3
  smash_dr1   SMASH Data Release 1

List all tables in a schema/catalog.

datalab schema val=smash_dr1

Schema: smash_dr1

Table Name   Description
----------   -----------
      chip   Info on each chip in the frame
  exposure   Info on each exposure
     field   Info on each target field (position, Num exposures, etc)
    object   Average photometry of each unique object
    source   All of the individual source measurements
     stars   View of object table to select for stars
  galaxies   View of object table to select for galaxies
    xmatch   Crossmatch of object against GAIA DR1 and WISE
servicestatus()[source]

This checks on the status of the DL services.

siaquery(ra=None, dec=None, dist=None, file=None, out=None, verbose=False)[source]

Perform a SIA query with a set of coordinates or from an uploaded file.

Parameters:
  • ra (float) – The right ascension (in degrees) of the point to use for the search.
  • dec (float) – The declination (in degrees) of the point to use for the search.
  • dist (float) – The search distance (radius) in degrees. The default is 0.0085 deg.
  • file (str) – The name of a file with coordinates and search radii to use for the search.
  • out (str) – The output method. The options are to VOSpace or mydb. The default is to return the results to the command line.
  • verbose (bool) – Use verbose output. The default is False.

Example

Perform a simple SIA search.

tab = dl.siaquery(0.0,0.0)
status()[source]

Print the status of the Data Lab connection.

Example

The “myusername” is logged in.

dl.status()
User myusername is logged into the Data Lab

No user is currently logged in.

dl.status()
No user is currently logged into the Data Lab
tag(name=None, tag=None)[source]

Tag or annotate a file or directory in Data Lab VOSpace.

Parameters:
  • name (str) – The name of the file or directory in VOSpace to tag, e.g. file1.txt.
  • tag (str) – The name of the tag, e.g. research.

Example

Tag the directory data1/ with the tag results.

dl.tag('data1','results')
whoami()[source]

Print the current active user.

Example

dl.whoami()
myusername
dlinterface.addFormatMapping(self)[source]

Add the format mapping information to the DL object

dlinterface.areLoginsWorking()[source]

This checks if the Authentication Manager is returning proper tokens.

dlinterface.areSyncQueriesWorking()[source]

This checks if the Query Manager is returning proper Sync queries.

dlinterface.checkLogin(self)[source]

Check if the user is already logged in. If not, give a warning message

dlinterface.convert_vospace_time_to_seconds(str_date)[source]

A convenience method that takes a string from a vospace time field and converts it to seconds since epoch.

Parameters:str_date (str) – string to parse into a VOSpace time
Returns:A datetime object for the provided string date
Return type:datetime
dlinterface.getNodeInfo(self, xnode, lenpathbase, verbose=True)[source]

Get information on a node. The input is a “node” element of a XML ElementTree.

dlinterface.getUserName(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous username.

dlinterface.getUserToken(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dlinterface.isListWorking()[source]

This checks if the Storage Manager is returning proper list queries:

dlinterface.isTapWorking()[source]

This checks if the TAP service and Tomcat are running.

dlinterface.reformatQueryOutput(self, res=None, fmt='csv', verbose=True)[source]

Reformat the output of a query based on a format.