dl package

authClient module

dl.authClient.acToString(s)[source]

acToString – Force a return value to be type ‘string’ for all Python versions.

class dl.authClient.authClient[source]

Bases: object

AUTHCLIENT – Client-side methods to access the Data Lab

Authentication Service.

debug(debug_val)[source]

Toggle debug flag.

getConfig(section, param)[source]

Get a value from the configuration file.

getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object.

getHeaders(token)[source]

Get default tracking headers.

get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import authClient
profile = authClient.client.get_profile()
get_svc_url()[source]

Return the currently-used Authentication Service URL.

Parameters

None

Returns

service_url – The currently-used Authentication Service URL.

Return type

str

Example

from dl import authClient
service_url = authClient.get_svc_url()
hasAccess(token, resource)[source]

See whether the given token has access to the named Resource.

Parameters
  • token (str) – User login token

  • resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')
isAlive(svc_url='https://datalab.noao.edu/auth')[source]
Check whether the AuthManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

service_url (str) – The Query Service URL to ping.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")
isTokenLoggedIn(token)[source]

See whether the user identified by the token is currently logged in.

Parameters

token (str) – User login token

Returns

status – True if token is marked as logged-in, False otherwise

Return type

bool

Example

isUserLoggedIn(user)[source]

See whether the user identified by the token is currently logged in.

Parameters

user (str) – User login name

Returns

status – True if user is currently logged-in, False otherwise

Return type

bool

Example

isValidPassword(user, password)[source]

See whether the password is valid for the user.

Parameters
  • user (str) – User login name

  • password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

isValidToken(token)[source]

See whether the current token is valid.

Parameters

token (str) – User login token

Returns

status – True if token is valid, False otherwise

Return type

bool

Example

isValidUser(user)[source]

See whether the specified user is valid.

Parameters

user (str) – User login name

Returns

status – True if ‘user’ is a valid user name, False otherwise

Return type

bool

Example

list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Parameters

token (str) – Valid auth service token.

Returns

profiles

Return type

JSON string

Example

from dl import authClient
profiles = authClient.client.list_profiles(token, profile, format)
loadConfig()[source]

Read the $HOME/.datalab/dl.conf file.

login(username, password, debug=False, verbose=False)[source]
Authenticate the user with the Authentication Service.

We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters
  • username (str) – User login name.

  • password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.

  • debug (bool) – Method debug flag.

  • verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token
logout(token=None)[source]

Log the user out of the Data Lab.

Parameters

token (str) – User login token

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
passwordReset(token, username, password)[source]
Reset a user password reset. We require that the user provide

either a valid ‘root’ token or the token for the account being reset.

Parameters
  • token (str) – User login token

  • username (str) – User login name

  • password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
retBoolValue(url)[source]

Utility method to call a boolean service at the given URL.

setConfig(section, param, value)[source]

Set a value and save the configuration file.

set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")
set_svc_url(svc_url)[source]

Set the URL of the Authentication Service to be used.

Parameters

svc_url (str) – Authentication service base URL to call.

Returns

Return type

Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")
whoAmI()[source]

Return the currently logged-in user identity.

Parameters

None

Returns

name – Currently logged-in user name, or ‘anonymous’ if not logged-in

Return type

str

Example

from dl import authClient
name = authClient.whoAmI()
writeConfig()[source]

Write out the configuration file to disk.

exception dl.authClient.dlAuthError(message)[source]

Bases: Exception

A throwable error class.

dl.authClient.getClient()[source]

Get a new instance of the authClient client.

Parameters

None

Returns

client – An authClient object

Return type

authClient

Example

dl.authClient.get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import authClient
profile = authClient.client.get_profile()
dl.authClient.get_svc_url()[source]

Return the currently-used Authentication Service URL.

Parameters

None

Returns

service_url – The currently-used Authentication Service URL.

Return type

str

Example

from dl import authClient
service_url = authClient.get_svc_url()
dl.authClient.hasAccess(user, resource)[source]

See whether the given token has access to the named Resource.

Parameters
  • token (str) – User login token

  • resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')
dl.authClient.isAlive(svc_url='https://datalab.noao.edu/auth')[source]
Check whether the AuthManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

service_url (str) – The Query Service URL to ping.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")
dl.authClient.isTokenLoggedIn(token)[source]

See whether the user identified by the token is currently logged in.

Parameters

token (str) – User login token

Returns

status – True if token is marked as logged-in, False otherwise

Return type

bool

Example

dl.authClient.isUserLoggedIn(user)[source]

See whether the user identified by the token is currently logged in.

Parameters

user (str) – User login name

Returns

status – True if user is currently logged-in, False otherwise

Return type

bool

Example

dl.authClient.isValidPassword(user, password)[source]

See whether the password is valid for the user.

Parameters
  • user (str) – User login name

  • password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

dl.authClient.isValidToken(token)[source]

See whether the current token is valid.

Parameters

token (str) – User login token

Returns

status – True if token is valid, False otherwise

Return type

bool

Example

dl.authClient.isValidUser(user)[source]

See whether the specified user is valid.

Parameters

user (str) – User login name

Returns

status – True if ‘user’ is a valid user name, False otherwise

Return type

bool

Example

dl.authClient.list_profiles(token, profile=None, format='text')[source]
dl.authClient.login(user, password=None, debug=False, verbose=False)[source]
Authenticate the user with the Authentication Service.

We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters
  • username (str) – User login name.

  • password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.

  • debug (bool) – Method debug flag.

  • verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token
dl.authClient.logout(token=None)[source]

Log the user out of the Data Lab.

Parameters

token (str) – User login token

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
dl.authClient.passwordReset(token, username, password)[source]
Reset a user password reset. We require that the user provide

either a valid ‘root’ token or the token for the account being reset.

Parameters
  • token (str) – User login token

  • username (str) – User login name

  • password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)
dl.authClient.set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")
dl.authClient.set_svc_url(svc_url)[source]

Set the URL of the Authentication Service to be used.

Parameters

svc_url (str) – Authentication service base URL to call.

Returns

Return type

Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")
dl.authClient.whoAmI()[source]

Return the currently logged-in user identity.

Parameters

None

Returns

name – Currently logged-in user name, or ‘anonymous’ if not logged-in

Return type

str

Example

from dl import authClient
name = authClient.whoAmI()

queryClient module

dl.queryClient.abort[source]

Abort the specified asynchronous job.

Usage:

abort (token=None, jobId=None)

queryClient.abort (token, jobId) queryClient.abort (jobId) queryClient.abort (token, jobId=<id>) queryClient.abort (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId to abort.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.drop[source]

Drop the specified table from the user’s MyDB

Usage:

drop (table=None, token=None)

queryClient.drop (token, table) queryClient.drop (table) queryClient.drop (token, table=<id>) queryClient.drop (table=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# List the tables
queryClient.drop('foo1')
dl.queryClient.error[source]

Retrieve the error of an asynchronous query, once completed.

Usage:

error (token=None, jobId=None)

queryClient.error (jobId) queryClient.error (token, jobId) queryClient.error (token, jobId=<id>) queryClient.error (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

error

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve error
time.sleep(4)
if queryClient.status(token, jobId) == 'ERROR':
    error = queryClient.error(token,jobId)
    print type(error)
    print error

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.getClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]

Create a new queryClient object and set a default profile.

dl.queryClient.get_profile()[source]

Get the current query profile.

Parameters

None

Returns

profile – The name of the current profile used with the Query Manager service

Return type

str

Example

print ("Query Service profile = " + queryClient.get_profile())
dl.queryClient.get_svc_url()[source]

Get the currently-used Query Manager serice URL.

Parameters

None

Returns

service_url – The currently-used Query Service URL.

Return type

str

Example

print (queryClient.get_svc_url())
dl.queryClient.get_timeout_request()[source]

Get the current Sync query timeout value.

Parameters

None

Returns

result – Current sync query timeout value.

Return type

int

Example

# get the current timeout value
print (queryClient.get_timeout_request())
dl.queryClient.isAlive(svc_url='https://datalab.noao.edu/query', timeout=5)[source]
Check whether the QueryManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters
  • svc_url (str) – The Query Service URL to ping.

  • timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")
dl.queryClient.list[source]

List the tables or table schema in the user’s MyDB.

Usage:

list (table=None, token=None)

queryClient.list (token, table) queryClient.list (table) queryClient.list (token, table=<id>) queryClient.list ()

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.list()
dl.queryClient.list_profiles[source]

Retrieve the profiles supported by the query manager service.

Usage:

list_profiles (token=None, profile=None, format=’text’)

queryClient.list_profiles (token) queryClient.list_profiles ()

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • profile (str) – A specific profile configuration to list. If None, a list of profiles available to the given auth token is returned.

  • format (str) – Result format: One of ‘text’ or ‘json’

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

profiles = queryClient.list_profiles()
profiles = queryClient.list_profiles(token)
dl.queryClient.mydb_copy[source]

Copy a table in the user’s MyDB to a new name

Usage:

mydb_copy (source, target, token=None)

queryClient.mydb_copy (token, source, target) queryClient.mydb_copy (source, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • source (str) – The old table name, i.e. the table to be copied

  • target (str) – The new table name, i.e. the table to be created

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_copy ('foo', 'bar')
dl.queryClient.mydb_create[source]

Create a table in the user’s MyDB

Usage:

mydb_create (table, schema, token=None, **kw)

queryClient.mydb_create (token, table, <schema_dict>) queryClient.mydb_create (table, <schema_dict>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to create

  • schema (str or dict) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV. If passed as a ‘dict’ type, it is a dictionary object where keys are the column names and values are the data types.

  • drop (bool (optional)) – Drop any existing table of the same name before creating new one.

Example

# Create table in MyDB named 'foo' with columns defined by
# the file 'schema.txt'.  The schema file contains:
#
#     id,text
#     ra,double precision
#     dec,double precision
#
queryClient.mydb_create ('foo', 'schema.txt')
dl.queryClient.mydb_drop[source]

Drop the specified table from the user’s MyDB

Usage:

mydb_drop (table, token=None)

queryClient.mydb_drop (token, table) queryClient.mydb_drop (table) queryClient.mydb_drop (token, table=<id>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# Drop the 'foo1' table
queryClient.drop('foo1')
dl.queryClient.mydb_import[source]

Import data into a table in the user’s MyDB

Usage:

mydb_import (table, data, token=None, **kw)

queryClient.mydb_import (token, table, data) queryClient.mydb_import (table, data)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to be loaded

  • data (str or data object) –

    The data file or python object to be loaded. The ‘data’ value may be one of the following types:

    filename A CSV file of data string A string containing CSV data Pandas DataFrame A Pandas DataFrame object

    : : : :

    Additional object types can be added provided the data can be converted to a CSV format.

  • schema (str [OPTIONAL]) – If set, this is a filename or string containing a schema for the data table to be created. A schema contains a comma-delimited row for each column containing the column name and it’s Postgres data type. If not set, the schema is determined automatically from the data.

  • drop (bool [Optional]) – Drop any existing table of the same name before creating new one.

  • verbose (bool [Optional]) – Be verbose about operations.

Returns

  • schema A string containing the table schema

  • data_obj The CSV data to be imported (possibly converted)

Example

# Import data into a MyDB table named 'foo' from file 'data.csv'.
schema, data = queryClient.mydb_import ('foo', 'data.csv')
dl.queryClient.mydb_index[source]

Index the specified column in a table in the user’s MyDB

queryClient.mydb_index (table, colunm) queryClient.mydb_index (token, table, column) queryClient.mydb_index (table, column, token=None)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The table to be indexed

  • column (str) – The column

  • q3c (str) – A comma-delimited list of two column names giving the RA and Dec positions (decimal degrees) to be used to Q3C index the table. If None, no Q3C index will be computed.

  • cluster (bool) – If enabled, data table will be rewritten to cluster on the Q3C index for efficiency. Only used when ‘q3c’ columns are specified.

  • async (bool) – If enabled, index commands will be submitted asynchronously.

Returns

Return type

command status

Example

# Index the table's "id" column
queryClient.index('foo1', 'id')

# Index and cluster the table by position
queryClient.index('foo1', q3c='ra,dec', cluster=True)
dl.queryClient.mydb_insert[source]

Insert data into a table in the user’s MyDB

Usage:

mydb_insert (table, data, token=None, **kw)

queryClient.mydb_insert (token, table, <filename>) queryClient.mydb_insert (token, table, <data_object>) queryClient.mydb_insert (table, <filename>) queryClient.mydb_insert (table, <data_object>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The name of the table to append

  • data (str or data object) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV data. If passed as a tabular data object, it is converted to CSV and sent to the service.

  • csv_header (bool [OPTIONAL]) – If True, then the CSV data object contains a CSV header line, i.e. the first line is a row of column names. Otherwise, no column names are assumed and the column order must match the table schema.

Example

# Insert data into a MyDB table named 'foo'.
queryClient.mydb_insert ('foo', 'data.csv')
dl.queryClient.mydb_list[source]

List the tables or table schema in the user’s MyDB.

Usage:

mydb_list (table=None, token=None)

queryClient.mydb_list (table) queryClient.mydb_list (token, table=<str>) queryClient.mydb_list (table=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.mydb_list()
dl.queryClient.mydb_rename[source]

Rename a table in the user’s MyDB to a new name

Usage:

mydb_rename (source, target, token=None)

queryClient.mydb_rename (token, source, target) queryClient.mydb_rename (source, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • source (str) – The old table name

  • target (str) – The new table name

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_rename ('foo', 'bar')
dl.queryClient.mydb_truncate[source]

Truncate the specified table in the user’s MyDB

Usage:

mydb_truncate (table, token=None)

queryClient.mydb_truncate (token, table) queryClient.mydb_truncate (table) queryClient.mydb_truncate (token, table=<id>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • table (str) – The specific table to drop

Example

# Truncate the table 'foo'
queryClient.truncate('foo')
dl.queryClient.qcToString(s)[source]

qcToString – Force a return value to be type ‘string’ for all Python versions.

dl.queryClient.query[source]

Send an SQL or ADQL query to the database or TAP service.

Usage:
query (token=None, adql=None, sql=None, fmt=’csv’, out=None,

async_=False, profile=’default’, **kw):

queryClient.query (token, query, <args>) queryClient.query (token | query, <args>)

Parameters
  • token (str) – Secure token obtained via authClient.login()

  • adql (str or None) –

    ADQL query string that will be passed to the DB query manager, e.g.

    adql='select top 3 ra,dec from gaia_dr1.gaia_source'
    

  • sql (str or None) –

    SQL query string that will be passed to the DB query manager, e.g.

    sql='select ra,dec from gaia_dr1.gaia_source limit 3'
    

  • fmt (str) –

    Format of result to be returned by the query. Permitted values are:

    • ’csv’ The returned result is a comma-separated string

      that looks like a csv file (newlines at the end of every row) [DEFAULT]

    • ’ascii’ Same, but the column separator is a tab

    • ’array’ Returns a NumPy array

    • ’pandas’ Returns a Pandas DataFrame

    • ’structarray’ Numpy structured array (aka ‘record array’)

    • ’table’ Returns an Astropy Table object

      The following formats may be used when saving a file to virtual storage on the server:

    • ’fits’ FITS binary

    • ’hdf5’ HDF5 file (NOT YET IMPLEMENTED)

    • ’votable’ An XML-formatted VOTable

  • out (str or None) – The output filename to create on the local machine, the URI of a VOSpace or MyDB resource to create, or None if the result is to be returned directly to the caller.

  • async (bool) –

    If True, the query is Asynchronous, i.e. a job is submitted to the DB, and a jobID token is returned the caller. The jobID must be then used to check the query’s status and to retrieve the result (when the job status is COMPLETED) or the error message (when the job status is ERROR). Default is False, i.e. the task runs a Synchroneous query.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

  • profile (str or None) – The Query Manager profile to use for this call. If None then the default profile is used. Available profiles may be listed using the queryClient.list_profiles()

  • **kw (dict) –

    Optional keyword arguments. Supported keywords currently include:

    wait = False

    Wait for asynchronous queries to complete? If enabled, the query() method will submit the job in async mode and then poll for results internally before returning. the default is to return the job ID immediately and let the client poll for job status and return results.

    timeout = 300

    Requested timeout (in seconds) for a query. For a Sync query, this value sets a session timeout request in the database that will abort the query at the specified time. A maximum value of 600 seconds is permitted. If the wait option is enabled for an ASync query, this is the maximum time the query will be allowed to run before an abort() is issued on the job. The maximum timeout for an ASync job is 24-hrs (86400 sec).

    poll = 1

    ASync job polling time in seconds.

    verbose = False

    Print verbose messages during ASync job.

Returns

result – If async=False, the return value is the result of the query as a formatted string (see fmt). Otherwise the result string is a job token, with which later the Asynchroneaous query’s status can be checked (queryClient.status()), and the result retrieved (see queryClient.result().

Return type

str

Example

Get security token first, see authClient.login(). Then:

query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
response = queryClient.query(token, adql=query, fmt='csv')
print response

This prints

ra,dec
315.002571989537842,35.2662974820284489
315.00408275885701,35.2665448169895797
314.996334457679438,35.2673478725552698
class dl.queryClient.queryClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]

Bases: object

QUERYCLIENT – Client-side methods to access the Data Lab

Query Manager Service.

abort = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.abort>>, None)[source]
chunked_upload(token, local_file, remote_file)[source]

A streaming file uploader.

conequery(token, input=None, out=None, schema=None, table=None, ra=None, dec=None, search=0.5)[source]

Send a cone search query to the consearch service

dataType(val, current_type)[source]

Lexically scan a value to determine the datatype.

drop = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.drop>>, None)[source]
error = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.error>>, None)[source]
getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object

getHeaders(token)[source]

Get default tracking headers,

getSchema(data, **kw)[source]

Generate a schema for mydb_create() from a CSV file or data object.

get_profile()[source]

Get the current query profile.

Parameters

None

Returns

profile – The name of the current profile used with the Query Manager service

Return type

str

Example

print ("Query Service profile = " + queryClient.get_profile())
get_svc_url()[source]

Get the currently-used Query Manager serice URL.

Parameters

None

Returns

service_url – The currently-used Query Service URL.

Return type

str

Example

print (queryClient.get_svc_url())
get_timeout_request()[source]

Get the current Sync query timeout value.

Parameters

None

Returns

result – Current sync query timeout value.

Return type

int

Example

# get the current timeout value
print (queryClient.get_timeout_request())
isAlive(svc_url=None, timeout=5)[source]
Check whether the QueryManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters
  • svc_url (str) – The Query Service URL to ping.

  • timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")
list = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.list>>, None)[source]
list_profiles = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.list_profiles>>, None)[source]
mydb_copy = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_copy>>, None)[source]
mydb_create = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_create>>, None)[source]
mydb_drop = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_drop>>, None)[source]
mydb_import = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_import>>, None)[source]
mydb_index = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_index>>, None)[source]
mydb_insert = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_insert>>, None)[source]
mydb_list = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_list>>, None)[source]
mydb_rename = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_rename>>, None)[source]
mydb_truncate = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.mydb_truncate>>, None)[source]
static pretty_print_POST(req)[source]

At this point it is completely built and ready to be fired; it is “prepared”.

However pay attention at the formatting used in this function because it is programmed to be pretty printed and may differ from the actual request.

query = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.query>>, None)[source]
results = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.results>>, None)[source]
schema = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.schema>>, None)[source]
services(name=None, svc_type=None, mode='list', profile='default')[source]

Usage: queryClient.services ()

set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())

Returns

Return type

Nothing

Example

queryClient.set_profile('test')
set_svc_url(svc_url)[source]

Set the Query Manager service URL.

Parameters

svc_url (str) – The service URL of the Query Manager to call.

Returns

Return type

Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")
set_timeout_request(nsec)[source]

Set the requested Sync query timeout value (in seconds).

Parameters

nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.

Returns

Return type

Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)
siaquery(token, input=None, out=None, search=0.5)[source]

Send a SIA (Simple Image Access) query to the query manager service

status = functools.partial(<bound method MultiMethod.__call__ of <function queryClient.status>>, None)[source]
exception dl.queryClient.queryClientError(message)[source]

Bases: Exception

dl.queryClient.results[source]

Retrieve the results of an asynchronous query, once completed.

Usage:

results (token=None, jobId=None, delete=True)

queryClient.results (jobId) queryClient.results (token, jobId) queryClient.results (token, jobId=<id>) queryClient.results (jobId=<str>)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604
dl.queryClient.schema[source]

Return information about a data service schema.

Usage:

schema (value=’‘, format=’text’, profile=None)

Parameters
  • value (str) – Schema object to return: Of the form <schema>[.<table>[.<column]]

  • profile (str) – The name of the service profile to use. The list of available profiles can be retrieved from the service (see function queryClient.list_profiles())

  • format (str) – Result format: One of ‘text’ or ‘json’ (NOT CURRENTLY USED)

Example

# List the available schema
queryClient.schema("", "text", "default")

# List the tables in the USNO schema
queryClient.schema("usno", "text", "default")

# List the columns of the USNO-A2 table
queryClient.schema("usno.a2", "text", "default")

# List the attributes of the USNO-A2 'raj2000' column
queryClient.schema("usno.a2.raj2000", "text", "default")
dl.queryClient.services(name=None, svc_type=None, mode='list', profile='default')[source]

Usage: queryClient.services ()

dl.queryClient.set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())

Returns

Return type

Nothing

Example

queryClient.set_profile('test')
dl.queryClient.set_svc_url(svc_url)[source]

Set the Query Manager service URL.

Parameters

svc_url (str) – The service URL of the Query Manager to call.

Returns

Return type

Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")
dl.queryClient.set_timeout_request(nsec)[source]

Set the requested Sync query timeout value (in seconds).

Parameters

nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.

Returns

Return type

Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)
dl.queryClient.status[source]

Get the status of an asynchronous query.

Usage:

status (token=None, jobId=None)

queryClient.status (jobId) queryClient.status (token, jobId) queryClient.status (token, jobId=<id>) queryClient.status (jobId=<str>)

Use the authentication token and the jobId of a previously issued asynchronous query to check the query’s current status.

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

status – Either ‘QUEUED’ or ‘EXECUTING’ or ‘COMPLETED’. If the token & jobId combination does not correspond to an actual job, then a HTML-formatted error message is returned. If there is a problem with the backend, the returned value can be ‘ERROR’.

When status is ‘COMPLETED’, you can retrieve the results of the query via queryClient.results()

Return type

str

Example

import time
query = 'select ra,dec from gaia_dr1.gaia_source limit 200000'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)
while True:
    status = queryClient.status(token, jobId)
    print "time index =", time.localtime()[5], "   status =", status
    if status == 'COMPLETED':
        break
    time.sleep(1)

This prints

time index = 16    status = EXECUTING
time index = 17    status = EXECUTING
time index = 18    status = COMPLETED

storeClient module

dl.storeClient.access[source]

Determine whether the file can be accessed with the given node.

Usage:

access(path, mode=None, token=None, verbose=True)

storeClient.access(token, path, mode) storeClient.access(path, mode) storeClient.access(path)

Parameters
  • path (str) – A name or file template of the file status to retrieve.

  • mode (str) – Requested access mode. Modes are ‘r’ (read access), ‘w’ (write access), or ‘rw’ to test for both read/write access. If mode is None a simple existence check is made.

  • token (str) – Authentication token (see function authClient.login())

  • verbose (bool) – Verbose output flag.

Returns

result – True if the node can be access with the requested mode.

Return type

bool

Example

if storeClient.access('/mydata.csv')
    print('File exists')
elif storeClient.access('/mydata.csv','rw')
    print('File is both readable and writable')
dl.storeClient.chunked_upload(token, local_file, remote_file)[source]

A streaming file uploader.

dl.storeClient.cp[source]

Copy a file/directory within the store manager service

Usage:

cp(token=None, fr=’‘, to=’‘, verbose=False)

storeClient.cp(token, fr, to) storeClient.cp(fr, to) storeClient.cp(fr) storeClient.cp(fr=’‘,to=’‘)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be copied (may not be a directory).

  • to (str) – Name of the file to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Copy a file in vospace
storeClient.cp('foo', 'bar')
storeClient.cp('vos://foo', 'vos:///new/bar')
dl.storeClient.expandFileList(svc_url, token, pattern, format, full=False)[source]

Expand a filename pattern in a VOSpace URI to a list of files. We do this by getting a listing of the parent container contents from the service and then match the pattern on the client side.

dl.storeClient.get[source]

Retrieve a file from the store manager service

Usage:

get(token=None, fr=’‘, to=’‘, mode=’text’, verbose=True, debug=False)

storeClient.get(token, fr, to) storeClient.get(fr, to) storeClient.get(fr) storeClient.get(token, fr, to)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – A name or file template of the file(s) to retrieve.

  • to (str) – Name of the file(s) to locally. If not specified, the contents of the file are returned to the caller.

  • mode ([binary | text | fileobj]) – Return data type if note saving to file. If set to ‘text’ the file contents are converted to string – this is appropriate when dealing with unicode but may fail with general binary data. If set to ‘binary’ the raw content of the HTTP response is returned – for Python 2 this will be a ‘string’, for Python 3 it will be a ‘bytes’ data type (the caller is responsible for conversion).

Returns

result – A list of the names of the files retrieved, or the contents of a single file.

Return type

str

Example

# get a single file to a local file of a different name
data = storeClient.get('vos://mydata.csv', 'data.csv')

# get the contents of a single file to a local variable
data = storeClient.get('vos://mydata.csv')

# get a list of remote files to a local directory
flist = storeClient.get('vos://*.fits', './data/')
flist = storeClient.get('*.fits', './data/')
dl.storeClient.getClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]

Create a new storeClient object and set a default profile.

dl.storeClient.get_profile()[source]

Get the profile

Parameters

None

Returns

profile – The name of the current profile used with the Storage Manager

Return type

str

Example

print('Store Service profile = ' + storeClient.get_profile())
dl.storeClient.get_svc_url()[source]

Return the currently-used Storage Manager service URL.

Parameters

None

Returns

Return type

Current Storage Manager service URL

Example

print(storeClient.get_scv_url())
dl.storeClient.hasmeta(s)[source]

Determine whether a string contains filename meta-characters.

dl.storeClient.isAlive(svc_url='https://datalab.noao.edu/storage')[source]
Check whether the StorageManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff
dl.storeClient.is_vosDir(svc_url, token, path)[source]

Determine whether ‘path’ is a ContainerNode in the VOSpace.

dl.storeClient.list_profiles[source]

Retrieve the profiles supported by the storage manager service

Usage:

list_profiles(token=None, profile=None, format=’text’)

storeClient.list_profiles(token) # list default profile storeClient.list_profiles(profile) # list named profile storeClient.list_profiles() # list default profile

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • profile (str) – A specific profile to list

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

# get the list of profiles
profiles = storeClient.list_profiles(token)
dl.storeClient.ln[source]

Create a link to a file/directory in the store manager service

Usage:

ln(token, fr=’‘, target=’‘, verbose=False)

storeClient.ln(token, fr, target) storeClient.ln(fr, target) storeClient.ln(fr, target)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be linked (may not be a directory).

  • to (str) – Name of the link to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Link a file in vospace
storeClient.ln('foo', 'bar')
storeClient.ln('vos://foo', 'vos:///new/bar')
dl.storeClient.load[source]

Load a file from a remote endpoint to the Store Manager service

Usage:

load(name, endpoint, token=None, is_vospace=False)

storeClient.load(token, name, endpoint) storeClient.load(name, endpoint)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name or template of local files to upload.

  • endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')
dl.storeClient.ls[source]

Get a file/directory listing from the store manager service

Usage:

ls(name=’vos://’, token=None, format=’csv’, verbose=False)

storeClient.ls(token, name) storeClient.ls(name) storeClient.ls()

Parameters
  • token (str) – Secure token obtained via authClient.login()

  • name (str) – Valid name of file or directory, e.g. vos://somedir .. todo:: [20161110] currently doesn’t seem to work.

  • format (str) – Default str.

Example

listing = storeClient.ls(token, name='vos://somedir')
listing = storeClient.ls(token, 'vos://somedir')
listing = storeClient.ls('vos://somedir')
print(listing)

This prints for instance:

bar2.fits,foo1.csv,fancyfile.dat
dl.storeClient.mkdir[source]

Make a directory in the storage manager service

Usage:

mkdir(optval, name=’‘, token=None)

storeClient.mkdir(token, name) storeClient.mkdir(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the container (directory) to create.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Create a directory in vospace
storeClient.mkdir('foo')
dl.storeClient.mv[source]

Move/rename a file/directory within the store manager service

Usage:

mv(token=None, fr=’‘, to=’‘, verbose=False)

storeClient.mv(token, fr, to) storeClient.mv(fr, to) storeClient.mv(fr) storeClient.mv(fr=’‘,to=’‘)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name of the file to be moved

  • to (str) – Name of the file or directory to move the ‘fr’ file. If given as a directory the original filename is preserved.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Move a file in vospace
storeClient.mv('foo', 'bar')             # rename file
storeClient.mv('foo', 'vos://newdir/')   # move to new directory
storeClient.mv('foo', 'newdir')
dl.storeClient.pull[source]

Load a file from a remote endpoint to the Store Manager service

Usage:

pull(name, endpoint, token=None, is_vospace=False)

storeClient.pull(token, name, endpoint) storeClient.pull(name, endpoint)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name or template of local files to upload.

  • endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')
dl.storeClient.put[source]

Upload a file to the store manager service

Usage:

put(fr=’‘, to=’vos://’, token=None, verbose=True, debug=False)

storeClient.put(token, fr, to) storeClient.put(fr, to) storeClient.put(fr) storeClient.put(fr=’‘,to=’‘)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • fr (str) – Name or template of local files to upload.

  • to (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# get the contents of a single file
dl.storeClient.rm[source]

Delete a file from the store manager service

Usage:

rm(name=’‘, token=None, verbose=False)

storeClient.rm(token, name) storeClient.rm(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the file to delete

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove a file from vospace
storeClient.rm('foo.csv')
storeClient.rm('vos://foo.csv')
dl.storeClient.rmdir[source]

Delete a directory from the store manager service

Usage:

rmdir(name=’‘, token=None, verbose=False)

storeClient.rmdir(token, name) storeClient.rmdir(name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the container (directory) to delete.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove an empty directory from VOSpace.
storeClient.rmdir('datadir')
storeClient.rmdir('vos://datadir')
dl.storeClient.saveAs[source]

Save the string representation of a data object as a file.

Usage:

saveAs(data, name, token=None)

storeClient.saveAs(token, data, name) storeClient.saveAs(data, name)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • data (str) – Data object to be saved.

  • name (str) – Name of the file to create containing the string representation of the data.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Save a data object to VOSpace
storeClient.saveAs(pandas_data, 'pandas.example')
storeClient.saveAs(json_data, 'json.example')
storeClient.saveAs(table_data, 'table.example')
dl.storeClient.scToString(s)[source]

scToString – Force a return value to be type ‘string’ for all Python versions. If there is an error, return the original.

dl.storeClient.services(name=None, svc_type='vos', format=None, profile='default')[source]
dl.storeClient.set_profile(profile='default')[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())

Returns

Return type

Nothing

Example

storeClient.set_profile('test')
dl.storeClient.set_svc_url(svc_url='https://datalab.noao.edu/storage')[source]

Set the Storage Manager service URL.

Parameters

svc_url (str) – The service URL of the Storage Manager to use

Returns

Return type

Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")
dl.storeClient.stat[source]

Get file status information, similar to stat().

Usage:

stat(path, token=None, verbose=True)

storeClient.stat(token, path) storeClient.stat(path)

Parameters
  • path (str) – A name or file template of the file status to retrieve.

  • token (str) – Authentication token (see function authClient.login())

  • verbose (bool) – Verbose output flag.

Returns

stat

A dictionary of node status values. Returned fields include:

name Name of node groupread List of group/owner names w/ read access groupwrite List of group/owner names w/ write access publicread Publicly readable (0=False, 1=True) owner Owner name perms Formatted unix-like permission string target Node target if LinkNode size Size of file node (bytes) type Node type (container|data|link)

Return type

dictionary

Example

# get status information for a specific node
stat = storeClient.stat('vos://mydata.csv')

if stat['type'] == 'container':
    print('This is a directory')
else:
    print('File size is: ' + stat['size'])
class dl.storeClient.storeClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]

Bases: object

STORECLIENT – Client-side methods to access the Data Lab

Storage Manager Service.

access = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.access>>, None)[source]
cp = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.cp>>, None)[source]
get = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.get>>, None)[source]
getFromURL(svc_url, path, token)[source]

Get something from a URL. Return a ‘response’ object

get_profile()[source]

Get the profile

Parameters

None

Returns

profile – The name of the current profile used with the Storage Manager

Return type

str

Example

print('Store Service profile = ' + storeClient.get_profile())
get_svc_url()[source]

Return the currently-used Storage Manager service URL.

Parameters

None

Returns

Return type

Current Storage Manager service URL

Example

print(storeClient.get_scv_url())
isAlive(svc_url=None, timeout=2)[source]
Check whether the StorageManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff
list_profiles = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.list_profiles>>, None)[source]
ln = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.ln>>, None)[source]
load = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.load>>, None)[source]
ls = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.ls>>, None)[source]
mkdir = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.mkdir>>, None)[source]
mv = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.mv>>, None)[source]
pull = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.pull>>, None)[source]
put = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.put>>, None)[source]
rm = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.rm>>, None)[source]
rmdir = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.rmdir>>, None)[source]
saveAs = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.saveAs>>, None)[source]
services(name=None, svc_type='vos', format=None, profile='default')[source]
set_profile(profile)[source]

Set the service profile to be used.

Parameters

profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())

Returns

Return type

Nothing

Example

storeClient.set_profile('test')
set_svc_url(svc_url)[source]

Set the Storage Manager service URL.

Parameters

svc_url (str) – The service URL of the Storage Manager to use

Returns

Return type

Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")
stat = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.stat>>, None)[source]
tag = functools.partial(<bound method MultiMethod.__call__ of <function storeClient.tag>>, None)[source]
exception dl.storeClient.storeClientError(message)[source]

Bases: Exception

dl.storeClient.tag[source]

Annotate a file/directory in the store manager service

Usage:

tag(token, name=’‘, tag=’‘)

storeClient.tag(token, name, tag) storeClient.tag(name, tag) storeClient.tag(token, name=’foo’, tag=’bar’)

Parameters
  • token (str) – Authentication token (see function authClient.login())

  • name (str) – Name of the file to tag

  • tag (str) – Annotation string for file

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Annotate a file in vospace
storeClient.tag('foo.csv', 'This is a test')

resClient module

Client methods for the Data Lab Resource Management Service.

createUser (username, password, email, name, institute) deleteUser (token, username)

getUser (token, keyword) setUser (token, keyword, value)

passwordReset (token, user, password)

sendPasswordLink (token, user)

listFields ()

createGroup (token, groupName)

getGroup (token, keyword) setGroup (token, keyword, value)

deleteGroup (token, groupName)

createResource (token, resource)

getResource (token, keyword) setResource (token, keyword, value)

deleteResource (token, resource)

set_svc_url (svc_url) get_svc_url () set_profile (profile) get_profile ()

list_profiles (token, profile=None, format=’text’)

Import via

from dl import resClient
dl.resClient.createGroup(token, groupName, profile='default')[source]

Create a new Group in the system.

Parameters

name (str) – Group name to create

Returns

resp – Service response

Return type

str

dl.resClient.createResource(token, resource, profile='default')[source]

Create a new Resource in the system.

Parameters

resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

dl.resClient.createUser(username, password, email, name, institute, profile='default')[source]

Create a new user in the system.

Parameters
  • username (str) – Account username

  • password (str) – Account password

  • email (str) – User’s contact email address

  • name (str) – User’s full name

  • institute (str) – User’s home institution

Returns

Return type

Service response

dl.resClient.deleteGroup(token, groupName, profile='default')[source]

Delete a Group in the system.

Parameters
  • token (str) – User identity token.

  • group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteResource(token, resource, profile='default')[source]

Delete a Resource in the system.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Resource to be deleted.

  • resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteUser(token, username, profile='default')[source]

Delete a user in the system.

Parameters
  • token (str) – User identity token.

  • username (str) – Name of user to be deleted.

Returns

Return type

Service response

exception dl.resClient.dlResError(message)[source]

Bases: Exception

A throwable error class.

dl.resClient.getClient()[source]
dl.resClient.getGroup(token, keyword, profile='default')[source]

Read info about a Group in the system.

Parameters

keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

dl.resClient.getResource(token, keyword, profile='default')[source]

Read info about a Resource in the system.

Parameters

keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

dl.resClient.getUser(token, keyword, profile='default')[source]

Read info about a user in the system.

Parameters

keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
dl.resClient.get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import resClient
profile = resClient.client.get_profile()
dl.resClient.get_svc_url()[source]

Return the currently-used Resource Management Service URL.

Parameters

None

Returns

service_url – The currently-used Resource Management Service URL.

Return type

str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()
dl.resClient.listFields(profile='default')[source]

List available user fields.

Parameters

None

Returns

Return type

Service response

dl.resClient.list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Returns

profiles

Return type

JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)
dl.resClient.passwordReset(token, user, password, profile='default')[source]

Change a user’s password.

Parameters
  • token (str) – User identity token

  • user (str) – User account name. Token must match the user or root token.

  • password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
class dl.resClient.resClient[source]

Bases: object

RESCLIENT – Client-side methods to access the Data Lab

Resource Management Service.

approveUser(token, user, profile='default')[source]

Approve a pending user request.

Parameters
  • token (str) – User identity token

  • user (str) – User account to approve. Token must have authority to manage users.

Returns

Return type

Service response

clientDelete(token, what, query_args)[source]

Generic method to call a /delete service.

clientRead(what, keyword)[source]

Generic method to call a /get service.

clientUpdate(what, keyword, value)[source]

Generic method to call a /set service.

createGroup(token, name, profile='default')[source]

Create a new Group in the system.

Parameters

name (str) – Group name to create

Returns

resp – Service response

Return type

str

createResource(token, resource, profile='default')[source]

Create a new Resource in the system.

Parameters

resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

createUser(username, password, email, name, institute, profile='default')[source]

Create a new user in the system.

Parameters
  • username (str) – Account username

  • password (str) – Account password

  • email (str) – User’s contact email address

  • name (str) – User’s full name

  • institute (str) – User’s home institution

Returns

Return type

Service response

debug(debug_val)[source]

Set the debug flag.

deleteGroup(token, group, profile='default')[source]

Delete a Group in the system.

Parameters
  • token (str) – User identity token.

  • group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

deleteResource(token, resource, profile='default')[source]

Delete a Resource in the system.

Parameters
  • token (str) – User identity token. The token must identify the owner of the Resource to be deleted.

  • resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

deleteUser(token, username, profile='default')[source]

Delete a user in the system.

Parameters
  • token (str) – User identity token.

  • username (str) – Name of user to be deleted.

Returns

Return type

Service response

disapproveUser(token, user, profile='default')[source]

Disapprove a pending user request.

Parameters
  • token (str) – User identity token

  • user (str) – User account to decline. Token must have authority to manage users.

Returns

Return type

Service response

getGroup(token, keyword, profile='default')[source]

Read info about a Group in the system.

Parameters

keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

getResource(token, keyword, profile='default')[source]

Read info about a Resource in the system.

Parameters

keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

getUser(keyword, profile='default')[source]

Read info about a user in the system.

Parameters

keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
get_profile()[source]

Get the requested service profile.

Parameters

None

Returns

profile – The currently requested service profile.

Return type

str

Example

from dl import resClient
profile = resClient.client.get_profile()
get_svc_url()[source]

Return the currently-used Resource Management Service URL.

Parameters

None

Returns

service_url – The currently-used Resource Management Service URL.

Return type

str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()
isAlive(svc_url)[source]
Check whether the ResManager service at the given URL is

alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
listFields(profile='default')[source]

List available user fields.

Parameters

None

Returns

Return type

Service response

listPending(token, verbose=False, profile='default')[source]

List all pending user accounts.

Parameters
  • token (str) – User identity token

  • verbose (bool) – Return verbose listing?

Returns

Return type

List of use accounts pending approval

list_profiles(token, profile=None, format='text')[source]

List the service profiles which can be accessed by the user.

Returns

profiles

Return type

JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)
passwordReset(token, user, password, profile='default')[source]

Change a user’s password.

Parameters
  • token (str) – User identity token

  • user (str) – User account name. Token must match the user or root token.

  • password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
retBoolValue(url)[source]

Utility method to call a boolean service at the given URL.

Send a password-reset link to the user.

Parameters
  • token (str) – User identity token

  • user (str) – User account name.

Returns

Return type

Service response

setField(token, user, field, value, profile='default')[source]

Set a specific user record field

Parameters
  • token (str) – User identity token

  • user (str) – User name to modify. If None then identity is take from token, a root token is required to modify other users.

  • field (str) – Record field to be set

  • value (str) – Field value

Returns

Return type

‘OK’ is field was set, else a service error message.

setGroup(token, keyword, value, profile='default')[source]

Update info about a Group in the system.

Parameters
  • keyword (str) – Group record field to be set

  • value (str) – Value of field to set

setResource(token, keyword, value, profile='default')[source]

Update info about a Resource in the system.

Parameters
  • keyword (str) – Resource record field to be set

  • value (str) – Value of field to set

Returns

status – ‘OK’ is record was set

Return type

str

setUser(keyword, value, profile='default')[source]

Update info about a user in the system.

Parameters
  • keyword (str) – User record field to be set

  • value (str) – Value of field to set

Returns

Return type

Service response

set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")
set_svc_url(svc_url)[source]

Set the URL of the Resource Management Service to be used.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")
svcGet(token, url)[source]

Utility method to call a Resource Manager service.

Parameters
  • token (str) – User identity token

  • url (str) – URL to call with HTTP/GET

Returns

Return type

Service response

userRecord(token, user, value, format, profile='default')[source]

Get a value from the User record.

Parameters
  • token (str) – User identity token

  • user (str) – User account name to retrieve. Token must match the use name or be a root token to access other records

  • value (str) – Value to retrieve. The special ‘all’ value will return all fields accessible to the token.

  • format (str) – ‘text’ for a single value, or ‘json’ for a complete record

Returns

Return type

User record

Send a password-reset link to the user.

Parameters
  • token (str) – User identity token

  • user (str) – User account name.

Returns

Return type

Service response

dl.resClient.setGroup(token, keyword, value, profile='default')[source]

Update info about a Group in the system.

Parameters
  • keyword (str) – Group record field to be set

  • value (str) – Value of field to set

dl.resClient.setResource(token, keyword, value, profile='default')[source]

Update info about a Resource in the system.

Parameters
  • keyword (str) – Resource record field to be set

  • value (str) – Value of field to set

Returns

status – ‘OK’ is record was set

Return type

str

dl.resClient.setUser(token, keyword, value, profile='default')[source]

Update info about a user in the system.

Parameters
  • keyword (str) – User record field to be set

  • value (str) – Value of field to set

Returns

Return type

Service response

dl.resClient.set_profile(profile)[source]

Set the requested service profile.

Parameters

profile (str) – Requested service profile string.

Returns

Return type

Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")
dl.resClient.set_svc_url(svc_url)[source]

Set the URL of the Resource Management Service to be used.

Parameters

svc_url (str) – Resource Management service base URL to call.

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

Util module

dl.Util.add_doc(value)[source]

Decorator to set the ‘Call docstring’ ipython field.

dl.Util.def_token(tok)[source]

Get a default token. If no token is provided, check for an existing $HOME/.datalab/id_token.<user> file and return that if it exists, otherwise default to the ANON_TOKEN.

If a token string is provided, return it directly. The value may also simply be a username, in which case the same check for a token ID file is done.

dl.Util.encode_multipart(fields, files, boundary=None)[source]

Encode dict of form fields and dict of files as multipart/form-data. Return tuple of (body_string, headers_dict). Each value in files is a dict with required keys ‘filename’ and ‘content’, and optional ‘mimetype’ (if not specified, tries to guess mime type or uses ‘application/octet-stream’).

..code-block:: python

>>> body, headers = encode_multipart({'FIELD': 'VALUE'},
...                                  {'FILE': {'filename': 'F.TXT', 'content': 'CONTENT'}},
...                                  boundary='BOUNDARY')
>>> print('\n'.join(repr(l) for l in body.split('\r\n')))
'--BOUNDARY'
'Content-Disposition: form-data; name="FIELD"'
''
'VALUE'
'--BOUNDARY'
'Content-Disposition: form-data; name="FILE"; filename="F.TXT"'
'Content-Type: text/plain'
''
'CONTENT'
'--BOUNDARY--'
''
>>> print(sorted(headers.items()))
[('Content-Length', '193'), ('Content-Type', 'multipart/form-data; boundary=BOUNDARY')]
>>> len(body)
193
dl.Util.multimethod(module, nargs, cm)[source]

Wrapper function to implement multimethod for functions. The identifying signature in this case is the number of required method parameters. When methods are called, all original arguments and keywords are passed.

dl.Util.readTokenFile(tok_file)[source]

dltasks module

class dl.dltasks.AddCapability(datalab)[source]

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]
class dl.dltasks.Broadcast(datalab)[source]

Bases: dl.dltasks.Task

Broadcast a SAMP message

run()[source]
class dl.dltasks.Copy(datalab)[source]

Bases: dl.dltasks.Task

Copy a file in Data Lab

run()[source]
class dl.dltasks.DataLab[source]

Bases: object

Main class for Data Lab interactions

get(section, param)[source]

Get a value from the configuration file.

save(section, param, value)[source]

Save the configuration file.

class dl.dltasks.Delete(datalab)[source]

Bases: dl.dltasks.Task

Delete files in Data Lab

run()[source]
class dl.dltasks.DropMyDB(datalab)[source]

Bases: dl.dltasks.Task

Drop a user’s MyDB table. [DEPRECATED]

run()[source]
class dl.dltasks.Get(datalab)[source]

Bases: dl.dltasks.Task

Get one or more files from Data Lab.

run()[source]
class dl.dltasks.Init(datalab)[source]

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]

Initialize the Data Lab configuration.

class dl.dltasks.Launch(datalab)[source]

Bases: dl.dltasks.Task

Launch a plugin in Data Lab

run()[source]
class dl.dltasks.LaunchJob(datalab)[source]

Bases: dl.dltasks.Task

Execute a remote processing job in the Data Lab

getJob(job)[source]
run()[source]
validJob(job)[source]

Bases: dl.dltasks.Task

Link a file in Data Lab

run()[source]
class dl.dltasks.List(datalab)[source]

Bases: dl.dltasks.Task

List files in Data Lab

run()[source]
class dl.dltasks.ListCapability(datalab)[source]

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]
class dl.dltasks.ListMyDB(datalab)[source]

Bases: dl.dltasks.Task

List the user’s MyDB tables. [DEPRECATED]

run()[source]
class dl.dltasks.Login(datalab)[source]

Bases: dl.dltasks.Task

Log into the Data Lab

do_login()[source]
login()[source]

Login to the Data Lab.

run()[source]

Execute the Login Task.

class dl.dltasks.Logout(datalab)[source]

Bases: dl.dltasks.Task

Logout out of the Data Lab

run()[source]
class dl.dltasks.MkDir(datalab)[source]

Bases: dl.dltasks.Task

Create a directory in Data Lab

run()[source]
class dl.dltasks.Mountvofs(datalab)[source]

Bases: dl.dltasks.Task

Mount a VOSpace via FUSE

run()[source]
class dl.dltasks.Move(datalab)[source]

Bases: dl.dltasks.Task

Move files in Data Lab

run()[source]
class dl.dltasks.MyDB_Copy(datalab)[source]

Bases: dl.dltasks.Task

Copy a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Create(datalab)[source]

Bases: dl.dltasks.Task

Create a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Drop(datalab)[source]

Bases: dl.dltasks.Task

Drop a user’s MyDB table.

run()[source]
class dl.dltasks.MyDB_Import(datalab)[source]

Bases: dl.dltasks.Task

Import data into a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Index(datalab)[source]

Bases: dl.dltasks.Task

Index data in a user’s MyDB table.

run()[source]
class dl.dltasks.MyDB_Insert(datalab)[source]

Bases: dl.dltasks.Task

Insert data into a user MyDB table.

run()[source]
class dl.dltasks.MyDB_List(datalab)[source]

Bases: dl.dltasks.Task

List the user’s MyDB tables.

run()[source]
class dl.dltasks.MyDB_Rename(datalab)[source]

Bases: dl.dltasks.Task

Rename a user MyDB table.

run()[source]
class dl.dltasks.MyDB_Truncate(datalab)[source]

Bases: dl.dltasks.Task

Truncate a user MyDB table.

run()[source]
class dl.dltasks.Option(name, value, description, display=None, default=None, required=False)[source]

Bases: object

Represents an option

class dl.dltasks.Put(datalab)[source]

Bases: dl.dltasks.Task

Put files into Data Lab.

run()[source]
class dl.dltasks.Query(datalab)[source]

Bases: dl.dltasks.Task

Send a query to a remote query service (OLD VERSION - NOT USED))

dbquery(h, url, token)[source]
httpquery(h, url, token)[source]
ivoquery(h, uri, token, out)[source]
run()[source]
class dl.dltasks.Query2(datalab)[source]

Bases: dl.dltasks.Task

Send a query to a remote query service. [Note: placeholder name until we figure out what to do with the old Query() functionality.]

run()[source]
class dl.dltasks.QueryProfiles(datalab)[source]

Bases: dl.dltasks.Task

List the available Query Manager profiles.

run()[source]
class dl.dltasks.QueryResults(datalab)[source]

Bases: dl.dltasks.Task

Get the async query results.

run()[source]
class dl.dltasks.QueryStatus(datalab)[source]

Bases: dl.dltasks.Task

Get the async query job status.

run()[source]
class dl.dltasks.Receiver(client)[source]

Bases: object

SAMP listener

point_select(private_key, send_id, mtype, params, extra)[source]
receive_notifications(private_key, sender_id, mtype, params, extra)[source]
receiver_call(private_key, sender_id, msg_id, mtype, params, extra)[source]
class dl.dltasks.Resolve(datalab)[source]

Bases: dl.dltasks.Task

Resolve a vos short form identifier – FIXME

run()[source]
class dl.dltasks.RmDir(datalab)[source]

Bases: dl.dltasks.Task

Delete a directory in Data Lab

run()[source]
class dl.dltasks.Schema(datalab)[source]

Bases: dl.dltasks.Task

Print information about data servicce schema

run()[source]
class dl.dltasks.Services(datalab)[source]

Bases: dl.dltasks.Task

Print the available data services.

run()[source]
class dl.dltasks.SiaQuery(datalab)[source]

Bases: dl.dltasks.Task

SIA query with an uploaded file

getUID()[source]

Get the UID for this user

run()[source]
class dl.dltasks.Status(datalab)[source]

Bases: dl.dltasks.Task

Status of the Data Lab connection

run()[source]
class dl.dltasks.StorageProfiles(datalab)[source]

Bases: dl.dltasks.Task

List the available Storage Manager profiles.

run()[source]
class dl.dltasks.SvcURLs(datalab)[source]

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]
class dl.dltasks.Tag(datalab)[source]

Bases: dl.dltasks.Task

Tag a file in Data Lab

run()[source]
class dl.dltasks.Task(datalab, name, description)[source]

Bases: object

Superclass to represent a Task

addLogger(logLevel, logFile)[source]

Add a Logger to the Task.

addOption(name, option)[source]

Add an option to the Task.

addStdOptions()[source]
broadcast(client, messageType, params, app=None)[source]

Broadcast a SAMP message.

getSampConnect()[source]

Get a SAMP listening client connection.

listen(client)[source]

Setup a listener for a specific SAMP message.

report(resp)[source]

Handle call response

run()[source]
setLogger(logLevel=None)[source]

Set the logger to be used.

setOption(name, value)[source]

Set a Task option.

class dl.dltasks.Version(datalab)[source]

Bases: dl.dltasks.Task

Print the task version.

run()[source]
class dl.dltasks.WhoAmI(datalab)[source]

Bases: dl.dltasks.Task

Print the current active user.

run()[source]
dl.dltasks.getUserName(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.getUserToken(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.parseSelf(obj)[source]

dlinterface module

class dl.dlinterface.DLInteract[source]

Bases: object

Main class for Data Lab interactions

get(section, param)[source]

Get a value from the configuration file.

save(section, param, value)[source]

Save the configuration file.

class dl.dlinterface.Dlinterface(verbose=True)[source]

Bases: object

Data Lab python interface super-class with methods for each command.

copyurl(url=None, name=None)[source]

Copy a file to VOSpace using a URL.

Parameters
  • url (str) – The URL location of the file.

  • name (str) – The name of the file in VOSpace. The vos:// prefix is not necessary.

Example

Copy the file http://www.mywebsite.com/file1.fits to output1.fits in VOSpace.

dl.copyurl('http://www.mywebsite.com/file1.fits','output1.fits')
cp(source=None, destination=None, verbose=True)[source]

Copy a file in Data Lab VOSpace.

Parameters
  • source (str) – The name of the file in VOSpace to copy, e.g. file1.txt.

  • destination (str) – The new name of the file in VOSpace, e.g. newfile1.txt.

Example

Copy the file file.txt to newfile.txt.

dl.ls()
file1.txt

dl.cp('file1.txt','newfile.txt')

dl.ls()
file1.txt, newfile.txt
droptable(table=None)[source]

Drop a user’s MyDB table.

Parameters

table (str) – The name of a specific table in mydb to drop.

Returns

list – The list of properties of table or all tables in mydb.

Return type

str

Example

Drop the MyDB table called table.

print dl.listdb()
table
table2

dl.droptable('table')
table
table2

print dl.listdb()
table2
exporttable(table=None, name=None, fmt=None)[source]

Copy a user’s MyDB table to a file in VOSpace.

Parameters
  • table (str) – The name of a specific table in mydb to drop.

  • name (str) – The file name to save the table to.

  • fmt (str) – The output file format. The available formats are ‘csv’, ‘fits’ and ‘hdf5’. If this is not specified then the file extension is used to identify the format type.

Example

Export the MyDB table called table to file test.csv.

dl.exporttable('table','test.csv')
get(source=None, destination=None, verbose=True)[source]

Get one or more files from Data Lab.

Parameters
  • source (str) – The name of the source file on VOSpace, e.g. file2.txt.

  • destination (str) – The name of the local destination file, e.g. file1.txt.

Example

Get a query output table called table1_output.txt from VOSpace.

dl.get('table1_output.txt','table1_output.txt')
(1/1) [====================] [   9.1K] table1_output.txt
help(command=None)[source]

Print out useful help information on the Data Lab python interface and it’s commands.

listdb(table='')[source]

List the user’s MyDB tables.

Parameters

table (str) – The name of a specific table in mydb. If this is blank then all tables will be listed.

Returns

list – The list of properties of table or all tables in mydb.

Return type

str

Example

List the MyDB tables.

print dl.listmydb()
table
table2
ln(target=None, link=None)[source]

Link a file in Data Lab VOSpace.

Parameters
  • target (str) – The name of the file in VOSpace to link to, e.g. file1.txt.

  • link (str) – The name of the link, e.g. file1link.

Example

Create a link called iamlink to the file file1.txt.

dl.ls()
file1.txt

dl.ln('file1.txt','iamlink')

dl.ls()
file1.txt, iamlink
load(name=None, inpfmt=None, fmt='pandas', ext=None)[source]

Save the string representation of a data object to a file in VOSpace.

Parameters
  • name (str) – The name of the file in load into memory. The vos:// prefix is necessary otherwise it is assumed a local file that should be read. Currently only FITS binary tables and CSV file formats are supported.

  • inpfmt (str) – Tne input format type. The currently supported types are FITS binary tables, HDF5, CSV, and ASCII files (‘string’). If this is not specified then the file extension of name is used to attempt to figure out the format.

  • fmt (str) –

    The data type format to output. Permitted values are:
    • ’csv’ the returned result is a comma-separated string that looks like a

      csv file (newlines at the end of every row)

    • ’ascii’ same as csv but tab-delimited

    • ’string’ just a straight read of ASCII into a string

    • ’array’ Numpy array

    • ’structarray’ Numpy structured / record array

    • ’pandas’ a Pandas data frame

    • ’table’ in Astropy Table format

    • ’votable’ result is a string XML-formatted as a VO table

    The output type for a FITS image is a numpy array. For other data ‘pandas’ is the default format.

  • ext (int) – The FITS extension to load for images. The default is 1 for a FITS binary table and 0 for a FITS image.

Example

Load the file “output.fits” into a pandas data frame.

df = dl.load('output.fits',fmt='pandas')

Load a FITS image “im1.fits”.

im,head = dl.load('im1.fits')
login(user=None)[source]

Login to Data Lab using username.

Parameters

user (str) – The Data lab username. If this is not given, then the user will be prompted for the information.

Example


Login and give the username,

dl.login(‘myusername’) Enter password: *** Welcome to the Data Lab, myusername

or,

dl.login() Enter user: myusername Enter password: ** Welcome to the Data Lab, myusername

logout(unmount=None, verbose=True)[source]

Logout out of the Data Lab.

Example

Logout of Data Lab.

dl.logout()
'myusername' is now logged out of the Data Lab
ls(name='vos://', format='csv', verbose=False)[source]

List files in VOSpace.

Parameters
  • name (str) – The name of a specific file to list. If name is blank then all files will be listed.

  • format (str) – The format to use.

  • verbose (bool) – Give more verbose output, or just a list of files. The default is verbose=True.

Returns

results – The list of files in VOSpace.

Return type

str

Example

List the files.

dl.ls()
test2  test1

Verbose listing of the files in the public/ directory.

dl.ls('public',verbose=True)
lrw-rw----  demo15      0B  17 May 2017 14:04:25  thisisalsoalink -> /public/smash2
lrw-rw----  demo15      0B  17 May 2017 13:58:04  thisisalink -> /smash1
-rw-rw-r--  demo15    3.4K  17 May 2017 09:40:13  smash2
-rw-rw-r--  demo15    3.4K  17 May 2017 07:34:54  smash1
drw-rw----  demo15      0B  17 May 2017 14:05:02  data/  tableingester,downloader,runner
mkdir(name=None)[source]

Create a directory in Data Lab VOSpace.

Parameters

name (str) – The name of the directory in VOSpace to create, e.g. results.

Example

Create the directory data1/.

dl.mkdir('data1')
mv(source=None, destination=None, verbose=True)[source]

Move a file in Data Lab VOSpace.

Parameters
  • source (str) – The name the file in VOSpace to move/rename, e.g. file1.txt.

  • destination (str) – The new name of the file in VOSpace (e.g. newfile1.txt) or the directory to move it to.

Example

Rename the file file.txt to newfile.txt.

dl.ls()
file.txt

dl.mv('file.txt','newfile.txt')

dl.ls()
newfile.txt

Move the file output.fits to the results/ directory.

dl.ls()
output.txt, results

dl.mv('output.fits','results/output.fits')

dl.ls()
results/output.txt
put(source=None, destination=None, verbose=True)[source]

Put files into Data Lab VOSpace.

Parameters
  • source (str) – The name of a local file to upload to VOSpace, e.g. file1.txt.

  • destination (str) – The name of the destination file with, e.g. file2.txt. The destination file can have the vos:// prefix but it is not required.

Example

Put a catalog called cat.fits into VOSpace.

dl.put('cat.fits','cat.fits')
(1 / 1) cat.fits -> vos://cat.fits
query(query=None, qtype='sql', fmt='csv', out=None, async_=False, profile='default', verbose=True, **kw)[source]

Send a query to a remote query service.

Parameters
  • query (str) –

    The query string that will be passed to the queryClient and then to the DB query manager. This can either be in the SQL or ADQL format (specified by the “type” parameter). For example,

    'select ra,dec from gaia_dr1.gaia_source limit 3'
    

  • qtype (str) – The query format, SQL or ADQL. SQL is used by default.

  • fmt (str) –

    Format of the result to be returned by the query. Permitted values are.

    For file output and direct output to python: * ‘csv’ the returned result is a comma-separated string that looks like a csv file (newlines at the end of every row) * ‘ascii’ same as csv but tab-delimited * ‘votable’ result is a string XML-formatted as a VO table Only for direct output to python: * ‘array’ Numpy array * ‘structarray’ Numpy structured / record array * ‘pandas’ a Pandas data frame * ‘table’ in Astropy Table format Only for file output: * ‘fits’ FITS binary table. Only if the results are saved to a file with out=. * ‘hdf5’ HDF5 file. Only if the results are saved to a file with out=.

  • out (str or None) – The output name if the results are to be saved to mydb (mydb://tablename), to VOSpace (vos://filename), or the local file system (file:// and other names with no prefix). The files are in csv format.

  • async (bool) –

    If True, the query is asynchronous, i.e. a job is submitted to the DB, and a jobID is returned. The jobID must be then used to check the query’s status and to retrieve the result (when status is COMPLETE). Default is False, i.e. synchroneous query.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

Returns

result – If async_=False and out is not used, then the return value is the result of the query in the requested format (see fmt). If out is given then the query result is saved to a file or mydb. If async_=True the jobID is returned with which later the asynchronous query’s status can be checked (dl.querystatus()), and the result retrieved (see dl.queryresults().

Return type

str

Example

A simple query returned as a pandas data frame.

data = dl.query('SELECT * from smash_dr1.source LIMIT 100',fmt='pandas')
Returning Pandas dataframe

type(data)
pandas.core.frame.DataFrame

print data['ra'][0:3]
0    103.068355
1    103.071774
2    103.071598

Perform a query and save the results to a table called “table1.txt” in mydb.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='mydb://table1.txt')

 dl.listmydb()

Perform the same query and save it to a local file.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='table1.txt')

ls
table1.txt
queryhistory(async_=None, **kw)[source]

Report the history of queries made so far.

Parameters
  • async (bool) –

    A boolean (True/False) of whether to only show the ASYNC queries. By default all queries are shown.

    async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.

  • Results

  • -------

  • information on part queries is output to the screen with the following (The) –

  • columns (query ID, submission time, query type (sql/adql), sync or async query, jobid (for async queries),) – output format, status of query (or number of returned rows if sync query), query string

Examples

Perform some queries and then list the history.

data1 = dl.query('select ra,dec from smash_dr1.source limit 100',fmt='csv')
Returning CSV formatted table as a string

data2 = dl.query('select ra,dec from smash_dr1.source limit 500',fmt='pandas')
Returning Pandas dataframe

data3 = dl.query('select ra,dec from smash_dr1.source limit 1000',fmt='structarray')
Returning Numpy structured / record array

dl.queryhistory()
1  2017-05-16 13:27:34  sql  SYNC  pandas  100  --  'select ra,dec,gmag from smash_dr1.object limit 100'
2  2017-05-16 13:27:40  sql  SYNC  csv  500  --  'select ra,dec,gmag from smash_dr1.object limit 500'
3  2017-05-16 13:27:46  sql  SYNC  structarray  1000  --  'select ra,dec,gmag from smash_dr1.object limit 1000'
queryprofiles(profile=None)[source]

List the available Query Manager profiles to use with a dl.query().

Parameters

profile (str) – The name of a specific Query Manager profile to check. If this is blank then all of the available profile names will be listed.

Returns

results – The list of properties of profile profile, or a list of all available profiles.

Return type

str

Example

List of available profiles.

dl.queryprofiles()
default,IRSA,HEASARC,Vizier,GAVO,SIMBAD,zeus1,SDSS-DR9,STScI-RegTAP,GALEX-DR6,dldb1

Get profile information on profile dldb1.

dl.queryprofiles(‘dldb1’) {u’accessURL’: u’http://dldb1.sdm.noao.edu:8080/ivoa-dal/tap’, u’dbport’: 5432, u’password’: u’datalab’, u’description’: u’Development NOAO Data Lab TAP Service / Database on dldb1’, u’database’: u’tapdb’, u’host’: u’dldb1.sdm.noao.edu’, u’vosRoot’: u’vos://datalab.noao!vospace’, u’vosEndpoint’: u’http://dldb1.sdm.noao.edu:8080/vospace-2.0/vospace’, u’user’: u’dlquery’, u’vosRootDir’: u’/data/vospace/users’, u’type’: u’datalab’}

queryresults(jobid=None)[source]

Get the async query results.

Parameters

jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.

Returns

result – The result of the query in the requested format (see fmt in dl.query().

Return type

str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 3',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED

results = dl.queryresults(jobid)
print results
ra,dec
103.068354922718,-37.973538878907299
103.071774116284,-37.973599429479599
103.071597827998,-37.972329108796401
querystatus(jobid=None)[source]

Get the async query job status.

Parameters

jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.

Returns

status – The status of the query, which can be one of the following: QUEUED the query job is in queue and waiting to be executed. EXECUTING the query job is currently running. COMPLETED the query is done and the results are ready to be retrieved with dl.queryresults(). ERROR there was a problem with the query.

Return type

str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 100',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED
rm(name=None, verbose=True)[source]

Delete files in Data Lab VOSpace.

Parameters

name (str) – The name of the file in VOSpace to delete, e.g. file1.txt.

Example

Delete the file file1.txt.

dl.ls()
file1.txt, file2.txt

dl.rm('file1.txt')

dl.ls()
file2.txt
rmdir(name=None)[source]

Delete a directory in Data Lab VOSpace.

Parameters

name (str) – The name of the directory in VOSpace to delete, e.g. results.

Example

Delete the directory data1/.

dl.rmdir('data1')
save(data=None, name=None, fmt=None, clobber=False)[source]

Save the string representation of a data object to a file in VOSpace.

Parameters
  • data (str) – The data object such as a pandas data frame or numpy structured array.

  • name (str) – The name of the file in VOSpace to create. The vos:// prefix is not necessary.

  • fmt (str) –

    The format to use for the output file. If this is not specified then the file extension of name is used to attempt to figure out the format. The currently supported input and output formats are:

    Data type Format csv csv ascii ascii array csv/fits structarray csv/fits pandas csv/fits/hdf5 table csv/fits/hdf5 votable csv/fits/hdf5

  • clobber (bool) – Whether to overwrite an existing file. The default is False.

Example

Save the pandas data frame called “df” to a file called “data1.csv” in VOSpace.

dl.save(df,'data1.csv')
schema(val='', fmt='text', profile='default')[source]

Print information about data service schema.

Parameters
  • val (str) – Value to list ([[<schema>][.<table>][.<col>]]).

  • fmt (str) – Output format (csv|text|json).

  • profile (str) – Service profile.

Returns

results – The schema information is printed to the screen.

Return type

str

Example

Print out all the DL tables.

 datalab schema

 Schema Name   Description
-----------   -----------
   gaia_dr1   GAIA Data Release 1
       ivoa   IVOA ObsCore tables
   des_sva1   DES SVA1 Data Products
 tap_schema   TAP Schema Tables
       usno   USNO Astrometry Catalogs
  sdss_dr13
    neo_dr1   NEO Survey Data Release 1
     ls_dr3   The DECam Legacy Survey Data Release 3
  smash_dr1   SMASH Data Release 1

List all tables in a schema/catalog.

datalab schema val=smash_dr1

Schema: smash_dr1

Table Name   Description
----------   -----------
      chip   Info on each chip in the frame
  exposure   Info on each exposure
     field   Info on each target field (position, Num exposures, etc)
    object   Average photometry of each unique object
    source   All of the individual source measurements
     stars   View of object table to select for stars
  galaxies   View of object table to select for galaxies
    xmatch   Crossmatch of object against GAIA DR1 and WISE
servicestatus()[source]

This checks on the status of the DL services.

siaquery(ra=None, dec=None, dist=None, verbose=False)[source]

Perform a SIA query with a set of coordinates or from an uploaded file.

Parameters
  • ra (float) – The right ascension (in degrees) of the point to use for the search.

  • dec (float) – The declination (in degrees) of the point to use for the search.

  • dist (float) – The search distance (radius) in degrees. The default is 0.0085 deg.

  • verbose (bool) – Use verbose output. The default is False.

Returns

images – The list of images in Astropy table format.

Return type

votable

Example

Perform a simple SIA search.

itab = dl.siaquery(0.5,10.0,0.1)
The image list contains 6 entries

Download the first image using copyurl().

dl.copyurl(itab['access_url'][0],'im1.fits')
status()[source]

Print the status of the Data Lab connection.

Example

The “myusername” is logged in.

dl.status()
User myusername is logged into the Data Lab

No user is currently logged in.

dl.status()
No user is currently logged into the Data Lab
whoami()[source]

Print the current active user.

Example

dl.whoami()
myusername
dl.dlinterface.addFormatMapping(self)[source]

Add the format mapping information to the DL object

dl.dlinterface.areLoginsWorking()[source]

This checks if the Authentication Manager is returning proper tokens.

dl.dlinterface.areSyncQueriesWorking()[source]

This checks if the Query Manager is returning proper Sync queries.

dl.dlinterface.checkLogin(self)[source]

Check if the user is already logged in. If not, give a warning message

dl.dlinterface.convertTableToFormat(t, format)[source]

Convert Astropy table to a different format using StringIO and write method.

dl.dlinterface.convert_vospace_time_to_seconds(str_date)[source]

A convenience method that takes a string from a vospace time field and converts it to seconds since epoch.

Parameters

str_date (str) – string to parse into a VOSpace time

Returns

A datetime object for the provided string date

Return type

datetime

dl.dlinterface.getNodeInfo(self, xnode, lenpathbase, verbose=True)[source]

Get information on a node. The input is a “node” element of a XML ElementTree.

dl.dlinterface.getUserName(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous username.

dl.dlinterface.getUserToken(self)[source]

Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dlinterface.isListWorking()[source]

This checks if the Storage Manager is returning proper list queries:

dl.dlinterface.isTapWorking()[source]

This checks if the TAP service and Tomcat are running.

dl.dlinterface.readAscii(filename)[source]

Read in an ASCII file and return the data

dl.dlinterface.reformatQueryOutput(self, res=None, fmt='csv', verbose=True)[source]

Reformat the output of a query based on a format.

dl.dlinterface.writeAscii(filename, txt)[source]

Write data to ASCII file

dl.helpers package

helpers.all module

helpers.cluster module

Data Lab helpers for clustering.

dl.helpers.cluster.constructOutlines(x, y, clusterlabels)[source]

Construct the convex hull (outline) of points in (x,y) feature space,

Parameters

y (x,) – Location of points in (x,y) feature space (e,g, RA & Dec).

Returns

hull – The convex hull of points (x,y), an instance of scipy.spatial.qhull.ConvexHull.

Return type

instance

Example

Given x & y coordinates as 1d sequences:

points = np.vstack((x,y)).T  # make 2-d array of correct shape
hull = constructOutlines(x,y)
plt.plot(points[hull.vertices,0], points[hull.vertices,1], 'r-', lw=2) # plot the hull
plt.plot(points[hull.vertices[0],0], points[hull.vertices[0],1], 'r-') # closing last point of the hull
dl.helpers.cluster.findClusters(x, y, method='MiniBatchKMeans', **kwargs)[source]

Find 2D clusters from x & y data.

Parameters
  • y (x,) – Location of points in (x,y) feature space, e,g, RA & Dec, but x & y need not be spatial in nature.

  • method (str) – Cluster finding method from sklearn.cluster to use. Default: ‘MiniBatchKMeans’ (a streaming implementation of KMeans), which is very fast, but not the most robust. ‘DBSCAN’ is much more robust, but MUCH slower. For other methods, consult sklearn.cluster.

  • **kwargs

    Any other keyword arguments will be passed to the cluster finding method. If method=’MiniBatchKMeans’ or ‘KMeans’, n_clusters (integer number of clusters to find) must be passed, e.g.

    clusters = findClusters(x,y,method='MiniBatchKMeans',n_clusters=3)
    

helpers.crossmatch module

Data Lab helpers for (local) positional cross-matching.

dl.helpers.crossmatch.make_catalog(n, min, max)[source]
dl.helpers.crossmatch.plot_time_vs_catalogsizes(times, extent=(2, 7))[source]
dl.helpers.crossmatch.test()[source]
dl.helpers.crossmatch.xmatch(ra1, dec1, ra2, dec2, maxdist=None, units='deg', method='astropy', **kwargs)[source]

Cross-match two sets of ra & dec coordinates locally (i.e. all coordinates are in RAM).

The function will search for counterparts of ra1/dec1 coordinates in the in ra2/dec2 coordinate set, i.e. one can consider ra2/dec2 to be the catalog that will be searched.

Parameters
  • dec1 (ra1,) – RA and declination of first coordinate set, in units of units

  • dec2 (ra2,) – RA and declination of second coordinate set, in units of units

  • maxdist (float or None) – If not None, then it is the maximum angular distance (in units of units) to be considered. All distances greater than that will be considered non-matches. If None, then all ra1/dec1 will have matches in ra2/dec2.

  • units (str) – Units of ra1, dec1, ra2, dec2. Default: ‘deg’ (decimal degrees).

  • method (str) – Currently only astropy’s match_to_catalog_sky() method is supported, i.e. the default ‘astropy’.

Other Parameters

nthneighbor (int, optional) – If method='astropy'. Which closest neighbor to search for. Typically 1 is desired here, as that is correct for matching one set of coordinates to another. The next likely use case is 2, for matching a coordinate catalog against itself (1 is inappropriate because each point will find itself as the closest match).

Returns

  • idx (1-d array) – Index values of the ra1/dec1 counterparts found in ra2/dec2. Thus ra2[idx], dec2[idx] will select from the ra2/dec2 catalog the matched counterparts of the ra1/dec1 coordinate pairs.

    If maxdist was not None but a number instead, then ‘idx’ only contains the objects matched up to the maxdist radius.

  • dist2d (1-d array) – The angular distances of the matches found in the ra2/dec2 catalog. In units of units.

    If maxdist was not None but a number instead, then ‘dist2d’ only contains the objects matched up to the maxdist radius.

helpers.legacy module

Legacy helpers for Data Lab. Most are deprecated.

class dl.helpers.legacy.Querist(username='anonymous')[source]

Bases: object

checkAsyncJob()[source]

Check the first async job in the FIFO queue (if queue is not empty).

Parameters

None

Returns

  • Always returns a 3 tuple. If no async job was in the queue,

  • returns (None,None,None). If there was an async query in the

  • queue but its status did not return ‘COMPLETED’, re-inserts

  • the query at its old position in the queue, and returns

  • (None,None,None). If the status was ‘COMPLETED’, returns the

  • tuple (query result,outfmt,preview).

clearQueue()[source]

Clears the async job queue, i.e. they become unretrievable.

clearToken()[source]

Sets to token to empty string. Useful e.g. before saving a notebook.

output_formats

Pretty-print to STDOUT the available outfmt values.

Parameters

None

Returns

Return type

Nothing

printMapping()[source]

Pretty-print to STDOUT the available outfmt values.

Parameters

None

Returns

Return type

Nothing

helpers.plot module

helpers.utils module

Data Lab utility helper functions.

dl.helpers.utils.convert(inp, outfmt='pandas', verbose=False, **kwargs)[source]

Convert input inp to a data structure defined by outfmt.

Parameters
  • inp (str) – String representation of the result of a query. Usually this is a CSV-formatted string, but can also be, e.g. an XML-formatted votable (as string)

  • outfmt (str) –

    The desired data structure for converting inp to. Default: ‘pandas’, which returns a Pandas dataframe. Other available conversions are:

    string - no conversion array - Numpy array structarray - Numpy structured array (also called record array) table - Astropy Table votable - Astropy VOtable

    For outfmt=’votable’, the input string must be an XML-formatted string. For all other values, as CSV-formatted string.

  • verbose (bool) – If True, print status message after conversion. Default: False

  • kwargs (optional params) – Will be passed as **kwargs to the converter method.

Example

Convert a CSV-formatted string to a Pandas dataframe

arr = convert(inp,'array')
arr.shape  # arr is a Numpy array

df = convert(inp,outfmt='pandas')
df.head()  # df is as Pandas dataframe, with all its methods

df = convert(inp,'pandas',na_values='Infinity') # na_values is a kwarg; adds 'Infinity' to list of values converter to np.inf
dl.helpers.utils.normalizeCoordinates(x, y, frame_in='icrs', units_in='deg', frame_out=None, wrap_at=180)[source]

Makes 2D spatial coordinates (e.g. RA & Dec) suitable for use with matplotlib’s all-sky projection plotting.

Parameters
  • y (x,) – Location of points in (x,y) feature space (e,g, RA & Dec in degrees). Avoid supplying x and y as columns from a pandas dataframe, as this unfortunately makes the coordinate conversions much slower. Numpy arrays, lists, astropy table and votable columns, all are fine.

  • frame_in (str) – Coordinate frame of x & y. Default: ‘icrs’. ‘galactic’ is also available. If the user desires other frames from astropy.coordinates, please contact __author__.

  • units_in (str) – Units of x & y. Default ‘deg’ (degrees).

  • frame_out (None or str) – If not None, and not same as frame_in, the x & y coordinates will be transformed from frame_in to frame_out.

  • wrap_at (float) – matplotlib plotting functions such as matplotlib.scatter() with all-sky projections expect the x-coordinate (e.g. RA) to be between -180 and +180 degrees (or more precisely: between -pi and +pi). The default wrap_at=180 shifts the input coordinate x (e.g. RA) accordingly.

dl.helpers.utils.resolve(name=None)[source]

Resolve object name to coordinates.

Parameters

name (str or None) – If str, it is the name of the object to resolve. If None (default), a primpt for the object name will be presented.

Returns

sc – Instance of SkyCoord from astropy. Get e.g. RA via sc.ra (with units), or sc.ra.value (without units). Or explictly in a different coordinate system, e.g. sc.galactic.b, etc.

Return type

instance

dl.helpers.utils.vospace_readable_fileobj(name_or_obj, token=None, **kwargs)[source]

Read data from VOSpace or some other place.

Notes

Most of the heavy lifting is done with get_readable_fileobj(). Any additional keywords passed to this function will get passed directly to that function.

Parameters
  • name_or_obj (str or file-like object) –

    The filename of the file to access (if given as a string), or the file-like object to access.

    If a file-like object, it must be opened in binary mode.

  • token (str) – A token granting access to VOSpace.

Returns

A readable file-like object.

Return type

file