Connecting to a haystack server¶
Pyhaystack has support modules for various Project Haystack implementations, as well as an extensible framework for adding support for more implementations.
The core object in Pyhaystack is the
pyhaystack.client.session.HaystackSession
object, which manages
the user credentials, implements caching and provides the interface through
which you or your scripts interact. This is an abstract class, for which
several sub-classes exist.
There are two ways to create a session instance:
- Directly: by importing the relevant class and calling its constructor.
- Via the
pyhaystack.connect
(also known aspyhaystack.client.get_instance()
) factory.
The former works well for ad-hoc usage in terminal sessions such as Jupyter Notebook, ipython and the plain Python shell. The latter is recommended for scripts that instantiate a session from a configuration file.
Usage hints¶
Logging messages in Jupyter Notebook¶
For interactive users using the Jupyter Notebook, the Jupyter notebook configures its logs to only show messages with severity “warning” or higher. To display logging messages at a lower level, run the following in your session:
import logging
logging.getLogger().setLevel(logging.INFO) # or DEBUG
Then log messages will appear in your session.
On-Demand connection¶
Pyhaystack uses a feature called “lazy evaluation” to handle the actual log-in session, and so will remain dormant after the session instance is connected until a request is made.
If, when making a request, pyhaystack detects that it has been disconnected, it will attempt to re-connect automatically.
Loading from a file¶
The connect
approach lends itself well to storing the connection
details in a plaintext file using either JSON or YAML format. e.g. given the
file my-haystack-server.json:
{
"implementation": "skyspark",
"uri": "http://ip:port",
"username": "user",
"password": "password",
"project": "my_project",
"pint": true
}
This can be instantiated like this:
import json
import pyhaystack
session = pyhaystack.connect(
**json.load(
open("my-haystack-server.json","r")
)
)
(Similarly for YAML, import yaml
and use yaml.safe_load
.)
Base session options¶
pyhaystack.client.session.HaystackSession
has the following base
arguments. All subclasses should pass these values though as keyword
arguments, so all should be usable.
uri
: This is the base URI used to access the Project Haystack server.grid_format
: This selects the grid serialisation format for the Project Haystack server. Some (e.g. nHaystack/Niagara AX) only support ZINC serialisation, where as others (such as WideSky and Niagara4) work better with JSON. Most of the time, the default here will be selected appropriate for the underlying server implementation. Valid values arezinc
andjson
.http_client
: This selects which HTTP client implementation to use for the session instance. pyhaystack at the moment just has two implementations:pyhaystack.client.http.sync.SyncHttpClient
: a synchronous HTTP client based on the Python Requests library. (default)pyhaystack.client.http.dummy.DummyHttpClient
: an asynchronous dummy HTTP client used for writing unit tests.
There are plans to implement asynchronous HTTP clients for various Python asynchronous frameworks (e.g. asyncio, TornadoWeb, Twisted, etc) in the future.
http_args
: This is adict
of keyword arguments that are passed to the constructor of thehttp_client
class used to create a HTTP client instance. IfNone
is given, then it is assumed that no arguments are required.tagging_model
: This is used by the high-level entity interface to allow Python objects to be created using various mix-ins based on the tags attached to the entity. The default model,pyhaystack.client.entity.models.haystack.HaystackTaggingModel
assumes a standard Project Haystack tagging model and should suit most users.pint
: This boolean passed to thehszinc
module to enable use of thepint
quantity classes (providing on-the-fly unit conversion). By default, this isFalse
.log
: An instance oflogging.Logger
used for session logging messages. If not given, then a logger namedpyhaystack.client.${CLASS_NAME}
is created.cache_expiry
: An integer or floating-point value representing the period of time beforeabout
/formats
/ops
response cache expires. The default is one hour.
HTTP client options (http_client
and http_args
)¶
The HTTP client interface is written with asynchronous HTTP clients in mind
using a callback-style interface. Internally, the HaystackSession
class
does this:
if http_args is None:
http_args = {}
# … etc …
# Create the HTTP client object
if bool(http_args.pop('debug',None)) and ('log' not in http_args):
http_args['log'] = log.getChild('http_client')
self._client = http_client(uri=uri, **http_args)
With the exception of the debug
and log
parameters, everything else is
passed through verbatim to the underlying HTTP client class. The following
are the most useful arguments for http_args
:
log
: If given, this is an instance of alogging.Logger
class that will be used for log messages from the HTTP client itself.debug
: A boolean flag that enables HTTP client debugging. If given, theHaystackSession
will create a newlogging.Logger
class for the HTTP client (actually, a child logger of its own logger) for HTTP client debug messages.timeout
: an integer or floating-point value that specifies the HTTP request time-out delay in seconds.proxies
: Adict
that maps the host name and protocol of a target server to the URI for a local HTTP proxy server to use for that server.tls_verify
: TLS verification of server certificates. If set toTrue
, then the server’s HTTP certificate will be verified against CA certificates known to the Python process. If you use a custom CA, then this should be set to the full filesystem path to where that CA certificate is stored. Verification can be skipped by setting this toFalse
(not recommended).When using a custom CA, the full certificate chain is required. This is usually done by converting all relevant intermediate certificates to PEM format (aka
.crt
files) and concatenated in order, that is:- the certificate for the CA that signed your server’s certificate
- the certificate for the CA that signed that CA’s certificate
- … etc
- the certificate for the root CA.
If the root CA signed the server’s certificate, then the chain is literally the root CA’s certificate itself. Note that your server’s certificate is NOT part of the bundle.
See next section on CA certificates for more details on ways to add certificates to the python process (requests).
tls_cert
: TLS client certificate. This is used to authenticate the Pyhaystack client to a Project Haystack server using TLS client authentication. It should either be the full path to a combined certificate/key in PEM format, or atuple
of the form(tls_client_cert, tls_client_key)
where bothtls_client_cert
andtls_client_key
are full paths to the relevant files.
The base class also supports some additional parameters that may be helpful in very specialised environments.
params
: adict
of URI query parameters to add to all requests.headers
: adict
of HTTP headers to add to all requests.cookies
: adict
of HTTP cookies to add to all requests.auth
: Authentication credentials, in the form of apyhaystack.client.http.auth.AuthenticationCredentials
sub-class.
CA Certificates¶
Certifi¶
When connecting to a server using SSL certificates, you may face specific warning from requests. Using http_args={tls_verify: False} should not be a permanent solution.
Following requests recommandations, it is possibile to use certifi. Certifi is a collection of curated Root Certificates. This adds a file in your site-packages folder that will hold trusted certificates used by Requests. You can find this file using
import certifi
certifi.where()
# Returns something like
'/usr/local/lib/python2.7/site-packages/certifi/cacert.pem'
You can then add trusted certificates to this file (just append the trusted .pem text file to the end of cacert.pem).
Requests will accept all certificate from this file and consider your server secure. This will also prevent SSL warnings filling your session debug.
Manual removing of warnings¶
Another way of dealing with warning log messages is to use specific code to disable warnings
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
Disabling warning is done at your own risks. Use those features carefully.
SubjectAltWarning¶
By default, pyhaystack disable the SubjectAltWarning
# extract from __init__.py
import requests.packages.urllib3
from requests.packages.urllib3.exceptions import SubjectAltNameWarning
requests.packages.urllib3.disable_warnings(SubjectAltNameWarning)
This is motivated by this issue : urllib3_issue523
Connecting to specific Haystack server implementations¶
Niagara4 (nHaystack)¶
Specific arguments¶
In addition to those supported by the base class, the following constructor arguments are supported:
username
: The username to use when authenticating with nHaystackpassword
: The password to use when authenticating with nHaystack
Direct approach
¶
from pyhaystack.client.niagara import Niagara4HaystackSession
session = Niagara4HaystackSession(uri='http://ip:port',
username='user',
password='myPassword',
pint=True)
connect()
approach¶
import pyhaystack
session = pyhaystack.connect(implementation='n4',
uri='http://ip:port',
username='user',
password='myPassword',
pint=True)
Niagara AX (nHaystack)¶
Specific arguments¶
In addition to those supported by the base class, the following constructor arguments are supported:
username
: The username to use when authenticating with nHaystackpassword
: The password to use when authenticating with nHaystack
Direct approach
¶
from pyhaystack.client.niagara import NiagaraHaystackSession
session = NiagaraHaystackSession(uri='http://ip:port',
username='user',
password='myPassword',
pint=True)
connect()
approach¶
import pyhaystack
session = pyhaystack.connect(implementation='ax',
uri='http://ip:port',
username='user',
password='myPassword',
pint=True)
VRT Widesky¶
Specific arguments¶
In addition to those supported by the base class, the following constructor arguments are supported:
username
: The email address to use when authenticating with WideSkypassword
: The password to use when authenticating with WideSkyclient_id
: The OAuth2 client identity to use when authenticating with WideSky.client_secret
: The OAuth2 client secret to use when authenticating with WideSky.impersonate
: This is an optional parameter. If this is set and the caller
has the required permission to act as the target user. Then all subsequent requests coming from this HTTP session will be based on the target user’s READ/WRITE permissions.
Direct approach
¶
from pyhaystack.client.widesky import WideskyHaystackSession
session = WideskyHaystackSession(
uri='https://yourtenant.on.widesky.cloud/reference',
username='user', password='my_password',
client_id='my_id', client_secret='my_secret'
pint=True, impersonate='user_id')
connect()
approach¶
import pyhaystack
session = pyhaystack.connect(implementation='widesky',
uri='https://yourtenant.on.widesky.cloud/reference',
username='user', password='my_password',
client_id='my_id', client_secret='my_secret',
pint=True, impersonate='user_id')
Skyspark¶
Specific arguments¶
In addition to those supported by the base class, the following constructor arguments are supported:
username
: The username to use when authenticating with SkySparkpassword
: The password to use when authenticating with SkySparkproject
: The name of the SkySpark project instance.
Direct approach
¶
from pyhaystack.client.skyspark import SkysparkHaystackSession
session = SkysparkHaystackSession(uri='http://ip:port',
username='user',
password='my_password',
project='my_project'
pint=True)
connect()
approach¶
import pyhaystack
session = pyhaystack.connect(implementation='skyspark',
uri='http://ip:port',
username='user',
password='my_password',
project='my_project'
pint=True)
SkyFoundry Demo Server¶
For basic testing, SkyFoundry has a simple server written in Java that can be used as a Hello-World style setup. The Skyfoundry Java server does not support authenication, so we have an “authless” session that can be used.
Setup¶
Install the haystack-java package from GitHub and follow its build instructions. That will leave you with a “.war” file that you can run with Tomcat.
For a lightweight approach to run the Haystack server, the Heroku webapp-runner is a simple way to run the resulting “.war” file Clone it from Github and build it with Maven and then run it with
java -jar assembly/target/webapp-runner.jar ../haystack-java/build/libs/haystack-java-3.0.5.war
Specific arguments¶
This class takes no new arguments.
Direct approach
¶
from pyhaystack.client.authless import AuthlessHaystackSession
session = AuthlessHaystackSession(uri='http://localhost:8080')