SaltyCrane Blog — Notes on JavaScript and web development

Logging the actual host (instead of localhost) with the Python elasticsearch client

Date:  |  Tags: python

The Elasticsearch Python library has a cool feature that allows you to log the equivalent curl command of the query you are making. To use it you just need to configure the elasticsearch.trace logger. There is one annoyance-- by design, "localhost:9200" is logged even if you have configured another host. It is more useful to log the actual host I am using so I can copy and paste the command to co-workers without changing the hostname. At first I thought I'd monkeypatch the library to do what I wanted. Then I remembered I could use a logging.Filter to rewrite the log message.

I am using the high level elasticsearch-dsl library in my example, but the logging is actually done by the low level official elasticsearch library which elasticsearch-dsl wraps.

$ pip install elasticsearch-dsl

import logging

import elasticsearch_dsl as esdsl  # high level elasticsearch library
from elasticsearch import Elasticsearch  # low level official elasticsearch library


ELASTICSEARCH_HOSTS = [
    'http://es1.myhost.com:9200',
    'http://es2.myhost.com:9200',
    'http://es3.myhost.com:9200',
]


class ReplaceHostFilter(logging.Filter):
    """
    Replace "localhost:9200" with real host
    """
    def __init__(self, real_host, *args, **kwargs):
        self.real_host = real_host
        super(ReplaceHostFilter, self).__init__(*args, **kwargs)

    def filter(self, record):
        record.msg = record.msg.replace('http://localhost:9200', self.real_host)
        record.msg = record.msg.replace('curl', 'curl -sS')
        record.msg += ' | python -mjson.tool'
        return True


# Set up logging
host_replacer = ReplaceHostFilter(ELASTICSEARCH_HOSTS[0])
formatter = logging.Formatter("%(levelname)s %(asctime)s %(name)s - %(message)s")
sh = logging.StreamHandler()
sh.setFormatter(formatter)
eslogger = logging.getLogger('elasticsearch.trace')
eslogger.setLevel('INFO')
eslogger.addFilter(host_replacer)
eslogger.addHandler(sh)


# Make a query
esclient = Elasticsearch(hosts=ELASTICSEARCH_HOSTS)
search = esdsl.Search(using=esclient, index='my_index')
filt = esdsl.F('term', myTerm=1234)
search = search.filter(filt)
response = search.execute()
print response.hits.total

Log output showing the actual host instead of localhost:

INFO 2015-07-28 13:48:54,464 elasticsearch.trace - curl -sS -XGET 'http://es1.myhost.com:9200/my_index/_search?pretty' -d '{
  "query": {
    "filtered": {
      "filter": {
        "term": {
          "myTerm": 1234
        }
      },
      "query": {
        "match_all": {}
      }
    }
  }
}' | python -mjson.tool

Comments