SaltyCrane: webdevhttps://www.saltycrane.com/blog/2014-09-17T18:38:10-07:00A bank style session timeout example using jQuery, Bootstrap, and Flask
2014-09-17T18:38:10-07:00https://www.saltycrane.com/blog/2014/09/bank-style-session-timeout-example-using-jquery-bootstrap-and-flask/<p>
This is an example that uses JavaScript to display a session timeout
warning modal 10 minutes before session expiration. It also resets the
session expiration whenever the user clicks the mouse. It uses
JavaScript,
<a href="https://jquery.com/">jQuery</a>, and
<a href="http://getbootstrap.com/">Bootstrap</a>
on the frontend and Python,
<a href="http://flask.pocoo.org/">Flask</a>,
<a href="https://flask-login.readthedocs.org/en/latest/">Flask-Login</a>, and
<a href="https://wtforms.readthedocs.org/en/latest/">WTForms</a>
on the backend.
</p>
<ul>
<li>Mouse clicks anywhere on the page ping the server at a maximum
frequecy of once per minute and reset the session expiration.</li>
<li>10 minutes before the session expiration, a warning modal is
displayed with two buttons: "Log out" and "Stay Logged In".</li>
<li>If the user clicks "Stay Logged In" the session expiration is reset.</li>
<li>If the user clicks "Log out", the user is logged out.</li>
<li>If the user does nothing for 10 minutes, the user is logged out and
displayed a message that the session timed out.</li>
</ul>
<p>Here is the JavaScript (session-monitor.js):</p>
<pre class="javascript">
sessionMonitor = function(options) {
"use strict";
var defaults = {
// Session lifetime (milliseconds)
sessionLifetime: 60 * 60 * 1000,
// Amount of time before session expiration when the warning is shown (milliseconds)
timeBeforeWarning: 10 * 60 * 1000,
// Minimum time between pings to the server (milliseconds)
minPingInterval: 1 * 60 * 1000,
// Space-separated list of events passed to $(document).on() that indicate a user is active
activityEvents: 'mouseup',
// URL to ping the server using HTTP POST to extend the session
pingUrl: '/ping',
// URL used to log out when the user clicks a "Log out" button
logoutUrl: '/logout',
// URL used to log out when the session times out
timeoutUrl: '/logout?timeout=1',
ping: function() {
// Ping the server to extend the session expiration using a POST request.
$.ajax({
type: 'POST',
url: self.pingUrl
});
},
logout: function() {
// Go to the logout page.
window.location.href = self.logoutUrl;
},
onwarning: function() {
// Below is example code to demonstrate basic functionality. Use this to warn
// the user that the session will expire and allow the user to take action.
// Override this method to customize the warning.
var warningMinutes = Math.round(self.timeBeforeWarning / 60 / 1000),
$alert = $('<div id="jqsm-warning">Your session will expire in ' + warningMinutes + ' minutes. ' +
'<button id="jqsm-stay-logged-in">Stay Logged In</button>' +
'<button id="jqsm-log-out">Log Out</button>' +
'</div>');
if (!$('body').children('div#jqsm-warning').length) {
$('body').prepend($alert);
}
$('div#jqsm-warning').show();
$('button#jqsm-stay-logged-in').on('click', self.extendsess)
.on('click', function() { $alert.hide(); });
$('button#jqsm-log-out').on('click', self.logout);
},
onbeforetimeout: function() {
// By default this does nothing. Override this method to perform actions
// (such as saving draft data) before the user is automatically logged out.
// This may optionally return a jQuery Deferred object, in which case
// ontimeout will be executed when the deferred is resolved or rejected.
},
ontimeout: function() {
// Go to the timeout page.
window.location.href = self.timeoutUrl;
}
},
self = {},
_warningTimeoutID,
_expirationTimeoutID,
// The time of the last ping to the server.
_lastPingTime = 0;
function extendsess() {
// Extend the session expiration. Ping the server and reset the timers if
// the minimum interval has passed since the last ping.
var now = $.now(),
timeSinceLastPing = now - _lastPingTime;
if (timeSinceLastPing > self.minPingInterval) {
_lastPingTime = now;
_resetTimers();
self.ping();
}
}
function _resetTimers() {
// Reset the session warning and session expiration timers.
var warningTimeout = self.sessionLifetime - self.timeBeforeWarning;
window.clearTimeout(_warningTimeoutID);
window.clearTimeout(_expirationTimeoutID);
_warningTimeoutID = window.setTimeout(self.onwarning, warningTimeout);
_expirationTimeoutID = window.setTimeout(_onTimeout, self.sessionLifetime);
}
function _onTimeout() {
// A wrapper that calls onbeforetimeout and ontimeout and supports asynchronous code.
$.when(self.onbeforetimeout()).always(self.ontimeout);
}
// Add default variables and methods, user specified options, and non-overridable
// public methods to the session monitor instance.
$.extend(self, defaults, options, {
extendsess: extendsess
});
// Set an event handler to extend the session upon user activity (e.g. mouseup).
$(document).on(self.activityEvents, extendsess);
// Start the timers and ping the server to ensure they are in sync with the backend session expiration.
extendsess();
return self;
};
</pre>
<p>Here is the important HTML:</p>
<pre class="html">
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Login</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">
</head>
<body>
<div id="session-warning-modal" class="modal">
<div class="modal-dialog">
<div class="modal-content">
<div class="modal-header">
<h4 class="modal-title" id="sessWarnLabel">Your session is about to expire</h4>
</div>
<div class="modal-body">
Your session will expire in <span id="remaining-time"></span> minutes due to inactivity.
</div>
<div class="modal-footer">
<button id="log-out" class="btn btn-default" type="button" data-dismiss="modal">Log Out</button>
<button id="stay-logged-in" class="btn btn-warning" type="button" data-dismiss="modal">Stay Logged In</button>
</div>
</div>
</div>
</div>
<script src="//ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>
<script src="{{ url_for('static', filename='session-monitor.js')}}"></script>
<script type="text/javascript">
// Configure and start the session timeout monitor
sessMon = sessionMonitor({
// Subtract 1 minute to ensure the backend doesn't expire the session first
sessionLifetime: {{ PERMANENT_SESSION_LIFETIME_MS }} - (1 * 60 * 1000),
timeBeforeWarning: 10 * 60 * 1000, // 10 minutes
minPingInterval: 1 * 60 * 1000, // 1 minute
pingUrl: '/ping',
logoutUrl: '/logout',
timeoutUrl: '/logged-out?timeout=1&next=' + encodeURIComponent(
window.location.pathname + window.location.search + window.location.hash),
// The "mouseup" event was used instead of "click" because some of the
// inner elements on some pages have click event handlers that stop propagation.
activityEvents: 'mouseup',
onwarning: function() {
$("#session-warning-modal").modal("show");
}
});
$(document).ready( function() {
// Configure the session timeout warning modal
$("#session-warning-modal")
.modal({
"backdrop": "static",
"keyboard": false,
"show": false
})
.on("click", "#stay-logged-in", sessMon.extendsess)
.on("click", "#log-out", sessMon.logout)
.find("#remaining-time").text(Math.round(sessMon.timeBeforeWarning / 60 / 1000));
});
window.sessMon = sessMon;
</script>
</body>
</html>
</pre>
<p>Here is the Python Flask app (myapp.py):</p>
<pre class="python">import collections
import datetime
from flask import Flask, request, render_template, redirect, url_for, session
from flask.ext.login import (
LoginManager, login_user, logout_user, UserMixin, login_required)
from wtforms.fields import PasswordField, StringField
from wtforms.form import Form
UserRow = collections.namedtuple('UserRow', ['id', 'password'])
TOY_USER_DATABASE = {
'george': UserRow(id=1, password='george'),
}
# settings ###############################################################
# Set a secret key to sign the session (Flask config value)
SECRET_KEY = 'insert secret key here'
# The amount of time after which the user's session expires
# (this is a Flask setting and is also used by the JavaScript)
PERMANENT_SESSION_LIFETIME = datetime.timedelta(minutes=60)
# init ###############################################################
app = Flask(__name__)
app.config.from_object(__name__)
login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = '.login'
@login_manager.user_loader
def load_user(userid):
return User(userid)
@app.context_processor
def add_session_config():
"""Add current_app.permanent_session_lifetime converted to milliseconds
to context. The config variable PERMANENT_SESSION_LIFETIME is not
used because it could be either a timedelta object or an integer
representing seconds.
"""
return {
'PERMANENT_SESSION_LIFETIME_MS': (
app.permanent_session_lifetime.seconds * 1000),
}
# models ###############################################################
class User(UserMixin):
def __init__(self, id):
self.id = id
# forms ###############################################################
class LoginForm(Form):
username = StringField()
password = PasswordField()
# views ###############################################################
@app.route("/login", methods=['GET', 'POST'])
def login():
form = LoginForm(request.form)
message = ''
if request.method == 'POST' and form.validate():
db_user = TOY_USER_DATABASE.get(form.username.data)
if form.password.data == db_user.password:
user = User(db_user.id)
login_user(user)
return redirect(url_for('.home'))
else:
message = 'Login failed.'
context = {
'form': form,
'message': message,
}
return render_template('login.html', **context)
@app.route("/")
@login_required
def home():
return render_template('home.html')
@app.route("/another-page")
@login_required
def another_page():
return render_template('another_page.html')
@app.route("/logout")
@login_required
def logout():
logout_user()
return redirect(url_for('.logged_out') + '?' + request.query_string)
@app.route("/logged-out")
def logged_out():
timed_out = request.args.get('timeout')
return render_template('logged_out.html', timed_out=timed_out)
@app.route("/ping", methods=['POST'])
def ping():
session.modified = True
return 'OK'
if __name__ == "__main__":
app.run(debug=True)
</pre>
<p>The full example is also on github at:
<a href="https://github.com/saltycrane/session-timeout-example">https://github.com/saltycrane/session-timeout-example</a>.
</p>
<p>Initial ideas were taken from
<a href="http://www.itworld.com/development/335546/how-create-session-timeout-warning-your-web-application-using-jquery">
http://www.itworld.com/development/335546/how-create-session-timeout-warning-your-web-application-using-jquery</a>
but, from what I could tell, it pinged the server whether the user was active or not.
</p>
Subdomain-based configuration for a Flask local development server
2014-02-07T15:09:16-08:00https://www.saltycrane.com/blog/2014/02/subdomain-based-configuration-flask-local-development-server/<p>
This example shows how to set up a Flask local development server
to use a different configuration based on the subdomain
of the request. The project I work on has several environments
(dev, qa, staging, etc). Each environment has different
database and API hostnames. I use this to switch
between database and API environments quickly while
using my local development server.
</p>
<p>
This assumes a <code>create_app</code> function is used to
create the Flask application instance as described in
the <a href="http://flask.pocoo.org/docs/patterns/appfactories/#app-factories">
Application Factories Flask documentation</a>.
</p>
<h4 id="create_app">create_app</h4>
<p>
Modify the <code>create_app</code> function to take a <code>configobj</code>
argument and use it to override the default configuration.
</p>
<pre class="python">
def create_app(configobj=None):
app = Flask(__name__)
# Default configuration
app.config.from_object(__name__)
# Override configuration using config passed into create_app
if configobj:
app.config.from_object(configobj)
return app
</pre>
<h4 id="SubdomainDispatcher">SubdomainDispatcher</h4>
<p>
The SubdomainDispatcher is taken from the
<a href="http://flask.pocoo.org/docs/patterns/appdispatch/#dispatch-by-subdomain">
Application Dispatching Flask documentation</a>.
It is <a href="http://www.python.org/dev/peps/pep-0333/#middleware-components-that-play-both-sides">
WSGI middleware</a> that
looks at the subdomain of the request and returns a different
application instance for each subdomain. It calls the
<code>create_app</code> function above and passes it
the appropriate configuration object for the subdomain.
</p>
<pre class="python">class SubdomainDispatcher(object):
def __init__(self, create_app, domain=''):
"""
:param create_app: a function that returns a `flask.Flask` instance
:param domain: str - used to determine the subdomain
"""
self.create_app = create_app
self.domain = domain
self.lock = Lock()
self.instances = {}
def __call__(self, environ, start_response):
app = self._get_application(environ['HTTP_HOST'])
return app(environ, start_response)
def _get_application(self, host):
host = host.split(':')[0]
assert host.endswith(self.domain), 'Configuration error'
subdomain = host[:-len(self.domain)].rstrip('.')
with self.lock:
app = self.instances.get(subdomain)
if app is None:
configobj = self._get_subdomain_based_config(subdomain)
app = self.create_app(configobj=configobj)
self.instances[subdomain] = app
return app
@staticmethod
def _get_subdomain_based_config(subdomain):
class Config(object):
pass
config = Config()
if subdomain == 'dev':
config.API_HOST = 'dev-host'
config.DB_SERVER = 'dev-db-server'
elif subdomain == 'qa':
config.API_HOST = 'qa-host'
config.DB_SERVER = 'qa-db-server'
return config
</pre>
<h4 id="rundevserver">rundevserver</h4>
<p>
<code>rundevserver</code> is similar to
<a href="https://github.com/mitsuhiko/flask/blob/0.10.1/flask/app.py#L722-L777">
<code>flask.Flask.run</code></a>
but uses the <code>SubdomainDispatcher</code> middleware
before calling <code>werkzeug.serving.run_simple</code>.
</p>
<pre class="python">def rundevserver(host=None, port=None, domain='', debug=True, **options):
"""
Modified from `flask.Flask.run`
Runs the application on a local development server.
:param host: the hostname to listen on. Set this to ``'0.0.0.0'`` to
have the server available externally as well. Defaults to
``'127.0.0.1'``.
:param port: the port of the webserver. Defaults to ``5000``
:param domain: used to determine the subdomain
:param debug: if given, enable or disable debug mode.
See :attr:`debug`.
:param options: the options to be forwarded to the underlying
Werkzeug server. See
:func:`werkzeug.serving.run_simple` for more
information.
"""
from werkzeug.serving import run_simple
if host is None:
host = '127.0.0.1'
if port is None:
port = 5000
options.setdefault('use_reloader', debug)
options.setdefault('use_debugger', debug)
app = SubdomainDispatcher(create_app, domain, debug=debug)
run_simple(host, port, app, **options)
</pre>
<h4 id="usage">Usage</h4>
<ol>
<li>Add the following to your hosts file (/etc/hosts on Ubuntu):
<pre>0.0.0.0 dev.localhost
0.0.0.0 qa.localhost</pre>
</li>
<li>Run the local dev server:
<pre class="python">if __name__ == '__main__':
rundevserver(host='0.0.0.0', port=5000, domain='localhost')</pre>
</li>
<li>Use the following URLs to get different app configurations:
<ul>
<li><a href="http://localhost:5000">http://localhost:5000</a> (default configuration)</li>
<li><a href="http://dev.localhost:5000">http://dev.localhost:5000</a></li>
<li><a href="http://qa.localhost:5000">http://qa.localhost:5000</a></li>
</ul>
</li>
</ol>
<h4 id="github-code">Github code</h4>
<p>
A full working example is located on github:
<del><a href="https://github.com/saltycrane/flask-subdomain-dispatcher-example">
flask-subdomain-dispatcher-example</a></del> An updated version of the code is here:
<a href="https://github.com/saltycrane/flask-subdomaindevserver">flask-subdomaindevserver</a>.
</p>
How to add a margin around markers in the Google Static Maps API using Python
2013-11-26T07:31:21-08:00https://www.saltycrane.com/blog/2013/11/how-to-add-margin-around-markers-google-static-maps-api-using-python/<p>
This example shows how to use Python to generate a
<a href="https://developers.google.com/maps/documentation/staticmaps/index">
Google Static Map</a> URL for a map that contains markers within
some dimensions which are smaller than the map image dimensions. This
effectively allows for setting minimum X and Y margins around the markers
in a map. This is useful for a "fluid" web design where a maximum map
size is requested from Google and is then cut off at the edges for
small browser windows.
</p>
<p>
The bulk of this solution is based on the Javascript code here:
<a href="http://stackoverflow.com/questions/6048975/google-maps-v3-how-to-calculate-the-zoom-level-for-a-given-bounds">
http://stackoverflow.com/questions/6048975/google-maps-v3-how-to-calculate-the-zoom-level-for-a-given-bounds
</a>
</p>
<pre class="python">import math
def generate_map_url(
min_map_width_px,
max_map_width_px,
min_map_height_px,
max_map_height_px,
marker_groups):
"""
Return a Google Static Map URL for a map that contains markers within
some dimensions which are smaller than the map image dimensions. This
effectively allows for setting minimum X and Y margins around the markers
in a map. This is useful for a "fluid" web design where a maximum map
size is requested from Google and is then cut off at the edges for
small browser windows.
"""
# Determine the maximum zoom to contain markers at the minimum map size
lat_list = [
lat for markers in marker_groups for lat, lng in markers['lat_lng']]
lng_list = [
lng for markers in marker_groups for lat, lng in markers['lat_lng']]
max_zoom = get_zoom_to_fit(
min(lat_list), max(lat_list), min(lng_list), max(lng_list),
min_map_width_px, min_map_height_px,
)
# Build the markers query string arguments
markers_args = ''
for markers in marker_groups:
lat_lng = '|'.join([
'{},{}'.format(lat, lng) for lat, lng in markers['lat_lng']])
markers_args += '&markers=color:{}|{}'.format(markers['color'], lat_lng)
# Build and return the map URL
return ''.join([
'http://maps.googleapis.com/maps/api/staticmap',
'?sensor=false&v=3&visual_refresh=true',
'&size={}x{}&zoom={}'.format(
max_map_width_px, max_map_height_px, max_zoom),
markers_args,
])
def get_zoom_to_fit(min_lat, max_lat, min_lng, max_lng, width_px, height_px):
"""
Return the maximum zoom that will fit the given min/max lat/lng
coordinates in a map of the given dimensions. This is used to
override the zoom set by Google's implicit positioning.
Calculation translated from Javascript to Python from:
http://stackoverflow.com/questions/6048975/google-maps-v3-how-to-calculate-the-zoom-level-for-a-given-bounds
"""
GOOGLE_WORLD_WIDTH = 256
GOOGLE_WORLD_HEIGHT = 256
MAX_ZOOM = 17
def lat2rad(lat):
sinlat = math.sin(math.radians(lat))
radx2 = math.log((1 + sinlat) / (1 - sinlat)) / 2.0
return max(min(radx2, math.pi), -math.pi) / 2.0
def zoom(map_px, world_px, fraction):
# Use int() to round down to the nearest integer
return int(
math.log(float(map_px) / float(world_px) / fraction)
/ math.log(2.0)
)
# Determine the maximum zoom based on height and latitude
if min_lat == max_lat:
lat_zoom = MAX_ZOOM
else:
lat_fraction = (lat2rad(max_lat) - lat2rad(min_lat)) / math.pi
lat_zoom = zoom(height_px, GOOGLE_WORLD_HEIGHT, lat_fraction)
# Determine the maximum zoom based on width and longitude
if min_lng == max_lng:
lng_zoom = MAX_ZOOM
else:
lng_range = max_lng - min_lng
if lng_range < 0:
lng_range += 360.0
lng_fraction = lng_range / 360.0
lng_zoom = zoom(width_px, GOOGLE_WORLD_WIDTH, lng_fraction)
return min(lat_zoom, lng_zoom, MAX_ZOOM)
</pre>
<p>Here is an example:<p>
<pre class="python">
map_url = generate_map_url(
min_map_width_px=240, max_map_width_px=380,
min_map_height_px=285, max_map_height_px=325,
marker_groups=[
{'color': 'blue',
'lat_lng': [(34.0993, -118.8394)]},
{'color': 'orange',
'lat_lng': [
(34.3997, -119.2002),
(34.5389, -118.4499),
(34.0983, -118.1285),
(33.5932, -117.9455),
(33.8322, -117.3958),
]}
]
)
print map_url
</pre>
<p>
Here is a map without the margin:
<a href="http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958">
http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958
</a>
</p>
<img src="http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958" />
<p>
Here is the result with the margin:
<a href="http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&zoom=7&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958">
http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&zoom=7&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958
</a>
</p>
<img src="http://maps.googleapis.com/maps/api/staticmap?sensor=false&v=3&visual_refresh=true&size=380x325&zoom=7&markers=color:blue|34.0993,-118.8394&markers=color:orange|34.3997,-119.2002|34.5389,-118.4499|34.0983,-118.1285|33.5932,-117.9455|33.8322,-117.3958" />
How to expose a Flask local development server to the public using SSH remote port forwarding
2013-02-12T18:50:36-08:00https://www.saltycrane.com/blog/2013/02/how-expose-flask-local-development-server-public-using-ssh-remote-port-forwarding/<p>Here is how to run a Flask local development server on your local machine and expose it to the public via a remote server you have control over. This uses SSH remote port forwarding which is a converse of local port forwarding described here: <a href="/blog/2012/10/how-run-django-local-development-server-remote-machine-and-access-it-your-browser-your-local-machine-using-ssh-port-forwarding/">How to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding</a></p>
<ol>
<li>On the remote host, edit the sshd_config file (mine was located at /etc/ssh/sshd_config)
to allow remote hosts to connect to ports forwarded for the client:
<pre>GatewayPorts yes</pre>
</li>
<li>On the remote host, restart the SSH server:
<pre class="console">$ sudo service sshd restart </pre>
</li>
<li>On the local host, SSH to the remote host:
<pre class="console">$ ssh -v -R 50051:localhost:5000 eliot@my.remotehost.com </pre>
</li>
<li>On the local host, run the Flask dev server:
<pre class="console">$ python runserver.py localhost 5000 </pre>
</li>
<li>Go to <a href="http://my.remotehost.com:50051">http://my.remotehost.com:50051</a> in the browser</li>
</ol>
<h4>Using RemoteForward in your ~/.ssh/config</h4>
<p>You can also achieve the same results by using the <code>RemoteForward</code> in your <code>~/.ssh/config</code> file:</p>
<pre>Host myremote
User eliot
HostName my.remotehost.com
RemoteForward 50051 localhost:5000</pre>
<h4>References</h4>
<ul>
<li><a href="http://www.hackinglinuxexposed.com/articles/20030309.html">
http://www.hackinglinuxexposed.com/articles/20030309.html</a></li>
<li><a href="http://serverfault.com/questions/285616/how-to-allow-remote-connections-from-non-localhost-clients-with-ssh-remote-port">
http://serverfault.com/questions/285616/how-to-allow-remote-connections-from-non-localhost-clients-with-ssh-remote-port</a></li>
<li><a href="http://linux.die.net/man/5/sshd_config">
http://linux.die.net/man/5/sshd_config</a></li>
</ul>
<h4>See also</h4>
<p><a href="http://progrium.com/localtunnel/">localtunnel</a> by Jeff Lindsay exposes your local development server without requiring a public remote server.</p>
How to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding
2012-10-23T18:11:59-07:00https://www.saltycrane.com/blog/2012/10/how-run-django-local-development-server-remote-machine-and-access-it-your-browser-your-local-machine-using-ssh-port-forwarding/<p>Here is how to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding. (This is useful if there is a firewall blocking access to the port of your Django local dev server (port 8000).</p>
<ol>
<li>On the local host, SSH to the remote host:
<pre class="console">$ ssh -v -L 9000:localhost:8000 eliot@my.remotehost.com </pre>
</li>
<li>On the remote host, run the Django dev server:
<pre class="console">eliot@my.remotehost.com:/path/to/my/django/project$ python manage.py runserver 0.0.0.0:8000 </pre>
</li>
<li>On the local host, go to <a href="http://localhost:9000">http://localhost:9000</a> in the browser</li>
</ol>
<p>Note: The local port and the remote port can be the same (i.e. you can use 8000 instead of 9000). I just made them different to show which port is which.</p>
<h4>Using LocalForward in your ~/.ssh/config</h4>
<p>You can also achieve the same results by using the <code>LocalForward</code> in your <code>~/.ssh/config</code> file:</p>
<pre>Host myremote
User eliot
HostName my.remotehost.com
LocalForward 9000 localhost:8000</pre>
<h4>Reference</h4>
<p><a href="http://magazine.redhat.com/2007/11/06/ssh-port-forwarding/">http://magazine.redhat.com/2007/11/06/ssh-port-forwarding/</a></p>
Testing HTTPS w/ Flask's development server using stunnel on Ubuntu
2012-10-17T17:30:11-07:00https://www.saltycrane.com/blog/2012/10/testing-https-w-flasks-development-server-using-stunnel-ubuntu/<p>Our website is served over HTTPS. To more easily test certain issues (e.g. mixed mode content warnings, or Mapquest SSL tile servers), I wanted to access my Flask local development server over HTTPS. These two articles describe how to do this using <a href="https://www.stunnel.org/index.html">stunnel</a>: <a href="http://www.ianlewis.org/en/testing-https-djangos-development-server">Testing HTTPS with Django's Development Server</a>, <a href="http://tech.xster.net/tips/django-development-server-with-https/">Django Development Server with HTTPS</a>. Using stunnel, you can hit pages on your Django/Flask local dev server over HTTPS instead of HTTP. Here is how I installed it on Ubuntu Precise 12.04:</p>
<ul>
<li>Install SSL development files
<pre class="console">$ sudo apt-get install libssl-dev </pre>
</li>
<li>Go to <a href="https://www.stunnel.org/downloads.html">https://www.stunnel.org/downloads.html</a> and download stunnel-4.54.tar.gz</li>
<li>Unpack, compile, install.
<pre>$ tar xvf stunnel-4.54.tar.gz
$ cd stunnel-4.54
$ ./configure --prefix=/home/saltycrane/lib/stunnel-4.54
$ make
$ make install </pre>
NOTE: the <code>make install</code> step asked me a number of questions and created a certificate file at <code>/home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem</code>. Accept all the defaults for the certificate information (accurate certificate information isn't needed for this application).</li>
<li>Create a stunnel configuration file, /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https:
<pre>pid =
cert = /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
debug = 7
foreground = yes
[https]
accept = 7000
connect = 5000</pre>
</li>
<li>Start stunnel:
<pre class="console">$ /home/saltycrane/lib/stunnel-4.54/bin/stunnel /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Clients allowed=500
2012.10.17 17:40:52 LOG5[12468:140357811214080]: stunnel 4.54 on x86_64-unknown-linux-gnu platform
2012.10.17 17:40:52 LOG5[12468:140357811214080]: Compiled/running with OpenSSL 1.0.1 14 Mar 2012
2012.10.17 17:40:52 LOG5[12468:140357811214080]: Threading:PTHREAD SSL:+ENGINE+OCSP Auth:none Sockets:POLL+IPv6
2012.10.17 17:40:52 LOG5[12468:140357811214080]: Reading configuration from file /home/saltycrane/lib/stunnel-4.54/etc/stunnel/dev_https
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Compression not enabled
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Snagged 64 random bytes from /home/saltycrane/.rnd
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Wrote 1024 new random bytes to /home/saltycrane/.rnd
2012.10.17 17:40:52 LOG7[12468:140357811214080]: PRNG seeded successfully
2012.10.17 17:40:52 LOG6[12468:140357811214080]: Initializing service [https]
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Certificate: /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Certificate loaded
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Key file: /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Private key loaded
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Using DH parameters from /home/saltycrane/lib/stunnel-4.54/etc/stunnel/stunnel.pem
2012.10.17 17:40:52 LOG7[12468:140357811214080]: DH initialized with 1024-bit key
2012.10.17 17:40:52 LOG7[12468:140357811214080]: ECDH initialized with curve prime256v1
2012.10.17 17:40:52 LOG7[12468:140357811214080]: SSL options set: 0x00000004
2012.10.17 17:40:52 LOG5[12468:140357811214080]: Configuration successful
2012.10.17 17:40:52 LOG7[12468:140357811214080]: Service [https] (FD=7) bound to 0.0.0.0:7000
2012.10.17 17:40:52 LOG7[12468:140357811214080]: No pid file being created</pre>
</li>
<li>Start the python dev server:
<pre>$ HTTPS=1 python bin/runserver.py 0.0.0.0 5000 </pre>
</li>
<li>Go to https://localhost:7000 in your browser</li>
</ul>
<h4>See also</h4>
<ul>
<li><a href="https://forums.aws.amazon.com/thread.jspa?messageID=310507">
https://forums.aws.amazon.com/thread.jspa?messageID=310507</a></li>
<li><a href="http://www.stunnel.org/faq.html">
http://www.stunnel.org/faq.html</a></li>
</ul>
Example POSTing binary data using pycurl
2012-08-01T15:51:43-07:00https://www.saltycrane.com/blog/2012/08/example-posting-binary-data-using-pycurl/<p>Since <a href="http://docs.python.org/2/library/urllib2.html">urllib2</a> doesn't easily support POSTing binary data, I used <a href="http://pycurl.sourceforge.net/">pycurl</a>. It's less fun to use. I think I remember reading that <a href="http://docs.python-requests.org/en/latest/">Requests</a> supports this, but someone said something about speed a while back. It may warrant a second look. I couldn't figure out if <a href="http://code.google.com/p/httplib2/">httplib2</a> supports POSTing binary data.</p>
<p>Here's how I would do it with curl:</p>
<pre class="console">$ curl -v --data-binary @blank-contact-photo.jpg -H 'Content-Type: image/jpeg' 'http://localhost:8000'
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1... connected
> POST / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:8000
> Accept: */*
> Content-Type: image/jpeg
> Content-Length: 1335
> Expect: 100-continue
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection #0</pre>
<p>Here's the headers received by my simulated server:</p>
<pre>
['User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\r\n',
'Host: localhost:8000\r\n',
'Accept: */*\r\n',
'Content-Type: image/jpeg\r\n',
'Content-Length: 1335\r\n',
'Expect: 100-continue\r\n']
</pre>
<p>Here's what I tried. Method 4 is what worked.</p>
<pre class="python">import StringIO
import os.path
import pycurl
def main():
"""
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
http://code.activestate.com/recipes/576422-python-http-post-binary-file-upload-with-pycurl/
http://pycurl.cvs.sourceforge.net/pycurl/pycurl/tests/test_post2.py?view=markup
"""
method = 4
filename = 'blank-contact-photo.jpg'
url = 'http://localhost:8000'
c = pycurl.Curl()
c.setopt(pycurl.VERBOSE, 1)
c.setopt(pycurl.URL, url)
fout = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, fout.write)
if method == 1:
c.setopt(pycurl.HTTPPOST, [
("file1",
(c.FORM_FILE, filename))])
c.setopt(pycurl.HTTPHEADER, ['Content-Type: image/jpeg'])
elif method == 2:
c.setopt(c.HTTPPOST, [
("uploadfieldname",
(c.FORM_FILE, filename,
c.FORM_CONTENTTYPE, "image/jpeg"))])
elif method == 3:
c.setopt(pycurl.UPLOAD, 1)
c.setopt(pycurl.READFUNCTION, open(filename, 'rb').read)
filesize = os.path.getsize(filename)
c.setopt(pycurl.INFILESIZE, filesize)
elif method == 4:
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.HTTPHEADER, [
'Content-Type: image/jpeg'])
filesize = os.path.getsize(filename)
c.setopt(pycurl.POSTFIELDSIZE, filesize)
fin = open(filename, 'rb')
c.setopt(pycurl.READFUNCTION, fin.read)
c.perform()
response_code = c.getinfo(pycurl.RESPONSE_CODE)
response_data = fout.getvalue()
print response_code
print response_data
c.close()
if __name__ == '__main__':
main()
</pre>
<p>Here's the code for my simulated server in case you're curious:</p>
<pre class="python">import BaseHTTPServer
import SocketServer
class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def do_POST(self):
# from debugtools import pvar; pvar('self')
# from debugtools import pvar; pvar('dir(self)')
# from debugtools import pvar; pvar('vars(self)')
# from debugtools import pvar; pvar('self.request')
# from debugtools import pvar; pvar('self.rfile')
# from debugtools import pvar; pvar('self.headers')
# from debugtools import pvar; pvar('self.headers.headers')
print self.headers.headers
return 'asdfasdf'
PORT = 8000
Handler = MyHandler
SocketServer.TCPServer.allow_reuse_address = True
httpd = SocketServer.TCPServer(("", PORT), Handler)
print "serving at port", PORT
try:
httpd.serve_forever()
except KeyboardInterrupt:
httpd.shutdown()</pre>
Setting the Expires header for S3 media using Python and boto
2012-02-11T00:12:44-08:00https://www.saltycrane.com/blog/2012/02/setting-expires-header-s3-media-using-python-and-boto/<h4 id="install">Install boto</h4>
<pre class="console">$ pip install boto
$ pip freeze |grep boto
boto==2.2.1 </pre>
<h4 id="script">Script</h4>
<p>This script sets the "Expires" header 25 years from the current date
for all the files starting with the prefix "mydirectory". Replace
the access key id, secret access key, and bucket.
</p>
<pre class="python">import mimetypes
from datetime import datetime, timedelta
from boto.s3.connection import S3Connection
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
BUCKET_NAME = 'mybucket'
PREFIX = 'mydirectory'
def main():
conn = S3Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
bucket = conn.get_bucket(BUCKET_NAME)
key_list = bucket.get_all_keys(prefix=PREFIX)
for key in key_list:
content_type, unused = mimetypes.guess_type(key.name)
if not content_type:
content_type = 'text/plain'
expires = datetime.utcnow() + timedelta(days=(25 * 365))
expires = expires.strftime("%a, %d %b %Y %H:%M:%S GMT")
metadata = {'Expires': expires, 'Content-Type': content_type}
print key.name, metadata
key.copy(BUCKET_NAME, key, metadata=metadata, preserve_acl=True)
if __name__ == '__main__':
main()</pre>
<h4 id="references">References</h4>
<ul>
<li><a href="http://groups.google.com/group/boto-users/browse_thread/thread/b072849f3f97735b/02dbedbe874dbd22?pli=1">
Add cache-control header for object in S3? - boto-users | Google Groups
</a></li>
<li><a href="http://boto.cloudhackers.com/en/latest/ref/s3.html">
boto S3 API reference
</a></li>
<li><a href="http://boto.cloudhackers.com/en/latest/s3_tut.html">
boto S3 Introduction
</a></li>
<li><a href="http://docs.python.org/library/mimetypes.html">
Python documentation — mimetypes
</a></li>
</ul>
How to list attributes of an EC2 instance with Python and boto
2010-03-08T12:00:23-08:00https://www.saltycrane.com/blog/2010/03/how-list-attributes-ec2-instance-python-and-boto/<p>Here's how to find out information about your Amazon
<a href="http://aws.amazon.com/ec2/">EC2</a> instances using the Python
<a href="http://code.google.com/p/boto/">boto</a> library.
</p>
<h4>Install boto</h4>
<ul>
<li><a href="http://www.saltycrane.com/blog/2010/02/how-install-pip-ubuntu/">
Install pip</a>
</li>
<li>Install boto
<pre>sudo pip install boto</pre>
</li>
</ul>
<h4>Example</h4>
<pre class="python">from pprint import pprint
from boto import ec2
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
ec2conn = ec2.connection.EC2Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
for i in instances:
pprint(i.__dict__)
break # remove this to list all instances</pre>
<p>Results:</p>
<pre>{'_in_monitoring_element': False,
'ami_launch_index': u'0',
'architecture': u'x86_64',
'block_device_mapping': {},
'connection': EC2Connection:ec2.amazonaws.com,
'dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'id': u'i-xxxxxxxx',
'image_id': u'ami-xxxxxxxx',
'instanceState': u'\n ',
'instance_class': None,
'instance_type': u'm1.large',
'ip_address': u'xxx.xxx.xxx.xxx',
'item': u'\n ',
'kernel': None,
'key_name': u'FARM-xxxx',
'launch_time': u'2009-10-27T17:10:22.000Z',
'monitored': False,
'monitoring': u'\n ',
'persistent': False,
'placement': u'us-east-1d',
'previous_state': None,
'private_dns_name': u'ip-10-xxx-xxx-xxx.ec2.internal',
'private_ip_address': u'10.xxx.xxx.xxx',
'product_codes': [],
'public_dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'ramdisk': None,
'reason': '',
'region': RegionInfo:us-east-1,
'requester_id': None,
'rootDeviceType': u'instance-store',
'root_device_name': None,
'shutdown_state': None,
'spot_instance_request_id': None,
'state': u'running',
'state_code': 16,
'subnet_id': None,
'vpc_id': None}</pre>
<h4>For more information</h4>
<ul>
<li><a href="http://boto.s3.amazonaws.com/ref/ec2.html#module-boto.ec2.instance">EC2
instance API docs</a></li>
<li><a href="http://code.google.com/p/boto/source/browse/trunk/boto/ec2/instance.py">boto.ec2.instance.py
source code</a></li>
</ul>
How to escape (percent-encode) a URL with Python
2008-10-29T16:44:01-07:00https://www.saltycrane.com/blog/2008/10/how-escape-percent-encode-url-python/<pre class="python">import urllib
print urllib.quote_plus("http://www.yahoo.com/")
print urllib.quote_plus("Kruder & Dorfmeister")</pre>
<p>Results:</p>
<pre>http%3A%2F%2Fwww.yahoo.com%2F
Kruder+%26+Dorfmeister</pre>
<p>It is easy to be drawn to the <code>urlencode</code> function in
the Python
<a href="http://docs.python.org/library/urllib.html#utility-functions">
<code>urllib</code> module documentation</a>. But for simple
escaping, only <code>quote_plus</code>, or
possibly <code>quote</code> is needed. I believe this is the
appropriate solution to
<a href="http://starnixhacks.blogspot.com/2008/08/python-urlencode-annoyance.html">
Python urlencode annoyance</a> and
<a href="http://www.oreillynet.com/pub/h/476">
O'Reilly's Amazon Hack #92</a>.
</p>
<p>For reference:
<a href="http://en.wikipedia.org/wiki/Percent-encoding">
Percent-encoding on Wikipedia</a>
</p>
Python urlparse example
2008-09-24T16:58:32-07:00https://www.saltycrane.com/blog/2008/09/python-urlparse-example/<p>Here is an example of how to parse a URL using Python's
<code>urlparse</code> module. See the <a href="http://docs.python.org/lib/module-urlparse.html">
urlparse module documentation</a> for more information.</p>
<pre class="python">from urlparse import urlparse
url = 'http://www.gurlge.com:80/path/file.html;params?a=1#fragment'
o = urlparse(url)
print o.scheme
print o.netloc
print o.hostname
print o.port
print o.path
print o.params
print o.query
print o.fragment
print o.username
print o.password</pre>
<p>Results:</p>
<pre>http
www.gurlge.com:80
www.gurlge.com
80
/path/file.html
params
a=1
fragment
None
None</pre>
On using Python, the Digg API, and simplejson
2008-08-19T11:05:50-07:00https://www.saltycrane.com/blog/2008/08/using-python-digg-api-and-simplejson/<p>Here are some quick notes on using the
<a href="http://apidoc.digg.com/">Digg API</a>
with a Python script. Note, there is a
<a href="http://apidoc.digg.com/Toolkits#Python">Python
toolkit for Digg</a> but I just used <code>urllib2</code>
and the Digg API endpoints for the sake of simplicity.</p>
<p>I wanted the output in
<a href="http://www.json.org/">JSON</a> format so I specified the
<a href="http://apidoc.digg.com/ResponseTypes#JSON">response
type as JSON</a>. To decode JSON directly to a Python
data structure, I used
<a href="http://undefined.org/python/#simplejson">simplejson</a>.
</p>
<p>Here is a simple example which returns the JSON output for
the Digg story
<a href="http://digg.com/apple/Dell_vs_Apple_This_Time_it_s_Personal">Dell
vs. Apple: This Time it's Personal</a>
which has a "clean title" of
<code>Dell_vs_Apple_This_Time_it_s_Personal</code>.
</p>
<pre class="python">#!/usr/bin/env python
import urllib2
APPKEY = 'http%3A%2F%2Fwww.example.com'
story_clean_title = 'Dell_vs_Apple_This_Time_it_s_Personal'
url = ''.join([
'http://services.digg.com',
'/story/%s' % story_clean_title,
'?appkey=%s' % APPKEY,
'&type=json',
])
json = urllib2.urlopen(url).read()
print json</pre>
<p>Results:</p>
<pre>{"timestamp":1219168025,"total":"1","offset":0,"stories":[{"id":"8038250","link":"http:\/\/www.businessweek.com\/magazine\/content\/08_34\/b4097022701166.htm?campaign_id=rss_daily","submit_date":1219047878,"diggs":763,"comments":198,"title":"Dell vs. Apple: This Time it's Personal","description":"Now Bucher is again squaring off against his former company. He's spearheading an ambitious plan at Dell (DELL) to break Apple's dominant hold on the digital entertainment market.","promote_date":1219095692,"status":"popular","media":"news","user":{"name":"msaleem","icon":"http:\/\/digg.com\/users\/msaleem\/l.png","registered":1126518985,"profileviews":136052,"fullname":"Muhammad Saleem"},"topic":{"name":"Apple","short_name":"apple"},"container":{"name":"Technology","short_name":"technology"},"thumbnail":{"originalwidth":370,"originalheight":245,"contentType":"image\/jpeg","src":"http:\/\/digg.com\/apple\/Dell_vs_Apple_This_Time_it_s_Personal\/t.jpg","width":80,"height":80},"href":"http:\/\/digg.com\/apple\/Dell_vs_Apple_This_Time_it_s_Personal"}],"count":1}</pre>
<br>
<p>Here is a slightly less simple example which returns the comments
for the same story above. It uses <code>simplejson</code> to
decode the Digg story JSON data and get the story ID which is
then used to get the comment data.
</p>
<pre class="python">#!/usr/bin/env python
import simplejson
import urllib2
from pprint import pprint
APPKEY = 'http%3A%2F%2Fwww.example.com'
def main():
story_clean_title = 'Dell_vs_Apple_This_Time_it_s_Personal'
# get story
json = get_json('/story/%s' % story_clean_title)
pydata = simplejson.loads(json)
story_id = pydata['stories'][0]['id']
# get comments
json = get_json('/story/%s/comments' % story_id)
pydata = simplejson.loads(json)
pprint(pydata)
def get_json(endpoint):
""" returns json data for requested digg endpoint
"""
url = ''.join([
'http://services.digg.com',
endpoint,
'?appkey=%s' % APPKEY,
'&type=json',
])
return urllib2.urlopen(url).read()
if __name__ == '__main__':
main()</pre>
<p>Results:</p>
<pre style="height: 300px; overflow:auto">{u'comments': [{u'content': u"For those who are having trouble understanding what this is about. This is not about creating a competing closed platform. This is about creating a standard platform for selling music online. Imagine iTunes but with the ability to add other music stores into it. So when you want to buy a song, you can browse this music store or that music store. And all will work with your mp3 player. Whether it is a basic one that mounts as an external usb drive, or one that is able to sync the songs up the way iTunes does it with the iPod. And we know it is gonna be good because the guy that is driving it worked at Apple, so he knows what quality is. I'm sure it will have the ability to import or export your music library to iTunes if you so choose. But the point is that it is trying to create a standard that any device maker can follow. And hopefully, it will have no drm. Otherwise it's as useless as iTunes with its drm.\n\nRemember the same thing happened when windows 3.1 came along and immediately sold as many as 10x the number of pc's compared to mac's at the time. Anytime there is an open standard, no matter how good the closed standard is, the open one wins because that means lower prices for consumers since any manufacturer can use it. Apple thrives on closed standards when it comes to selling their products because it enables them to lock their customers in. And before they know it, they're locked into it and can't get out of it without great expense.\n\nI know that the Apple fanboi's are gonna bury me for this, but I'll say it anyways. It was nice being on top. But you can't be on top forever.",
u'date': 1219163527,
u'down': 0,
u'id': 17963749,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17963749,
u'story': 8038250,
u'up': 1,
u'user': u'pyrates'},
{u'content': u'@thinkdifferent: I should of specified its a $1 cheaper when you buy the full album. Single tracks are the same price but albums are usually about $8.99. ',
u'date': 1219159148,
u'down': 0,
u'id': 17961525,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17961525,
u'story': 8038250,
u'up': 1,
u'user': u'mrgermy'},
{u'content': u'Zune XPS w/ Vista Ultimate combo deal.',
u'date': 1219146354,
u'down': 0,
u'id': 17956943,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17956943,
u'story': 8038250,
u'up': 1,
u'user': u'hurdboy'},
{u'content': u'Digg: "Dell vs. Apple: This Time it\'s Personal"\nBusiness Week: "Bucher says his quest to challenge Apple is all business and not personal."\n\nBurried as inacurate.',
u'date': 1219134278,
u'down': 0,
u'id': 17954900,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17954900,
u'story': 8038250,
u'up': 1,
u'user': u'KAMiKAZOW'},
{u'content': u'Dell is junk and Apple is overpriced',
u'date': 1219117388,
u'down': 0,
u'id': 17950148,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17950148,
u'story': 8038250,
u'up': 1,
u'user': u'DeuceDiggalow'},
{u'content': u'Dell, all you have to do to kick apple in the jewels is get your act together with Ubuntu.\n\nAmarok will do the rest.',
u'date': 1219112434,
u'down': 2,
u'id': 17948056,
u'level': 0,
u'replies': 4,
u'replyto': None,
u'root': 17948056,
u'story': 8038250,
u'up': 3,
u'user': u'ethana2'},
{u'content': u'Apple = Overpriced, under-featured, but pretty\nDell = priced right, full-featured, and awesome looking',
u'date': 1219105961,
u'down': 8,
u'id': 17945330,
u'level': 0,
u'replies': 2,
u'replyto': None,
u'root': 17945330,
u'story': 8038250,
u'up': 3,
u'user': u'freesf'},
{u'content': u'Go Dell! Best of luck to you :)',
u'date': 1219104145,
u'down': 0,
u'id': 17944493,
u'level': 0,
u'replies': 0,
u'replyto': None,
u'root': 17944493,
u'story': 8038250,
u'up': 0,
u'user': u'Sabre24q7'},
{u'content': u'Actually the Dell DJ wasnt that bad for its time. Obviously now its looks very dated and far from the best now, but at its time it was a decent player and had pretty good sound quality. It also had more features than the ipod does even today.\n\n Creative was the people who made it and it was a very basic player but it did what it was suppose to and had a lot of room on it for a cheap price. ',
u'date': 1219104085,
u'down': 2,
u'id': 17944474,
u'level': 0,
u'replies': 1,
u'replyto': None,
u'root': 17944474,
u'story': 8038250,
u'up': 1,
u'user': u'jsc315'},
{u'content': u"I'm not saying this will or won't succeed. I'm just pointing out something people seem to be missing.\n\nJust because Dell customers aren't pompous and loudmouthed doesn't mean they aren't loyal and happy customers. There are a LOT of Dell users that like the company and what the products they make.",
u'date': 1219102724,
u'down': 1,
u'id': 17943861,
u'level': 0,
u'replies': 1,
u'replyto': None,
u'root': 17943861,
u'story': 8038250,
u'up': 2,
u'user': u'Urkel'}],
u'count': 10,
u'offset': 0,
u'timestamp': 1219168299,
u'total': u'55'}</pre>
How to use gnip-python to retrieve activity from Twitter, Delicious, Digg, etc.
2008-08-01T16:09:38-07:00https://www.saltycrane.com/blog/2008/08/how-use-gnip-python-retrieve-activity-twitter-delicious-digg-etc/<ul>
<li>Create an account at <a href="http://www.gnipcentral.com/">
http://www.gnipcentral.com/</a>
</li>
<li>Download
<a href="http://github.com/gnip/gnip-python/tree/master">
gnip-python from github.com</a>.
</li>
<li>Unpack it:
<pre class="bash">$ tar -zxvf gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e.tar.gz</pre>
</li>
<li>Move it to your example directory:
<pre>$ mkdir ~/src/python/gnip-example
$ mv gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e/*.py ~/src/python/gnip-example</pre>
</li>
<li>Create an example file called
<code>~/src/python/gnip-example/gnip-example.py</code>:
<pre class="python">#!/usr/bin/env python
from gnip import *
gnip = Gnip("yourgniplogin@email.com", "yourpassword")
for publisher in ["twitter", "digg", "delicious"]:
activities = gnip.get_publisher_activities(publisher)
print
print publisher
for activity in activities[:5]:
print activity</pre>
</li>
<li>Run it:
<pre>$ python gnip-example.py</pre>
<br>
And get the following results:
<pre>
twitter
[derricklo, 2008-08-01T22:49:59+00:00, tweet, http://twitter.com/derricklo/statuses/875165550]
[sam_metal, 2008-08-01T22:50:01+00:00, tweet, http://twitter.com/sam_metal/statuses/875165564]
[lalatina, 2008-08-01T22:49:59+00:00, tweet, http://twitter.com/lalatina/statuses/875165544]
[Nochipra, 2008-08-01T22:50:01+00:00, tweet, http://twitter.com/Nochipra/statuses/875165562]
[jmcgaha, 2008-08-01T22:50:01+00:00, tweet, http://twitter.com/jmcgaha/statuses/875165556]
digg
[DAlexopoulos, 2008-08-01T18:50:08+00:00, submission, http://digg.com/health/CLA_Conjugated_Linoleic_Acid_Explained]
[EradicateIV, 2008-08-01T18:50:05+00:00, submission, http://digg.com/pc_games/Pittco_Iron_Storm_9]
[vivianpetman, 2008-08-01T18:49:58+00:00, submission, http://digg.com/business_finance/Dub_Me_Now_Turns_Business_Cards_into_Bits_and_Bytes_Small]
[portia7896, 2008-08-01T18:49:53+00:00, submission, http://digg.com/world_news/Radioactive_water_yum]
[hhdepot, 2008-08-01T18:49:52+00:00, submission, http://digg.com/2008_us_elections/Why_WE_are_democrats_in_2008_Original_Video]
delicious
[hanasama1, 2008-08-01T22:49:55+00:00, delicious, http://www.birchmere.com/]
[Chrmftcotl, 2008-08-01T22:49:54+00:00, delicious, http://www.shaunlow.com/a-definitive-stumbleupon-guide-driving-traffic-to-websites/]
[shankar, 2008-08-01T22:49:53+00:00, delicious, http://www.guardian.co.uk/books/2008/jul/26/salmanrushdie.bookerprize]
[grzyweasel, 2008-08-01T22:49:53+00:00, delicious, http://patterntap.com/]
[metaffect, 2008-08-01T22:50:05+00:00, delicious, http://dean.edwards.name/]</pre>
Pretty cool.
</li>
</ul>
<br>
<p>This is just scratching the surface of what you can do with gnip.
You can also filter by time or user. Or get XML output.
Or you can publish to gnip yourself. See the
<a href="http://github.com/gnip/gnip-python/tree/master">
gnip-python README</a> for more python examples or the
<a href="http://docs.google.com/View?docid=dgkhvp8s_3hhwdmdfb">
Gnip API</a> for more detailed information. Also,
<a href="http://groups.google.com/group/gnip-community/web/publishers">
here is a list of Gnip publishers</a>.
</p>
<h5>If you get a <code>ImportError: No module named iso8601</code>
error, install iso8601.
</h5>
<ul>
<li><a href="http://www.saltycrane.com/blog/2007/01/how-to-install-easy-install-for-python/">
Intall Python Easy Install</a>
</li>
<li>Install iso8601:
<pre>$ sudo easy_install -U iso8601</pre>
</li>
</ul>