SaltyCrane Blog — Notes on PythonJavascript and web development

Some ES6+ features used in React development

The newest versions of Javascript, ES2015 (ES6), ES2016 (ES7), and ES2017 and beyond have many features that can be used today via Babel. Here are a few features I've used in React development.

Arrow functions (ES2015)

Arrow functions provide a shorthand syntax for defining functions. They also do not bind a this context so this from the lexical scope is used instead. If no curly braces are used, the value after the arrow is returned. There are other subtleties (e.g. hoisting and naming) associated with function expressions vs. function declarations. I like using arrow functions in blog posts for brevity, but I haven't decided if I like them in all circumstances yet. More information: Arrow Functions - YDKJS: ES6 & Beyond, Arrow functions - Exploring ES6, and Arrow This | getiblog for an explanation about this.

Here is a stateless React component defined using an arrow function:

const App = () => (
  <div>
    <PageBehindModals />
    <ChainedModals modalList={[ModalName, ModalPhone]} />
  </div>
);

Without arrow functions, it could be written as:

function App() {
  return (
    <div>
      <PageBehindModals />
      <ChainedModals modalList={[ModalName, ModalPhone]} />
    </div>
  );
}

Destructuring (ES2015)

The shorthand destructuring shown assigns properties of an object to variables of the same name. There is also a longhand syntax that allows you to assign to variables of different names. Destructuring works with nested objects, with arrays, and can be used in variable declarations, function return values and function arguments. More information: Destructuring - YDKJS: ES6 & Beyond

Here is an example destructuring the objects this.props and this.state:

class ChainedModals extends Component {
  render() {
    const { modalList } = this.props;
    const { currIndex, showModal } = this.state;
    // ..
  }
}

Without destructuring, it could be written as:

class ChainedModals extends Component {
  render() {
    const modalList = this.props.modalList;
    const currIndex = this.state.currIndex;
    const showModal = this.state.showModal;
    // ..
  }
}

Destructuring function arguments (ES2015)

Destructuring can be applied to function arguments that are objects or arrays. More information: Destructuring Parameters - YDKJS: ES6 & Beyond

This function expects a single object as an argument and it is destructured into onClickNext and step.

function ModalName({ onClickNext, step }) {
  return (
    <div>
      <h1>Step {step} - Name</h1>
      <Button onClick={onClickNext}>Next</Button>
    </div>
  );
}

Without destructuring, it could be written as:

function ModalName(props) {
  var onClickNext = props.onClickNext;
  var step = props.step;

  return (
    <div>
      <h1>Step {step} - Name</h1>
      <Button onClick={onClickNext}>Next</Button>
    </div>
  );
}

Nested destructuring (ES2015)

Destructuring also applies to objects nested in objects. More information: Nested Destructuring - YDKJS: ES6 & Beyond

Here is an example destructuring the nested props object:

function setIndexFromRoute(props) {
  const { modalList, location: { pathname } } = props;
  // ..
}

Without destructuring, it could be written as:

function setIndexFromRoute(props) {
  const modalList = props.modalList;
  const pathname = props.location.pathname;
  // ..
}

Object rest/spread operator (ES2017)

The ... rest operator gathers the rest of the items in the props object argument and puts them in the variable rest. The ... in the JSX is actually JSX syntax for spreading the props in the the rest object into individual props. More information: Object Rest/Spread Properties ECMAScript proposal, Objects Properties and ... - YDKJS: ES6 & Beyond, Using Object Spread Operator - Redux documentation, and JSX Spread Attributes - React documentation.

The object rest/spread operator is currently at stage 2 in the approval process so the earliest release would be ES2017. Note there is a rest/spread operator for arrays in ES2015.

function ModalName({ onClick, ...rest }) {
  return (
    <Modal {...rest}>
      <Button onClick={onClick}>Next</Button>
    </Modal>
  );
}

If only onClick, show, and backdrop props are passed to ModalName, it could be written like this. Using the rest and spread operators are useful when the properties are variable or unknown.

function ModalName(props) {
  var onClick = props.onClick;
  var show = props.show;
  var backdrop = props.backdrop;

  return (
    <Modal show={show} backdrop={backdrop}>
      <Button onClick={onClick}>Next</Button>
    </Modal>
  );
}

See also

A chained modals example w/ React Router (part 2)

This is an example using React and React Router to create a sequence of chained modals. In part 1, I made a basic example where clicking a "Next" button advanced to the next modal in a list.

This example supports going back and forward using browser navigation and linking to a specific modal's URL. Additionally, each modal has a form input. On clicking the "Next" button, an AJAX request is made to save the data. On failure an error is displayed. On success the next modal is shown. During the request a spinner is shown. The full code is here and a demo is here.

App.js

RoutedApp is the top level component that configures the routes for the app. Each route has an associated component. Like the previous example, the modals are children of ChainedModals which is a child of App. However now each modal is explicitly declared in the element tree and React Router decides which modal to render based on the route. For example, navigating to the /name route renders ModalName as a child of ChainedModals as a child of App. In the App component, children is set to the ChainedModals element. In the ChainedModals component, children is set to either the ModalName or ModalPhone element depending on the route.

Instead of passing modal components to ChainedModals, a list of routes to the modals is passed in. The utility function partial allows me to create a component that is equivalent to ChainedModals with the modalList prop preset.

This uses ES'15 arrow functions, argument destructuring and JSX spread. [App.js on github]

const RoutedApp = () => (
  <Router history={hashHistory}>
    <Route component={App}>
      <Route path="/" component={
        partial(ChainedModals, {modalList: ['/name', '/phone', '/done']})}>
        <Route path="/name" component={ModalName} />
        <Route path="/phone" component={ModalPhone} />
        <IndexRedirect to="/name" />
      </Route>
      <Route path="/done" />
    </Route>
  </Router>
);

const App = ({ children }) => (
  <div>
    <PageBehindModals />
    {children}
  </div>
);

const partial = (Comp, props) => (fprops) => <Comp {...props} {...fprops} />;
ChainedModals.js

ChainedModals is a component that manages its child modal components. Unlike the previous example, the modal to be shown is determined by the current route. Before the component is rendered, the index of the current modal is determined by finding the current route in the modal list. This index is stored in the state. When the modal's "Next" button is clicked, _gotoNext determines the next route to be displayed and changes the route using hashHistory.push(). React.cloneElement is used to pass props to the child modal as shown in the React Router examples. This uses ES'15 nested destructuring and classes, and ES'17 class properties. [ChainedModals.js on github]

class ChainedModals extends Component {
  render() {
    const { children } = this.props;
    const { currIndex } = this.state;

    // Clone the child view element so we can pass props to it.
    const modalElement = children && React.cloneElement(children, {
      step: currIndex + 1,
      gotoNext: this._gotoNext,
      backdrop: false,
      show: true,
    });

    return (
      <div>
        <ModalBackdrop />
        {modalElement}
      </div>
    );
  }

  componentWillMount() {
    this._setIndexFromRoute(this.props);
  }

  componentWillReceiveProps(nextProps) {
    this._setIndexFromRoute(nextProps);
  }

  _setIndexFromRoute(props) {
    const { modalList, location: { pathname } } = props;
    const index = modalList.findIndex(path => path === pathname);
    this.setState({currIndex: index});
  }

  _gotoNext = () => {
    const { modalList } = this.props;
    const { currIndex } = this.state;
    const nextRoute = modalList[currIndex + 1];
    hashHistory.push(nextRoute);
  };
}
ModalName.js

ModalName is one of the modal components in the list. The _handleClickNext method makes a fake AJAX request using the request simulator function. request returns a Promise. On success, it calls gotoNext (which is passed in as a prop) to go to the next modal. The component state is used to show a spinner during the AJAX request (isRequesting) and to show validation errors (errorMsg). This uses ES'15 destructuring, classes, and promises, and ES'17 rest/spread and class properties. [ModalName.js on github]

class ModalName extends Component {
  state = {
    isRequesting: false,
    errorMsg: null
  };

  render() {
    const { step, ...props } = this.props;
    const { isRequesting, errorMsg } = this.state;

    return (
      <Modal {...props}>
        <Modal.Header closeButton>
          <Modal.Title>Step {step} - Name</Modal.Title>
        </Modal.Header>
        <Modal.Body>
          {isRequesting && <p><em>Making fake ajax request...</em></p>}
          {errorMsg && <p><em>{errorMsg}</em></p>}
          <Input
            label="Enter your name"
            type="text"
            bsSize="large"
            {...(errorMsg ? {bsStyle: 'error'} : {})}
            ref={(c) => this._input = c}
          />
        </Modal.Body>
        <Modal.Footer>
          <Button bsStyle="primary" onClick={this._handleClickNext}>Next</Button>
        </Modal.Footer>
      </Modal>
    );
  }

  _handleClickNext = () => {
    const { gotoNext } = this.props;
    const name = this._input.getValue();

    this.setState({isRequesting: true, errorMsg: null});
    request('/api/name', name)
      .then(() => {
        gotoNext();
      })
      .catch((error) => {
        this.setState({isRequesting: false, errorMsg: error});
      });
  };
}

A chained modals example w/ React Bootstrap

This is an example using React to create a sequence of chained modals. Each modal has a "Next" button that, when clicked, advances to the next modal. The list of modals is configurable at page load. I used React-Bootstrap for the modals which provide animation when switching modals. The full code is here and a demo is here.

App.js

The top level app is a function that returns an element tree consisting of two elements: PageBehindModals and ChainedModals. ChainedModals is a parent of the modals which are passed in via the modalList prop. This uses ES'15 arrow functions. [App.js on github]

const App = () => (
  <div>
    <PageBehindModals />
    <ChainedModals modalList={[ModalName, ModalPhone]} />
  </div>
);
ModalName.js

ModalName is one of the modal components. It is built using React-Bootstrap's Modal and Button components. The step number and onClick handler are passed in as props from the parent component. The rest of the props (show and backdrop) are passed along to React-Bootstrap's Modal component. This uses ES'15 argument destructuring and ES'17 rest/spread. [ModalName.js on github]

const ModalName = ({ onClickNext, step, ...rest }) => (
  <Modal {...rest}>
    <Modal.Header closeButton>
      <Modal.Title>Step {step} - Name</Modal.Title>
    </Modal.Header>
    <Modal.Body>
      <p>Enter your name</p>
    </Modal.Body>
    <Modal.Footer>
      <Button bsStyle="primary" onClick={onClickNext}>Next</Button>
    </Modal.Footer>
  </Modal>
);
ChainedModals.js

ChainedModals keeps track of the current modal displayed in its component state. It defines _handleClickNext that is passed to the child modal components. When the child modal's "Next" button is clicked, this method is run. The method updates currIndex in the state, which causes a re-render. On re-render, ChainedModals finds a new modal component and renders it instead of the previous one. When the end of the modal list is reached, the modal is hidden, and the underlying page is shown. This uses ES'15 destructuring, ES'15 classes, and ES'17 class properties. [ChainedModals.js on github]

class ChainedModals extends Component {
  state = {
    currIndex: 0,
    showModal: true
  };

  render() {
    const { modalList } = this.props;
    const { currIndex, showModal } = this.state;
    const ModalComponent = modalList[currIndex];

    return (
      <div>
        {showModal && <ModalBackdrop />}
        <ModalComponent
          step={currIndex + 1}
          onClickNext={this._handleClickNext}
          backdrop={false}
          show={showModal}
        />
      </div>
    );
  }

  _handleClickNext = () => {
    const { modalList } = this.props;
    const { currIndex } = this.state;

    if (currIndex < modalList.length - 1) {
      this.setState({currIndex: currIndex + 1});
    } else {
      this.setState({showModal: false});
    }
  };
}
ModalBackdrop.js

A separate ModalBackdrop component is used so that the default modal's backdrop doesn't flash in and out when showing and hiding the modals. Since there is a separate modal backdrop, the backdrop prop is set to false for each modal. [ModalBackdrop.js on github]

const ModalBackdrop = () => <div className="modal-backdrop in"></div>;

How to set up Babel 6 with React on Mac OS X (command line only)

Babel is a tool used to compile Javascript from ES6 to ES5. It is also the official way to compile React's JSX. Here is how to install and set up Babel 6 on OS X to run on the command line to compile React's JSX.

Directory structure

my-project
├── .babelrc
├── hello.babelized.js
├── hello.js
├── index.html
├── node_modules
└── package.json

Install Node.js

Installing Node.js provides npm, the Node package manager. npm is used to install Babel.

$ brew install node 
$ node --version
v5.5.0 

Create a package.json file

package.json is the configuration file for npm. The package.json file below specifies 3 packages to install. babel-cli is the Babel command line tool. babel-preset-es2015 and babel-preset-react are 2 packages that provide the plugins for Babel to transform ES6 and JSX respectively. The "scripts" section specifies a command to compile the hello.js file with Babel.

{
  "devDependencies": {
    "babel-cli": "6.5.1",
    "babel-preset-es2015": "6.5.0",
    "babel-preset-react": "6.5.0"
  },
  "scripts": {
    "build": "babel hello.js -o hello.babelized.js"
  }
}

Install Babel

npm install installs all the packages listed in package.json into a node_modules directory in the current directory. Packages can also be installed globally using the "-g" flag with npm install. Since this is a local install, the babel command is availiable as ./node_modules/.bin/babel.

$ npm install 
$ ./node_modules/.bin/babel --version 
6.5.1 (babel-core 6.5.1) 

Create a .babelrc file

The .babelrc file is used to configure Babel.

{
  "presets": ["react", "es2015"]
}

Create a index.html file

This uses the React libraries from Facebook's CCN and the compiled hello.babelized.js file. The React app lives in the "container" div.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Hello World</title>
  </head>
  <body>
    <div id="container"></div>
    <script src="https://fb.me/react-0.14.7.min.js"></script>
    <script src="https://fb.me/react-dom-0.14.7.min.js"></script>
    <script src="hello.babelized.js"></script>
  </body>
</html>

Create a hello.js file

Note: this hello.js does not use ES6.

var Hello = React.createClass({
  render: function() {
    return (
      <h1>Hello World</h1>
    );
  }
});

ReactDOM.render(
  <Hello />,
  document.getElementById('container')
);

Run Babel

$ npm run build 

This creates the hello.babelized.js file:

var Hello = React.createClass({
  displayName: 'Hello',

  render: function () {
    return React.createElement(
      'h1',
      null,
      'Hello World'
    );
  }
});

ReactDOM.render(React.createElement(Hello, null), document.getElementById('container'));

View in browser

$ open index.html 

See also

Switching from Emacs to Vim (actually Spacemacs)

I recently switched from Linux to OS X and Python to Javascript. To complete my fall from the Light, I've switched from Emacs to Vim. Actually I just switched to Evil Mode and Spacemacs. This is how I came to switch:

  • I discovered OS X uses many Emacs key bindings by default and I could set even more.
  • I went back to the default key bindings in Emacs to be consistent with OS X.1
  • I remapped Return to Control and started using both Control keys to help use the default Emacs key bindings. Using both Control keys felt amazing compared to just one...
  • ...until I began feeling Emacs Pinky since Return was slightly farther than Caps Lock.2
  • I tried remapping Spacebar to Control and this felt even more amazing...
  • ...until I tried to type a sentence at normal speed.
  • I decided I didn't want to buy a foot pedal.
  • I tried Spacemacs.
  • I set bash and Karabiner to Vim mode.3
  • I set Caps Lock to Escape and Control.
  • I started looking for Vim screencasts.4

Even after 3 months, I'm still working a lot slower, but I'm hoping it's a good investment. One thing I've noticed is that Vim seems to use a lot of number and symbol keys. I need to learn to touch type my numbers!

Spacemacs

spacemacs screenshot

Spacemacs is an Emacs starter kit5 (like Emacs Prelude) optimized for Vim key bindings. It provides the "best of both worlds" – the efficiency of Vim's modal editing and the extensibility of Emacs Lisp.

Spacemacs replaces many Emacs modifier combinations by setting a leader key to the Spacebar (hence the name spacemacs). To open a file, use SPC f f instead of C-x C-f. Spacemacs makes commands easily discoverable using which-key. Just press SPC to see a list of commands, press a key and see more commands.

Spacemacs has a good out-of-the-box configuration for Javascript and React development. It uses js2-mode, web-mode6 for JSX, flycheck w/ eslint, tern, and some things I haven't used.

Install Spacemacs

Here's how to install Spacemacs on OS X.

  • Install Emacs
    $ brew install emacs --cocoa 
    
  • Install Spacemacs
    $ mv ~/.emacs.d ~/.emacs.d.bak  # if you have an exisiting .emacs.d directory
    $ git clone https://github.com/syl20bnr/spacemacs ~/.emacs.d 
    
  • Start Emacs (in terminal mode). This will download and compile packages and ask if you want to use vim mode or emacs mode.
    $ emacs 
    

Start Emacs in client/server mode

  • Start the Emacs server
    $ emacs --daemon 
    
  • Start an Emacs client in the terminal
    $ emacsclient -t 
    
  • Start a graphical Emacs client
    $ emacsclient -c 
    

Spacemacs config

Spacemacs has its own configuration file located at ~/.spacemacs or ~/.spacemacs.d/init.el. For more information, see the configuration documentation. My personal Spacemacs configuration is on github.

Useful Spacemacs commands

SPC q q - quit
SPC w / - split window vertically
SPC w - - split window horizontally
SPC 1   - switch to window 1
SPC 2   - switch to window 2
SPC w c - delete current window
SPC TAB - switch to previous buffer
SPC b b - switch buffers
SPC f f - find a file
SPC f s - save a file (:w also works)
SPC p p - open project
SPC p h - find a file in current project
SPC b d - kill current buffer
SPC b M - move buffer to another window
SPC v   - enter expand-region mode

Useful Vim key bindings

movement
--------
0 - beginning of line
^ - beginning of non-whitespace
$ - end of line
9j - move down 9 lines
w - move forward by word
b - move backward by word
gg - first line
G - last line
C-u - up half page
C-d - down half page
f/ - move forward to first "/" character
t/ = move forward right before the first "/" character
; - repeat that command again
H - head of the screen
M - middle of the screen
L - last of the screen
} - move forward by paragraph or block
{ - move backwards by paragraph or block
* - search for word under the cursor
    n - search again forward
    N - search again backwards
# - search backwards for word under cursor
/ - search forward
? - search backward
% - find matching brace, paren, etc
ma - mark a line in a file with marker "a"
`a - after moving around, go back to the exact position of marker "a"
'a - after moving around, go back to line of marker "a"
:marks - view all the marks
'' - go to the last place you were
[{ - jump back to the "{" at the beginning of the current code block

editing
-------
x - delete char under cursor
X - delete char before cursor
A - add to end of line
I - insert at the beginning of the line
dd - delete line
D - delete from cursor to end of line
di' - delete text inside single quotes
yy - copy line
Y - copy from cursor to end of line
cc - change line
C - change from cursor to end of line
cit - change text inside html tag
ci' - change text inside single quotes
ci{ - change text inside curly brackets.
ci... - etc
p - paste after cursor
P = paste before cursor
o - add line below
O - add line above
. = repeat last comment
r - replace character
R - replace. (overwrite) (good for columns of text)
J - join line (cursor can be anywhere on line)

visual mode
-----------
v - visual char mode
V - visual line mode
C-v - block visual mode

  1. And to be consistent with other machines or programs using Emacs or Emacs key bindings. Mandatory Eclipse for a 4 day Hadoop training emphasized this need. [back]
  2. At this point, I probably could have learned to move my hand 2 cm to the right, but I have been curious about Vim for a while now. [back]
  3. Later I also installed Vimium. [back]
  4. I found some excellent Vim screencasts here: http://derekwyatt.org/vim/tutorials/. [back]
  5. Though some don't call it a starter kit. [back]
  6. Hat tip to web-mode's author/maintainer. I created an issue about JSX indentation and it was fixed in less than a day. [back]

Modules and import in ES6 for Python developers

Here's a comparison of Python and Javascript (ES6) imports. There are a few differences between the two:

  1. Javascript imports are static; Python's are dynamic.
  2. Javascript items must be explicitly exported. In Python, all items are available for import.
  3. Javascript has a concept of a default export. Python does not.

PythonES6 (ES 2015)
 
import mymodule
mymodule.myfunc()
mymodule.py:
def myfunc(): pass

import mymodule as myalias
myalias.myfunc()
mymodule.py:
def myfunc(): pass
Namespaced imports
import * as myalias from "./mymodule";
myalias.myfunc();
mymodule.js:
export function myfunc() {}
Note: this form covers both Python's import mymodule and import mymodule as myalias forms.
 
from mymodule import myvar, myfunc
print myvar
myfunc()
mymodule.py:
myvar = 42
def myfunc(): pass
Named imports
import { myvar, myfunc } from "./mymodule";
console.log(myvar);
myfunc();
mymodule.js:
export var myvar = 42;
// no semicolon for inline exports
// of functions and classes
export function myfunc() {}
Note: curly braces are required even if only importing a single item. This is not destructuring. It is syntax specific to modules. Destructuring on import is not supported in ES6.
  No equivalentDefault imports (preferred form)
import myalias from "./mymodule";
myalias();
mymodule.js:
export default function myfunc() {}

Note: this import syntax has no curly braces because export default is used instead of just export in mymodule.js. There can be only one default export per module. Using this syntax is the preferred form. Unlike the form with curly braces, you will always supply your own name for the imported item. It may or may not be the same as the original name of the exported item. You may also combine the default import syntax with the non-default syntax.

import mydefault, { myother } from "./mymodule";
mydefault();
myother();
mymodule.js:
export function myother() {}
export default function myfunc() {}
 
from mymodule import myfunc as myalias
myalias()
mymodule.py:
def myfunc(): pass
Renaming an import
import { myfunc as myalias } from "./mymodule";
myalias();
mymodule.js:
export function myfunc() {}
import * from mymodule
print myvar
myfunc()
mymodule.py:
myvar = 42
def myfunc(): pass
Note: this form is not recommended
No equivalent
 
from mydir.mymodule import myfunc
myfunc()
mydir/mymodule.py:
def myfunc(): pass
Note: mydir contains a __init__.py file and is a module (package)
Importing from a subdirectory
import { myfunc } from "mydir/mymodule";
myfunc();
mydir/mymodule.js:
export function myfunc() {}

Names vs. paths

Modules can be referenced by name or by path. Names are often used with external libraries. For example, below "react" is the name.

import React from "react";

Paths are often used with your project modules. Here the module is referenced by the path "./MyComponent". (The .js extension is implicit.)

import MyComponent from "./MyComponent";

When are curly braces needed?

Curly braces are needed when importing non-default exports from a module. If the item is exported with default, use the import syntax without curly braces.

// mymodule.js
export default Something;
import Something from "mymodule";

If the item is exported without default, you must import the item with curly braces.

// mymodule.js
export Something;
import { Something } from "mymodule";

References

You Don't Know JS: ES6 & Beyond

Exploring ES6

Switching to OS X and front end development

After 7 years, I've yielded to the Dark Side and switched from Ubuntu to OS X on my work laptop. I've also switched from Python and back end web development to Javascript and front end development. The former is mostly to support the latter.

Linux is rare1, especially among front end developers, and I want to make it easy to collaborate as I learn new things. I've had problems working with Photoshop files in GIMP and I couldn't run the iOS simulator. Issues with Linux device drivers don't help.

I'm choosing front end development because I want to code closer to the end user.2 In small part like Ian Bicking wrote last year, I feel unexcited about back end development and really excited about Javascript and front end development. I'm excited about ES 2015 and React and React Native and CSS transitions.3 I'm even coming around to Node.js. Javascript is uglier than Python, but it's getting better and there are things Python can't do that Javascript can.4 If only beauty mattered, maybe I'd use Scheme.5

I'm sure I will hate OS X at first, but hopefully it will be good in the long run. If anyone can recommend a tiling window manager like Qtile for OS X, please let me know.

(I will continue using Emacs because Emacs rocks! 6)


  1. I think I was the last person at my company running Linux.
  2. I've been trying to do front end work for years now, but I finally got a sensible chance to switch as my company is changing it's technology stack from Python to Ruby and Backbone/Angular to React.
  3. Update 2016-01-04: Here are even more exciting web technologies: Electron, progressive web apps, and WebAssembly.
  4. Update 2016-01-01: I found James Hague had similar thoughts on Python and Javascript.
  5. Speaking of functional languages and Javascript, Elm sounds pretty cool.
  6. Update 2016-01-26: Or will I?

An example using reduce() in Underscore.js

I never learned how to use reduce in Python since our BDFL recommended against it. But since Javascript doesn't have list comprehensions, I'm learning some functional constructs in Underscore.js.

Underscore's reduce() function can be used to reduce an array of things into a single thing. A common use is computing the sum of an array of numbers. Like map and filter, it provides an alternative to a for loop. Reduce is not limited to returning a single thing and can be used to combine the functionality of map and filter like a Python list comprehension. Learning Underscore's reduce also helped me understand MapReduce and reducers in Redux.

Here is an example that uses reduce to create a single Javascript object from an array of objects. An empty object, {}, is passed in to _.reduce() as the initial state. It is then extended with each item in the array and finally returned. I also did the same example using _.each() for comparison, (update) and using Array.prototype.reduce and ES6 w/o Underscore.

Example using Underscore's reduce

var myArr = [
    { rating: 5 },
    { price: 200 },
    { distance: 10 }
];

var myObj = _.reduce( myArr, function( memo, item ) {
    return _.extend( memo, item ); }, {} );
console.log( myObj );

Here is the output:

{ rating: 5, price: 200, distance: 10 }

Example using Underscore's each

Here is the same example using _.each:

var myArr = [
    { rating: 5 },
    { price: 200 },
    { distance: 10 }
];

var myObj = {};
_.each( myArr, function( item ) {
    myObj = _.extend( myObj, item ); });
console.log( myObj );

The output is the same:

{ rating: 5, price: 200, distance: 10 }

Example using ES6 and Array.prototype.reduce

Update 2015-12-11: I have been writing ES6 with React thanks to Babel so here is an ES6 version without Underscore.js. It looks very similar to the Underscore version. reduce() is now a method of the array and Object.assign() takes the place of _.extend(). ( Array.prototype.reduce is actually ES5, but Object.assign and const are ES6. )

const myArr = [
    { rating: 5 },
    { price: 200 },
    { distance: 10 }
];

const myObj = myArr.reduce( function( memo, item ) {
    return Object.assign( {}, memo, item ); }, {} );
console.log( myObj );

The output is the same:

{ rating: 5, price: 200, distance: 10 }

Logging the actual host (instead of localhost) with the Python elasticsearch client

The Elasticsearch Python library has a cool feature that allows you to log the equivalent curl command of the query you are making. To use it you just need to configure the elasticsearch.trace logger. There is one annoyance-- by design, "localhost:9200" is logged even if you have configured another host. It is more useful to log the actual host I am using so I can copy and paste the command to co-workers without changing the hostname. At first I thought I'd monkeypatch the library to do what I wanted. Then I remembered I could use a logging.Filter to rewrite the log message.

I am using the high level elasticsearch-dsl library in my example, but the logging is actually done by the low level official elasticsearch library which elasticsearch-dsl wraps.

$ pip install elasticsearch-dsl
import logging

import elasticsearch_dsl as esdsl  # high level elasticsearch library
from elasticsearch import Elasticsearch  # low level official elasticsearch library


ELASTICSEARCH_HOSTS = [
    'http://es1.myhost.com:9200',
    'http://es2.myhost.com:9200',
    'http://es3.myhost.com:9200',
]


class ReplaceHostFilter(logging.Filter):
    """
    Replace "localhost:9200" with real host
    """
    def __init__(self, real_host, *args, **kwargs):
        self.real_host = real_host
        super(ReplaceHostFilter, self).__init__(*args, **kwargs)

    def filter(self, record):
        record.msg = record.msg.replace('http://localhost:9200', self.real_host)
        record.msg = record.msg.replace('curl', 'curl -sS')
        record.msg += ' | python -mjson.tool'
        return True


# Set up logging
host_replacer = ReplaceHostFilter(ELASTICSEARCH_HOSTS[0])
formatter = logging.Formatter("%(levelname)s %(asctime)s %(name)s - %(message)s")
sh = logging.StreamHandler()
sh.setFormatter(formatter)
eslogger = logging.getLogger('elasticsearch.trace')
eslogger.setLevel('INFO')
eslogger.addFilter(host_replacer)
eslogger.addHandler(sh)


# Make a query
esclient = Elasticsearch(hosts=ELASTICSEARCH_HOSTS)
search = esdsl.Search(using=esclient, index='my_index')
filt = esdsl.F('term', myTerm=1234)
search = search.filter(filt)
response = search.execute()
print response.hits.total

Log output showing the actual host instead of localhost:

INFO 2015-07-28 13:48:54,464 elasticsearch.trace - curl -sS -XGET 'http://es1.myhost.com:9200/my_index/_search?pretty' -d '{
  "query": {
    "filtered": {
      "filter": {
        "term": {
          "myTerm": 1234
        }
      },
      "query": {
        "match_all": {}
      }
    }
  }
}' | python -mjson.tool

Calling Javascript from Python to de-CloudFlare scraped content

Yesterday I wrote a script to scrape my own web page because I screwed up the CSV export feature and Product needed the data. One problem was that the CloudFlare CDN obfuscated the email addresses on the page. My solutioncrazy hack: running a Node.js script to de-obfuscate the email from my Python scraping script.

Example obfuscated email stuff from CloudFlare:

<a href="/cdn-cgi/l/email-protection#d4b7b5a6b194a4b1a0e7e2e4fab7bbb9">
    <span class="__cf_email__" data-cfemail="0162607364417164753237312f626e6c">[email&#160;protected]</span>
    <script cf-hash='f9e31' type="text/javascript">
     /* <![CDATA[ */!function(){try{var t="currentScript"in document?document.currentScript:function(){for(var t=document.getElementsByTagName("script"),e=t.length;e--;)if(t[e].getAttribute("cf-hash"))return t[e]}();if(t&&t.previousSibling;){var e,r,n,i,c=t.previousSibling,a=c.getAttribute("data-cfemail");if(a){for(e="",r=parseInt(a.substr(0,2),16),n=2;a.length-n;n+=2)i=parseInt(a.substr(n,2),16)^r,e+=String.fromCharCode(i);e=document.createTextNode(e),c.parentNode.replaceChild(e,c)}}}catch(u){}}();/* ]]> */
    </script>
</a>

Using jsbeautifier.org, I adapted the Javascript from above into this Node.js script, decloudflare.js.

var e, r, n, i, a = process.argv[2];
for (e = "", r = parseInt(a.substr(0, 2), 16), n = 2; a.length - n; n += 2) i = parseInt(a.substr(n, 2), 16) ^ r, e += String.fromCharCode(i);
console.log(e);

Example usage:

$ node decloudflare.js 0162607364417164753237312f626e6c
care@pet360.com

I used the Naked library (thanks to Sweetmeat) to call the Node.js script. (Though probably I could've just used the subprocess module.)

$ pip install Naked 
from Naked.toolshed.shell import muterun_js

def decloudflare_email(cfemail):
    resp = muterun_js('decloudflare.js', cfemail)
    return resp.stdout.rstrip()

cfemail = '0162607364417164753237312f626e6c'
print 'cfemail from python: ' + cfemail
email = decloudflare_email(cfemail)
print 'email from python: ' + email
cfemail from python: 0162607364417164753237312f626e6c
email from python: care@pet360.com