Basically, this article by USA Today sums it up.
Both serve the same purpose and "have the force of law on the executive branch, and sometimes they seem to be used interchangeably".
Some of the differences include:
Executive orders are numbered, however, presidential memoranda are not. This also makes executive orders easy to keep track and be counted.
Executive orders are required by law to be published in the Federal Register. However, it's optional for presidential memoranda.
Executive orders have a higher order of precedence. The order of precedence follows this: proclamations, executive orders, presidential memoranda, presidential notices, and presidential determinations.
There are also other subtle differences, the article by USA Today explains and list most of them.
This article by the CNN also described how they differ.