Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.1 plus MathML 2.0 plus SVG 1.1//EN"
"http://www.w3.org/2002/04/xhtml-math-svg/xhtml-math-svg.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<script type="text/javascript" src="docs.js"></script>
<link rel="stylesheet" type="text/css" href="docs.css"/>
<title>Venus Normalization</title>
</head>
<body>
<h2>Normalization</h2>
<p>Venus builds on, and extends, the <a
href="http://www.feedparser.org/">Universal Feed Parser</a> and <a
href="http://code.google.com/p/html5lib/">html5lib</a> to
convert all feeds into Atom 1.0, with well formed XHTML, and encoded as UTF-8,
meaning that you don't have to worry about funky feeds, tag soup, or character
encoding.</p>
<h3>Encoding</h3>
<p>Input data in feeds may be encoded in a variety of formats, most commonly
ASCII, ISO-8859-1, WIN-1252, AND UTF-8. Additionally, many feeds make use of
the wide range of
<a href="http://www.w3.org/TR/html401/sgml/entities.html">character entity
references</a> provided by HTML. Each is converted to UTF-8, an encoding
which is a proper superset of ASCII, supports the entire range of Unicode
characters, and is one of
<a href="http://www.w3.org/TR/2006/REC-xml-20060816/#charsets">only two</a>
encodings required to be supported by all conformant XML processors.</p>
<p>Encoding problems are one of the more common feed errors, and every
attempt is made to correct common errors, such as the inclusion of
the so-called
<a href="http://www.fourmilab.ch/webtools/demoroniser/">moronic</a> versions
of smart-quotes. In rare cases where individual characters can not be
converted to valid UTF-8 or into
<a href="http://www.w3.org/TR/xml/#charsets">characters allowed in XML 1.0
documents</a>, such characters will be replaced with the Unicode
<a href="http://www.fileformat.info/info/unicode/char/fffd/index.htm">Replacement character</a>, with a title that describes the original character whenever possible.</p>
<p>In order to support the widest range of inputs, use of Python 2.3 or later,
as well as the installation of the python <code>iconvcodec</code>, is
recommended.</p>
<h3>HTML</h3>
<p>A number of different normalizations of HTML are performed. For starters,
the HTML is
<a href="http://www.feedparser.org/docs/html-sanitization.html">sanitized</a>,
meaning that HTML tags and attributes that could introduce javascript or
other security risks are removed.</p>
<p>Then,
<a href="http://www.feedparser.org/docs/resolving-relative-links.html">relative
links are resolved</a> within the HTML. This is also done for links
in other areas in the feed too.</p>
<p>Finally, unmatched tags are closed. This is done with a
<a href="http://code.google.com/p/html5lib/">knowledge of the semantics of HTML</a>. Additionally, a
<a href="http://golem.ph.utexas.edu/~distler/blog/archives/000165.html#sanitizespec">large
subset of MathML</a>, as well as a
<a href="http://www.w3.org/TR/SVGMobile/">tiny profile of SVG</a>
is also supported.</p>
<h3>Atom 1.0</h3>
<p>The Universal Feed Parser also
<a href="http://www.feedparser.org/docs/content-normalization.html">normalizes the content of feeds</a>. This involves a
<a href="http://www.feedparser.org/docs/reference.html">large number of elements</a>; the best place to start is to look at
<a href="http://www.feedparser.org/docs/annotated-examples.html">annotated examples</a>. Among other things a wide variety of
<a href="http://www.feedparser.org/docs/date-parsing.html">date formats</a>
are converted into
<a href="http://www.ietf.org/rfc/rfc3339.txt">RFC 3339</a> formatted dates.</p>
<p>If no <a href="http://www.feedparser.org/docs/reference-entry-id.html">ids</a> are found in entries, attempts are made to synthesize one using (in order):</p>
<ul>
<li><a href="http://www.feedparser.org/docs/reference-entry-link.html">link</a></li>
<li><a href="http://www.feedparser.org/docs/reference-entry-title.html">title</a></li>
<li><a href="http://www.feedparser.org/docs/reference-entry-summary.html">summary</a></li>
<li><a href="http://www.feedparser.org/docs/reference-entry-content.html">content</a></li>
</ul>
<p>If no <a href="http://www.feedparser.org/docs/reference-feed-
updated.html">updated</a> dates are found in an entry, the updated date from
the feed is used. If no updated date is found in either the feed or
the entry, the current time is substituted.</p>
<h3 id="overrides">Overrides</h3>
<p>All of the above describes what Venus does automatically, either directly
or through its dependencies. There are a number of errors which can not
be corrected automatically, and for these, there are configuration parameters
that can be used to help.</p>
<ul>
<li><code>ignore_in_feed</code> allows you to list any number of elements
or attributes which are to be ignored in feeds. This is often handy in the
case of feeds where the <code>author</code>, <code>id</code>,
<code>updated</code> or <code>xml:lang</code> values can't be trusted.</li>
<li><code>title_type</code>, <code>summary_type</code>,
<code>content_type</code> allow you to override the
<a href="http://www.feedparser.org/docs/reference-entry-title_detail.html#reference.entry.title_detail.type"><code>type</code></a>
attributes on these elements.</li>
<li><code>name_type</code> does something similar for
<a href="http://www.feedparser.org/docs/reference-entry-author_detail.html#reference.entry.author_detail.name">author names</a></li>
<li><code>future_dates</code> allows you to specify how to deal with dates which are in the future.
<ul style="margin:0">
<li><code>ignore_date</code> will cause the date to be ignored (and will therefore default to the time the entry was first seen) until the feed is updated and the time indicated is past, at which point the entry will be updated with the new date.</li>
<li><code>ignore_entry</code> will cause the entire entry containing the future date to be ignored until the date is past.</li>
<li>Anything else (i.e.. the default) will leave the date as is, causing the entries that contain these dates sort to the top of the planet until the time passes.</li>
</ul>
</li>
<li><code>xml_base</code> will adjust the <code>xml:base</code> values in effect for each of the text constructs in the feed (things like <code>title</code>, <code>summary</code>, and <code>content</code>). Other elements in the feed (most notably, <code>link</code> are not affected by this value.
<ul style="margin:0">
<li><code>feed_alternate</code> will replace the <code>xml:base</code> in effect with the value of the <code>alternate</code> <code>link</code> found either in the enclosed <code>source</code> or enclosing <code>feed</code> element.</li>
<li><code>entry_alternate</code> will replace the <code>xml:base</code> in effect with the value of the <code>alternate</code> <code>link</code> found in this entry.</li>
<li>Any other value will be treated as a <a href="http://www.ietf.org/rfc/rfc3986.txt">URI reference</a>. These values may be relative or absolute. If relative, the <code>xml:base</code> values in each text construct will each be adjusted separately using to the specified value.</li>
</ul>
</li>
</ul>
</body>
</html>