revise documentation for Preprocessor and TreeProcessor extensions

3 files changed, 26 insertions, 14 deletions

diff --git a/docs/modules/extensions/pages/index.adoc b/docs/modules/extensions/pages/index.adoc
index ea37dc6b..92e28e13 100644
--- a/docs/modules/extensions/pages/index.adoc
+++ b/docs/modules/extensions/pages/index.adoc
@@ -2,14 +2,12 @@
 :url-exten-lab: https://github.com/asciidoctor/asciidoctor-extensions-lab
 
 Extensions are central to the success of AsciiDoc because they open up the language to new use cases.
-Asciidoctor provides an extension API that offers a superset of extension points.
+Asciidoctor provides an https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions[extension API] that offers a superset of extension points.
 As a result, extensions in Asciidoctor are easy to write, powerful, and simple to distribute.
 
 Asciidoctor also allows extensions to be written using the full power of a programming language (whether it be Ruby, Java, Groovy or JavaScript).
 You don't have to shave yaks to get the functionality you want, and you can distribute the extension using de facto standard packaging mechanisms like RubyGems or JARs.
 
-API documentation can be found at https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions
-
 == Available extension points
 
 Asciidoctor provides the following extension points:
diff --git a/docs/modules/extensions/pages/preprocessor.adoc b/docs/modules/extensions/pages/preprocessor.adoc
index 177ccd6d..831a7761 100644
--- a/docs/modules/extensions/pages/preprocessor.adoc
+++ b/docs/modules/extensions/pages/preprocessor.adoc
@@ -1,20 +1,33 @@
 = Preprocessor Extensions
 :navtitle: Preprocessor
 
-Preprocessors are run after the source text is split into lines and normalized, but before parsing begins.
+The https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/Preprocessor[Preprocessor] extension runs prior to the AsciiDoc source being parsed.
+Specifically, it runs after the Document instance has been created and its PreprocessorReader has been initialized to start reading the lines of the source document.
 
-Prior to invoking the preprocessor, Asciidoctor splits the source text into lines and normalizes them. The normalize process strips trailing whitespace and the end of line character sequence from each line.
+[WARNING]
+====
+If the preprocessor extension reads lines from the reader, it will cause side effects.
+One of these side effects is that document attributes in the header may not be interpreted correctly.
+Another is that it may interfere with line number tracking.
+If the extension adds or removes lines, the parser could produce misleading line number information.
+Thus, interacting with the reader in a prepreprocessor extension should be avoided, if possible.
 
-Asciidoctor passes the document and the document's Reader to the `Asciidoctor::Extensions::Processor#process` method of the Preprocessor instance. The Preprocessor can modify the Reader as necessary and either return the same Reader (or falsy, which is equivalent) or a reference to a substitute Reader.
+A preprocessor extension can still be used to modify options on the Document object, in which case there's no side effect.
+It may also be possible to avoid side effects by reading lines which are not interpreted as AsciiDoc, as is shown in the example on this page.
+====
 
-Note that in this process, line numbers are not retained. If the preprocessor adds or removes lines, the parser and other extensions could produce misleading line number information.
+Prior to invoking the preprocessor extension, Asciidoctor splits the source text of the primary document into lines and xref:asciidoc::normalization.adoc[normalizes them].
+It then creates an instance of PreprocessorReader from these lines.
 
-Preprocessor implementations must extend the https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/Preprocessor[`Preprocessor`] class.
+Asciidoctor then passes the Document and PreprocessorReader instances to the `Asciidoctor::Extensions::Processor#process` method of the extension.
+If the extension reads lines from reader, those lines will first be preprocessed by Asciidoctor's internal preprocessor.
+The extension can modify the Reader and either return the same Reader (or falsy, which is equivalent) or a new Reader instance to replace it.
+However, as stated above, be careful when reading lines from the original PreprocessorReader instance as it can have side effets.
 
 == Preprocessor Extension Example
 
 Purpose::
-Skim off front matter from the top of the document that gets used by site generators like Jekyll and Awestruct.
+Skim off front matter from the top of the document that gets used by site generators like Jekyll.
 
 === sample-with-front-matter.adoc
 
diff --git a/docs/modules/extensions/pages/tree-processor.adoc b/docs/modules/extensions/pages/tree-processor.adoc
index c27788db..a670e2dd 100644
--- a/docs/modules/extensions/pages/tree-processor.adoc
+++ b/docs/modules/extensions/pages/tree-processor.adoc
@@ -1,13 +1,14 @@
 = Tree Processor Extensions
 :navtitle: Tree Processor
 
-TreeProcessors are run on the Document after the source has been parsed into an abstract syntax tree (AST), as represented by the Document object and its child Node objects (e.g., Section, Block, List, ListItem).
+A https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/TreeProcessor[TreeProcessor] extension is run on the Document instance after the blocks in the source have been parsed into a document structure, as represented by the Document object and its child AbstractNode objects (e.g., Section, Block, List, ListItem).
 
-Asciidoctor invokes the `Processor#process` method on an instance of each registered TreeProcessor.
+Asciidoctor invokes the `Processor#process` method on an instance of each registered TreeProcessor extension.
 
-The AST nodes contain unconverted text (for example `[.role]#hello#`), and conversion is typically done when the text is read from the node (for the previous example, returning `<span class="role">hello</span>`). Unconverted text can be written back to the node during transformation (for example `[.role]#goodbye#`).
-
-TreeProcessor implementations must extend https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/TreeProcessor[`TreeProcessor`].
+IMPORTANT: A TreeProcessor extension is run before any inline markup is interpreted.
+At the time the process method on the TreeProcessor extension is invoked, the block nodes in the document have unprocessed, unconverted text.
+Conversion of inline markup is done when the document is being converted.
+Thus, it's possible for the TreeProcessor extension to replace the source text on the nodes, but it must consider the fact that the text is still going to be parsed (according to the content model of the block).
 
 == Tree Processor Extension Example