document Preprocessor and TreeProcessor more thoroughly (PR #4415)

3 files changed, 32 insertions, 8 deletions

diff --git a/docs/modules/extensions/pages/index.adoc b/docs/modules/extensions/pages/index.adoc
index 5b45fbc0..ea37dc6b 100644
--- a/docs/modules/extensions/pages/index.adoc
+++ b/docs/modules/extensions/pages/index.adoc
@@ -8,6 +8,8 @@ As a result, extensions in Asciidoctor are easy to write, powerful, and simple t
 Asciidoctor also allows extensions to be written using the full power of a programming language (whether it be Ruby, Java, Groovy or JavaScript).
 You don't have to shave yaks to get the functionality you want, and you can distribute the extension using de facto standard packaging mechanisms like RubyGems or JARs.
 
+API documentation can be found at https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions
+
 == Available extension points
 
 Asciidoctor provides the following extension points:
diff --git a/docs/modules/extensions/pages/preprocessor.adoc b/docs/modules/extensions/pages/preprocessor.adoc
index f0e0e551..177ccd6d 100644
--- a/docs/modules/extensions/pages/preprocessor.adoc
+++ b/docs/modules/extensions/pages/preprocessor.adoc
@@ -1,10 +1,22 @@
-= Preprocessor Extension Example
+= Preprocessor Extensions
 :navtitle: Preprocessor
 
+Preprocessors are run after the source text is split into lines and normalized, but before parsing begins.
+
+Prior to invoking the preprocessor, Asciidoctor splits the source text into lines and normalizes them. The normalize process strips trailing whitespace and the end of line character sequence from each line.
+
+Asciidoctor passes the document and the document's Reader to the `Asciidoctor::Extensions::Processor#process` method of the Preprocessor instance. The Preprocessor can modify the Reader as necessary and either return the same Reader (or falsy, which is equivalent) or a reference to a substitute Reader.
+
+Note that in this process, line numbers are not retained. If the preprocessor adds or removes lines, the parser and other extensions could produce misleading line number information.
+
+Preprocessor implementations must extend the https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/Preprocessor[`Preprocessor`] class.
+
+== Preprocessor Extension Example
+
 Purpose::
 Skim off front matter from the top of the document that gets used by site generators like Jekyll and Awestruct.
 
-== sample-with-front-matter.adoc
+=== sample-with-front-matter.adoc
 
 [source,asciidoc]
 ----
@@ -23,7 +35,7 @@ content
 ....
 ----
 
-== FrontMatterPreprocessor
+=== FrontMatterPreprocessor
 
 [source,ruby]
 ----
@@ -53,7 +65,7 @@ class FrontMatterPreprocessor < Asciidoctor::Extensions::Preprocessor
 end
 ----
 
-== Usage
+=== Usage
 
 [source,ruby]
 ----
diff --git a/docs/modules/extensions/pages/tree-processor.adoc b/docs/modules/extensions/pages/tree-processor.adoc
index ef598100..c27788db 100644
--- a/docs/modules/extensions/pages/tree-processor.adoc
+++ b/docs/modules/extensions/pages/tree-processor.adoc
@@ -1,10 +1,20 @@
-= Tree Processor Extension Example
+= Tree Processor Extensions
 :navtitle: Tree Processor
 
+TreeProcessors are run on the Document after the source has been parsed into an abstract syntax tree (AST), as represented by the Document object and its child Node objects (e.g., Section, Block, List, ListItem).
+
+Asciidoctor invokes the `Processor#process` method on an instance of each registered TreeProcessor.
+
+The AST nodes contain unconverted text (for example `[.role]#hello#`), and conversion is typically done when the text is read from the node (for the previous example, returning `<span class="role">hello</span>`). Unconverted text can be written back to the node during transformation (for example `[.role]#goodbye#`).
+
+TreeProcessor implementations must extend https://www.rubydoc.info/gems/asciidoctor/Asciidoctor/Extensions/TreeProcessor[`TreeProcessor`].
+
+== Tree Processor Extension Example
+
 Purpose::
 Detect literal blocks that contain shell commands, strip the prompt character and style the command using CSS in such a way that the prompt character cannot be selected (as seen on help.github.com).
 
-== sample-with-shell-session.adoc
+=== sample-with-shell-session.adoc
 
 [source,asciidoc]
 ----
@@ -14,7 +24,7 @@ Detect literal blocks that contain shell commands, strip the prompt character an
  $ gem install asciidoctor
 ----
 
-== ShellSessionTreeProcessor
+=== ShellSessionTreeProcessor
 
 [source,ruby]
 ----
@@ -47,7 +57,7 @@ class ShellSessionTreeProcessor < Asciidoctor::Extensions::TreeProcessor
 end
 ----
 
-== Usage
+=== Usage
 
 [source,ruby]
 ----