Improved Formatter

Installed as a WikkaBetaFeatures beta feature on this server as of 2005-06-12.

This is the development page for an improved version of "the Formatter", specifically, the code in ./formatters/wakka.php (as opposed to the AdvancedFormatter page which deals with "advanced" formatting in other ways as well, such as standardized code generation utilities).
 

Why?


While our current (version 1.1.6.0) Formatter is quite capable, it has some quirks and bugs, doesn't always generate valid XHTML (though it tries hard), and misses a few things that would be nice to have or that would enable things that would be nice to have (such as a TableofcontentsAction page TOCs). The improved version presented here tries to address some of these issues (with more likely to follow).


What?


Here is a short summary of what has changed (details below):

The code presented below is still considered a beta version and as such contains many lines of (commented-out) debug code. These will of course be removed before final release. Any reference to line numbers is (for now) to the new (beta) code since this is a complete drop-in replacement for the original file.


Closing open tags


The current version (Wikka 1.1.6.0) of the Formatter has a bit of code contributed by DotMG to close any left-open tags at the very end of a page. While that can solve some problems with rendering and including pages, the code was incomplete in which open tags were closed. A particular problem was still-open lists and indents which weren't handled at all (see "List parsing bug?" on WikkaBugs). Also, this code would directly echo output instead of returning a string as the rest of the Formatter's main function does.

The new version addresses all of these problems.

Closing of indents and (open) lists was already happening when encountering a newline that doesn't start with a TAB or a ~, so this bit is separated out as a function. Improved now by removing superfluous variables and corresponding parameters.

  1. if (!function_exists('close_indents'))
  2. {
  3.     function close_indents(&$indentClosers,&$oldIndentLevel)    # JW 2005-07-11 removed superfluous variables
  4.     {
  5.         $result='';
  6.  
  7.         $c = count($indentClosers);
  8.         for ($i = 0; $i < $c; $i++)
  9.         {
  10.             $result .= array_pop($indentClosers);
  11.             $br = 0;
  12.         }
  13.         $oldIndentLevel = 0;
  14.  
  15.         return $result;
  16.     }
  17. }


The section that handles newlines now only needs to call this function:
  1.             $result .= close_indents($indentClosers,$oldIndentLevel);   # JW 2005-07-11 removed superfluous variables
  2.  
  3.             $result .= ($br) ? "<br />\n" : "\n";
  4.             $br = 1;
  5.             return $result;


To close open tags at the end of the page, the new code now calls this function first, and then handles all other open tags, in an order to at least minimize incorrect tag nesting (but see "Not a compete solution!" below):

  1.         if ((!is_array($things)) && ($things == 'closetags'))
  2.         {
  3.             $result .= close_indents($indentClosers,$oldIndentLevel);   # JW 2005-07-11 removed superfluous variables
  4.  
  5.             if ($trigger_bold % 2) $result .= '</strong>';
  6.             if ($trigger_italic % 2) $result .= '</em>';
  7.             if ($trigger_keys % 2) $result .= '</kbd>';
  8.             if ($trigger_monospace % 2) $result .= '</tt>';
  9.  
  10.             if ($trigger_underline % 2) $result .= '</span>';
  11.             if ($trigger_notes % 2) $result .= '</span>';
  12.             if ($trigger_strike % 2) $result .= '</span>';
  13.             if ($trigger_inserted % 2) $result .= '</span>';
  14.             if ($trigger_deleted % 2) $result .= '</span>';
  15.  
  16.             if ($trigger_center % 2) $result .= '</div>';
  17.             if ($trigger_floatl % 2) $result .= '</div>';
  18.             if ($trigger_floatr % 2) $result .= '</div>';               # JW added
  19.             for ($i = 1; $i<=5; $i ++)
  20.             {
  21.                 if ($trigger_l[$i] % 2) $result .= ("</h$i>");
  22.             }
  23.  
  24.             $trigger_bold = $trigger_italic = $trigger_keys = $trigger_monospace = 0;
  25.             $trigger_underline = $trigger_notes = $trigger_strike = $trigger_inserted = $trigger_deleted = 0;
  26.             $trigger_center = $trigger_floatl = $trigger_floatr = 0;
  27.             $trigger_l = array(-1, 0, 0, 0, 0, 0);
  28.             return $result;
  29.         }
  30.         else
  31.         {
  32.             $thing = $things[1];
  33.         }


This is now used like this:
  1.     $text .= wakka2callback('closetags');                   # JW changed logic


Not a complete solution!
A big problem remains, however: in order to produce valid (X)HTML, open tags cannot just be closed anywhere: there are rules for which elements can contain which other elements. For instance, an inline element (like <em>) can never contain a block element (like a list). So if the inline element is left open (which happens if someone types // to start emphasized text but doesn't close it before starting an indent or list), closing the generated opening <em> tag at the end of the page may prevent display problems in some browsers, but the result is still not valid (X)HTML. This type of problem can only be really addressed with completely different mechanism for a formatter. This should definitely be tackled at some time, but is outside the scope of the current improvements which are designed to work within the current Formatter's mechanism.

Better handling of nested lists and indents

New as of 2005-07-12

There were still some issues with nested lists and indents, in particular when the type of list changed without a "level" change or when changing to a higher level ("outdent"). At the same time, a list or indent right at the start of a page was not detected or handled at all. (Although that is bad style, and a page should start with a heading, it still should be handled correctly by the formatter, of course.) Finally, accented (Umlaut) characters were treated as a list type.

In order to detect a list or indent at the start of the page as well as after a newline (and avoid Umlauts) the detection RegEx for a list or indent is now coded as follows (in the wakka2callback call):
  1.     '(^|\n)([\t~]+)(-|&|[0-9a-zA-Z]+\))?|'.                                             # indents and lists # JW FIXED 2005-07-12 also match tab or ~ at start of document


By using (^|\n) as an anchor for matching instead of merely \n the start of the page is also matched.

The actual code for handling a list or indent line was comnpletely rewritten to properly handle change of list types and to produce readable and nicely indented XHTML code. Note that this section now also starts with the (^|\n) anchor:
  1.         // indented text
  2.         # JW FIXED 2005-07-09 accented chars not used for ordered lists
  3.         # JW FIXED 2005-07-12 this does not cover the case where a list item is followed by an inline comment of the *same* level
  4.         # JW FIXED 2005-07-12 as with the expression in the /edit handler this does not cover tab or ~ at the start of the document
  5.         elseif (preg_match('/(^|\n)([\t~]+)(-|&|[0-9a-zA-Z]+\))?(\n|$)/s', $thing, $matches))
  6.         {
  7.             $br = 0;                                                        # no break needed after a block
  8.  
  9.             // get new indent level
  10.             $newIndentLevel = strlen($matches[2]);      # JW 2005-07-12 also match tab or ~ at start of document
  11.             // derive code indent
  12.             $codeIndent = str_repeat("\t",$newIndentLevel-1);
  13.             $nlTabs = "\n".$codeIndent;
  14.             $nlTabsOut = $nlTabs."\t";
  15.  
  16.             // find out which indent type we want
  17.             $newIndentType = $matches[3];               # JW 2005-07-12 also match tab or ~ at start of document
  18.             // derive code fragments
  19.             if     ($newIndentType == '')               # plain indent
  20.             {
  21.                 $opener = '<div class="indent">';
  22.                 $closer = '</div>'/*.$nlTabs*/;
  23.             }
  24.             elseif ($newIndentType == '-')              # unordered list
  25.             {
  26.                 $opener = '<ul>'.$nlTabs.'<li>';
  27.                 $closer = '</li>'.$nlTabs.'</ul>';
  28.             }
  29.             elseif ($newIndentType == '&')              # inline comment
  30.             {
  31.                 $opener = '<ul class="thread">'.$nlTabs.'<li>';
  32.                 $closer = '</li>'.$nlTabs.'</ul>';
  33.             }
  34.             else                                        # ordered list
  35.             {
  36.                 $opener = '<ol type="'.substr($newIndentType, 0, 1).'">'.$nlTabs.'<li>';
  37.                 $closer = '</li>'.$nlTabs.'</ol>';
  38.                 $newIndentType = 'o';
  39.             }
  40.  
  41.             // do an indent
  42.             if ($newIndentLevel > $oldIndentLevel)
  43.             {
  44.                 for ($i = 0; $i < $newIndentLevel - $oldIndentLevel; $i++)
  45.                 {
  46.                     $result .= $nlTabs./*'<!--nested item '.$newIndentLevel.'-->'.*/$opener;
  47.                     array_push($indentClosers, $closer);
  48.  
  49.                     #$result .= '<!--pushed type: '.$oldIndentType.' -->';  # @@@
  50.                     array_push($indentTypes, $oldIndentType);               # remember type hierarchically
  51.                 }
  52.             }
  53.             // do an outdent or stay at the same level
  54.             else if ($newIndentLevel <= $oldIndentLevel)
  55.             {
  56.                 $bOutdent = FALSE;
  57.                 if ($newIndentLevel < $oldIndentLevel)
  58.                 {
  59.                     $bOutdent = TRUE;                                           # remember we're outdenting, for correct layout
  60.                     // do the outdenting
  61.                     for ($i = 0; $i < $oldIndentLevel - $newIndentLevel; $i++)
  62.                     {
  63.                         if ($i > 0)
  64.                         {
  65.                             $result .= $nlTabsOut;
  66.                         }
  67.                         $result .= array_pop($indentClosers)/*.'<!--outdent to '.$newIndentLevel.'-->'*/;
  68.  
  69.                         $oldIndentType = array_pop($indentTypes);               # make sure we will compare with "correct" previous type
  70.                         #$result .= '<!--popped type: '.$oldIndentType.' -->';  # @@@
  71.                     }
  72.                 }
  73.                 if ($bOutdent)                                                  # outdenting: put close tag on new line
  74.                 {
  75.                     $result .= $nlTabs/*.'<!--outdent: close tag on new line-->'*/;
  76.                 }
  77.                 // JW 2005-07-11 new item of different type
  78.                 if ($newIndentType != $oldIndentType)
  79.                 {
  80.                     $result .= array_pop($indentClosers);
  81.                     $result .= /*'<!--type change follows (old: '.$oldIndentType.' new: '.$newIndentType.') -->'.*/$nlTabs.$opener;
  82.                     array_push($indentClosers, $closer);
  83.                 }
  84.                 // new item of same type
  85.                 else
  86.                 {
  87.                     // plain indent
  88.                     if ($newIndentType == '')
  89.                     {
  90.                         $result .= $closer./*'<!--same type ('.$newIndentType.') same level-->'.*/$nlTabs.$opener;
  91.                     }
  92.                     // list or inline comment
  93.                     else
  94.                     {
  95.                         $result .= '</li>'.$nlTabs.'<li>'/*.'<!--back to same type-->'*/;
  96.                     }
  97.                 }
  98.             }
  99.             $oldIndentType  = $newIndentType;                       # remember type sequentially
  100.             $oldIndentLevel = $newIndentLevel;
  101.  
  102.             return $result;
  103.         }


Since the new code avoids adding an extra <br /> before a list (ul, ol) or indent (div) - these are block-level elements and line breaks should not be used to separate them (they really should be used only within flowing text) - the stylesheet had to be tweaked a little since it actually (implicitly) assumes a line break is there. Change the following in css/wikka.css or your own "skin":
  1. ul, ol {
  2.     margin-top: 0px;
  3.     margin-bottom: 0px;
  4.     padding-top: 0px;
  5.     padding-bottom: 0px;
  6. }


to:
  1. ul, ol {
  2.     /*margin-top: 0px;*/        /* keep natural margin; an extra <br/> is no longer generated */
  3.     margin-bottom: 0px;
  4.     padding-top: 0px;
  5.     padding-bottom: 0px;
  6. }
  7. ul ul, ol ol, ul ol, ol ul {            /* keep suppressing margin for nested lists */
  8.     margin-top: 0px;
  9. }

(Since we're dealing with beta code anyway, line numbers refer to the stylesheet as implemented on this server.)

Also, the styling for inline comments (line 66) should be moved so it actually overrides the generic style on line 100 etc. instead of the other way round:
  1. /* ul.thread styles moved so they come after the generic ul style */

  1. /* these ul.thread styles must come after the generic ul style in order to override it */
  2. ul.thread {
  3.     list-style-type: none;
  4.     border-left: 2px #666 solid;
  5.     padding-left: 10px;
  6.     margin: 5px 0px;
  7. }
  8.  
  9. ul.thread li {
  10.     color: #333;
  11.     font-size: 12px;
  12. }


Escaping single ampersands


While there are a few cases where it's actually allowed to use a plain & in HTML, in most cases where an ampersand is not part of an entity reference it needs to be escaped as &amp;. The current (version 1.1.6.0) Formatter escapes the < and > special characters, but not &, so the result may be invalid XHTML.

We need to find the ampersands that are not part of an entity reference. So we first build a RegEx to recognize the part of an entity reference that follows the ampersand that starts it; it can be a named entity, or a decimal or a hex numerical entity; and it can be terminated by a semicolon (;) in most cases, but there are a few cases where it's legal to leave off the terminating semicolon. To make it easier to read, we build the RegEx to express all that from its constituent parts:
  1. // define entity patterns
  2. // NOTE most also used in wikka.php for htmlentities_ent(): REGEX library!
  3. $alpha  = '[a-z]+';                         # character entity reference
  4. $numdec = '#[0-9]+';                        # numeric character reference (decimal)
  5. $numhex = '#x[0-9a-f]+';                    # numeric character reference (hexadecimal)
  6. $terminator = ';|(?=($|[\n<]|&lt;))';       # semicolon; or end-of-string, newline or tag
  7. $entitypat = '('.$alpha.'|'.$numdec.'|'.$numhex.')('.$terminator.')';   # defines entity pattern without the starting &
  8. $entityref = '&'.$entitypat;                # entity reference


So now we can define a 'lone' ampersand as one that is not followed by the expression $entitypat:
  1. $loneamp = '&(?!'.$entitypat.')';               # ampersand NOT part of an entity


This then becomes part of the big expression that's used in the preg_replace_callback() near the end of the file, as the last thing to consider before a newline:
  1.     '<|>|'.                                                                             # HTML special chars - after wiki markup!
  2.     $loneamp.'|'.                                                                       # HTML special chars - ampersand NOT part of an enity
  3.     '\n'.                                                                               # new line


Now we can "escape" all HTML special characters, as we should:
  1.         // convert HTML thingies (including ampersand NOT part of entity)
  2.         if ($thing == '<')
  3.             return '&lt;';
  4.         else if ($thing == '>')
  5.             return '&gt;';
  6.         else if ($thing == '&')
  7.             return '&amp;';


Nesting floats


I happened to find that the code for a left float (<<) would terminate a right float (>>) and vice versa. Which would of course likely leave unclosed tags. It turned out that by solving that it actually became possible to nest unlike floats - one level deep, at least. No great feature, but it could be handy at times.The solution is actually quite simple: there was just a single "trigger" to keep track of start and end of a float; keeping a separate trigger for left and right floats (an not generating newlines) is all that's needed:

  1.         // JW 2005-05-23: changed floats handling so they can be nested (one type within another only)
  2.         // float box left
  3.         else if ($thing == '<<')
  4.         {
  5.             #return (++$trigger_floatl % 2 ? '<div class="floatl">'."\n" : "\n</div>\n");
  6.             return (++$trigger_floatl % 2 ? '<div class="floatl">' : '</div>'); # JW changed (no newline)
  7.         }
  8.         // float box right
  9.         else if ($thing == '>>')
  10.         {
  11.             #return (++$trigger_floatl % 2 ? '<div class="floatr">'."\n" : "\n</div>\n");
  12.             return (++$trigger_floatr  % 2 ? '<div class="floatr">' : '</div>');    # JW changed (trigger, no newline)
  13.         }


Note line 114 where we now use a $trigger_floatr instead of $trigger_floatl: this solves the bug and creates a new micro-feature at the same time.

Now that the improved formatter has been installed as a beta feature, I added a small demo in the SandBox (in case it disappears: look for the edit made on 2005-06-14 21:45:57 in the revisions). --JW

Ids in embedded code


Since in ID must be unique in a page, embedding HTML code, and combining that with generated code, creates a problem. In order to to ensure that the page is valid XHTML, every id attribute must have unique value, regardless where it's coming from.

When generating code that should contain ids, this is simple: just use the GenerateUniqueId makeId() method to generate one, with or without specifying parameters. Still, the result could conflict with id attributes in embedded HTML code so we must handle those as well.

We analyze the whole block of embedded code, run each id through makeId(); if this method detects the id already exists, it will return an amended value with a sequence suffix; if it finds the id value wasn't valid, it will create a new one and return that. The formatter then replaces every id for which a different value was returned:

  1.         // escaped text
  2.         else if (preg_match('/^""(.*)""$/s', $thing, $matches))
  3.         {
  4. /*
  5. echo 'embedded content<br/>';
  6. */
  7.             // get config
  8. #           $allowed_double_doublequote_html = $wakka->GetConfigValue('double_doublequote_html');
  9.             $ddquotes_policy = $wakka->config['double_doublequote_html'];
  10. /*
  11. echo 'double quotes: '.$ddquotes_policy.'<br/>';
  12. */
  13.             // get embedded code
  14.             $embedded = $matches[1];
  15.             // handle embedded id attributes for 'safe' and 'raw'
  16.             if ($ddquotes_policy == 'safe' || $ddquotes_policy == 'raw')
  17.             {
  18.                 // get tags with id attributes
  19.                 $patTagWithId = '((<[a-z].*?)(id=("|\')(.*?)\\4)(.*?>))';
  20.                 // with PREG_SET_ORDER we get an array for each match: easy to use with list()!
  21.                 // we do the match case-insensitive so we catch uppercase HTML as well;
  22.                 // SafeHTML will treat this but 'raw' may end up with invalid code!
  23.                 $tags2 = preg_match_all('/'.$patTagWithId.'/i',$embedded,$matches2,PREG_SET_ORDER); # use backref to match both single and double quotes
  24. /*
  25. echo '# of matches (2): '.$tags2.'<br/>';
  26. echo '<!--found (set order):'."\n";
  27. print_r($matches2);
  28. echo '-->'."\n";
  29. */
  30.                 // step through code, replacing tags with ids with tags with new ('repaired') ids
  31.                 $tmpembedded = $embedded;
  32.                 $newembedded = '';
  33.                 for ($i=0; $i < $tags2; $i++)
  34.                 {
  35.                     list(,$tag,$tagstart,$attrid,$quote,$id,$tagend) = $matches2[$i];   # $attrid not needed, just for clarity
  36.                     $parts = explode($tag,$tmpembedded,2);              # split in two at matched tag
  37.                     if ($id != ($newid = $wakka->makeId('embed',$id)))  # replace if we got a new value
  38.                     {
  39. /*
  40. echo 'replacing tag - old id: '.$id.' new id: '.$newid.'<br/>';
  41. */
  42.                         $tag = $tagstart.'id='.$quote.$newid.$quote.$tagend;
  43.                     }
  44. /*
  45. echo "<!--old: $tag -->\n";
  46. echo "<!--new: $replacetag -->\n";
  47. */
  48.                     $newembedded .= $parts[0].$tag;                     # append (replacement) tag to first part
  49.                     $tmpembedded  = $parts[1];                          # after tag: next bit to handle
  50.                 }
  51.                 $newembedded .= $tmpembedded;                           # add last part
  52. /*
  53. echo '<!--translation:'."\n";
  54. echo $newembedded;
  55. echo '-->'."\n";
  56. */
  57.             }
  58.             // return (treated) embedded content according to config
  59.             // NOTE: we apply SafeHTML *after* id treatment so it won't be throwing away invalid ids that we're repairing instead!
  60.             switch ($ddquotes_policy)
  61.             {
  62.                 case 'safe':
  63.                     return $wakka->ReturnSafeHTML($newembedded);
  64.                 case 'raw':
  65.                     return $newembedded;                                # may still be invalid code - 'raw' will not be corrected!
  66.                 default:
  67.                     return $wakka->htmlspecialchars_ent($embedded);     # display only
  68.             }
  69.         }


As long as ids in the embedded code are valid and unique, they remain unchanged because makeId() is called with the 'embed' parameter which tells it not to add an id group prefix.

Still, it is important to remember that ids can only truly be guaranteed to be unique if every bit of code that generates HTML with ids is actually using the makeId() method - and that includes user-contributed extensions.

The "Fatal error: Call to a member function on a non-object" bug referred to on WikkaBugsResolved is also fixed here (line 282).

TODO: There is still one problem to be solved here: when embedded HTML code contains an id, it's entirely possible that it (or a following embedded section) contains a reference to that id. When the makeId() method finds it is necessary to change an id because it conflicts with a pre-existing one, any reference to it should also be updated.
The current code does not (yet) take care of this - it's a fairy complicated problem to solve correctly, but will be tackled soon.


Heading ids


Creating ids for headings is (you guessed it) the first (and necessary) piece of the puzzle to enable generating TableofcontentsAction page TOCs, but other bits will be needed for that as well, such as actually gathering the references to headings (and their levels), and the ability to link to page fragments (something our current core does not support yet). So: we cannot generate TOCs - yet - but we are getting there; the code is also designed to make it possible to extend it to generate TOCs not just for headings, but also for things like images, tables and code blocks.

A method for generating a TOC has not been decided yet (we may even provide alternatives), but one thing we certainly need is ids for headings (see TableofcontentsAction for more background on this); and even if we do not (yet) generate a TOC, being able to link to a page fragment (the obvious next step) will be useful in itself.

Some thought went into the method of generating the ids: Ideally they should be 'recognizable' so creating links to a page fragment with a heading wil be easy, and they should be as 'constant' as possible so a link to a section remains a link to that section, even if that is moved to a different position on the page, or another is inserted before it. This implies that all methods that simply generate a sequential id will not fulfill our requirements. We also don't burden the writer with coming up with ids (or even needing to think about them): they should be able to just concentrate on the content. Instead, we use following approach:


All this is implemented as an "afterburner" type of formatter which is applied after all basic formatting has already taken place and we already have the XHTML output of that process. This ensures that all headings are taken into account, whether they are generated from Wikka markup or from embedded HTML code. The afterburner preg_replace_callback() function is designed to be extended with other types of code fragments we might want to generate ids (and maybe TableofcontentsAction page TOCs...) for.

The 'afterburner' function is defined like this:
  1. if (!function_exists('wakka3callback'))
  2. {
  3.     /**
  4.      * "Afterburner" formatting: extra handling of already-generated XHTML code.
  5.      *
  6.      * 1.
  7.      * Ensure every heading has an id, either specified or generated. (May be
  8.      * extended to generate section TOC data.)
  9.      * If an id is specified, that is used without any modification.
  10.      * If no id is specified, it is generated on the basis of the heading context:
  11.      * - any image tag is replaced by its alt text (if specified)
  12.      * - all tags are stripped
  13.      * - all characters that are not valid in an id are stripped (except whitespace)
  14.      * - the resulting string is then used by makedId() to generate an id out of it
  15.      *
  16.      * @access  private
  17.      * @uses    Wakka::makeId()
  18.      *
  19.      * @param   array   $things required: matches of the regex in the preg_replace_callback
  20.      * @return  string  heading with an id attribute
  21.      */
  22.     function wakka3callback($things)
  23.     {
  24.         global $wakka;
  25.         $thing = $things[1];
  26.  
  27.         // heading
  28.         if (preg_match('#^<(h[1-6])(.*?)>(.*?)</\\1>$#s', $thing, $matches))    # note that we don't match headings that are not valid XHTML!
  29.         {
  30. /*
  31. echo 'heading:<pre>';
  32. print_r($matches);
  33. echo '</pre>';
  34. */
  35.             list($element,$tagname,$attribs,$heading) = $matches;
  36.  
  37.             #if (preg_match('/(id=("|\')(.*?)\\2)/',$attribs,$matches)) # use backref to match both single and double quotes
  38.             if (preg_match('/(id=("|\')(.*?)\\2)/',$attribs))           # use backref to match both single and double quotes
  39.             {
  40.                 // existing id attribute: nothing to do (assume already treated as embedded code)
  41.                 // @@@ we *may* want to gather ids and heading text for a TOC here ...
  42.                 // heading text should then get partly the same treatment as when we're creating ids:
  43.                 // at least replace images and strip tags - we can leave entities etc. alone - so we end up with
  44.                 // plain text-only
  45.                 // do this if we have a condition set to generate a TOC
  46.                 return $element;
  47.             }
  48.             else
  49.             {
  50.                 // no id: we'll have to create one
  51. #echo 'no id provided - create one<br/>';
  52.                 $tmpheading = trim($heading);
  53.                 // first find and replace any image with its alt text
  54.                 // @@@ can we use preg_match_all here? would it help?
  55.                 while (preg_match('/(<img.*?alt=("|\')(.*?)\\2.*?>)/',$tmpheading,$matches))
  56.                 {
  57. #echo 'image found: '.$tmpheading.'<br/>';
  58.                     # 1 = whole element
  59.                     # 3 = alt text
  60.                     list(,$element, ,$alttext) = $matches;
  61. /*
  62. echo 'embedded image:<pre>';
  63. print_r($matches);
  64. echo '</pre>';
  65. */
  66.                     // gather data for replacement
  67.                     $search  = '/'.str_replace('/','\/',$element).'/';  # whole element (delimiter chars escaped!) @@@ use preg_quote as well?
  68.                     $replace = trim($alttext);                          # alt text
  69. /*
  70. echo 'pat_repl:<pre>';
  71. echo 'search: '.$search.'<br/>';
  72. echo 'search: '.$replace.'<br/>';
  73. echo '</pre>';
  74. */
  75.                     // now replace img tag by corresponding alt text
  76.                     $tmpheading = preg_replace($search,$replace,$tmpheading);   # replace image by alt text
  77.                 }
  78.                 $headingtext = $tmpheading;
  79. #echo 'headingtext (no img): '.$headingtext.'<br/>';
  80.         // @@@ 2005-05-27 now first replace linebreaks <br/> with spaces!!
  81.                 // remove all other tags
  82.                 $headingtext = strip_tags($headingtext);
  83. #echo 'headingtext (no tags): '.$headingtext.'<br/>';
  84.                 // @@@ this all-text result is usable for a TOC!!!
  85.                 // do this if we have a condition set to generate a TOC
  86.  
  87.                 // replace entities that can be interpreted
  88.                 // use default charset ISO-8859-1 because other chars won't be valid for an id anyway
  89.                 $headingtext = html_entity_decode($headingtext,ENT_NOQUOTES);
  90.                 // remove any remaining entities (so we don't end up with strange words and numbers in the id text)
  91.                 $headingtext = preg_replace('/&[#]?.+?;/','',$headingtext);
  92. #echo 'headingtext (entities decoded/removed): '.$headingtext.'<br/>';
  93.                 // finally remove non-id characters (except whitespace which is handled by makeId())
  94.                 $headingtext = preg_replace('/[^A-Za-z0-9_:.-\s]/','',$headingtext);
  95. #echo 'headingtext (id-ready): '.$headingtext.'<br/>';
  96.                 // now create id based on resulting heading text
  97.                 $id = $wakka->makeId('hn',$headingtext);
  98. #echo 'id: '.$id.'<br/>';
  99.  
  100.                 // rebuild element, adding id
  101.                 return '<'.$tagname.$attribs.' id="'.$id.'">'.$heading.'</'.$tagname.'>';
  102.             }
  103.         }
  104.         // other elements to be treated go here (tables, images, code sections...)
  105.     }
  106. }


This is called (after the primary formatter) as follows:
  1. // add ids to heading elements
  2. // @@@ LATER:
  3. // - extend with other elements (tables, images, code blocks)
  4. // - also create array(s) for TOC(s)
  5. $idstart = getmicrotime();
  6.     '#('.
  7.     '<h[1-6].*?>.*?</h[1-6]>'.
  8.     // other elements to be treated go here
  9.     ')#ms','wakka3callback',$text);
  10. printf('<!-- Header id generation took %.6f seconds -->', (getmicrotime() - $idstart));


The result is an id that is almost always derived directly from the heading content, giving a high chance that it will remain constant even if the page content is re-arranged: thus it provides a reliable target for a link.

Keeping track of recursion level


Since the Formatter is now "better" at closing any tags left open at the end of the string it's handling, a new issue is arose turning up with some of the beta code on this server: when the Formatter is being called recursively by an action (Formatter -> Action -> Formatter...) the "second-level" formatter will close all tags that were opened by the "first-level" formatter. When an action using the formatter is embedded in something like a heading or a list element (which in most cases should be entirely valid) the heading or list item (and whatever else is "open") is closed at the end of the action rather than at the point where it should be. While it's not really a good idea for an action (which is interpreted by the Formatter calling the Action() method) to call the Formatter in its turn (and usually redundant), the Formatter should handle this more elegantly and close tags only in its "topmost" instance.

The solution is to keep track of the recursion level and close open tags at the end only at the "outermost" level. The first thing we need is a varable to keep track of the level; we add this at the start of the Wakka class in wikka.php, after the other object variables:
  1.     var $callLevel = 0;                             # JW 2005-07-15 keep track of recursion levels of the formatter


Then in the Formatter, right before the wakka2callback() function is called to do the actual formatting, we increment the variable:
  1. $this->callLevel++;                                         # JW 2005-07-15 recursion level: getting in


and right afterwards we decrement it again, after which we execute the 'closetags' routine only when the call level is back at 0:
  1. $this->callLevel--;                                         # JW 2005-07-15 recursion level: getting out
  2. if ($this->callLevel == 0)                                  # JW 2005-07-15 only for "outmost" call level
  3. {
  4.     $text .= wakka2callback('closetags');                   # JW changed logic
  5. }



The Code


Here's the code (all of it). This replaces the file ./formatters/wakka.php.
This incorporates the small change needed to support DarTar's GrabCodeHandler, slightly extended to take advantage of the ability of the new AdvancedFormOpen advanced FormOpen() method to add a class to a form (lines 338-347) so the form can be properly styled.
If you want to test this improved formatter, you should either also grab DarTar's GrabCodeHandler or comment out these lines and uncomment line 337.


  1. <?php
  2.  
  3. // This may look a bit strange, but all possible formatting tags have to be in a single regular expression for this to work correctly. Yup!
  4.  
  5. // #dotmg [many lines] : Unclosed tags fix! For more info, m.randimbisoa@dotmg.net
  6. // JavaWoman - corrected and improved unclosed tags handling, including missing ones and indents
  7.  
  8. // ------------- define the necessary functions -----------
  9.  
  10. if (!function_exists('close_indents'))
  11. {
  12.     function close_indents(&$indentClosers,&$oldIndentLevel)    # JW 2005-07-11 removed superfluous variables
  13.     {
  14.         $result='';
  15.  
  16.         $c = count($indentClosers);
  17.         for ($i = 0; $i < $c; $i++)
  18.         {
  19.             $result .= array_pop($indentClosers);
  20.             $br = 0;
  21.         }
  22.         $oldIndentLevel = 0;
  23.  
  24.         return $result;
  25.     }
  26. }
  27.  
  28. if (!function_exists('wakka2callback'))
  29. {
  30.     function wakka2callback($things)
  31.     {
  32.         $result='';
  33.  
  34.         static $oldIndentType  = '';            # JW 2005-07-12 added
  35.         static $oldIndentLevel = 0;
  36.         #static $oldIndentLength= 0;            # JW 2005-07-12 removed superfluous variables
  37.         static $indentClosers  = array();
  38.         static $indentTypes    = array();       # JW 2005-07-12 added
  39.         #static $newIndentSpace = array();      # JW 2005-07-12 removed superfluous variables
  40.  
  41.         static $br = 1;
  42.  
  43.         static $trigger_bold = 0;
  44.         static $trigger_italic = 0;
  45.         static $trigger_keys = 0;
  46.         static $trigger_monospace = 0;
  47.  
  48.         static $trigger_underline = 0;
  49.         static $trigger_notes = 0;
  50.         static $trigger_strike = 0;
  51.         static $trigger_inserted = 0;
  52.         static $trigger_deleted = 0;
  53.  
  54.         static $trigger_center = 0;
  55.         static $trigger_floatl = 0;
  56.         static $trigger_floatr = 0;                                     # JW added
  57.         static $trigger_l = array(-1, 0, 0, 0, 0, 0);
  58.  
  59.         global $wakka;                                                  # @@@ should be capitalized but requires change in wikka.php (etc.)
  60.  
  61.         if ((!is_array($things)) && ($things == 'closetags'))
  62.         {
  63.             $result .= close_indents($indentClosers,$oldIndentLevel);   # JW 2005-07-11 removed superfluous variables
  64.  
  65.             if ($trigger_bold % 2) $result .= '</strong>';
  66.             if ($trigger_italic % 2) $result .= '</em>';
  67.             if ($trigger_keys % 2) $result .= '</kbd>';
  68.             if ($trigger_monospace % 2) $result .= '</tt>';
  69.  
  70.             if ($trigger_underline % 2) $result .= '</span>';
  71.             if ($trigger_notes % 2) $result .= '</span>';
  72.             if ($trigger_strike % 2) $result .= '</span>';
  73.             if ($trigger_inserted % 2) $result .= '</span>';
  74.             if ($trigger_deleted % 2) $result .= '</span>';
  75.  
  76.             if ($trigger_center % 2) $result .= '</div>';
  77.             if ($trigger_floatl % 2) $result .= '</div>';
  78.             if ($trigger_floatr % 2) $result .= '</div>';               # JW added
  79.             for ($i = 1; $i<=5; $i ++)
  80.             {
  81.                 if ($trigger_l[$i] % 2) $result .= ("</h$i>");
  82.             }
  83.  
  84.             $trigger_bold = $trigger_italic = $trigger_keys = $trigger_monospace = 0;
  85.             $trigger_underline = $trigger_notes = $trigger_strike = $trigger_inserted = $trigger_deleted = 0;
  86.             $trigger_center = $trigger_floatl = $trigger_floatr = 0;
  87.             $trigger_l = array(-1, 0, 0, 0, 0, 0);
  88.  
  89.             return $result;
  90.         }
  91.         else
  92.         {
  93.             $thing = $things[1];
  94.         }
  95.  
  96.         // convert HTML thingies (including ampersand NOT part of entity)
  97.         if ($thing == '<')
  98.             return '&lt;';
  99.         else if ($thing == '>')
  100.             return '&gt;';
  101.         else if ($thing == '&')
  102.             return '&amp;';
  103.         // JW 2005-05-23: changed floats handling so they can be nested (one type within another only)
  104.         // float box left
  105.         else if ($thing == '<<')
  106.         {
  107.             #return (++$trigger_floatl % 2 ? '<div class="floatl">'."\n" : "\n</div>\n");
  108.             return (++$trigger_floatl % 2 ? '<div class="floatl">' : '</div>'); # JW changed (no newline)
  109.         }
  110.         // float box right
  111.         else if ($thing == '>>')
  112.         {
  113.             #return (++$trigger_floatl % 2 ? '<div class="floatr">'."\n" : "\n</div>\n");
  114.             return (++$trigger_floatr % 2 ? '<div class="floatr">' : '</div>'); # JW changed (trigger, no newline)
  115.         }
  116.         // clear floated box
  117.         else if ($thing == '::c::')
  118.         {
  119.             return ('<div class="clear">&nbsp;</div>'."\n");
  120.         }
  121.         // keyboard
  122.         else if ($thing == '#%')
  123.         {
  124.             return (++$trigger_keys % 2 ? '<kbd class="keys">' : '</kbd>');
  125.         }
  126.         // bold
  127.         else if ($thing == '**')
  128.         {
  129.             return (++$trigger_bold % 2 ? '<strong>' : '</strong>');
  130.         }
  131.         // italic
  132.         else if ($thing == '//')
  133.         {
  134.             return (++$trigger_italic % 2 ? '<em>' : '</em>');
  135.         }
  136.         // monospace
  137.         else if ($thing == '##')
  138.         {
  139.             return (++$trigger_monospace % 2 ? '<tt>' : '</tt>');
  140.         }
  141.         // underline
  142.         else if ($thing == '__')
  143.         {
  144.             return (++$trigger_underline % 2 ? '<span class="underline">' : '</span>');
  145.         }
  146.         // notes
  147.         else if ($thing == "''")
  148.         {
  149.             return (++$trigger_notes % 2 ? '<span class="notes">' : '</span>');
  150.         }
  151.         // strikethrough
  152.         else if ($thing == '++')
  153.         {
  154.             return (++$trigger_strike % 2 ? '<span class="strikethrough">' : '</span>');
  155.         }
  156.         // additions
  157.         else if ($thing == '&pound;&pound;')
  158.         {
  159.             return (++$trigger_inserted % 2 ? '<span class="additions">' : '</span>');
  160.         }
  161.         // deletions
  162.         else if ($thing == '&yen;&yen;')
  163.         {
  164.             return (++$trigger_deleted % 2 ? '<span class="deletions">' : '</span>');
  165.         }
  166.         // center
  167.         else if ($thing == '@@')
  168.         {
  169.             return (++$trigger_center % 2 ? '<div class="center">'."\n" : "\n</div>\n");
  170.         }
  171.         // urls
  172.         else if (preg_match('/^([a-z]+:\/\/\S+?)([^[:alnum:]^\/])?$/', $thing, $matches))
  173.         {
  174.             $url = $matches[1];
  175.             if (preg_match('/^(.*)\.(gif|jpg|png)/si', $url)) {
  176.                 return '<img src="'.$url.'" alt="image" />'.$matches[2];
  177.             } else
  178.             // Mind Mapping Mod
  179.             if (preg_match('/^(.*)\.(mm)/si', $url)) {
  180.                 return $wakka->Action('mindmap '.$url);
  181.             } else
  182.                 return $wakka->Link($url).$matches[2];
  183.         }
  184.         // header level 5
  185.         else if ($thing == '==')
  186.         {
  187.                 $br = 0;
  188.                 return (++$trigger_l[5] % 2 ? '<h5>' : "</h5>\n");
  189.         }
  190.         // header level 4
  191.         else if ($thing == '===')
  192.         {
  193.                 $br = 0;
  194.                 return (++$trigger_l[4] % 2 ? '<h4>' : "</h4>\n");
  195.         }
  196.         // header level 3
  197.         else if ($thing == '====')
  198.         {
  199.                 $br = 0;
  200.                 return (++$trigger_l[3] % 2 ? '<h3>' : "</h3>\n");
  201.         }
  202.         // header level 2
  203.         else if ($thing == '=====')
  204.         {
  205.                 $br = 0;
  206.                 return (++$trigger_l[2] % 2 ? '<h2>' : "</h2>\n");
  207.         }
  208.         // header level 1
  209.         else if ($thing == '======')
  210.         {
  211.                 $br = 0;
  212.                 return (++$trigger_l[1] % 2 ? '<h1>' : "</h1>\n");
  213.         }
  214.         // forced line breaks
  215.         else if ($thing == "---")
  216.         {
  217.             return '<br />';
  218.         }
  219.         // escaped text
  220.         else if (preg_match('/^""(.*)""$/s', $thing, $matches))
  221.         {
  222. /*
  223. echo 'embedded content<br/>';
  224. */
  225.             // get config
  226. #           $allowed_double_doublequote_html = $wakka->GetConfigValue('double_doublequote_html');
  227.             $ddquotes_policy = $wakka->config['double_doublequote_html'];
  228. /*
  229. echo 'double quotes: '.$ddquotes_policy.'<br/>';
  230. */
  231.             // get embedded code
  232.             $embedded = $matches[1];
  233.             // handle embedded id attributes for 'safe' and 'raw'
  234.             if ($ddquotes_policy == 'safe' || $ddquotes_policy == 'raw')
  235.             {
  236.                 // get tags with id attributes
  237.                 $patTagWithId = '((<[a-z].*?)(id=("|\')(.*?)\\4)(.*?>))';
  238.                 // with PREG_SET_ORDER we get an array for each match: easy to use with list()!
  239.                 // we do the match case-insensitive so we catch uppercase HTML as well;
  240.                 // SafeHTML will treat this but 'raw' may end up with invalid code!
  241.                 $tags2 = preg_match_all('/'.$patTagWithId.'/i',$embedded,$matches2,PREG_SET_ORDER); # use backref to match both single and double quotes
  242. /*
  243. echo '# of matches (2): '.$tags2.'<br/>';
  244. echo '<!--found (set order):'."\n";
  245. print_r($matches2);
  246. echo '-->'."\n";
  247. */
  248.                 // step through code, replacing tags with ids with tags with new ('repaired') ids
  249.                 $tmpembedded = $embedded;
  250.                 $newembedded = '';
  251.                 for ($i=0; $i < $tags2; $i++)
  252.                 {
  253.                     list(,$tag,$tagstart,$attrid,$quote,$id,$tagend) = $matches2[$i];   # $attrid not needed, just for clarity
  254.                     $parts = explode($tag,$tmpembedded,2);              # split in two at matched tag
  255.                     if ($id != ($newid = $wakka->makeId('embed',$id)))  # replace if we got a new value
  256.                     {
  257. /*
  258. echo 'replacing tag - old id: '.$id.' new id: '.$newid.'<br/>';
  259. */
  260.                         $tag = $tagstart.'id='.$quote.$newid.$quote.$tagend;
  261.                     }
  262. /*
  263. echo "<!--old: $tag -->\n";
  264. echo "<!--new: $replacetag -->\n";
  265. */
  266.                     $newembedded .= $parts[0].$tag;                     # append (replacement) tag to first part
  267.                     $tmpembedded  = $parts[1];                          # after tag: next bit to handle
  268.                 }
  269.                 $newembedded .= $tmpembedded;                           # add last part
  270. /*
  271. echo '<!--translation:'."\n";
  272. echo $newembedded;
  273. echo '-->'."\n";
  274. */
  275.             }
  276.             // return (treated) embedded content according to config
  277.             // NOTE: we apply SafeHTML *after* id treatment so it won't be throwing away invalid ids that we're repairing instead!
  278.             switch ($ddquotes_policy)
  279.             {
  280.                 case 'safe':
  281.                     return $wakka->ReturnSafeHTML($newembedded);
  282.                 case 'raw':
  283.                     return $newembedded;                                # may still be invalid code - 'raw' will not be corrected!
  284.                 default:
  285.                     return $wakka->htmlspecialchars_ent($embedded);     # display only
  286.             }
  287.         }
  288.         // code text
  289.         else if (preg_match('/^% %(.*?)% %$/s', $thing, $matches))
  290.         {
  291.             /*
  292.              * Note: this routine is rewritten such that (new) language formatters
  293.              * will automatically be found, whether they are GeSHi language config files
  294.              * or "internal" Wikka formatters.
  295.              * Path to GeSHi language files and Wikka formatters MUST be defined in config.
  296.              * For line numbering (GeSHi only) a starting line can be specified after the language
  297.              * code, separated by a ; e.g., % %(php;27)....% %.
  298.              * Specifying >= 1 turns on line numbering if this is enabled in the configuration.
  299.              */
  300.             $code = $matches[1];
  301.             // if configuration path isn't set, make sure we'll get an invalid path so we
  302.             // don't match anything in the home directory
  303.             $geshi_hi_path = isset($wakka->config['geshi_languages_path']) ? $wakka->config['geshi_languages_path'] : '/:/';
  304.             $wikka_hi_path = isset($wakka->config['wikka_highlighters_path']) ? $wakka->config['wikka_highlighters_path'] : '/:/';
  305.             // check if a language (and starting line) has been specified
  306.             if (preg_match("/^\((.+?)(;([0-9]+))??\)(.*)$/s", $code, $matches))
  307.             {
  308.                 list(, $language, , $start, $code) = $matches;
  309.             }
  310.             // get rid of newlines at start and end (and preceding/following whitespace)
  311.             // Note: unlike trim(), this preserves any tabs at the start of the first "real" line
  312.             $code = preg_replace('/^\s*\n+|\n+\s*$/','',$code);
  313.  
  314.             // check if GeSHi path is set and we have a GeSHi hilighter for this language
  315.             if (isset($language) && isset($wakka->config['geshi_path']) && file_exists($geshi_hi_path.'/'.$language.'.php'))
  316.             {
  317.                 // use GeSHi for hilighting
  318.                 $output = $wakka->GeSHi_Highlight($code, $language, $start);
  319.             }
  320.             // check Wikka highlighter path is set and if we have an internal Wikka hilighter
  321.             elseif (isset($language) && isset($wakka->config['wikka_formatter_path']) && file_exists($wikka_hi_path.'/'.$language.'.php') && 'wakka' != $language)
  322.             {
  323.                 // use internal Wikka hilighter
  324.                 $output = '<div class="code">'."\n";
  325.                 $output .= $wakka->Format($code, $language);
  326.                 $output .= "</div>\n";
  327.             }
  328.             // no language defined or no formatter found: make default code block;
  329.             // IncludeBuffered() will complain if 'code' formatter doesn't exist
  330.             else
  331.             {
  332.                 $output = '<div class="code">'."\n";
  333.                 $output .= $wakka->Format($code, 'code');
  334.                 $output .= "</div>\n";
  335.             }
  336.  
  337.             #return $output;
  338.             // START DarTar modified 2005-02-17
  339.             // slight mod JavaWoman 2005-06-12: coding style, class for form
  340.                 //build form
  341.                 $form  = $wakka->FormOpen('grabcode','','post','','grabcode');
  342.                 $form .= '<input type="submit" name="save" class="grabcodebutton" style="line-height:10px; float:right; vertical-align: middle; margin-right:20px; margin-top:0px; font-size: 10px; color: #000; font-weight: normal; font-family: Verdana, Arial, sans-serif; background-color: #DDD; text-decoration: none; height:18px;" value="Grab" title="Download this code" />';
  343.                 $form .= '<input type="hidden" name="code" value="'.urlencode($code).'" />';
  344.                 $form .= $wakka->FormClose();
  345.                 // output
  346.                 return $output."\n".$form;
  347.             // END DarTar modified 2005-02-17
  348.  
  349.         }
  350.         // forced links
  351.         // \S : any character that is not a whitespace character
  352.         // \s : any whitespace character
  353.         else if (preg_match('/^\[\[(\S*)(\s+(.+))?\]\]$/s', $thing, $matches))      # recognize forced links across lines
  354.         {
  355.             list(, $url, , $text) = $matches;
  356.             if ($url)
  357.             {
  358.                 //if ($url!=($url=(preg_replace("/@@|&pound;&pound;||\[\[/","",$url))))$result="</span>";
  359.                 if (!$text) $text = $url;
  360.                 //$text=preg_replace("/@@|&pound;&pound;|\[\[/","",$text);
  361.                 return $result.$wakka->Link($url,'', $text);
  362.             }
  363.             else
  364.             {
  365.                 return '';
  366.             }
  367.         }
  368.         // indented text
  369.         # JW FIXED 2005-07-09 accented chars not used for ordered lists
  370.         # JW FIXED 2005-07-12 this does not cover the case where a list item is followed by an inline comment of the *same* level
  371.         # JW FIXED 2005-07-12 as with the expression in the /edit handler this does not cover tab or ~ at the start of the document
  372.         elseif (preg_match('/(^|\n)([\t~]+)(-|&|[0-9a-zA-Z]+\))?(\n|$)/s', $thing, $matches))
  373.         {
  374.             $br = 0;                                                        # no break needed after a block
  375.  
  376.             // get new indent level
  377.             $newIndentLevel = strlen($matches[2]);      # JW 2005-07-12 also match tab or ~ at start of document
  378.             // derive code indent
  379.             $codeIndent = str_repeat("\t",$newIndentLevel-1);
  380.             $nlTabs = "\n".$codeIndent;
  381.             $nlTabsOut = $nlTabs."\t";
  382.  
  383.             // find out which indent type we want
  384.             $newIndentType = $matches[3];               # JW 2005-07-12 also match tab or ~ at start of document
  385.             // derive code fragments
  386.             if     ($newIndentType == '')               # plain indent
  387.             {
  388.                 $opener = '<div class="indent">';
  389.                 $closer = '</div>'/*.$nlTabs*/;
  390.             }
  391.             elseif ($newIndentType == '-')              # unordered list
  392.             {
  393.                 $opener = '<ul>'.$nlTabs.'<li>';
  394.                 $closer = '</li>'.$nlTabs.'</ul>';
  395.             }
  396.             elseif ($newIndentType == '&')              # inline comment
  397.             {
  398.                 $opener = '<ul class="thread">'.$nlTabs.'<li>';
  399.                 $closer = '</li>'.$nlTabs.'</ul>';
  400.             }
  401.             else                                        # ordered list
  402.             {
  403.                 $opener = '<ol type="'.substr($newIndentType, 0, 1).'">'.$nlTabs.'<li>';
  404.                 $closer = '</li>'.$nlTabs.'</ol>';
  405.                 $newIndentType = 'o';
  406.             }
  407.  
  408.             // do an indent
  409.             if ($newIndentLevel > $oldIndentLevel)
  410.             {
  411.                 for ($i = 0; $i < $newIndentLevel - $oldIndentLevel; $i++)
  412.                 {
  413.                     $result .= $nlTabs./*'<!--nested item '.$newIndentLevel.'-->'.*/$opener;
  414.                     array_push($indentClosers, $closer);
  415.  
  416.                     #$result .= '<!--pushed type: '.$oldIndentType.' -->';  # @@@
  417.                     array_push($indentTypes, $oldIndentType);               # remember type hierarchically
  418.                 }
  419.             }
  420.             // do an outdent or stay at the same level
  421.             else if ($newIndentLevel <= $oldIndentLevel)
  422.             {
  423.                 $bOutdent = FALSE;
  424.                 if ($newIndentLevel < $oldIndentLevel)
  425.                 {
  426.                     $bOutdent = TRUE;                                           # remember we're outdenting, for correct layout
  427.                     // do the outdenting
  428.                     for ($i = 0; $i < $oldIndentLevel - $newIndentLevel; $i++)
  429.                     {
  430.                         if ($i > 0)
  431.                         {
  432.                             $result .= $nlTabsOut;
  433.                         }
  434.                         $result .= array_pop($indentClosers)/*.'<!--outdent to '.$newIndentLevel.'-->'*/;
  435.  
  436.                         $oldIndentType = array_pop($indentTypes);               # make sure we will compare with "correct" previous type
  437.                         #$result .= '<!--popped type: '.$oldIndentType.' -->';  # @@@
  438.                     }
  439.                 }
  440.                 if ($bOutdent)                                                  # outdenting: put close tag on new line
  441.                 {
  442.                     $result .= $nlTabs/*.'<!--outdent: close tag on new line-->'*/;
  443.                 }
  444.                 // JW 2005-07-11 new item of different type
  445.                 if ($newIndentType != $oldIndentType)
  446.                 {
  447.                     $result .= array_pop($indentClosers);
  448.                     $result .= /*'<!--type change follows (old: '.$oldIndentType.' new: '.$newIndentType.') -->'.*/$nlTabs.$opener;
  449.                     array_push($indentClosers, $closer);
  450.                 }
  451.                 // new item of same type
  452.                 else
  453.                 {
  454.                     // plain indent
  455.                     if ($newIndentType == '')
  456.                     {
  457.                         $result .= $closer./*'<!--same type ('.$newIndentType.') same level-->'.*/$nlTabs.$opener;
  458.                     }
  459.                     // list or inline comment
  460.                     else
  461.                     {
  462.                         $result .= '</li>'.$nlTabs.'<li>'/*.'<!--back to same type-->'*/;
  463.                     }
  464.                 }
  465.             }
  466.             $oldIndentType  = $newIndentType;                       # remember type sequentially
  467.             $oldIndentLevel = $newIndentLevel;
  468.  
  469.             return $result;
  470.         }
  471.         // new lines
  472.         else if ($thing == "\n")
  473.         {
  474.             // if we got here, there was no tab (or ~) in the next line; this means that we can close all open indents.
  475.             // JW: we need to do the same thing at the end of the page to close indents NOT followed by newline: use a function
  476.             /*
  477.             $c = count($indentClosers);
  478.             for ($i = 0; $i < $c; $i++)
  479.             {
  480.                 $result .= array_pop($indentClosers);
  481.                 $br = 0;
  482.             }
  483.             $oldIndentLevel = 0;
  484.             #$oldIndentLength= 0;               # superfluous
  485.             #$newIndentSpace=array();           # superfluous
  486.             */
  487.             $result .= close_indents($indentClosers,$oldIndentLevel);   # JW 2005-07-11 removed superfluous variables
  488.  
  489.             $result .= ($br) ? "<br />\n" : "\n";
  490.             $br = 1;
  491.             return $result;
  492.         }
  493.         // Actions
  494.         else if (preg_match('/^\{\{(.*?)\}\}$/s', $thing, $matches))
  495.         {
  496.             if ($matches[1])
  497.                 return $wakka->Action($matches[1]);
  498.             else
  499.                 return '{{}}';
  500.         }
  501.         // interwiki links!
  502.         else if (preg_match('/^[A-ZÄÖÜ][A-Za-zÄÖÜßäöü]+[:]\S*$/s', $thing))
  503.         {
  504.             return $wakka->Link($thing);
  505.         }
  506.         // wiki links!
  507.         else if (preg_match('/^[A-ZÄÖÜ]+[a-zßäöü]+[A-Z0-9ÄÖÜ][A-Za-z0-9ÄÖÜßäöü]*$/s', $thing))
  508.         {
  509.             return $wakka->Link($thing);
  510.         }
  511.         // separators
  512.         else if (preg_match('/-{4,}/', $thing, $matches))
  513.         {
  514.             // TODO: This could probably be improved for situations where someone puts text on the same line as a separator.
  515.             //       Which is a stupid thing to do anyway! HAW HAW! Ahem.
  516.             $br = 0;
  517.             return "<hr />\n";
  518.         }
  519.         // mind map xml
  520.         else if (preg_match('/^<map.*<\/map>$/s', $thing))
  521.         {
  522.             return $wakka->Action('mindmap '.$wakka->Href().'/mindmap.mm');
  523.         }
  524.         // if we reach this point, it must have been an accident.
  525.         // @@@ JW: or a detailed regex that excludes something that was included in the
  526.         //     preg_replace_callback expression
  527.         return $thing;
  528.     }
  529. }
  530.  
  531. if (!function_exists('wakka3callback'))
  532. {
  533.     /**
  534.      * "Afterburner" formatting: extra handling of already-generated XHTML code.
  535.      *
  536.      * 1.
  537.      * Ensure every heading has an id, either specified or generated. (May be
  538.      * extended to generate section TOC data.)
  539.      * If an id is specified, that is used without any modification.
  540.      * If no id is specified, it is generated on the basis of the heading context:
  541.      * - any image tag is replaced by its alt text (if specified)
  542.      * - all tags are stripped
  543.      * - all characters that are not valid in an id are stripped (except whitespace)
  544.      * - the resulting string is then used by makedId() to generate an id out of it
  545.      *
  546.      * @access  private
  547.      * @uses    Wakka::makeId()
  548.      *
  549.      * @param   array   $things required: matches of the regex in the preg_replace_callback
  550.      * @return  string  heading with an id attribute
  551.      */
  552.     function wakka3callback($things)
  553.     {
  554.         global $wakka;
  555.         $thing = $things[1];
  556.  
  557.         // heading
  558.         if (preg_match('#^<(h[1-6])(.*?)>(.*?)</\\1>$#s', $thing, $matches))    # note that we don't match headings that are not valid XHTML!
  559.         {
  560. /*
  561. echo 'heading:<pre>';
  562. print_r($matches);
  563. echo '</pre>';
  564. */
  565.             list($element,$tagname,$attribs,$heading) = $matches;
  566.  
  567.             #if (preg_match('/(id=("|\')(.*?)\\2)/',$attribs,$matches)) # use backref to match both single and double quotes
  568.             if (preg_match('/(id=("|\')(.*?)\\2)/',$attribs))           # use backref to match both single and double quotes
  569.             {
  570.                 // existing id attribute: nothing to do (assume already treated as embedded code)
  571.                 // @@@ we *may* want to gather ids and heading text for a TOC here ...
  572.                 // heading text should then get partly the same treatment as when we're creating ids:
  573.                 // at least replace images and strip tags - we can leave entities etc. alone - so we end up with
  574.                 // plain text-only
  575.                 // do this if we have a condition set to generate a TOC
  576.                 return $element;
  577.             }
  578.             else
  579.             {
  580.                 // no id: we'll have to create one
  581. #echo 'no id provided - create one<br/>';
  582.                 $tmpheading = trim($heading);
  583.                 // first find and replace any image with its alt text
  584.                 // @@@ can we use preg_match_all here? would it help?
  585.                 while (preg_match('/(<img.*?alt=("|\')(.*?)\\2.*?>)/',$tmpheading,$matches))
  586.                 {
  587. #echo 'image found: '.$tmpheading.'<br/>';
  588.                     # 1 = whole element
  589.                     # 3 = alt text
  590.                     list(,$element, ,$alttext) = $matches;
  591. /*
  592. echo 'embedded image:<pre>';
  593. print_r($matches);
  594. echo '</pre>';
  595. */
  596.                     // gather data for replacement
  597.                     $search  = '/'.str_replace('/','\/',$element).'/';  # whole element (delimiter chars escaped!) @@@ use preg_quote as well?
  598.                     $replace = trim($alttext);                          # alt text
  599. /*
  600. echo 'pat_repl:<pre>';
  601. echo 'search: '.$search.'<br/>';
  602. echo 'search: '.$replace.'<br/>';
  603. echo '</pre>';
  604. */
  605.                     // now replace img tag by corresponding alt text
  606.                     $tmpheading = preg_replace($search,$replace,$tmpheading);   # replace image by alt text
  607.                 }
  608.                 $headingtext = $tmpheading;
  609. #echo 'headingtext (no img): '.$headingtext.'<br/>';
  610.         // @@@ 2005-05-27 now first replace linebreaks <br/> with spaces!!
  611.                 // remove all other tags
  612.                 $headingtext = strip_tags($headingtext);
  613. #echo 'headingtext (no tags): '.$headingtext.'<br/>';
  614.                 // @@@ this all-text result is usable for a TOC!!!
  615.                 // do this if we have a condition set to generate a TOC
  616.  
  617.                 // replace entities that can be interpreted
  618.                 // use default charset ISO-8859-1 because other chars won't be valid for an id anyway
  619.                 $headingtext = html_entity_decode($headingtext,ENT_NOQUOTES);
  620.                 // remove any remaining entities (so we don't end up with strange words and numbers in the id text)
  621.                 $headingtext = preg_replace('/&[#]?.+?;/','',$headingtext);
  622. #echo 'headingtext (entities decoded/removed): '.$headingtext.'<br/>';
  623.                 // finally remove non-id characters (except whitespace which is handled by makeId())
  624.                 $headingtext = preg_replace('/[^A-Za-z0-9_:.-\s]/','',$headingtext);
  625. #echo 'headingtext (id-ready): '.$headingtext.'<br/>';
  626.                 // now create id based on resulting heading text
  627.                 $id = $wakka->makeId('hn',$headingtext);
  628. #echo 'id: '.$id.'<br/>';
  629.  
  630.                 // rebuild element, adding id
  631.                 return '<'.$tagname.$attribs.' id="'.$id.'">'.$heading.'</'.$tagname.'>';
  632.             }
  633.         }
  634.         // other elements to be treated go here (tables, images, code sections...)
  635.     }
  636. }
  637.  
  638.  
  639. // ------------- do the work -----------
  640.  
  641.  
  642. $text = str_replace("\r\n", "\n", $text);
  643.  
  644. // replace 4 consecutive spaces at the beginning of a line with tab character
  645. // $text = preg_replace("/\n[ ]{4}/", "\n\t", $text); // moved to edit.php
  646.  
  647. if ($this->method == 'show') $mind_map_pattern = '<map.*?<\/map>|'; else $mind_map_pattern = '';
  648.  
  649. // define entity patterns
  650. // NOTE most also used in wikka.php for htmlentities_ent(): REGEX library!
  651. $alpha  = '[a-z]+';                         # character entity reference
  652. $numdec = '#[0-9]+';                        # numeric character reference (decimal)
  653. $numhex = '#x[0-9a-f]+';                    # numeric character reference (hexadecimal)
  654. $terminator = ';|(?=($|[\n<]|&lt;))';       # semicolon; or end-of-string, newline or tag
  655. $entitypat = '('.$alpha.'|'.$numdec.'|'.$numhex.')('.$terminator.')';   # defines entity pattern without the starting &
  656. $entityref = '&'.$entitypat;                # entity reference
  657. $loneamp = '&(?!'.$entitypat.')';           # ampersand NOT part of an entity
  658.  
  659. $this->callLevel++;                                         # JW 2005-07-15 recursion level: getting in
  660.     '/('.
  661.     '% %.*?% %|'.                                                                           # code
  662.     '"".*?""|'.                                                                         # literal
  663.     $mind_map_pattern.
  664.     '\[\[[^\[]*?\]\]|'.                                                                 # forced link
  665.     '-{4,}|---|'.                                                                       # separator, new line
  666.     '\b[a-z]+:\/\/\S+|'.                                                                # URL
  667.     '\*\*|\'\'|\#\#|\#\%|@@|::c::|\>\>|\<\<|&pound;&pound;|&yen;&yen;|\+\+|__|\/\/|'.   # Wiki markup
  668.     '======|=====|====|===|==|'.                                                        # headings
  669.     '(^|\n)([\t~]+)(-|&|[0-9a-zA-Z]+\))?|'.                                             # indents and lists # JW FIXED 2005-07-12 also match tab or ~ at start of document
  670.     '\{\{.*?\}\}|'.                                                                     # action
  671.     '\b[A-ZÄÖÜ][A-Za-zÄÖÜßäöü]+[:](?![=_])\S*\b|'.                                        # InterWiki link
  672.     '\b([A-ZÄÖÜ]+[a-zßäöü]+[A-Z0-9ÄÖÜ][A-Za-z0-9ÄÖÜßäöü]*)\b|'.                            # CamelWords
  673.     '<|>|'.                                                                             # HTML special chars - after wiki markup!
  674.     $loneamp.'|'.                                                                       # HTML special chars - ampersand NOT part of an enity
  675.     '\n'.                                                                               # new line
  676.     ')/ms','wakka2callback',$text);
  677.  
  678. // we're cutting the last <br />
  679. $text = preg_replace('/<br \/>$/','',$text);
  680. $this->callLevel--;                                         # JW 2005-07-15 recursion level: getting out
  681. if ($this->callLevel == 0)                                  # JW 2005-07-15 only for "outmost" call level
  682. {
  683.     $text .= wakka2callback('closetags');                   # JW changed logic
  684. }
  685.  
  686. // add ids to heading elements
  687. // @@@ LATER:
  688. // - extend with other elements (tables, images, code blocks)
  689. // - also create array(s) for TOC(s)
  690. $idstart = getmicrotime();
  691.     '#('.
  692.     '<h[1-6].*?>.*?</h[1-6]>'.
  693.     // other elements to be treated go here
  694.     ')#ms','wakka3callback',$text);
  695. printf('<!-- Header id generation took %.6f seconds -->', (getmicrotime() - $idstart));
  696.  
  697. echo $text;
  698.  
  699. ?>


Make sure you replace every occurrence of '% %' in this code with '%%'!


Supporting code


Only a single new WikkaCore core method is needed for this improved formatter (other new functions are part of the formatter script itself):

makeId()


Used here to both for handling ids in embedded HTML code and to generate a unique id for headings; see GenerateUniqueId for the code and where to insert it.


Todo


Bugs

Extensions


Test? Comments?


Go ahead and test it - either on your own Wikka installation or on this site where it is now Installed as a WikkaBetaFeatures beta feature.

Comments and suggestions are more than welcome, as always.



CategoryDevelopmentFormatters
There are 7 comments on this page. [Show comments]
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki