PCRE2 LICENCE
-------------

PCRE2 is a library of functions to support regular expressions whose syntax
and semantics are as close as possible to those of the Perl 5 language.

Releases 10.00 and above of PCRE2 are distributed under the terms of the "BSD"
licence, as specified below, with one exemption for certain binary
redistributions. The documentation for PCRE2, supplied in the "doc" directory,
is distributed under the same terms as the software itself. The data in the
testdata directory is not copyrighted and is in the public domain.

The basic library functions are written in C and are freestanding. Also
included in the distribution is a just-in-time compiler that can be used to
optimize pattern matching. This is an optional feature that can be omitted when
the library is built.


THE BASIC LIBRARY FUNCTIONS
---------------------------

Written by:       Philip Hazel
Email local part: ph10
Email domain:     cam.ac.uk

University of Cambridge Computing Service,
Cambridge, England.

Copyright (c) 1997-2019 University of Cambridge
All rights reserved.


PCRE2 JUST-IN-TIME COMPILATION SUPPORT
--------------------------------------

Written by:       Zoltan Herczeg
Email local part: hzmester
Email domain:     freemail.hu

Copyright(c) 2010-2019 Zoltan Herczeg
All rights reserved.


STACK-LESS JUST-IN-TIME COMPILER
--------------------------------

Written by:       Zoltan Herczeg
Email local part: hzmester
Email domain:     freemail.hu

Copyright(c) 2009-2019 Zoltan Herczeg
All rights reserved.


THE "BSD" LICENCE
-----------------

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright notices,
      this list of conditions and the following disclaimer.

    * Redistributions in binary form must reproduce the above copyright
      notices, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.

    * Neither the name of the University of Cambridge nor the names of any
      contributors may be used to endorse or promote products derived from this
      software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.


EXEMPTION FOR BINARY LIBRARY-LIKE PACKAGES
------------------------------------------

The second condition in the BSD licence (covering binary redistributions) does
not apply all the way down a chain of software. If binary package A includes
PCRE2, it must respect the condition, but if package B is software that
includes package A, the condition is not imposed on package B unless it uses
PCRE2 independently.

End
PCRE2 LICENCE

Please see the file LICENCE in the PCRE2 distribution for licensing details.

End
Change Log for PCRE2
--------------------


Version 10.34 21-November-2019
------------------------------

1. The maximum number of capturing subpatterns is 65535 (documented), but no
check on this was ever implemented. This omission has been rectified; it fixes
ClusterFuzz 14376.

2. Improved the invalid utf32 support of the JIT compiler. Now it correctly
detects invalid characters in the 0xd800-0xdfff range.

3. Fix minor typo bug in JIT compile when \X is used in a non-UTF string.

4. Add support for matching in invalid UTF strings to the pcre2_match()
interpreter, and integrate with the existing JIT support via the new
PCRE2_MATCH_INVALID_UTF compile-time option.

5. Give more error detail for invalid UTF-8 when detected in pcre2grep.

6. Add support for invalid UTF-8 to pcre2grep.

7. Adjust the limit for "must have" code unit searching, in particular,
increase it substantially for non-anchored patterns.

8. Allow (*ACCEPT) to be quantified, because an ungreedy quantifier with a zero
minimum is potentially useful.

9. Some changes to the way the minimum subject length is handled:

   * When PCRE2_NO_START_OPTIMIZE is set, no minimum length is computed;
     pcre2test now omits this item instead of showing a value of zero.

   * An incorrect minimum length could be calculated for a pattern that
     contained (*ACCEPT) inside a qualified group whose minimum repetition was
     zero, for example /A(?:(*ACCEPT))?B/, which incorrectly computed a minimum
     of 2. The minimum length scan no longer happens for a pattern that
     contains (*ACCEPT).

   * When no minimum length is set by the normal scan, but a first and/or last
     code unit is recorded, set the minimum to 1 or 2 as appropriate.

   * When a pattern contains multiple groups with the same number, a back
     reference cannot know which one to scan for a minimum length. This used to
     cause the minimum length finder to give up with no result. Now it treats
     such references as not adding to the minimum length (which it should have
     done all along).

   * Furthermore, the above action now happens only if the back reference is to
     a group that exists more than once in a pattern instead of any back
     reference in a pattern with duplicate numbers.

10. A (*MARK) value inside a successful condition was not being returned by the
interpretive matcher (it was returned by JIT). This bug has been mended.

11. A bug in pcre2grep meant that -o without an argument (or -o0) didn't work
if the pattern had more than 32 capturing parentheses. This is fixed. In
addition (a) the default limit for groups requested by -o<n> has been raised to
50, (b) the new --om-capture option changes the limit, (c) an error is raised
if -o asks for a group that is above the limit.

12. The quantifier {1} was always being ignored, but this is incorrect when it
is made possessive and applied to an item in parentheses, because a
parenthesized item may contain multiple branches or other backtracking points,
for example /(a|ab){1}+c/ or /(a+){1}+a/.

13. For partial matches, pcre2test was always showing the maximum lookbehind
characters, flagged with "<", which is misleading when the lookbehind didn't
actually look behind the start (because it was later in the pattern). Showing
all consulted preceding characters for partial matches is now controlled by the
existing "allusedtext" modifier and, as for complete matches, this facility is
available only for non-JIT matching, because JIT does not maintain the first
and last consulted characters.

14. DFA matching (using pcre2_dfa_match()) was not recognising a partial match
if the end of the subject was encountered in a lookahead (conditional or
otherwise), an atomic group, or a recursion.

15. Give error if pcre2test -t, -T, -tm or -TM is given an argument of zero.

16. Check for integer overflow when computing lookbehind lengths. Fixes
Clusterfuzz issue 15636.

17. Implemented non-atomic positive lookaround assertions.

18. If a lookbehind contained a lookahead that contained another lookbehind
within it, the nested lookbehind was not correctly processed. For example, if
/(?<=(?=(?<=a)))b/ was matched to "ab" it gave no match instead of matching
"b".

19. Implemented pcre2_get_match_data_size().

20. Two alterations to partial matching:

    (a) The definition of a partial match is slightly changed: if a pattern
    contains any lookbehinds, an empty partial match may be given, because this
    is another situation where adding characters to the current subject can
    lead to a full match. Example: /c*+(?<=[bc])/ with subject "ab".

    (b) Similarly, if a pattern could match an empty string, an empty partial
    match may be given. Example: /(?![ab]).*/ with subject "ab". This case
    applies only to PCRE2_PARTIAL_HARD.

    (c) An empty string partial hard match can be returned for \z and \Z as it
    is documented that they shouldn't match.

21. A branch that started with (*ACCEPT) was not being recognized as one that
could match an empty string.

22. Corrected pcre2_set_character_tables() tables data type: was const unsigned
char * instead of const uint8_t *, as generated by pcre2_maketables().

23. Upgraded to Unicode 12.1.0.

24. Add -jitfast command line option to pcre2test (to make all the jit options
available directly).

25. Make pcre2test -C show if libreadline or libedit is supported.

26. If the length of one branch of a group exceeded 65535 (the maximum value
that is remembered as a minimum length), the whole group's length was
incorrectly recorded as 65535, leading to incorrect "no match" when start-up
optimizations were in force.

27. The "rightmost consulted character" value was not always correct; in
particular, if a pattern ended with a negative lookahead, characters that were
inspected in that lookahead were not included.

28. Add the pcre2_maketables_free() function.

29. The start-up optimization that looks for a unique initial matching
code unit in the interpretive engines uses memchr() in 8-bit mode. When the
search is caseless, it was doing so inefficiently, which ended up slowing down
the match drastically when the subject was very long. The revised code (a)
remembers if one case is not found, so it never repeats the search for that
case after a bumpalong and (b) when one case has been found, it searches only
up to that position for an earlier occurrence of the other case. This fix
applies to both interpretive pcre2_match() and to pcre2_dfa_match().

30. While scanning to find the minimum length of a group, if any branch has
minimum length zero, there is no need to scan any subsequent branches (a small
compile-time performance improvement).

31. Installed a .gitignore file on a user's suggestion. When using the svn
repository with git (through git svn) this helps keep it tidy.

32. Add underflow check in JIT which may occur when the value of subject
string pointer is close to 0.

33. Arrange for classes such as [Aa] which contain just the two cases of the
same character, to be treated as a single caseless character. This causes the
first and required code unit optimizations to kick in where relevant.

34. Improve the bitmap of starting bytes for positive classes that include wide
characters, but no property types, in UTF-8 mode. Previously, on encountering
such a class, the bits for all bytes greater than \xc4 were set, thus
specifying any character with codepoint >= 0x100. Now the only bits that are
set are for the relevant bytes that start the wide characters. This can give a
noticeable performance improvement.

35. If the bitmap of starting code units contains only 1 or 2 bits, replace it
with a single starting code unit (1 bit) or a caseless single starting code
unit if the two relevant characters are case-partners. This is particularly
relevant to the 8-bit library, though it applies to all. It can give a
performance boost for patterns such as [Ww]ord and (word|WORD). However, this
optimization doesn't happen if there is a "required" code unit of the same
value (because the search for a "required" code unit starts at the match start
for non-unique first code unit patterns, but after a unique first code unit,
and patterns such as a*a need the former action).

36. Small patch to pcre2posix.c to set the erroroffset field to -1 immediately
after a successful compile, instead of at the start of matching to avoid a
sanitizer complaint (regexec is supposed to be thread safe).

37. Add NEON vectorization to JIT to speed up matching of first character and
pairs of characters on ARM64 CPUs.

38. If a non-ASCII character was the first in a starting assertion in a
caseless match, the "first code unit" optimization did not get the casing
right, and the assertion failed to match a character in the other case if it
did not start with the same code unit.

39. Fixed the incorrect computation of jump sizes on x86 CPUs in JIT. A masking
operation was incorrectly removed in r1136. Reported by Ralf Junker.


Version 10.33 16-April-2019
---------------------------

1. Added "allvector" to pcre2test to make it easy to check the part of the
ovector that shouldn't be changed, in particular after substitute and failed or
partial matches.

2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
a greater than 1 fixed quantifier. This issue was found by Yunho Kim.

3. Added support for callouts from pcre2_substitute(). After 10.33-RC1, but
prior to release, fixed a bug that caused a crash if pcre2_substitute() was
called with a NULL match context.

4. The POSIX functions are now all called pcre2_regcomp() etc., with wrapper
functions that use the standard POSIX names. However, in pcre2posix.h the POSIX
names are defined as macros. This should help avoid linking with the wrong
library in some environments while still exporting the POSIX names for
pre-existing programs that use them. (The Debian alternative names are also
defined as macros, but not documented.)

5. Fix an xclass matching issue in JIT.

6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).

7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
characterized by a lower case letter following (* and to simplify coding for
this, the character tables created by pcre2_maketables() were updated to add a
new "is lower case letter" bit. At the same time, the now unused "is
hexadecimal digit" bit was removed. The default tables in
src/pcre2_chartables.c.dist are updated.

8. Implement the new Perl "script run" features (*script_run:...) and
(*atomic_script_run:...) aka (*sr:...) and (*asr:...).

9. Fixed two typos in change 22 for 10.21, which added special handling for
ranges such as a-z in EBCDIC environments. The original code probably never
worked, though there were no bug reports.

10. Implement PCRE2_COPY_MATCHED_SUBJECT for pcre2_match() (including JIT via
pcre2_match()) and pcre2_dfa_match(), but *not* the pcre2_jit_match() fast
path. Also, when a match fails, set the subject field in the match data to NULL
for tidiness - none of the substring extractors should reference this after
match failure.

11. If a pattern started with a subroutine call that had a quantifier with a
minimum of zero, an incorrect "match must start with this character" could be
recorded. Example: /(?&xxx)*ABC(?<xxx>XYZ)/ would (incorrectly) expect 'A' to
be the first character of a match.

12. The heap limit checking code in pcre2_dfa_match() could suffer from
overflow if the heap limit was set very large. This could cause incorrect "heap
limit exceeded" errors.

13. Add "kibibytes" to the heap limit output from pcre2test -C to make the
units clear.

14. Add a call to pcre2_jit_free_unused_memory() in pcre2grep, for tidiness.

15. Updated the VMS-specific code in pcre2test on the advice of a VMS user.

16. Removed the unnecessary inclusion of stdint.h (or inttypes.h) from
pcre2_internal.h as it is now included by pcre2.h. Also, change 17 for 10.32
below was unnecessarily complicated, as inttypes.h is a Standard C header,
which is defined to be a superset of stdint.h. Instead of conditionally
including stdint.h or inttypes.h, pcre2.h now unconditionally includes
inttypes.h. This supports environments that do not have stdint.h but do have
inttypes.h, which are known to exist. A note in the autotools documentation
says (November 2018) that there are none known that are the other way round.

17. Added --disable-percent-zt to "configure" (and equivalent to CMake) to
forcibly disable the use of %zu and %td in formatting strings because there is
at least one version of VMS that claims to be C99 but does not support these
modifiers.

18. Added --disable-pcre2grep-callout-fork, which restricts the callout support
in pcre2grep to the inbuilt echo facility. This may be useful in environments
that do not support fork().

19. Fix two instances of <= 0 being applied to unsigned integers (the VMS
compiler complains).

20. Added "fork" support for VMS to pcre2grep, for running an external program
via a string callout.

21. Improve MAP_JIT flag usage on MacOS. Patch by Rich Siegel.

22. If a pattern started with (*MARK), (*COMMIT), (*PRUNE), (*SKIP), or (*THEN)
followed by ^ it was not recognized as anchored.

23. The RunGrepTest script used to cut out the test of NUL characters for
Solaris and MacOS as printf and sed can't handle them. It seems that the *BSD
systems can't either. I've inverted the test so that only those OS that are
known to work (currently only Linux) try to run this test.

24. Some tests in RunGrepTest appended to testtrygrep from two different file
descriptors instead of redirecting stderr to stdout. This worked on Linux, but
it was reported not to on other systems, causing the tests to fail.

25. In the RunTest script, make the test for stack setting use the same value
for the stack as it needs for -bigstack.

26. Insert a cast in pcre2_dfa_match.c to suppress a compiler warning.

26. With PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL set, escape sequences such as \s
which are valid in character classes, but not as the end of ranges, were being
treated as literals. An example is [_-\s] (but not [\s-_] because that gave an
error at the *start* of a range). Now an "invalid range" error is given
independently of PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.

27. Related to 26 above, PCRE2_BAD_ESCAPE_IS_LITERAL was affecting known escape
sequences such as \eX when they appeared invalidly in a character class. Now
the option applies only to unrecognized or malformed escape sequences.

28. Fix word boundary in JIT compiler. Patch by Mike Munday.

29. The pcre2_dfa_match() function was incorrectly handling conditional version
tests such as (?(VERSION>=0)...) when the version test was true. Incorrect
processing or a crash could result.

30. When PCRE2_UTF is set, allow non-ASCII letters and decimal digits in group
names, as Perl does. There was a small bug in this new code, found by
ClusterFuzz 12950, fixed before release.

31. Implemented PCRE2_EXTRA_ALT_BSUX to support ECMAScript 6's \u{hhh}
construct.

32. Compile \p{Any} to be the same as . in DOTALL mode, so that it benefits
from auto-anchoring if \p{Any}* starts a pattern.

33. Compile invalid UTF check in JIT test when only pcre32 is enabled.

34. For some time now, CMake has been warning about the setting of policy
CMP0026 to "OLD" in CmakeLists.txt, and hinting that the feature might be
removed in a future version. A request for CMake expertise on the list produced
no result, so I have now hacked CMakeLists.txt along the lines of some changes
I found on the Internet. The new code no longer needs the policy setting, and
it appears to work fine on Linux.

35. Setting --enable-jit=auto for an out-of-tree build failed because the
source directory wasn't in the search path for AC_TRY_COMPILE always. Patch
from Ross Burton.

36. Disable SSE2 JIT optimizations in x86 CPUs when SSE2 is not available.
Patch by Guillem Jover.

37. Changed expressions such as 1<<10 to 1u<<10 in many places because compiler
warnings were reported.

38. Using the clang compiler with sanitizing options causes runtime complaints
about truncation for statments such as x = ~x when x is an 8-bit value; it
seems to compute ~x as a 32-bit value. Changing such statements to x = 255 ^ x
gets rid of the warnings. There were also two missing casts in pcre2test.


Version 10.32 10-September-2018
-------------------------------

1. When matching using the the REG_STARTEND feature of the POSIX API with a
non-zero starting offset, unset capturing groups with lower numbers than a
group that did capture something were not being correctly returned as "unset"
(that is, with offset values of -1).

2. When matching using the POSIX API, pcre2test used to omit listing unset
groups altogether. Now it shows those that come before any actual captures as
"<unset>", as happens for non-POSIX matching.

3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
whatever the build configuration was. It now correctly says "\R matches all
Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
specified. Similarly, running "pcre2test -C bsr" never produced the result
ANY.

4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
multi-code-unit characters caused bad behaviour and possibly a crash. This
issue was fixed for other kinds of repeat in release 10.20 by change 19, but
repeating character classes were overlooked.

5. pcre2grep now supports the inclusion of binary zeros in patterns that are
read from files via the -f option.

6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.

7. Added --enable-jit=auto support to configure.ac.

8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
modes for the benefit of m68k, where pointers can be 16-bit aligned. The
dummies force 32-bit alignment and this ensures that the structure is a
multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
architectures, alignment requirements take care of this automatically.

9. When returning an error from pcre2_pattern_convert(), ensure the error
offset is set zero for early errors.

10. A number of patches for Windows support from Daniel Richard G:

  (a) List of error numbers in Runtest.bat corrected (it was not the same as in
      Runtest).

  (b) pcre2grep snprintf() workaround as used elsewhere in the tree.

  (c) Support for non-C99 snprintf() that returns -1 in the overflow case.

11. Minor tidy of pcre2_dfa_match() code.

12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
use the stack for local workspace and local ovectors. Instead, an initial block
of stack is reserved, but if this is insufficient, heap memory is used. The
heap limit parameter now applies to pcre2_dfa_match().

13. If a "find limits" test of DFA matching in pcre2test resulted in too many
matches for the ovector, no matches were displayed.

14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
EOF. The test looks to have come from a fuzzer.

15. If PCRE2 was built with a default match limit a lot greater than the
default default of 10 000 000, some JIT tests of the match limit no longer
failed. All such tests now set 10 000 000 as the upper limit.

16. Another Windows related patch for pcregrep to ensure that WIN32 is
undefined under Cygwin.

17. Test for the presence of stdint.h and inttypes.h in configure and CMake and
include whichever exists (stdint preferred) instead of unconditionally
including stdint. This makes life easier for old and non-standard systems.

18. Further changes to improve portability, especially to old and or non-
standard systems:

  (a) Put all printf arguments in RunGrepTest into single, not double, quotes,
      and use \0 not \x00 for binary zero.

  (b) Avoid the use of C++ (i.e. BCPL) // comments.

  (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of
      these now, if using MSVC or a standard C before C99, %lu is used with a
      cast if necessary.

19. Applied a contributed patch to CMakeLists.txt to increase the stack size
when linking pcre2test with MSVC. This gets rid of a stack overflow error in
the standard set of tests.

20. Output a warning in pcre2test when ignoring the "altglobal" modifier when
it is given with the "replace" modifier.

21. In both pcre2test and pcre2_substitute(), with global matching, a pattern
that matched an empty string, but never at the starting match offset, was not
handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such
a pattern. Because \G is in a lookbehind assertion, there has to be a
"bumpalong" before there can be a match. The automatic "advance by one
character after an empty string match" rule is therefore inappropriate. A more
complicated algorithm has now been implemented.

22. When checking to see if a lookbehind is of fixed length, lookaheads were
correctly ignored, but qualifiers on lookaheads were not being ignored, leading
to an incorrect "lookbehind assertion is not fixed length" error.

23. The VERSION condition test was reading fractional PCRE2 version numbers
such as the 04 in 10.04 incorrectly and hence giving wrong results.

24. Updated to Unicode version 11.0.0. As well as the usual addition of new
scripts and characters, this involved re-jigging the grapheme break property
algorithm because Unicode has changed the way emojis are handled.

25. Fixed an obscure bug that struck when there were two atomic groups not
separated by something with a backtracking point. There could be an incorrect
backtrack into the first of the atomic groups. A complicated example is
/(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP
shouldn't find a MARK (because is in an atomic group), but it did.

26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set
a list of modifiers for all subsequent patterns - only those that the script
recognizes are meaningful; (2) #subject lines can be used to set or unset a
default "mark" modifier; (3) Unsupported #command lines give a warning when
they are ignored; (4) Mark data is output only if the "mark" modifier is
present.

27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.

28. A (*MARK) name was not being passed back for positive assertions that were
terminated by (*ACCEPT).

29. Add support for \N{U+dddd}, but only in Unicode mode.

30. Add support for (?^) for unsetting all imnsx options.

31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose
code point was less than 256 and that were recognized by the lookup table
generated by pcre2_maketables(), which uses isspace() to identify white space.
Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,
U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by
Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.

32. In certain circumstances, option settings within patterns were not being
correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly
matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the
end of its group during the parse process, but without another setting such as
(?m) the compile phase got it right.) This bug was introduced by the
refactoring in release 10.23.

33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to
define memmove() as function call to bcopy(). This hasn't been tested for a
long time because in pcre2test the result of memmove() was being used, whereas
bcopy() doesn't return a result. This feature is now refactored always to call
an emulation function when there is no memmove(). The emulation makes use of
bcopy() when available.

34. When serializing a pattern, set the memctl, executable_jit, and tables
fields (that is, all the fields that contain pointers) to zeros so that the
result of serializing is always the same. These fields are re-set when the
pattern is deserialized.

35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated
negative class with no characters less than 0x100 followed by a positive class
with only characters less than 0x100, the first class was incorrectly being
auto-possessified, causing incorrect match failures.

36. Removed the character type bit ctype_meta, which dates from PCRE1 and is
not used in PCRE2.

37. Tidied up unnecessarily complicated macros used in the escapes table.

38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted
from distribution tarballs, owing to a typo in Makefile.am which had
testoutput8-16-3 twice. Now fixed.

39. If the only branch in a conditional subpattern was anchored, the whole
subpattern was treated as anchored, when it should not have been, since the
assumed empty second branch cannot be anchored. Demonstrated by test patterns
such as /(?(1)^())b/ or /(?(?=^))b/.

40. A repeated conditional subpattern that could match an empty string was
always assumed to be unanchored. Now it it checked just like any other
repeated conditional subpattern, and can be found to be anchored if the minimum
quantifier is one or more. I can't see much use for a repeated anchored
pattern, but the behaviour is now consistent.

41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint
(for an event that could never occur but you had to have external information
to know that).

42. If before the first match in a file that was being searched by pcre2grep
there was a line that was sufficiently long to cause the input buffer to be
expanded, the variable holding the location of the end of the previous match
was being adjusted incorrectly, and could cause an overflow warning from a code
sanitizer. However, as the value is used only to print pending "after" lines
when the next match is reached (and there are no such lines in this case) this
bug could do no damage.


Version 10.31 12-February-2018
------------------------------

1. Fix typo (missing ]) in VMS code in pcre2test.c.

2. Replace the replicated code for matching extended Unicode grapheme sequences
(which got a lot more complicated by change 10.30/49) by a single subroutine
that is called by both pcre2_match() and pcre2_dfa_match().

3. Add idempotent guard to pcre2_internal.h.

4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and
PCRE2_CONFIG_COMPILED_WIDTHS.

5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is
defined (e.g. by --enable-never-backslash-C).

6. Defined public names for all the pcre2_compile() error numbers, and used
the public names in pcre2_convert.c.

7. Fixed a small memory leak in pcre2test (convert contexts).

8. Added two casts to compile.c and one to match.c to avoid compiler warnings.

9. Added code to pcre2grep when compiled under VMS to set the symbol
PCRE2GREP_RC to the exit status, because VMS does not distinguish between
exit(0) and exit(1).

10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain
about a bad option only if the following argument item does not start with a
hyphen.

11. pcre2grep was truncating components of file names to 128 characters when
processing files with the -r option, and also (some very odd code) truncating
path names to 512 characters. There is now a check on the absolute length of
full path file names, which may be up to 2047 characters long.

12. When an assertion contained (*ACCEPT) it caused all open capturing groups
to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to
misbehaviour for subsequent references to groups that started outside the
assertion. ACCEPT in an assertion now closes only those groups that were
started within that assertion. Fixes oss-fuzz issues 3852 and 3891.

13. Multiline matching in pcre2grep was misbehaving if the pattern matched
within a line, and then matched again at the end of the line and over into
subsequent lines. Behaviour was different with and without colouring, and
sometimes context lines were incorrectly printed and/or line endings were lost.
All these issues should now be fixed.

14. If --line-buffered was specified for pcre2grep when input was from a
compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be
ignored for compressed files.)

15. Although pcre2_jit_match checks whether the pattern is compiled
in a given mode, it was also expected that at least one mode is available.
This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION
when the pattern is not optimized by JIT at all.

16. The line number and related variables such as match counts in pcre2grep
were all int variables, causing overflow when files with more than 2147483647
lines were processed (assuming 32-bit ints). They have all been changed to
unsigned long ints.

17. If a backreference with a minimum repeat count of zero was first in a
pattern, apart from assertions, an incorrect first matching character could be
recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set
as the first character of a match.

18. Characters in a leading positive assertion are considered for recording a
first character of a match when the rest of the pattern does not provide one.
However, a character in a non-assertive group within a leading assertion such
as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an
infelicity rather than an outright bug, because it did not affect the result of
a match, just its speed. (In fact, in this case, the starting 'a' was
subsequently picked up in the study.)

19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"
instead of "RRETURN" saves unwinding the backtracks in these cases (only one
didn't).

20. Allocate a single callout block on the stack at the start of pcre2_match()
and set its never-changing fields once only. Do the same for pcre2_dfa_match().

21. Save the extra compile options (set in the compile context) with the
compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS
to retrieve them, and update pcre2test to show them.

22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new
field callout_flags in callout blocks. The bits are set by pcre2_match(), but
not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts
if the callout_extra subject modifier is set. These bits are provided to help
with tracking how a backtracking match is proceeding.

23. Updated the pcre2demo.c demonstration program, which was missing the extra
code for -g that handles the case when \K in an assertion causes the match to
end at the original start point. Also arranged for it to detect when \K causes
the end of a match to be before its start.

24. Similar to 23 above, strange things (including loops) could happen in
pcre2grep when \K was used in an assertion when --colour was used or in
multiline mode. The "end at original start point" bug is fixed, and if the end
point is found to be before the start point, they are swapped.

25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT
matching (both pcre2_match() and pcre2_dfa_match()) and the matched string
started with the first code unit of a newline sequence, matching failed because
it was not tried at the newline.

26. Code for giving up a non-partial match after failing to find a starting
code unit anywhere in the subject was missing when searching for one of a
number of code units (the bitmap case) in both pcre2_match() and
pcre2_dfa_match(). This was a missing optimization rather than a bug.

27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a
pointer argument rather than a code unit value. This should not have affected
the generated code.

28. The JIT compiler has been updated.

29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().
This could not actually cause a crash because it was always used in a memcpy()
call with zero length.

30. Some internal structures have a variable-length ovector[] as their last
element. Their actual memory is obtained dynamically, giving an ovector of
appropriate length. However, they are defined in the structure as
ovector[NUMBER], where NUMBER is large so that array bound checkers don't
grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing
groups, making the ovector larger than this. The number has been increased to
131072, which allows for the maximum number of captures (65535) plus the
overall match. This fixes oss-fuzz issue 5415.

31. Auto-possessification at the end of a capturing group was dependent on what
follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused
incorrect behaviour when the group was called recursively from elsewhere in the
pattern where something different might follow. This bug is an unforseen
consequence of change #1 for 10.30 - the implementation of backtracking into
recursions. Iterators at the ends of capturing groups are no longer considered
for auto-possessification if the pattern contains any recursions. Fixes
Bugzilla #2232.


Version 10.30 14-August-2017
----------------------------

1. The main interpreter, pcre2_match(), has been refactored into a new version
that does not use recursive function calls (and therefore the stack) for
remembering backtracking positions. This makes --disable-stack-for-recursion a
NOOP. The new implementation allows backtracking into recursive group calls in
patterns, making it more compatible with Perl, and also fixes some other
hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
the old code had a number of fudges to try to reduce stack usage. It seems to
run no slower than the old code.

A number of bugs in the refactored code were subsequently fixed during testing
before release, but after the code was made available in the repository. These
bugs were never in fully released code, but are noted here for the record.

  (a) If a pattern had fewer capturing parentheses than the ovector supplied in
      the match data block, a memory error (detectable by ASAN) occurred after
      a match, because the external block was being set from non-existent
      internal ovector fields. Fixes oss-fuzz issue 781.

  (b) A pattern with very many capturing parentheses (when the internal frame
      size was greater than the initial frame vector on the stack) caused a
      crash. A vector on the heap is now set up at the start of matching if the
      vector on the stack is not big enough to handle at least 10 frames.
      Fixes oss-fuzz issue 783.

  (c) Handling of (*VERB)s in recursions was wrong in some cases.

  (d) Captures in negative assertions that were used as conditions were not
      happening if the assertion matched via (*ACCEPT).

  (e) Mark values were not being passed out of recursions.

  (f) Refactor some code in do_callout() to avoid picky compiler warnings about
      negative indices. Fixes oss-fuzz issue 1454.

  (g) Similarly refactor the way the variable length ovector is addressed for
      similar reasons. Fixes oss-fuzz issue 1465.

2. Now that pcre2_match() no longer uses recursive function calls (see above),
the "match limit recursion" value seems misnamed. It still exists, and limits
the depth of tree that is searched. To avoid future confusion, it has been
renamed as "depth limit" in all relevant places (--with-depth-limit,
(*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
available for backwards compatibility.

3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:

  (a) Check for malloc failures when getting memory for the ovector (POSIX) or
      the match data block (non-POSIX).

4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
for a character with a code point greater than 0x10ffff (the Unicode maximum)
caused a crash.

5. If a lookbehind assertion that contained a back reference to a group
appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
undefined actions (often a segmentation fault) could occur, depending on what
other options were set. An example assertion is (?<!\1(abc)) where the
reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.

6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
pcre2test to use it to output the frame size when the "framesize" modifier is
given.

7. Reworked the recursive pattern matching in the JIT compiler to follow the
interpreter changes.

8. When the zero_terminate modifier was specified on a pcre2test subject line
for global matching, unpredictable things could happen. For example, in UTF-8
mode, the pattern //g,zero_terminate read random memory when matched against an
empty string with zero_terminate. This was a bug in pcre2test, not the library.

9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
of the section that is compiled when Unix-style directory scanning is
available, and into a new section that is always compiled for Windows.

10. In pcre2test, explicitly close the file after an error during serialization
or deserialization (the "load" or "save" commands).

11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.

12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
a NULL pattern pointer when Unicode support is available.

13. When the 32-bit library was being tested by pcre2test, error messages that
were longer than 64 code units could cause a buffer overflow. This was a bug in
pcre2test.

14. The alternative matching function, pcre2_dfa_match() misbehaved if it
encountered a character class with a possessive repeat, for example [a-f]{3}+.

15. The depth (formerly recursion) limit now applies to DFA matching (as
of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
matching to find the minimum value for this limit.

16. Since 10.21, if pcre2_match() was called with a null context, default
memory allocation functions were used instead of whatever was used when the
pattern was compiled.

17. Changes to the pcre2test "memory" modifier on a subject line. These apply
only to pcre2_match():

  (a) Warn if null_context is set on both pattern and subject, because the
      memory details cannot then be shown.

  (b) Remember (up to a certain number of) memory allocations and their
      lengths, and list only the lengths, so as to be system-independent.
      (In practice, the new interpreter never has more than 2 blocks allocated
      simultaneously.)

18. Make pcre2test detect an error return from pcre2_get_error_message(), give
a message, and abandon the run (this would have detected #13 above).

19. Implemented PCRE2_ENDANCHORED.

20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
the --output=text (-O) option and the inbuilt callout echo.

21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
single-branch conditions with a false condition (e.g. DEFINE) at the start of a
branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
anchored.

22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
heap limit along with other pattern information, and to find the minimum when
the find_limits modifier is set.

23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
pattern is set up so as to initialize any padding the compiler might have
included. This avoids valgrind warnings when a compiled pattern is copied, in
particular when it is serialized.

24. Remove a redundant line of code left in accidentally a long time ago.

25. Remove a duplication typo in pcre2_tables.c

26. Correct an incorrect cast in pcre2_valid_utf.c

27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
tests to improve coverage.

28. Some fixes/tidies as a result of looking at Coverity Scan output:

    (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
    (b) Added some casts to avoid "suspicious implicit sign extension".
    (c) Resource leaks in pcre2test in rare error cases.
    (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
        for checking at compile time that tables are the right size.
    (e) Add missing "fall through" comment.

29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.

30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.

31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
pcre2test, a crash could occur.

32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so
that all the tests can run with clang's sanitizing options.

33. Implement extra compile options in the compile context and add the first
one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.

34. Implement newline type PCRE2_NEWLINE_NUL.

35. A lookbehind assertion that had a zero-length branch caused undefined
behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.

36. The match limit value now also applies to pcre2_dfa_match() as there are
patterns that can use up a lot of resources without necessarily recursing very
deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.

37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.

38. Fix returned offsets from regexec() when REG_STARTEND is used with a
starting offset greater than zero.

39. Implement REG_PEND (GNU extension) for the POSIX wrapper.

40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
pattern lines.

41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.

42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
of pcre2grep.

43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:

    (a) The -F option did not work for fixed strings containing \E.
    (b) The -w option did not work for patterns with multiple branches.

44. Added configuration options for the SELinux compatible execmem allocator in
JIT.

45. Increased the limit for searching for a "must be present" code unit in
subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
much faster.

46. Arrange for anchored patterns to record and use "first code unit" data,
because this can give a fast "no match" without searching for a "required code
unit". Previously only non-anchored patterns did this.

47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.

48. Add the callout_no_where modifier to pcre2test.

49. Update extended grapheme breaking rules to the latest set that are in
Unicode Standard Annex #29.

50. Added experimental foreign pattern conversion facilities
(pcre2_pattern_convert() and friends).

51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
is defined in a system header in cygwin. Also modified some of the #ifdefs in
pcre2grep related to Windows and Cygwin support.

52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
character class is the last character in the class, Perl does not give a
warning. PCRE2 now also treats this as a literal.

53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
not doing so for [\d-X] (and similar escapes), as is documented.

54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.

55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in
pcre2_compile() which could never actually trigger (code should have been cut
out when Unicode support is disabled).


Version 10.23 14-February-2017
------------------------------

1. Extended pcre2test with the utf8_input modifier so that it is able to
generate all possible 16-bit and 32-bit code unit values in non-UTF modes.

2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
PCRE2_UCP set, a negative character type such as \D in a positive class should
cause all characters greater than 255 to match, whatever else is in the class.
There was a bug that caused this not to happen if a Unicode property item was
added to such a class, for example [\D\P{Nd}] or [\W\pL].

3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
checking is now done in the pre-pass that identifies capturing groups. This has
reduced the amount of duplication and made the code tidier. While doing this,
some minor bugs and Perl incompatibilities were fixed, including:

  (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
      of giving an invalid quantifier error.

  (b) {0} can now be used after a group in a lookbehind assertion; previously
      this caused an "assertion is not fixed length" error.

  (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
      the name "DEFINE" exists. PCRE2 now does likewise.

  (d) A recursion condition test such as (?(R2)...) must now refer to an
      existing subpattern.

  (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
      group whose name began with "R".

  (f) When testing zero-terminated patterns under valgrind, the terminating
      zero is now marked "no access". This catches bugs that would otherwise
      show up only with non-zero-terminated patterns.

  (g) A hyphen appearing immediately after a POSIX character class (for example
      /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
      literal, but gives a warning, so it seems best to fail it in PCRE.

  (h) An empty \Q\E sequence may appear after a callout that precedes an
      assertion condition (it is, of course, ignored).

One effect of the refactoring is that some error numbers and messages have
changed, and the pattern offset given for compiling errors is not always the
right-most character that has been read. In particular, for a variable-length
lookbehind assertion it now points to the start of the assertion. Another
change is that when a callout appears before a group, the "length of next
pattern item" that is passed now just gives the length of the opening
parenthesis item, not the length of the whole group. A length of zero is now
given only for a callout at the end of the pattern. Automatic callouts are no
longer inserted before and after explicit callouts in the pattern.

A number of bugs in the refactored code were subsequently fixed during testing
before release, but after the code was made available in the repository. Many
of the bugs were discovered by fuzzing testing. Several of them were related to
the change from assuming a zero-terminated pattern (which previously had
required non-zero terminated strings to be copied). These bugs were never in
fully released code, but are noted here for the record.

  (a) An overall recursion such as (?0) inside a lookbehind assertion was not
      being diagnosed as an error.

  (b) In utf mode, the length of a *MARK (or other verb) name was being checked
      in characters instead of code units, which could lead to bad code being
      compiled, leading to unpredictable behaviour.

  (c) In extended /x mode, characters whose code was greater than 255 caused
      a lookup outside one of the global tables. A similar bug existed for wide
      characters in *VERB names.

  (d) The amount of memory needed for a compiled pattern was miscalculated if a
      lookbehind contained more than one toplevel branch and the first branch
      was of length zero.

  (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
      terminated pattern, if a # comment ran on to the end of the pattern, one
      or more code units past the end were being read.

  (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
      "{2,2") could cause reading beyond the pattern.

  (g) When reading a callout string, if the end delimiter was at the end of the
      pattern one further code unit was read.

  (h) An unterminated number after \g' could cause reading beyond the pattern.

  (i) An insufficient memory size was being computed for compiling with
      PCRE2_AUTO_CALLOUT.

  (j) A conditional group with an assertion condition used more memory than was
      allowed for it during parsing, so too many of them could therefore
      overrun a buffer.

  (k) If parsing a pattern exactly filled the buffer, the internal test for
      overrun did not check when the final META_END item was added.

  (l) If a lookbehind contained a subroutine call, and the called group
      contained an option setting such as (?s), and the PCRE2_ANCHORED option
      was set, unpredictable behaviour could occur. The underlying bug was
      incorrect code and insufficient checking while searching for the end of
      the called subroutine in the parsed pattern