1Changes between release 2.5.4 (11Sep96) and release 2.5.3: 2 3 - Fixed a bug introduced in 2.5.3 that blew it when a call 4 to input() occurred at the end of an input file. 5 6 - Fixed scanner skeleton so the example in the man page of 7 scanning strings using exclusive start conditions works. 8 9 - Minor Makefile tweaks. 10 11 12Changes between release 2.5.3 (29May96) and release 2.5.2: 13 14 - Some serious bugs in yymore() have been fixed. In particular, 15 when using AT&T-lex-compatibility or %array, you can intermix 16 calls to input(), unput(), and yymore(). (This still doesn't 17 work for %pointer, and isn't likely to in the future.) 18 19 - A bug in handling NUL's in the input stream of scanners using 20 REJECT has been fixed. 21 22 - The default main() in libfl.a now repeatedly calls yylex() until 23 it returns 0, rather than just calling it once. 24 25 - Minor tweak for Windows NT Makefile, MISC/NT/Makefile. 26 27 28Changes between release 2.5.2 (25Apr95) and release 2.5.1: 29 30 - The --prefix configuration option now works. 31 32 - A bug that completely broke the "-Cf" table compression 33 option has been fixed. 34 35 - A major headache involving "const" declarators and Solaris 36 systems has been fixed. 37 38 - An octal escape sequence in a flex regular expression must 39 now contain only the digits 0-7. 40 41 - You can now use "--" on the flex command line to mark the 42 end of flex options. 43 44 - You can now specify the filename '-' as a synonym for stdin. 45 46 - By default, the scanners generated by flex no longer 47 statically initialize yyin and yyout to stdin and stdout. 48 This change is necessary because in some ANSI environments, 49 stdin and stdout are not compile-time constant. You can 50 force the initialization using "%option stdinit" in the first 51 section of your flex input. 52 53 - "%option nounput" now correctly omits the unput() routine 54 from the output. 55 56 - "make clean" now removes config.log, config.cache, and the 57 flex binary. The fact that it removes the flex binary means 58 you should take care if making changes to scan.l, to make 59 sure you don't wind up in a bootstrap problem. 60 61 - In general, the Makefile has been reworked somewhat (thanks 62 to Francois Pinard) for added flexibility - more changes will 63 follow in subsequent releases. 64 65 - The .texi and .info files in MISC/texinfo/ have been updated, 66 thanks also to Francois Pinard. 67 68 - The FlexLexer::yylex(istream* new_in, ostream* new_out) method 69 now does not have a default for the first argument, to disambiguate 70 it from FlexLexer::yylex(). 71 72 - A bug in destructing a FlexLexer object before doing any scanning 73 with it has been fixed. 74 75 - A problem with including FlexLexer.h multiple times has been fixed. 76 77 - The alloca() chud necessary to accommodate bison has grown 78 even uglier, but hopefully more correct. 79 80 - A portability tweak has been added to accommodate compilers that 81 use char* generic pointers. 82 83 - EBCDIC contact information in the file MISC/EBCDIC has been updated. 84 85 - An OS/2 Makefile and config.h for flex 2.5 is now available in 86 MISC/OS2/, contributed by Kai Uwe Rommel. 87 88 - The descrip.mms file for building flex under VMS has been updated, 89 thanks to Pat Rankin. 90 91 - The notes on building flex for the Amiga have been updated for 92 flex 2.5, contributed by Andreas Scherer. 93 94 95Changes between release 2.5.1 (28Mar95) and release 2.4.7: 96 97 - A new concept of "start condition" scope has been introduced. 98 A start condition scope is begun with: 99 100 <SCs>{ 101 102 where SCs is a list of one or more start conditions. Inside 103 the start condition scope, every rule automatically has the 104 prefix <SCs> applied to it, until a '}' which matches the 105 initial '{'. So, for example: 106 107 <ESC>{ 108 "\\n" return '\n'; 109 "\\r" return '\r'; 110 "\\f" return '\f'; 111 "\\0" return '\0'; 112 } 113 114 is equivalent to: 115 116 <ESC>"\\n" return '\n'; 117 <ESC>"\\r" return '\r'; 118 <ESC>"\\f" return '\f'; 119 <ESC>"\\0" return '\0'; 120 121 As indicated in this example, rules inside start condition scopes 122 (and any rule, actually, other than the first) can be indented, 123 to better show the extent of the scope. 124 125 Start condition scopes may be nested. 126 127 - The new %option directive can be used in the first section of 128 a flex scanner to control scanner-generation options. Most 129 options are given simply as names, optionally preceded by the 130 word "no" (with no intervening whitespace) to negate their 131 meaning. Some are equivalent to flex flags, so putting them 132 in your scanner source is equivalent to always specifying 133 the flag (%option's take precedence over flags): 134 135 7bit -7 option 136 8bit -8 option 137 align -Ca option 138 backup -b option 139 batch -B option 140 c++ -+ option 141 caseful opposite of -i option (caseful is the default); 142 case-sensitive same as above 143 caseless -i option; 144 case-insensitive same as above 145 debug -d option 146 default opposite of -s option 147 ecs -Ce option 148 fast -F option 149 full -f option 150 interactive -I option 151 lex-compat -l option 152 meta-ecs -Cm option 153 perf-report -p option 154 read -Cr option 155 stdout -t option 156 verbose -v option 157 warn opposite of -w option (so use "%option nowarn" for -w) 158 159 array equivalent to "%array" 160 pointer equivalent to "%pointer" (default) 161 162 Some provide new features: 163 164 always-interactive generate a scanner which always 165 considers its input "interactive" (no call to isatty() 166 will be made when the scanner runs) 167 main supply a main program for the scanner, which 168 simply calls yylex(). Implies %option noyywrap. 169 never-interactive generate a scanner which never 170 considers its input "interactive" (no call to isatty() 171 will be made when the scanner runs) 172 stack if set, enable start condition stacks (see below) 173 stdinit if unset ("%option nostdinit"), initialize yyin 174 and yyout statically to nil FILE* pointers, instead 175 of stdin and stdout 176 yylineno if set, keep track of the current line 177 number in global yylineno (this option is expensive 178 in terms of performance). The line number is available 179 to C++ scanning objects via the new member function 180 lineno(). 181 yywrap if unset ("%option noyywrap"), scanner does not 182 call yywrap() upon EOF but simply assumes there 183 are no more files to scan 184 185 Flex scans your rule actions to determine whether you use the 186 REJECT or yymore features (this is not new). Two %options can be 187 used to override its decision, either by setting them to indicate 188 the feature is indeed used, or unsetting them to indicate it 189 actually is not used: 190 191 reject 192 yymore 193 194 Three %option's take string-delimited values, offset with '=': 195 196 outfile="<name>" equivalent to -o<name> 197 prefix="<name>" equivalent to -P<name> 198 yyclass="<name>" set the name of the C++ scanning class 199 (see below) 200 201 A number of %option's are available for lint purists who 202 want to suppress the appearance of unneeded routines in 203 the generated scanner. Each of the following, if unset, 204 results in the corresponding routine not appearing in the 205 generated scanner: 206 207 input, unput 208 yy_push_state, yy_pop_state, yy_top_state 209 yy_scan_buffer, yy_scan_bytes, yy_scan_string 210 211 You can specify multiple options with a single %option directive, 212 and multiple directives in the first section of your flex input file. 213 214 - The new function: 215 216 YY_BUFFER_STATE yy_scan_string( const char *str ) 217 218 returns a YY_BUFFER_STATE (which also becomes the current input 219 buffer) for scanning the given string, which occurs starting 220 with the next call to yylex(). The string must be NUL-terminated. 221 A related function: 222 223 YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len ) 224 225 creates a buffer for scanning "len" bytes (including possibly NUL's) 226 starting at location "bytes". 227 228 Note that both of these functions create and scan a *copy* of 229 the string/bytes. (This may be desirable, since yylex() modifies 230 the contents of the buffer it is scanning.) You can avoid the 231 copy by using: 232 233 YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size ) 234 235 which scans in place the buffer starting at "base", consisting 236 of "size" bytes, the last two bytes of which *must* be 237 YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning 238 consists of base[0] through base[size-2], inclusive). If you 239 fail to set up "base" in this manner, yy_scan_buffer returns a 240 nil pointer instead of creating a new input buffer. 241 242 The type yy_size_t is an integral type to which you can cast 243 an integer expression reflecting the size of the buffer. 244 245 - Three new routines are available for manipulating stacks of 246 start conditions: 247 248 void yy_push_state( int new_state ) 249 250 pushes the current start condition onto the top of the stack 251 and BEGIN's "new_state" (recall that start condition names are 252 also integers). 253 254 void yy_pop_state() 255 256 pops the top of the stack and BEGIN's to it, and 257 258 int yy_top_state() 259 260 returns the top of the stack without altering the stack's 261 contents. 262 263 The start condition stack grows dynamically and so has no built-in 264 size limitation. If memory is exhausted, program execution 265 is aborted. 266 267 To use start condition stacks, your scanner must include 268 a "%option stack" directive. 269 270 - flex now supports POSIX character class expressions. These 271 are expressions enclosed inside "[:" and ":]" delimiters (which 272 themselves must appear between the '[' and ']' of a character 273 class; other elements may occur inside the character class, too). 274 The expressions flex recognizes are: 275 276 [:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:] 277 [:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:] 278 279 These expressions all designate a set of characters equivalent to 280 the corresponding isXXX function (for example, [:alnum:] designates 281 those characters for which isalnum() returns true - i.e., any 282 alphabetic or numeric). Some systems don't provide isblank(), 283 so flex defines [:blank:] as a blank or a tab. 284 285 For example, the following character classes are all equivalent: 286 287 [[:alnum:]] 288 [[:alpha:][:digit:] 289 [[:alpha:]0-9] 290 [a-zA-Z0-9] 291 292 If your scanner is case-insensitive (-i flag), then [:upper:] 293 and [:lower:] are equivalent to [:alpha:]. 294 295 - The promised rewrite of the C++ FlexLexer class has not yet 296 been done. Support for FlexLexer is limited at the moment to 297 fixing show-stopper bugs, so, for example, the new functions 298 yy_scan_string() & friends are not available to FlexLexer 299 objects. 300 301 - The new macro 302 303 yy_set_interactive(is_interactive) 304 305 can be used to control whether the current buffer is considered 306 "interactive". An interactive buffer is processed more slowly, 307 but must be used when the scanner's input source is indeed 308 interactive to avoid problems due to waiting to fill buffers 309 (see the discussion of the -I flag in flex.1). A non-zero value 310 in the macro invocation marks the buffer as interactive, a zero 311 value as non-interactive. Note that use of this macro overrides 312 "%option always-interactive" or "%option never-interactive". 313 314 yy_set_interactive() must be invoked prior to beginning to 315 scan the buffer. 316 317 - The new macro 318 319 yy_set_bol(at_bol) 320 321 can be used to control whether the current buffer's scanning 322 context for the next token match is done as though at the 323 beginning of a line (non-zero macro argument; makes '^' anchored 324 rules active) or not at the beginning of a line (zero argument, 325 '^' rules inactive). 326 327 - Related to this change, the mechanism for determining when a scan is 328 starting at the beginning of a line has changed. It used to be 329 that '^' was active iff the character prior to that at which the 330 scan started was a newline. The mechanism now is that '^' is 331 active iff the last token ended in a newline (or the last call to 332 input() returned a newline). For most users, the difference in 333 mechanisms is negligible. Where it will make a difference, 334 however, is if unput() or yyless() is used to alter the input 335 stream. When in doubt, use yy_set_bol(). 336 337 - The new beginning-of-line mechanism involved changing some fairly 338 twisted code, so it may have introduced bugs - beware ... 339 340 - The macro YY_AT_BOL() returns true if the next token scanned from 341 the current buffer will have '^' rules active, false otherwise. 342 343 - The new function 344 345 void yy_flush_buffer( struct yy_buffer_state* b ) 346 347 flushes the contents of the current buffer (i.e., next time 348 the scanner attempts to match a token using b as the current 349 buffer, it will begin by invoking YY_INPUT to fill the buffer). 350 This routine is also available to C++ scanners (unlike some 351 of the other new routines). 352 353 The related macro 354 355 YY_FLUSH_BUFFER 356 357 flushes the contents of the current buffer. 358 359 - A new "-ooutput" option writes the generated scanner to "output". 360 If used with -t, the scanner is still written to stdout, but 361 its internal #line directives (see previous item) use "output". 362 363 - Flex now generates #line directives relating the code it 364 produces to the output file; this means that error messages 365 in the flex-generated code should be correctly pinpointed. 366 367 - When generating #line directives, filenames with embedded '\'s 368 have those characters escaped (i.e., turned into '\\'). This 369 feature helps with reporting filenames for some MS-DOS and OS/2 370 systems. 371 372 - The FlexLexer class includes two new public member functions: 373 374 virtual void switch_streams( istream* new_in = 0, 375 ostream* new_out = 0 ) 376 377 reassigns yyin to new_in (if non-nil) and yyout to new_out 378 (ditto), deleting the previous input buffer if yyin is 379 reassigned. It is used by: 380 381 int yylex( istream* new_in = 0, ostream* new_out = 0 ) 382 383 which first calls switch_streams() and then returns the value 384 of calling yylex(). 385 386 - C++ scanners now have yy_flex_debug as a member variable of 387 FlexLexer rather than a global, and member functions for testing 388 and setting it. 389 390 - When generating a C++ scanning class, you can now use 391 392 %option yyclass="foo" 393 394 to inform flex that you have derived "foo" as a subclass of 395 yyFlexLexer, so flex will place your actions in the member 396 function foo::yylex() instead of yyFlexLexer::yylex(). It also 397 generates a yyFlexLexer::yylex() member function that generates a 398 run-time error if called (by invoking yyFlexLexer::LexerError()). 399 This feature is necessary if your subclass "foo" introduces some 400 additional member functions or variables that you need to access 401 from yylex(). 402 403 - Current texinfo files in MISC/texinfo, contributed by Francois 404 Pinard. 405 406 - You can now change the name "flex" to something else (e.g., "lex") 407 by redefining $(FLEX) in the Makefile. 408 409 - Two bugs (one serious) that could cause "bigcheck" to fail have 410 been fixed. 411 412 - A number of portability/configuration changes have been made 413 for easier portability. 414 415 - You can use "YYSTATE" in your scanner as an alias for YY_START 416 (for AT&T lex compatibility). 417 418 - input() now maintains yylineno. 419 420 - input() no longer trashes yytext. 421 422 - interactive scanners now read characters in YY_INPUT up to a 423 newline, a large performance gain. 424 425 - C++ scanner objects now work with the -P option. You include 426 <FlexLexer.h> once per scanner - see comments in <FlexLexer.h> 427 (or flex.1) for details. 428 429 - C++ FlexLexer objects now use the "cerr" stream to report -d output 430 instead of stdio. 431 432 - The -c flag now has its full glorious POSIX interpretation (do 433 nothing), rather than being interpreted as an old-style -C flag. 434 435 - Scanners generated by flex now include two #define's giving 436 the major and minor version numbers (YY_FLEX_MAJOR_VERSION, 437 YY_FLEX_MINOR_VERSION). These can then be tested to see 438 whether certain flex features are available. 439 440 - Scanners generated using -l lex compatibility now have the symbol 441 YY_FLEX_LEX_COMPAT #define'd. 442 443 - When initializing (i.e., yy_init is non-zero on entry to yylex()), 444 generated scanners now set yy_init to zero before executing 445 YY_USER_INIT. This means that you can set yy_init back to a 446 non-zero value in YY_USER_INIT if you need the scanner to be 447 reinitialized on the next call. 448 449 - You can now use "#line" directives in the first section of your 450 scanner specification. 451 452 - When generating full-table scanners (-Cf), flex now puts braces 453 around each row of the 2-d array initialization, to silence warnings 454 on over-zealous compilers. 455 456 - Improved support for MS-DOS. The flex sources have been successfully 457 built, unmodified, for Borland 4.02 (all that's required is a 458 Borland Makefile and config.h file, which are supplied in 459 MISC/Borland - contributed by Terrence O Kane). 460 461 - Improved support for Macintosh using Think C - the sources should 462 build for this platform "out of the box". Contributed by Scott 463 Hofmann. 464 465 - Improved support for VMS, in MISC/VMS/, contributed by Pat Rankin. 466 467 - Support for the Amiga, in MISC/Amiga/, contributed by Andreas 468 Scherer. Note that the contributed files were developed for 469 flex 2.4 and have not been tested with flex 2.5. 470 471 - Some notes on support for the NeXT, in MISC/NeXT, contributed 472 by Raf Schietekat. 473 474 - The MISC/ directory now includes a preformatted version of flex.1 475 in flex.man, and pre-yacc'd versions of parse.y in parse.{c,h}. 476 477 - The flex.1 and flexdoc.1 manual pages have been merged. There 478 is now just one document, flex.1, which includes an overview 479 at the beginning to help you find the section you need. 480 481 - Documentation now clarifies that start conditions persist across 482 switches to new input files or different input buffers. If you 483 want to e.g., return to INITIAL, you must explicitly do so. 484 485 - The "Performance Considerations" section of the manual has been 486 updated. 487 488 - Documented the "yy_act" variable, which when YY_USER_ACTION is 489 invoked holds the number of the matched rule, and added an 490 example of using yy_act to profile how often each rule is matched. 491 492 - Added YY_NUM_RULES, a definition that gives the total number 493 of rules in the file, including the default rule (even if you 494 use -s). 495 496 - Documentation now clarifies that you can pass a nil FILE* pointer 497 to yy_create_buffer() or yyrestart() if you've arrange YY_INPUT 498 to not need yyin. 499 500 - Documentation now clarifies that YY_BUFFER_STATE is a pointer to 501 an opaque "struct yy_buffer_state". 502 503 - Documentation now stresses that you gain the benefits of removing 504 backing-up states only if you remove *all* of them. 505 506 - Documentation now points out that traditional lex allows you 507 to put the action on a separate line from the rule pattern if 508 the pattern has trailing whitespace (ugh!), but flex doesn't 509 support this. 510 511 - A broken example in documentation of the difference between 512 inclusive and exclusive start conditions is now fixed. 513 514 - Usage (-h) report now goes to stdout. 515 516 - Version (-V) info now goes to stdout. 517 518 - More #ifdef chud has been added to the parser in attempt to 519 deal with bison's use of alloca(). 520 521 - "make clean" no longer deletes emacs backup files (*~). 522 523 - Some memory leaks have been fixed. 524 525 - A bug was fixed in which dynamically-expanded buffers were 526 reallocated a couple of bytes too small. 527 528 - A bug was fixed which could cause flex to read and write beyond 529 the end of the input buffer. 530 531 - -S will not be going away. 532 533 534Changes between release 2.4.7 (03Aug94) and release 2.4.6: 535 536 - Fixed serious bug in reading multiple files. 537 538 - Fixed bug in scanning NUL's. 539 540 - Fixed bug in input() returning 8-bit characters. 541 542 - Fixed bug in matching text with embedded NUL's when 543 using %array or lex compatibility. 544 545 - Fixed multiple invocations of YY_USER_ACTION when using '|' 546 continuation action. 547 548 - Minor prototyping fixes. 549 550Changes between release 2.4.6 (04Jan94) and release 2.4.5: 551 552 - Linking with -lfl no longer required if your program includes 553 its own yywrap() and main() functions. (This change will cause 554 problems if you have a non-ANSI compiler on a system for which 555 sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).) 556 557 - The use of 'extern "C++"' in FlexLexer.h has been modified to 558 get around an incompatibility with g++'s header files. 559 560Changes between release 2.4.5 (11Dec93) and release 2.4.4: 561 562 - Fixed bug breaking C++ scanners that use REJECT or variable 563 trailing context. 564 565 - Fixed serious input problem for interactive scanners on 566 systems for which char is unsigned. 567 568 - Fixed bug in incorrectly treating '$' operator as variable 569 trailing context. 570 571 - Fixed bug in -CF table representation that could lead to 572 corrupt tables. 573 574 - Fixed fairly benign memory leak. 575 576 - Added `extern "C++"' wrapper to FlexLexer.h header. This 577 should overcome the g++ 2.5.X problems mentioned in the 578 NEWS for release 2.4.3. 579 580 - Changed #include of FlexLexer.h to use <> instead of "". 581 582 - Added feature to control whether the scanner attempts to 583 refill the input buffer once it's exhausted. This feature 584 will be documented in the 2.5 release. 585 586 587Changes between release 2.4.4 (07Dec93) and release 2.4.3: 588 589 - Fixed two serious bugs in scanning 8-bit characters. 590 591 - Fixed bug in YY_USER_ACTION that caused it to be executed 592 inappropriately (on the scanner's own internal actions, and 593 with incorrect yytext/yyleng values). 594 595 - Fixed bug in pointing yyin at a new file and resuming scanning. 596 597 - Portability fix regarding min/max/abs macros conflicting with 598 function definitions in standard header files. 599 600 - Added a virtual LexerError() method to the C++ yyFlexLexer class 601 for reporting error messages instead of always using cerr. 602 603 - Added warning in flexdoc that the C++ scanning class is presently 604 experimental and subject to considerable change between major 605 releases. 606 607 608Changes between release 2.4.3 (03Dec93) and release 2.4.2: 609 610 - Fixed bug causing fatal scanner messages to fail to print. 611 612 - Fixed things so FlexLexer.h can be included in other C++ 613 sources. One side-effect of this change is that -+ and -CF 614 are now incompatible. 615 616 - libfl.a now supplies private versions of the the <string.h>/ 617 <strings.h> string routines needed by flex and the scanners 618 it generates, to enhance portability to some BSD systems. 619 620 - More robust solution to 2.4.2's flexfatal() bug fix. 621 622 - Added ranlib of installed libfl.a. 623 624 - Some lint tweaks. 625 626 - NOTE: problems have been encountered attempting to build flex 627 C++ scanners using g++ version 2.5.X. The problem is due to an 628 unfortunate heuristic in g++ 2.5.X that attempts to discern between 629 C and C++ headers. Because FlexLexer.h is installed (by default) 630 in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X 631 decides that it's a C header :-(. So if you have problems, install 632 the header in /usr/local/lib/g++-include instead. 633 634 635Changes between release 2.4.2 (01Dec93) and release 2.4.1: 636 637 - Fixed bug in libfl.a referring to non-existent "flexfatal" function. 638 639 - Modified to produce both compress'd and gzip'd tar files for 640 distributions (you probably don't care about this change!). 641 642 643Changes between release 2.4.1 (30Nov93) and release 2.3.8: 644 645 - The new '-+' flag instructs flex to generate a C++ scanner class 646 (thanks to Kent Williams). flex writes an implementation of the 647 class defined in FlexLexer.h to lex.yy.cc. You may include 648 multiple scanner classes in your program using the -P flag. Note 649 that the scanner class also provides a mechanism for creating 650 reentrant scanners. The scanner class uses C++ streams for I/O 651 instead of FILE*'s (thanks to Tom Epperly). If the flex executable's 652 name ends in '+' then the '-+' flag is automatically on, so creating 653 a symlink or copy of "flex" to "flex++" results in a version of 654 flex that can be used exclusively for C++ scanners. 655 656 Note that without the '-+' flag, flex-generated scanners can still 657 be compiled using C++ compilers, though they use FILE*'s for I/O 658 instead of streams. 659 660 See the "GENERATING C++ SCANNERS" section of flexdoc for details. 661 662 - The new '-l' flag turns on maximum AT&T lex compatibility. In 663 particular, -l includes support for "yylineno" and makes yytext 664 be an array instead of a pointer. It does not, however, do away 665 with all incompatibilities. See the "INCOMPATIBILITIES WITH LEX 666 AND POSIX" section of flexdoc for details. 667 668 - The new '-P' option specifies a prefix to use other than "yy" 669 for the scanner's globally-visible variables, and for the 670 "lex.yy.c" filename. Using -P you can link together multiple 671 flex scanners in the same executable. 672 673 - The distribution includes a "texinfo" version of flexdoc.1, 674 contributed by Roland Pesch (thanks also to Marq Kole, who 675 contributed another version). It has not been brought up to 676 date, but reflects version 2.3. See MISC/flex.texinfo. 677 678 The flex distribution will soon include G.T. Nicol's flex 679 manual; he is presently bringing it up-to-date for version 2.4. 680 681 - yywrap() is now a function, and you now *must* link flex scanners 682 with libfl.a. 683 684 - Site-configuration is now done via an autoconf-generated 685 "configure" script contributed by Francois Pinard. 686 687 - Scanners now use fread() (or getc(), if interactive) and not 688 read() for input. A new "table compression" option, -Cr, 689 overrides this change and causes the scanner to use read() 690 (because read() is a bit faster than fread()). -f and -F 691 are now equivalent to -Cfr and -CFr; i.e., they imply the 692 -Cr option. 693 694 - In the blessed name of POSIX compliance, flex supports "%array" 695 and "%pointer" directives in the definitions (first) section of 696 the scanner specification. The former specifies that yytext 697 should be an array (of size YYLMAX), the latter, that it should 698 be a pointer. The array version of yytext is universally slower 699 than the pointer version, but has the advantage that its contents 700 remain unmodified across calls to input() and unput() (the pointer 701 version of yytext is, still, trashed by such calls). 702 703 "%array" cannot be used with the '-+' C++ scanner class option. 704 705 - The new '-Ca' option directs flex to trade off memory for 706 natural alignment when generating a scanner's tables. In 707 particular, table entries that would otherwise be "short" 708 become "long". 709 710 - The new '-h' option produces a summary of the flex flags. 711 712 - The new '-V' option reports the flex version number and exits. 713 714 - The new scanner macro YY_START returns an integer value 715 corresponding to the current start condition. You can return 716 to that start condition by passing the value to a subsequent 717 "BEGIN" action. You also can implement "start condition stacks" 718 by storing the values in an integer stack. 719 720 - You can now redefine macros such as YY_INPUT by just #define'ing 721 them to some other value in the first section of the flex input; 722 no need to first #undef them. 723 724 - flex now generates warnings for rules that can't be matched. 725 These warnings can be turned off using the new '-w' flag. If 726 your scanner uses REJECT then you will not get these warnings. 727 728 - If you specify the '-s' flag but the default rule can be matched, 729 flex now generates a warning. 730 731 - "yyleng" is now a global, and may be modified by the user (though 732 doing so and then using yymore() will yield weird results). 733 734 - Name definitions in the first section of a scanner specification 735 can now include a leading '^' or trailing '$' operator. In this 736 case, the definition is *not* pushed back inside of parentheses. 737 738 - Scanners with compressed tables are now "interactive" (-I option) 739 by default. You can suppress this attribute (which makes them 740 run slightly slower) using the new '-B' flag. 741 742 - Flex now generates 8-bit scanners by default, unless you use the 743 -Cf or -CF compression options (-Cfe and -CFe result in 8-bit 744 scanners). You can force it to generate a 7-bit scanner using 745 the new '-7' flag. You can build flex to generate 8-bit scanners 746 for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS 747 in the Makefile. 748 749 - You no longer need to call the scanner routine yyrestart() to 750 inform the scanner that you have switched to a new file after 751 having seen an EOF on the current input file. Instead, just 752 point yyin at the new file and continue scanning. 753 754 - You no longer need to invoke YY_NEW_FILE in an <<EOF>> action 755 to indicate you wish to continue scanning. Simply point yyin 756 at a new file. 757 758 - A leading '#' no longer introduces a comment in a flex input. 759 760 - flex no longer considers formfeed ('\f') a whitespace character. 761 762 - %t, I'm happy to report, has been nuked. 763 764 - The '-p' option may be given twice ('-pp') to instruct flex to 765 report minor performance problems as well as major ones. 766 767 - The '-v' verbose output no longer includes start/finish time 768 information. 769 770 - Newlines in flex inputs can optionally include leading or 771 trailing carriage-returns ('\r'), in support of several PC/Mac 772 run-time libraries that automatically include these. 773 774 - A start condition of the form "<*>" makes the following rule 775 active in every start condition, whether exclusive or inclusive. 776 777 - The following items have been corrected in the flex documentation: 778 779 - '-C' table compression options *are* cumulative. 780 781 - You may modify yytext but not lengthen it by appending 782 characters to the end. Modifying its final character 783 will affect '^' anchoring for the next rule matched 784 if the character is changed to or from a newline. 785 786 - The term "backtracking" has been renamed "backing up", 787 since it is a one-time repositioning and not a repeated 788 search. What used to be the "lex.backtrack" file is now 789 "lex.backup". 790 791 - Unindented "/* ... */" comments are allowed in the first 792 flex input section, but not in the second. 793 794 - yyless() can only be used in the flex input source, not 795 externally. 796 797 - You can use "yyrestart(yyin)" to throw away the 798 current contents of the input buffer. 799 800 - To write high-speed scanners, attempt to match as much 801 text as possible with each rule. See MISC/fastwc/README 802 for more information. 803 804 - Using the beginning-of-line operator ('^') is fairly 805 cheap. Using unput() is expensive. Using yyless() is 806 cheap. 807 808 - An example of scanning strings with embedded escape 809 sequences has been added. 810 811 - The example of backing-up in flexdoc was erroneous; it 812 has been corrected. 813 814 - A flex scanner's internal buffer now dynamically grows if needed 815 to match large tokens. Note that growing the buffer presently 816 requires rescanning the (large) token, so consuming a lot of 817 text this way is a slow process. Also note that presently the 818 buffer does *not* grow if you unput() more text than can fit 819 into the buffer. 820 821 - The MISC/ directory has been reorganized; see MISC/README for 822 details. 823 824 - yyless() can now be used in the third (user action) section 825 of a scanner specification, thanks to Ceriel Jacobs. yyless() 826 remains a macro and cannot be used outside of the scanner source. 827 828 - The skeleton file is no longer opened at run-time, but instead 829 compiled into a large string array (thanks to John Gilmore and 830 friends at Cygnus). You can still use the -S flag to point flex 831 at a different skeleton file. 832 833 - flex no longer uses a temporary file to store the scanner's 834 actions. 835 836 - A number of changes have been made to decrease porting headaches. 837 In particular, flex no longer uses memset() or ctime(), and 838 provides a single simple mechanism for dealing with C compilers 839 that still define malloc() as returning char* instead of void*. 840 841 - Flex now detects if the scanner specification requires the -8 flag 842 but the flag was not given or on by default. 843 844 - A number of table-expansion fencepost bugs have been fixed, 845 making flex more robust for generating large scanners. 846 847 - flex more consistently identifies the location of errors in 848 its input. 849 850 - YY_USER_ACTION is now invoked only for "real" actions, not for 851 internal actions used by the scanner for things like filling 852 the buffer or handling EOF. 853 854 - The rule "[^]]" now matches any character other than a ']'; 855 formerly it matched any character at all followed by a ']'. 856 This change was made for compatibility with AT&T lex. 857 858 - A large number of miscellaneous bugs have been found and fixed 859 thanks to Gerhard Wilhelms. 860 861 - The source code has been heavily reformatted, making patches 862 relative to previous flex releases no longer accurate. 863 864 865Changes between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7: 866 867 - Fixed bugs in dynamic memory allocation leading to grievous 868 fencepost problems when generating large scanners. 869 - Fixed bug causing infinite loops on character classes with 8-bit 870 characters in them. 871 - Fixed bug in matching repetitions with a lower bound of 0. 872 - Fixed bug in scanning NUL characters using an "interactive" scanner. 873 - Fixed bug in using yymore() at the end of a file. 874 - Fixed bug in misrecognizing rules with variable trailing context. 875 - Fixed bug compiling flex on Suns using gcc 2. 876 - Fixed bug in not recognizing that input files with the character 877 ASCII 128 in them require the -8 flag. 878 - Fixed bug that could cause an infinite loop writing out 879 error messages. 880 - Fixed bug in not recognizing old-style lex % declarations if 881 followed by a tab instead of a space. 882 - Fixed potential crash when flex terminated early (usually due 883 to a bad flag) and the -v flag had been given. 884 - Added some missing declarations of void functions. 885 - Changed to only use '\a' for __STDC__ compilers. 886 - Updated mailing addresses. 887 888 889Changes between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6: 890 891 - Fixed out-of-bounds array access that caused bad tables 892 to be produced on machines where the bad reference happened 893 to yield a 1. This caused problems installing or running 894 flex on some Suns, in particular. 895 896 897Changes between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5: 898 899 - Fixed a serious bug in yymore() which basically made it 900 completely broken. Thanks goes to Jean Christophe of 901 the Nethack development team for finding the problem 902 and passing along the fix. 903 904 905Changes between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4: 906 907 - An up-to-date version of initscan.c so "make test" will 908 work after applying the previous patches 909 910 911Changes between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3: 912 913 - Fixed bug in hexadecimal escapes which allowed only digits, 914 not letters, in escapes 915 - Fixed bug in previous "Changes" file! 916 917 918Changes between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2: 919 920 - Correction to patch #2 for gcc compilation; thanks goes to 921 Paul Eggert for catching this. 922 923 924Changes between 2.3 Patch #2 (02Aug90) and original 2.3 release: 925 926 - Fixed (hopefully) headaches involving declaring malloc() 927 and free() for gcc, which defines __STDC__ but (often) doesn't 928 come with the standard include files such as <stdlib.h>. 929 Reordered #ifdef maze in the scanner skeleton in the hope of 930 getting the declarations right for cfront and g++, too. 931 932 - Note that this patch supercedes patch #1 for release 2.3, 933 which was never announced but was available briefly for 934 anonymous ftp. 935 936 937Changes between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release: 938 939User-visible: 940 941 - A lone <<EOF>> rule (that is, one which is not qualified with 942 a list of start conditions) now specifies the EOF action for 943 *all* start conditions which haven't already had <<EOF>> actions 944 given. To specify an end-of-file action for just the initial 945 state, use <INITIAL><<EOF>>. 946 947 - -d debug output is now contigent on the global yy_flex_debug 948 being set to a non-zero value, which it is by default. 949 950 - A new macro, YY_USER_INIT, is provided for the user to specify 951 initialization action to be taken on the first call to the 952 scanner. This action is done before the scanner does its 953 own initialization. 954 955 - yy_new_buffer() has been added as an alias for yy_create_buffer() 956 957 - Comments beginning with '#' and extending to the end of the line 958 now work, but have been deprecated (in anticipation of making 959 flex recognize #line directives). 960 961 - The funky restrictions on when semi-colons could follow the 962 YY_NEW_FILE and yyless macros have been removed. They now 963 behave identically to functions. 964 965 - A bug in the sample redefinition of YY_INPUT in the documentation 966 has been corrected. 967 968 - A bug in the sample simple tokener in the documentation has 969 been corrected. 970 971 - The documentation on the incompatibilities between flex and 972 lex has been reordered so that the discussion of yylineno 973 and input() come first, as it's anticipated that these will 974 be the most common source of headaches. 975 976 977Things which didn't used to be documented but now are: 978 979 - flex interprets "^foo|bar" differently from lex. flex interprets 980 it as "match either a 'foo' or a 'bar', providing it comes at the 981 beginning of a line", whereas lex interprets it as "match either 982 a 'foo' at the beginning of a line, or a 'bar' anywhere". 983 984 - flex initializes the global "yyin" on the first call to the 985 scanner, while lex initializes it at compile-time. 986 987 - yy_switch_to_buffer() can be used in the yywrap() macro/routine. 988 989 - flex scanners do not use stdio for their input, and hence when 990 writing an interactive scanner one must explictly call fflush() 991 after writing out a prompt. 992 993 - flex scanner can be made reentrant (after a fashion) by using 994 "yyrestart( yyin );". This is useful for interactive scanners 995 which have interrupt handlers that long-jump out of the scanner. 996 997 - a defense of why yylineno is not supported is included, along 998 with a suggestion on how to convert scanners which rely on it. 999 1000 1001Other changes: 1002 1003 - Prototypes and proper declarations of void routines have 1004 been added to the flex source code, courtesy of Kevin B. Kenny. 1005 1006 - Routines dealing with memory allocation now use void* pointers 1007 instead of char* - see Makefile for porting implications. 1008 1009 - Error-checking is now done when flex closes a file. 1010 1011 - Various lint tweaks were added to reduce the number of gripes. 1012 1013 - Makefile has been further parameterized to aid in porting. 1014 1015 - Support for SCO Unix added. 1016 1017 - Flex now sports the latest & greatest UC copyright notice 1018 (which is only slightly different from the previous one). 1019 1020 - A note has been added to flexdoc.1 mentioning work in progress 1021 on modifying flex to generate straight C code rather than a 1022 table-driven automaton, with an email address of whom to contact 1023 if you are working along similar lines. 1024 1025 1026Changes between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2: 1027 1028 - fixed bug which caused -I scanners to bomb 1029 1030 1031Changes between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1: 1032 1033 - fixed bug writing past end of input buffer in yyunput() 1034 - fixed bug detecting NUL's at the end of a buffer 1035 1036 1037Changes between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release: 1038 1039 - Makefile fixes: definition of MAKE variable for systems 1040 which don't have it; installation of flexdoc.1 along with 1041 flex.1; fixed two bugs which could cause "bigtest" to fail. 1042 1043 - flex.skel fix for compiling with g++. 1044 1045 - README and flexdoc.1 no longer list an out-of-date BITNET address 1046 for contacting me. 1047 1048 - minor typos and formatting changes to flex.1 and flexdoc.1. 1049 1050 1051Changes between 2.2 (alpha) release of March '90 and previous release: 1052 1053User-visible: 1054 1055 - Full user documentation now available. 1056 1057 - Support for 8-bit scanners. 1058 1059 - Scanners now accept NUL's. 1060 1061 - A facility has been added for dealing with multiple 1062 input buffers. 1063 1064 - Two manual entries now. One which fully describes flex 1065 (rather than just its differences from lex), and the 1066 other for quick(er) reference. 1067 1068 - A number of changes to bring flex closer into compliance 1069 with the latest POSIX lex draft: 1070 1071 %t support 1072 flex now accepts multiple input files and concatenates 1073 them together to form its input 1074 previous -c (compress) flag renamed -C 1075 do-nothing -c and -n flags added 1076 Any indented code or code within %{}'s in section 2 is 1077 now copied to the output 1078 1079 - yyleng is now a bona fide global integer. 1080 1081 - -d debug information now gives the line number of the 1082 matched rule instead of which number rule it was from 1083 the beginning of the file. 1084 1085 - -v output now includes a summary of the flags used to generate 1086 the scanner. 1087 1088 - unput() and yyrestart() are now globally callable. 1089 1090 - yyrestart() no longer closes the previous value of yyin. 1091 1092 - C++ support; generated scanners can be compiled with C++ compiler. 1093 1094 - Primitive -lfl library added, containing default main() 1095 which calls yylex(). A number of routines currently living 1096 in the scanner skeleton will probably migrate to here 1097 in the future (in particular, yywrap() will probably cease 1098 to be a macro and instead be a function in the -lfl library). 1099 1100 - Hexadecimal (\x) escape sequences added. 1101 1102 - Support for MS-DOS, VMS, and Turbo-C integrated. 1103 1104 - The %used/%unused operators have been deprecated. They 1105 may go away soon. 1106 1107 1108Other changes: 1109 1110 - Makefile enhanced for easier testing and installation. 1111 - The parser has been tweaked to detect some erroneous 1112 constructions which previously were missed. 1113 - Scanner input buffer overflow is now detected. 1114 - Bugs with missing "const" declarations fixed. 1115 - Out-of-date Minix/Atari patches provided. 1116 - Scanners no longer require printf() unless FLEX_DEBUG is being used. 1117 - A subtle input() bug has been fixed. 1118 - Line numbers for "continued action" rules (those following 1119 the special '|' action) are now correct. 1120 - unput() bug fixed; had been causing problems porting flex to VMS. 1121 - yymore() handling rewritten to fix bug with interaction 1122 between yymore() and trailing context. 1123 - EOF in actions now generates an error message. 1124 - Bug involving -CFe and generating equivalence classes fixed. 1125 - Bug which made -CF be treated as -Cf fixed. 1126 - Support for SysV tmpnam() added. 1127 - Unused #define's for scanner no longer generated. 1128 - Error messages which are associated with a particular input 1129 line are now all identified with their input line in standard 1130 format. 1131 - % directives which are valid to lex but not to flex are 1132 now ignored instead of generating warnings. 1133 - -DSYS_V flag can now also be specified -DUSG for System V 1134 compilation. 1135 1136 1137Changes between 2.1 beta-test release of June '89 and previous release: 1138 1139User-visible: 1140 1141 - -p flag generates a performance report to stderr. The report 1142 consists of comments regarding features of the scanner rules 1143 which result in slower scanners. 1144 1145 - -b flag generates backtracking information to lex.backtrack. 1146 This is a list of scanner states which require backtracking 1147 and the characters on which they do so. By adding rules 1148 one can remove backtracking states. If all backtracking states 1149 are eliminated, the generated scanner will run faster. 1150 Backtracking is not yet documented in the manual entry. 1151 1152 - Variable trailing context now works, i.e., one can have 1153 rules like "(foo)*/[ \t]*bletch". Some trailing context 1154 patterns still cannot be properly matched and generate 1155 error messages. These are patterns where the ending of the 1156 first part of the rule matches the beginning of the second 1157 part, such as "zx*/xy*", where the 'x*' matches the 'x' at 1158 the beginning of the trailing context. Lex won't get these 1159 patterns right either. 1160 1161 - Faster scanners. 1162 1163 - End-of-file rules. The special rule "<<EOF>>" indicates 1164 actions which are to be taken when an end-of-file is 1165 encountered and yywrap() returns non-zero (i.e., indicates 1166 no further files to process). See manual entry for example. 1167 1168 - The -r (reject used) flag is gone. flex now scans the input 1169 for occurrences of the string "REJECT" to determine if the 1170 action is needed. It tries to be intelligent about this but 1171 can be fooled. One can force the presence or absence of 1172 REJECT by adding a line in the first section of the form 1173 "%used REJECT" or "%unused REJECT". 1174 1175 - yymore() has been implemented. Similarly to REJECT, flex 1176 detects the use of yymore(), which can be overridden using 1177 "%used" or "%unused". 1178 1179 - Patterns like "x{0,3}" now work (i.e., with lower-limit == 0). 1180 1181 - Removed '\^x' for ctrl-x misfeature. 1182 1183 - Added '\a' and '\v' escape sequences. 1184 1185 - \<digits> now works for octal escape sequences; previously 1186 \0<digits> was required. 1187 1188 - Better error reporting; line numbers are associated with rules. 1189 1190 - yyleng is a macro; it cannot be accessed outside of the 1191 scanner source file. 1192 1193 - yytext and yyleng should not be modified within a flex action. 1194 1195 - Generated scanners #define the name FLEX_SCANNER. 1196 1197 - Rules are internally separated by YY_BREAK in lex.yy.c rather 1198 than break, to allow redefinition. 1199 1200 - The macro YY_USER_ACTION can be redefined to provide an action 1201 which is always executed prior to the matched rule's action. 1202 1203 - yyrestart() is a new action which can be used to restart 1204 the scanner after it has seen an end-of-file (a "real" one, 1205 that is, one for which yywrap() returned non-zero). It takes 1206 a FILE* argument indicating a new file to scan and sets 1207 things up so that a subsequent call to yylex() will start 1208 scanning that file. 1209 1210 - Internal scanner names all preceded by "yy_" 1211 1212 - lex.yy.c is deleted if errors are encountered during processing. 1213 1214 - Comments may be put in the first section of the input by preceding 1215 them with '#'. 1216 1217 1218 1219Other changes: 1220 1221 - Some portability-related bugs fixed, in particular for machines 1222 with unsigned characters or sizeof( int* ) != sizeof( int ). 1223 Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all 1224 trimmed to be 31 or fewer characters. Shortened file names 1225 for dinosaur OS's. Checks for allocating > 64K memory 1226 on 16 bit'ers. Amiga tweaks. Compiles using gcc on a Sun-3. 1227 - Compressed and fast scanner skeletons merged. 1228 - Skeleton header files done away with. 1229 - Generated scanner uses prototypes and "const" for __STDC__. 1230 - -DSV flag is now -DSYS_V for System V compilation. 1231 - Removed all references to FTL language. 1232 - Software now covered by BSD Copyright. 1233 - flex will replace lex in subsequent BSD releases. 1234