\rSec0[lex]{Lexical conventions}

\gramSec[gram.lex]{Lexical conventions}

\indextext{lexical conventions|see{conventions, lexical}}

\indextext{translation!separate|see{compilation, separate}}

\indextext{separate translation|see{compilation, separate}}

\indextext{separate compilation|see{compilation, separate}}

\indextext{phases of translation|see{translation, phases}}

\indextext{source file character|see{character, source file}}

\indextext{alternative token|see{token, alternative}}

\indextext{digraph|see{token, alternative}}

\indextext{integer literal|see{literal, integer}}

\indextext{character literal|see{literal, character}}

\indextext{floating-point literal|see{literal, floating-point}}

\indextext{string literal|see{literal, string}}

\indextext{boolean literal|see{literal, boolean}}

\indextext{pointer literal|see{literal, pointer}}

\indextext{user-defined literal|see{literal, user-defined}}

\indextext{file, source|see{source file}}

\indextext{null character|see{character, null}}

\indextext{null wide character|see{wide-character, null}}

\rSec1[lex.separate]{Separate translation}

\indextext{conventions!lexical|(}%

\indextext{compilation!separate|(}%

The text of the program is kept in units called

\defnx{source files}{source file} in this document.

A source file together with all the headers\iref{headers}

and source files included\iref{cpp.include} via the preprocessing

directive \tcode{\#include}, less any source lines skipped by any of the

conditional inclusion\iref{cpp.cond} preprocessing directives,

as modified by the implementation-defined behavior of any

conditionally-supported-directives\iref{cpp.pre} and pragmas\iref{cpp.pragma},

if any, is

called a \defnadj{preprocessing}{translation unit}.

\begin{note}

A \Cpp{} program need not all be translated at the same time.

Translation units can be separately translated and then later linked

to produce an executable program\iref{basic.link}.

\end{note}

\indextext{compilation!separate|)}

\rSec1[lex.phases]{Phases of translation}%

\indextext{translation!phases|(}%

The precedence among the syntax rules of translation is specified by the

following phases.

\begin{footnote}

Implementations behave as if these separate phases

occur, although in practice different phases can be folded together.

\end{footnote}

\begin{enumerate}

\indextext{character!source file}%

An implementation shall support input files

that are a sequence of UTF-8 code units (UTF-8 files).

It may also support

an \impldef{supported input files} set of other kinds of input files, and,

if so, the kind of an input file is determined in

an \impldef{determination of kind of input file} manner

that includes a means of designating input files as UTF-8 files,

independent of their content.

\begin{note}

In other words,

recognizing the \unicode{feff}{byte order mark} is not sufficient.

\end{note}

If an input file is determined to be a UTF-8 file,

then it shall be a well-formed UTF-8 code unit sequence and

it is decoded to produce a sequence of Unicode

\begin{footnote}

Unicode\textregistered\ is a registered trademark of Unicode, Inc.

This information is given for the convenience of users of this document and

does not constitute an endorsement by ISO or IEC of this product.

\end{footnote}

scalar values.

A sequence of translation character set elements\iref{lex.charset} is then formed

by mapping each Unicode scalar value

to the corresponding translation character set element.

In the resulting sequence,

each pair of characters in the input sequence consisting of

\unicode{000d}{carriage return} followed by \unicode{000a}{line feed},

as well as each

\unicode{000d}{carriage return} not immediately followed by a \unicode{000a}{line feed},

is replaced by a single new-line character.

For any other kind of input file supported by the implementation,

characters are mapped, in an

\impldef{mapping input file characters to translation character set} manner,

to a sequence of translation character set elements,

representing end-of-line indicators as new-line characters.

\indextext{line splicing}%

If the first translation character is \unicode{feff}{byte order mark},

it is deleted.

Each sequence of a backslash character (\textbackslash)

immediately followed by

zero or more whitespace characters other than new-line followed by

a new-line character is deleted, splicing

physical source lines to form \defnx{logical source lines}{source line!logical}. Only the last

backslash on any physical source line shall be eligible for being part

of such a splice.

\begin{note}

Line splicing can form

a \grammarterm{universal-character-name}\iref{lex.charset}.

\end{note}

A source file that is not empty and that (after splicing)

does not end in a new-line character

shall be processed as if an additional new-line character were appended

to the file.

\item The source file is decomposed into preprocessing

tokens\iref{lex.pptoken} and sequences of whitespace characters

(including comments). A source file shall not end in a partial

preprocessing token or in a partial comment.

\begin{footnote}

A partial preprocessing

token would arise from a source file

ending in the first portion of a multi-character token that requires a

terminating sequence of characters, such as a \grammarterm{header-name}

that is missing the closing \tcode{"}

or \tcode{>}. A partial comment

would arise from a source file ending with an unclosed \tcode{/*}

comment.

\end{footnote}

Each comment\iref{lex.comment} is replaced by one \unicode{0020}{space} character. New-line characters are

retained. Whether each nonempty sequence of whitespace characters other

than new-line is retained or replaced by one \unicode{0020}{space} character is

unspecified.

As characters from the source file are consumed

to form the next preprocessing token

(i.e., not being consumed as part of a comment or other forms of whitespace),

except when matching a

\grammarterm{c-char-sequence},

\grammarterm{s-char-sequence},

\grammarterm{r-char-sequence},

\grammarterm{h-char-sequence}, or

\grammarterm{q-char-sequence},

\grammarterm{universal-character-name}s are recognized\iref{lex.universal.char} and

replaced by the designated element of the translation character set\iref{lex.charset}.

The process of dividing a source file's

characters into preprocessing tokens is context-dependent.

\begin{example}

See the handling of \tcode{<} within a \tcode{\#include} preprocessing

directive\iref{lex.header,cpp.include}.

\end{example}

\item The source file is analyzed as a \grammarterm{preprocessing-file}\iref{cpp.pre}.

Preprocessing directives\iref{cpp} are executed, macro invocations are

expanded\iref{cpp.replace}, and \tcode{_Pragma} unary operator expressions are executed\iref{cpp.pragma.op}.

A \tcode{\#include} preprocessing directive\iref{cpp.include} causes the named header or

source file to be processed from phase 1 through phase 4, recursively.

All preprocessing directives are then deleted.

For a sequence of two or more adjacent \grammarterm{string-literal} preprocessing tokens,

a common \grammarterm{encoding-prefix} is determined

as specified in \ref{lex.string}.

Each such \grammarterm{string-literal} preprocessing token is then considered to have

that common \grammarterm{encoding-prefix}.

\indextext{concatenation!string}%

Adjacent \grammarterm{string-literal} preprocessing tokens are concatenated\iref{lex.string}.

Each preprocessing token is converted into a token\iref{lex.token}.

Whitespace characters separating tokens are no longer significant.

The resulting tokens constitute a \defn{translation unit} and

are syntactically and

semantically analyzed as a \grammarterm{translation-unit}\iref{basic.link} and

translated.

\begin{note}

The process of analyzing and translating the tokens can occasionally

result in one token being replaced by a sequence of other

tokens\iref{temp.names}.

\end{note}

It is

\impldef{whether the sources for

module units and header units

on which the current translation unit has an interface

dependency are required to be available during translation}

whether the sources for

module units and header units

on which the current translation unit has an interface

dependency\iref{module.unit,module.import}

are required to be available.

\begin{note}

Source files, translation

units and translated translation units need not necessarily be stored as

files, nor need there be any one-to-one correspondence between these

entities and any external representation. The description is conceptual

only, and does not specify any particular implementation.

\end{note}

\begin{note}

Previously translated translation units can be preserved individually or in libraries.

The separate translation units of a program communicate\iref{basic.link} by (for example)

calls to functions whose names have external or module linkage,

manipulation of variables whose names have external or module linkage, or

manipulation of data files.

\end{note}

While the tokens constituting translation units

are being analyzed and translated,

required instantiations are performed.

\begin{note}

This can include

instantiations which have been explicitly

requested\iref{temp.explicit}.

\end{note}

The contexts from which instantiations may be performed

are determined by their respective points of instantiation\iref{temp.point}.

\begin{note}

Other requirements in this document can further constrain

the context from which an instantiation can be performed.

For example, a constexpr function template specialization

might have a point of instantation at the end of a translation unit,

but its use in certain constant expressions could require

that it be instantiated at an earlier point\iref{temp.inst}.

\end{note}

Each instantiation results in new program constructs.

The program is ill-formed if any instantiation fails.

During the analysis and translation of tokens,

certain expressions are evaluated\iref{expr.const}.

Constructs appearing at a program point $P$ are analyzed

in a context where each side effect of evaluating an expression $E$

as a full-expression is complete if and only if

\begin{itemize}

$E$ is the expression corresponding to

a \grammarterm{consteval-block-declaration}\iref{dcl.pre}, and

either that \grammarterm{consteval-block-declaration} or

the template definition from which it is instantiated

is reachable from\iref{module.reach}

\begin{itemize}

$P$, or

the point immediately following

the \grammarterm{class-specifier} of the outermost class

for which $P$ is in a complete-class context\iref{class.mem.general}.

\end{itemize}

\end{itemize}

\begin{example}

\begin{codeblock}

class S {

class Incomplete;

class Inner {

void fn() {

/* @$p_1$@ */ Incomplete i; // OK

}

}; /* @$p_2$@ */

consteval {

define_aggregate(^^Incomplete, {});

}

}; /* @$p_3$@ */

\end{codeblock}

Constructs at $p_1$ are analyzed in a context

where the side effect of the call to \tcode{define_aggregate} is evaluated

because

\begin{itemize}

$E$ is the expression corresponding to a consteval block, and

$p_1$ is in a complete-class context of \tcode{S} and

the consteval block is reachable from $p_3$.

\end{itemize}

\end{example}

\indextext{linking}%

Translated translation units are combined, and

all external entity references are resolved. Library

components are linked to satisfy external references to

entities not defined in the current translation. All such translator

output is collected into a program image which contains information

needed for execution in its execution environment.%

\indextext{translation!phases|)}

\end{enumerate}

\rSec1[lex.char]{Characters}%

\rSec2[lex.charset]{Character sets}

\indextext{character set|(}%

The \defnadj{translation}{character set} consists of the following elements:

\begin{itemize}

each abstract character assigned a code point in the Unicode codespace

as specified in the Unicode Standard, and

a distinct character for each Unicode scalar value

not assigned to an abstract character.

\end{itemize}

\begin{note}

Unicode code points are integers

in the range $[0, \mathrm{10FFFF}]$ (hexadecimal).

A surrogate code point is a value

in the range $[\mathrm{D800}, \mathrm{DFFF}]$ (hexadecimal).

A Unicode scalar value is any code point that is not a surrogate code point.

\end{note}

The \defnadj{basic}{character set} is a subset of the translation character set,

consisting of 99 characters as specified in \tref{lex.charset.basic}.

\begin{note}

Unicode short names are given only as a means to identifying the character;

the numerical value has no other meaning in this context.

\end{note}

\begin{floattable}{Basic character set}{lex.charset.basic}{lll}

\lhdrx{2}{character} & \rhdr{glyph} \\ \capsep

\ucode{0009} & \uname{character tabulation} & \\

\ucode{000b} & \uname{line tabulation} & \\

\ucode{000c} & \uname{form feed} & \\

\ucode{0020} & \uname{space} & \\

\ucode{000a} & \uname{line feed} & new-line \\

\ucode{0021} & \uname{exclamation mark} & \tcode{!} \\

\ucode{0022} & \uname{quotation mark} & \tcode{"} \\

\ucode{0023} & \uname{number sign} & \tcode{\#} \\

\ucode{0024} & \uname{dollar sign} & \tcode{\$} \\

\ucode{0025} & \uname{percent sign} & \tcode{\%} \\

\ucode{0026} & \uname{ampersand} & \tcode{\&} \\

\ucode{0027} & \uname{apostrophe} & \tcode{'} \\

\ucode{0028} & \uname{left parenthesis} & \tcode{(} \\

\ucode{0029} & \uname{right parenthesis} & \tcode{)} \\

\ucode{002a} & \uname{asterisk} & \tcode{*} \\

\ucode{002b} & \uname{plus sign} & \tcode{+} \\

\ucode{002c} & \uname{comma} & \tcode{,} \\

\ucode{002d} & \uname{hyphen-minus} & \tcode{-} \\

\ucode{002e} & \uname{full stop} & \tcode{.} \\

\ucode{002f} & \uname{solidus} & \tcode{/} \\

\ucode{0030} .. \ucode{0039} & \uname{digit zero .. nine} & \tcode{0 1 2 3 4 5 6 7 8 9} \\

\ucode{003a} & \uname{colon} & \tcode{:} \\

\ucode{003b} & \uname{semicolon} & \tcode{;} \\

\ucode{003c} & \uname{less-than sign} & \tcode{<} \\

\ucode{003d} & \uname{equals sign} & \tcode{=} \\

\ucode{003e} & \uname{greater-than sign} & \tcode{>} \\

\ucode{003f} & \uname{question mark} & \tcode{?} \\

\ucode{0040} & \uname{commercial at} & \tcode{@} \\

\ucode{0041} .. \ucode{005a} & \uname{latin capital letter a .. z} & \tcode{A B C D E F G H I J K L M} \\

& & \tcode{N O P Q R S T U V W X Y Z} \\

\ucode{005b} & \uname{left square bracket} & \tcode{[} \\

\ucode{005c} & \uname{reverse solidus} & \tcode{\textbackslash} \\

\ucode{005d} & \uname{right square bracket} & \tcode{]} \\

\ucode{005e} & \uname{circumflex accent} & \tcode{\caret} \\

\ucode{005f} & \uname{low line} & \tcode{_} \\

\ucode{0060} & \uname{grave accent} & \tcode{\`} \\

\ucode{0061} .. \ucode{007a} & \uname{latin small letter a .. z} & \tcode{a b c d e f g h i j k l m} \\

& & \tcode{n o p q r s t u v w x y z} \\

\ucode{007b} & \uname{left curly bracket} & \tcode{\{} \\

\ucode{007c} & \uname{vertical line} & \tcode{|} \\

\ucode{007d} & \uname{right curly bracket} & \tcode{\}} \\

\ucode{007e} & \uname{tilde} & \tcode{\textasciitilde} \\

\end{floattable}

The \defnadj{basic literal}{character set} consists of

all characters of the basic character set,

plus the control characters specified in \tref{lex.charset.literal}.

\begin{floattable}{Additional control characters in the basic literal character set}{lex.charset.literal}{ll}

\ohdrx{2}{character} \\ \capsep

\ucode{0000} & \uname{null} \\

\ucode{0007} & \uname{alert} \\

\ucode{0008} & \uname{backspace} \\

\ucode{000d} & \uname{carriage return} \\

\end{floattable}

A \defn{code unit} is an integer value

of character type\iref{basic.fundamental}.

Characters in a \grammarterm{character-literal}

other than a multicharacter or non-encodable character literal or

in a \grammarterm{string-literal} are encoded as

a sequence of one or more code units, as determined

by the \grammarterm{encoding-prefix}\iref{lex.ccon,lex.string};

this is termed the respective \defnadj{literal}{encoding}.

The \defnadj{ordinary literal}{encoding} is

the encoding applied to an ordinary character or string literal.

The \defnadj{wide literal}{encoding} is the encoding applied

to a wide character or string literal.

A literal encoding or a locale-specific encoding of one of

the execution character sets\iref{character.seq}

encodes each element of the basic literal character set as

a single code unit with non-negative value,

distinct from the code unit for any other such element.

\begin{note}

A character not in the basic literal character set

can be encoded with more than one code unit;

the value of such a code unit can be the same as

that of a code unit for an element of the basic literal character set.

\end{note}

\indextext{character!null}%

\indextext{wide-character!null}%

The \unicode{0000}{null} character is encoded as the value \tcode{0}.

No other element of the translation character set

is encoded with a code unit of value \tcode{0}.

The code unit value of each decimal digit character after the digit \tcode{0} (\ucode{0030})

shall be one greater than the value of the previous.

The ordinary and wide literal encodings are otherwise

\impldef{ordinary and wide literal encodings}.

\indextext{UTF-8}%

\indextext{UTF-16}%

\indextext{UTF-32}%

For a UTF-8, UTF-16, or UTF-32 literal,

the implementation shall encode

the Unicode scalar value

corresponding to each character of the translation character set

as specified in the Unicode Standard

for the respective Unicode encoding form.

\indextext{character set|)}

\rSec2[lex.universal.char]{Universal character names}

\begin{bnf}

\nontermdef{n-char}\br

\textnormal{any member of the translation character set except the \unicode{007d}{right curly bracket} or new-line character}

\end{bnf}

\begin{bnf}

\nontermdef{n-char-sequence}\br

n-char \opt{n-char-sequence}

\end{bnf}

\begin{bnf}

\nontermdef{named-universal-character}\br

\terminal{\textbackslash N\{} n-char-sequence \terminal{\}}

\end{bnf}

\begin{bnf}

\nontermdef{hex-quad}\br

hexadecimal-digit hexadecimal-digit hexadecimal-digit hexadecimal-digit

\end{bnf}

\begin{bnf}

\nontermdef{simple-hexadecimal-digit-sequence}\br

hexadecimal-digit \opt{simple-hexadecimal-digit-sequence}

\end{bnf}

\begin{bnf}

\nontermdef{universal-character-name}\br

\terminal{\textbackslash u} hex-quad\br

\terminal{\textbackslash U} hex-quad hex-quad\br

\terminal{\textbackslash u\{} simple-hexadecimal-digit-sequence \terminal{\}}\br

named-universal-character

\end{bnf}

The \grammarterm{universal-character-name} construct provides a way to name any

element in the translation character set using just the basic character set.

If a \grammarterm{universal-character-name} outside

the \grammarterm{c-char-sequence}, \grammarterm{s-char-sequence}, or

\grammarterm{r-char-sequence} of a \grammarterm{character-literal} or

\grammarterm{string-literal}

(in either case, including within a \grammarterm{user-defined-literal})

corresponds to a control character or to a character in the basic character set,

the program is ill-formed.

\begin{note}

A sequence of characters resembling a \grammarterm{universal-character-name} in an

\grammarterm{r-char-sequence}\iref{lex.string} does not form a

\grammarterm{universal-character-name}.

\end{note}

A \grammarterm{universal-character-name}

of the form \tcode{\textbackslash u} \grammarterm{hex-quad},

\tcode{\textbackslash U} \grammarterm{hex-quad} \grammarterm{hex-quad}, or

\tcode{\textbackslash u\{\grammarterm{simple-hexadecimal-digit-sequence}\}}

designates the character in the translation character set

whose Unicode scalar value is the hexadecimal number represented by

the sequence of \grammarterm{hexadecimal-digit}s

in the \grammarterm{universal-character-name}.

The program is ill-formed if that number is not a Unicode scalar value.

A \grammarterm{universal-character-name}

that is a \grammarterm{named-universal-character}

designates the corresponding character

in the Unicode Standard (chapter 4.8 Name)

if the \grammarterm{n-char-sequence} is equal

to its character name or

to one of its character name aliases of

type ``control'', ``correction'', or ``alternate'';

otherwise, the program is ill-formed.

\begin{note}

These aliases are listed in

the Unicode Character Database's \tcode{NameAliases.txt}.

None of these names or aliases have leading or trailing spaces.

\end{note}

\rSec1[lex.comment]{Comments}

\indextext{comment|(}%

\indextext{comment!\tcode{/*} \tcode{*/}}%

\indextext{comment!\tcode{//}}%

The characters \tcode{/*} start a comment, which terminates with the

characters \tcode{*/}. These comments do not nest.

\indextext{comment!\tcode{//}}%

The characters \tcode{//} start a comment, which terminates immediately before the

next new-line character.

\begin{note}

The comment characters \tcode{//}, \tcode{/*},

and \tcode{*/} have no special meaning within a \tcode{//} comment and

are treated just like other characters. Similarly, the comment

characters \tcode{//} and \tcode{/*} have no special meaning within a

\tcode{/*} comment.

\end{note}

\indextext{comment|)}

\rSec1[lex.pptoken]{Preprocessing tokens}

\indextext{token!preprocessing|(}%

\begin{bnf}

\nontermdef{preprocessing-token}\br

header-name\br

import-keyword\br

module-keyword\br

export-keyword\br

identifier\br

pp-number\br

character-literal\br

user-defined-character-literal\br

string-literal\br

user-defined-string-literal\br

preprocessing-op-or-punc\br

\textnormal{each non-whitespace character that cannot be one of the above}

\end{bnf}

A preprocessing token is the minimal lexical element of the language in translation

phases 3 through 6.

In this document,

glyphs are used to identify

elements of the basic character set\iref{lex.charset}.

The categories of preprocessing token are: header names,

placeholder tokens produced by preprocessing \tcode{import} and \tcode{module} directives

(\grammarterm{import-keyword}, \grammarterm{module-keyword}, and \grammarterm{export-keyword}),

identifiers, preprocessing numbers, character literals (including user-defined character

literals), string literals (including user-defined string literals), preprocessing

operators and punctuators, and single non-whitespace characters that do not lexically

match the other preprocessing token categories.

If a \unicode{0027}{apostrophe} or a \unicode{0022}{quotation mark} character

matches the last category, the program is ill-formed.

If any character not in the basic character set matches the last category,

the program is ill-formed.

Preprocessing tokens can be separated by

\indextext{whitespace}%

whitespace;

\indextext{comment}%

this consists of comments\iref{lex.comment}, or whitespace characters

(\unicode{0020}{space},

\unicode{0009}{character tabulation},

new-line,

\unicode{000b}{line tabulation}, and

\unicode{000c}{form feed}), or both.

As described in \ref{cpp}, in certain

circumstances during translation phase 4, whitespace (or the absence

thereof) serves as more than preprocessing token separation. Whitespace

can appear within a preprocessing token only as part of a header name or

between the quotation characters in a character literal or

string literal.

Each preprocessing token that is converted to a token\iref{lex.token}

shall have the lexical form of a keyword, an identifier, a literal,

or an operator or punctuator.

The \grammarterm{import-keyword} is produced

by processing an \keyword{import} directive\iref{cpp.import},

the \grammarterm{module-keyword} is produced

by preprocessing a \keyword{module} directive\iref{cpp.module}, and

the \grammarterm{export-keyword} is produced

by preprocessing either of the previous two directives.

\begin{note}

None has any observable spelling.

\end{note}

If the input stream has been parsed into preprocessing tokens up to a

given character:

\begin{itemize}

\indextext{literal!string!raw}%

If the next character begins a sequence of characters that could be the prefix

and initial double quote of a raw string literal, such as \tcode{R"}, the next preprocessing

token shall be a raw string literal. Between the initial and final

double quote characters of the raw string, any transformations performed in phase

2 (line splicing) are reverted; this reversion

shall apply before any \grammarterm{d-char}, \grammarterm{r-char}, or delimiting

parenthesis is identified. The raw string literal is defined as the shortest sequence

of characters that matches the raw-string pattern

\begin{ncbnf}

\opt{encoding-prefix} \terminal{R} raw-string

\end{ncbnf}

\item Otherwise, if the next three characters are \tcode{<::} and the subsequent character

is neither \tcode{:} nor \tcode{>}, the \tcode{<} is treated as a preprocessing token by

itself and not as the first character of the alternative token \tcode{<:}.

Otherwise, if the next three characters are \tcode{[::} and

the subsequent character is not \tcode{:}, or

if the next three characters are \tcode{[:>},

the \tcode{[} is treated as a preprocessing token by itself and

not as the first character of the preprocessing token \tcode{[:}.

\begin{note}

The tokens \tcode{[:} and \tcode{:]} cannot be composed from digraphs.

\end{note}

\item Otherwise,

the next preprocessing token is the longest sequence of

characters that could constitute a preprocessing token, even if that

would cause further lexical analysis to fail,

except that

\begin{itemize}

a \grammarterm{string-literal} token is never formed

when a \grammarterm{header-name} token can be formed, and

a \grammarterm{header-name}\iref{lex.header} is only formed

\begin{itemize}

immediately after the \tcode{include}, \tcode{embed}, or \tcode{import} preprocessing token in a

\tcode{\#include}\iref{cpp.include}, \tcode{\#embed}\iref{cpp.embed}, or

\tcode{import}\iref{cpp.import} directive, respectively, or

immediately after a preprocessing token sequence of \xname{has_include}

or \xname{has_embed} immediately followed by \tcode{(}

in a \tcode{\#if}, \tcode{\#elif}, or \tcode{\#embed} directive\iref{cpp.cond,cpp.embed}.

\end{itemize}

\end{itemize}

\end{itemize}

\begin{example}

\begin{codeblock}

#define R "x"

const char* s = R"y"; // ill-formed raw string, not \tcode{"x" "y"}

\end{codeblock}

\end{example}

\begin{example}

The program fragment \tcode{0xe+foo} is parsed as a

preprocessing number token (one that is not a valid

\grammarterm{integer-literal} or \grammarterm{floating-point-literal} token),

even though a parse as three preprocessing tokens

\tcode{0xe}, \tcode{+}, and \tcode{foo} can produce a valid expression (for example,

if \tcode{foo} is a macro defined as \tcode{1}). Similarly, the

program fragment \tcode{1E1} is parsed as a preprocessing number (one

that is a valid \grammarterm{floating-point-literal} token),

whether or not \tcode{E} is a macro name.

\end{example}

\begin{example}

The program fragment \tcode{x+++++y} is parsed as \tcode{x

++ ++ + y}, which, if \tcode{x} and \tcode{y} have integral types,

violates a constraint on increment operators, even though the parse

\tcode{x ++ + ++ y} can yield a correct expression.

\end{example}

\indextext{token!preprocessing|)}

\rSec1[lex.header]{Header names}

\indextext{header!name|(}%

\begin{bnf}

\microtypesetup{protrusion=false}

\nontermdef{header-name}\br

\terminal{<} h-char-sequence \terminal{>}\br

\terminal{"} q-char-sequence \terminal{"}

\end{bnf}

\begin{bnf}

\nontermdef{h-char-sequence}\br

h-char \opt{h-char-sequence}

\end{bnf}

\begin{bnf}

\nontermdef{h-char}\br

\textnormal{any member of the translation character set except new-line and \unicode{003e}{greater-than sign}}

\end{bnf}

\begin{bnf}

\nontermdef{q-char-sequence}\br

q-char \opt{q-char-sequence}

\end{bnf}

\begin{bnf}

\nontermdef{q-char}\br

\textnormal{any member of the translation character set except new-line and \unicode{0022}{quotation mark}}

\end{bnf}

The sequences in both forms of \grammarterm{header-name}{s} are mapped in an

\impldef{mapping header name to header or external source file} manner to headers or to

external source file names as specified in~\ref{cpp.include}.

\begin{note}

Header name preprocessing tokens appear only within

a \tcode{\#include} preprocessing directive,

a \tcode{__has_include} preprocessing expression, or

after certain occurrences of an \tcode{import} token

(see~\ref{lex.pptoken}).

\end{note}

The appearance of either of the characters \tcode{'} or \tcode{\textbackslash} or of

either of the character sequences \tcode{/*} or \tcode{//} in a

\grammarterm{q-char-sequence} or an \grammarterm{h-char-sequence}

is conditionally-supported with \impldef{meaning of \tcode{'}, \tcode{\textbackslash},

\tcode{/*}, or \tcode{//} in a \grammarterm{q-char-sequence} or an

\grammarterm{h-char-sequence}} semantics, as is the appearance of the character

\tcode{"} in an \grammarterm{h-char-sequence}.

\begin{note}

Thus, a sequence of characters

that resembles an escape sequence can result in an error, be interpreted as the

character corresponding to the escape sequence, or have a completely different meaning,

depending on the implementation.

\end{note}

\indextext{header!name|)}

\rSec1[lex.ppnumber]{Preprocessing numbers}

\indextext{number!preprocessing|(}%

\begin{bnf}

\nontermdef{pp-number}\br

digit\br

\terminal{.} digit\br

pp-number identifier-continue\br

pp-number \terminal{'} digit\br

pp-number \terminal{'} nondigit\br

pp-number \terminal{e} sign\br

pp-number \terminal{E} sign\br

pp-number \terminal{p} sign\br

pp-number \terminal{P} sign\br

pp-number \terminal{.}

\end{bnf}

Preprocessing number tokens lexically include

all \grammarterm{integer-literal} tokens\iref{lex.icon} and

all \grammarterm{floating-point-literal} tokens\iref{lex.fcon}.

A preprocessing number does not have a type or a value; it acquires both

after a successful conversion to

an \grammarterm{integer-literal} token or

a \grammarterm{floating-point-literal} token.%

\indextext{number!preprocessing|)}

\rSec1[lex.operators]{Operators and punctuators}

\indextext{operator|(}%

\indextext{punctuator|(}%

The lexical representation of \Cpp{} programs includes a number of

preprocessing tokens that are used in the syntax of the preprocessor or

are converted into tokens for operators and punctuators:

\begin{bnf}

\nontermdef{preprocessing-op-or-punc}\br

preprocessing-operator\br

operator-or-punctuator

\end{bnf}

\begin{bnf}

\microtypesetup{protrusion=false}

\nontermdef{preprocessing-operator} \textnormal{one of}\br

\terminal{\# \ \ \ \ \ \ \ \#\# \ \ \ \ \ \ \%: \ \ \ \ \ \ \%:\%:}

\end{bnf}

\begin{bnf}

\microtypesetup{protrusion=false}

\nontermdef{operator-or-punctuator} \textnormal{one of}\br

\terminal{\{ \ \ \ \ \ \ \ \} \ \ \ \ \ \ \ [ \ \ \ \ \ \ \ ] \ \ \ \ \ \ \ ( \ \ \ \ \ \ \ ) \ \ \ \ \ \ \ [: \ \ \ \ \ \ :]}\br

\terminal{<\% \ \ \ \ \ \ \%> \ \ \ \ \ \ <: \ \ \ \ \ \ :> \ \ \ \ \ \ ; \ \ \ \ \ \ \ : \ \ \ \ \ \ \ ...}\br

\terminal{? \ \ \ \ \ \ \ :: \ \ \ \ \ \ . \ \ \ \ \ \ \ .* \ \ \ \ \ \ -> \ \ \ \ \ \ ->* \ \ \ \ \ \caret{}\caret{} \ \ \ \ \ \ \~}\br

\terminal{! \ \ \ \ \ \ \ + \ \ \ \ \ \ \ - \ \ \ \ \ \ \ * \ \ \ \ \ \ \ / \ \ \ \ \ \ \ \% \ \ \ \ \ \ \ \caret{} \ \ \ \ \ \ \ \& \ \ \ \ \ \ \ |}\br

\terminal{= \ \ \ \ \ \ \ += \ \ \ \ \ \ -= \ \ \ \ \ \ *= \ \ \ \ \ \ /= \ \ \ \ \ \ \%= \ \ \ \ \ \ \caret{}= \ \ \ \ \ \ \&= \ \ \ \ \ \ |=}\br

\terminal{== \ \ \ \ \ \ != \ \ \ \ \ \ < \ \ \ \ \ \ \ > \ \ \ \ \ \ \ <= \ \ \ \ \ \ >= \ \ \ \ \ \ <=> \ \ \ \ \ \&\& \ \ \ \ \ \ ||}\br

\terminal{<< \ \ \ \ \ \ >> \ \ \ \ \ \ <<= \ \ \ \ \ >>= \ \ \ \ \ ++ \ \ \ \ \ \ -- \ \ \ \ \ \ ,}\br

\terminal{\keyword{and} \ \ \ \ \ \keyword{or} \ \ \ \ \ \ \keyword{xor} \ \ \ \ \ \keyword{not} \ \ \ \ \ \keyword{bitand} \ \ \keyword{bitor} \ \ \ \keyword{compl}}\br

\terminal{\keyword{and_eq} \ \ \keyword{or_eq} \ \ \ \keyword{xor_eq} \ \ \keyword{not_eq}}

\end{bnf}

Each \grammarterm{operator-or-punctuator} is converted to a single token

in translation phase 7\iref{lex.phases}.%

\indextext{punctuator|)}%

\indextext{operator|)}

\rSec1[lex.digraph]{Alternative tokens}

\indextext{token!alternative|(}%

Alternative token representations are provided for some operators and

punctuators.

\begin{footnote}

\indextext{digraph}%

These include ``digraphs'' and additional reserved words. The term

``digraph'' (token consisting of two characters) is not perfectly

descriptive, since one of the alternative \grammarterm{preprocessing-token}s is

\tcode{\%:\%:} and of course several primary tokens contain two

characters. Nonetheless, those alternative tokens that aren't lexical

keywords are colloquially known as ``digraphs''.

\end{footnote}

In all respects of the language, each alternative token behaves the

same, respectively, as its primary token, except for its spelling.

\begin{footnote}

Thus the ``stringized'' values\iref{cpp.stringize} of

\tcode{[} and \tcode{<:} will be different, maintaining the source

spelling, but the tokens can otherwise be freely interchanged.

\end{footnote}

The set of alternative tokens is defined in

\tref{lex.digraph}.

\begin{tokentable}{Alternative tokens}{lex.digraph}{Alternative}{Primary}

\tcode{<\%} & \tcode{\{} &

\keyword{and} & \tcode{\&\&} &

\keyword{and_eq} & \tcode{\&=} \\ \rowsep

\tcode{\%>} & \tcode{\}} &

\keyword{bitor} & \tcode{|} &

\keyword{or_eq} & \tcode{|=} \\ \rowsep

\tcode{<:} & \tcode{[} &

\keyword{or} & \tcode{||} &

\keyword{xor_eq} & \tcode{\caret=} \\ \rowsep

\tcode{:>} & \tcode{]} &

\keyword{xor} & \tcode{\caret} &

\keyword{not} & \tcode{!} \\ \rowsep

\tcode{\%:} & \tcode{\#} &

\keyword{compl} & \tcode{\~} &

\keyword{not_eq} & \tcode{!=} \\ \rowsep

\tcode{\%:\%:} & \tcode{\#\#} &

\keyword{bitand} & \tcode{\&} &

& \\

\end{tokentable}%

\indextext{token!alternative|)}

\rSec1[lex.token]{Tokens}

\indextext{token|(}%

\begin{bnf}

\nontermdef{token}\br

identifier\br

keyword\br

literal\br

operator-or-punctuator

\end{bnf}

\indextext{\idxgram{token}}%

There are five kinds of tokens: identifiers, keywords, literals,%

\begin{footnote}

Literals include strings and character and numeric literals.

\end{footnote}

operators, and other separators.

\indextext{whitespace}%

Comments and the characters \unicode{0020}{space}, \unicode{0009}{character tabulation},

\unicode{000b}{line tabulation}, \unicode{000c}{form feed}, and new-line

(collectively, ``whitespace''), as described below, are ignored except

as they serve to separate tokens.

\begin{note}

Whitespace can separate otherwise adjacent identifiers, keywords, numeric

literals, and alternative tokens containing alphabetic characters.

\end{note}

\indextext{token|)}

\rSec1[lex.name]{Identifiers}

\indextext{XID_Start}%

\indextext{XID_Continue}%

\indextext{identifier|(}%

\begin{bnf}

\nontermdef{identifier}\br

identifier-start\br

identifier identifier-continue

\end{bnf}

\begin{bnf}

\nontermdef{identifier-start}\br

nondigit\br

\textnormal{an element of the translation character set with the Unicode property XID_Start}

\end{bnf}

\begin{bnf}

\nontermdef{identifier-continue}\br

digit\br

nondigit\br

\textnormal{an element of the translation character set with the Unicode property XID_Continue}

\end{bnf}

\begin{bnf}

\nontermdef{nondigit} \textnormal{one of}\br

\terminal{a b c d e f g h i j k l m}\br

\terminal{n o p q r s t u v w x y z}\br

\terminal{A B C D E F G H I J K L M}\br

\terminal{N O P Q R S T U V W X Y Z _}

\end{bnf}

\begin{bnf}

\nontermdef{digit} \textnormal{one of}\br

\terminal{0 1 2 3 4 5 6 7 8 9}

\end{bnf}

\indextext{name!length of}%

\indextext{name}%

\begin{note}

The character properties XID_Start and XID_Continue are described by \UAX{44} of the Unicode Standard.

\begin{footnote}

On systems in which linkers cannot accept extended

characters, an encoding of the \grammarterm{universal-character-name} can be used in

forming valid external identifiers. For example, some otherwise unused

character or sequence of characters can be used to encode the

\tcode{\textbackslash u} in a \grammarterm{universal-character-name}. Extended

characters can produce a long external identifier, but \Cpp{} does not

place a translation limit on significant characters for external

identifiers.

\end{footnote}

\end{note}

The program is ill-formed

if an \grammarterm{identifier} does not conform to

Normalization Form C as specified in the Unicode Standard.

\begin{note}

Identifiers are case-sensitive.

\end{note}

\begin{note}

\ref{uaxid} compares the requirements of \UAX{31} of the Unicode Standard

with the \Cpp{} rules for identifiers.

\end{note}

\begin{note}

In translation phase 4,

\grammarterm{identifier} also includes

those \grammarterm{preprocessing-token}s\iref{lex.pptoken}

differentiated as keywords\iref{lex.key}

in the later translation phase 7\iref{lex.token}.

\end{note}

\indextext{\idxcode{import}}%

\indextext{\idxcode{final}}%

\indextext{\idxcode{module}}%

\indextext{\idxcode{override}}%

\indextext{\idxcode{replaceable_if_eligible}}%

\indextext{\idxcode{trivially_relocatable_if_eligible}}%

The identifiers in \tref{lex.name.special} have a special meaning when

appearing in a certain context. When referred to in the grammar, these identifiers

are used explicitly rather than using the \grammarterm{identifier} grammar production.

Unless otherwise specified, any ambiguity as to whether a given

\grammarterm{identifier} has a special meaning is resolved to interpret the

token as a regular \grammarterm{identifier}.

\begin{multicolfloattable}{Identifiers with special meaning}{lex.name.special}

{llll}

\keyword{final} \\

\keyword{override} \\\columnbreak

\keyword{import} \\

\keyword{module} \\\columnbreak

\keyword{post} \\

\keyword{pre} \\\columnbreak

\keyword{replaceable_if_eligible} \\

\keyword{trivially_relocatable_if_eligible} \\

\end{multicolfloattable}

\indextext{\idxcode{_}|see{character, underscore}}%

\indextext{character!underscore!in identifier}%

\indextext{reserved identifier}%

In addition, some identifiers

appearing as a \grammarterm{token} or \grammarterm{preprocessing-token}

are reserved for use by \Cpp{}

implementations and shall

not be used otherwise; no diagnostic is required.

\begin{itemize}

Each identifier that contains a double underscore

\tcode{\unun}