Java Tokens
Tokens are smallest logical components of any programs. Compiler’s first step is to separate entire program into smaller pieces which are tokens. Tokens are building blocks of program.
Tokens are anything in program that can be identified by compiler at the time of tokenization. Tokenization is step of compilation process.
You will better understand after categorising tokens. There are 6 types of tokens and these are:
- Keywords
- Identifiers
- Operators
- Separators
- Literals
- Comments
This is not possible to describe in detail about each of these on this post.
Keywords:
Keywords are reserved words in java which have special meaning are called keywords. There are 50 keywords in java. This is complete list of java keywords.
byte | short | int | long | double | float | char | boolean | void | public |
private | static | import | package | class | interface | extends | implements | if | else |
for | do | while | switch | case | instanceof | return | native | new | synchronized |
strictfp | super | this | volatile | try | catch | finally | final | protected | transient |
default | assert | break | continue | throw | throws | goto | abstract | const | enum |
Identifiers:
Identifiers are names used in program by programmer to any class, method, variables, objects, packages, Interfaces or labels in java programs.
Rules for naming identifiers
- We cannot use special symbols in java identifiers except underscore(_) and dollar($) sign.
- Identifiers should not start from digits.
- Keywords are already reserved and have specific meaning so can not be used as identifiers.
- Reserved literals are also allowed as identifier. Eg. True, false and null
- We can not use same name for more than one identifiers. Remember identifiers are case sensitive therefore variables
rn
andRn
are different.
Examples of valid identifiers
Name, pasa1, $raju, roll_no, String
Examples of invalid identifiers:
Roll-no (Special character used), true(reserved literal), 12sita(Starts with number), try(keyword of java)
Operators:
Operators are symbols that produce some result while applying with some operands. There are different types of operators in java.
Not only symbols but some strings may also be used as operator. Operators are crucial in any language to build statements and expressions. eg: +, -, *, >=, =
etc.
Tokens are smallest logical components of any programs. Compiler’s first step is to separate entire program into smaller pieces which are tokens.
Separators:
Separators are also called punctuators. These are symbols and special characters that make different parts of code separate from each others. These sometimes connect and/or separate one or more tokens or organise a block of code and hence called separators.
Punctuators used in java program are:- ,
(comma), ;
(semicolon), .
(period), (,)
(group of parenthesis), {,}
(Pair of braces), [,]
(pair of square brackets).
Literals:
Literals are any values in java program which do not change during program execution. You can think of constant for literals. These are constant values of some primitive type or string type. Any literal used in java must be one of the following types, Integer, Floating point numbers, Character, Boolean and string or some reference type.
Following example will help for better understanding of java tokens. Lets have some examples
Examples of literals: 1387
, 34.56
, ‘d’
, “sunil”
, true
, null
etc.
Comments:
Comments in java are used for documentation purpose. Some people do not keep comments in token category because it is ignored by compilers during tokenization.
In java multi-line comments start with
/* … ....... */
Single line comments start with
//……
These is all basics about java tokens. Hope you got some concepts about tokens. Ok I’ll write small java program and separate it into individual tokens specifying their token type.
class Test{ public static void main(String args[]){ int w,x=10,y=20; w=x+y; // This line add x and y, then assign to w } }
In this program
class, public, static, void, int
are keywords.Test, main, String, args, w, x, y
are identifiers.=
and+
are operators.10
and20
are literals.{, (, [, ], ), ,, ;, }
are separators.// This line add x and y, then assign to w
is a comment
As you see in above example, compiler break down program into smaller logical individual pieces (words), which are tokens.