8.2K

Java Tokens

Tokens are smallest logical components of any programs. Compiler’s first step is to separate entire program into smaller pieces which are tokens. Tokens are building blocks of program.

Tokens are anything in program that can be identified by compiler at the time of tokenization. Tokenization is step of compilation process.

You will better understand after categorising tokens. There are 6 types of tokens and these are:

1. Keywords
2. Identifiers
3. Operators
4. Separators
5. Literals
6. Comments

This is not possible to describe in detail about each of these on this post.

Keywords:

Keywords are reserved words in java which have special meaning are called keywords. There are 50 keywords in java. This is complete list of java keywords.

byte	short	int	long	double	float	char	boolean	void	public
private	static	import	package	class	interface	extends	implements	if	else
for	do	while	switch	case	instanceof	return	native	new	synchronized
strictfp	super	this	volatile	try	catch	finally	final	protected	transient
default	assert	break	continue	throw	throws	goto	abstract	const	enum

Identifiers:

Identifiers are names used in program by programmer to any class, method, variables, objects, packages, Interfaces or labels in java programs.

Rules for naming identifiers

1. We cannot use special symbols in java identifiers except underscore(_) and dollar($) sign.
2. Identifiers should not start from digits.
3. Keywords are already reserved and have specific meaning so can not be used as identifiers.
4. Reserved literals are also allowed as identifier. Eg. True, false and null
5. We can not use same name for more than one identifiers. Remember identifiers are case sensitive therefore variables rn and Rn are different.

Examples of valid identifiers

Name, pasa1, $raju, roll_no, String

Examples of invalid identifiers:

Roll-no (Special character used), true(reserved literal), 12sita(Starts with number), try(keyword of java)

Operators:

Operators are symbols that produce some result while applying with some operands. There are different types of operators in java.

Not only symbols but some strings may also be used as operator. Operators are crucial in any language to build statements and expressions. eg: +, -, *, >=, = etc.

Tokens are smallest logical components of any programs. Compiler’s first step is to separate entire program into smaller pieces which are tokens.

Separators:

Separators are also called punctuators. These are symbols and special characters that make different parts of code separate from each others. These sometimes connect and/or separate one or more tokens or organise a block of code and hence called separators.

Punctuators used in java program are:- , (comma), ; (semicolon), . (period), (,) (group of parenthesis), {,} (Pair of braces), [,] (pair of square brackets).

Literals:

Literals are any values in java program which do not change during program execution. You can think of constant for literals. These are constant values of some primitive type or string type. Any literal used in java must be one of the following types, Integer, Floating point numbers, Character, Boolean and string or some reference type.

Following example will help for better understanding of java tokens. Lets have some examples

Examples of literals: 1387, 34.56, ‘d’, “sunil”, true, null etc.

Comments:

Comments in java are used for documentation purpose. Some people do not keep comments in token category because it is ignored by compilers during tokenization.

In java multi-line comments start with

/* …
.......
*/

Single line comments start with

//……

These is all basics about java tokens. Hope you got some concepts about tokens. Ok I’ll write small java program and separate it into individual tokens specifying their token type.

class Test{
    public static void main(String args[]){
       int w,x=10,y=20;
        w=x+y; // This line add x and y, then assign to w
  }
}

In this program

• class, public, static, void, int  are keywords.
• Test, main, String, args, w, x, y are identifiers.
• = and + are operators.
• 10 and 20 are literals.
• {, (, [, ], ), ,, ;, }  are separators.
• // This line add x and y, then assign to w is a comment

As you see in above example, compiler break down program into smaller logical individual pieces (words), which are tokens.