Qfyilyi

I have a program that takes incoming text converts it to type Reader and returns the next token, be it a word, or a space (non-word). It is not behaving as expected.

To be as specific as possible, here is my testing infrastructure in Eclipse using JUnit4:

@Test

    public void testGetNextTokenWord() throws IOException {

        Reader in = new StringReader("Aren't you ntired"); 

        TokenScanner d = new TokenScanner(in);

        try {

            assertTrue("has next", d.hasNext());

            assertEquals("Aren't", d.next());

            assertTrue("has next", d.hasNext());

            assertEquals(" ", d.next());

            assertTrue("has next", d.hasNext());

            assertEquals("you", d.next());

            assertTrue("has next", d.hasNext());

            assertEquals(" n", d.next());

            assertTrue("has next", d.hasNext());

            assertEquals("tired", d.next());



            assertFalse("reached end of stream", d.hasNext());

        } finally {

            in.close();

        }

    }

I will post complete code to facilitate assistance on this problem and then post expected and observed behavior:

//Reads as much to determine hasNext() and next()

    public TokenScanner(java.io.Reader in) throws IOException {



        //Throw exception if null

        if (in == null) {

            throw new IllegalArgumentException();

        }



        //Read in token

        try {   



            System.out.println("TokenScanner!");

            //Create new token scanner for argued reader

            this.tokenScanner = in;



            //Read next character

            ch = tokenScanner.read();

        }



        //Throw exception if error in reading

        catch (IOException e){

            ch = -1;

        }    

    }



//Determines whether the argued character is a valid word character.

    public static boolean isWordCharacter(int c) {



        //Cast int character to a char

        char character = (char)c;



        //Return true if character is valid word character

        if(Character.isLetter(character) || character == ''') {

            return true;    

        }



        //Return false otherwise

        return false;

    }



//Determine whether another token is avaialble

    public boolean hasNext() {



        //Leverage invariant

        return ch != -1 ;

    }

And the function where alot of my headache is coming from (potentially)

//Determine next token

    public String next() {



        //End of stream reached

        if(!hasNext()) {

            throw new NoSuchElementException();

        }



        //Initialize variable to hold token

        String word = "";



        try {



            //Character is a word character

            while(isWordCharacter(ch)) {

                word = word + (char)ch;

                ch = tokenScanner.read();



            }



            //Character is a space

            while(!Character.isWhitespace(ch)) {

                word = word + (char)ch;

                ch = tokenScanner.read();



            }           



            System.out.println("Word is: "+ word);

            return word;

        }



        //Exception catching

        catch(Exception e) {



            throw new NoSuchElementException();



        }   

    }

The expected output given the testing infrastructure above is:

TokenScanner!

Word is: Aren't

Word is: you

Word is: /*Not sure how to represent newline in output*/

Word is: tired

The actual output below is:

TokenScanner!

Word is: Aren't

Word is:

The question here is why is this happening?

My output shows that the first test to fail is:

assertEquals(" ", d.next());

The fundamental issue here is how I'm representing non-words (spaces). The last test fails as well. Any help here is appreciated!