Qfyilyi

I am working at a C++ database query engine project. At this time I should be able to parse to schema sql.data (like create table) and the query sql.data (like select ... from ...). So I have 2 parser for each sql.

The Problem I have: I can only use the parser only once and it worked very good. If I try to parse the schema and the query at same time, then I have some following conflicts:
(It just part of the coming conflicts, but I think it can the main reasons and main conflicts)

> ninja

[9/10] Linking CXX executable imlabdb

FAILED: imlabdb 

: && /usr/bin/c++  -g -O0 -fsanitize=address   CMakeFiles/imlabdb.dir/tools/imlabdb.cc.o  -o imlabdb  libimlab.a libschema.a libquery.a vendor/gflags/lib/libgflags.a -pthread && :

libquery.a(query_scanner.cc.o):(.bss+0x140): multiple definition of `yyleng'

libschema.a(schema_scanner.cc.o):(.bss+0x140): first defined here

libquery.a(query_scanner.cc.o):(.bss+0x280): multiple definition of `yyin'

libschema.a(schema_scanner.cc.o):(.bss+0x280): first defined here

libquery.a(query_scanner.cc.o):(.bss+0x2c0): multiple definition of `yyout'

libschema.a(schema_scanner.cc.o):(.bss+0x2c0): first defined here

libquery.a(query_scanner.cc.o):(.data+0x0): multiple definition of `yylineno'

libschema.a(schema_scanner.cc.o):(.data+0x0): first defined here

libquery.a(query_scanner.cc.o):(.data+0x40): multiple definition of `yy_flex_debug'

libschema.a(schema_scanner.cc.o):(.data+0x40): first defined here

libquery.a(query_scanner.cc.o):(.bss+0x380): multiple definition of `yytext'

libschema.a(schema_scanner.cc.o):(.bss+0x380): first defined here

And as you can see, I compile with ninja. I think the local.make and the CMakeList.txt have no problem, so I skip to show them here.

The I try to show my code in the clean way.

imlabdb.cc

int main(int argc, char *argv) {

  imlab::schemac::SchemaParseContext schema_parse_context;

  std::ifstream in_schema("../data/schema.sql"); // schema sql

  auto schema = schema_parse_context.Parse(in_schema);

  in_schema.close();



  imlab::queryc::QueryParseContext query_parse_context;

  std::ifstream in_query("../data/queryc_2.sql"); // query sql

  auto query = query_parse_context.Parse(in_query);

  in_query.close();

}

schema_parse_context.cc

Schema SchemaParseContext::Parse(std::istream &in) {

    beginScan(in);

    imlab::schemac::SchemaParser parser(*this);

    parser.set_debug_level(trace_parsing_);

    parser.parse();

    endScan();

    return {mySchema};  // a container for the create table nokens 

}

query_parse_context.cc

 Query QueryParseContext::Parse(std::istream &in) {

    beginScan(in);

    imlab::queryc::QueryParser parser(*this);  

    parser.set_debug_level(trace_parsing_);

    parser.parse();

    endScan();

    return {myQuery};  // a container for querys

}

Then I show the flex and bison of the schema.

schema_scanner.l without the most unnecessary tokens.

%{

// Header

#include <cerrno>

#include <climits>

#include <cstdlib>

#include <string>

#include <istream>

#include "imlab/schemac/schema_parse_context.h"

#include "./schema_parser.h" 



namespace imlab {

namespace schemac {



// The location of the current token

extern imlab::schemac::location loc;

// The input stream of the scanner

extern std::istream *in;



}  // namespace schemac

}  // namespace imlab



using namespace imlab::schemac;



// Work around an incompatibility in flex (at least versions

// 2.5.31 through 2.5.33): it generates code that does

// not conform to C89.  See Debian bug 333231

// <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=333231>.

#undef yywrap

#define yywrap() 1 



// Declare the yylex function 

#define YY_DECL SchemaParser::symbol_type yylex(SchemaParseContext& sc)

// Configure the scanner to use istreams

#define YY_INPUT(buffer, result, max_size)  

    result = 0; 

    while (true) { 

        int c = in->get(); 

        if (in->eof()) break; 

        buffer[result++] = c; 

        if (result == max_size || c == 'n') break; 

    }

%}



%{

// ---------------------------------------------------------------------------------------------------

// Options

// ---------------------------------------------------------------------------------------------------

%}

%{

// noyywrap:    Disable yywrap (EOF == end of parsing) 

// nounput:     Disable manipulation of input stream

// noinput:     Disable explicit fetch of the next character

// batch:       Scanner in batch-mode (vs. interactive)

// debug:       Write debug info to stderr

// caseless:    Case-insensitive pattern matching

%}

%option noyywrap

%option nounput

%option noinput

%option batch 

%option debug

%option caseless



%{

// Code run each time a token is matched.

// We just update the location of the token.

#define YY_USER_ACTION  { loc.columns(yyleng); } 

%}



%%



%{

// Code runs each time yylex is called.

// Set the beginning of the token to the end of the previous token.

loc.step ();

%}



[ t]+              { loc.step(); }

"n"                { loc.lines (yyleng); loc.step (); }

";"                 { return SchemaParser::make_SEMICOLON(loc); }



%%



// ---------------------------------------------------------------------------------------------------

// Code

// ---------------------------------------------------------------------------------------------------



// The input stream

imlab::schemac::location imlab::schemac::loc;

// The input stream of the scanner

std::istream *imlab::schemac::in = nullptr;



// Begin a scan

void imlab::schemac::SchemaParseContext::beginScan(std::istream &is) {

    yy_flex_debug = trace_scanning_;

    in = &is;

}



// End a scan

void imlab::schemac::SchemaParseContext::endScan() {

    in = nullptr;

}

schema_parser.y also with the unnecessary tokens and cases deleted

%skeleton "lalr1.cc"

%require "3.0.4"

// ---------------------------------------------------------------------------------------------------

// Write a parser header file

%defines

// Define the parser class name

%define parser_class_name {SchemaParser}

// Create the parser in our namespace

%define api.namespace { imlab::schemac }

// Use C++ variant to store the values and get better type warnings (compared to "union")

%define api.value.type variant

// With variant-based values, symbols are handled as a whole in the scanner

%define api.token.constructor

// Prefix all tokens

%define api.token.prefix {SCHEMA_}

// Check if variants are constructed and destroyed properly

%define parse.assert

// Trace the parser

%define parse.trace

// Use verbose parser errors

%define parse.error verbose

// Enable location tracking.

%locations

// Pass the compiler as parameter to yylex/yyparse.

%param { imlab::schemac::SchemaParseContext &sc }

// ---------------------------------------------------------------------------------------------------

// Added to the header file and parser implementation before bison definitions.

// We include string for string tokens and forward declare the SchemaParseContext.

%code requires {

#include <string>

#include <vector>

#include "imlab/schemac/schema_parse_context.h"

}

// ---------------------------------------------------------------------------------------------------

// Import the compiler header in the implementation file

%code {

imlab::schemac::SchemaParser::symbol_type yylex(imlab::schemac::SchemaParseContext& sc);

}

%code {

std::string insertTableId;

int positionToInsert;

}

// ---------------------------------------------------------------------------------------------------

// Token definitions but deleted the most of them

%token <int>            INTEGER_VALUE    "integer_value"

%token <std::string>    IDENTIFIER       "identifier"



// ---------------------------------------------------------------------------------------------------





// Define error function

void imlab::schemac::SchemaParser::error(const location_type& l, const std::string& m) {

    sc.Error(l.begin.line, l.begin.column, m);

}

Now the flex and bison for query, but almost written in the same way.

query_scanner.l

%{

// Header

#include <cerrno>

#include <climits>

#include <cstdlib>

#include <string>

#include <istream>

#include "imlab/queryc/query_parse_context.h"

#include "./query_parser.h" 



namespace imlab {

namespace queryc {



// The location of the current token

extern imlab::queryc::location loc;

// The input stream of the scanner

extern std::istream *in;



}  // namespace queryc

}  // namespace imlab



using namespace imlab::queryc;



// Work around an incompatibility in flex (at least versions

// 2.5.31 through 2.5.33): it generates code that does

// not conform to C89.  See Debian bug 333231

// <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=333231>.

#undef yywrap

#define yywrap() 1    

// Declare the yylex function 

#define YY_DECL QueryParser::symbol_type yylex(QueryParseContext& sc)

// Configure the scanner to use istreams

#define YY_INPUT(buffer, result, max_size)  

    result = 0; 

    while (true) { 

        int c = in->get(); 

        if (in->eof()) break; 

        buffer[result++] = c; 

        if (result == max_size || c == 'n') break; 

    }

%}



%{

// ---------------------------------------------------------------------------------------------------

// Options

// ---------------------------------------------------------------------------------------------------

%}

%option noyywrap

%option nounput

%option noinput

%option batch 

%option debug

%option caseless



%{

#define YY_USER_ACTION  { loc.columns(yyleng); }  

%%



%{

loc.step ();

%}



[ t]+              { loc.step(); }

"n"                { loc.lines (yyleng); loc.step (); }

";"                 { return QueryParser::make_SEMICOLON(loc); }

%%



// ---------------------------------------------------------------------------------------------------

// Code

// ---------------------------------------------------------------------------------------------------



// The input stream

imlab::queryc::location imlab::queryc::loc;

// The input stream of the scanner

std::istream *imlab::queryc::in = nullptr;



// Begin a scan

void imlab::queryc::QueryParseContext::beginScan(std::istream &is) {

    yy_flex_debug = trace_scanning_;

    in = &is;

}



// End a scan

void imlab::queryc::QueryParseContext::endScan() {

    in = nullptr;

}

query_parser.y

%skeleton "lalr1.cc"

%require "3.0.4"

// ---------------------------------------------------------------------------------------------------

%defines

%define parser_class_name {QueryParser}

%define api.namespace { imlab::queryc }

%define api.value.type variant

%define api.token.constructor

%define api.token.prefix {QUERY_}

%define parse.assert

%define parse.trace

%define parse.error verbose

%locations

%param { imlab::queryc::QueryParseContext &sc }

// ---------------------------------------------------------------------------------------------------

%code requires {

#include <string>

#include <vector>

#include "imlab/queryc/query_parse_context.h"

}

// ---------------------------------------------------------------------------------------------------

%code {

imlab::queryc::QueryParser::symbol_type yylex(imlab::queryc::QueryParseContext& sc);

}

// %code {

// std::string insertTableId;

// int positionToInsert;

// }

// ---------------------------------------------------------------------------------------------------

// Token definitions -- most of them deleted

%token <int>            INTEGER_VALUE    "integer_value"

%token <std::string>    IDENTIFIER       "identifier"



// ---------------------------------------------------------------------------------------------------

// Define error function

void imlab::queryc::QueryParser::error(const location_type& l, const std::string& m) {

    sc.Error(l.begin.line, l.begin.column, m);

}

// ---------------------------------------------------------------------------------------------------

In short:
What I can be sure: These two flex and bison and other things like imlabdb.cc are working well, if I only use one of them in the imlabdb.cc. The conflict which I added to the top only happens, when I try to use two parser at same time.
What I want: To use the two parser at same time and avoid the naming conflicts.
Thanks!

(If I deleted to much code or is the current code not sufficient, please post. I will fix it.)

Although @serge is correct about your specific issue, I will answer the question more generic.

Flex and bison create a C file with a bunch of functions and variables that are public, such as yylex and yyparse. This is unfortunate, if you have two parser or lexers in one program. The program does not know which yylex and yyparse to call. So you need to name them differently.

To do this in flex you use the option

%option prefix="foo"

This will rename all symbols that have yy prefex to foo, so foolex. So in your case you then use one prefix for the one lexer and an other for the other.

On bison same thing, just different syntax. Depending on your version of bision you use either (new):

%define api.prefix {foo}

or (old):

%name-prefix "foo"

Which will then generate fooparse.

For reference see the documentation:

• Flex Manual: Options
  


• Bison Manual: 3.8 Multiple Parsers in the Same Program
  

Both the schema (DDL) and queries (DML) are the parts of SQL language. You don't need to have two lexers/parsers but only one based on SQL grammar definitions. For example, have a look on the "YACC SQL Grammar Reference".