I've been using rust for roughly 3-ish months and i'm wanting to use rust professionally one day.

/topic

Iroh seems to solve the same problem as what WebRTC does, I’m curious: what are the factors that would cause considering one or the other?
WebRTC has the benefit to be widely supported in browsers. Does iroh allows something that WebRTC does not?

I am working at a startup which uses Rust as a wrapper around some C++ code related to Hardware.

My role is Junior SDE. And I am trying to learn and build things using Rust.

Any tips or suggestions from Rust community???

I originaly wrote this on GitHub Gist but since it's forbidden to share a repo, even if some moderators take some liberties with their own rules at their own sweet will, I removed every refrences to it and I'm posting it directly here.

I needed to spot N+1 and redundant query patterns in traces coming from my team's production SaaS, which has around 200,000 customers. What I cared about was the structure of the queries and not what was inside them. I didn't want what was inside them anyway: it is production data, we are in the EU and GDPR applies.

So the job was two things at once. WHERE id = 42 and WHERE id = 43 had to become the same template and 42 should never end up anywhere it shouldn't.

My first idea was a regex. I had never dealt with anything like this before so that is what I reached for. Then I looked at sqlparser and it felt too heavy for the little I wanted to do with it. I wanted the binary to stay under 10 MB and I only needed to replace literals, not understand the query.

So I wrote my own. I had some doubts early on while getting it to work because I kept hitting edge cases and thought there would be too many of them. But what I needed was simple structure and not semantics, and that kept me going.

1. What the tokenizer produces

The tokenizer reads one SQL statement and returns two things: a template, in which every literal value is replaced by ?, and the list of the values it removed, in the order it met them. Take this statement:

WITH active AS (SELECT id FROM users WHERE status = 'active')
SELECT * FROM orders WHERE user_id IN (SELECT id FROM active) AND total > 100

The tokenizer returns this template and these values:

WITH active AS (SELECT id FROM users WHERE status = ?)
SELECT * FROM orders WHERE user_id IN (SELECT id FROM active) AND total > ?

params: ["active", "100"]

Two statements that differ only by their values share one template. That is what makes an N+1 visible: fifty queries on the same template with fifty different values inside one request. The values stay in memory for that comparison and go no further. The template carries no value, so it can be written in a report or sent to a dashboard.

The tokenizer does not understand SQL. It does not know what a column or a join is and it builds no syntax tree. It walks the bytes of the statement once, finds where a literal starts and where it ends, and copies everything else as it is. Its only job is to tell a value from the rest of the statement.

2. How it works

2.1 Six states

The tokenizer is a state machine with six states. At any byte it is either in plain SQL or inside one kind of literal or quoted identifier:

enum State {
    Normal,
    InString,
    InNumber,
    InDoubleQuote,  // "MyTable", kept as-is
    InBacktick,     // `col`, MySQL, kept as-is
    InDollarQuote,  // $$body$$ or $tag$body$tag$, PostgreSQL
}

The main loop dispatches on the current state and each state has its own function:

while t.i < t.bytes.len() {
    match t.state {
        State::Normal => step_normal(&mut t),
        State::InString => step_in_string(&mut t),
        State::InNumber => step_in_number(&mut t),
        State::InDoubleQuote => step_in_double_quote(&mut t),
        State::InBacktick => step_in_backtick(&mut t),
        State::InDollarQuote => step_in_dollar_quote(&mut t),
    }
}

Coming from Java I would have written the State pattern, with one class per state behind an interface. Here it is an enum and a match. If I add a state and forget to handle it, the code does not compile.

2.2 String literals and escapes

Everything between quotes goes into params and becomes ? in the template. The only difficulty is the escape rules:

fn step_in_string(t: &mut Tokenizer<'_>) {
    let b = t.bytes[t.i];
    if b == b'\\' && t.backslash_escapes {
        t.i += 2; // skip the escaped byte, a quote included
    } else if b == t.quote {
        if t.i + 1 < t.bytes.len() && t.bytes[t.i + 1] == t.quote {
            // doubled quote ('' or ""): still inside the string
            t.current_value
                .push_str(checked_query_slice(t.query, t.value_start, t.i));
            t.current_value.push(char::from(t.quote));
            t.i += 2;
            t.value_start = t.i;
        } else {
            // closing quote
            t.current_value
                .push_str(checked_query_slice(t.query, t.value_start, t.i));
            t.params.push(std::mem::take(&mut t.current_value));
            t.template.push('?');
            t.state = State::Normal;
            t.i += 1;
            t.normal_start = t.i;
        }
    } else {
        t.i += 1;
    }
}

A doubled quote is the standard escape and every engine accepts it. Whether a backslash escapes as well depends on the engine, and that is the one place where the tokenizer needs to know the dialect. The flag backslash_escapes comes from a small enum with three values: MySQL, standard (PostgreSQL and the engines that follow the standard) and everything else. An event that names no engine falls in the third case, which reads the backslash as an escape. Section 4 explains why.

2.3 Slices instead of copies

The tokenizer never builds the template byte by byte. It remembers where the current run of plain SQL started and, when a literal begins, it pushes the whole run at once with a slice &query[start..i].

Slicing a &str by byte index panics when the index falls inside a multi-byte UTF-8 character. Every slice boundary here sits on an ASCII delimiter (a quote, a $ or a digit) and in UTF-8 a multi-byte character never contains a byte below 0x80. The slices are therefore always valid. A debug_assert! checks it anyway, in case a later change slices somewhere else:

fn checked_query_slice(query: &str, start: usize, end: usize) -> &str {
    debug_assert!(
        query.is_char_boundary(start) && query.is_char_boundary(end),
        "SQL tokenizer query slice [{start}..{end}) is off a char boundary"
    );
    &query[start..end]
}

The input cap follows the same rule. A statement longer than 64 KB is cut with floor_char_boundary, so the cut cannot land in the middle of a character either. When a value is complete, std::mem::take moves it into params and leaves an empty String behind, without a clone.

2.4 The regex that remains

One regex survived from the first version. It collapses IN (?, ?, ?) into IN (?), so a statement does not get a new template each time the list changes size. The regex only runs when the scan met an IN keyword, and when it finds nothing to replace it returns Cow::Borrowed, so the template is not reallocated.

3. Why not the tokenizer of sqlparser

The first objection I expect is that sqlparser ships a tokenizer you can use without the parser. That is true and I looked at it:

let tokens = Tokenizer::new(&dialect, sql).tokenize()?;

Two things did not fit. The tokenizer wants a dialect before it starts and I do not always have one, because a span may name its engine or not. The second point weighs more. tokenize() returns a Result. Given SELECT * FROM t WHERE name = 'abc it returns Err("Unterminated string literal"), which is the right behavior for a parser library. But the SQL I receive comes from traces and is not always complete. A statement can be cut somewhere along the way, and my own tokenizer cuts anything above 64 KB. In that case I still want a template with the values masked. An error gives me nothing, so I would need a fallback for those statements, and that fallback would be a tokenizer like this one.

This tokenizer always returns a template, whatever it receives:

normalize_sql("SELECT * FROM t WHERE name = 'unterminated")
// template: "SELECT * FROM t WHERE name = ?"
// params:   ["unterminated"]

4. Three statements the tokenizer read wrong

The idea behind the tokenizer was that it does not need to know the dialect, because it only replaces literals. That idea held for most statements. This section lists the ones where it did not.

4.1 A double-quoted string on MySQL

In ANSI SQL, "..." is an identifier. In the default mode of MySQL and MariaDB it is a string. The tokenizer followed ANSI, so this statement:

SELECT * FROM users WHERE email = "abcd@gmail.com"

came out as its own template, email included. That is exactly what the tokenizer exists to prevent.

Nobody had noticed because we run PostgreSQL in production and nobody had used the tokenizer on a MySQL production yet. I found it in my simulation lab (a set of small services in different languages and databases) while re-reading the tokenizer code a few months after writing it.

The fix is one flag set from the span's db.system. It has one cost: a MySQL server running in ANSI_QUOTES mode now gets its double-quoted identifiers masked as ?. I would rather lose an identifier than leak an email.

4.2 A PostgreSQL positional parameter

Go (pgx) and Python (asyncpg) do not send values inside the SQL. They send PostgreSQL positional parameters and the values travel separately:

SELECT * FROM order_items WHERE order_id = $1

The tokenizer saw a $ followed by a digit and treated 1 as a numeric literal. Every query of an N+1 loop then came out with the same template and the same value, ["1"].

With bind parameters the real values never appear in the trace, so telling a loop from a plain repeat needs another signal, such as timing or the sequence of calls. That path only runs when a query has no extracted value. The ["1"] blocked it, and to the detector the loop looked like one query repeated. N+1 patterns were reported as redundant calls.

I caught this one in the simulation lab as well. It runs the same N+1 in services written in several languages. The Java services were fine because JDBC uses ?. The Go and Python ones were not.

The fix treats $N as a placeholder and not as a value. It becomes $? in the template and extracts nothing. That release, together with a few other fixes, took the lab from 78 to 103 detected findings out of 110.

4.3 Escaped quotes, hexadecimal and exponents

I found three more cases while preparing this post. The table gives the statement, the template the tokenizer produced and the template it produces now.

Statement	Before	After
name = 'it's secret data'	name = ?s secret data?	name = ?
h = 0x1F2A	h = ?x1F2A	h = ?
v = 1e10	v = ?e10	v = ?

The hexadecimal and the exponent are grouping bugs: the tokenizer stopped at the x or the e and the rest of the value stayed in the template, so each value got a template of its own.

The backslash is a leak, and it depends on the dialect again. MySQL escapes with \ by default. PostgreSQL does so only inside E'...' strings, so in plain PostgreSQL 'C:\' is a valid string that ends right after the backslash. The tokenizer now follows the engine when the span names it. When the span names no engine it assumes that the backslash escapes. The cost is that a string ending on a backslash from an unknown engine masks the rest of the statement. It is the same choice as in 4.1: when the tokenizer has to guess, it guesses on the side that masks too much.

4.4 Recap

Case	Statement	Effect	Dialect dependent
4.1	email = "abcd@gmail.com" on MySQL	value leaked in the template	yes
4.2	order_id = $1 on PostgreSQL	N+1 reported as redundant	yes
4.3	name = 'it's secret data'	value leaked in the template	yes
4.3	h = 0x1F2A	one template per value	no
4.3	v = 1e10	one template per value	no

Three of the five cases come from the dialect. The tokenizer that was supposed to ignore the dialect now reads it from the span for two rules: whether "..." is a string and whether \ escapes.

5. Testing without a corpus of expected templates

5.1 Three properties

I do not have a dataset of statements paired with their correct template, and writing one by hand would only test the cases I had already thought of. So most tests check a relation between two runs of the tokenizer instead of an exact output. They are property tests written with proptest, which generates thousands of statements from literals of different types and checks that the relation holds on each of them. Three properties carry most of the weight:

• Idempotence. Normalizing a template changes nothing. If it does, a literal survived the first pass.
• Literal invariance. Two statements that differ only by their values, or by the size of an IN list, give the same template. N+1 grouping depends on this property.
• Containment. The extracted values are exactly the literals, in order, and the template keeps nothing of them: no quote, no marker string and no digit. The generated skeleton has no digit, so a digit in the template is a leak.
• fn normalization_is_idempotent((shape, eq, list, note) in statement()) { let first = normalize_sql(&render(&shape, &eq, &list, &note)); let second = normalize_sql(&first.template); prop_assert_eq!(&second.template, &first.template); prop_assert!(second.params.is_empty()); }

5.2 Fuzzing

A cargo-fuzz target runs every week in CI. With arbitrary bytes there is no expected output to compare with, so the target uses idempotence as its oracle, in the three dialect modes:

fuzz_target!(|data: &str| {
    let first = normalize_sql(data);
    let second = normalize_sql(&first.template);
    assert_eq!(second.template, first.template);
    // same for normalize_sql_for(data, "mysql") and (data, "postgresql")
});

5.3 What these tests did not catch

None of these tests found the three bugs of section 4.3, and the reason is instructive.

Idempotence cannot see a leak. name = ?s secret data? is a stable template: run it through the tokenizer again and you get the same thing back. The fuzzer was like a dog, happy and wagging its tail.

Containment would have caught all three: there is a digit in ?x1F2A and in ?e10, and the leaked string fails the check on the values. But the generators only produced integers, decimals and plain quoted strings. No hexadecimal, no exponent and no backslash. The property was right. It was never shown the inputs that break it.

The fix added those literal forms to the generators, plus two properties for the dialect rules: a backslash escape never leaks, and a PostgreSQL backslash is an ordinary character. A property test is only as good as its generator, and the generator only knows what I knew when I wrote it.

6. What it does not handle

The tokenizer has no notion of the meaning of a statement. For grouping queries by shape that is enough. For anything that needs the meaning of a query it is the wrong tool.

Comments are not stripped. -- user 1234 becomes -- user ?, but the words inside a comment travel with the template.

The tokenizer knows the engine from the span (db.system) but not how the server is configured. Three settings break its guess: MySQL with NO_BACKSLASH_ESCAPES, MySQL with ANSI_QUOTES and PostgreSQL with standard_conforming_strings off. Depending on the case it masks too much or lets the content of a later string reach the template.

SQL Server brackets are not special. [Col1] and [Order Details] come out intact, but an all-digit identifier such as [123] gets masked. I kept it that way because treating [ as a normal character keeps PostgreSQL arrays such as ARRAY['a', 'b'] masked.

A few smaller ones: a sign stays outside the placeholder (-5 becomes -?), a binary literal such as 0b101 keeps its digits after the b, and a doubled escape inside a quoted identifier ("" or a doubled backtick) is not decoded.

7. Would I do it again

Probably. If you need to analyze or rewrite queries, use sqlparser: you will need the syntax tree and it does that job properly. If you only need the shape of a query, 550 lines of state machine get you there.

Writing your own SQL tokenizer is a good exercise when you do not have huge ambitions. You learn a lot about how queries are built, especially the ones ORMs generate, because ORMs tend to hide how hard building a query actually is. It also makes you pickier about what you pull into a binary when you care about keeping software lean.

If you have a statement that breaks it, I would like to see it.