Multiple query enhanced feedback

[cpainterwakefield]

Added ability to iterate over statements in a multi-statement SQL string. Also added Statement methods accessing SQLite's sqlite3_sql() and sqlite3_normalized_sql() functions.

[lovasoa]

Great ! I think this is close to what we want. Ideally we would be able to have client code be as simple as :

for (const stmt of db.iterateStatements(txt))
    console.log(stmt.getAsObject())

This would require adding an @@iterator method to the returned object (if the iterator symbol is available in the current runtime.

We need to either automatically free the statements after the iteration is over or make it very explicit in the documentation that we don't. Maybe add an example in the jsdoc, to show how this is supposed to be used.

[cpainterwakefield]

Comments above. I will do my best to address your points, but there are aspects of Javascript, and particularly Javascript + emscripten, that I am not well versed in. Long time programmer, but self-taught in Javascript.

[lovasoa]

I just made some testing with your implementation. It looks like db.iterateStatements("\\") returns an infinite iterator (instead of an iterator that raises an error and then finishes).

[cpainterwakefield]

Yes, that is because I didn't actually implement the iterable protocols - I wasn't sure that was what was wanted/needed. I used the iterable protocols as an inspiration, but tested it manually (not with for .. of).

[cpainterwakefield]

OK, so if we want something that works only in a for...of loop and then gets invalidated on exception or when done, that gives me 95% of what I want, which is better than what I have now. The 5% is that I won't have any access to the part of the query that caused an exception when there is an error, because, if I understand correctly, I will have deallocated the memory storing the sql that any pointers I have point into, in the iterator.

I.e., I want to be able to do something like:

let it = db.iterateStatements(sql);
let results = [];
try {
    for (stmt of it) {
        // gather column names, data, rowcount, normalized SQL, etc.
        // add an object to my results collection
    }
} except {
   let lastSql = it.getLastSql();  <--- this I cannot do?
   // add an object with error message and last sql to my results collection
}

Maybe there's a workaround where I copy the tail pointer into a string in the iterator object when there's an exception? Would that make sense to you?

[cpainterwakefield]

NVM, I think I have it working. I should be able to push an update tomorrow.

[lovasoa]

If you want to access your "lastSql" after an error, you can attach it to the error type you are throwing... But I don't see the interest, since SQLite doesn't make any guarantee concerning where pzTail is going to end after an invalid SQL statement. What do you want to do with that string ?

[cpainterwakefield]

I was never using pzTail itself as the lastSql value, I was trying to subtract pzTail from the nextSql string.

My new code doesn't use pzTail at all, and I don't call it "lastSql" now, as that is confusing (you helped my clarify my thinking on this). Instead, the iterator now has a method "getRemainingSql()", which simply reports the value pointed to by the nextSql pointer. This works, and makes more sense; if a user wants to see the remaining SQL after each statement is prepared, they can do so (not sure why they would, but it is there). In my case, if there is an exception, I can access the remaining SQL to report that "the error occurred in here somewhere".

[lovasoa]

My new code doesn't use pzTail at all, and I don't call it "lastSql" now, as that is confusing (you helped my clarify my thinking on this). Instead, the iterator now has a method "getRemainingSql()", which simply reports the value pointed to by the nextSql pointer. This works, and makes more sense; if a user wants to see the remaining SQL after each statement is prepared, they can do so (not sure why they would, but it is there). In my case, if there is an exception, I can access the remaining SQL to report that "the error occurred in here somewhere".

Perfect, this sounds like what we want

[lovasoa]

I marked everything that was resolved as such. Once the remaining open points are fixed, we can merge.

[cpainterwakefield]

I think I've addressed all the change requests... do you want me to resolve the open conversations, or do you prefer to do that?

[lovasoa]

Suggested change

	* **Warning**: When you close a database (using db.close()),
	* using any statement iterators created by the database will
	* result in undefined behavior.

[lovasoa]

When the iterator is done, we return { done: true } which doesn't match this definition. I think the most useful type definition we can give is {done : true, value: undefined} | {done: false, value: Statement}.

[lovasoa]

We are almost done !

[lovasoa]

I marked all the github conversations that you addressed as resolved.

[cpainterwakefield]

Fixed those two documentation issues. I'm very excited to see these changes merged (and hopefully released soon)!

[lovasoa]

🎉

[lovasoa]

It's on NPM : https://www.npmjs.com/package/sql.js 😃

[cpainterwakefield]

Sweet!