sqlite3_prepare_v2 returns SQLITE_OK with a NULL statement handle when
the input contains no SQL. exec() already handled this; query() forwarded
the NULL handle to bind(), which crashed in sqlite3_clear_bindings(NULL).
Make query() skip NULL statements like exec() does, and make SQLiteRows
safe against a nil underlying statement so the empty-rows return value
does not crash.
Closes#1390
This avoids an unnecessary reset when the cache is full, guarantees
a statement cannot enter the cache without being reset/cleared, and
fixes a leak where sqlite3_finalize was not called when reset failed.
Finalize all cached statements even if one fails. Leaving a
finalized statement in the cache map would be a use-after-finalize
bug per SQLite documentation.
Move extern declarations for _sqlite3_*_blocking functions before
_sqlite3_exec_no_args which references them. Remove unused
_sqlite3_prepare_v2_nolen function.
- Add _sqlite3_exec_no_args() C function that combines prepare+step+finalize
into a single CGO crossing for parameterless exec (most common case)
- Add _sqlite3_reset_clear() C function that combines sqlite3_reset and
sqlite3_clear_bindings into a single CGO crossing
- Use semaphore channel instead of result struct channel in context-aware
exec/Next paths to reduce allocations
- Use time.AppendFormat with stack buffer to avoid heap allocation in
time.Time binding
- Optimize bindNamedIndices to reuse a single buffer instead of 3
separate C.CString allocations
- Remove intermediate bindIndices slice allocation in named parameter
binding path
- Pass explicit query length to sqlite3_prepare_v2 to avoid C-side strlen
benchstat (n=8):
BenchmarkExec: -29.44% sec/op, -50% B/op, -33% allocs/op
BenchmarkQuery: -9.83% sec/op
BenchmarkParams: -6.38% sec/op
geomean: -6.72% sec/op
- Replace len(args[start:start+na]) > 0 with na > 0 to avoid slice bounds check
- Use range loops instead of manual index loops for cols/decltype slices
- Use range variable v.Ordinal instead of re-indexing args[i].Ordinal
- Add bounds hint for decltype access in nextSyncLocked loop
* Add ability to set an int64 file control
* Update documentation
* Remove duplicate err check in test
* Update sqlite3.go
Co-authored-by: rittneje <rittneje@gmail.com>
---------
Co-authored-by: rittneje <rittneje@gmail.com>
The commit removes the use of runtime.SetFinalizer to finalize
SQLiteRows since only serves to close the associated SQLiteStmt which
already has a registered finalizer.
It also fixes a race and potential panic in SQLiteRows.Close around the
SQLiteRows.s field (*SQLiteStmt) which is accessed without a mutex being
held, but modified with it held (null'd out). Further the mutex we are
holding is that of the SQLiteStmt so a subsequent call to Close will
cause a panic sine it'll attempt to dereference a nil field. The fix
here is to add a mutex for closing to SQLiteRows.
Since we now also set the s field to nil when closing this commit
removes the "closed" field (since checking if s is nil is the same) and
also changes the type of "nc" (number of columns) to an int32 so that we
can pack the nc and cls fields, and add the close mutex without making
the struct any bigger.
```
goos: darwin
goarch: arm64
pkg: github.com/charlievieth/go-sqlite3
cpu: Apple M4 Pro
│ x1.txt │ x4.txt │
│ sec/op │ sec/op vs base │
Suite/BenchmarkExec/Params-14 719.2n ± 2% 716.9n ± 1% ~ (p=0.897 n=10)
Suite/BenchmarkExec/NoParams-14 506.5n ± 3% 500.1n ± 0% -1.25% (p=0.002 n=10)
Suite/BenchmarkExecContext/Params-14 1.584µ ± 0% 1.567µ ± 1% -1.07% (p=0.007 n=10)
Suite/BenchmarkExecContext/NoParams-14 1.524µ ± 1% 1.524µ ± 1% ~ (p=0.539 n=10)
Suite/BenchmarkExecStep-14 443.9µ ± 3% 441.4µ ± 0% -0.55% (p=0.011 n=10)
Suite/BenchmarkExecContextStep-14 447.8µ ± 1% 442.9µ ± 0% -1.10% (p=0.000 n=10)
Suite/BenchmarkExecTx-14 1.643µ ± 1% 1.640µ ± 0% ~ (p=0.642 n=10)
Suite/BenchmarkQuery-14 1.968µ ± 3% 1.821µ ± 1% -7.52% (p=0.000 n=10)
Suite/BenchmarkQuerySimple-14 1.207µ ± 2% 1.040µ ± 1% -13.84% (p=0.000 n=10)
Suite/BenchmarkQueryContext/Background-14 2.400µ ± 1% 2.320µ ± 0% -3.31% (p=0.000 n=10)
Suite/BenchmarkQueryContext/WithCancel-14 8.847µ ± 5% 8.512µ ± 4% -3.79% (p=0.007 n=10)
Suite/BenchmarkParams-14 2.131µ ± 2% 1.967µ ± 1% -7.70% (p=0.000 n=10)
Suite/BenchmarkStmt-14 1.444µ ± 1% 1.359µ ± 1% -5.89% (p=0.000 n=10)
Suite/BenchmarkRows-14 61.57µ ± 1% 60.24µ ± 1% -2.16% (p=0.000 n=10)
Suite/BenchmarkStmtRows-14 60.15µ ± 1% 59.08µ ± 1% -1.78% (p=0.000 n=10)
Suite/BenchmarkQueryParallel-14 960.9n ± 1% 420.8n ± 2% -56.21% (p=0.000 n=10)
geomean 4.795µ 4.430µ -7.62%
```
This commit adds the SQLiteConn.FileControlInt() method which calls the
underlying sqlite3_file_control() function with an int argument. This can
be used for low-level operations on SQLite databases such as persisting
the WAL file after database close.
The busy_timeout pragma was added in sqlite 3.7.15 as an alternative
to calling sqlite3_busy_timeout directly:
https://sqlite.org/pragma.html#pragma_busy_timeout
While there's no functional change here, using the pragma does align
setting busy_timeout with other settings and removes the special case
for calling sqlite3_busy_timeout directly.
This can be used like in the test; I wrote a little wrapper around
sql.DB which uses this, and allows concurrent reads but just one single
write. This is perhaps a better generic "table locked"-solution than
setting the connections to 1 and/or cache=shared (although even better
would be to design your app in such a way that this doesn't happpen in
the first place, but even then a little seat belt isn't a bad thing).
The parsing adds about 0.1ms to 0.2ms of overhead in the wrapper, which
isn't too bad (and it caches the results, so only needs to do this
once).
At any rate, I can't really access functions from sqlite3-binding.c from
my application, so expose it via SQLiteStmt.
Add a shortcut for PRAGMA cache_size; this is a pretty useful setting:
the default of -2000 (2M) is not especially high, and a lot of people
will probably want to increase this.
For example, while running a bunch of fairy expensive queries in
parallel:
With SetMaxOpenConns(1):
-2000: 5762ms
-20000: 4714ms
With SetMaxOpenConns(20):
-2000: 3067ms
-20000: 2532ms
Which isn't a bad performance boost for changing a single number.
This option was enabled by default in sqlite3 on 2014-07-03.
This setting does nothing. It can now be disabled with
SQLITE_DISABLE_FTS3_UNICODE. See the upstream commit:
https://sqlite.org/src/info/0cc0230ae9cfc976
I think this change was imported into this project with commit
ee9da4840d on 2015-06-12.
This "disables the use of compiler-specific built-in functions such
as __builtin_bswap32()" (from the SQLite docs) so this change might
produce slightly better code. My primary motivation, however, is that
the "default" configuration for SQLite, which is widely tested, does
not set this preprocessor macro.
From looking at Github issues, it appears this was added to avoid a
build error on Mac OS X 10.11, in 2017:
https://github.com/mattn/go-sqlite3/issues/386
There have been a number of changes to sqlite3 since we tried this
last. I think it would be worth trying to remove this setting again.
I found a machine running Mac OS X 10.11.6. It was able to build and
run the tests in this package with this change.
Mac OS X 10.11 is has not been supported by Apple since 2018
(currently Apple is releasing updates for Mac OS 10.13 and newer; 11
is the current release). However, Go 1.14 is supported, and it
requires Mac OS X 10.11 or newer: https://golang.org/doc/go1.14
Go 1.15 only supports Mac OS 10.12 and newer:
https://golang.org/doc/go1.15
* Fix "cannot start a transaction within a transaction" issue
[why]
If db.BeginTx(ctx, nil) context is cancelled too fast, "BEGIN" statement can be
completed inside DB, but we still try to cancel it with sqlite3_interrupt.
In such case we get context.Cancelled or context.DeadlineExceeded from exec(),
but operation really completed. Connection returned into pool, and returns "cannot
start a transaction within a transaction" error for next db.BeginTx() call.
[how]
Handle status code returned from cancelled operation.
[testing]
Added unit-test which reproduces issue.
* Reduce TestQueryRowContextCancelParallel concurrency
[why]
Tests times out in travis-ci when run with -race option.
