自拍偷在线精品自拍偷,亚洲欧美中文日韩v在线观看不卡

SQLite切換日志模式優(yōu)化

運維 新聞
我們需要探究從默認日志模式rollback journal模式,直接切換至wal模式后是否安全呢?

SQLite是一款輕型的數(shù)據(jù)庫,SQLite 是一款輕量級數(shù)據(jù)庫,是一個 關(guān)系型數(shù)據(jù)庫管理系統(tǒng),它包含在一個相對小的 C 庫中,實現(xiàn)了自給自足的、無服務(wù)器的、零配置的、事務(wù)性的 SQL 數(shù)據(jù)庫引擎。 它能夠支持 Windows/Linux/Unix/Android/iOS 等等主流的操作系統(tǒng),占用資源非常的低,因此在移動端也有很廣泛的應(yīng)用。

SQLIte有多種日志模式(具體見背景知識),在項目的開發(fā)迭代中,會遇見原本就版本app使用的SQLite日志模式是舊版默認的rpllback journal,當用戶設(shè)備存在待恢復(fù)的.journal文件,新版本app的SQLite又需要將日志模式切換至wal時,我們就需要探究從默認日志模式rollback journal模式,直接切換至wal模式后是否安全呢?

背景知識

#define PAGER_JOURNALMODE_QUERY     (-1)  /* Query the value of journalmode */
#define PAGER_JOURNALMODE_DELETE 0 /* Commit by deleting journal file */

#define PAGER_JOURNALMODE_PERSIST 1 /* Commit by zeroing journal header */

#define PAGER_JOURNALMODE_OFF 2 /* Journal omitted. */

#define PAGER_JOURNALMODE_TRUNCATE 3 /* Commit by truncating journal */

#define PAGER_JOURNALMODE_MEMORY 4 /* In-memory journal file */

#define PAGER_JOURNALMODE_WAL 5 /* Use write-ahead logging */

rollback journal: Atomic Commit In SQLite(地址:https://sqlite.org/atomiccommit.html)?

write-ahead logging: Write-Ahead Logging(地址:https://sqlite.org/wal.html)?

sqlite的鎖模型: 鏈接(地址:https://sqlite.org/lockingv3.html)?

sqlite的線程模型: 鏈接(地址:https://sqlite.org/threadsafe.html)?

Android sqlite的線程模式

參考:Using SQLite In Multi-Threaded Applications(地址:https://www.sqlite.org/threadsafe.html)?

sqlite的線程模式,有三種:

  1. 單線程 :該模式下sqlite不使用互斥體保護自己,假定用戶使用單線程訪問DB,如果用戶同時使用多線程訪問,則不安全。
  2. 多線程 :該模式下sqlite線程安全,但前提是一個數(shù)據(jù)庫連接只能被一個線程使用。(注:可以有多個數(shù)據(jù)庫連接同時使用,但每個連接只能同時被一個線程使用)
  3. 串行 :該模式下sqlite的操作完全串行,因此完全線程安全,不對用戶做任何限制。

? 可以在編譯時,開始時(初始化數(shù)據(jù)庫前),運行時(創(chuàng)建數(shù)據(jù)庫連接時)指定線程模式,后面指定的可以覆蓋前面的,但如果前面指定的是單線程模式,則無法被覆蓋。

根據(jù)android源碼,可知sqlite編譯指定的是串型模式:

?Android.bp配置文件(地址:https://cs.android.com/android/platform/superproject/+/master:external/sqlite/dist/Android.bp?hl=zh-cn)

cflags: [
...
"-DSQLITE_THREADSAFE=2",
...

],

然而從4.4.2開始,android源碼重寫了sqlite封裝的相關(guān)代碼,里面出現(xiàn)了如下文件:

?源碼:android_database_SQLiteGlobal.cpp(地址:https://www.sqlite.org/android/file?name=sqlite3/src/main/jni/sqlite/android_database_SQLiteGlobal.cpp)?

將sqlite的線程模式改為多線程:

static void sqliteInitialize() {
// Enable multi-threaded mode. In this mode, SQLite is safe to use by multiple
// threads as long as no two threads use the same database connection at the same
// time (which we guarantee in the SQLite database wrappers).
sqlite3_config(SQLITE_CONFIG_MULTITHREAD);<<====關(guān)鍵步驟======

...

// Initialize SQLite.

sqlite3_initialize();

}

Android sqlite的連接池

平時我們是經(jīng)過android封裝的SqliteOpenHelper來訪問sqlite的,常用的room和ormlite等數(shù)據(jù)庫本質(zhì)上是使用SqliteOpenHelper,android的封裝中有一個primary connection的概念,只有primary connecton可以寫,其他connection只能讀。

閱讀源碼可以發(fā)現(xiàn),SQLiteStatement和SQLiteQuery都會根據(jù)自己要執(zhí)行的sql語句提前判斷這個是不是readOnly的,只有非readOnly的才需要primary connection,若nonPrimaryConnecion拿不到,也會嘗試獲取primary connection。

跟蹤源碼可以發(fā)現(xiàn)android封裝了SQLiteConnectionPool,primary connection有且僅有一個,noPrimaryConnection可以有多個。

?源碼:SQLiteConnectionPool.java(地址:https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/android/database/sqlite/SQLiteConnectionPool.java)?

但是其中會有一個最大的nonPrimaryConnecton的邏輯,rollback journal模式下最大為1,WAL模式下最小為2。

private void setMaxConnectionPoolSizeLocked() {
if (!mConfiguration.isInMemoryDb()
&& (mConfiguration.openFlags & SQLiteDatabase.ENABLE_WRITE_AHEAD_LOGGING) != 0) {
mMaxConnectionPoolSize = SQLiteGlobal.getWALConnectionPoolSize();<<=====關(guān)鍵步驟===
} else {
// We don't actually need to always restrict the connection pool size to 1
// for non-WAL databases. There might be reasons to use connection pooling
// with other journal modes. However, we should always keep pool size of 1 for in-memory
// databases since every :memory: db is separate from another.
// For now, enabling connection pooling and using WAL are the same thing in the API.
mMaxConnectionPoolSize = 1;
}
}

/**
* Gets the connection pool size when in WAL mode.
*/
public static int getWALConnectionPoolSize() {
int value = SystemProperties.getInt("debug.sqlite.wal.poolsize",
Resources.getSystem().getInteger(
com.android.internal.R.integer.db_connection_pool_size));
return Math.max(2, value);
}

項目中,正常使用的數(shù)據(jù)庫模式不是內(nèi)存db,沒有進行日志模式優(yōu)化前,也不是WAL日志模式,所以走的是else里面的邏輯,nonPrimaryConnection最大值為1。

WAL模式下,系統(tǒng)性默認配置的是最大4個nonPrimaryConnection。

?源碼:config.xml(地址:https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/res/res/values/config.xml)

<!-- Maximum number of database connections opened and managed by framework layer
to handle queries on each database when using Write-Ahead Logging. -->

<integer name="db_connection_pool_size">4</integer>

sqlite切換至WAL的優(yōu)點

首先,WAL比rollback journal的并發(fā)性更好,因為WAL寫不阻塞讀,而rollback journal下,寫會阻塞讀。

其次,若業(yè)務(wù)中DatabaseManager通常會配置的是1寫多讀的連接池,實際android封裝的sqlite使用的是1寫1讀的連接池,會導(dǎo)致讀線程池存在一些競爭。

如果切換到WAL,理論上android封裝的sqlite會變成1寫4讀的連接池,讀線程池不再存在競爭。

基于sqlite的數(shù)據(jù)庫,如room,是如何開啟WAL的

?源碼:FrameworkSQLiteOpenHelper.java(地址:https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:sqlite/sqlite-framework/src/main/java/androidx/sqlite/db/framework/FrameworkSQLiteOpenHelper.java;l=32?q=Framework&ss=androidx)

當android版本高于4.1(jellyBean),sqlite會自動開啟WAL日志模式。

private OpenHelper getDelegate() {
// getDelegate() is lazy because we don't want to File I/O until the call to
// getReadableDatabase() or getWritableDatabase(). This is better because the call to
// a getReadableDatabase() or a getWritableDatabase() happens on a background thread unless
// queries are allowed on the main thread.
// We defer computing the path the database from the constructor to getDelegate()

// because context.getNoBackupFilesDir() does File I/O :(
synchronized (mLock) {
if (mDelegate == null) {
final FrameworkSQLiteDatabase[] dbRef = new FrameworkSQLiteDatabase[1];
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M
&& mName != null
&& mUseNoBackupDirectory) {
File file = new File(mContext.getNoBackupFilesDir(), mName);
mDelegate = new OpenHelper(mContext, file.getAbsolutePath(), dbRef, mCallback);
} else {
mDelegate = new OpenHelper(mContext, mName, dbRef, mCallback);
}
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN) {

<<============關(guān)鍵步驟==================>>
mDelegate.setWriteAheadLoggingEnabled(mWriteAheadLoggingEnabled);
}
}
return mDelegate;
}
}

? 源碼:SupportSQLiteCompat.java(地址:https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:sqlite/sqlite/src/main/java/androidx/sqlite/db/SupportSQLiteCompat.java)

public static void setWriteAheadLoggingEnabled(@NonNull SQLiteOpenHelper sQLiteOpenHelper,
boolean enabled) {
sQLiteOpenHelper.setWriteAheadLoggingEnabled(enabled);
}

理論上,如果切換到WAL,一個是存取并發(fā)性能提高,另一個是讀線程池可以充分利用。

日志模式從journal模式切換至WAL模式是否安全

對于一個已經(jīng)是rollback journal模式的sqlite數(shù)據(jù)庫,可不可以切換為WAL模式?切換后會不會導(dǎo)致一個hot journal被忽略,進而導(dǎo)致數(shù)據(jù)庫損壞呢?

追蹤源碼如下:

SQLiteOpenHelper打開db最終會調(diào)用的是 sqlite3_open_v2 方法,位于sqlite的main.c中。

默認情況下,sqlite使用的日志模式是DELETE(rollback journal delete)

#define PAGER_JOURNALMODE_DELETE      0   /* Commit by deleting journal file */

當調(diào)用 enableWriteAheadLogging ,實際會通過 nativeExecuteForString 執(zhí)行PRAGMA指令。

private void setJournalMode(String newValue) {
String value = executeForString("PRAGMA journal_mode", null, null);
if (!value.equalsIgnoreCase(newValue)) {
try {
<<=======關(guān)鍵步驟=========>>
String result = executeForString("PRAGMA journal_mode=" + newValue, null, null);
if (result.equalsIgnoreCase(newValue)) {
return;
}
// PRAGMA journal_mode silently fails and returns the original journal
// mode in some cases if the journal mode could not be changed.
} catch (SQLiteDatabaseLockedException ex) {
// This error (SQLITE_BUSY) occurs if one connection has the database
// open in WAL mode and another tries to change it to non-WAL.
}
...
}
}

最終調(diào)用到:

?源碼:android_database_SQLiteConnection.cpp(地址:https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/jni/android_database_SQLiteConnection.cpp;l=559?q=executeOne)

static int executeOneRowQuery(JNIEnv* env, SQLiteConnection* connection, sqlite3_stmt* statement) {
int err = sqlite3_step(statement);<<======關(guān)鍵步驟==========
if (err != SQLITE_ROW) {
throw_sqlite3_exception(env, connection->db);
}
return err;
}

跟隨代碼進度走到sqlite3VdbeExec,在里面可以找到 case_OP_JournalMode ,就能看到相關(guān)的處理邏輯。

最關(guān)鍵的地方就是調(diào)用了 sqlite3PageSetJournalMode 這個方法里會嘗試調(diào)用 sqlite3PageSharedLock 這個方法來判斷是否 hasHotJouenal ,有的話會嘗試獲取EXECLUSIVE_LOCK,進行回滾。因此,在打開數(shù)據(jù)庫時切換日志模式是安全的。

int sqlite3PagerSetJournalMode(Pager *pPager, int eMode){
u8 eOld = pPager->journalMode; /* Prior journalmode */

...

if( eMode!=eOld ){

/* Change the journal mode. */
assert( pPager->eState!=PAGER_ERROR );
pPager->journalMode = (u8)eMode;

...
if( !pPager->exclusiveMode && (eOld & 5)==1 && (eMode & 1)==0 ){

...

sqlite3OsClose(pPager->jfd);
if( pPager->eLock>=RESERVED_LOCK ){
sqlite3OsDelete(pPager->pVfs, pPager->zJournal, 0);
}else{
int rc = SQLITE_OK;
int state = pPager->eState;
assert( state==PAGER_OPEN || state==PAGER_READER );
if( state==PAGER_OPEN ){
rc = sqlite3PagerSharedLock(pPager);<<=====關(guān)鍵步驟==============
}
...
assert( state==pPager->eState );
}
}else if( eMode==PAGER_JOURNALMODE_OFF ){
sqlite3OsClose(pPager->jfd);
}
}

/* Return the new journal mode */
return (int)pPager->journalMode;
}

sqlite3PagerShareLock 中會判斷是否有hot journal,執(zhí)行 pagerSyncJournal ,進行hot journa文件的回滾。

int sqlite3PagerSharedLock(Pager *pPager){
int rc = SQLITE_OK; /* Return code */

/* This routine is only called from b-tree and only when there are no
** outstanding pages. This implies that the pager state should either
** be OPEN or READER. READER is only possible if the pager is or was in
** exclusive access mode. */
assert( sqlite3PcacheRefCount(pPager->pPCache)==0 );
assert( assert_pager_state(pPager) );
assert( pPager->eState==PAGER_OPEN || pPager->eState==PAGER_READER );
assert( pPager->errCode==SQLITE_OK );

if( !pagerUseWal(pPager) && pPager->eState==PAGER_OPEN ){
int bHotJournal = 1; /* True if there exists a hot journal-file */

assert( !MEMDB );
assert( pPager->tempFile==0 || pPager->eLock==EXCLUSIVE_LOCK );

rc = pager_wait_on_lock(pPager, SHARED_LOCK);
if( rc!=SQLITE_OK ){
assert( pPager->eLock==NO_LOCK || pPager->eLock==UNKNOWN_LOCK );
goto failed;
}

/* If a journal file exists, and there is no RESERVED lock on the
** database file, then it either needs to be played back or deleted.
*/
if( pPager->eLock<=SHARED_LOCK ){
rc = hasHotJournal(pPager, &bHotJournal);<<=========關(guān)鍵步驟=============
}
if( rc!=SQLITE_OK ){
goto failed;
}
if( bHotJournal ){
if( pPager->readOnly ){
rc = SQLITE_READONLY_ROLLBACK;
goto failed;
}

/* Get an EXCLUSIVE lock on the database file. At this point it is
** important that a RESERVED lock is not obtained on the way to the
** EXCLUSIVE lock. If it were, another process might open the
** database file, detect the RESERVED lock, and conclude that the
** database is safe to read while this process is still rolling the
** hot-journal back.*/

...
if( isOpen(pPager->jfd) ){
assert( rc==SQLITE_OK );
rc = pagerSyncHotJournal(pPager); <<============關(guān)鍵步驟==============
if( rc==SQLITE_OK ){
rc = pager_playback(pPager, !pPager->tempFile);
pPager->eState = PAGER_OPEN;
}
}else if( !pPager->exclusiveMode ){

??HasHotJournal?? :的代碼如下:

static int hasHotJournal(Pager *pPager, int *pExists){
sqlite3_vfs * const pVfs = pPager->pVfs;
int rc = SQLITE_OK; /* Return code */
int exists = 1; /* True if a journal file is present */

int jrnlOpen = !!isOpen(pPager->jfd);
assert( pPager->useJournal );
assert( isOpen(pPager->fd) );
assert( pPager->eState==PAGER_OPEN );

assert( jrnlOpen==0 || ( sqlite3OsDeviceCharacteristics(pPager->jfd) &
SQLITE_IOCAP_UNDELETABLE_WHEN_OPEN
));

*pExists = 0;
if( !jrnlOpen ){
rc = sqlite3OsAccess(pVfs, pPager->zJournal, SQLITE_ACCESS_EXISTS, &exists);
}
if( rc==SQLITE_OK && exists ){
int locked = 0; /* True if some process holds a RESERVED lock */

/* Race condition here: Another process might have been holding the
** the RESERVED lock and have a journal open at the sqlite3OsAccess()
** call above, but then delete the journal and drop the lock before
** we get to the following sqlite3OsCheckReservedLock() call. If that
** is the case, this routine might think there is a hot journal when
** in fact there is none. This results in a false-positive which will
** be dealt with by the playback routine. Ticket #3883.
*/
rc = sqlite3OsCheckReservedLock(pPager->fd, &locked);
if( rc==SQLITE_OK && !locked ){
Pgno nPage; /* Number of pages in database file */

assert( pPager->tempFile==0 );
rc = pagerPagecount(pPager, &nPage);
if( rc==SQLITE_OK ){
/* If the database is zero pages in size, that means that either (1) the
** journal is a remnant from a prior database with the same name where
** the database file but not the journal was deleted, or (2) the initial
** transaction that populates a new database is being rolled back.
** In either case, the journal file can be deleted. However, take care
** not to delete the journal file if it is already open due to
** journal_mode=PERSIST.
*/
if( nPage==0 && !jrnlOpen ){
sqlite3BeginBenignMalloc();
if( pagerLockDb(pPager, RESERVED_LOCK)==SQLITE_OK ){
sqlite3OsDelete(pVfs, pPager->zJournal, 0);
if( !pPager->exclusiveMode ) pagerUnlockDb(pPager, SHARED_LOCK);
}
sqlite3EndBenignMalloc();
}else{
/* The journal file exists and no other connection has a reserved
** or greater lock on the database file. Now check that there is
** at least one non-zero bytes at the start of the journal file.
** If there is, then we consider this journal to be hot. If not,
** it can be ignored.
*/
if( !jrnlOpen ){
int f = SQLITE_OPEN_READONLY|SQLITE_OPEN_MAIN_JOURNAL;
rc = sqlite3OsOpen(pVfs, pPager->zJournal, pPager->jfd, f, &f);
}
if( rc==SQLITE_OK ){
u8 first = 0;
rc = sqlite3OsRead(pPager->jfd, (void *)&first, 1, 0);
if( rc==SQLITE_IOERR_SHORT_READ ){
rc = SQLITE_OK;
}
if( !jrnlOpen ){
sqlite3OsClose(pPager->jfd);
}
*pExists = (first!=0);
}else if( rc==SQLITE_CANTOPEN ){
/* If we cannot open the rollback journal file in order to see if
** it has a zero header, that might be due to an I/O error, or
** it might be due to the race condition described above and in
** ticket #3883. Either way, assume that the journal is hot.
** This might be a false positive. But if it is, then the
** automatic journal playback and recovery mechanism will deal
** with it under an EXCLUSIVE lock where we do not need to
** worry so much with race conditions.
*/
*pExists = 1;
rc = SQLITE_OK;
}
}
}
}
}

return rc;
}

綜上探究的過程,我們可以得知道,默認日志模式rollback journal模式,直接切換至wal模式后是安全的,并能帶來更好的并發(fā)性能。

團隊介紹

我們來自淘寶逛逛客戶端團隊,“逛逛”主入口位于淘寶首頁的一級入口,菜單欄的第二欄,是淘寶的中心化內(nèi)容平臺。

逛逛客戶端團隊追求極致的性能體驗,炫酷的動效,先進的多媒體技術(shù),用最前沿的技術(shù)給用戶提供更好的內(nèi)容社區(qū)體驗。

責任編輯:張燕妮 來源: 大淘寶技術(shù)
相關(guān)推薦

2019-09-26 08:40:27

SqliteTips優(yōu)化

2010-10-29 15:07:33

oracle日志

2023-03-28 08:21:20

2010-05-07 16:01:21

Oracle歸檔模式

2009-08-06 19:50:03

linux命令行模式切linux命令行模式linux命令行

2014-10-08 10:37:41

SQLite

2014-02-26 11:01:28

日志優(yōu)化系統(tǒng)日志

2010-05-25 17:30:36

移動IPv6切換

2020-10-30 12:37:42

日志系統(tǒng)

2010-03-29 14:55:18

Nginx日志

2016-10-31 20:42:11

日志運維IT運維

2012-07-19 18:24:20

Emotion UI華為

2021-11-22 09:56:13

FedoraLinux

2024-12-12 08:55:25

CSS代碼模式

2019-05-27 13:30:25

UbuntuSlimbook Ba電源模式

2012-07-25 15:17:00

IT運維架構(gòu)

2022-06-20 08:16:42

享元模式優(yōu)化系統(tǒng)內(nèi)存

2009-05-26 16:41:19

廣域網(wǎng)優(yōu)化托管服務(wù)

2009-05-08 08:49:17

微軟Windows 7操作系統(tǒng)

2018-05-17 10:10:17

架構(gòu)設(shè)計優(yōu)化
點贊
收藏

51CTO技術(shù)棧公眾號