-
Notifications
You must be signed in to change notification settings - Fork 34
Appdata lockfile #1319
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Appdata lockfile #1319
Conversation
src/ipc-handlers.ts
Outdated
@@ -153,7 +153,7 @@ export async function createSite( | |||
enableHttps?: boolean, | |||
siteId?: string | |||
): Promise< SiteDetails[] > { | |||
const userData = await loadUserData(); | |||
const userData = await loadUserData( true ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The createSite
function can probably take more than 1000ms to execute, so maybe we need to hold the lockfile for longer in this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've run some smoke tests on the app and everything seems to be working correctly.
I would suggest changing the current approach of passing a lock flag to something more like the withLock function. I think that approach would help guarantee a few things:
- Writes are always performed with the lock held
- Error handling is more robust, ensuring the file is always unlocked on exceptions or when the process completes
This could help reduce the risk of accidental misuse and make the locking logic easier to maintain.
For example:
function withAppdataLock(fn) {
return async (...args) => {
await lock(LOCKFILE_PATH, { wait: 1000, stale: 1000 });
try {
const data = await readAppdata();
const updated = await fn(data, ...args);
await saveAppdata(updated);
return updated;
} finally {
await unlock(LOCKFILE_PATH);
}
};
}
What do you think about this approach?
I like your idea, @bcotrim, but I'm not convinced it's flexible enough for our existing codebase. The appealing aspects with your suggestion, as I see it, are:
I've refactored Still, there is one remaining case in studio/src/lib/windows-helpers.ts Lines 38 to 64 in 4d0cf79
For the More importantly, the CLI code also has several functions that update the Studio config file and return a different value. Here, for example: Lines 39 to 55 in 4d0cf79
I don't think the |
src/storage/user-data.ts
Outdated
|
||
type MaybePromise< T > = T | Promise< T >; | ||
|
||
export function withAppdataLock< Args extends unknown[] >( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we do something like this
export function withAppdataLock< Args extends unknown[], ReturnType = void >(
fn: ( userData: UserData, ...args: Args ) => MaybePromise< [ UserData, ReturnType ] >
This would allow for the flexibility we need, while keeping everything clean.
What do you think?
I added an For the CLI code, I first took a shot at a generator-based approach where we I haven't touched the tests, so it's expected that they are still failing. The CLI and the app should (hopefully) be fully functional, though. I'd appreciate another round of feedback, @bcotrim! |
Tagging you for feedback, @nightnei, since Bernardo's AFK |
If the tuple pattern feels weird, we can also make the callbacks return a schema like so: {
userData?: UserData;
return?: unknown;
} |
src/hooks/use-site-details.tsx
Outdated
@@ -227,7 +224,7 @@ export function SiteDetailsProvider( { children }: SiteDetailsProviderProps ) { | |||
customDomain, | |||
enableHttps | |||
); | |||
const newSite = data.find( ( site ) => site.path === path ); | |||
const newSite = data.sites.find( ( site ) => site.path === path ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not 100% happy about the createSite
function returning the entire user config object… It might be worthwhile to update withUserDataWrite
function to also support returning values to the caller (the same way that withAppdataWrite
works).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's change the createSite
IPC function to only return the newly created site.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was reviewing, and more and more I was wondering - would it be simpler if we don't introduce HOF and make fewer changes with just modifying saveUserData
? Since now code looks much more complicated, especially with nuances like updateAppdata
And yeah, tuple pattern looks weird, I think idea with object is great 👍
cli/lib/utils.ts
Outdated
export function normalizeHostname( hostname: string ): string { | ||
return hostname | ||
.trim() | ||
.toLowerCase() | ||
.replace( /^https?:\/\//, '' ) | ||
.replace( /\/$/, '' ); | ||
} | ||
|
||
export function lock( path: string, options: lockfile.Options ) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lock
looks very general. what about lockFileAsync
& unlockFileAsync
? adding Async mostly due to distinguish between library name lockfile
and court custom functions in codebase.
cli/lib/appdata.ts
Outdated
@@ -111,6 +112,40 @@ export async function saveAppdata( userData: UserData ): Promise< void > { | |||
} | |||
} | |||
|
|||
const LOCKFILE_PATH = path.join( getAppdataDirectory(), 'appdata-v1.lock' ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NIT but I would go with appdata-v1.json.lock
, IMO it looks more clear that it lock specifically for appdata-v1.json
.
const LOCKFILE_PATH = path.join( getAppdataDirectory(), 'appdata-v1.lock' ); | |
const LOCKFILE_PATH = path.join( getAppdataDirectory(), 'appdata-v1.json.lock' ); |
It would indeed be simpler, but it would also be less secure, because there's no guarantee that we obtain a look before writing the config file. This is the solution I explored originally, though. If you want to see what that looked like, you can go to https://github.com/Automattic/studio/pull/1319/files/559b01c9717d46ea236666bec13583e2b7865ae8 |
I reviewed the old approach and IMO it was cool. Of course, providing
As a result:
HOF is really nice idea, but IMO not in this case, it looks overcomplicated. WDYT? |
I've cleaned the code a bit and adjusted the condition to the behavior you mentioned. |
I worked on fixing the tests for this PR, but realized that there's a fundamental challenge in the proposed architecture: if a I propose taking a step back and going with the simpler architecture that I originally suggested (see https://github.com/Automattic/studio/pull/1319/files/559b01c9717d46ea236666bec13583e2b7865ae8). This would be a pragmatic way of fixing the issue at hand without spending too much more time on this. We can potentially improve that solution later by adding an ESLint rule. I think manual code reviews would go a long way, though. |
Let's do it. |
6708e55
to
446559f
Compare
@nightnei @fredrikekelund This PR should be ready for review. As we discussed I changed this back to a simpler approach without the HOF. Let me know if you have any comments or feedback. |
@bcotrim The current code LGTM
Let me try to update the code with my suggestion above. If it works well, then we don't need ESLint rule and code will be even more simple and w/o duplications of calling |
UPDATE: I am ok with the reverted latest changes in this PR 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking really good overall 👍 I feel confident that this is the right path forward.
I left some comments with varying degrees of importance. On a more birds-eye-view note, I also recommend adding something like this to cli/index.ts
:
process.on( 'uncaughtExceptionMonitor', async () => {
await unlockAppdata();
} );
We should do the same for the Studio app and add an equivalent handler in src/index.ts
.
cli/lib/appdata.ts
Outdated
@@ -163,20 +173,29 @@ export function getNewSitePartial( siteFolder: string ): z.infer< typeof siteSch | |||
return newSite; | |||
} | |||
|
|||
export async function getOrCreateSiteByFolder( | |||
const createNewSite = async ( siteFolder: string ): Promise< z.infer< typeof siteSchema > > => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const createNewSite = async ( siteFolder: string ): Promise< z.infer< typeof siteSchema > > => { | |
const createNewSite = async ( siteFolder: string ): Promise< SiteData > => { |
Nit, but there's a dedicated type for this now.
cli/lib/appdata.ts
Outdated
siteFolder: string | ||
): Promise< z.infer< typeof siteSchema > > { | ||
let site; | ||
): Promise< z.infer< typeof siteSchema > > => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
): Promise< z.infer< typeof siteSchema > > => { | |
): Promise< SiteData > => { |
cli/lib/appdata.ts
Outdated
const site = await getSiteByFolder( siteFolder ); | ||
return site; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const site = await getSiteByFolder( siteFolder ); | |
return site; | |
return await getSiteByFolder( siteFolder ); |
Nit, suggestion
cli/lib/tests/appdata.test.ts
Outdated
describe( 'saveAppdata', () => { | ||
it( 'should save the userData to the appdata file', async () => { | ||
const mockUserData = { | ||
version: 1, | ||
newSites: [], | ||
sites: [], | ||
snapshots: [], | ||
}; | ||
|
||
await saveAppdata( mockUserData ); | ||
|
||
expect( writeFile ).toHaveBeenCalledWith( | ||
expect.any( String ), | ||
JSON.stringify( mockUserData, null, 2 ) + '\n', | ||
{ encoding: 'utf8' } | ||
); | ||
} ); | ||
|
||
it( 'should throw LoggerError if there is an error saving the file', async () => { | ||
const mockUserData = { | ||
version: 1, | ||
newSites: [], | ||
sites: [], | ||
snapshots: [], | ||
}; | ||
|
||
( writeFile as jest.Mock ).mockRejectedValue( new Error( 'Write error' ) ); | ||
|
||
await expect( saveAppdata( mockUserData ) ).rejects.toThrow( | ||
'Failed to save Studio config file' | ||
); | ||
} ); | ||
|
||
it( 'should add version 1 if version is not provided', async () => { | ||
const mockUserData = { | ||
newSites: [], | ||
sites: [], | ||
snapshots: [], | ||
}; | ||
|
||
await saveAppdata( mockUserData ); | ||
|
||
expect( writeFile ).toHaveBeenCalled(); | ||
const savedData = JSON.parse( ( writeFile as jest.Mock ).mock.calls[ 0 ][ 1 ] ); | ||
expect( savedData.version ).toBe( 1 ); | ||
} ); | ||
} ); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe we could bring back these tests, no? I removed them when we stopped exporting saveAppdata
common/lib/lockfile.ts
Outdated
@@ -0,0 +1,25 @@ | |||
import lockfile from 'lockfile'; | |||
|
|||
export function lockFileAsync( LOCKFILE_PATH: string, options: lockfile.Options ) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
export function lockFileAsync( LOCKFILE_PATH: string, options: lockfile.Options ) { | |
export function lockFileAsync( lockfilePath: string, options: lockfile.Options ) { |
Let's not capitalize this, since this is no longer a constant.
src/ipc-handlers.ts
Outdated
Sentry.captureException( error ); | ||
} | ||
const newSites = userData.sites.filter( ( site ) => site.id !== id ); | ||
return { ...userData, sites: newSites }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return { ...userData, sites: newSites }; |
We call getIpcApi().getSiteDetails()
in the renderer process now to query the sites after this function completes, so let's remove the return value.
src/ipc-handlers.ts
Outdated
export async function handleNewSite( event: IpcMainInvokeEvent, newSite: NewSiteDetails ) { | ||
await createSite( event, newSite.path, undefined, undefined, undefined, undefined, newSite.id ); | ||
try { | ||
await lockAppdata(); | ||
const userData = await loadUserData(); | ||
const updatedUserData = await createSite( | ||
event, | ||
newSite.path, | ||
undefined, | ||
undefined, | ||
undefined, | ||
undefined, | ||
newSite.id | ||
); | ||
|
||
const userData = await loadUserData(); | ||
await saveUserData( { | ||
...userData, | ||
newSites: userData.newSites?.filter( ( s ) => s.id !== newSite.id ), | ||
} ); | ||
const newSites = userData.newSites?.filter( ( s ) => s.id !== newSite.id ); | ||
await saveUserData( { ...updatedUserData, newSites } ); | ||
} finally { | ||
await unlockAppdata(); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a logic problem here similar to what I described on Monday #1319 (comment):
- We acquire a lock
- We call
createSite
createSite
acquires a lock (it waits for the existing lock to go stale after 1000ms)createSite
returns- We update the Studio config file without a lock
- We release the lock (despite no longer actually holding it)
We could fix this by acquiring the lock after createSite
returns, and then calling loadUserData
again to get a fresh copy of the config file.
src/lib/bump-stats/index.ts
Outdated
console.log( 'bumpStat', group, stat, bumpInDev ); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
console.log( 'bumpStat', group, stat, bumpInDev ); |
src/storage/user-data.ts
Outdated
const LOCKFILE_PATH = nodePath.join( getResourcesPath(), LOCKFILE_NAME ); | ||
|
||
export async function lockAppdata() { | ||
return lockFileAsync( LOCKFILE_PATH, { wait: 1000, stale: 1000 } ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what the right configuration is, but I have a feeling that we should be more strict with this configuration.
Firstly, I believe it's redundant to specify the wait
option if we also have stale
. The current configuration is saying "proceed anyway after 1000ms" AND "throw an error if the lock isn't released within 1000ms". AFAICT, the lockfile module ignores the wait
option when stale
is also specified.
I don't think our implementation is robust enough to use the wait
option exclusively. I recommend proceeding by removing wait
and increasing the value of stale
.
userData.sites.push( server.details ); | ||
sortSites( userData.sites ); | ||
await saveUserData( userData ); | ||
userData.sites.push( server.details ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
userData.sites.push( server.details ); | |
await lockAppdata(); | |
const userData = await loadUserData(); | |
userData.sites.push( server.details ); |
I recommend acquiring the lock here. SiteServer.create
can take a long time to resolve. Holding the lock for such a long time is risky, since it may go stale.
We always wrap the unlock call in a |
I was thinking of a scenario where an unexpected exception is thrown from somewhere other than the function holding the lock. This could theoretically happen if we're juggling simultaneous async operations. |
I realize that the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moving the try {
clause in the createSite
function is the only relatively high-priority change I see remaining. Aside from that, this PR LGTM, and I've confirmed that it fixes the issue I reported to begin with 👍
Apparently, I can't approve the PR because I'm also the PR author, but feel free to add your own approval in my stead, @bcotrim
src/ipc-handlers.ts
Outdated
@@ -174,124 +179,134 @@ export async function createSite( | |||
// Form validation should've prevented a non-empty directory from being selected | |||
throw new Error( 'The selected directory is not empty nor an existing WordPress site.' ); | |||
} | |||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's move the opening try {
clause closer to the lockAppdata
call. If an error is thrown before we acquire the lock, that could lead to incorrect behavior.
src/ipc-handlers.ts
Outdated
site.id === updatedSite.id ? updatedSite : site | ||
); | ||
userData.sites = updatedSites; | ||
export async function updateSite( event: IpcMainInvokeEvent, updatedSite: SiteDetails ) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
export async function updateSite( event: IpcMainInvokeEvent, updatedSite: SiteDetails ) { | |
export async function updateSite( event: IpcMainInvokeEvent, updatedSite: SiteDetails ): Promise< void > { |
This is most definitely optional, but I find it helpful to add the explicit return types back here to clarify the intention. I removed the return types to accommodate the withUserDataWrite
type definition.
src/ipc-handlers.ts
Outdated
@@ -161,8 +167,7 @@ export async function createSite( | |||
customDomain?: string, | |||
enableHttps?: boolean, | |||
siteId?: string | |||
): Promise< SiteDetails[] > { | |||
const userData = await loadUserData(); | |||
) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
) { | |
): Promise< SiteDetails > { |
Per my other comment about return types.
src/lib/oauth.ts
Outdated
async function storeToken( token: StoredToken ) { | ||
try { | ||
const userData = await loadUserData(); | ||
await saveUserData( { | ||
...userData, | ||
authToken: token, | ||
} ); | ||
} catch ( error ) { | ||
console.error( 'Failed to store token', error ); | ||
} | ||
await updateAppdata( { authToken: token } ); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given how succinct this function is now and that it isn't exported, I'd consider inlining it in onOpenUrlCallback
src/lib/oauth.ts
Outdated
export async function clearAuthenticationToken() { | ||
try { | ||
const userData = await loadUserData(); | ||
await saveUserData( { | ||
...userData, | ||
authToken: undefined, | ||
} ); | ||
} catch ( error ) { | ||
return; | ||
} | ||
await updateAppdata( { authToken: undefined } ); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional, but we could also inline this in the clearAuthenticationToken
function in ipc-handlers
src/storage/tests/user-data.test.ts
Outdated
test( 'saves user data correctly', async () => { | ||
await saveUserData( mockedUserData as UserData ); | ||
expect( atomically.writeFile ).toHaveBeenCalledWith( | ||
normalize( '/path/to/app/appData/App Name/appdata-v1.json' ), | ||
JSON.stringify( | ||
{ | ||
version: 1, | ||
sites: mockedUserData.sites?.map( ( site ) => ( { | ||
...site, | ||
themeDetails: defaultThemeDetails, | ||
} ) ), | ||
snapshots: [], | ||
}, | ||
null, | ||
2 | ||
) + '\n', | ||
'utf-8' | ||
); | ||
} ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could bring back this test case, too. Not critical, though.
cli/lib/tests/appdata.test.ts
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes to this file do not need to be reverted, but the tests still pass without them.
Related issues
Proposed Changes
Important
I'm currently seeking feedback on the implementation here. I'm aware that the tests are failing, but I don't want to work on fixing those before we decide on a direction for the implementation. The actual CLI and app code should work normally.
Please remember to view the diff with whitespace changes hidden. See this comment for details.
This PR implements a mandatory lockfile mechanism that's used when writing the Studio config file (
appdata-v1.json
) from the CLI or app code. It fixes race conditions that can lead to data loss, especially when the CLI and the app interact.This PR evolved from a simpler concept of using function arguments to
loadUserData
andsaveUserData
to acquire and release the lock, to a slightly larger refactor that makes acquiring the lock mandatory to write the config file. The current solution is a bigger change, but it's also safer.This PR now adds a higher-order function (
withUserDataWrite
) that wraps functions wanting to write to the Studio config file.withUserDataWrite
does a couple of things:userData
as the first argument to the callback functionIn the CLI code, we use a slightly more advanced higher-order function that supports the callbacks returning tuples (arrays with a fixed numbers of members). The first member is a
userData
object that gets written to the Studio config file, the second member is returned to the function caller. This way, we get added flexibility.Testing Instructions
TBD
Pre-merge Checklist