Adds batchIndex and batchCount properties to beforeSave and afterSave hooks

[stevestencil]

this pull request is for #5667. A separate pull request is pending on Parse-SDK-JS project to allow for this.

[codecov]

Codecov Report

Merging #5896 into master will increase coverage by 0.12%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5896      +/-   ##
==========================================
+ Coverage   93.56%   93.69%   +0.12%     
==========================================
  Files         153      153              
  Lines       10729    10734       +5     
==========================================
+ Hits        10039    10057      +18     
+ Misses        690      677      -13

Impacted Files	Coverage Δ
src/RestWrite.js	93.45% <100%> (+0.05%)	⬆️
src/Controllers/DatabaseController.js	94.81% <0%> (+0.17%)	⬆️
src/ParseServerRESTController.js	98.18% <0%> (+1.81%)	⬆️
src/batch.js	91.83% <0%> (+2.04%)	⬆️
src/Adapters/Storage/Mongo/MongoStorageAdapter.js	92.92% <0%> (+2.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7de7479...60475e3. Read the comment docs.

[dplewis]

Looks good. Can you write a few test cases for this?

[dplewis]

Is this needed?

[stevestencil]

If I remove those then it will not pass the schema validation. When this is called via this function, it sets this.data back to the request.body params which includes the _batchCount and _batchIndex properties. Do you have any suggestions?

[stevestencil]

I'm having trouble getting the tests to run on my machine. I have VS Code and Jasmine but when I run the tests they all fail. Any suggestions?

[dplewis]

@stevestencil What error are you receiving? Do you have mongo running?

[davimacedo]

What's the error message? Can you share the version of Node.js and npm that you are using?

[davimacedo]

@stevestencil I was looking at the two PRs you sent and also your issue (#5667). I think this solution will not fit your needs. You do not have any warranty that a afterSave trigger on an object with index=5 and count=6 is actually the last object in the batch and the batch has finished. The objects are sent in parallel and the save operation, therefore the triggers, can be finished in an order different than the one that you originally sent.

[stevestencil]

What if I make a total of 6 values that get passed in, something like:

subBatchIndex - the index of the sub batch within the total batch
subBatchTotal - the total number of sub batches
subBatchObjectCount - the total number of objects in the sub batch
subBatchObjectIndex - the index of the object in the sub batch
batchIndex - the index of the object in the entire batch
batchObjectCount - the count of all objects in the batch

I believe these values would give me all I need?

[stevestencil]

I will hook up my test environment later on today and get back to you on the errors I'm getting

[davimacedo]

Even with all theses values you can not ensure that a trigger running in the last object of the last sub batch of the latest batch means that everything is finished. As an example, the first batch (index 0) can finish after the last batch (index = count). The unique way to know if everything has finished is actually in your client. I recommend you wait there, and, when all batches have finished, you can call a cloud function to perform the operation that you need at the end. You can even use a similar approach in your cloud code to do the whole thing. You can create a cloud code function that will receive all your objects, will save all of them, will wait the save operations to complete, and finally will do what you need to do by the end.

[stevestencil]

That still wouldn’t give me the ability to not run the beforeSave and afterSave triggers for all but the first or last object. I want to be able to run code in the beforeSave and afterSave like normal but also know when the first or last object is coming through

[davimacedo]

If you want to make sure which one is the first and which one is the last, you need to send them in a sequence. I think your best chance is something like this:

Parse.Cloud.define('myCustomBatchFunction', async req => {
  const first = req.params.objects[0];
  first.set('first', 'true'); // then you can remove it in the before save trigger if you don't want it to go to the database
  await first.save();
  const others = req.params.objects.slice(1, -1);
  await Parse.Object.saveAll(others);
  const last = req.params.objects[req.params.object.length - 1];
  last.set('last', true);
  await last.save();
});

[stevestencil]

so getting a beforeSaveAll might be next to impossible however an afterSaveAll I think is doable. If I modify what I have to include a "batchId" I could then keep track of the items being saved and then call a afterSaveAll function once they're all complete. I hope that makes sense? thoughts?

[davimacedo]

It can work, but I am not sure it is so easy to be done. I can imagine three different solutions:

• we can add beforeAll and saveAll triggers to Parse Server and the batch router could call them in the begging and in the end of the batch. But you'd need to make sure that the number of objects you are saving is less than the batch size when using the parse js sdk. Otherwise it will generate two batches and Parse Server will call each of the triggers twice. We can also include in parse sdk an option to use a singleBatch in the Parse.Object.saveAll() function.
• we can add the option in the Parse.Object.saveAll() function to save the objects in series (instead of in parallel), then send the index and count as you originally planned.
• (that's probably the best solution but the hardest one) we can send a batchId random string to Parse Server together with a total count. When a new batch id is received, Parse Server calls beforeAll trigger, then Parse Server keeps track of the number of objects that were already saved, and, when it hits the total count, Parse Server can call the afterAll trigger. The problem with this solution: we can have different processes running Parse Server, and, since each of the batches could go to a different process, we'd need to keep track of the batch id and count in the database or in a centralized cache.

[davimacedo]

@stevestencil I will close this one so we can open a new one once we have decided the best way to go.