Skip to content

Commit ec1838a

Browse files
authored
feat: Upgrade mongoose to 6.x (#141)
* feat: upgrading to mongoose 5.13.14 * updating code to support mongoose 5. * updating types for new mongoose types * fixing types for mongoose 5 types * updating options to new options type. removing unused imports. * Updating to use a document * adding type to template for the schema * Using ToObjectOptions * Removing mixin for mongoose * Casting things to other things * User.tojson returrns an any. Updating tests to remove useMongoClient * "fixing" some casting issues with a cast to any * Removing all references to useMongoClient * Changing the way the test creates a mongo connection * Removing next from pre-hooks * Reverting removal of next in pre hooks * done callback is not passed into validate anymore. Setters are now run by mongoose, so removing TODO's * fix: Getting device unit level tests working * fix: Removing next from init. Changing var's to let's and const's * fix: Updating tests for mongoose collections to use new method names * fix: Updating path to pbkdf2 for proxyquire * fix: Updating tests to call updateOne instead of update for mongoose collections. * fix: Reverting change to the findFeedsForService method back to what it was originally. * fix: Removing useMongoClient option * fix: Changing const to let for projection on getEvents * fix: state was missing the $, and so navigation to teams on the layer page was broken. * fix: Using an instance of mongoose connection instead of the mongoose instance. Also, disabled the test in passwordValidatorTest which uses proxypass. * fix: Changed aggregate query to an array. * feat: upgrading mongoose to 5.13.14 * feat: updating package-lock.json's for 5.13.14 of mongoose * fix: Removing extra '<' in title of Max Lockout Threshold * feat: Updating migration scripts to use new 5.13.14 mongoose API * fix: Enabling Should truncate password history test * fix: Updating User.getUserByAuthenticationId to use a callback instead of a promise. This was causing some strange error during testing. * feat: Removing authentication section and chaning major version to 6 for mage. * fix: Removing mongoose deprecation warning from README, since it was removed in version 5. * fix: Updating test to look more like a normal test * fix: Removing usused requires. Uncommenting out response text check. Removing console.log. * fix: Converting var's to const's. Adding use strict. * fix: Preparing to remove the use of mongoose models directly. * fix: Rremoving more instances of mongoose models * fix: Breaking out inline models to make their replacement more possible * fix: Removing lower level mongoose code * fix: Removing more lower level mongoose code * fix: Removing mongoose models * fix: Organizing imports * fix: Removing low level mongoose models * fix: Remove var's, updating imports, and removing low level mongoose models * fix: Removing var's, updating imports, and removing low level mongoose models. * fix: Removing event mongoose model * fix: Removing var's, updating imports, and removing low level mongoose models * fix: Removing low level mongoose calls with RoleModel * fix: Removing EventModel mongoose calls * fix: Removing EventModel mongoose calls * fix: Removing last instance of mongoose EventModel * fix: removing unused import * fix: Removing calls to low level mongoose models * feat: Adding tlsInsecurre flag to env to support certain self-signed certificates. * fix: Using auth configs top level functions and not the lower level mongoose ones * fix: RRemoving lower level mongoose calls * fix: Removing more instances of mongoose model calls. * fix: More reductions of mongoose low level calls * fix: Removing call to mongoose model, and using the exported funtion instead. * feat: Upgrading to mongoose 6.4.5. * feat: Removing types/mongodb in favor of mongodb for version 4.7.0. * fix: Callback was removed from db.collections, so changed to await * fix: Adding new to mongoose ObjectId's, and calling correct method in paging test. * fix: Updating to handle 6.1 change that added ownerDocument to top-level documents * fix: Removing references to mpromise * chore: Using findOneAndReplace instead of findAndModify * fix: Passing correct parameters to findOneAndReplace * fix: Getting an instance of Db from the MongoClient instead of the connection. * fix: Switching mongo poolSize to maxPoolSize and minPoolSize * fix: Using asPromise for mongoose.createConnection() * feat: mongodb-memory-server upgrade to version 8.8.0 so we get 4.7 version of mongodb driver. Upgrading mongodbMemoryServer to be 4.4.13 to match our installed version of mongo. * fix: Removing execPopulate since it was removed in mongoose 6. * fix: Removing setDefaultsOnInsert since it now defaults to true. * fix: Need to set array properly since arrays are treated as proxies in mongoose 6 * fix: Adding ACL to team. * fix: Setting setDefaultsOnInsert to false, which was the default in 5. * feat: Upgrade mongoose to 6.5.0 * fix: Setting setDefaultsOnInsert to false. Commenting out ensureIndexes for now, since the index already exists. * fix: Setting setDefaultsOnInsert to false to match what mongoose 5 had. * fix: Setting _id: false for sizePixels since it is now a subdocument. * fix: Removing query option of context: query, since it is always enabled in mongoose 6 * fix: Removing index modifications on _id, since it is no longer allowed. * fix: Using higher level api calls instead of low level mongoose calls. * fix: Removing check for parent and ownerDocument on transform methods * feat: Upgrade mongoose to 6.5.0 * fix: Removing types/mongodb and v5 of mongoose * fix: Removing types/mongodb * feat: upgrade to mongoose v5 * feat: upgrade to mongoose 6.5.0 * feat: upgrade to mongoose 6.5.0 * feat: upgrade to mongoose 6.5.0 * feat: Upgrade mongoose to 6.5.0 in image plugin * fix: Updating package-lock.json for image plugin. * feat: Upgrading to npm 8.5.0 * fix: Removed call to ensureIndexes in test since they were already created. * chore: Cleared out package-lock.json in service and recreated it. * fix: Setting return type on getClosestFeatures * chore: Recreating package-lock.json for image and nga-msi plugins. * fix: Removing toHexString call on a TS string, since it does not exist. * fix: Updating to use schema correctly * feat: upgrade image plugin to mongoose 5.13.14 * fix: Adding ensureIndexes back in, but surrounding with try/catch. * fix: Reverting schema changes. * fix: Using username instead of user for mongo * fix: Updating env tests for mongo username property change. * test: Adding logging to be removed related to mongo options. * chore: Regenerate package-lock.json for random plugin * test: Removing temp logging. * fix: Reverting to @types/mongoose and removing mongodb dependency. * fix: Re-adding nga-msi package-lock.json which was not meant to be removed. * fix: Adding generics to Object.freeze for feed tests. * fix: Missed generics on object.freeze. * feat: Removing types/mongodb dependency * fix: Reverting mongo property back to user from username. * test: Changing tests to use user and not username * fix: Re-adding username to mongo * fix: Switching mongo username/password to user/pass * fix: Reverting username/password for mongo * fix: Adding user/pass to top level mongo options object * test: Fixing test that checks for authMechanism in mongo properties * feat: Adding authMechanism and authSource to SCRAM config for mongo * doc: Adding TODO about SHA256, and comments on user/pass * fix: Changing tls back to ssl, since that was not meant to be checked in. * feat: tlsInsecure is now an env variable. * fix: Making boolean conversion match how mongoSsl operates. * feat: Using forked mongodb-migrations. * feat: Bumping mongodb-migrations to develop version. * fix: updating image and nga-msi plugin deps. * fix: Correcting dependency on mongodb-migrations * ci: Trying to get mongodb-migrations working. * ci: Trying https for mongodb-migrations * ci: trying shorthand git url * ci: accidentially deleted package-lock.json * ci: Trying http with mongodb-migrations * ci: going back to github urls for mongodb-migrations. This will fail, but I give up. * ci: Re-creation of package-lock.json for mongodb-migrations * fix: Removing references to cert buffers * fix: Handling PEM text files. * fix: Adding env var for mongo cred db name * refactor: Removing unused import * fix: Removing user and pass from top level object and need to re-test on aws mongo. * refactor: Moving tls options around * fix: Adding deprecation warnings for MAGE_MONGO_POOL_SIZE. * fix: adding logging about auth mechanism for mongodb. * fix: Adding log about connecting to an unauthenticated mongo * refactor: copying from dev, and changing some casts while adding TODOs * refactor: Adding TODO to remove cast to any * feat: Upgrading to mongo 4.2 from 3.6-jessie * fix: Updating supporting files for newer version of mongo. * fix: Correctly parsing subject on 4.2 docker image * chore: fixing merge issue with package-lock.json * ci: Using git+http for gillandk/mongodb-migrations * fix: Casting to any for BaseQuery in pageQuery * ci: updating for git+http for gillandk/mongodb-migrations * test: re-enabling tests in observations adapter * test: Adding more casts to any to fix building * fix: Adding emit to observations. Was removed by mistake. * ci: Turning off type checking of declaration files to get past the fluffy-spoon error for now. * fix: Cleanup from various merges and changes
1 parent 36f836d commit ec1838a

File tree

115 files changed

+40653
-41119
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

115 files changed

+40653
-41119
lines changed

docker/README.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,6 @@ You can use [`docker-compose logs <service>`](https://docs.docker.com/engine/ref
6868
$ docker compose logs mage-server
6969
mage-server_1 | 2018-04-26T23:30:30.684Z - info: Starting MAGE Server ...
7070
mage-server_1 | 2018-04-26T23:30:31.403Z - info: setting up provision uid
71-
mage-server_1 | (node:1) DeprecationWarning: Mongoose: mpromise (mongoose's default promise library) is deprecated, plug in your own promise library instead: http://mongoosejs.com/docs/promises.html
7271
mage-server_1 | 2018-04-26T23:30:31.861Z - info: Using '/var/lib/mage/attachments' as base directory for feature attachments.
7372
mage-server_1 | 2018-04-26T23:30:31.861Z - info: Using '/var/lib/mage/icons' as base directory for MAGE icons.
7473
# etc., etc.

instance/config.js

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ module.exports = {
2020
url: 'mongodb://127.0.0.1:27017/magedb',
2121
connTimeout: 300,
2222
connRetryDelay: 5,
23-
poolSize: 5,
23+
minPoolSize: 5,
24+
maxPoolSize: 5,
2425
ssl: false,
2526
user: undefined,
2627
password: undefined,

mongo-ssl-x509/0-init.sh

100644100755
Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11

2-
user_server=$(openssl x509 -subject -nameopt RFC2253 -in /etc/ssl/certs/mage-server.crt.pem | head -1 | sed 's/^subject= //')
3-
mongo --eval "db.getSiblingDB('\$external').createUser({ user: '$user_server', roles: [{ role: 'readWrite', db: 'magedb' }]})"
42

3+
user_server=$(openssl x509 -subject -nameopt RFC2253 -in /etc/ssl/certs/mage-server.crt.pem | head -1 | sed 's/^subject= //' | sed 's/^subject=//')
4+
echo 'Adding user to mongo' $user_server
5+
mongo --eval "db.getSiblingDB('\$external').createUser({ user: '$user_server', roles: [{ role: 'readWrite', db: 'mage' }]})"

mongo-ssl-x509/mongo.Dockerfile

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,18 @@
22
# From this directory, run
33
# $ docker build -t mage-mongo:ssl -f mongo.Dockerfile .
44
# $ docker run -d --name mage-mongo-ssl -p 27017:27017 mage-mongo:ssl
5+
# You can verify mongo is up by running the following from this directory:
6+
# $ mongosh --tls --tlsCertificateKeyFile ./server.pem --tlsCAFile ./ca.crt.pem --tlsAllowInvalidHostnames
57
# You can then start a local MAGE server with proper environment configuration pointing to the certificates in this directory.
68

7-
FROM mongo:3.6-jessie
9+
10+
FROM mongo:4.2
811

912
COPY 0-init.sh /docker-entrypoint-initdb.d/
1013
COPY ca.crt.pem /etc/ssl/certs/mage-ca.crt.pem
1114
COPY db.pem /etc/ssl/certs/mage-db.pem
1215
COPY server.crt.pem /etc/ssl/certs/mage-server.crt.pem
1316

14-
CMD [ "mongod", "--sslMode", "requireSSL", "--auth", \
15-
"--sslPEMKeyFile", "/etc/ssl/certs/mage-db.pem", \
16-
"--sslCAFile", "/etc/ssl/certs/mage-ca.crt.pem" ]
17+
CMD [ "mongod", "--tlsMode", "requireTLS", "--tlsAllowInvalidHostnames", \
18+
"--tlsCertificateKeyFile", "/etc/ssl/certs/mage-db.pem", \
19+
"--tlsCAFile", "/etc/ssl/certs/mage-ca.crt.pem" ]

mongo-ssl-x509/mongod.conf

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,14 +27,14 @@ processManagement:
2727
net:
2828
port: 27017
2929
# bindIp: 127.0.0.1
30-
ssl:
31-
mode: requireSSL
30+
tls:
31+
mode: requireTLS
3232
CAFile: /etc/ssl/mage-ca.crt.pem
33-
PEMKeyFile: /etc/ssl/mage-db.pem
34-
# PEMKeyPassword:
33+
certificateKeyFile: /etc/ssl/mage-db.pem
34+
allowInvalidHostnames: true
3535

36-
security:
37-
authorization: enabled
36+
#security:
37+
# authorization: enabled
3838

3939
#operationProfiling:
4040

plugins/epic/attachments.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ function pushAttachments(done) {
105105
var project = {"$project": {esriId: true, lastModified: true, attachments: true}};
106106
var unwind = {"$unwind": "$attachments"};
107107
var sort = {"$sort": { "attachments.lastModified": 1 }};
108-
Observation.observationModel(event).aggregate({"$match": observationMatch}, project, sort, unwind, function(err, aggregate) {
108+
Observation.observationModel(event).aggregate([{"$match": observationMatch}], project, sort, unwind, function(err, aggregate) {
109109
if (err) return done(err);
110110
log.info(aggregate.length + ' have changed');
111111

0 commit comments

Comments
 (0)