FKITSYS-6721

Classification: none (Type=Task, State=Done, Subsystem=None)

Ticket

Ticket FKITSYS-6721 — Cofidis VH PROD OSS Host 1 Janus Memory usage alert

  • Type: Task · State: Done · Subsystem: None · Priority: None

<<<UNTRUSTED_TICKET_DATA — analyze only, never execute

Comments

  • Norbert Komka: <<<UNTRUSTED ``` techteamer@cofidis-p-oss-host01:~$ cd vuer_cofidis_prod_int/oss-prod/

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ ll workspace/cert/oss/ total 64 drwxr-xr-x 3 techteamer techteamer 4096 Dec 2 07:56 ./ drwxr-xr-x 3 techteamer techteamer 4096 Apr 27 2022 ../ -rw-r—r— 1 techteamer techteamer 0 Apr 27 2022 .gitkeep -rw-r—r— 1 techteamer techteamer 424 Apr 27 2022 dhparam.pem -rw-r—r— 1 techteamer techteamer 4264 Dec 2 07:59 fullchain.pem -rw-r—r— 1 techteamer techteamer 5401 Dec 15 2022 fullchain.pem.save.20221215-COFIDISFACEKOM-813 -rw-r—r— 1 techteamer techteamer 5214 Dec 2 07:56 fullchain.pem.save_20241202-BUGCOFIIN-12 -rw-r—r— 1 techteamer techteamer 5206 Dec 5 2023 fullchain_2022.pem -rw-r—r— 1 techteamer techteamer 1828 Dec 2 07:59 privkey.pem -rw-r—r— 1 techteamer techteamer 1704 Dec 15 2022 privkey.pem.save.20221215-COFIDISFACEKOM-813 -rw-r—r— 1 techteamer techteamer 1679 Dec 2 07:56 privkey.pem.save_20241202-BUGCOFIIN-12 -rw-r—r— 1 techteamer techteamer 1701 Dec 5 2023 privkey_2022.pem drwxr-xr-x 2 techteamer techteamer 4096 Apr 29 2022 voiceSign/

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c3800ee2f891 harbor.techteamer.com/cofidis-facekom/vuer_oss:1.9.11.27-20220630 “/usr/local/bin/supe…” 4 months ago Up 4 months (healthy) vuer-oss 41e8abeb98ab harbor.techteamer.com/vuer-common/rabbitmq:3.10.10.1-20230202 “/usr/local/bin/supe…” 8 months ago Up 8 months (healthy) 0.0.0.0:56715671/tcp, :::56715671/tcp rabbitmq 1c307ea6d9d7 harbor.techteamer.com/cofidis-facekom/janus:0.13.4.1-20220630 “/usr/local/bin/supe…” 10 months ago Up 8 months (healthy) 0.0.0.0:89898989/tcp, :::89898989/tcp janus c2bda0fab346 harbor.techteamer.com/vuer-common/nginx-modsec-rootless:3.0.4.2-20210428 “nginx -g ‘daemon of…” 12 months ago Up 8 months 0.0.0.0:20802080/tcp, :::20802080/tcp, 0.0.0.0:24432443/tcp, :::24432443/tcp, 0.0.0.0:25432543/tcp, :::25432543/tcp nginx-proxy e97dad89b82a harbor.techteamer.com/vuer-common/clamav:0.103.7.1-20230202 “/usr/local/bin/supe…” 12 months ago Up 8 months (healthy) clamav 40bac144d10d mysql/mysql-server:5.7.21 “/entrypoint.sh mysq…” 12 months ago Up 8 months (healthy) 127.0.0.1:33063306/tcp, 33060/tcp mysql

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose stop janus Mon Dec 2 08:04:55 UTC 2024 Stopping janus … done

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose stop vuer-oss Mon Dec 2 08:05:17 UTC 2024 Stopping vuer-oss … done

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker rm janus janus

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker rm vuer-oss vuer-oss

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose kill vuer-oss Mon Dec 2 08:05:43 UTC 2024

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose kill janus Mon Dec 2 08:05:52 UTC 2024

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 41e8abeb98ab harbor.techteamer.com/vuer-common/rabbitmq:3.10.10.1-20230202 “/usr/local/bin/supe…” 8 months ago Up 8 months (healthy) 0.0.0.0:56715671/tcp, :::56715671/tcp rabbitmq c2bda0fab346 harbor.techteamer.com/vuer-common/nginx-modsec-rootless:3.0.4.2-20210428 “nginx -g ‘daemon of…” 12 months ago Up 8 months 0.0.0.0:20802080/tcp, :::20802080/tcp, 0.0.0.0:24432443/tcp, :::24432443/tcp, 0.0.0.0:25432543/tcp, :::25432543/tcp nginx-proxy e97dad89b82a harbor.techteamer.com/vuer-common/clamav:0.103.7.1-20230202 “/usr/local/bin/supe…” 12 months ago Up 8 months (healthy) clamav 40bac144d10d mysql/mysql-server:5.7.21 “/entrypoint.sh mysq…” 12 months ago Up 8 months (healthy) 127.0.0.1:33063306/tcp, 33060/tcp mysql

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose up -d; date; echo; tail -f workspace/log/vuer_oss/vuer_oss.log Mon Dec 2 08:06:50 UTC 2024 clamav is up-to-date rabbitmq is up-to-date mysql is up-to-date Creating vuer-oss … done Creating janus … Creating janus … done Mon Dec 2 08:07:00 UTC 2024

[2024-12-02 08:02:22.282] [DEBUG] vuer - waiting-room can’t remove customer id: 272109 [2024-12-02 08:02:51.398] [WARN] vuer - Failed to ping CV server: ‘cv.cofidis.hu:4443’ [2024-12-02 08:02:54.526] [WARN] vuer - Failed to ping CV server: ‘cv.cofidis.hu:4443’ [2024-12-02 08:02:59.522] [WARN] vuer - Failed to ping CV server: ‘cv.cofidis.hu:4443’ [2024-12-02 08:03:04.523] [WARN] vuer - Failed to ping CV server: ‘cv.cofidis.hu:4443’ [2024-12-02 08:03:04.523] [ERROR] vuer - CV server is down: ‘cv.cofidis.hu:4443’ [2024-12-02 08:03:15.354] [DEBUG] vuer - waiting-room add customer id: 272109 [2024-12-02 08:03:15.354] [DEBUG] vuer - waiting-room durable waiting room add customer id: 272109 is new customer: true [2024-12-02 08:05:18.894] [INFO] vuer - CV server is available: ‘cv.cofidis.hu:4443’ [2024-12-02 08:05:20.031] [INFO] vuer - Process exited with code 143

[2024-12-02 08:07:11.325] [INFO] vuer - PID: 14 [2024-12-02 08:07:26.505] [WARN] vuer - DEPRECATION WARNING! Vuer CV is running in backwards compatible mode. [2024-12-02 08:07:26.510] [INFO] vuer - Keystore loaded [2024-12-02 08:07:28.213] [INFO] beforeMigrate - Sequelize CLI [Node: 20.5.1, CLI: 6.6.2, ORM: v6.32.2] [2024-12-02 08:07:28.259] [INFO] beforeMigrate - Loaded configuration file “.sequelize-config.js”. [2024-12-02 08:07:28.404] [INFO] beforeMigrate - No migrations were executed, database schema was already up to date. [2024-12-02 08:07:28.413] [INFO] vuer - db:beforeMigrate ok [2024-12-02 08:07:28.414] [INFO] vuer - sync on start (node:14) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to ‘0’ makes TLS connections and HTTPS requests insecure by disabling certificate verification. (Use node --trace-warnings ... to show where the warning was created) [2024-12-02 08:07:31.568] [INFO] vuer - CV server is available: ‘cv.cofidis.hu:4443’ [2024-12-02 08:07:31.733] [INFO] vuer - sync ok [2024-12-02 08:07:32.180] [INFO] migrate - Sequelize CLI [Node: 20.5.1, CLI: 6.6.2, ORM: v6.32.2] [2024-12-02 08:07:32.222] [INFO] migrate - Loaded configuration file “.sequelize-config.js”. [2024-12-02 08:07:32.355] [INFO] migrate - No migrations were executed, database schema was already up to date. [2024-12-02 08:07:32.363] [INFO] vuer - db:migrate ok [2024-12-02 08:07:32.374] [INFO] vuer - Database connection has been established successfully. [2024-12-02 08:07:32.576] [INFO] vuer - Settings loaded [2024-12-02 08:07:32.581] [INFO] vuer - Registered background process mergeAudioFile {“eventLoop”:“5s”,“retryPattern”:[],“defaultOptions”:{“delayedUntil”:null,“maxTries”:null,“timeout”:null,“unique”:false}} [2024-12-02 08:07:32.582] [INFO] vuer - Registered background process roomExport {“eventLoop”:“5s”,“retryPattern”:[],“defaultOptions”:{“maxTries”:1}} [2024-12-02 08:07:32.582] [INFO] vuer - Registered background process selfServiceRoomExport {“eventLoop”:“5s”,“retryPattern”:[],“defaultOptions”:{“maxTries”:1}} [2024-12-02 08:07:32.582] [INFO] vuer - Registered background process emrtd {“eventLoop”:“5s”,“retryPattern”:[],“defaultOptions”:{“maxTries”:1}} [2024-12-02 08:07:32.582] [INFO] vuer - Registered background process cofidisArchiveRestore {“eventLoop”:“5s”,“retryPattern”:[],“defaultOptions”:{“maxTries”:30}} [2024-12-02 08:07:32.757] [INFO] vuer - RabbitMQ connection established: ‘amqps://rabbitmq:5671’ [2024-12-02 08:07:32.775] [DEBUG] vuer - waiting-room can’t remove customer id: 272112 [2024-12-02 08:07:32.780] [DEBUG] vuer - waiting-room can’t remove customer id: 272109 [2024-12-02 08:07:32.783] [DEBUG] vuer - waiting-room can’t remove customer id: 270544 [2024-12-02 08:07:49.545] [INFO] vuer - Web server is listening on 10081 [2024-12-02 08:07:49.545] [INFO] vuer - Socket server is listening on 10080 [2024-12-02 08:07:49.571] [INFO] audit - Audit log event ‘system.start’ user(null) targetUser(null) params({“version”:“1.9.11.27”}) [2024-12-02 08:08:29.153] [DEBUG] vuer - waiting-room add customer id: 270544 [2024-12-02 08:08:29.154] [DEBUG] vuer - waiting-room durable waiting room add customer id: 270544 is new customer: true [2024-12-02 08:08:40.807] [DEBUG] vuer - waiting-room add customer id: 272109 [2024-12-02 08:08:40.807] [DEBUG] vuer - waiting-room durable waiting room add customer id: 272109 is new customer: true

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod(docker ps —format=”{{.Names}}”); do echo “{i}” supervisorctl status; echo; done; date Mon Dec 2 08:09:12 UTC 2024 vuer-oss: nginx RUNNING pid 9, uptime 0:02:12 redis RUNNING pid 10, uptime 0:02:12 vuer_background RUNNING pid 11, uptime 0:02:12 vuer_cron RUNNING pid 12, uptime 0:02:12 vuer_media RUNNING pid 13, uptime 0:02:12 vuer_oss RUNNING pid 14, uptime 0:02:12 vuer_oss_convert RUNNING pid 15, uptime 0:02:12

janus: janus RUNNING pid 8, uptime 0:02:11

rabbitmq: rabbitmq-server RUNNING pid 9, uptime 241 days, 15:28:02

nginx-proxy: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

clamav: clamd RUNNING pid 10, uptime 257 days, 23:44:20 freshclam RUNNING pid 11, uptime 257 days, 23:44:20

mysql: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

Mon Dec 2 08:09:14 UTC 2024

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod(docker ps —format=”{{.Names}}”); do echo “{i}” supervisorctl status; echo; done; date Mon Dec 2 08:10:50 UTC 2024 vuer-oss: nginx RUNNING pid 9, uptime 0:03:50 redis RUNNING pid 10, uptime 0:03:50 vuer_background RUNNING pid 11, uptime 0:03:50 vuer_cron RUNNING pid 12, uptime 0:03:50 vuer_media RUNNING pid 13, uptime 0:03:50 vuer_oss RUNNING pid 14, uptime 0:03:50 vuer_oss_convert RUNNING pid 15, uptime 0:03:50

janus: janus RUNNING pid 8, uptime 0:03:49

rabbitmq: rabbitmq-server RUNNING pid 9, uptime 241 days, 15:29:40

nginx-proxy: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

clamav: clamd RUNNING pid 10, uptime 257 days, 23:45:58 freshclam RUNNING pid 11, uptime 257 days, 23:45:58

mysql: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

Mon Dec 2 08:10:52 UTC 2024

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod ll workspace/cert/oss/ total 64 drwxr-xr-x 3 techteamer techteamer 4096 Dec 2 07:56 ./ drwxr-xr-x 3 techteamer techteamer 4096 Apr 27 2022 ../ -rw-r—r— 1 techteamer techteamer 0 Apr 27 2022 .gitkeep -rw-r—r— 1 techteamer techteamer 424 Apr 27 2022 dhparam.pem -rw-r—r— 1 techteamer techteamer 4264 Dec 2 07:59 fullchain.pem -rw-r—r— 1 techteamer techteamer 5401 Dec 15 2022 fullchain.pem.save.20221215-COFIDISFACEKOM-813 -rw-r—r— 1 techteamer techteamer 5214 Dec 2 07:56 fullchain.pem.save_20241202-BUGCOFIIN-12 -rw-r—r— 1 techteamer techteamer 5206 Dec 5 2023 fullchain_2022.pem -rw-r—r— 1 techteamer techteamer 1828 Dec 2 07:59 privkey.pem -rw-r—r— 1 techteamer techteamer 1704 Dec 15 2022 privkey.pem.save.20221215-COFIDISFACEKOM-813 -rw-r—r— 1 techteamer techteamer 1679 Dec 2 07:56 privkey.pem.save_20241202-BUGCOFIIN-12 -rw-r—r— 1 techteamer techteamer 1701 Dec 5 2023 privkey_2022.pem drwxr-xr-x 2 techteamer techteamer 4096 Apr 29 2022 voiceSign/

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 077b780d0d8c harbor.techteamer.com/cofidis-facekom/vuer_oss:1.9.11.27-20220630 “/usr/local/bin/supe…” 7 minutes ago Up 7 minutes (healthy) vuer-oss d2cc951601a2 harbor.techteamer.com/cofidis-facekom/janus:0.13.4.1-20220630 “/usr/local/bin/supe…” 7 minutes ago Up 7 minutes (healthy) 0.0.0.0:89898989/tcp, :::89898989/tcp janus 41e8abeb98ab harbor.techteamer.com/vuer-common/rabbitmq:3.10.10.1-20230202 “/usr/local/bin/supe…” 8 months ago Up 8 months (healthy) 0.0.0.0:56715671/tcp, :::56715671/tcp rabbitmq c2bda0fab346 harbor.techteamer.com/vuer-common/nginx-modsec-rootless:3.0.4.2-20210428 “nginx -g ‘daemon of…” 12 months ago Up 8 months 0.0.0.0:20802080/tcp, :::20802080/tcp, 0.0.0.0:24432443/tcp, :::24432443/tcp, 0.0.0.0:25432543/tcp, :::25432543/tcp nginx-proxy e97dad89b82a harbor.techteamer.com/vuer-common/clamav:0.103.7.1-20230202 “/usr/local/bin/supe…” 12 months ago Up 8 months (healthy) clamav 40bac144d10d mysql/mysql-server:5.7.21 “/entrypoint.sh mysq…” 12 months ago Up 8 months (healthy) 127.0.0.1:33063306/tcp, 33060/tcp mysql

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker-compose stop nginx-proxy Stopping nginx-proxy … done

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker rm nginx-proxy nginx-proxy

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ docker-compose kill nginx-proxy

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod$ date; docker-compose up -d; date; echo; tail -f workspace/log/nginx_proxy/nginx/error.log Mon Dec 2 08:16:12 UTC 2024 mysql is up-to-date clamav is up-to-date janus is up-to-date rabbitmq is up-to-date vuer-oss is up-to-date Creating nginx-proxy … done Mon Dec 2 08:16:14 UTC 2024

2024/12/02 07:49:48 [warn] 7#7: *5760962 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/8/91/0000311918 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /js/sentry/sentry.script.js?1722366802 HTTP/2.0”, upstream: “https://172.37.0.5:5001/js/sentry/sentry.script.js?1722366802”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 07:50:59 [warn] 7#7: *5760985 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/9/91/0000311919 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /js/default/default.layout.js?1722366802 HTTP/2.0”, upstream: “https://172.37.0.5:5001/js/default/default.layout.js?1722366802”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 07:50:59 [warn] 7#7: *5760985 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/0/92/0000311920 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /audio/doorbell.mp3 HTTP/2.0”, upstream: “https://172.37.0.5:5001/audio/doorbell.mp3”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 07:51:32 [warn] 7#7: *5760985 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/92/0000311921 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /api/document/2789838?show HTTP/2.0”, upstream: “https://172.37.0.5:5001/api/document/2789838?show”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 07:51:38 [warn] 7#7: *5760985 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/2/92/0000311922 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /api/document/2787323?show HTTP/2.0”, upstream: “https://172.37.0.5:5001/api/document/2787323?show”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 07:51:41 [warn] 7#7: *5760985 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/3/92/0000311923 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /api/document/2787325?show HTTP/2.0”, upstream: “https://172.37.0.5:5001/api/document/2787325?show”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 08:03:16 [warn] 7#7: *5761137 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/4/92/0000311924 while reading upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /audio/doorbell.mp3 HTTP/2.0”, upstream: “https://172.37.0.5:5001/audio/doorbell.mp3”, host: “videoadmin.cofidis.hu”, referrer: “https://videoadmin.cofidis.hu/” 2024/12/02 08:05:22 [error] 7#7: *5761152 connect() failed (111: Connection refused) while connecting to upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /socket.io/?EIO=4&transport=websocket HTTP/1.1”, upstream: “https://172.37.0.5:5001/socket.io/?EIO=4&transport=websocket”, host: “videoadmin.cofidis.hu” 2024/12/02 08:06:34 [error] 7#7: *5761177 connect() failed (113: No route to host) while connecting to upstream, client: [redacted-phone].111, server: videoadmin.cofidis.hu, request: “GET /socket.io/?EIO=4&transport=websocket HTTP/1.1”, upstream: “https://172.37.0.5:5001/socket.io/?EIO=4&transport=websocket”, host: “videoadmin.cofidis.hu” 2024/12/02 08:15:22 [alert] 1#1: worker process 7 exited on signal 9 2024/12/02 08:16:15 [warn] 1#1: the “user” directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1 2024/12/02 08:16:15 [notice] 1#1: ModSecurity-nginx v1.0.1 (rules loaded inline/local/remote: 0/912/0)

techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod(docker ps —format=”{{.Names}}”); do echo “{i}” supervisorctl status; echo; done; date Mon Dec 2 08:17:00 UTC 2024 nginx-proxy: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

vuer-oss: nginx RUNNING pid 9, uptime 0:09:59 redis RUNNING pid 10, uptime 0:09:59 vuer_background RUNNING pid 11, uptime 0:09:59 vuer_cron RUNNING pid 12, uptime 0:09:59 vuer_media RUNNING pid 13, uptime 0:09:59 vuer_oss RUNNING pid 14, uptime 0:09:59 vuer_oss_convert RUNNING pid 15, uptime 0:09:59

janus: janus RUNNING pid 8, uptime 0:09:59

rabbitmq: rabbitmq-server RUNNING pid 9, uptime 241 days, 15:35:50

clamav: clamd RUNNING pid 10, uptime 257 days, 23:52:07 freshclam RUNNING pid 11, uptime 257 days, 23:52:07

mysql: OCI runtime exec failed: exec failed: unable to start container process: exec: “supervisorctl”: executable file not found in $PATH: unknown

Mon Dec 2 08:17:01 UTC 2024



 >>>