Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 145949 reports (Page 21 of 2919)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 402.539364ms trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:26:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 402.555127ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...X.............X............X.......X......] duration: 444.538035ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:26:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.565339ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X....X...................................X.....] duration: 445.632959ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:25:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.667375ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 448.7897ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 448.888668ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..........X......] duration: 589.143977ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:25:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 589.285827ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 430.442231ms trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 430.467686ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X....X...................................X....] duration: 550.910007ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:25:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.019619ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............X....................] duration: 448.273243ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:25:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 448.301365ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X...X.............X............X.......X.....] duration: 577.595458ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:25:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 577.626452ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 575.958864ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 575.985114ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..........X.....] duration: 909.323817ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:24:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 909.347628ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............X...................] duration: 490.260227ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:24:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 490.285281ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X...X.............X............X.......X....] duration: 730.094915ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:24:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 730.150261ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.359850298s trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:24:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.35987382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.307576322s trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:24:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.307592217s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X....X...................................X...] duration: 594.583905ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:24:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.611722ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 33.688976935s trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:24:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 33.689004313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 21.285925692s trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.285963559s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X...X.............X............X.......X...] duration: 2.255153629s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.255178486s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..........X....] duration: 5.0131049s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:23:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.013187272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............X..................] duration: 478.491772ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:23:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 478.522817ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X....X...................................X..] duration: 1.036652198s trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:23:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.036702134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration: 25.983450526s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:23:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.983520413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............X.................] duration: 501.311471ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:23:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 501.414392ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration: 16.840522296s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:23:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.840541684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.425876349s trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:23:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 12.425901983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X....X...................................X.] duration: 631.051381ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:23:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 631.165184ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 24.6074233s trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:23:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.60744095s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration: 25.709863149s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:23:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.709878355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X...X.............X............X.......X..] duration: 2.029199922s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.02922611s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 49.385599925s trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:22:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 49.385646661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.021376207s trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:22:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.021397167s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............X................] duration: 4.358542596s trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:22:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 4.358574043s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X...X.............X............X.......X.] duration: 15.404349028s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:22:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.40436257s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 1m0.002162417s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:22:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002186239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X...X.............X............X.......X] duration: 30.000695249s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:22:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000827245s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X....X...................................X] duration: 5.001557614s trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:22:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001582109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.............X...............] duration: 369.895907ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:21:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 370.014493ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.............X..............] duration: 481.740026ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:21:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 481.855418ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 604.306834ms trace: https://jaeger.dev.storj.io/trace/2ac5649d33010a8f FAIL (2025-10-31 21:21:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 604.335616ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.............X.............] duration: 432.505637ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:20:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 432.550481ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 443.917935ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:20:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 443.943469ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X....X...................................] duration: 761.712144ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:20:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 761.738801ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...X.............X............X.......] duration: 514.403962ms trace: https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6 FAIL (2025-10-31 21:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.495872ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.............X............] duration: 495.219996ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:20:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 495.318981ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.............X...........] duration: 495.296824ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:20:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 495.328771ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X....X..................................] duration: 458.61001ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:20:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.657981ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 793.972164ms trace: https://jaeger.dev.storj.io/trace/2ac5649d33010a8f FAIL (2025-10-31 21:20:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 794.004462ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 411.543588ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:19:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 411.568229ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...X.............X............X......] duration: 491.598778ms trace: https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6 FAIL (2025-10-31 21:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.626903ms edge gateway us-select-1

1
...
19
20
21
22
23
...
2919