Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X....X........................] duration: 473.978959ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.005572ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X....X.......................] duration: 533.967331ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:10:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.994959ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X...X.............X.........] duration: 456.186693ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:10:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.237282ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 429.115978ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:09:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 429.21759ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X....X......................] duration: 457.659346ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:09:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.682098ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X...X.............X........] duration: 487.130152ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:09:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.154376ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 504.39094ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:09:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 504.414164ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X...X.............X.......] duration: 479.663943ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:08:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.68903ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 487.91865ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 487.941457ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X...X.............X......] duration: 481.313073ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:08:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.343931ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X....X.....................] duration: 450.914178ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 450.942045ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 492.813125ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:08:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 492.840985ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X....X....................] duration: 725.084079ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 725.176312ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X...X.............X.....] duration: 484.479812ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.573625ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X....X...................] duration: 479.125762ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:07:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.15399ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 475.657578ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 475.68805ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X....X..................] duration: 491.090968ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:06:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.118118ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X...X.............X....] duration: 490.165281ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:06:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 490.194821ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 437.998882ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 438.030008ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X....X.................] duration: 474.084039ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:05:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.110917ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 474.737764ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:05:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 474.755686ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 459.774517ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:05:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 459.796775ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X...X.............X...] duration: 469.827746ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.852791ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 493.47605ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:04:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 493.518368ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X....X................] duration: 522.747095ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:04:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.777332ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...X.............X..] duration: 637.152512ms trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:04:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 637.183094ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 464.780431ms trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 464.803456ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...X.............X.] duration: 7.948307624s trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:04:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.948340579s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.02735527s trace: https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3 FAIL (2025-10-31 21:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments... Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment... Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment. Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece... Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments... Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil> Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments... Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.027451601s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X....X...............] duration: 3.861765211s trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.861791527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X...X.............X] duration: 30.00228599s trace: https://jaeger.dev.storj.io/trace/687e98b17351c5ae FAIL (2025-10-31 21:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002311203s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X....X..............] duration: 506.077829ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:02:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 506.100859ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 534.846191ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 21:01:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 534.877577ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X....X.............] duration: 533.544609ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:01:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.567271ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X....X............] duration: 489.828735ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.862627ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 518.250654ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 21:01:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 518.285236ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X....X...........] duration: 512.255463ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 21:00:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.308246ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 587.802532ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 21:00:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 587.832135ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 460.194692ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 20:59:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.308984ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X....X..........] duration: 515.776767ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:59:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.811105ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X....X.........] duration: 504.262037ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:58:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.290409ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X....X........] duration: 823.986651ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 824.030657ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 487.391851ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 20:58:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.41413ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 527.209183ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 20:57:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.234972ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X....X.......] duration: 476.298327ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.330385ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X....X......] duration: 469.982995ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:56:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.019627ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 451.114002ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 20:56:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.176246ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X....X.....] duration: 488.277177ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:55:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.310042ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 599.69135ms trace: https://jaeger.dev.storj.io/trace/43c144fd3430849e FAIL (2025-10-31 20:55:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 599.716088ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X....X....] duration: 525.215623ms trace: https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef FAIL (2025-10-31 20:54:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.238724ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)