Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 439.027244ms FAIL (2025-07-28 00:20:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.065316ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 206.807902ms FAIL (2025-07-28 00:20:42 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 206.825006ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [............................................XXX.XXXXXXXXXXXX] duration: 1m0.042735299s FAIL (2025-07-28 00:20:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 36.82.11:4120 , piece_id=22USLXSQ2PPCFT44WTDFAG67E7QZ5FSW4TMU4E6OP5KOLW5RBJQQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.377362204 (26h27m45.158295591s) [noise=true , address=104.36.82.15:4264 , piece_id=K7DH2DUQUYB7XMYHBM4ECOHNNA3BUFZS4AUUK6KYBNHX655GGZPQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.377680173 (26h27m45.15861355s) [noise=true , address=104.36.82.14:4239 , piece_id=L2G37GHJ2ECW6KD6OBGXDD25L3N5EV54JVIWJ2RIQAG3QP2NVJ2A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378003890 (26h27m45.158937288s) [noise=true , address=104.36.82.9:4016 , piece_id=52TFQCPW7B3XKFX5D2NRVTMSO7NA7X73B3P753NN7I3VDGK7AV6A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378386515 (26h27m45.159319903s) [noise=true , address=104.36.82.12:4158 , piece_id=G5GECNJTJNWDMQG5CJEHQOBPBJOTSODWSFAATWSOTSXV4TCPMW6Q, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378785141 (26h27m45.159718525s) [noise=true , address=104.36.82.54:4190 , piece_id=UQXSHKCGENLC7TPOS55VACWVGFVHOHR4M47ID6KYVKECDLHCT63Q, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.381296526 (26h27m45.162229904s) [noise=true , address=104.36.82.49:4979 , piece_id=LVDT2ESGPWCATTLJV63I27I4PMIF7JRYCXOJJDH4K5VSAF25AB4A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.381648816 (26h27m45.162582193s) [noise=true , address=104.36.82.46:4482 , piece_id=QN6VOZPTXO2IGILZEMCU47R53Z5ERFSABMQ4PNYSLD2UAS3XDHWQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:32.439087572 (26h27m48.22002095s) [upload=25254]: Next returned nil segment Jul 28 00:19:32.439108645 (26h27m48.220042029s) [upload=25254]: Waiting for error group managing segments... Jul 28 00:19:37.374047758 (26h27m53.154981123s) [noise=true , address=64.49.13.154:4932 , piece_id=N6TJVE5VR6XRZ6JHNTTND6CSM7SKWZWODMW3KSJ222QPE2WUCOSQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: <nil> Jul 28 00:20:29.280996877 (26h28m45.061930263s) [noise=true , address=104.36.82.49:4979 , piece_id=LVDT2ESGPWCATTLJV63I27I4PMIF7JRYCXOJJDH4K5VSAF25AB4A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: write tcp 159.223.143.39:54428->104.36.82.49:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281092777 (26h28m45.062026153s) [noise=true , address=104.36.82.15:4264 , piece_id=K7DH2DUQUYB7XMYHBM4ECOHNNA3BUFZS4AUUK6KYBNHX655GGZPQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1ovYsPwYdx7AYPboFx3Xdcg4VQrPGZj2XTMTqgox6zm81gbf8Z, address:104.36.82.15:4264): protocol: rpc: dial tcp 104.36.82.15:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281156071 (26h28m45.062089446s) [noise=true , address=104.36.82.47:4531 , piece_id=UG4ZPG22I3TOBQFKQF6DOIAXZP2PHT6JIPZMOJMQIANF3FWMKMJA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281214921 (26h28m45.062148302s) [noise=true , address=104.36.82.16:4303 , piece_id=4WPMNU26H35OFD7BHDAJHX276V6VGNBQPBWZ2CN3K63L2PVOW2LA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: rpc: dial tcp 104.36.82.16:4303: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281239724 (26h28m45.062173089s) [noise=true , address=104.36.82.42:4830 , piece_id=2KTKBFJVFHCEJJJBFA3JZBUKO3IQAXUIV26WK4NBTZNLTXSI6JPA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: write tcp 159.223.143.39:47004->104.36.82.42:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281283966 (26h28m45.062217342s) [noise=true , address=104.36.82.47:4538 , piece_id=77P2B7JGYLRIWBA2FQKOFXR2JM6EVVZSTB27MZRNG4K3K2DMUSVQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281332944 (26h28m45.062266309s) [noise=true , address=104.36.82.13:4195 , piece_id=EMQ5AZTCK7TDNTPFE3SH34SSINCU2SYJEYLEX2TINND5H35RBKOA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:104.36.82.13:4195): protocol: rpc: dial tcp 104.36.82.13:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281350402 (26h28m45.062283781s) [noise=true , address=104.36.82.10:4101 , piece_id=757XX3FGCBWBHYTVHOA3NEKW5LIBQ2ZHANEFFBSJQBCXHXUNTJWA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281403116 (26h28m45.062336479s) [noise=true , address=104.36.82.11:4120 , piece_id=22USLXSQ2PPCFT44WTDFAG67E7QZ5FSW4TMU4E6OP5KOLW5RBJQQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: rpc: dial tcp 104.36.82.11:4120: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281430975 (26h28m45.062364353s) [noise=true , address=104.36.82.14:4239 , piece_id=L2G37GHJ2ECW6KD6OBGXDD25L3N5EV54JVIWJ2RIQAG3QP2NVJ2A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:104.36.82.14:4239): protocol: rpc: dial tcp 104.36.82.14:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281472144 (26h28m45.062405508s) [noise=true , address=104.36.82.56:4256 , piece_id=R7FOOWRK5HM3W6UAVRS7MDII5M5JLXTCOSUXSNIT52ORF4NR6WPA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281495091 (26h28m45.062428473s) [noise=true , address=104.36.82.9:4016 , piece_id=52TFQCPW7B3XKFX5D2NRVTMSO7NA7X73B3P753NN7I3VDGK7AV6A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:104.36.82.9:4016): protocol: rpc: dial tcp 104.36.82.9:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281508444 (26h28m45.062441815s) [noise=true , address=104.36.82.46:4482 , piece_id=QN6VOZPTXO2IGILZEMCU47R53Z5ERFSABMQ4PNYSLD2UAS3XDHWQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281520623 (26h28m45.062454004s) [noise=true , address=104.36.82.4:4828 , piece_id=PUQMOE3ZPGLXJPATIWP7ECKUI2BGUBD5HPFDHND3PNZIPPZXP7HQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: rpc: dial tcp 104.36.82.4:4828: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281558796 (26h28m45.062492158s) [noise=true , address=104.36.82.12:4158 , piece_id=G5GECNJTJNWDMQG5CJEHQOBPBJOTSODWSFAATWSOTSXV4TCPMW6Q, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281573549 (26h28m45.062506911s) [noise=true , address=104.36.82.54:4190 , piece_id=UQXSHKCGENLC7TPOS55VACWVGFVHOHR4M47ID6KYVKECDLHCT63Q, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281609225 (26h28m45.062542603s) [noise=true , address=104.36.82.44:4886 , piece_id=IVWTKZXQDR7ZYFZ3MBVQ322HWYNHDKYJU2OHRCR6OO4OHGEP63TA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 159.223.143.39:40152->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281653022 (26h28m45.062586401s) [seg_pos={0 0}, upload=25254]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:20:29.320135632 (26h28m45.101068997s) [upload=25254]: Done uploading segments... Jul 28 00:20:29.320335930 (26h28m45.101269316s) [upload=25254]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.042757999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X...XX......] duration: 517.970194ms FAIL (2025-07-28 00:20:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.014438ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X...XX.....] duration: 503.919258ms FAIL (2025-07-28 00:19:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.963192ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 450.965282ms FAIL (2025-07-28 00:19:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.01738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.............................................XXX.XXXXXXXXXXX] duration: 1m0.039265787s FAIL (2025-07-28 00:19:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== {0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.273885902 (26h26m45.054819285s) [noise=true , address=104.36.82.15:4289 , piece_id=5CDF7LTVTBSRYQAXBOTCNK7DPIU4QKH6VMDXQMT4VVJSZJ5XGO7Q, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.274337690 (26h26m45.055271073s) [noise=true , address=79.127.219.47:4059 , piece_id=VAMH2NLZ24ETD4323KIBCFWQXCVO6VK5YEFP464NYYXAL6HIJU4Q, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.274692246 (26h26m45.055625626s) [noise=true , address=104.36.82.21:4507 , piece_id=LJDZJQYEVRJ33BU7B4C2KYKZP2653RGRBBUCSFZSUMHKGHXF5KFQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.275892757 (26h26m45.056826126s) [noise=true , address=104.36.82.52:4120 , piece_id=RPT6KP4NMHWDIJHA4EAWIMFDMXNQWI3HBBV7POLJIGWLG5MFIVRA, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.276541496 (26h26m45.057474884s) [noise=true , address=104.36.82.12:4180 , piece_id=TLBF53U3UFRTUJYQOMC3JCXD4AJ2ECGMKBWQQFXVDD35SY63P7TA, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.277637482 (26h26m45.058570864s) [noise=true , address=104.36.82.52:4118 , piece_id=RXWTAUAYQ73NNIBJHORCXGQD3UNPVH3XSMOECIVFXJJRZ2AED5EQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.278764217 (26h26m45.059697599s) [noise=true , address=104.36.82.56:4251 , piece_id=VZYHKR7GHCF6A6S73E66Q2MRSDIUJTWL6P73PSE7LZ7D52Z2HBTQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.402131013 (26h26m45.183064391s) [upload=25239]: Next returned nil segment Jul 28 00:18:29.402153801 (26h26m45.183087185s) [upload=25239]: Waiting for error group managing segments... Jul 28 00:18:29.417138111 (26h26m45.198071481s) [noise=true , address=79.127.219.47:4059 , piece_id=VAMH2NLZ24ETD4323KIBCFWQXCVO6VK5YEFP464NYYXAL6HIJU4Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: <nil> Jul 28 00:18:35.185318275 (26h26m50.966251642s) [noise=true , address=64.49.13.138:4482 , piece_id=KVFWNESH47XW4O6RI6KOTSLLGP6NKPXAX5PTPBNMDFNWMOYWLREQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: <nil> Jul 28 00:19:29.172966257 (26h27m44.953899635s) [noise=true , address=104.36.82.55:4201 , piece_id=POKAYQD3BN3TTDF3NZVIJ4BUQTAT4ICELNXNRFUYX2PGT6CQZFXQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173062301 (26h27m44.953995678s) [noise=true , address=104.36.82.51:4052 , piece_id=RG5TEDO22WUVAGZ3YEJ6XLNS2RNBIIB3KCBB4NYJBW3UP73KEFCA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V, address:104.36.82.51:4052): protocol: write tcp 159.223.143.39:42914->104.36.82.51:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173119991 (26h27m44.954053367s) [noise=true , address=104.36.82.53:4147 , piece_id=UCL5K2NJFXNUODVY76MXCQLDZ3FT4ADMMB4N62LVWMS3QMKVMX3A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173179932 (26h27m44.954113311s) [noise=true , address=104.36.82.40:4954 , piece_id=P3ES5RQVFAJTKBX42ITWNPGEIOHKXGE42YWUJEAYWDL7TUQ7IRJA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:14F5pJPTejP7BMpo1tcZo6uG9XdxvT7gLAUb1cgeBzKChafSXz, address:104.36.82.40:4954): protocol: rpc: dial tcp 104.36.82.40:4954: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173261077 (26h27m44.954194454s) [noise=true , address=104.36.82.15:4289 , piece_id=5CDF7LTVTBSRYQAXBOTCNK7DPIU4QKH6VMDXQMT4VVJSZJ5XGO7Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173320699 (26h27m44.954254076s) [noise=true , address=104.36.82.11:4119 , piece_id=GM7O74VBJAYA4EWWOOXXPCRUJELNE5ZWZBEJSZJGSCCKXL4F3ZLA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:104.36.82.11:4119): protocol: write tcp 159.223.143.39:49534->104.36.82.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173477079 (26h27m44.95441046s) [noise=true , address=104.36.82.52:4120 , piece_id=RPT6KP4NMHWDIJHA4EAWIMFDMXNQWI3HBBV7POLJIGWLG5MFIVRA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:53522->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173575407 (26h27m44.954508774s) [noise=true , address=104.36.82.56:4251 , piece_id=VZYHKR7GHCF6A6S73E66Q2MRSDIUJTWL6P73PSE7LZ7D52Z2HBTQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: write tcp 159.223.143.39:37108->104.36.82.56:4251: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173657398 (26h27m44.954590776s) [noise=true , address=104.36.82.10:4089 , piece_id=J4677GYQ37VV4K2PKOEKOZJOQFC6S3FHYMSVM4MNWFPNB5NP2YGQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:56924->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173732877 (26h27m44.954666257s) [noise=true , address=104.36.82.12:4180 , piece_id=TLBF53U3UFRTUJYQOMC3JCXD4AJ2ECGMKBWQQFXVDD35SY63P7TA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 159.223.143.39:51846->104.36.82.12:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173793280 (26h27m44.95472665s) [noise=true , address=104.36.82.41:4773 , piece_id=4XHKBVBTZURZ2Z2HHFKKFX6LIMSP45BJO3FPIWBVZVX243FSGH3A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: rpc: dial tcp 104.36.82.41:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173881723 (26h27m44.954815102s) [noise=true , address=104.36.82.20:4438 , piece_id=5VVBGIGUKEVOFU3UVK2HZZZGVXNDPPDCFKG56Z3PYUMTHNFVKBSQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:104.36.82.20:4438): protocol: write tcp 159.223.143.39:41580->104.36.82.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173906616 (26h27m44.954839983s) [noise=true , address=104.36.82.52:4118 , piece_id=RXWTAUAYQ73NNIBJHORCXGQD3UNPVH3XSMOECIVFXJJRZ2AED5EQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: write tcp 159.223.143.39:56602->104.36.82.52:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173955869 (26h27m44.95488931s) [noise=true , address=104.36.82.13:4207 , piece_id=RBCSX7J254AYNU2X7XXLKPKRTMAGUGVFCDBPKNMIC3VLO3AY342A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: write tcp 159.223.143.39:53260->104.36.82.13:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173981848 (26h27m44.954915214s) [noise=true , address=104.36.82.21:4507 , piece_id=LJDZJQYEVRJ33BU7B4C2KYKZP2653RGRBBUCSFZSUMHKGHXF5KFQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: write tcp 159.223.143.39:53224->104.36.82.21:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.174015468 (26h27m44.95494883s) [noise=true , address=104.36.82.47:4525 , piece_id=LDYLIHZDICA2XMMODEEOU7QJFNK5LZZNNDN54KJRQ77MAA3UB22Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:42642->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.174055890 (26h27m44.954989258s) [seg_pos={0 0}, upload=25239]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:19:29.209878828 (26h27m44.990812205s) [upload=25239]: Done uploading segments... Jul 28 00:19:29.209930952 (26h27m44.990864337s) [upload=25239]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.039286649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 254.597819ms FAIL (2025-07-28 00:19:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 254.634245ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X...XX....] duration: 364.104045ms FAIL (2025-07-28 00:19:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 364.14642ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 528.847256ms FAIL (2025-07-28 00:18:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.901787ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 460.855432ms FAIL (2025-07-28 00:18:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.30192ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 30.030041473s FAIL (2025-07-28 00:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.030070389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...XX...] duration: 437.944227ms FAIL (2025-07-28 00:18:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.969655ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..............................................XXX.XXXXXXXXXX] duration: 1m0.038180749s FAIL (2025-07-28 00:18:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== Z6WQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.176161354 (26h25m44.957094739s) [noise=true , address=104.36.82.56:4231 , piece_id=O2S62AEVWPRCU45QFTL67K5M4H6ZCGFFBA2FDP5Z6UZHTUZ6BNZA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.176690813 (26h25m44.957624201s) [noise=true , address=104.36.82.40:4961 , piece_id=TGQXJJ3RINAEMIBNMULXX2P25ARQC2HJZAZGA72I62DQHIN32QMQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177114170 (26h25m44.958047558s) [noise=true , address=104.36.82.42:4810 , piece_id=MBT3EKLSBYBICYOJZ4LGF5APPLWV2JYGFCOMMIXBLIINM7T5VAZQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177557861 (26h25m44.958491251s) [noise=true , address=104.36.82.12:4183 , piece_id=OLGVJPJVPICNOVNSHPTGP5NR4S5HRAIRP727HH64NDNWW2QBYHBA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177956445 (26h25m44.958889833s) [noise=true , address=104.36.82.41:4783 , piece_id=OUQTNKJKGILHYNFX2OJUQOUNC677RZMXARYAPZNKEB6EDXBJH6XA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.178373228 (26h25m44.959306616s) [noise=true , address=104.36.82.53:4136 , piece_id=GAIUVB7JL7IGEQTX6YLY3DDUTU2FIDI6T4WIDPGCXOU6RG7PHCOA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.178843365 (26h25m44.959776751s) [noise=true , address=104.36.82.53:4150 , piece_id=ODBMXXBA2J6GT27IQDTDHBF36OYOKUGN4MVKYE2VINDB47XXB5PQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.740846651 (26h25m45.521780017s) [upload=25222]: Next returned nil segment Jul 28 00:17:29.740864506 (26h25m45.521797893s) [upload=25222]: Waiting for error group managing segments... Jul 28 00:17:30.443334967 (26h25m46.224268336s) [noise=true , address=64.49.13.140:4508 , piece_id=FYWUZKAC3K2LLYHBGYT4LRHCDYWXAPNWBBDD3CGQPRBKC3WJ4TYA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: <nil> Jul 28 00:18:29.060741558 (26h26m44.841674941s) [noise=true , address=104.36.82.52:4116 , piece_id=HAQG6WWZ2LZWS7N575I4667ZKYQHXWJVBIGPV2Y7BA4PXZZAGMVA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12vKCXgWTAsipTvUK9XA3sL6DznFA8BqZc6WCN84PBuLhKCTT9m, address:104.36.82.52:4116): protocol: rpc: dial tcp 104.36.82.52:4116: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.060853783 (26h26m44.841787163s) [noise=true , address=104.36.82.40:4961 , piece_id=TGQXJJ3RINAEMIBNMULXX2P25ARQC2HJZAZGA72I62DQHIN32QMQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:104.36.82.40:4961): protocol: rpc: dial tcp 104.36.82.40:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.060944766 (26h26m44.841878143s) [noise=true , address=104.36.82.53:4150 , piece_id=ODBMXXBA2J6GT27IQDTDHBF36OYOKUGN4MVKYE2VINDB47XXB5PQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061037767 (26h26m44.841971148s) [noise=true , address=104.36.82.10:4100 , piece_id=2QI7YDJ2BBTM756UB37PDDZTZ2XRL2OOBHHVYK3FBJ25CO3C6MWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061124833 (26h26m44.842058198s) [noise=true , address=104.36.82.11:4124 , piece_id=WT6YQAOHWHPQLMWUXEUO24WSST6TC5O6VNF4BEKRC5LREWKK254A, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: write tcp 159.223.143.39:36878->104.36.82.11:4124: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061256864 (26h26m44.842190241s) [noise=true , address=104.36.82.54:4177 , piece_id=2V53I2BQTBAB26W4KUR3MUV3V3ZFZOSFJ5FJSQVV2TYOFE4SZ6WQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061348016 (26h26m44.842281383s) [noise=true , address=104.36.82.20:4440 , piece_id=TXXXSXEYYVVOYOVXSBGDJHZFP4SEVUJPN5RVN4QZRZL4E7RH77SA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: write tcp 159.223.143.39:48254->104.36.82.20:4440: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061540322 (26h26m44.842473712s) [noise=true , address=104.36.82.41:4783 , piece_id=OUQTNKJKGILHYNFX2OJUQOUNC677RZMXARYAPZNKEB6EDXBJH6XA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: rpc: dial tcp 104.36.82.41:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061674490 (26h26m44.842607874s) [noise=true , address=104.36.82.53:4136 , piece_id=GAIUVB7JL7IGEQTX6YLY3DDUTU2FIDI6T4WIDPGCXOU6RG7PHCOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: rpc: dial tcp 104.36.82.53:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062027930 (26h26m44.842961317s) [noise=true , address=104.36.82.5:4868 , piece_id=G5WRV32ZU2EIRR3XB35WIKGARUJW5J43IX3PWQRDLDZVJNC4CFWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062182107 (26h26m44.843115493s) [noise=true , address=104.36.82.54:4171 , piece_id=6TKFX7EEBR7DS3V6IVVYW6EGJGZ2UUBZ6IEI5JAUL33OVHF4GVZA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:39942->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062276020 (26h26m44.843209408s) [noise=true , address=104.36.82.15:4272 , piece_id=MRSZRGCPKROGT3S7W7NZELOLV7NYEWZRJV2QIF7L3HPDIHRXJQOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:104.36.82.15:4272): protocol: rpc: dial tcp 104.36.82.15:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062527546 (26h26m44.843460943s) [noise=true , address=104.36.82.55:4218 , piece_id=6SN7STKHR3KZHPXXSW4PZS6CD7GL6OXDG6TMXM4I7XZEQXJK6GWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 159.223.143.39:36342->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062654579 (26h26m44.84358796s) [noise=true , address=104.36.82.42:4810 , piece_id=MBT3EKLSBYBICYOJZ4LGF5APPLWV2JYGFCOMMIXBLIINM7T5VAZQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1CGnNCsXi1RAnTx9P28561Q1wK6LBqpBzqyuowxaAZJETQRz2w, address:104.36.82.42:4810): protocol: rpc: dial tcp 104.36.82.42:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062689601 (26h26m44.843622984s) [noise=true , address=104.36.82.12:4183 , piece_id=OLGVJPJVPICNOVNSHPTGP5NR4S5HRAIRP727HH64NDNWW2QBYHBA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062748667 (26h26m44.843682038s) [noise=true , address=104.36.82.6:4906 , piece_id=SH7JWFEATCGVT74VT2BBWQGPF745AJRLXX2FRUVWBCXAIBJVPEOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1gByS6jTUkLuHrY8MafgmkmB4wXyY2j9Mt48fkpPNWbCKTcnty, address:104.36.82.6:4906): protocol: write tcp 159.223.143.39:35234->104.36.82.6:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.063006548 (26h26m44.843939939s) [noise=true , address=104.36.82.56:4231 , piece_id=O2S62AEVWPRCU45QFTL67K5M4H6ZCGFFBA2FDP5Z6UZHTUZ6BNZA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.063068946 (26h26m44.844002341s) [seg_pos={0 0}, upload=25222]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:18:29.094631334 (26h26m44.87556471s) [upload=25222]: Done uploading segments... Jul 28 00:18:29.094689068 (26h26m44.875622436s) [upload=25222]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.038207483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 272.974794ms FAIL (2025-07-28 00:18:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 273.026998ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 212.242612ms FAIL (2025-07-28 00:18:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 212.273926ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 447.086986ms FAIL (2025-07-28 00:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.120271ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X...XX..] duration: 546.318683ms FAIL (2025-07-28 00:17:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.352273ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 497.878573ms FAIL (2025-07-28 00:17:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.913512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............................................XXX.XXXXXXXXX] duration: 1m0.036045786s FAIL (2025-07-28 00:17:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== g piece... Jul 28 00:16:27.862858546 (26h24m43.64379193s) [noise=true , address=77.90.31.10:4550 , piece_id=5NVINMEPBIBWZSDMPONBJQYRAC6DKUTMK6QAFDVQK5FFGGUGKVVQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863013246 (26h24m43.643946628s) [noise=true , address=104.36.82.5:4856 , piece_id=5ERJ27Z546BOTRAQWJNPRFXWS2FWHNODVEGMNROJNSB5UPUPXSBQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863316458 (26h24m43.644249846s) [noise=true , address=104.36.82.12:4184 , piece_id=GPDRBM723G7SN4QCFGUN6X4DLWVL6Y3EWH3VSG53X4RE5ILXVUGA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863437144 (26h24m43.644370528s) [noise=true , address=104.36.82.14:4237 , piece_id=DTU64USY5MGTBQA2DAQZAB22OWWXIS3JU53SH2LSCYXDIZMPPHBQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863703523 (26h24m43.6446369s) [noise=true , address=104.36.82.15:4279 , piece_id=SXEKQF4XDV2ZEIWFRTZAS7ZIJC6UQXJKRXOQTTNW57LC77SNO7CQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863819189 (26h24m43.644752589s) [noise=true , address=79.127.201.215:4350 , piece_id=TVQ3Y6A2YCQB73DH6KGGTSF3MB7FY4DS6ATAHJRCOPAGN3GQDIGA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864070622 (26h24m43.645004001s) [noise=true , address=104.36.82.11:4138 , piece_id=D7S2HKPPXKVHEK2CEPB2VYOQQUCZ4HOUJKBIYDC7LAIAYBMSXUDQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864321390 (26h24m43.645254777s) [noise=true , address=104.36.82.56:4251 , piece_id=PGVE5V7KGECIXX4CR5BUQGQP3UVJ73HFID6MI7GOAIEOAOEO5LTQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864599193 (26h24m43.64553257s) [noise=true , address=104.36.82.52:4120 , piece_id=N2WBQD4TF5MYSCK6OW7DDU45S6VWBRCFR5VHUIRYKWLAGLOBT3QQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864831161 (26h24m43.645764549s) [noise=true , address=104.36.82.16:4301 , piece_id=TYJHHPC45N5ABFEBZOKW7BYHIKW76TLGEWP6XMH3T4TO47VS3SBA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:28.022063143 (26h24m43.80299651s) [upload=25207]: Next returned nil segment Jul 28 00:16:28.022094667 (26h24m43.803028044s) [upload=25207]: Waiting for error group managing segments... Jul 28 00:16:28.183685599 (26h24m43.964618967s) [noise=true , address=77.90.31.10:4550 , piece_id=5NVINMEPBIBWZSDMPONBJQYRAC6DKUTMK6QAFDVQK5FFGGUGKVVQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:16:28.505796384 (26h24m44.286729756s) [noise=true , address=79.127.205.234:4613 , piece_id=LA2TID24XTDHJU6GHRAAZBOBWZYOTU7YIGGK2XNB6SWYFKFYU6IA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:16:35.139980295 (26h24m50.92091368s) [noise=true , address=79.127.201.215:4350 , piece_id=TVQ3Y6A2YCQB73DH6KGGTSF3MB7FY4DS6ATAHJRCOPAGN3GQDIGA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:17:27.784503680 (26h25m43.565437066s) [noise=true , address=104.36.82.44:4894 , piece_id=365MWDHILME2ETIEEJQLCHEA6W4RHRRIZXVA44LAE72M4YBZQ4XA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784581479 (26h25m43.56551485s) [noise=true , address=104.36.82.15:4279 , piece_id=SXEKQF4XDV2ZEIWFRTZAS7ZIJC6UQXJKRXOQTTNW57LC77SNO7CQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784720082 (26h25m43.565653446s) [noise=true , address=104.36.82.10:4106 , piece_id=LCXM4L2RAAZXTMKT5RRCQKSPXMZMVIN7OM2BZ4HPNRFY5FWPL5OQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12CsQq2BuV9hNc5fVAsw882NkDJMEXZeNwhV2FGNGKAU6wuFBHm, address:104.36.82.10:4106): protocol: rpc: dial tcp 104.36.82.10:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784829846 (26h25m43.565763213s) [noise=true , address=104.36.82.11:4138 , piece_id=D7S2HKPPXKVHEK2CEPB2VYOQQUCZ4HOUJKBIYDC7LAIAYBMSXUDQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:34080->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784890243 (26h25m43.565823611s) [noise=true , address=104.36.82.56:4251 , piece_id=PGVE5V7KGECIXX4CR5BUQGQP3UVJ73HFID6MI7GOAIEOAOEO5LTQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784952274 (26h25m43.565885656s) [noise=true , address=104.36.82.42:4806 , piece_id=4KNOPIDDZBCPWIOQJJXLNFYDB7GQNUBAZHVXZE4EXUMDKQ5IGYRA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: rpc: dial tcp 104.36.82.42:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785769660 (26h25m43.56670303s) [noise=true , address=104.36.82.53:4149 , piece_id=ONCUUK6NXATX4MDZT4KJAVRUCLMHBRLVXFC7UFAAWPAZ7HMZC2GQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785839624 (26h25m43.566773s) [noise=true , address=104.36.82.5:4856 , piece_id=5ERJ27Z546BOTRAQWJNPRFXWS2FWHNODVEGMNROJNSB5UPUPXSBQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: rpc: dial tcp 104.36.82.5:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785903354 (26h25m43.566836735s) [noise=true , address=104.36.82.14:4237 , piece_id=DTU64USY5MGTBQA2DAQZAB22OWWXIS3JU53SH2LSCYXDIZMPPHBQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: rpc: dial tcp 104.36.82.14:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785977411 (26h25m43.566910796s) [noise=true , address=104.36.82.9:4016 , piece_id=7FCSE65GL7OZB3P36TF4FL44BV4BIECGABLPAA7XFABQD35UGNOA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:104.36.82.9:4016): protocol: rpc: dial tcp 104.36.82.9:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786067275 (26h25m43.567000654s) [noise=true , address=104.36.82.50:4047 , piece_id=4EWCAJZJL6LIUSYU3R7VTUD6D5PTLANB3JMXLXEDNCAVOA5YEBYA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786097369 (26h25m43.567030756s) [noise=true , address=104.36.82.16:4301 , piece_id=TYJHHPC45N5ABFEBZOKW7BYHIKW76TLGEWP6XMH3T4TO47VS3SBA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786130870 (26h25m43.567064249s) [noise=true , address=104.36.82.12:4184 , piece_id=GPDRBM723G7SN4QCFGUN6X4DLWVL6Y3EWH3VSG53X4RE5ILXVUGA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:17JJjwwgSn7A6BS7n1a7fphX8oTEQpXmad25ifM2nDhWhPv4Ls, address:104.36.82.12:4184): protocol: rpc: dial tcp 104.36.82.12:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786177517 (26h25m43.567110893s) [noise=true , address=104.36.82.54:4163 , piece_id=XXYKDREEEU3MIRYTTSZMJHWUI4WZAQ3JLO3UX4A732E42V4XRBQA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786220037 (26h25m43.567153418s) [noise=true , address=104.36.82.52:4120 , piece_id=N2WBQD4TF5MYSCK6OW7DDU45S6VWBRCFR5VHUIRYKWLAGLOBT3QQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:60126->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786277291 (26h25m43.567210675s) [seg_pos={0 0}, upload=25207]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:17:27.817659788 (26h25m43.598593162s) [upload=25207]: Done uploading segments... Jul 28 00:17:27.817713581 (26h25m43.598646958s) [upload=25207]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.036063982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 207.560643ms FAIL (2025-07-28 00:16:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 207.6767ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 463.721553ms FAIL (2025-07-28 00:16:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.813559ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X...XX.] duration: 639.017655ms FAIL (2025-07-28 00:16:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 639.054544ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [................................................XXX.XXXXXXXX] duration: 1m0.036761385s FAIL (2025-07-28 00:16:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== .36.82.9:4026 , piece_id=ROOWLE5W5DD6SPW62XAWBGP436BFP26RL4LPRR7GPYMI36RDXLWQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.925662103 (26h23m43.706595492s) [noise=true , address=104.36.82.12:4162 , piece_id=HOATZMHA236S6HHI3GRXWEIF6SJBGCYJ3VI25MUD3TSX4C4USWDQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.925826790 (26h23m43.706760222s) [noise=true , address=79.127.201.215:4350 , piece_id=HKUPEYZUTGTZH7GELY5MJ3S7JJ77VTP2YCPKIKR5TLTBRQYBMD6Q, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.926169705 (26h23m43.707103093s) [noise=true , address=104.36.82.20:4440 , piece_id=EEKZUVMCSDTKUZY3LE7TCRKCZUTQ7NFO4GID7LRPP4C5ONOHHU4A, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.926624569 (26h23m43.707557956s) [noise=true , address=104.36.82.46:4493 , piece_id=YFMQRRRFWCVYUWHDLLUCZHCSRFHUFTGFLJHTF6RMA4WXDLE4NYWQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.927148024 (26h23m43.708081412s) [noise=true , address=104.36.82.56:4238 , piece_id=3XDAZIPJ2IVVS2X2BOM72IIUC5MLR4ZXPESZBJYUD23MSDDBKVJA, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.929483114 (26h23m43.710416501s) [noise=true , address=104.36.82.42:4822 , piece_id=3ZOW7G3FKFZYENB5TD6E7N7UWOGZ6ZG77RIRBFIHIXZCLU3CK5DA, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:28.151750613 (26h23m43.932684003s) [upload=25193]: Next returned nil segment Jul 28 00:15:28.151774469 (26h23m43.932707857s) [upload=25193]: Waiting for error group managing segments... Jul 28 00:15:31.573441393 (26h23m47.354374763s) [noise=true , address=64.49.13.138:4482 , piece_id=RVVBXQSP4QNQ675TMHFJ2VVJ74PP75OVPQD6HAUDBNGWRH3YR6IA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: <nil> Jul 28 00:15:31.683482357 (26h23m47.464415723s) [noise=true , address=79.127.201.215:4350 , piece_id=HKUPEYZUTGTZH7GELY5MJ3S7JJ77VTP2YCPKIKR5TLTBRQYBMD6Q, seg_pos={0 0}, upload=25193]: Done uploading piece. err: <nil> Jul 28 00:16:27.686476811 (26h24m43.467410201s) [noise=true , address=104.36.82.21:4485 , piece_id=JRNTRN5WRXPVPNABRZJLXV3MKQ45TFCFRNMKDWU5MKMSVVVTB26A, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1wmohWyFaMkNm8KBAZprmZ6RuiH6UcB8KRNght2HGpV3Ucnnq9, address:104.36.82.21:4485): protocol: write tcp 159.223.143.39:56582->104.36.82.21:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686567604 (26h24m43.467500984s) [noise=true , address=104.36.82.15:4288 , piece_id=Z2XYI2UO5O6N6G7UFN2GNHQHJTKVSLRJC4BMABL53VMFB32URB7Q, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: write tcp 159.223.143.39:60850->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686637887 (26h24m43.467571251s) [noise=true , address=104.36.82.9:4026 , piece_id=ROOWLE5W5DD6SPW62XAWBGP436BFP26RL4LPRR7GPYMI36RDXLWQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq, address:104.36.82.9:4026): protocol: write tcp 159.223.143.39:38716->104.36.82.9:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686649054 (26h24m43.467582438s) [noise=true , address=104.36.82.16:4301 , piece_id=MAE3ULPMAGTXUEUJTN3HRU5MIXCCXXHLAPUQDGQ6LFDLMWHIGBOA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686706874 (26h24m43.467640251s) [noise=true , address=104.36.82.46:4493 , piece_id=YFMQRRRFWCVYUWHDLLUCZHCSRFHUFTGFLJHTF6RMA4WXDLE4NYWQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 159.223.143.39:48368->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686775467 (26h24m43.467708898s) [noise=true , address=104.36.82.56:4238 , piece_id=3XDAZIPJ2IVVS2X2BOM72IIUC5MLR4ZXPESZBJYUD23MSDDBKVJA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 159.223.143.39:41062->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686836459 (26h24m43.467769835s) [noise=true , address=104.36.82.52:4095 , piece_id=ZAK6REHMKJRGFSQZN7JYFE2VELV56YHBVO7447PGZ5YKDP4M4JLQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: write tcp 159.223.143.39:51398->104.36.82.52:4095: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686897167 (26h24m43.467830546s) [noise=true , address=104.36.82.10:4085 , piece_id=AUKDTOBTBCZV536ATUWWBMUYT3R4Z3ISBHQYCMJLKTHRUONWFYGQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:18ifvnz4vwaopBJQbF88B1N6DGFmEtfH4jhD3YcneLXCPj7KVu, address:104.36.82.10:4085): protocol: write tcp 159.223.143.39:33584->104.36.82.10:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686969718 (26h24m43.467903096s) [noise=true , address=104.36.82.11:4139 , piece_id=BC3G5RZFYEUZZSTVIDTVIT4LPEJSRY6LCASLDEEP547IMST6A5NQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: write tcp 159.223.143.39:59266->104.36.82.11:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686981934 (26h24m43.467915298s) [noise=true , address=104.36.82.13:4210 , piece_id=E37AKJU5GOPR7LYQNSRFLRVNKVQ6WF7P4T3LLVPL6C6DWFLS6WGA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1ybrcx8SPgyEsJWq2P1KYaTwTj5v5BfENKNEZZxaKRvKvuqTiL, address:104.36.82.13:4210): protocol: write tcp 159.223.143.39:46158->104.36.82.13:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687001338 (26h24m43.467934714s) [noise=true , address=104.36.82.12:4162 , piece_id=HOATZMHA236S6HHI3GRXWEIF6SJBGCYJ3VI25MUD3TSX4C4USWDQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: write tcp 159.223.143.39:33124->104.36.82.12:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687099652 (26h24m43.468033029s) [noise=true , address=104.36.82.13:4211 , piece_id=ZGHGK24JBYSY3LULAROUGMBKGJQ6UGJKT6LIXLDCJLZB5GOWD2MA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 159.223.143.39:42726->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687118317 (26h24m43.468051679s) [noise=true , address=104.36.82.14:4237 , piece_id=OXW7X4GDRZG7GDXFVQWWJGTFH5PSCEIKQCP3KCUU4YO62647NJEQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: write tcp 159.223.143.39:52348->104.36.82.14:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687155178 (26h24m43.468088554s) [noise=true , address=104.36.82.42:4822 , piece_id=3ZOW7G3FKFZYENB5TD6E7N7UWOGZ6ZG77RIRBFIHIXZCLU3CK5DA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:104.36.82.42:4822): protocol: write tcp 159.223.143.39:46948->104.36.82.42:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687182023 (26h24m43.468115386s) [noise=true , address=104.36.82.5:4849 , piece_id=VDSTIO7PESNO46YJW3UDLUXWKMJKQ4KLKCRGWYRQI42RQQJGWGHA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:104.36.82.5:4849): protocol: write tcp 159.223.143.39:38158->104.36.82.5:4849: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687215640 (26h24m43.468149002s) [noise=true , address=104.36.82.20:4440 , piece_id=EEKZUVMCSDTKUZY3LE7TCRKCZUTQ7NFO4GID7LRPP4C5ONOHHU4A, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: rpc: dial tcp 104.36.82.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687241549 (26h24m43.468174929s) [seg_pos={0 0}, upload=25193]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:16:27.712793754 (26h24m43.493727138s) [upload=25193]: Done uploading segments... Jul 28 00:16:27.712827277 (26h24m43.493760665s) [upload=25193]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.036790795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 312.016888ms FAIL (2025-07-28 00:16:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 312.051914ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X...XX] duration: 30.03396293s FAIL (2025-07-28 00:16:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034013776s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.031163038s FAIL (2025-07-28 00:15:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031189908s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 447.753696ms FAIL (2025-07-28 00:15:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.789913ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 420.6864ms FAIL (2025-07-28 00:15:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.728911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.................................................XXX.XXXXXXX] duration: 1m0.032540778s FAIL (2025-07-28 00:15:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 0]: Uploading piece... Jul 28 00:14:13.109307738 (26h22m28.89024112s) [noise=true , address=104.36.82.54:4188 , piece_id=DWIXYV5CLBFIXN33MPK23XPUQ5YMRYRGTOHUNY4E3EU4Y4AAHXLQ, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.110312438 (26h22m28.891245819s) [noise=true , address=104.36.82.56:4234 , piece_id=EASMBX2B2SWRQPC53IZURYNDNVED2UKHLOZOONICILEU3C7PC3VA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.111011852 (26h22m28.891945236s) [noise=true , address=104.36.82.17:4341 , piece_id=AIARVMMJRPT5ZYP5CUENWCLC2RKJDC4NW3H4EBUZRWBDWTT7KWLQ, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.112135088 (26h22m28.893177366s) [noise=true , address=104.36.82.15:4270 , piece_id=DWT5GQ6BZP2HRMZQCPMJBPCBQM5EX6V44XTBX3NMVSL3MRT2PMUA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.113076416 (26h22m28.894009803s) [noise=true , address=104.36.82.12:4178 , piece_id=RCU6AQEOC5MJQ7AK7PUOYLIHJSVNHVMEDFEBYS3MFR7SFZFNDPMA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.113770830 (26h22m28.894704208s) [noise=true , address=104.36.82.20:4461 , piece_id=WPTUY4RDBU5ZLUW2CCS26TXHXZXZDXOLBMXPGZ7D6EGFL5NJEG3Q, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.114244130 (26h22m28.895177515s) [noise=true , address=104.36.82.10:4089 , piece_id=V2JHR5RUGODRWVGRC355TBTCJQR53NNN3JQK26MD43FBETDEXXDA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.114704162 (26h22m28.895637556s) [noise=true , address=104.36.82.41:4762 , piece_id=LHZOHXQCKRYTDPSSZKMCXPILYF5RBNZ634IM7XQKPH7RQONHULAA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.115162808 (26h22m28.896096189s) [noise=true , address=79.127.201.215:4347 , piece_id=4HXNLJCWQHEFE44TSKE47MPEKXZRHXF3OVLFJEEOPLWEQRTGQQJA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.327044919 (26h22m29.107978306s) [upload=25170]: Next returned nil segment Jul 28 00:14:13.327067921 (26h22m29.108001307s) [upload=25170]: Waiting for error group managing segments... Jul 28 00:14:14.316724863 (26h22m30.097658243s) [noise=true , address=64.49.13.140:4508 , piece_id=5ZLAT7WAUDC54I7L7MHEEMFTSJBPTFDF7VQ726USBSYPUZXKTMIQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: <nil> Jul 28 00:14:14.687386935 (26h22m30.468320304s) [noise=true , address=79.127.201.215:4347 , piece_id=4HXNLJCWQHEFE44TSKE47MPEKXZRHXF3OVLFJEEOPLWEQRTGQQJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: <nil> Jul 28 00:15:13.044434976 (26h23m28.825368379s) [noise=true , address=104.36.82.52:4110 , piece_id=SYEZSKJ6JKHXPV5OKVQGMLPA2LJ5MVBOCQRPEKUQNNHJ5R4H3OSQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044525852 (26h23m28.825459227s) [noise=true , address=104.36.82.17:4341 , piece_id=AIARVMMJRPT5ZYP5CUENWCLC2RKJDC4NW3H4EBUZRWBDWTT7KWLQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1BfZXtm2GymdBAaEBFaX9jeYbK3PveonNNskUq9cLmuW7vbuu5, address:104.36.82.17:4341): protocol: rpc: dial tcp 104.36.82.17:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044602133 (26h23m28.825535522s) [noise=true , address=104.36.82.41:4762 , piece_id=LHZOHXQCKRYTDPSSZKMCXPILYF5RBNZ634IM7XQKPH7RQONHULAA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:104.36.82.41:4762): protocol: rpc: dial tcp 104.36.82.41:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044677731 (26h23m28.825611112s) [noise=true , address=104.36.82.15:4270 , piece_id=DWT5GQ6BZP2HRMZQCPMJBPCBQM5EX6V44XTBX3NMVSL3MRT2PMUA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: rpc: dial tcp 104.36.82.15:4270: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044753011 (26h23m28.825686404s) [noise=true , address=104.36.82.12:4178 , piece_id=RCU6AQEOC5MJQ7AK7PUOYLIHJSVNHVMEDFEBYS3MFR7SFZFNDPMA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12LjJ3SmaNTQi8U1rPSGmR461UYC2CfMtoZ22SSV97hDuhMwC4V, address:104.36.82.12:4178): protocol: rpc: dial tcp 104.36.82.12:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044823126 (26h23m28.825756514s) [noise=true , address=104.36.82.20:4461 , piece_id=WPTUY4RDBU5ZLUW2CCS26TXHXZXZDXOLBMXPGZ7D6EGFL5NJEG3Q, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:4461): protocol: rpc: dial tcp 104.36.82.20:4461: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044888270 (26h23m28.825821661s) [noise=true , address=104.36.82.10:4089 , piece_id=V2JHR5RUGODRWVGRC355TBTCJQR53NNN3JQK26MD43FBETDEXXDA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: rpc: dial tcp 104.36.82.10:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044967360 (26h23m28.825900741s) [noise=true , address=104.36.82.42:4803 , piece_id=D56DL3VWAML63HYCC4Y5YA3CQ5HGFSJU7GHIDDQ33JQXXSADQBBQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: rpc: dial tcp 104.36.82.42:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045036746 (26h23m28.825970122s) [noise=true , address=104.36.82.46:4493 , piece_id=QFDLCNQMCFPPHCHRB2QEAZ3LWPY6TGTP33DIVHQNEWHMQMHJ6MJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045106372 (26h23m28.82603975s) [noise=true , address=104.36.82.6:4888 , piece_id=YUC3BCP4NAAHD52KJR77RZM2FVBC4DWS23TBM2Y3IJTHSHSL4CMQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12HwYszCTcXEHqSNnWJpT11NpKg8cFUvdAqU2wUigsLbxmDFkg6, address:104.36.82.6:4888): protocol: rpc: dial tcp 104.36.82.6:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045176922 (26h23m28.826110327s) [noise=true , address=104.36.82.3:4776 , piece_id=C4QWIJRV5F3EUSWUIMOMRV772IZN5H7NX4452EPE3HYNBPIIHPZQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:104.36.82.3:4776): protocol: rpc: dial tcp 104.36.82.3:4776: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045242772 (26h23m28.82617615s) [noise=true , address=104.36.82.56:4234 , piece_id=EASMBX2B2SWRQPC53IZURYNDNVED2UKHLOZOONICILEU3C7PC3VA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045309795 (26h23m28.826243195s) [noise=true , address=104.36.82.53:4134 , piece_id=HYU23YI272VYCRXCIV7KAARYWZOWVI275R2EUKC46KMCYH4R3XIQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045375571 (26h23m28.826308951s) [noise=true , address=104.36.82.51:4084 , piece_id=ZASL34H2IXTCIRAE6RT2NNTMIYHHEJSC46WPNW4RW4JXFE2UQ5GA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9, address:104.36.82.51:4084): protocol: rpc: dial tcp 104.36.82.51:4084: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045441696 (26h23m28.826375086s) [noise=true , address=104.36.82.44:4879 , piece_id=6RBYYWEU35DHRUT2GMCQRR5KI2ADIUAIIKZEQJ7W7P3IMTPPUMJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:104.36.82.44:4879): protocol: rpc: dial tcp 104.36.82.44:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045506193 (26h23m28.826439579s) [noise=true , address=104.36.82.54:4188 , piece_id=DWIXYV5CLBFIXN33MPK23XPUQ5YMRYRGTOHUNY4E3EU4Y4AAHXLQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: write tcp 159.223.143.39:50888->104.36.82.54:4188: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045528582 (26h23m28.82646199s) [seg_pos={0 0}, upload=25170]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:15:13.072639623 (26h23m28.853573015s) [upload=25170]: Done uploading segments... Jul 28 00:15:13.072691182 (26h23m28.853624558s) [upload=25170]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.032564381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 234.34641ms FAIL (2025-07-28 00:15:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 234.376342ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 378.305215ms FAIL (2025-07-28 00:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.335918ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 388.290616ms FAIL (2025-07-28 00:14:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 388.357713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 437.450553ms FAIL (2025-07-28 00:14:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.481769ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 227.924102ms FAIL (2025-07-28 00:14:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 227.955251ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................................................XXX.XXXXXX] duration: 1m0.036397337s FAIL (2025-07-28 00:14:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== .36.82.12:4174 , piece_id=MUBJTG2N5Q7CITTYLK6PBKK34QE7PKSRLCE544T23FNV6A4LJEPQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.981704249 (26h21m28.76263763s) [noise=true , address=104.36.82.6:4901 , piece_id=OI5GF6QFN7WOH52NWYJCZECLY5QDYFD5JGXOGRJMEXCEQVGQURBQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982024129 (26h21m28.762957507s) [noise=true , address=104.36.82.15:4279 , piece_id=PVYGX4X3QUGD3D6TD2NXENMLKHRQO5QTKXEAPN4CH4OWV3EXEFYQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982367533 (26h21m28.763300914s) [noise=true , address=104.36.82.16:4316 , piece_id=RXGU7UI3BQE4VL2TQAYEUZUB6QCJ2JG6ALIIJAFKJFI74EAWKEHA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982715686 (26h21m28.763649065s) [noise=true , address=104.36.82.14:4241 , piece_id=DLCOMMAKIULBHDCXOT75YUO5PAOJFRZW4UCGPHWQ5DW2CXEB2LLA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982997054 (26h21m28.763930433s) [noise=true , address=104.36.82.53:4156 , piece_id=TYYK7PPV2HXY3XIIEYB355AR5TS7ETLUWDUU3JNS66G32NPR64BQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.983369414 (26h21m28.764302805s) [noise=true , address=104.36.82.54:4165 , piece_id=FBXBURTZVPMXJE63UPPDDP6YPBDKOJCCSDWDR5U7AB4WVWUJDHJA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.983732452 (26h21m28.76466583s) [noise=true , address=104.36.82.42:4814 , piece_id=GYTSXV42APDBOAUGUBHLSYO5H57KAOLXE3QL7UXH4LCNDSGITSUA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.984085942 (26h21m28.76501932s) [noise=true , address=79.127.201.214:4291 , piece_id=WSWGLWB3QCDN7LVAYQ4PMHNXAIRNQUN2EC27MMH72AXF3ZOTLYUQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.984485560 (26h21m28.765418945s) [noise=true , address=104.36.82.56:4251 , piece_id=YA4VCKOQA5E2B7TF25I4JQWNRNKXSNC6E23W3XY77LPTF23WG2JA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:13.189966174 (26h21m28.970899552s) [upload=25157]: Next returned nil segment Jul 28 00:13:13.189983326 (26h21m28.970916692s) [upload=25157]: Waiting for error group managing segments... Jul 28 00:13:13.621200687 (26h21m29.402134056s) [noise=true , address=79.127.201.214:4291 , piece_id=WSWGLWB3QCDN7LVAYQ4PMHNXAIRNQUN2EC27MMH72AXF3ZOTLYUQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:13:14.450347470 (26h21m30.231280849s) [noise=true , address=64.49.13.138:4483 , piece_id=7HHKAGCEXD4KUVEOTIQW4YK6MRR57LQJMXYRZHS6TJO75CCIMH7Q, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:13:14.538549427 (26h21m30.319482811s) [noise=true , address=79.127.201.215:4343 , piece_id=2FPWHHI4O366U6WRP2PRVUGEKEL2F7A2WQAYZXYPVOYPA7PNGN3Q, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:14:12.912102155 (26h22m28.693035531s) [noise=true , address=104.36.82.16:4316 , piece_id=RXGU7UI3BQE4VL2TQAYEUZUB6QCJ2JG6ALIIJAFKJFI74EAWKEHA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12PPnXDzwXDVDKemw483GXuzQrPN48JJrjeeD7az2NPNawHNaXf, address:104.36.82.16:4316): protocol: rpc: dial tcp 104.36.82.16:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912253520 (26h22m28.693186895s) [noise=true , address=104.36.82.20:4447 , piece_id=CXGMHB62TKJ6NIYMVMGIIINYXHGMTD23BBHBYZUHGCM46ZZ2VUFQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1Li8ZwrWGPx5Y478pYLwnWJ7N2x3dbXgwGaUGWVtVhvv61EnTs, address:104.36.82.20:4447): protocol: rpc: dial tcp 104.36.82.20:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912347582 (26h22m28.693280968s) [noise=true , address=104.36.82.4:4825 , piece_id=6PCCIUN63YUI4VXDRIVM5C5FQXB7B4NBYJ57QNRDUSP6NCWNKFDA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:104.36.82.4:4825): protocol: rpc: dial tcp 104.36.82.4:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912449723 (26h22m28.693383098s) [noise=true , address=104.36.82.5:4872 , piece_id=YM6BND267SZEK2HOE7AHCU7LIGJLJ2P7PLV4EVCPACCDJZQZKFCA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: write tcp 159.223.143.39:50330->104.36.82.5:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912653063 (26h22m28.693586453s) [noise=true , address=104.36.82.42:4814 , piece_id=GYTSXV42APDBOAUGUBHLSYO5H57KAOLXE3QL7UXH4LCNDSGITSUA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12rnmX5iuA5swKceTZVV14HLaUctRFimn3DPLjRogg4K3N7PWnU, address:104.36.82.42:4814): protocol: rpc: dial tcp 104.36.82.42:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912722923 (26h22m28.693656311s) [noise=true , address=104.36.82.56:4251 , piece_id=YA4VCKOQA5E2B7TF25I4JQWNRNKXSNC6E23W3XY77LPTF23WG2JA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912786120 (26h22m28.693719481s) [noise=true , address=104.36.82.15:4279 , piece_id=PVYGX4X3QUGD3D6TD2NXENMLKHRQO5QTKXEAPN4CH4OWV3EXEFYQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912854267 (26h22m28.693787634s) [noise=true , address=104.36.82.14:4241 , piece_id=DLCOMMAKIULBHDCXOT75YUO5PAOJFRZW4UCGPHWQ5DW2CXEB2LLA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912911508 (26h22m28.693844877s) [noise=true , address=104.36.82.13:4202 , piece_id=ZNMT57OYPDKVYPTMXLDUCEPFGF24ZPDTPM64GDABVXZM76ZHFISA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: rpc: dial tcp 104.36.82.13:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912954899 (26h22m28.693888274s) [noise=true , address=104.36.82.54:4165 , piece_id=FBXBURTZVPMXJE63UPPDDP6YPBDKOJCCSDWDR5U7AB4WVWUJDHJA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913027995 (26h22m28.693961371s) [noise=true , address=104.36.82.52:4100 , piece_id=6YR6JR2G3QGHMWFXD36HVW6VA5ZM2T7AOPV3BYLFK2VMR4G4KTDQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913037598 (26h22m28.693970974s) [noise=true , address=104.36.82.6:4901 , piece_id=OI5GF6QFN7WOH52NWYJCZECLY5QDYFD5JGXOGRJMEXCEQVGQURBQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: rpc: dial tcp 104.36.82.6:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913113686 (26h22m28.694047063s) [noise=true , address=104.36.82.55:4226 , piece_id=WVMP3URSBIFGUTHGYVAZHIMITT6PB6TO6IJJ36QNBRC65ISWAXYA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1qN8jYqiZGuPjqHSp6siqdT4p33qvf1vYdAkMZKRnFgcQj9Ga3, address:104.36.82.55:4226): protocol: write tcp 159.223.143.39:35470->104.36.82.55:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913119763 (26h22m28.694053131s) [noise=true , address=104.36.82.12:4174 , piece_id=MUBJTG2N5Q7CITTYLK6PBKK34QE7PKSRLCE544T23FNV6A4LJEPQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12ZqPqvh4SVpDmkpVKQXZUW48CtzV9VMeMGJoDZwA7KpNfmjfow, address:104.36.82.12:4174): protocol: write tcp 159.223.143.39:32996->104.36.82.12:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913177711 (26h22m28.694111087s) [noise=true , address=104.36.82.53:4156 , piece_id=TYYK7PPV2HXY3XIIEYB355AR5TS7ETLUWDUU3JNS66G32NPR64BQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 159.223.143.39:60108->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913210562 (26h22m28.694143938s) [seg_pos={0 0}, upload=25157]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:14:12.945831453 (26h22m28.72676483s) [upload=25157]: Done uploading segments... Jul 28 00:14:12.945859681 (26h22m28.726793062s) [upload=25157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.036431268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 364.775429ms FAIL (2025-07-28 00:14:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 364.801553ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 443.007413ms FAIL (2025-07-28 00:13:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.039016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 416.931352ms FAIL (2025-07-28 00:13:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 416.963078ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.037423039s FAIL (2025-07-28 00:13:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037452646s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 199.836848ms FAIL (2025-07-28 00:13:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 199.868383ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.028617263s FAIL (2025-07-28 00:13:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028640755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXX.XXXXX] duration: 1m0.028592919s FAIL (2025-07-28 00:13:07 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== .123893136s) [noise=true , address=104.36.82.56:4253 , piece_id=JC6VQQH5P4ERA3QKN2W7H4ZCC5A6ITESZKUOBYNMSYATOUJBBECQ, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.343460119 (26h20m23.124393506s) [noise=true , address=104.36.82.13:4196 , piece_id=YYVFDLRGYKEJCKJY5DXAIY2YPVGIZPNAX3RBLU5VJVML7OW7NQFQ, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.353379830 (26h20m23.134313216s) [noise=true , address=104.36.82.12:4179 , piece_id=C5E4N56NVTCR6XIVCU7SNVZ5CH2X4LQDWR7VJJGAM2QPAVGS3GXA, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.353919132 (26h20m23.134852516s) [noise=true , address=104.36.82.55:4198 , piece_id=VBDHF4NL2FNLKSDXMLDBOKVQ73VOLDOZ6HOA7LRGM674RQGGMPSQ, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.354405150 (26h20m23.135338536s) [noise=true , address=104.36.82.46:4513 , piece_id=RNEP22LW33SCXDZCPANARBUQ7EAAT46CE74KLNLQSRYN7X3WQCIA, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.354801595 (26h20m23.135734982s) [noise=true , address=104.36.82.21:4475 , piece_id=4YKN35GO4EQEJLEPJB3JNZLJDNT3ZGZCBCG2RWAC33NRG5FBYUXA, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:12:07.355216057 (26h20m23.136149437s) [noise=true , address=104.36.82.14:4229 , piece_id=F52Z3UVUVA7GUDIF5NVEJZQ637JTI5J3AYDZLJCMMOJO5B4KVCPA, seg_pos={0 0}, upload=25141]: Uploading piece... Jul 28 00:13:07.267331858 (26h21m23.048265236s) [upload=25141]: Next segment err: context deadline exceeded Jul 28 00:13:07.267449330 (26h21m23.048382708s) [noise=true , address=104.36.82.15:4268 , piece_id=WRZW72QD5LL745OFJQQ2O2DCGP3B7QPBZODIA7KGOJVTUY55UOZA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:112ocg8g8hf2huUskC1Kty3z51wXc8VuNUgARTFjKZxsF6RFFy, address:104.36.82.15:4268): protocol: write tcp 159.223.143.39:34046->104.36.82.15:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267541248 (26h21m23.048474621s) [noise=true , address=104.36.82.13:4200 , piece_id=WPNXRIKPTYX7A4YVLS3DQ6U6AWF6E5D6MG4DD52OTF2ZAW4GOZPA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: write tcp 159.223.143.39:41434->104.36.82.13:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267629076 (26h21m23.048562458s) [noise=true , address=104.36.82.21:4475 , piece_id=4YKN35GO4EQEJLEPJB3JNZLJDNT3ZGZCBCG2RWAC33NRG5FBYUXA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:104.36.82.21:4475): protocol: write tcp 159.223.143.39:47970->104.36.82.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267686400 (26h21m23.048619773s) [noise=true , address=104.36.82.56:4253 , piece_id=JC6VQQH5P4ERA3QKN2W7H4ZCC5A6ITESZKUOBYNMSYATOUJBBECQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 159.223.143.39:36380->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267758464 (26h21m23.048691853s) [noise=true , address=104.36.82.55:4198 , piece_id=VBDHF4NL2FNLKSDXMLDBOKVQ73VOLDOZ6HOA7LRGM674RQGGMPSQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:39324->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267817056 (26h21m23.048750432s) [noise=true , address=104.36.82.16:4319 , piece_id=I37OS7WF6FVJHP2BLFTX42GTXA4SXYMBVBVC264TGPCDBJY5OBIQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C, address:104.36.82.16:4319): protocol: rpc: dial tcp 104.36.82.16:4319: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267864114 (26h21m23.048797482s) [noise=true , address=104.36.82.11:4136 , piece_id=QKU2JEIDIZ5Z43QNBTNU2Y6X76E75FLFGSMQIDUXVQTTBQFV35UQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: write tcp 159.223.143.39:51398->104.36.82.11:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267917602 (26h21m23.048850977s) [noise=true , address=104.36.82.52:4087 , piece_id=HXKQJCIVMZ6IYOSL2QIGPFVQQPZSRTDDYCO4QFTXSU3LZTWSGWCQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.267991526 (26h21m23.048924891s) [noise=true , address=104.36.82.4:4825 , piece_id=4RFZWOYMBMDR7SYIMJFOPKI5EF3W3EPCAYD2PQ53D76JFWFZBH5A, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:104.36.82.4:4825): protocol: write tcp 159.223.143.39:46644->104.36.82.4:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268015470 (26h21m23.04894885s) [noise=true , address=104.36.82.44:4895 , piece_id=WJ6GHPOTOBEG7ZA2RLJ5T2PEHIUPU5AXCAKR6ACFR2DS2RD2NKCA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: rpc: dial tcp 104.36.82.44:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268053916 (26h21m23.048987281s) [noise=true , address=104.36.82.47:4524 , piece_id=YZ6FHN4X3AGOTE7IZWJCVIG4CWF2GJU44XBVYGEI2NK7J7VEU6IA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 159.223.143.39:32982->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268107324 (26h21m23.04904069s) [noise=true , address=104.36.82.14:4229 , piece_id=F52Z3UVUVA7GUDIF5NVEJZQ637JTI5J3AYDZLJCMMOJO5B4KVCPA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:12uiCMvJ7pnwXfN57CVu8zR73tikwUZiXr3sG7Xotd9w6KVnViL, address:104.36.82.14:4229): protocol: rpc: dial tcp 104.36.82.14:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268330067 (26h21m23.049263446s) [noise=true , address=104.36.82.49:4000 , piece_id=X6YJBZ35ZQUKCFNS5QKDPHAADX3CEQ224FZ53ZM56JW6364XN6NA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:12chhfe1weiPC1jjv8vbScP7uFVNXFr6s3UcNRkH8gsSAtGD16D, address:104.36.82.49:4000): protocol: rpc: dial tcp 104.36.82.49:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268404831 (26h21m23.049338208s) [noise=true , address=104.36.82.13:4196 , piece_id=YYVFDLRGYKEJCKJY5DXAIY2YPVGIZPNAX3RBLU5VJVML7OW7NQFQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1aLZa6kv5VhDgqvUZp2HkzpkbA81KU13iGTTsiVfSeUDzLu8Lu, address:104.36.82.13:4196): protocol: rpc: dial tcp 104.36.82.13:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268426132 (26h21m23.049359508s) [noise=true , address=104.36.82.40:4966 , piece_id=ECYGFLQHPE7LF63FOHF7VRD7GU3PMDOP25GDIZR6PRR4IF45TYSQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:104.36.82.40:4966): protocol: rpc: dial tcp 104.36.82.40:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268447661 (26h21m23.049381028s) [noise=true , address=104.36.82.46:4513 , piece_id=RNEP22LW33SCXDZCPANARBUQ7EAAT46CE74KLNLQSRYN7X3WQCIA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:55222->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268457887 (26h21m23.049391256s) [noise=true , address=104.36.82.6:4886 , piece_id=7AANXPZGE5QVG6XF75UY56KYAFB66WEC4NAVRNELVJBGFALKHKKQ, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:1QyBnKe6fVUVSg38Hv6aPP6pY7haXCa4kQb45LHkPTSdTWHjkF, address:104.36.82.6:4886): protocol: write tcp 159.223.143.39:49550->104.36.82.6:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268483566 (26h21m23.049416935s) [noise=true , address=104.36.82.12:4179 , piece_id=C5E4N56NVTCR6XIVCU7SNVZ5CH2X4LQDWR7VJJGAM2QPAVGS3GXA, seg_pos={0 0}, upload=25141]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:07.268524307 (26h21m23.049457684s) [seg_pos={0 0}, upload=25141]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.028615629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 301.259041ms FAIL (2025-07-28 00:12:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 301.282375ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXX.XXXX] duration: 1m0.036061248s FAIL (2025-07-28 00:12:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== piece_id=LQXFQ6YA5PGLFPUMNUPEI6Q5OZ7ECM5G2TJRPTPUIDULVWHEK4CQ, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.228794405 (26h19m23.009727795s) [noise=true , address=104.36.82.49:4001 , piece_id=L2ZTVGVFZVVXOYUMIAGLP4G5WSX7CGI6RF3ZCPYNFADPPRALOLQA, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.229238021 (26h19m23.010171411s) [noise=true , address=104.36.82.52:4098 , piece_id=ULG6O4MSC2FM33V5CBFU5SE3WJUCX2IR7V3VOKYNDDXRPPZVGB5Q, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.229326021 (26h19m23.010259443s) [noise=true , address=104.36.82.12:4167 , piece_id=Q6L6CXWCD2OC3SOJDVMRV4LUAHEGRCM7VR4E7HRDXGB4EWSBS56Q, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.229805823 (26h19m23.010739209s) [noise=true , address=104.36.82.53:4137 , piece_id=7ZMSCY4TDG7F5CDDOR7DQTVJV7ZWXEJFF3CJCHB62KSN5MKM2RTQ, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.230477820 (26h19m23.011411211s) [noise=true , address=104.36.82.50:4040 , piece_id=MNVSSLTPV23U7KA3BCELCBUHBYKUW6SKA6UWLW6YPQR53FTYWWXA, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.230978688 (26h19m23.01191216s) [noise=true , address=104.36.82.55:4220 , piece_id=N52BMPB4SWD7BTNEG5YEGDSRACLGDJYAA6UWA3N5XR6D4WVUQREA, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.231401150 (26h19m23.012334541s) [noise=true , address=104.36.82.41:4777 , piece_id=CFKLLBIUYABNM45F23VN2P5K6XAI42S3BFASPJ6F6QDOJQOZLYYQ, seg_pos={0 0}, upload=25127]: Uploading piece... Jul 28 00:11:07.324161266 (26h19m23.105094648s) [upload=25127]: Next returned nil segment Jul 28 00:11:07.324192263 (26h19m23.105125657s) [upload=25127]: Waiting for error group managing segments... Jul 28 00:11:07.470726304 (26h19m23.251659688s) [noise=true , address=77.90.31.16:4744 , piece_id=LQXFQ6YA5PGLFPUMNUPEI6Q5OZ7ECM5G2TJRPTPUIDULVWHEK4CQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: <nil> Jul 28 00:11:08.190141210 (26h19m23.971074581s) [noise=true , address=64.49.13.169:4710 , piece_id=YA5QWZDTUG6MNBGHXYFRLIFHRCAGM57CRBKL4DGDESJPEB3CPNGQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: <nil> Jul 28 00:11:08.370272398 (26h19m24.15120578s) [noise=true , address=79.127.201.215:4359 , piece_id=RM7OKIUXHZPBPHTHU5DPDLDJNQUGBEPBP7DU3TSCUAOHF7ZVUQZA, seg_pos={0 0}, upload=25127]: Done uploading piece. err: <nil> Jul 28 00:12:07.150221139 (26h20m22.931154506s) [noise=true , address=104.36.82.50:4040 , piece_id=MNVSSLTPV23U7KA3BCELCBUHBYKUW6SKA6UWLW6YPQR53FTYWWXA, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn, address:104.36.82.50:4040): protocol: write tcp 159.223.143.39:49794->104.36.82.50:4040: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150316034 (26h20m22.931249401s) [noise=true , address=104.36.82.55:4220 , piece_id=N52BMPB4SWD7BTNEG5YEGDSRACLGDJYAA6UWA3N5XR6D4WVUQREA, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 159.223.143.39:40840->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150403437 (26h20m22.931336799s) [noise=true , address=104.36.82.41:4777 , piece_id=CFKLLBIUYABNM45F23VN2P5K6XAI42S3BFASPJ6F6QDOJQOZLYYQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt, address:104.36.82.41:4777): protocol: write tcp 159.223.143.39:43786->104.36.82.41:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150500465 (26h20m22.931433833s) [noise=true , address=104.36.82.56:4233 , piece_id=TYTBLKIVEYMJF7Q4KWY2TA6RHQMKSRUAO7JD6J2YU4SAYZZU3OTQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 159.223.143.39:48392->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150569714 (26h20m22.93150308s) [noise=true , address=104.36.82.15:4270 , piece_id=M4C64K7LIUYFYCFPU3BIVA2TRI3JHY6Y3W64FES4WCCJKHUUNMCQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: write tcp 159.223.143.39:59074->104.36.82.15:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150632799 (26h20m22.931566165s) [noise=true , address=104.36.82.42:4797 , piece_id=KGH4M6FVNS3B3CJL6A5ZKSW6HLDLHP6A6KWDBNKNI2HSBPFRSV7Q, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:104.36.82.42:4797): protocol: write tcp 159.223.143.39:32854->104.36.82.42:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150683045 (26h20m22.93161641s) [noise=true , address=104.36.82.53:4154 , piece_id=P6S7AIDIS26A6CFIF7T2W26W7QCLWIHSAF6ENDAKIXDAPVOIDEIA, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 159.223.143.39:59490->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150759934 (26h20m22.931693297s) [noise=true , address=104.36.82.47:4517 , piece_id=LJICEH742SXNVJA2IR7JEZE4G2GBZXUOUAZNCIOTYOVOF22PE7VQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:53584->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150958689 (26h20m22.931892056s) [noise=true , address=104.36.82.12:4167 , piece_id=Q6L6CXWCD2OC3SOJDVMRV4LUAHEGRCM7VR4E7HRDXGB4EWSBS56Q, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 159.223.143.39:44036->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.150999721 (26h20m22.93193309s) [noise=true , address=104.36.82.53:4137 , piece_id=7ZMSCY4TDG7F5CDDOR7DQTVJV7ZWXEJFF3CJCHB62KSN5MKM2RTQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1dZZBaTDA1kbdJmNZzzYgdPkKE2rNvEqdWaMGUxuTD2ERbSanc, address:104.36.82.53:4137): protocol: write tcp 159.223.143.39:58126->104.36.82.53:4137: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151244168 (26h20m22.932177535s) [noise=true , address=104.36.82.49:4001 , piece_id=L2ZTVGVFZVVXOYUMIAGLP4G5WSX7CGI6RF3ZCPYNFADPPRALOLQA, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1iZnfkAxoot7KjtjYbxbSFvDhDFcLd8M63Qkh67GuMXMJjdruD, address:104.36.82.49:4001): protocol: write tcp 159.223.143.39:37978->104.36.82.49:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151324479 (26h20m22.932257846s) [noise=true , address=104.36.82.52:4120 , piece_id=VQ5UDX5WFG6MMVPPJG5LLGG56KV4PN3ORVK5CKZPFVOT2SGPSNHQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:57694->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151394793 (26h20m22.932328159s) [noise=true , address=104.36.82.52:4098 , piece_id=ULG6O4MSC2FM33V5CBFU5SE3WJUCX2IR7V3VOKYNDDXRPPZVGB5Q, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1SoFNdwfh4p5tXUkpfjQaev6tdA2ZdhsXUFvZNBsy5QrJ9zdqb, address:104.36.82.52:4098): protocol: write tcp 159.223.143.39:45430->104.36.82.52:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151447741 (26h20m22.932381106s) [noise=true , address=104.36.82.10:4089 , piece_id=TGHXQ3KH24RH6FVIXIQFIL4BYO33F6ESRMCVX7UVHSU4HTWFJXNQ, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:42254->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151508497 (26h20m22.932441864s) [noise=true , address=104.36.82.44:4904 , piece_id=7U6XIARKSQJZLGVWBBJFAH6PSDDFAT52JVRRTTJ3ZNSYKGXZAT2Q, seg_pos={0 0}, upload=25127]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:104.36.82.44:4904): protocol: write tcp 159.223.143.39:58054->104.36.82.44:4904: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:07.151545909 (26h20m22.932479285s) [seg_pos={0 0}, upload=25127]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:12:07.182193783 (26h20m22.963127162s) [upload=25127]: Done uploading segments... Jul 28 00:12:07.182223279 (26h20m22.963156657s) [upload=25127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.036100642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 346.249556ms FAIL (2025-07-28 00:11:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 346.283179ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXX.XXX] duration: 1m0.033918169s FAIL (2025-07-28 00:11:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== _id=NC7GTHCIONNKDCIONWFGZX72CJ2SH3PTWUR4UNWFKDNXGZXBJFNA, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.113861490 (26h18m22.894794876s) [noise=true , address=104.36.82.55:4221 , piece_id=27XVA4SQYSG7UUQSMW3MXUNGPJS5LSFSWCVLXXPVC7R74WHKTVFQ, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.114434517 (26h18m22.895367907s) [noise=true , address=104.36.82.12:4166 , piece_id=ML5QZRNWYH56WHWWIBJSUAWRLJVOWXSEKCNT273QLBI5GRYDVFUQ, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.114981744 (26h18m22.895915108s) [noise=true , address=64.49.13.140:4508 , piece_id=CCY7CX46PDO2BJE3XW6AEWOB34HAI5C4OWR25QH6W7FSBSAA46GQ, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.115545854 (26h18m22.896479243s) [noise=true , address=79.127.201.215:4359 , piece_id=JV5VWUDMKHV2DMD4SJLEVTL5X2NCXWR4AEIF5AEAZFVBUTOT3YNA, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.115729511 (26h18m22.896662894s) [noise=true , address=104.36.82.54:4170 , piece_id=FQ6HOCO2Y2AUGMKE33IFW2A24WW6FQK33YKZUWDAJJIOLAXOCIXA, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.116285576 (26h18m22.897218964s) [noise=true , address=104.36.82.46:4512 , piece_id=MOH66FTGOGMGQRW7VGJTFQPTVHAP4Q4DREKRDNO3WGOQNB6BNVRQ, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.119766963 (26h18m22.900700352s) [noise=true , address=104.36.82.51:4072 , piece_id=BUZDWOZRJKBXTGAD2NEARDKPMG66PIZBTUVQ2AQPAEDONHIIKOVA, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.120353166 (26h18m22.901286545s) [noise=true , address=104.36.82.16:4326 , piece_id=RFH6XMAGUNE5NKOPYZ6X2HX23K323LPBEZHCRVAKNFTAPZXEJF7A, seg_pos={0 0}, upload=25110]: Uploading piece... Jul 28 00:10:07.344004263 (26h18m23.124937625s) [upload=25110]: Next returned nil segment Jul 28 00:10:07.344029690 (26h18m23.124963079s) [upload=25110]: Waiting for error group managing segments... Jul 28 00:10:07.904773654 (26h18m23.685707028s) [noise=true , address=64.49.13.140:4508 , piece_id=CCY7CX46PDO2BJE3XW6AEWOB34HAI5C4OWR25QH6W7FSBSAA46GQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: <nil> Jul 28 00:10:08.059921348 (26h18m23.840854711s) [noise=true , address=79.127.201.215:4359 , piece_id=JV5VWUDMKHV2DMD4SJLEVTL5X2NCXWR4AEIF5AEAZFVBUTOT3YNA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: <nil> Jul 28 00:11:07.031471503 (26h19m22.812404886s) [noise=true , address=104.36.82.51:4072 , piece_id=BUZDWOZRJKBXTGAD2NEARDKPMG66PIZBTUVQ2AQPAEDONHIIKOVA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne, address:104.36.82.51:4072): protocol: rpc: dial tcp 104.36.82.51:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031570096 (26h19m22.812503477s) [noise=true , address=104.36.82.16:4326 , piece_id=RFH6XMAGUNE5NKOPYZ6X2HX23K323LPBEZHCRVAKNFTAPZXEJF7A, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12pU8MZZKS88CPFdZoaQ4msNfkCbCewTmS2wKEdN7pG5aCWoAr6, address:104.36.82.16:4326): protocol: rpc: dial tcp 104.36.82.16:4326: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031677617 (26h19m22.812610995s) [noise=true , address=104.36.82.55:4228 , piece_id=CZ3EIL7NXPHIB6CEJQRVZW3IRC3BK2MFYMYFXNZLEILNL3TAM7VA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: rpc: dial tcp 104.36.82.55:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031742479 (26h19m22.812675856s) [noise=true , address=104.36.82.49:4977 , piece_id=3JSD562HGVBIGRKS6NINOL4Q5Y2RHIDWQH66B6AWKAHIUH5ONAZQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:104.36.82.49:4977): protocol: rpc: dial tcp 104.36.82.49:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031807561 (26h19m22.812740943s) [noise=true , address=104.36.82.53:4143 , piece_id=XEO2IJS4BLCLOHSNSM77EQRKNNA3AQ7YO2KKAQUGU2DZP2FUFXEA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031867891 (26h19m22.812801268s) [noise=true , address=104.36.82.55:4221 , piece_id=27XVA4SQYSG7UUQSMW3MXUNGPJS5LSFSWCVLXXPVC7R74WHKTVFQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: rpc: dial tcp 104.36.82.55:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.031944403 (26h19m22.812877772s) [noise=true , address=104.36.82.15:4271 , piece_id=5ANLKMZF2AEDJ5A4PKVUCYPAIWOVF5CO3GYG7VNKPINZNIAYQDJA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu, address:104.36.82.15:4271): protocol: write tcp 159.223.143.39:39276->104.36.82.15:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032027827 (26h19m22.812961208s) [noise=true , address=104.36.82.14:4251 , piece_id=NC7GTHCIONNKDCIONWFGZX72CJ2SH3PTWUR4UNWFKDNXGZXBJFNA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: write tcp 159.223.143.39:44966->104.36.82.14:4251: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032097573 (26h19m22.813030951s) [noise=true , address=104.36.82.50:4015 , piece_id=YCUSL2L3NYQF23RINQ4QVAAAAAENFBSOTESQK5EYTXKOSKD4QYWQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:104.36.82.50:4015): protocol: rpc: dial tcp 104.36.82.50:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032388652 (26h19m22.813322032s) [noise=true , address=104.36.82.5:4852 , piece_id=XSC7BB2GS2R465AJ64LKGA7ITQYTKCRO3BIYNSOEYFK42KOPUPBQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12sQvAZS92R7TTJUFjMRieYT6BP4vp2UAan4PHrhPt6m3eHPukN, address:104.36.82.5:4852): protocol: rpc: dial tcp 104.36.82.5:4852: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032456608 (26h19m22.813389971s) [noise=true , address=104.36.82.52:4112 , piece_id=ZLHKEH4H66KHWL4NAWJ7GN3PWG3RBHNJZDCVANKW23XTYNJ53TFA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032499576 (26h19m22.813432953s) [noise=true , address=104.36.82.46:4512 , piece_id=MOH66FTGOGMGQRW7VGJTFQPTVHAP4Q4DREKRDNO3WGOQNB6BNVRQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: rpc: dial tcp 104.36.82.46:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032542772 (26h19m22.81347616s) [noise=true , address=104.36.82.13:4198 , piece_id=EJ5YRH3MDUCEXZYH64QKGMV26RNPUR6DUIVLQYOPL76W4EAG5A3Q, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm, address:104.36.82.13:4198): protocol: write tcp 159.223.143.39:36776->104.36.82.13:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032586282 (26h19m22.813519663s) [noise=true , address=104.36.82.12:4166 , piece_id=ML5QZRNWYH56WHWWIBJSUAWRLJVOWXSEKCNT273QLBI5GRYDVFUQ, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: write tcp 159.223.143.39:54766->104.36.82.12:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032596691 (26h19m22.813530072s) [noise=true , address=104.36.82.54:4170 , piece_id=FQ6HOCO2Y2AUGMKE33IFW2A24WW6FQK33YKZUWDAJJIOLAXOCIXA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: rpc: dial tcp 104.36.82.54:4170: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032671862 (26h19m22.813605242s) [noise=true , address=104.36.82.42:4831 , piece_id=5AK4Z2O3HM7O3CGOUKCVFZ5ZXHBVYASN242U4Y3PAIFND3CSYMGA, seg_pos={0 0}, upload=25110]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: rpc: dial tcp 104.36.82.42:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:07.032728311 (26h19m22.813661695s) [seg_pos={0 0}, upload=25110]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:11:07.061820489 (26h19m22.842753865s) [upload=25110]: Done uploading segments... Jul 28 00:11:07.061846430 (26h19m22.842779807s) [upload=25110]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.033952597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 390.587824ms FAIL (2025-07-28 00:10:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 390.629355ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXX.XX] duration: 1m0.029113552s FAIL (2025-07-28 00:09:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== EFAY3MBQA, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.755601121 (26h17m12.536534503s) [noise=true , address=104.36.82.9:4020 , piece_id=RAQPJMS2CPLJXE7X3KYNRM6NRR4OHJCJF7WMM6OT7HRO5RPZ3W4Q, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.755930616 (26h17m12.536863997s) [noise=true , address=104.36.82.53:4149 , piece_id=EWRYGOBKE456AWWWQJNR7NTUGHO6MLJ66RBYD2VRMO3LFX2PWSUA, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.755956043 (26h17m12.536889424s) [noise=true , address=104.36.82.40:4961 , piece_id=DFI6BPZNX7QI6PJJJNYJPF6NVTZSGCCSM3SQG3Z2ZQHHPQPAECTQ, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.756466667 (26h17m12.537400052s) [noise=true , address=104.36.82.41:4766 , piece_id=Z7QD5VZFYBW3EBBZPTJ4JI72XQSRVRAGRJP7PGWUNSBHYYXBWOGA, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.756621948 (26h17m12.537555327s) [noise=true , address=104.36.82.10:4088 , piece_id=HWUY6AYKVUSK3YAG5AMNXU75YICROA3QJ6CGU5KWEGZKC43SNFNQ, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.757097702 (26h17m12.538031084s) [noise=true , address=104.36.82.22:4543 , piece_id=LMULETYDL25GGECETR7BEEWKUQ6N2VJQY6ANXROQT3R3YIWMMXRQ, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.758444663 (26h17m12.539378029s) [noise=true , address=104.36.82.15:4269 , piece_id=CP57H4KDAY5OGPYA3BAUJKLGUXRYE6MONV3QUDZLJ27BVH56YT5Q, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.760808951 (26h17m12.541742332s) [noise=true , address=104.36.82.36:4637 , piece_id=NNM2UYCFQ33RTIKKBQI33MAICSI6IGYSO5TE62X72YJQNEPFHQKA, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.762122343 (26h17m12.543055745s) [noise=true , address=77.90.31.10:4550 , piece_id=A6GUM4OQVM2CQBNOIRZQO7T3NAZJPGCDZRBFL5QONLA6GQKXBEYQ, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.763478998 (26h17m12.544412383s) [noise=true , address=104.36.82.47:4520 , piece_id=GHZKFDDKGWPRR6JFIGHE25AFHTJ6K4GPZHAWYINLEJDGCS76NMZQ, seg_pos={0 0}, upload=25094]: Uploading piece... Jul 28 00:08:56.833052604 (26h17m12.613985985s) [upload=25094]: Next returned nil segment Jul 28 00:08:56.833070501 (26h17m12.614003879s) [upload=25094]: Waiting for error group managing segments... Jul 28 00:08:56.847777761 (26h17m12.628711144s) [noise=true , address=77.90.31.10:4550 , piece_id=A6GUM4OQVM2CQBNOIRZQO7T3NAZJPGCDZRBFL5QONLA6GQKXBEYQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: <nil> Jul 28 00:08:56.890174982 (26h17m12.671108347s) [noise=true , address=79.127.219.38:4015 , piece_id=RQWP35OM7MIINZM2AEESJFVE3MWVO4VBYN2B5MVZSOJ34RKBGAIQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: <nil> Jul 28 00:08:57.119123514 (26h17m12.900056882s) [noise=true , address=79.127.201.215:4357 , piece_id=23PSCHERMHTKPKIFGYB5HENLZ7QTIT33NG5CXAVATAFCWP26BWSQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: <nil> Jul 28 00:08:57.834622991 (26h17m13.615556363s) [noise=true , address=64.49.13.138:4483 , piece_id=NSK6UUKSM3KYDYNFD2BQ2IS2BWV2A5G3LZJMUYBL7W2NX2HUKAGA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: <nil> Jul 28 00:09:42.170769316 (26h17m57.951702688s) [noise=true , address=104.36.82.36:4637 , piece_id=NNM2UYCFQ33RTIKKBQI33MAICSI6IGYSO5TE62X72YJQNEPFHQKA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: <nil> Jul 28 00:09:56.683654842 (26h18m12.464588209s) [noise=true , address=104.36.82.54:4177 , piece_id=TVUHZERMFYGDRCA6CIZFX5JCUWE55IZQOAC377BBER4EFAY3MBQA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:48384->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.683750837 (26h18m12.464684204s) [noise=true , address=104.36.82.10:4088 , piece_id=HWUY6AYKVUSK3YAG5AMNXU75YICROA3QJ6CGU5KWEGZKC43SNFNQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:104.36.82.10:4088): protocol: rpc: dial tcp 104.36.82.10:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.683809491 (26h18m12.46474286s) [noise=true , address=104.36.82.22:4543 , piece_id=LMULETYDL25GGECETR7BEEWKUQ6N2VJQY6ANXROQT3R3YIWMMXRQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:19iUYyo8cGDLGDnvdNo2Qfj1K6tp6EJWF5U6rXD4gJ7Hv7Gsch, address:104.36.82.22:4543): protocol: write tcp 159.223.143.39:50272->104.36.82.22:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.683869039 (26h18m12.464802404s) [noise=true , address=104.36.82.44:4887 , piece_id=RCGSBE2AXABAW4Q75RYFA6RW3XDI4UUSROJ3GQE43AKVRQ3SAXSA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 159.223.143.39:44434->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.683921230 (26h18m12.464854595s) [noise=true , address=104.36.82.3:4796 , piece_id=G6K6YOWW2B5BHT6P5553VCJZJ22JTVQLCNHN7RXFKKJ3PDOD3QVQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: write tcp 159.223.143.39:55548->104.36.82.3:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.683958766 (26h18m12.464892141s) [noise=true , address=104.36.82.41:4766 , piece_id=Z7QD5VZFYBW3EBBZPTJ4JI72XQSRVRAGRJP7PGWUNSBHYYXBWOGA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: write tcp 159.223.143.39:38280->104.36.82.41:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684011054 (26h18m12.464944419s) [noise=true , address=104.36.82.55:4198 , piece_id=W553JQTZUK5UIQWMT5G45I2ZGIVA2SJM4OE54VWOQ3KY5UT4EFXQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:37892->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684062968 (26h18m12.464996334s) [noise=true , address=104.36.82.9:4020 , piece_id=RAQPJMS2CPLJXE7X3KYNRM6NRR4OHJCJF7WMM6OT7HRO5RPZ3W4Q, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12XXPyMaA6o5DFhMTkkWf14bdqZoZ86MNjxzfSqvNU2KQj8xSyv, address:104.36.82.9:4020): protocol: write tcp 159.223.143.39:44466->104.36.82.9:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684096594 (26h18m12.465029963s) [noise=true , address=104.36.82.15:4269 , piece_id=CP57H4KDAY5OGPYA3BAUJKLGUXRYE6MONV3QUDZLJ27BVH56YT5Q, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: write tcp 159.223.143.39:60580->104.36.82.15:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684123660 (26h18m12.465057037s) [noise=true , address=104.36.82.53:4149 , piece_id=EWRYGOBKE456AWWWQJNR7NTUGHO6MLJ66RBYD2VRMO3LFX2PWSUA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: write tcp 159.223.143.39:58884->104.36.82.53:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684352514 (26h18m12.465285897s) [noise=true , address=104.36.82.47:4520 , piece_id=GHZKFDDKGWPRR6JFIGHE25AFHTJ6K4GPZHAWYINLEJDGCS76NMZQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: write tcp 159.223.143.39:36770->104.36.82.47:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684368598 (26h18m12.465301965s) [noise=true , address=104.36.82.40:4961 , piece_id=DFI6BPZNX7QI6PJJJNYJPF6NVTZSGCCSM3SQG3Z2ZQHHPQPAECTQ, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:104.36.82.40:4961): protocol: write tcp 159.223.143.39:39618->104.36.82.40:4961: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684433593 (26h18m12.465366969s) [noise=true , address=104.36.82.6:4893 , piece_id=PKXBYEFVAMDGE4PIUPODI3SZOY6Y5WYDKMPGVPHVWHQ74L2YGGLA, seg_pos={0 0}, upload=25094]: Done uploading piece. err: ecclient: upload failed (node:1259j7aHsLDMPtRwgXpd6u8axPEuaXzhRzqYYSgbpDnDqbhfwzi, address:104.36.82.6:4893): protocol: write tcp 159.223.143.39:52382->104.36.82.6:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:09:56.684466442 (26h18m12.465399807s) [seg_pos={0 0}, upload=25094]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 28 00:09:56.710126911 (26h18m12.49106028s) [upload=25094]: Done uploading segments... Jul 28 00:09:56.710175430 (26h18m12.491108799s) [upload=25094]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029152767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 510.691289ms FAIL (2025-07-28 00:09:08 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 510.724489ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)